Tumblelog by Soup.io
Newer posts are loading.
You are at the newest post.
Click here to check if anything new just came in.

February 10 2017

23:33

Migrating from owncloud 9.1 to nextcloud 11

First one should ask though: why? My main motivation was that many of the apps I use were easily available in the nextcloud store, while with owncloud I had to manually pull them from github.
Additionally some of the app authors migrated to nextcloud and did not provide further updates for owncloud.

Another reason is this:

the graphs above show the number of commits for owncloud and nextcloud. Owncloud has taken a very noticeable hit here after the fork – even though they deny it.

From the user perspective the lack of contribution is visible for instance in the admin interface where with nextcloud you get a nice log browser and system stats while with owncloud you do not. Furthermore the nextcloud android app handles Auto-Upload much better and generally seems more polished – I think one can expect nextcloud to advance faster in general.

Migrating

For migrating you can follow the excellent instructions of Jos Poortvliet.

In my case owncloud 9.1 was installed on Ubuntu in /var/www/owncloud and I put nextcloud 11 to /var/www/nextcloud. Then the following steps had to be applied:

  1. put owncloud in maintenance mode
    sudo -u www-data php occ maintenance:mode --on
  2. copy over the config.php
    cp /var/www/owncloud/config/config.php /var/www/nextcloud/config/
  3. adapt the path in config.php
    # from 
    'path' => '/var/www/owncloud/apps',
    # to
    'path' => '/var/www/nextcloud/apps',
  4. adapt the path in crontab
    sudo crontab -u www-data -e
  5. adapt the paths in the apache config
  6. run the upgrade script which takes care of the actual migration. Then disable the maintanance mode.
    sudo -u www-data php occ upgrade
    sudo -u www-data php occ maintenance:mode --off

and thats it.

0 Add to favourites0 Bury
00:00

Working on an Android tablet, 2017 edition

Back in 2013 I was working exclusively on an Android tablet. Then with the NoFlo Kickstarter I needed a device with a desktop browser. What followed were brief periods working on a Chromebook, on a 12” MacBook, and even an iPad Pro.

But from April 2016 onwards I’ve been again working with an Android device. Some people have asked me about my setup, and so here is an update.

Information technology

Why work on a tablet?

When I started on this path in 2013, using a tablet for “real work” was considered crazy. While every story on tablet productivity still brings out the people claiming it is not a real computer for real work, using tablets for real work is becoming more and more common.

A big contributor to this has been the plethora of work-oriented tablets and convertibles released since then. Microsoft’s popular Surface Pro line brought the PC to tablet form factor, and Apple’s iPad Pro devices gave the iPad a keyboard.

Here are couple of great posts talking about how it feels to work on an iPad:

With all the activity going on, one could claim using a tablet for work has been normalized. But why work on a tablet instead of a “real computer”? Here are some reasons, at least for me:

Free of legacy cruft

Desktop operating systems have become clunky. Window management. File management. Multiple ways to discover, install, and uninstall applications. Broken notification mechanisms.

With a tablet you can bypass pretty much all of that, and jump into a simpler, cleaner interface designed for the modern connected world.

I think this is also the reason driving some developers back to Linux and tiling window managers — cutting manual tweaking and staying focused.

Amazing endurance

Admittedly, laptop battery life has increased a lot since 2013. But with some manufacturers using this an excuse to ship thinner devices, tablets still win the endurance game.

With my current work tablet, I’m customarily getting 12 or more hours of usage. This means I can power through the typical long days of a startup founder without having to plug in. And when traveling, I really don’t have to care where power sockets are located on trains, airplanes, and conference centers.

Low power usage also means that I can really get a lot of more runtime by utilizing the mobile battery pack I originally bought to use with my phone. While I’ve never actually had to try this, back-of-the-envelope math claims I should be able to get a full workweek from the combo without plugging in.

Work and play

The other aspect of using a tablet is that it becomes a very nice content consumption device after I’m done working. Simply disconnect the keyboard and lean back, and the same device you used for writing software becomes a great e-reader, video player, or a gaming machine.

Livestreaming a SpaceX launch

This combined with the battery life has meant that I’ve actually stopped carrying a Kindle with me. While an e-ink screen is still nicer to read, not needing an extra device has its benefits, especially for a frequent one-bag traveller.

The setup

I’m writing this on a Pixel C, a 10.2” Android tablet made by Google. I got the device last spring when there were developer discounts available at ramp-up to the Android 7 release, and have been using it full-time since.

Software

My Android homescreen

Surprisingly little has changed in my software use since 2013 — I still spend the most of the time writing software in either Flowhub or terminal. Here are the apps I use on daily basis:

Looking back to the situation in early 2013, the biggest change is that Slack has pretty much killed work email.

Termux is a new app that has done a lot to improve the local development situation. By starting the app you get a very nice Linux chroot environment where a lot of software is only a quick apt install away.

Since much of my non-Flowhub work is done in tmux and vim, I get the exactly same working environment on both local chroot and cloud machines by simply installing my dotfiles on each of them.

Keyboard

Laptop tablet

When I’m on the road I’m using the Pixel C keyboard. This doubles as a screen protector, and provides a reasonable laptop-like typing environment. It attaches to the tablet with very strong magnets and allows a good amount of flexibility on the screen angles.

However, when stationary, no laptop keyboard compares to a real mechanical keyboard. When I’m in the office I use a Filco MiniLa Air, a bluetooth keyboard with quiet-ish Cherry MX brown switches.

Desktop tablet

This tenkeyless (60%) keyboard is extremely comfortable to type on. However, the sturdy metal case means that it is a little too big and heavy to carry on a daily basis.

In practice I’ve only taken with me when there has been a longer trip where I know that I’ll be doing a lot of typing. To solve this, I’m actually looking to build a more compact custom mechanical keyboard so I could always have it with me.

Comparison with iOS

So, why work on Android instead of getting an iPad Pro? I’ve actually worked on both, and here are my reasons:

  • Communications between apps: while iOS has extensions now, the ability to send data from an app to another is still a hit-or-miss. Android had intents from day one, meaning pretty much any app can talk to any other app
  • Standard charging: all of my other devices charge with the same USB-C chargers and cables. iPads still use the proprietary Lightnight plug, requiring custom dongles for everything
  • Standard accessories: this boils down to USB-C just like charging. With Android I can plug in a network adapter or even a mouse, and it’ll just work
  • Ecosystem lock-in: we’re moving to a world where everything — from household electronics to cars — is either locked to the Apple ecosystem or following standards. I don’t want to be locked to a single vendor for everything digital
  • Browser choice: with iOS you only get one web renderer, the rather dated Safari. On Android I can choose between Chrome, Firefox, or any other browser that has been ported to the platform

Of course, iOS has its own benefits. Apple has a stronger stance on privacy than Google. And there is more well-made tablet software available for iPads than Android. But when almost everything I use is available on the web, this doesn’t matter that much.

The future

Hacking on the c-base patio

As a software developer working on Android tablets, the weakest point of the platform is still that there are no browser developer tools available. This was a problem in 2013, and it is still a problem now.

From my conversations with some Chrome developers, it seems Google has very little interest in addressing this. However, there is a bright spot: the new breed of convertible Chromebooks being released now. And they run Android apps:

Chrome OS is another clean, legacy free, modern computing interface. With these new devices you get the combination of a full desktop browser and the ability to run all Android tablet software.

The Samsung Chromebook Pro/Plus mentioned above is definitely interesting. A high-res 12” screen and a digital pen which I see as something very promising for visual programming purposes.

However, given that I already have a great mechanical keyboard, I’d love a device that shipped without an attached keyboard. We’ll see what kind of devices get out later this year.

0 Add to favourites0 Bury

February 09 2017

20:07

How to expose a QList in a ViewModel to QML

MyPlugin/MyPlugin.cpp:

#include <ViewModels/MyListClass.h>
#include <ViewModels/DisplayViewModel.h>

qmlRegisterUncreatableType<MyListClass>( a_uri, 1, 0, "MyListClass",
         "Use access via DisplayViewModel instead");
qmlRegisterType<DisplayViewModel>( a_uri, 1, 0, "DisplayViewModel");

Utils/MyQMLListUtils.h

#define MY_DECLARE_QML_LIST(type, name, owner, prop) \
QQmlListProperty<type> name(){ \
   return QQmlListProperty<type>( \
               this, 0,&owner::count##typeList, \
               &owner::at##typeList); \
} \
static int count##typeList(QQmlListProperty<type>*property){ \
   owner *m = qobject_cast<owner *>(property->object); \
   return m->prop.size(); \
} \
static type *at##typeList( \
        QQmlListProperty<type>*property, int index){ \
   owner *m = qobject_cast<owner *>(property->object); \
   return m->prop[index]; \
}

ViewModels/DisplayViewModel.h

#ifndef DISPLAYVIEWMODEL_H
#define DISPLAYVIEWMODEL_H

#include <QObject>
#include <QtQml>
#include <ViewModels/MyListClass.h>
#include <Utils/MyQMLListUtils.h>

class DisplayViewModel : public QObject
{
    Q_OBJECT

    Q_PROPERTY(constQString title READ title WRITE setTitle NOTIFY titleChanged )
    Q_PROPERTY(constQList<MyListClass*> objects READ objects
                                          NOTIFY objectsChanged ) 
    Q_PROPERTY( QQmlListProperty<MyListClass> objectList READ objectList
                                              NOTIFY objectsChanged )
public:
    explicit DisplayViewModel( QObject *a_parent = nullptr );
    explicit DisplayViewModel( const QString &a_title,
                               QList<MyListClass*> a_objects,
                               QObject *a_parent = nullptr );
    const QString title()
        { return m_title; }
    void setTitle( const QString &a_title ); 
    const QList<MyListClass*> objects ()
        { return m_objects; } 
    Q_INVOKABLE void appendObject( MyListClass *a_object);
    Q_INVOKABLE void deleteObject( MyListClass *a_object);
    Q_INVOKABLE void reset( );

protected:
    MY_DECLARE_QML_LIST(MyListClass, objectList, DisplayViewModel, m_objects)

signals:
    void titleChanged();
    void objectsChanged();

private:
    QString m_title;
    QList<MyListObject*> m_objects;
};

#endif// DISPLAYVIEWMODEL_H

DisplayViewModel.cpp

#include "DisplayViewModel.h"

DisplayViewModel::DisplayViewModel( const QString &a_title,
                                    QList<MyListClass*> a_objects,
                                    QObject *a_parent )
    : QObject ( a_parent )
    , m_title ( a_title )
    , m_objects ( a_objects )
{
    foreach (MyListClass* mobject, m_objects) {
        mobject->setParent (this);
    }
}

void DisplayViewModel::setTitle (const QString &a_title )
{
    if ( m_title != a_title ) {
        m_title = a_title;
        emit titleChanged();
    }
}

void DisplayViewModel::reset( )
{
    foreach ( MyListClass *mobject, m_objects ) {
        mobject->deleteLater();
    }
    m_objects.clear();
    emit objectsChanged();
}

void DisplayViewModel::appendObject( MyListClass *a_object )
{
    a_object->setParent( this );
    m_objects.append( a_object );
    emit objectsChanged();
}

void DisplayViewModel::deleteObject( MyListClass *a_object )
{
    if (m_objects.contains( a_object )) {
        m_objects.removeOne( a_object );
        a_object->deleteLater();
        emit objectsChanged();
    }
}

Tester.cpp

#include <ViewModels/DisplayViewModel.h>
#include <ViewModels/MyListClass.h>

QList<MyListClass*> objectList;
for( int i = 0; i < 100 ; ++i ) {
    objectList.append ( new MyListClass (i) );
}
DisplayViewModel *viewModel = new DisplayViewModel (objectList);
viewModel->appendObject ( new MyListClass (101) );

Display.qml

import QtQuick 2.5
import MyPlugin 1.0

Repeater { 
    property DisplayViewModel viewModel: DisplayViewModel { } 
    model: viewModel.objectList
    delegate: Item {
        property MyListClass object: modelData
        Text {
            text: object.property
        }
    }
}
0 Add to favourites0 Bury

February 08 2017

08:52

QEMU and the qcow2 metadata checks

When choosing a disk image format for your virtual machine one of the factors to take into considerations is its I/O performance. In this post I’ll talk a bit about the internals of qcow2 and about one of the aspects that can affect its performance under QEMU: its consistency checks.

As you probably know, qcow2 is QEMU’s native file format. The first thing that I’d like to highlight is that this format is perfectly fine in most cases and its I/O performance is comparable to that of a raw file. When it isn’t, chances are that this is due to an insufficiently large L2 cache. In one of my previous blog posts I wrote about the qcow2 L2 cache and how to tune it, so if your virtual disk is too slow, you should go there first.

I also recommend Max Reitz and Kevin Wolf’s qcow2: why (not)? talk from KVM Forum 2015, where they talk about a lot of internal details and show some performance tests.

qcow2 clusters: data and metadata

A qcow2 file is organized into units of constant size called clusters. The cluster size defaults to 64KB, but a different value can be set when creating a new image:

qemu-img create -f qcow2 -o cluster_size=128K hd.qcow2 4G

Clusters can contain either data or metadata. A qcow2 file grows dynamically and only allocates space when it is actually needed, so apart from the header there’s no fixed location for any of the data and metadata clusters: they can appear mixed anywhere in the file.

Here’s an example of what it looks like internally:

In this example we can see the most important types of clusters that a qcow2 file can have:

  • Header: this one contains basic information such as the virtual size of the image, the version number, and pointers to where the rest of the metadata is located, among other things.
  • Data clusters: the data that the virtual machine sees.
  • L1 and L2 tables: a two-level structure that maps the virtual disk that the guest can see to the actual location of the data clusters in the qcow2 file.
  • Refcount table and blocks: a two-level structure with a reference count for each data cluster. Internal snapshots use this: a cluster with a reference count >= 2 means that it’s used by other snapshots, and therefore any modifications require a copy-on-write operation.

Metadata overlap checks

In order to detect corruption when writing to qcow2 images QEMU (since v1.7) performs several sanity checks. They verify that QEMU does not try to overwrite sections of the file that are already being used for metadata. If this happens, the image is marked as corrupted and further access is prevented.

Although in most cases these checks are innocuous, under certain scenarios they can have a negative impact on disk write performance. This depends a lot on the case, and I want to insist that in most scenarios it doesn’t have any effect. When it does, the general rule is that you’ll have more chances of noticing it if the storage backend is very fast or if the qcow2 image is very large.

In these cases, and if I/O performance is critical for you, you might want to consider tweaking the images a bit or disabling some of these checks, so let’s take a look at them. There are currently eight different checks. They’re named after the metadata sections that they check, and can be divided into the following categories:

  1. Checks that run in constant time. These are equally fast for all kinds of images and I don’t think they’re worth disabling.
    • main-header
    • active-l1
    • refcount-table
    • snapshot-table
  2. Checks that run in variable time but don’t need to read anything from disk.
    • refcount-block
    • active-l2
    • inactive-l1
  3. Checks that need to read data from disk. There is just one check here and it’s only needed if there are internal snapshots.
    • inactive-l2

By default all tests are enabled except for the last one (inactive-l2), because it needs to read data from disk.

Disabling the overlap checks

Tests can be disabled or enabled from the command line using the following syntax:

-drive file=hd.qcow2,overlap-check.inactive-l2=on
-drive file=hd.qcow2,overlap-check.snapshot-table=off

It’s also possible to select the group of checks that you want to enable using the following syntax:

-drive file=hd.qcow2,overlap-check.template=none
-drive file=hd.qcow2,overlap-check.template=constant
-drive file=hd.qcow2,overlap-check.template=cached
-drive file=hd.qcow2,overlap-check.template=all

Here, none means that no tests are enabled, constant enables all tests from group 1, cached enables all tests from groups 1 and 2, and all enables all of them.

As I explained in the previous section, if you’re worried about I/O performance then the checks that are probably worth evaluating are refcount-block, active-l2 and inactive-l1. I’m not counting inactive-l2 because it’s off by default. Let’s look at the other three:

  • inactive-l1: This is a variable length check because it depends on the number of internal snapshots in the qcow2 image. However its performance impact is likely to be negligible in all cases so I don’t think it’s worth bothering with.
  • active-l2: This check depends on the virtual size of the image, and on the percentage that has already been allocated. This check might have some impact if the image is very large (several hundred GBs or more). In that case one way to deal with it is to create an image with a larger cluster size. This also has the nice side effect of reducing the amount of memory needed for the L2 cache.
  • refcount-block: This check depends on the actual size of the qcow2 file and it’s independent from its virtual size. This check is relatively expensive even for small images, so if you notice performance problems chances are that they are due to this one. The good news is that we have been working on optimizing it, so if it’s slowing down your VMs the problem might go away completely in QEMU 2.9.

Conclusion

The qcow2 consistency checks are useful to detect data corruption, but they can affect write performance.

If you’re unsure and you want to check it quickly, open an image with overlap-check.template=none and see for yourself, but remember again that this will only affect write operations. To obtain more reliable results you should also open the image with cache=none in order to perform direct I/O and bypass the page cache. I’ve seen performance increases of 50% and more, but whether you’ll see them depends a lot on your setup. In many cases you won’t notice any difference.

I hope this post was useful to learn a bit more about the qcow2 format. There are other things that can help QEMU perform better, and I’ll probably come back to them in future posts, so stay tuned!

Acknowledgments

My work in QEMU is sponsored by Outscale and has been made possible by Igalia and the help of the rest of the QEMU development team.

0 Add to favourites0 Bury

January 05 2017

00:00

Process API for NoFlo components

It has been a while that I’ve written about flow-based programming — but now that I’m putting most of my time to Flowhub things are moving really quickly.

One example is the new component API in NoFlo that has been emerging over the last year or so.

Most of the work described here was done by Vladimir Sibirov from The Grid team.

Introducing the Process API

NoFlo programs consist of graphs where different nodes are connected together. These nodes can themselves be graphs, or they can be components written in JavaScript.

A NoFlo component is simply a JavaScript module that provides a certain interface that allows NoFlo to run it. In the early days there was little convention on how to write components, but over time some conventions emerged, and with them helpers to build well-behaved components more easily.

Now with the upcoming NoFlo 0.8 release we’ve taken the best ideas from those helpers and rolled them back into the noflo.Component base class.

So, how does a component written using the Process API look like?

// Load the NoFlo interface
var noflo = require('noflo');
// Also load any other dependencies you have
var fs = require('fs');

// Implement the getComponent function that NoFlo's component loader
// uses to instantiate components to the program
exports.getComponent = function () {
  // Start by instantiating a component
  var c = new noflo.Component();

  // Provide some metadata, including icon for visual editors
  c.description = 'Reads a file from the filesystem';
  c.icon = 'file';

  // Declare the ports you want your component to have, including
  // their data types
  c.inPorts.add('in', {
    datatype: 'string'
  });
  c.outPorts.add('out', {
    datatype: 'string'
  });
  c.outPorts.add('error', {
    datatype: 'object'
  });

  // Implement the processing function that gets called when the
  // inport buffers have packets available
  c.process(function (input, output) {
    // Precondition: check that the "in" port has a data packet.
    // Not necessary for single-inport components but added here
    // for the sake of demonstration
    if (!input.hasData('in')) {
      return;
    }

    // Since the preconditions matched, we can read from the inport
    // buffer and start processing
    var filePath = input.getData('in');
    fs.readFile(filePath, 'utf-8', (err, contents) {
      // In case of errors we can just pass the error to the "error"
      // outport
      if (err) {
        output.done(err);
        return;
      }

      // Send the file contents to the "out" port
      output.send({
        out: contents
      });
      // Tell NoFlo we've finished processing
      output.done();
    });
  });

  // Finally return to component to the loader
  return c;
}

Most of this is still the same component API we’ve had for quite a while: instantiation, component metadata, port declarations. What is new is the process function and that is what we’ll focus on.

When is process called?

NoFlo components call their processing function whenever they’ve received packets to any of their regular inports.

In general any new information packets received by the component cause the process function to trigger. However, there are some exceptions:

  • Non-triggering ports don’t cause the function to be called
  • Ports that have been set to forward brackets don’t cause the function to be called on bracket IPs, only on data

Handling preconditions

When the processing function is called, the first job is to determine if the component has received enough data to act. These “firing rules” can be used for checking things like:

  • When having multiple inports, do all of them contain data packets?
  • If multiple input packets are to be processed together, are all of them available?
  • If receiving a stream of packets is the complete stream available?
  • Any input synchronization needs in general

The NoFlo component input handler provides methods for checking the contents of the input buffer. Each of these return a boolean if the conditions are matched:

  • input.has('portname') whether an input buffer contains packets of any type
  • input.hasData('portname') whether an input buffer contains data packets
  • input.hasStream('portname') whether an input buffer contains at least one complete stream of packets

For convenience, has and hasData can be used to check multiple ports at the same time. For example:

// Fail precondition check unless both inports have a data packet
if (!input.hasData('in1', 'in2')) return;

For more complex checking it is also possible to pass a validation function to the has method. This function will get called for each information packet in the port(s) buffer:

// We want to process only when color is green
var validator = function (packet) {
  if (packet.data.color === 'green') {
    return true;
  }
  return false;
}
// Run all packets in in1 and in2 through the validator to
// check that our firing conditions are met
if (!input.has('in1', 'in2', validator)) return;

The firing rules should be checked in the beginning of the processing function before we start actually reading packets from the buffer. At that stage you can simply finish the run with a return.

Processing packets

Once your preconditions have been met, it is time to read packets from the buffers and start doing work with them.

For reading packets there are equivalent get functions to the has functions used above:

  • input.get('portname') read the first packet from the port’s buffer
  • input.getData('portname') read the first data packet, discarding preceding bracket IPs if any
  • input.getStream('portname') read a whole stream of packets from the port’s buffer

For get and getStream you receive whole IP objects. For convenience, getData returns just the data payload of the data packet.

When you have read the packets you want to work with, the next step is to do whatever your component is supposed to do. Do some simple data processing, call some remote API function, or whatever. NoFlo doesn’t really care whether this is done synchronously or asynchronously.

Note: once you read packets from an inport, the component activates. After this it is necessary to finish the process by calling output.done() when you’re done.

Sending packets

While the component is active, it can send packets to any number of outports using the output.send method. This method accepts a map of port names and information packets.

output.send({
  out1: new noflo.IP('data', "some data"),
  out2: new noflo.IP('data', [1, 2, 3])
});

For data packets you can also just send the data as-is, and NoFlo will wrap it to an information packet.

Once you’ve finished processing, simply call output.done() to deactivate the component. There is also a convenience method that is a combination of send and done. This is useful for simple components:

c.process(function (input, output) {
  var data = input.getData('in');
  // We just add one to the number we received and send it out
  output.sendDone({
    out: data + 1
  });
});

In normal situations there packets are transmitted immediately. However, when working on individual packets that are part of a stream, NoFlo components keep an output buffer to ensure that packets from the stream are transmitted in original order.

Component lifecycle

In addition to making input processing easier, the other big aspect of the Process API is to help formalize NoFlo’s component and program lifecycle.

NoFlo program lifecycle

The component lifecycle is quite similar to the program lifecycle shown above. There are three states:

  • Initialized: the component has been instantiated in a NoFlo graph
  • Activated: the component has read some data from inport buffers and is processing it
  • Deactivated: all processing has finished

Once all components in a NoFlo network have deactivated, the whole program is finished.

Components are only allowed to do work and send packets when they’re activated. They shouldn’t do any work before receiving input packets, and should not send anything after deactivating.

Generator components

Regular NoFlo components only send data associated with input packets they’ve received. One exception is generators, a class of components that can send packets whenever something happens.

Some examples of generators include:

  • Network servers that listen to requests
  • Components that wait for user input like mouse clicks or text entry
  • Timer loops

The same rules of “only send when activated” apply also to generators. However, they can utilize the processing context to self-activate as needed:

exports.getComponent = function () {
 var c = new noflo.Component();
 c.inPorts.add('start', { datatype: 'bang' });
 c.inPorts.add('stop', { datatype: 'bang' });
 c.outPorts.add('out', { datatype: 'bang' });
 // Generators generally want to send data immediately and
 // not buffer
 c.autoOrdering = false;

 // Helper function for clearing a running timer loop
 var cleanup = function () {
   // Clear the timer
   clearInterval(c.timer.interval);
   // Then deactivate the long-running context
   c.timer.deactivate();
   c.timer = null;
 }

 // Receive the context together with input and output
 c.process(function (input, output, context) {
   if (input.hasData('start')) {
     // We've received a packet to the "start" port
     // Stop the previous interval and deactivate it, if any
     if (c.timer) {
       cleanup();
     }
     // Activate the context by reading the packet
     input.getData('start');
     // Set the activated context to component so it can
     // be deactivated from the outside
     c.timer = context
     // Start generating packets
     c.timer.interval = setInterval(function () {
       // Send a packet
       output.send({
         out: true
       });
     }, 100);
     // Since we keep the generator running we don't
     // call done here
   }

   if (input.hasData('stop')) {
     // We've received a packet to the "stop" port
     input.getData('stop');
     if (!c.timer) {
       // No timers running, we can just finish here
       output.done();
       return;
     }
     // Stop the interval and deactivate
     cleanup();
     // Also call done for this one
     output.done();
   }
 });

 // We also may need to clear the timer at network shutdown
 c.shutdown = function () {
   if (c.timer) {
     // Stop the interval and deactivate
     cleanup();
   }
   c.emit('end');
   c.started = false;
 }
}

Time to prepare

NoFlo 0.7 included a preview version of the Process API. However, last week during the 33C3 conference we finished some tricky bits related to process lifecycle and automatic bracket forwarding that make it more useful for real-life NoFlo applications.

These improvements will land in NoFlo 0.8, due out soon.

So, if you’re maintaining a NoFlo application, now is a good time to give the git version a spin and look at porting your components to the new API. Make sure to report any issues you encounter!

We’re currently migrating all the hundred-plus NoFlo open source modules to latest build and testing process so that they can be easily updated to the new APIs when they land.

1 Add to favourites0 Bury

December 22 2016

01:26

OGRECave 1.10 release

The 1.10.0 release of the OGRECave fork was just created. This means that the code is considered stable enough for general usage and the current interfaces will be supported in subsequent patch releases (i.e. 1.10.1, 1.10.2 …).

SampleBrowser running GLES2 on desktop

This release represents more than 3 years of work from various contributors when compared to the previous 1.9 release. At the time of writing it contains all commits from the bitbucket version as well as many fork specific features and fixes.

If you are reading about the fork for the first time and wonder why it was created, see this blog post. For a comparison between the github and bitbucket version see this log.

For a general overview of the 1.10 features when compared to 1.9, see the OGRECave 1.10 release notes.

The highlights probably are:

  • upstream Python bindings as an component
  • improved GL3+/ GLES2 renderers
  • A new HLMS Component implementing physically based shading
  • SDL2 based input handling
  • Bites Component for rapid prototyping of applications
  • Emscripten platform support

For further information see the github page of the fork.

0 Add to favourites0 Bury

November 28 2016

21:57
Nokia and their standard batteries

October 28 2016

11:41

The 1st Maemo Developer Regatta - The starting line

The Maemo Community e.V. proudly presents

The 1st Maemo Developer Regatta - The starting line

sponsored by Jolla


The wait is over!

The first Maemo Coding Competition for all Maemo derived systems, including but not limited to Maemo and SailfishOS.

This year's competition has the following three categories: Something new, Fixing/Updating and Beginner. Whether you are an experienced developer, porter, hacker or just a beginner on your very first hacks/codes, we have a category that suits you and your coding skills! If you are new to programming or want to learn, here's the excuse to finally make something. Anything. Entries for Maemo & Mer-based devices are eligible.

The timetable for the competition is the following: The competitions begins on Tuesday, 1st of November, and will be running for 3 months until Tuesday 31st of January 2016. You can enroll into the competition at any time, however, all the entries for the competition must be submitted no later than by 23:59 on Tuesday, 31st of January(UTC time). Testing/reviewing will take two weeks, from 01.02.2017 to 15.02.2017 and voting will run from 16th of February until 28th of February 2017, 23:59 UTC.

Follow the links to the wiki below for more information.

Devices & Platforms
Categories
How to Submit an Entry
Prizes
Participants
Rules
Voting
Developer Resources

If you would like to have an App, a Port, a Fix, a Hack or etc. but you can't code it yourself, don't worry, just write your ideas in here! Many developers would be happy to get a new idea input.

We are relying on your donations towards the Community Prize Fund.

The entry is free of charge.

Thanks for your interest in taking part, we can't wait to see what you will develop! :)

You still have questions? Contact us!

  • council<*at*>maemo<*dot*>org
  • This thread

Regards

Maemo Community e.V.

0 Add to favourites0 Bury

September 11 2016

17:35

Maemo Community e.V. - new Invitation to the General Assembly 2016

Maemo Community e.V. - Einladung zur Mitgliederversammlung 2016

Sehr geehrtes Mitglied,

aufgrund zu geringer Teilnehmerzahl und daraus resultierender Beschlussunfähigkeit, musste die letzte ordentliche Mitgliederversammlung am 04.09.2016 vorzeitig beendet werden.
Die Mitgliederversammlung findet nun am 09.10.2016 14:00 UTC im IRC Freenode Channel #maemo-meeting statt.
*Bitte sorgt dafür, dass ihr diesmal anwesend seid.*

Auf der Tagesordnung stehen folgende Themen:
1.    Begrüßung durch die Vorsitzende des Vorstands
2.    Feststellung der ordnungsgemäßen Einberufung und der Beschlussfähigkeit der Mitgliederversammlung
3.    Entgegennahme des Jahresberichts für das abgelaufene Geschäftsjahr und Entlastung des Vorstands
4.    Neuwahl des Vorstandes
5.    Änderung der Satzung §7 und §8, und der Vereinsordnung §1.1, §3.2 und §4.1. Erläuterung im Anhang [1].
6.    Verschiedenes

Anträge auf Ergänzungen der Tagesordnung müssen bis eine Woche vor der Versammlung schriftlich beim Vorstand eingereicht werden (§ 9 Abs. 2 der Satzung).

Mit freundlichen Grüßen
Win7Mac/Gido Griese


########################## English ##########################

Maemo Community e.V. - Invitation to the General Assembly 2016

Dear Member,

due to too low number of participants and the resulting lack of a quorum, our last regular General Assembly on 2016-09-04 had to be ceased prematurely.
The General Meeting will now be held on Sunday, 2016-10-09 at 14:00 UTC on IRC Freenode channel #maemo-meeting.
*Please make sure to join the meeting this time.*

Unless any further issues are raised, the agenda includes the following topics:
1.    Welcome by the Chairman of the Board
2.    Determination of the proper convocation and the quorum of the General Assembly
3.    Acceptance of the annual report for the fiscal year and actions of the Executive
4.    Election of the Board Directors
5.    Amendment of Articles of Association §7 and §8, Association Rules §1.1, §3.2 and §4.1 - please see appendix for details [1].
6.    Any other business

Requests for additions to the agenda must be submitted to the Board in writing one week prior to the meeting (§ 9.2 of the Statutes).

Yours sincerely
Win7Mac/Gido Griese


[1] http://maemo.org/community/council/referendum_2015

Anhang / APPENDIX

** Änderungen der Satzung / Amendments to the Articles of Association **

§7 Vorstand (Board of directors)
 (5) Der Vorstand führt die Beschlüsse der aktiven Mitgliederversammlung und der des Vorstandsrats aus, sofern dem Vorstandsrat diese Entscheidungshoheit laut Satzung oder Vereinsordnung eingeräumt wird.

 (5) The Board of Directors carries out the resolutions of the General Assembly and of the Council in so far as this decision's jurisdiction is delegated to Council according to the statutes.

§8 Vorstandsrat (Council)
 (4) Aufgaben des Vorstandsrats sowie Regelungen zu dessen Arbeit, inklusive der Einberufung von Sitzungen, deren Ablauf und die Durchführung von Abstimmungen, sind in der Vereinsordnung festgelegt.

 (4) The council's duties, rules to announce meetings, their proceedings and executions of votes are regulated by the Association Rules.


** Änderungen der Vereinsordnung / Amendments to the Association Rules **

§ 1.1
  (2) § 4.1 (Vorstandsratswahlen) dieser Vereinsordnung kann durch ein Referendum der passiven Mitgliederversammlung geändert oder aufgehoben werden. Änderungen dieser Wahlordnung durch die aktive Mitgliederversammlung müssen durch Beschluss der passiven Mitgliederversammlung bestätigt werden (Referendum). Sollte die Änderung von der passiven Mitgliederversammlung abgelehnt werden, kann der Änderungsvorschlag überarbeitet und erneut zur Abstimmung gebracht werden.

  (2) § 4.1 (Council/election rules) of these association rules can be changed or dismissed with a referendum by the Passive Members' Meeting. Changes to these election rules by the General Assembly must be confirmed by a resolution of the passive members meeting (referendum). Should the amendment be rejected by the passive members meeting, the proposed amendment can be revised and brought again to the vote.


§3.2 Vorstandsrat /(council)

 (7) Aufgaben und Zuständigkeiten des Vorstandsrats sind wie folgt:
 a) Vertretung der assoziierten Mitglieder und ihrer Interessen
 b) Beratung des Vorstands in allgemeinen Belangen
 c) Der Vorstandsrat ist für die Durchführung aller Wahlen zuständig

 (7) Tasks and competences of the Council are as follows:
 a) Representation of the associated members (garage account) and their interests.
 b) Counseling of the Board of directors in general matters
 c) Council is responsible to execute all elections


§ 4.1 Vorstandsratswahlen / Council Elections

(6) Kandidaten mit einem professionellen bzw. witschaftlichen Interesse in Maemo, z.B. Mitarbeiter von Firmen, die in Bezug zu Maemo Hard- oder Software stehen, müssen ihr Interesse bei der Nominierung erklären. Bleibt dies aus, kann der Vorstandsrat die Kandidatur des Mitglieds für ungültig erklären und somit ausschliessen.

(6) Nominees with a professional interest in Maemo, such as working for a company involved in Maemo-related software development - must declare their interest when advertising their nomination. Failure to do so may result in the Board of Directors, or the outgoing Council, declaring their nomination invalid and so bar them from standing in the current election.

(9) Das Datum der nächsten Wahl soll mindestens 4 Wochen vorher auf maemo.org bekannt gegeben werden.

(9) The election date must be publicised at least 4 weeks in advance of the election.

(11) Sollten nach Ende der Nominierungsphase weniger als 4 Kandidaten zur Wahl stehen, kann die Wahl nicht abgehalten werden.
a) Die Nominierungsphase wird dann um 4 Wochen verlängert und die Wahl entsprechend verschoben.
b) Sollten nach der verlängerten Nominierungsphase weniger als 4 Kandidaten zur Wahl stehen, bestimmt der scheidende Vorstandsrat zusammen mit dem Vorstand das weitere Vorgehen. Eine Lösung, die zu weiteren Kandidaten und einer ordentlichen Wahl führt, soll dabei bevorzugt werden, andere Möglichkeiten werden nicht ausgeschlossen.

(11) If there are less than 4 candidates when the nominations close, the election can’t be held.
a) The nomination period will be extended by 4 weeks and the election postponed similarly.
b) If, after 4 weeks's delay, there are still fewer than 4 candidates; the outgoing council will decide - in conjunction with Board of Directors - about how to proceed. The preferred solution is to encourage more members of the maemo.org community to participate and re-run the election, but all options are open.

(12) Änderungen der Vorstandsratswahlen müssen durch Beschluss durch die aktive Mitgliederversammlung oder die passive Mitgliederversammlung bestätigt werden.
a) Die Teilnahme an Referenda unterliegt den gleichen Bestimmungen, die für Vorstandsratswahlen gelten.
b) Bevor ein Referendum zur Abstimmung gelangen kann, muss ein entsprechender Entwurf für mindestens 4 Wochen zur Diskussion stehen.
c) Der Zeitraum, in welchem über das Referendum abgestimmt werden kann, soll genauso lang sein, wie bei regulären Vorstandsratswahlen.

(12)Changes to any of the above rules must be approved by a resolution of the Passive Members' Meeting (community referendum) or General Assembly.
a) Voting in such, will be open to anyone eligible to vote in the respective meeting.
b) The referendum options must be debated for a minimum of 4 weeks prior to the referendum.
c) Referendum voting will be open for the same length of time as the council elections.

1 Add to favourites0 Bury

August 30 2016

08:41

2016-08-09 Meeting Minutes

Meeting held 2016-08-09 on FreeNode, channel #maemo-meeting (logs)

Attending: Win7Mac, eekkelund, juiceme, chem|st

Partial:

Absent: pichlo, reinob

Summary of topics (ordered by discussion):

  • Topic Coding Competition

(Topic Coding Competition):


Action Items:
  • old items:
    • Coding competition planning (eekkelund)
    • Set up ftp/sftp site for CC (reinob)
    • Create twitter account for maemo community(eekkelund)
  • new items:

Solved Action Items:
  • The next GA meeting should be announced soon.
0 Add to favourites0 Bury
08:39

2016-08-02 Meeting Minutes

Meeting held 2016-08-02 on FreeNode, channel #maemo-meeting (logs)

Attending: pichlo, Win7Mac, eekkelund, reinob, juiceme

Partial:

Absent:

Summary of topics (ordered by discussion):

  • Topic warfare replaced all SSL certificates
  • Topic MC e.V.

(Topic warfare replaced all SSL certificates):

  • All maemo.org certs should be replaced now as the others were to expire today (except wiki)

(Topic MC e.V.):

  • GA meeting invitation posted
  • juiceme has papers for tax-excemption status. He will print those out

Action Items:
  • old items:
    • Coding competition planning (eekkelund)
    • Set up ftp/sftp site for CC (reinob)
    • Create twitter account for maemo community(eekkelund)
  • new items:

Solved Action Items:
  • The next GA meeting should be announced soon.
0 Add to favourites0 Bury

August 18 2016

06:39

2016-07-26 Meeting Minutes

Meeting held 2016-07-26 on FreeNode, channel #maemo-meeting (logs)

Attending: pichlo, Win7Mac, eekkelund, reinob, juiceme

Partial: chem|st

Absent:

Summary of topics (ordered by discussion):


(Topic Approve pending project request on https://garage.maemo.org/admin/approve-pending.php):

  • juiceme added Council to garage admin group
  • juiceme approved project request

(Topic Coding Competition):

  • Postponed CC by one month
  • Submitting an entry: "we could also set up an ftp site where we place whatever comes via mailing list, in case people prefer to download it like that" reinob could handle this

(Topic Twitter account):

  • Maemo community does not have twitter account
  • eekkelund will make one (Maemo Community, @maemo_org)

(Topic MC e.V.):

  • Choocing GA meeting dates
  • Court fillings
  • Win7Mac&reinob will candidate for board

Action Items:
  • old items:
    • Coding competition planning (eekkelund)
    • The next GA meeting should be announced soon.
  • new items:
    • Set up ftp/sftp site for CC (reinob)
    • Create twitter account for maemo community(eekkelund)

Solved Action Items:
Find out if https is doable
0 Add to favourites0 Bury
06:38

2016-07-19 Meeting Minutes

Meeting held 2016-07-19 on FreeNode, channel #maemo-meeting (logs)

Attending: Win7Mac, eekkelund, reinob

Partial:

Absent: pichlo, juiceme

Summary of topics (ordered by discussion):

  • Topic Coding Competition

(Topic Coding Competition):

  • How to submit an entry? Thread to TMO/email
  • Categories discussion, dropped wishlist.
  • Updated the Wiki page

Action Items:
  • old items:
    • Coding competition planning (eekkelund)
    • The next GA meeting should be announced soon.
  • new items:

Solved Action Items:
Find out if https is doable
0 Add to favourites0 Bury
06:36

2016-07-12 Meeting Minutes

Meeting held 2016-07-12 on FreeNode, channel #maemo-meeting (logs)

Attending: eekkelund, pichlo, reinob

Partial:

Absent: Win7Mac, juiceme

Summary of topics (ordered by discussion):

  • Topic Coding Competition

(Topic Coding Competition):

  • Donations, board has PayPal and bank account
  • Timeframe discussion
  • Categories discussion, 3 main categories: Something new, Fixing/Updating and Beginner.
  • Rules discussion
  • eekkelund has edited wiki page

Action Items:
  • old items:
    • Coding competition planning (eekkelund)
    • The next GA meeting should be announced soon.
  • new items:

Solved Action Items:
Find out if https is doable
0 Add to favourites0 Bury
06:35

2016-07-05 Meeting Minutes

Meeting held 2016-07-05 on FreeNode, channel #maemo-meeting (logs)

Attending: juiceme, eekkelund, pichlo, Win7Mac, reinob

Partial:

Absent:

Summary of topics (ordered by discussion):

  • Topic read-only option for TOR endpoints
  • Topic Coding Competition

(Topic read-only option for TOR endpoints):

  • There is thread in TMO
  • maemo.org should have read-only option for tor endpoints
  • Or if TMO is set up as hidden service then read-write access should be OK

(Topic Coding Competition):

  • eekkelund will start to write wiki page
  • Board will handle the prizes/donations
  • Timeframe discussion
  • Platforms discussion
  • Categories discussion
  • Voting, same as in elections

Action Items:
  • old items:
    • The next GA meeting should be announced soon.
  • new items:
    • Coding competition planning (eekkelund)

Solved Action Items:
Find out if https is doable
0 Add to favourites0 Bury

August 05 2016

12:24

On OGRE versions

Currently one can choose between the following OGRE versions
1.9, 1.10, 2.0 and 2.1

However the versioning scheme has become completely arbitrary while still resembling semantic versioning.
As a consequence somebody even had to put a “What version to choose?” guide on the OGRE homepage.

Unfortunately the guide confuses more than it helps:

  • First of all, forget about 1.9. It lacks most of the bugfixes that are in 1.10 and therefore is less stable. Even if you rely on third-party add-ons – 1.10 is the way to go. The pointless change that broke backward compatibility got reverted in the github fork.
  • If you do not like tracking a moving target and want stability, go with 1.10. It contains most bugfixes and the existing API does not change. This is actually your only choice if you want to rely on third-party add-ons or have existing code using OGRE. Furthermore you get the largest Platform support including HTML5 and Mobile.
  • You can mostly forget about 2.0. It is neither backward compatible with 1.10 nor forward compatible with 2.1. One should rather think of it as 2.1 beta (2.0 beta when using semantic versioning).
    Feature-wise it  boosts the node updating performance by about the factor of 5x by exploiting cache locality (see 1.9 review by Mike Acton). However it lacks many fixes that are in 1.10 and it is unmaintained so things will not improve.
  • Finally 2.1 is where all the new and shiny things are. However it is backwards incompatible to 1.10 and 2.0 so calling it 3.0 would be more appropriate.
    Feature-wise you get the high node updating performance of 2.0 and automatic batching for rendering. The batching results in a higher GPU utilization so with 2.1 rendering speed is bound by the GPU performance. With 2.0 and 1.x on the other hand the GPU was often stalled because of inefficient material and geometry changes. However even 2.1 lacks some bugfixes that are in 1.10 and more importantly it is a moving target as the API still evolves. Also it only works on desktop platforms.

You might wonder why 2.1 while having the latest features, lacks some bugfixes of older releases. Read on..

Lack of branching concept

Usually projects have a default master or trunk branch where all development eventually ends up and optionally some maintenance and bleeding edge branches. See this for a small overview of the according workflows.

Now OGRE has the following branches: v1-9, v1-10, v2-0, v2-1 and default. By common sense default would contain the most recent code while the other branches would be maintenance branches of the respective releases. But not today! Actually all of these branches are under active development with 2.1 containing the latest features and default representing the current stable version. (actually default and 1.10 are pretty much the same)

As there is no clear code flow between the branches this means that branches are diverging. Fixes for issues in one branch do not distribute across the others. Examples:

As you can see we got all possible directions here. The situation can not be easily resolved by successive merging branches into each other.

With the github fork, I therefore manually compared the different branches and cherry-picked all fixed into master (v1-10).

This is why I initially called it the stable fork. Even though it contains new features as discussed in my last post, the API is backwards compatible to existing code/ add-ons and it aggregates all bugfixes in one branch.

0 Add to favourites0 Bury

July 30 2016

16:11

Maemo Community e.V. - Invitation to the General Assembly 2016

Maemo Community e.V. - Einladung zur Mitgliederversammlung 2016

Sehr geehrtes Mitglied, unsere diesjährige ordentliche Mitgliederversammlung findet am 04.09.2016 16:00 UTC im IRC Freenode Channel #maemo-meeting statt.

Auf der Tagesordnung stehen folgende Themen:
1. Begrüßung durch die Vorsitzende des Vorstands
2. Feststellung der ordnungsgemäßen Einberufung und der Beschlussfähigkeit der Mitgliederversammlung
3. Entgegennahme des Jahresberichts für das abgelaufene Geschäftsjahr und Entlastung des Vorstands
4. Neuwahl des Vorstandes
5. Änderung der Satzung §7 und §8, und der Vereinsordnung §1.1, §3.2 und §4.1. Erläuterung im Anhang.
6. Verschiedenes

Anträge auf Ergänzungen der Tagesordnung müssen bis eine Woche vor der Versammlung schriftlich beim Vorstand eingereicht werden (§ 9 Abs. 2 der Satzung).

Mit freundlichen Grüßen
Win7Mac/Gido Griese

############################ English##########################

Maemo Community e.V. - Invitation to the General Assembly 2016

Dear Member, our Annual General Meeting will be held on Sunday, 2016-09-04 at 16:00 UTC on IRC Freenode channel #maemo-meeting.

Unless any further issues are raised, the agenda includes the following topics:
1. Welcome by the Chairman of the Board
2. Determination of the proper convocation and the quorum of the General Assembly
3. Acceptance of the annual report for the fiscal year and actions of the Executive
4. Election of the Board Directors
5. Amendment of Articles of Association §7 and §8, Association Rules §1.1, §3.2 and $4.1 - please see appendix for details [1].
6. Any other business

Requests for additions to the agenda must be submitted to the Board in writing one week prior to the meeting (§ 9 para. 2 of the Statutes).

Yours sincerely
Win7Mac/Gido Griese


[1]http://maemo.org/community/council/referendum_2015/


Anhang / APPENDIX

* Änderungen der Satzung/Amendments to the Articles of Association:
§7 Vorstand (Board of directors)
(5) Der Vorstand führt die Beschlüsse der aktiven Mitgliederversammlung und der des Vorstandsrats aus, sofern dem Vorstandsrat diese Entscheidungshoheit laut Satzung oder Vereinsordnung eingeräumt wird.
(5) The Board of Directors carries out the resolutions of the General Assembly and of the Council in so far as this decision's jurisdiction is delegated to Council according to the statutes.

§8 Vorstandsrat (Council)
(4) Aufgaben des Vorstandsrats sowie Regelungen zu dessen Arbeit, inklusive der Einberufung von Sitzungen, deren Ablauf und die Durchführung von Abstimmungen, sind in der Vereinsordnung festgelegt.
(4) The council's duties, rules to announce meetings, their proceedings and executions of votes are regulated by the Association Rules.

* Änderungen der Vereinsordnung/Amendments to the Association Rules
§ 1.1
(2) § 4.1 (Vorstandsratswahlen) dieser Vereinsordnung kann durch ein Referendum der passiven Mitgliederversammlung geändert oder aufgehoben werden. Änderungen dieser Wahlordnung durch die aktive Mitgliederversammlung müssen durch Beschluss der passiven Mitgliederversammlung bestätigt werden (Referendum). Sollte die Änderung von der passiven Mitgliederversammlung abgelehnt werden, kann der Änderungsvorschlag überarbeitet und erneut zur Abstimmung gebracht werden.

(2) § 4.1 (Council election rules) of these association rules can be changed or dismissed with a referendum by the Passive Members' Meeting. Changes to these election rules by the General Assembly must be confirmed by a resolution of the passive members meeting (referendum). Should the amendment be rejected by the passive members meeting, the proposed amendment can be revised and brought again to the vote.

§3.2 Vorstandsrat (council)

(7) Aufgaben und Zuständigkeiten des Vorstandsrats sind wie folgt:
a) Vertretung der assoziierten Mitglieder und ihrer Interessen
b) Beratung des Vorstands in allgemeinen Belangen
c) Der Vorstandsrat ist für die Durchführung aller Wahlen zuständig

(7) Tasks and competences of the Council are as follows:
a) Representation of the associated members (garage account) and their interests.
b) Counseling of the Board of directors in general matters
c) Council is responsible to execute all elections

§ 4.1 Vorstandsratswahlen (Council Elections)

(6) Kandidaten mit einem professionellen bzw. witschaftlichen Interesse in Maemo, z.B. Mitarbeiter von Firmen, die in Bezug zu Maemo Hard- oder Software stehen, müssen ihr Interesse bei der Nominierung erklären. Bleibt dies aus, kann der Vorstandsrat die Kandidatur des Mitglieds für ungültig erklären und somit ausschliessen.

(6) Nominees with a professional interest in Maemo, such as working for a company involved in Maemo-related software development - must declare their interest when advertising their nomination. Failure to do so may result in the Board of Directors, or the outgoing Council, declaring their nomination invalid and so bar them from standing in the current election.

(9) Das Datum der nächsten Wahl soll mindestens 4 Wochen vorher auf maemo.org bekannt gegeben werden.

(9) The election date must be publicised at least 4 weeks in advance of the election.

(11) Sollten nach Ende der Nominierungsphase weniger als 4 Kandidaten zur Wahl stehen, kann die Wahl nicht abgehalten werden.
a) Die Nominierungsphase wird dann um 4 Wochen verlängert und die Wahl entsprechend verschoben.
b) Sollten nach der verlängerten Nominierungsphase weniger als 4 Kandidaten zur Wahl stehen, bestimmt der scheidende Vorstandsrat zusammen mit dem Vorstand das weitere Vorgehen. Eine Lösung, die zu weiteren Kandidaten und einer ordentlichen Wahl führt, soll dabei bevorzugt werden, andere Möglichkeiten werden nicht ausgeschlossen.

(11) If there are less than 4 candidates when the nominations close, the election can’t be held.
a) The nomination period will be extended by 4 weeks and the election postponed similarly.
b) If, after 4 weeks's delay, there are still fewer than 4 candidates; the outgoing council will decide - in conjunction with Board of Directors - about how to proceed. The preferred solution is to encourage more members of the maemo.org community to participate and re-run the election, but all options are open.

(12) Änderungen der Vorstandsratswahlen müssen durch Beschluss durch die aktive Mitgliederversammlung oder die passive Mitgliederversammlung bestätigt werden.
a) Die Teilnahme an Referenda unterliegt den gleichen Bestimmungen, die für Vorstandsratswahlen gelten.
b) Bevor ein Referendum zur Abstimmung gelangen kann, muss ein entsprechender Entwurf für mindestens 4 Wochen zur Diskussion stehen.
c) Der Zeitraum, in welchem über das Referendum abgestimmt werden kann, soll genauso lang sein, wie bei regulären Vorstandsratswahlen.

(12)Changes to any of the above rules must be approved by a resolution of the Passive Members' Meeting (community referendum) or General Assembly.
a) Voting in such, will be open to anyone eligible to vote in the respective meeting.
b) The referendum options must be debated for a minimum of 4 weeks prior to the referendum.
c) Referendum voting will be open for the same length of time as the council elections.1 Add to favourites0 Bury

July 28 2016

12:42

Truly huge files and the problem of continuous virtual address space

As we all know does mmap, or even worse on Windows CreateFileMapping, need contiguous virtual address space for a given mapping size. That can become a problem when you want to load a file of a gigabyte with mmap.

The solution is of course to mmap the big file using multiple mappings. For example like adapting yesterday’s demo this way:

void FileModel::setFileName(const QString &fileName)
{
    ...
    if (m_file->open(QIODevice::ReadOnly)) {
        if (m_file->size() > MAX_MAP_SIZE) {
            m_mapSize = MAX_MAP_SIZE;
            m_file_maps.resize(1 + m_file->size() / MAX_MAP_SIZE, nullptr);
        } else {
            m_mapSize = static_cast(m_file->size());
            m_file_maps.resize(1, nullptr);
        }
        ...
    } else {
        m_index->open(QFile::ReadOnly);
        m_rowCount = m_index->size() / 4;
    }
    m_file_maps[0] = m_file->map(0, m_mapSize, QFileDevice::NoOptions);
    qDebug() map(0, m_index->size(), QFileDevice::NoOptions);

    beginResetModel();
    endResetModel();
    emit fileNameChanged();
}

And in the data() function:

QVariant FileModel::data( const QModelIndex& index, int role ) const
{
    QVariant ret;
    ...
    quint32 mapIndex = pos_i / MAX_MAP_SIZE;
    quint32 map_pos_i = pos_i % MAX_MAP_SIZE;
    quint32 map_end_i = end_i % MAX_MAP_SIZE;
    uchar* map_file = m_file_maps[mapIndex];
    if (map_file == nullptr)
        map_file = m_file_maps[mapIndex] = m_file->map(mapIndex * m_mapSize, m_mapSize, QFileDevice::NoOptions);
    position = m_file_maps[mapIndex] + map_pos_i;
    if (position) {
            const int length = static_cast(end_i - pos_i);
            char *buffer = (char*) alloca(length+1);
            if (map_end_i >= map_pos_i)
                strncpy (buffer, (char*) position, length);
            else {
                const uchar *position2 = m_file_maps[mapIndex+1];
                if (position2 == nullptr) {
                    position2 = m_file_maps[mapIndex+1] = m_file->map((mapIndex+1) *
                         m_mapSize, m_mapSize, QFileDevice::NoOptions);
                }
                strncpy (buffer, (char*) position, MAX_MAP_SIZE - map_pos_i);
                strncpy (buffer + (MAX_MAP_SIZE - map_pos_i), (char*) position2, map_end_i);
            }
            buffer[length] = 0;
            ret = QVariant(QString(buffer));
        }
    }
    return ret;
}

You could also not use mmap for the very big source text file and use m_file.seek(map_pos_i) and m_file.read(buffer, length). The most important mapping is of course the index one, as the reading of the individual lines can also be done fast enough with normal read() calls (as long as you don’t have to do it for each and every line of the very big file and as long as you know in a O(1) way where the QAbstractListModel’s index.row()’s data is).

But you already knew that. Right?

0 Add to favourites0 Bury

July 26 2016

19:15

Loading truly truly huge text files with a QAbstractListModel

Sometimes people want to do crazy stuff like loading a gigabyte sized plain text file into a Qt view that can handle QAbstractListModel. Like for example a QML ListView. You know, the kind of files you generate with this commando:

base64 /dev/urandom | head -c 100000000 > /tmp/file.txt

But, how do they do it?

FileModel.h

So we will make a custom QAbstractListModel. Its private member fields I will explain later:

#ifndef FILEMODEL_H
#define FILEMODEL_H

#include <QObject>
#include <QVariant>
#include <QAbstractListModel>
#include <QFile>

class FileModel: public QAbstractListModel {
    Q_OBJECT

    Q_PROPERTY(QString fileName READ fileName WRITE setFileName NOTIFY fileNameChanged )
public:
    explicit FileModel( QObject* a_parent = nullptr );
    virtual ~FileModel();

    int columnCount(const QModelIndex &parent) const;
    int rowCount( const QModelIndex& parent =  QModelIndex() ) const Q_DECL_OVERRIDE;
    QVariant data( const QModelIndex& index, int role = Qt::DisplayRole ) const  Q_DECL_OVERRIDE;
    QVariant headerData( int section, Qt::Orientation orientation,
                         int role = Qt::DisplayRole ) const  Q_DECL_OVERRIDE;
    void setFileName(const QString &fileName);
    QString fileName () const
        { return m_file->fileName(); }
signals:
    void fileNameChanged();
private:
    QFile *m_file, *m_index;
    uchar *map_file;
    uchar *map_index;
    int m_rowCount;
    void clear();
};

#endif// FILEMODEL_H

FileModel.cpp

We will basically scan the very big source text file for newline characters. We’ll write the offsets of those to a file suffixed with “.mmap”. We’ll use that new file as a sort of “partition table” for the very big source text file, in the data() function of QAbstractListModel. But instead of sectors and files, it points to newlines.

The reason why the scanner itself isn’t using the mmap’s address space is because apparently reading blocks of 4kb is faster than reading each and every byte from the mmap in search of \n characters. Or at least on my hardware it was.

You should probably do the scanning in small qEventLoop iterations (make sure to use nonblocking reads, then) or in a thread, as your very big source text file can be on a unreliable or slow I/O device. Plus it’s very big, else you wouldn’t be doing this (please promise me to just read the entire text file in memory unless it’s hundreds of megabytes in size: don’t micro optimize your silly homework notepad.exe clone).

Note that this is demo code with a lot of bugs like not checking for \r and god knows what memory leaks and stuff was remaining when it suddenly worked. I leave it to the reader to improve this. An example is that you should check for validity of the “.mmap” file: your very big source text file might have changed since the newline partition table was made.

Knowing that I’ll soon find this all over the place without any of its bugs fixed, here it comes ..

#include "FileModel.h"

#include <QDebug>

#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <pthread.h>
#include <unistd.h>

FileModel::FileModel( QObject* a_parent )
    : QAbstractListModel( a_parent )
    , m_file (nullptr)
    , m_index(nullptr)
    , m_rowCount ( 0 ) { }

FileModel::~FileModel() { clear(); }

void FileModel::clear()
{
    if (m_file) {
        if (m_file->isOpen() && map_file != nullptr)
            m_file->unmap(map_file);
        delete m_file;
    }
    if (m_index) {
        if (m_index->isOpen() && map_index != nullptr)
            m_index->unmap(map_index);
        delete m_index;
    }
}

void FileModel::setFileName(const QString &fileName)
{
   clear();
   m_rowCount = 0;
   m_file = new QFile(fileName);
   int cur = 0;
   m_index = new QFile(m_file->fileName() + ".mmap");
   if (m_file->open(QIODevice::ReadOnly)) {
       if (!m_index->exists()) {
           char rbuffer[4096];
           m_index->open(QIODevice::WriteOnly);
           char nulbuffer[4];
           int idxnul = 0;
           memset( nulbuffer +0, idxnul >> 24 & 0xff, 1 );
           memset( nulbuffer +1, idxnul >> 16 & 0xff, 1 );
           memset( nulbuffer +2, idxnul >>  8 & 0xff, 1 );
           memset( nulbuffer +3, idxnul >>  0 & 0xff, 1 );
           m_index->write( nulbuffer, sizeof(quint32));
           qDebug() << "Indexing to" << m_index->fileName();
           while (!m_file->atEnd()) {
               int in = m_file->read(rbuffer, 4096);
               if (in == -1)
                   break;
               char *newline = (char*) 1;
               char *last = rbuffer;
               while (newline != 0) {
                   newline = strchr ( last, '\n');
                   if (newline != 0) {
                     char buffer[4];
                     int idx = cur + (newline - rbuffer);
                     memset( buffer +0, idx >> 24 & 0xff, 1 );
                     memset( buffer +1, idx >> 16 & 0xff, 1 );
                     memset( buffer +2, idx >>  8 & 0xff, 1 );
                     memset( buffer +3, idx >>  0 & 0xff, 1 );
                     m_index->write( buffer, sizeof(quint32));
                     m_rowCount++;
                     last = newline + 1;
                  }
               }
               cur += in;
           }
           m_index->close();
           m_index->open(QFile::ReadOnly);
           qDebug() << "done";
       } else {
           m_index->open(QFile::ReadOnly);
           m_rowCount = m_index->size() / 4;
       }
       map_file= m_file->map(0, m_file->size(), QFileDevice::NoOptions);
       qDebug() << "Done loading " << m_rowCount << " lines";
       map_index = m_index->map(0, m_index->size(), QFileDevice::NoOptions);
   }
   beginResetModel();
   endResetModel();
   emit fileNameChanged();
}

static quint32
read_uint32 (const quint8 *data)
{
    return data[0] << 24 |
           data[1] << 16 |
           data[2] << 8 |
           data[3];
}

int FileModel::rowCount( const QModelIndex& parent ) const
{
    Q_UNUSED( parent );
    return m_rowCount;
}

int FileModel::columnCount(const QModelIndex &parent) const
{
    Q_UNUSED( parent );
    return 1;
}

QVariant FileModel::data( const QModelIndex& index, int role ) const
{
    if( !index.isValid() )
        return QVariant();
    if (role == Qt::DisplayRole) {
        QVariant ret;
        quint32 pos_i = read_uint32(map_index + ( 4 * index.row() ) );
        quint32 end_i;
        if ( index.row() == m_rowCount-1 )
            end_i = m_file->size();
        else
            end_i = read_uint32(map_index + ( 4 * (index.row()+1) ) );
        uchar *position;
        position = map_file +  pos_i;
        uchar *end = map_file + end_i;
        int length = end - position;
        char *buffer = (char*) alloca(length +1);
        memset (buffer, 0, length+1);
        strncpy (buffer, (char*) position, length);
        ret = QVariant(QString(buffer));
        return ret;
    }
    return QVariant();
}

QVariant FileModel::headerData( int section, Qt::Orientation orientation, int role ) const
{
    Q_UNUSED(section);
    Q_UNUSED(orientation);
    if (role != Qt::DisplayRole)
           return QVariant();
    return QString("header");
}

main.cpp

#include <QGuiApplication>
#include <QQmlApplicationEngine>
#include <QtQml>// qmlRegisterType

#include "FileModel.h"

int main(int argc, char *argv[])
{
    QGuiApplication app(argc, argv);
    qmlRegisterType<FileModel>( "FileModel", 1, 0, "FileModel" );
    QQmlApplicationEngine engine;
    engine.load(QUrl(QStringLiteral("qrc:/main.qml")));
    return app.exec();
}

main.qml

import QtQuick 2.3
import QtQuick.Window 2.2
import FileModel 1.0

Window {
    visible: true

    FileModel { id: fileModel }
    ListView {
        id: list
        anchors.fill: parent
        delegate: Text { text: display }
        MouseArea {
            anchors.fill: parent
            onClicked: {
                list.model = fileModel
                fileModel.fileName = "/tmp/file.txt"
            }
        }
    }
}

profile.pro

TEMPLATE = app
QT += qml quick
CONFIG += c++11
SOURCES += main.cpp \
    FileModel.cpp
RESOURCES += qml.qrc
HEADERS += \
    FileModel.h

qml.qrc

<RCC>
    <qresource prefix="/">
        <file>main.qml</file>
    </qresource>
</RCC>
0 Add to favourites0 Bury

July 09 2016

13:45

Creating PyGTK app snaps with snapcraft

Snap is a new packaging format introduced by Ubuntu as an successor to dpkg aka debian package. It offers sandboxing and transactional updates and thus is a competitor to the flatpak format and resembles docker images.

As with every new technology the weakest point of working with snaps is the documentation. Your best bet so far is the snappy-playpen repository.

There are also some rough edges regarding desktop integration and python interoperability, so this is what the post will be about.

I will introduce some quircks that were needed to get teatime running, which is written in Python3 and uses Unity and GTK3 via GObject introspection.

The most important thing to be aware of is that snaps are similar to containers in that each snap has its own rootfs and only restricted access outside of it. This is basically what the sandboxing is about.
However a typical desktop application needs to know quite a lot about the outside world:

  • It must know which theme the user currently uses, and after that it also needs to access the theme files.
  • For saving anything it needs access to /home
  • If it should access the internet it needs system level access as well; like querying whether there actually is an active internet connection

To declare that we want to write to home, play back sound and use unity features we use the plugs keyword like

apps:
    teatime:
         # ...
         plugs: [unity7, home, pulseaudio]

However we must also tell our app to look for the according libraries inside its snap instead of the system paths. For this one must change quite a few environment variables manually. Fortunately Ubuntu provides wrapper scripts that take care of this for us. They are called desktop-launchers.

To use the launcher the configures the GTK3 environment we have to extend the teatime part like this:

apps:
    teatime:
        command: desktop-launch $SNAP/usr/share/teatime/teatime.py
        # ...
parts:
    teatime:
         # ...
         after: [desktop/gtk3]

The desktop-launch script takes care of telling PyGTK where the GI repository files are.

You can see the full snapcraft.yml here.

 

Update:

Before my fix, one had to use this rather lengthy startup command

env GI_TYPELIB_PATH=$SNAP/usr/lib/girepository-1.0:$SNAP/usr/lib/x86_64-linux-gnu/girepository-1.0 desktop-launch $SNAP/usr/share/teatime/teatime.py

which hard-coded the architecture.

/Update

After this teatime will start, but the paths still have to be fixed. Inside a snap “/” still refers to the system root, so all absolute paths must be prefixed with $SNAP.

Actually I think the design of flatpak is more elegant in this regard where “/” points to the local rootfs and one does not have to change absolute paths. To bring in the system parts flatpak uses bind mounts.

Conclusions

Once you get the hang of how snaps work, packaging becomes quite straightforward, however currently there are still some drawbacks

  • the snap package results in 120MB compared to a 12KB deb. This is actually a feature as shipping all the dependencies makes the snap installable on every linux distribution. However I hope that we can get this down by introducing shared frameworks (like GTK3, Python) that do not have to be included in the app package.
  • Due to another issue, your snap has only the C locale available and thus will not be localized.
  • [Update: fixed] Unity desktop notifications do not work. You will get a DBus exception at the corresponding call.
  • [Update: fixed] The shipped .desktop file is not hooked up with the system, so you can only launch the app via the command line.
0 Add to favourites0 Bury
Older posts are this way If this message doesn't go away, click anywhere on the page to continue loading posts.
Could not load more posts
Maybe Soup is currently being updated? I'll try again automatically in a few seconds...
Just a second, loading more posts...
You've reached the end.

Don't be the product, buy the product!

Schweinderl