24 hours a day, 7 days a week, 365 days per year...

November 24, 2015

2015-11-24 Tuesday

  • More struggling with main-loops; product team call. Discovered that the new scheduler has the (non-optimal) idea of setting a MAX_UINT64 timeout in milliseconds to the glib main-loop timeout; and this is really a non-clever idea.
  • Also (thanks Dimstar) managed to find the SRPMs for openSUSE 13.2 which was surprisingly hard (though finding the source in the build-service is easy).
  • Plugged away; tested on windows; Norbert kindly tested on Mac; all good (encouragingly). Read stories for babes.

SDN/NFV DevRoom at FOSDEM: Deadline approaching!

We extended the deadline for the SDN/NFV DevRoom at FOSDEM to Wednesday, November 25th recently – and we now have the makings of a great line-up!

To date, I have received proposals about open source VNFs, dataplane acceleration and accelerated virtual switching, an overview of routing on the internet that looks fascinating, open switch design and operating systems, traffic generation and testing, and network overlays.

I am still interested in having a few more NFV focussed presentations, and one or two additional SDN controller projects – and any other topics you might think would tickle our fancy! Just over 24 hours until the deadline.

Fetching and Extending Update Access Key expiry date for IBM's POWER8 Firmware

On IBM's POWER8 systems there was no way to have Update Access Keys updated till now.

After the updated POWER8 firmware, with my commits merged to upstream candidate "next" branch of powerpc-utils's activate_firmware utility last week, and with the ongoing efforts of the Electronic Service Agent team based on my work, it will soon be possible to make the update happen on the fly on provisioning a new update-access-key to the utility.

IBM Power8 customers will also be able to look at the current UAK Expiry date to make a decision as to when to update the system with a new key and verify whether date has been updated after providing the key or not.

Quoting the man page the options added to activate_firmware utility are:

-e [keyfile]
When used with this option, the command either fetches the current Update Access
Key expiry date or sets the Update Access Key expiry date if is

The activate_firmware utility will not directly be used by the end user but by an update_flash script to view or extend the firmware entitlement date by providing a new update-access-key.


November 23, 2015

2015-11-23 Monday

  • A day of E-mail, and lots of calls; 1:1's, partner team meeting, bit of hackery; team meeting; more hackery until late - trying to unwind a lot of evil in the VCL main-loop; we badly need a real 'idle' concept; starting to get to that.

Concurso Universitario de Software Libre 2015-2016

Long time no post!


A quick reminder for students (*) in Spain interested in participating in this year’s CUSL, the deadline for the project proposals has been extended until December 1st:

You’re still on time to submit a proposal!

* Universidad, bachiller, ciclos de grado medio…

Filed under: FreeDesktop Planet, GNOME Planet, GNU Planet, Meetings, Planets, Uncategorized

Improve Stratup time of Logs

It's been a long time that I haven't written a blog. :-(

Recently(actually I a long time ago) I noticed that it take me a very long time to open Logs. I did use a timer to see how long it takes me exactly. And I was surprised to know it takes me more than 4 minutes! It's a lot. I really cannot stand it any more. Then I began to investigate what's wrong with Logs.

I implemented a feature that viewing the latest five boots in this summer as part of my GSoC project. The commit can be found here. In this patch, to get the latest 5 boots, I fetched all the boots available in the system. Why I do this? I learned to do this from part of the source codes of journalctl. I have more than 700 boots in my laptop. So it takes quite a while for Logs to iterate those 700+ boots. And of course the same result for "journalctl --list-boots". It also takes me more than 4 minutes to get the output.

Logs only show users the latest 5 boots. So obviously it's not necessary to fetch all available boots. Then I investigated the latest source code of journalctl and found a piece of improved source code.

Well, the general idea to fix it is, to iterate boots from the latest boot to the earliest one. But we only iterate 5 times, and that's enough.

Firstly, I used sd_journal_seek_tail() to seek to the end of the journal(that's also the last log entry of the latest boot) and get the realtime timestamp of the current log entry. Then get the latest boot's boot ID and sd_journal_add_match() to only filter log entries from the specific boot. sd_journal_seek_head() will be used to seek to the first log entry of the latest boot, then we get the realtime timestamp using sd_journal_get_realtime_usec(). So right now we finished the operations for a single boot, and then I use sd_journal_previous() to seek to the last log entry of the second latest boot. Then just follow step above to get the next few available boots.

The commit for this can be found here.

After this patch, the startup time of Logs for those who have hundreds of boots in their system has been improved significantly. At least it only takes me a few seconds to open Logs.

November 22, 2015

From “power on” to “gtk GUI usable” as fast as possible

In my Raspberry Pi + Hifiberry-amp + Pitft project I’ve been looking how to speed up the boot process.

[edit] I updated various parts of this post with suggestions from the comments. [/edit]

The original boot process took around 60 seconds from “power on” until my gtk app was visible and responding. I used the original (Wheezy based) raspbian, with autologon on tty1, and in the .bash_profile I called startx. In /home/pi/.xinitrc I started my application without starting a display manager:
exec /usr/bin/python /home/pi/

(b.t.w. the PYTHONIOENCODING is there otherwise python will give an exception when calling print with a non-ascii unicode character)

By removing a few services from the init process I removed 10 seconds from the boot process. Still 50 seconds. But further options on Raspbian Wheezy are limited.

Next step was to move to Raspbian Jessie, which has systemd. This gave an unwanted side effect: the hifiberry was detected correctly, but did not produce any sound anymore. Removing a line added for the pitft device_tree=bcm2708-rpi-b-plus.dtb from /boot/config.txt fixed this problem. And the pitft is still working.

In systemd I can start X11 as soon as userspace is running. However this generated an error – the touchscreen device has to be available first. The touchscreen input is in udev configured as /dev/input/touchscreen. I created /etc/systemd/system/xinit-login.service with the following content:
ExecStart=/bin/su pi -l -c /usr/bin/xinit -- VT08

[edit] with suggestions from the comments in now looks like:


and .xinitrc is now a hardlink to the gtk application [/edit]

This makes my xinit session start directly after the touchscreen device is available. This reduced the startup time to 28 seconds. Much better! (b.t.w. I haven’t look if all entries in xinit-login.service are correct, perhaps the contents can be improved).

Systemd has a good tool called systemd-analyze which analyzes the boot process. That way you can easily see which parts take too much time. That helped me to strip off another 2 seconds. An interesting question is how the udev config can be improved to create the touchscreen device earlier in the boot process. Another interesting question is how X11 can be configured to start faster, or how python + gtk3 can be configured to start faster. If you have any suggestions please leave them in the comments.

Next thing was to make the system readonly. I installed busybox-syslogd instead of rsyslogd, since it can do in-memory logging. [edit] I now switched to systemd-journald for in memory logging [/edit]

I analyzed which directories changed after a reboot, and moved those to a tmpfs in-memory filesystem. The /etc/fstab now looks like:
proc /proc proc defaults 0 0
/dev/mmcblk0p1 /boot vfat defaults,ro 0 2
/dev/mmcblk0p2 / ext4 defaults,noatime,ro 0 1
tmpfs /tmp tmpfs defaults,noatime 0 0
tmpfs /var/lock tmpfs defaults,noatime 0 0
tmpfs /var/lib/dhcpcd5 tmpfs defaults,noatime 0 0
tmpfs /var/log tmpfs defaults,noatime 0 0

Possibly this can be optimized with a single tmpfs mount and symlinks to reduce the mount time of 4 tmpfs filesystems.

[edit] did that, they are all symlinks to /tmp now [/edit]

The only remaining issue that I didn’t solve yet is handling of /etc/resolv.conf when moving the setup to a different network. Because /etc/ is readonly the resolv.conf is now fixed for my local wifi network.

[edit] disabling the UART, and moving from the debian networking scripts to systemd-networkd reduced the boot time to 21 seconds! [/edit]

Task-oriented desktop and applications?

I wanted to add a comment to Allan’s blog post The next big thing, but too late! comments are already closed (side note, I think it’s a general problem on or it’s a WordPress issue, or both).

So, at Flock 2014, Langdon White talked about “Fedora for Developers” (video), with the following idea, among other things: Desktop integration with “tasks” (with saved state).

It reminded me about a discussion that I had with gedit developers back in 2011. We talked about profiles: being able to create different gedit profiles, for example one for the C language and one for Vala. Different gedit instances could be launched with different profiles. They would behave as independent apps, with different settings, different plugins activated, a different UI state, and so on. Of course a profile could be created for each programming project, or… task.

But a task often involves launching several applications. So to resume a complete task quickly, in one step, it either needs some desktop integration, or another application could be responsible to create tasks and resume them. But either way it needs coordination with applications.

November 21, 2015

CMake: dependencies between targets and files and custom commands

As I said in my last post about CMake, targets are everything in CMake. Unfortunately, not everything is a target though!

If you’ve tried do anything non-trivial in CMake using the add_custom_command() command, you may have got stuck in this horrible swamp of confusion. If you want to generate some kind of file at build time, in some manner other than compiling C or C++ code, then you need to use a custom command to generate the file. But files aren’t targets and have all sorts of exciting limitations to make you forget everything you ever new about dependency management.

What makes it so hard is that there’s not one limitation, but several. Here is a hopefully complete list of things you might want to do in CMake that involve custom commands and custom targets depending on each other, and some explainations as to why things don’t work the way that you might expect.

1. Dependencies between targets

point1-verticalThis is CMake at its simplest (and best).

cmake_minimum_required(VERSION 3.2)

add_library(foo foo.c)

add_executable(bar bar.c)
target_link_libraries(bar foo)

You have a library, and a program that depends on it. When you run CMake, both of them get built. Ideal! This is great!

What is “all”, in the dependency graph to the left? It’s a built in target, and it’s the default target. There are also “install” and “test” targets built in (but no “clean” target).

2. Custom targets

If your project is a good one then maybe you use a documentation tool like GTK-Doc or Doxygen to generate documentation from the code.

This is where add_custom_command() enters your life. You may live to regret ever letting it in.

cmake_minimum_required(VERSION 3.2)

        doxygen docs/Doxyfile
        cmake -E touch docs/doxygen.stamp
        "Generating API documentation with Doxygen"

We have to create a ‘stamp’ file because Doxygen generates lots of different files, and we can’t really tell CMake what to expect. But actually, here’s what to expect: nothing! If you build this, you get no output. Nothing depends on the documentation, so it isn’t built.

So we need to add a dependency between docs/doxygen.stamp and the “all” target. How about using add_dependencies()? No, you can’t use that with any of the built in targets. But as a special case, you can use add_custom_target(ALL) to create a new target attached to the “all” target:

    docs ALL
    DEPENDS docs/doxygen.stamp


In practice, you might also want to make the custom command depend on all your source code, so it gets regenerated every time you change the code. Or, you might want to remove the ALL from your custom target, so that you have to explicitly run make docs to generate the documentation.

This is also discussed here.

3. Custom commands in different directories

Another use case for add_custom_command() is generating source code files using 3rd party tools.

### Toplevel CMakeLists.txt
cmake_minimum_required(VERSION 3.2)


### src/CMakeLists.txt
        cmake -E echo "Generate my C code" > foo.c

### tests/CMakeLists.txt
        test-foo.c ${CMAKE_CURRENT_BINARY_DIR}/../src/foo.c

    NAME test-foo
    COMMAND test-foo

How does this work? Actually it doesn’t! You’ll see the following error when you run CMake:

CMake Error at tests/CMakeLists.txt:1 (add_executable):
  Cannot find source file:


  Tried extensions .c .C .c++ .cc .cpp .cxx .m .M .mm .h .hh .h++ .hm .hpp
  .hxx .in .txx
CMake Error: CMake can not determine linker language for target: test-foo
CMake Error: Cannot determine link language for target "test-foo".

Congratulations, you’ve hit bug 14633! The fun thing here is that generated files don’t behave anything like targets. Actually they can only be referenced in the file that contains the corresponding add_custom_command() call. So when we refer to the generated foo.c in tests/CMakeLists.txt, CMake actually has no idea where it could come from, so it raises an error.


As the corresponding FAQ entry describes, there are two things you need to do to work around this limitation.

The first is to wrap your custom command in a custom target. Are you noticing a pattern yet? Most of the workarounds here are going to involve wrapping custom commands in custom targets. In src/CMakeLists.txt, you do this:


Then, in tests/CMakeLists.txt, you can add a dependency between “test-foo” and “generate-foo”:

add_dependency(test-foo generate-foo)

That’s enough to ensure that foo.c now gets generated before the build of test-foo begins, which is obviously important. If you try to run CMake now, you’ll hit the same error, because CMake still has no idea where that generated foo.c file might come from. The workaround here is to manually set the GENERATED target property:



Note that this is a bit of a contrived example. In most cases, the correct solution is to do this:

### src/CMakeLists.txt
add_library(foo foo.c)

### tests/CMakeLists.txt
target_link_libraries(test-foo foo)

Then you don’t have to worry about any of the nonsense above, because libraries are proper targets, and you can use them anywhere.

Even if it’s not practical to make a library containing ‘foo.c’, there must be some other target that links against it in the same directory that it is generated in. So instead of creating a “generate-foo” target, you can make “test-foo” depend on whatever other target links to “foo.c”.

4. Custom commands and parallel make

I came into this issue while doing something pretty unusual with CMake: wrapping a series of Buildroot builds. Imagine my delight at discovering that, when parallel make was used, my CMake-generated Makefile was running the same Buildroot build multiple times at the same time! That is not what I wanted!

It turns out this is a pretty common issue. The crux of it is that with the “Unix Makefiles” backend, multiple toplevel targets run as an independent, parallel make processes. Files aren’t targets, and unless something is a target then it doesn’t get propagated around like you would expect.

Here is the test case:

cmake_minimum_required(VERSION 3.2)

    OUTPUT gen
    COMMAND sleep 1
    COMMAND cmake -E echo Hello > gen

    my-all-1 ALL DEPENDS gen

    my-all-2 ALL DEPENDS gen

If you generate a Makefile from this and run make -j 2, you’ll see the following:

Scanning dependencies of target my-all-2
Scanning dependencies of target my-all-1
[ 50%] Generating gen
[100%] Generating gen
[100%] Built target my-all-2
[100%] Built target my-all-1

If creating ‘gen’ takes a long time, then you really don’t want it to happen multiple times! It may even cause disasters, for example running make twice at once in the same Buildroot build tree is not pretty at all.


As explained in bug 10082, the solution is (guess what!) to wrap the custom command in a custom target!

add_custom_target(make-gen DEPENDS gen)


Then you change the custom targets to depend on “make-gen”, instead of the file ‘gen’. Except! Be careful when doing that — because there is another trap waiting for you!

5. File-level dependencies of custom targets are not propagated

If you read the documentation of add_custom_command() closely, and you look at the DEPENDS keyword argument, you’ll see this text:

If DEPENDS specifies any target (created by the add_custom_target(), add_executable(), or add_library() command) a target-level dependency is created to make sure the target is built before any target using this custom command. Additionally, if the target is an executable or library a file-level dependency is created to cause the custom command to re-run whenever the target is recompiled.

This sounds quite nice, like more or less what you would expect. But the important bit of information here is what CMake doesn’t do: when a custom target depends on another custom target, all the file level dependencies are completely ignored.

Here’s your final example for the evening:

cmake_minimum_required(VERSION 3.2)


    OUTPUT gen1
    COMMAND cmake -E echo ${SPECIAL_TEXT} > gen1

    DEPENDS gen1

    OUTPUT gen2
    DEPENDS gen1-wrapper
    COMMAND cmake -E copy gen1 gen2

    all-generated ALL
    DEPENDS gen2

This is subtly wrong, even though you did what you were told, and wrapped the custom command in a custom target.

The first time you build it:

Scanning dependencies of target gen1-wrapper
[ 50%] Generating gen1
[ 50%] Built target gen1-wrapper
Scanning dependencies of target all-generated
[100%] Generating gen2
[100%] Built target all-generated

But then touch the file ‘gen1’, or overwrite it with something other text, or change the value of SPECIAL_TEXT in CMakeLists.txt to something else, and you will see this:

[ 50%] Generating gen1
[ 50%] Built target gen1-wrapper
[100%] Built target all-generated

There’s no file-level dependency created between ‘gen2’ and ‘gen1’, so ‘gen2’ never gets updated, and things get all weird.


You can’t just depend on gen1 instead of gen1-wrapper, because it may end up being built multiple times! See the previous point. Instead, you need to depend on the “gen1-wrapper” target and the file itself:

    OUTPUT gen2
    DEPENDS gen1-wrapper gen1
    COMMAND cmake -E copy gen1 gen2

As the documentation says, this only applies to targets wrapping add_custom_command() output. If ‘gen1’ was a library created with add_library, things would work how you expect.



Maybe I just have a blunt head, but I found all of this quite difficult to work out. I can understand why CMake works this way, but I think there is plenty of room for improvement in the documentation where this is explained. Hopefully this guide has gone some way to making things clearer.

If you have any other dependency-related traps in CMake that you’ve hit, please comment and I’ll add them to this list…

November 20, 2015

Kubernetes from the ground up

I really loved reading Git from the bottom up when I was learning Git, which starts by showing how all the pieces fit together. Starting with the basics and gradually working towards the big picture is a great way to understand any complex piece of technology.

Recently I’ve been working with Kubernetes, a fantastic cluster manager. Like Git it is tremendously powerful, but the learning curve can be quite steep.

But there is hope. Kamal Marhubi has written a great series of articles that take the same approach: start from the basic building blocks, build with those.

Currently available:

Highly recommended.


Comments | More on | @rubenv on Twitter

gspell news

  • gspell is now fully hosted on!
  • There is now a mailing list.
  • In addition to LaTeXila, gspell is now also used by gedit (5,800 lines removed, yay!).
  • The 0.1.x version has branched and is meant to be installed alongside GNOME 3.18.
  • If everything goes fine, the 1.0 version will be released at the same time as GNOME 3.20, with a stable API.

API reviews are more than welcome!

Stay tuned.

Introducing sd-event

The Event Loop API of libsystemd

When we began working on systemd we built it around a hand-written ad-hoc event loop, wrapping Linux epoll. The more our project grew the more we realized the limitations of using raw epoll:

  • As we used timerfd for our timer events, each event source cost one file descriptor and we had many of them! File descriptors are a scarce resource on UNIX, as RLIMIT_NOFILE is typically set to 1024 or similar, limiting the number of available file descriptors per process to 1021, which isn't particularly a lot.

  • Ordering of event dispatching became a nightmare. In many cases, we wanted to make sure that a certain kind of event would always be dispatched before another kind of event, if both happen at the same time. For example, when the last process of a service dies, we might be notified about that via a SIGCHLD signal, via an sd_notify() "STATUS=" message, and via a control group notification. We wanted to get these events in the right order, to know when it's safe to process and subsequently release the runtime data systemd keeps about the service or process: it shouldn't be done if there are still events about it pending.

  • For each program we added to the systemd project we noticed we were adding similar code, over and over again, to work with epoll's complex interfaces. For example, finding the right file descriptor and callback function to dispatch an epoll event to, without running into invalidated pointer issues is outright difficult and requires non-trivial code.

  • Integrating child process watching into our event loops was much more complex than one could hope, and even more so if child process events should be ordered against each other and unrelated kinds of events.

Eventually, we started working on sd-bus. At the same time we decided to seize the opportunity, put together a proper event loop API in C, and then not only port sd-bus on top of it, but also the rest of systemd. The result of this is sd-event. After almost two years of development we declared sd-event stable in systemd version 221, and published it as official API of libsystemd.


sd-event.h, of course, is not the first event loop API around, and it doesn't implement any really novel concepts. When we started working on it we tried to do our homework, and checked the various existing event loop APIs, maybe looking for candidates to adopt instead of doing our own, and to learn about the strengths and weaknesses of the various implementations existing. Ultimately, we found no implementation that could deliver what we needed, or where it would be easy to add the missing bits: as usual in the systemd project, we wanted something that allows us access to all the Linux-specific bits, instead of limiting itself to the least common denominator of UNIX. We weren't looking for an abstraction API, but simply one that makes epoll usable in system code.

With this blog story I'd like to take the opportunity to introduce you to sd-event, and explain why it might be a good candidate to adopt as event loop implementation in your project, too.

So, here are some features it provides:

  • I/O event sources, based on epoll's file descriptor watching, including edge triggered events (EPOLLET). See sd_event_add_io(3).

  • Timer event sources, based on timerfd_create(), supporting the CLOCK_MONOTONIC, CLOCK_REALTIME, CLOCK_BOOTIME clocks, as well as the CLOCK_REALTIME_ALARM and CLOCK_BOOTTIME_ALARM clocks that can resume the system from suspend. When creating timer events a required accuracy parameter may be specified which allows coalescing of timer events to minimize power consumption. For each clock only a single timer file descriptor is kept, and all timer events are multiplexed with a priority queue. See sd_event_add_time(3).

  • UNIX process signal events, based on signalfd(2), including full support for real-time signals, and queued parameters. See sd_event_add_signal(3).

  • Child process state change events, based on waitid(2). See sd_event_add_child(3).

  • Static event sources, of three types: defer, post and exit, for invoking calls in each event loop, after other event sources or at event loop termination. See sd_event_add_defer(3).

  • Event sources may be assigned a 64bit priority value, that controls the order in which event sources are dispatched if multiple are pending simultanously. See sd_event_source_set_priority(3).

  • The event loop may automatically send watchdog notification messages to the service manager. See sd_event_set_watchdog(3).

  • The event loop may be integrated into foreign event loops, such as the GLib one. The event loop API is hence composable, the same way the underlying epoll logic is. See sd_event_get_fd(3) for an example.

  • The API is fully OOM safe.

  • A complete set of documentation in UNIX man page format is available, with sd-event(3) as the entry page.

  • It's pretty widely available, and requires no extra dependencies. Since systemd is built on it, most major distributions ship the library in their default install set.

  • After two years of development, and after being used in all of systemd's components, it has received a fair share of testing already, even though we only recently decided to declare it stable and turned it into a public API.

Note that sd-event has some potential drawbacks too:

  • If portability is essential to you, sd-event is not your best option. sd-event is a wrapper around Linux-specific APIs, and that's visible in the API. For example: our event callbacks receive structures defined by Linux-specific APIs such as signalfd.

  • It's a low-level C API, and it doesn't isolate you from the OS underpinnings. While I like to think that it is relatively nice and easy to use from C, it doesn't compromise on exposing the low-level functionality. It just fills the gaps in what's missing between epoll, timerfd, signalfd and related concepts, and it does not hide that away.

Either way, I believe that sd-event is a great choice when looking for an event loop API, in particular if you work on system-level software and embedded, where functionality like timer coalescing or watchdog support matter.

Getting Started

Here's a short example how to use sd-event in a simple daemon. In this example, we'll not just use sd-event.h, but also sd-daemon.h to implement a system service.

#include <alloca.h>
#include <endian.h>
#include <errno.h>
#include <netinet/in.h>
#include <signal.h>
#include <stdbool.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/ioctl.h>
#include <sys/socket.h>
#include <unistd.h>

#include <systemd/sd-daemon.h>
#include <systemd/sd-event.h>

static int io_handler(sd_event_source *es, int fd, uint32_t revents, void *userdata) {
        void *buffer;
        ssize_t n;
        int sz;

        /* UDP enforces a somewhat reasonable maximum datagram size of 64K, we can just allocate the buffer on the stack */
        if (ioctl(fd, FIONREAD, &sz) < 0)
                return -errno;
        buffer = alloca(sz);

        n = recv(fd, buffer, sz, 0);
        if (n < 0) {
                if (errno == EAGAIN)
                        return 0;

                return -errno;

        if (n == 5 && memcmp(buffer, "EXIT\n", 5) == 0) {
                /* Request a clean exit */
                sd_event_exit(sd_event_source_get_event(es), 0);
                return 0;

        fwrite(buffer, 1, n, stdout);
        return 0;

int main(int argc, char *argv[]) {
        union {
                struct sockaddr_in in;
                struct sockaddr sa;
        } sa;
        sd_event_source *event_source = NULL;
        sd_event *event = NULL;
        int fd = -1, r;
        sigset_t ss;

        r = sd_event_default(&event);
        if (r < 0)
                goto finish;

        if (sigemptyset(&ss) < 0 ||
            sigaddset(&ss, SIGTERM) < 0 ||
            sigaddset(&ss, SIGINT) < 0) {
                r = -errno;
                goto finish;

        /* Block SIGTERM first, so that the event loop can handle it */
        if (sigprocmask(SIG_BLOCK, &ss, NULL) < 0) {
                r = -errno;
                goto finish;

        /* Let's make use of the default handler and "floating" reference features of sd_event_add_signal() */
        r = sd_event_add_signal(event, NULL, SIGTERM, NULL, NULL);
        if (r < 0)
                goto finish;
        r = sd_event_add_signal(event, NULL, SIGINT, NULL, NULL);
        if (r < 0)
                goto finish;

        /* Enable automatic service watchdog support */
        r = sd_event_set_watchdog(event, true);
        if (r < 0)
                goto finish;

        if (fd < 0) {
                r = -errno;
                goto finish;
        } = (struct sockaddr_in) {
                .sin_family = AF_INET,
                .sin_port = htobe16(7777),
        if (bind(fd, &, sizeof(sa)) < 0) {
                r = -errno;
                goto finish;

        r = sd_event_add_io(event, &event_source, fd, EPOLLIN, io_handler, NULL);
        if (r < 0)
                goto finish;

        (void) sd_notifyf(false,
                          "STATUS=Daemon startup completed, processing events.");

        r = sd_event_loop(event);

        event_source = sd_event_source_unref(event_source);
        event = sd_event_unref(event);

        if (fd >= 0)
                (void) close(fd);

        if (r < 0)
                fprintf(stderr, "Failure: %s\n", strerror(-r));

        return r < 0 ? EXIT_FAILURE : EXIT_SUCCESS;

The example above shows how to write a minimal UDP/IP server, that listens on port 7777. Whenever a datagram is received it outputs its contents to STDOUT, unless it is precisely the string EXIT\n in which case the service exits. The service will react to SIGTERM and SIGINT and do a clean exit then. It also notifies the service manager about its completed startup, if it runs under a service manager. Finally, it sends watchdog keep-alive messages to the service manager if it asked for that, and if it runs under a service manager.

When run as systemd service this service's STDOUT will be connected to the logging framework of course, which means the service can act as a minimal UDP-based remote logging service.

To compile and link this example, save it as event-example.c, then run:

$ gcc event-example.c -o event-example `pkg-config --cflags --libs libsystemd`

For a first test, simply run the resulting binary from the command line, and test it against the following netcat command line:

$ nc -u localhost 7777

For the sake of brevity error checking is minimal, and in a real-world application should, of course, be more comprehensive. However, it hopefully gets the idea across how to write a daemon that reacts to external events with sd-event.

For further details on the functions used in the example above, please consult the manual pages: sd-event(3), sd_event_exit(3), sd_event_source_get_event(3), sd_event_default(3), sd_event_add_signal(3), sd_event_set_watchdog(3), sd_event_add_io(3), sd_notifyf(3), sd_event_loop(3), sd_event_source_unref(3), sd_event_unref(3).


So, is this the event loop to end all other event loops? Certainly not. I actually believe in "event loop plurality". There are many reasons for that, but most importantly: sd-event is supposed to be an event loop suitable for writing a wide range of applications, but it's definitely not going to solve all event loop problems. For example, while the priority logic is important for many usecase it comes with drawbacks for others: if not used carefully high-priority event sources can easily starve low-priority event sources. Also, in order to implement the priority logic, sd-event needs to linearly iterate through the event structures returned by epoll_wait(2) to sort the events by their priority, resulting in worst case O(n*log(n)) complexity on each event loop wakeup (for n = number of file descriptors). Then, to implement priorities fully, sd-event only dispatches a single event before going back to the kernel and asking for new events. sd-event will hence not provide the theoretically possible best scalability to huge numbers of file descriptors. Of course, this could be optimized, by improving epoll, and making it support how todays's event loops actually work (after, all, this is the problem set all event loops that implement priorities -- including GLib's -- have to deal with), but even then: the design of sd-event is focussed on running one event loop per thread, and it dispatches events strictly ordered. In many other important usecases a very different design is preferable: one where events are distributed to a set of worker threads and are dispatched out-of-order.

Hence, don't mistake sd-event for what it isn't. It's not supposed to unify everybody on a single event loop. It's just supposed to be a very good implementation of an event loop suitable for a large part of the typical usecases.

Note that our APIs, including sd-bus, integrate nicely into sd-event event loops, but do not require it, and may be integrated into other event loops too, as long as they support watching for time and I/O events.

And that's all for now. If you are considering using sd-event for your project and need help or have questions, please direct them to the systemd mailing list.

A GTK+ update

You may have noticed that GTK+ master has a large number of changes in the CSS area. As some like to put it:

Oh NO! they’re breaking themes again!

Which is certainly one way to look at it, but I think it misses the point a little bit – since the effort is actually meant to make life easier for anybody who wants to change the appearance of GTK+ widgets.

What changes are we making ?

As a first step, we are introducing CSS nodes. A CSS node has an element name, a state and it can have style classes.  Each widget has one or more CSS nodes, and they are organized in a tree.  GTK+’s CSS machinery matches CSS selectors on this CSS node tree, and as a result, each node carries a full set of CSS properties.

The transition to CSS nodes is mostly done in GTK+ 3.19.2.

In a  second step, we will integrate CSS nodes into size allocation and rendering. This will bring consistent support for margins, padding and min-width/height for all widgets. This step is currently being prepared on the wip/otte/gadget branch.

None of this is exposed as API yet  (with some small exceptions, such as gtk_widget_path_iter_set_object_name ) . It needs more time to prove itself before we are ready to offer this as stable API.

Why are we doing this ?

There are a number of reasons for doing these changes. In no particular order,

  • Element names and style classes use by widgets for their CSS nodes (and their tree relationships) are now documented and provide a stable interface for themes and custom CSS.
  • Element names provide a level of abstraction over the direct use of type names, and will allow reuse of CSS. You wil be able to add a “check” node to your own widget, whereas you can’t call your own widget GtkCheckButton.
  • CSS nodes are permanent. This implies that they can carry state, e.g. for running CSS animations. They can also notify on state changes.Spinning
  • CSS nodes can explored and manipulated in GtkInspector.Inspect!
  • CSS nodes (in particular, the second phase) force us to clean up all the irregularities of the current widget rendering and move to a much more regular model, where every node draws a background and a  frame. Fancy Switches
  • Positional selectors such as :first-child now work everywhere, including parts of a widget. You can e.g., use :nth-child(even) to style notebook tabs.Fancy Tabs

What do you have to do ?

If you maintain a theme or custom CSS for an application,  update your selectors to use the documented element names and classes. The changes in Adwaita can serve as an example for the kind of changes that are needed.

If you use GTK+’s style machinery to render non-widget content (like mutter does for window decorations), use
gtk_widget_path_iter_set_object_name to make your widget paths look like the widgets they imitate.

If you have code that pokes colors out of GtkStyleContext to do your own rendering, you should really port to use the gtk_render APIs and custom CSS. If you want to keep your existing code working for now, you need to minimally fix gtk_style_context_get_color calls to use the correct state.  Ie. go from

gtk_style_context_get_color (context,


gtk_style_context_save (context);
gtk_style_context_set_state (contexts,
gtk_style_context_get_color (context,
          gtk_style_context_get_state (context),
gtk_style_context_restore (context);

What else is new ?

Support for help overlays Shortcuts got merged into GTK+ and is available in 3.19.1. We’ve gotten some feedback from early adopters and added a few more features, so things should be in good shape for wider adoption: please add it to your favorite GTK+ application, and update this page if you do so.

Wayland support is progressing. So far in 3.19, the focus has been on straightening out issues with window and popup sizing and positioning. Many of the remaining gaps will be fixed by integrating the protocols that have been under discussion for a while now: DND actions, tablet support, pointer lock,…  Jonas just revamped the Wayland protocol development process, which should lead to faster progress on this front. For more details, see Jonas’ mail.

FAD G11N Event Report

FAD G11N started with a lot of enthusiasm and energy. It was fully packed with discussions and hackathons. 

Discussions points 

  • How to get started with new contributors? How to get new contributors?
  • Release based statistics for Fedora (in terms of translations)
  • Fedora release schedule w.r.t to translations deadlines 
  • Zanata survey and how to improve 
  • Fedora g11n structure 
  • Fedora 24 planning 
To attract new comers and get them started, one of the suggestions was to document the whole process and my suggestion was create a online course on and I can help if someone wants to do it.

During hackathons everyone worked on various projects and demonstrated it at the end of FAD.
It was productive because for some projects some people worked together and it was for the first time fedora g11n group met in person. 

I have worked on zanata python client with Suanand, we tested z-p-c with latest zanata server and identified missing features.

Another thing that I worked on is hunspell dictionary infra project. It's difficult to manage hunspell dictionaries. So one thought is to create online portal for it. I created a prototype for it so that and it can found on
I will work on this project to create single page app.

And Finally FAD ends with happy faces and looking forward to see it happen again


November 19, 2015

Pitivi 0.95 — Enfant Suisse

Hey everyone! It’s time for a new Pitivi release, 0.95. This one packs a lot of bugfixes and architectural work to further stabilize the GES backend. In this blog post, I’ll give you an overview of the new and interesting stuff this release brings, coming out from a year of hard work. It’s pretty epic and you’re in for a few surprises, so I suggest listening to this song while you’re reading this blog post.

Engine rework: completed.

Those of you who attended my talk at GUADEC 2013 might remember this particular slide:

kill gnonlin

Well, it’s done now. It’s dead and buried.

This is something I’ve had on my mind for so long, I was even having nightmares about it—literally. To give you an idea just how ancient gnonlin was from an architectural standpoint, it was created fourteen years ago, merely six months after the first release of GStreamer itself. Well, y’know, a lot of stuff happens in 13-14 years.

So, over the past year, Mathieu and Thibault gradually refactored GNonLin into NLE, the new non-linear engine inside GES. For details, see the previous two blog posts about our War Against Deadlocks: the story about the mixing elements and the story about the new engine using them (replacing gnonlin).

The resulting improvements in reliability are not only palpable in daily use, they are actually quantifiable with the results our GES gst-validate test suite runs:

  • In the 1.4 series: 154 tests pass out of 198 (22.2% failures)
  • With the 1.6 release: 198 tests pass out of 198
— "What's going on? Give me a sitrep!" — "The tests… they all pass!" — "What?!"

— “What’s going on? Give me a sitrep!”
— “The tests… they all pass!”
— “What?!”

Now 100% GTK, with new horizons

pitivi 0.95

We were hitting various limitations and bugs (such as this) in Clutter, the library we used to display and animate the project’s timeline. Eventually we came to a point where we had to change strategy and port the timeline to use pure GTK+ widgets, with Matplotlib for drawing the keyframes on clips. Quite some work went into the new timeline.

The viewer (the widget that shows the main video preview, above the timeline) using glimagesink was causing too many problems related to embedding in the X window. We switched to the new GtkSink instead, which also allowed us to test gtkglsink at the same time, as they are compatible.

Thanks to the new GTK timeline, we have a little surprise to show here: technically, Pitivi can also work on Mac OS X now. This is not an April Fool’s joke.

Some notes about the experiment are sitting there if you’re curious. At this time, we are not supporting the Mac OS version officially, because we don’t have the resources for that (yet?). I was told that we should be able to make something available for testing a Mac build once we reach 1.0. Want to make it happen sooner? Feel free to join us and to work on that.

Wait, that’s not all. These changes also allow us to make Pitivi work with the GDK Broadway backend, meaning we can even run Pitivi in a web browser! Yep, you heard that right. Pitivi in a web browser. What could possibly go wrong? ;)

Spit polishing

An improvement we’re quite happy about is that you can finally drag & drop a file from a different app directly to the timeline, to create a clip.

The layers’ representation changed somewhat. Previously, an audio-video clip would be displayed as two separate clips in the timeline, one for video and one for audio, on two separate layers. At times it was pretty awkward. While porting the timeline, Thibault simplified the layers model to have the notion of generic layers, in which audio-video clips are represented as a unified clip object. This also means that there is no more wasted space if the layer has only video or only audio.

Also worth mentioning:

  • We have resurrected the transformation box feature, but the UI is currently very primitive. See the Clip properties > Transformation section when a clip is selected on the timeline. You can also drag the image in the viewer to change the position of the selected clip at the current position and you can use the mouse wheel to zoom in/out.
  • While editing a project, every operation is saved in a scenario file. These can be used when reporting bugs. See how to use scenarios for reporting complicated bugs easily (or if you’re feeling geeky, the details about how the scenarios are used to automatically test the GES backend).
  • You can now copy/paste clips in the timeline.
  • We’re now compatible with smaller screen resolutions (such as 1024×768) again
  • We removed a bunch of widgets in the layer controls. They were placeholders for future features, we should put them back once the features actually become available.
  • Undo/redo has been disabled until we add unit tests and make sure it works properly. Until then you can Ctrl+S regularly.
  • See also the release notes for 0.95.

Infrastructure changes

  • The Pitivi team migrated from Bugzilla to Phabricator for bug/task tracking.
  • We now have a script to setup the development environment from the latest daily bundle. This hybrid approach makes it very easy for new developers to start hacking on Pitivi’s Python side without needing to build the rest.
  • It was difficult for us to keep using Dogtail, so we moved all the integration tests to GstValidate.
  • Some of you have suggested that we compress the all-in-one bundles using XZ, and so we did. Our packages are now 20% lighter than uncompressed tarballs, so they will take less time to download (which is nice if you’re using the dailies to test).
  • With some help from Jeffrey Schroeder, I have finally upgraded our MediaWiki instance to the latest stable release. We hadn’t upgraded it in four years (thankfully it was fairly locked down so we did not run into trouble), in big part because it was not version-controlled and thus was a pain in the butt to manage. I should be able to do a better job at keeping it up-to-date from now on.

Where do we stand on the fundraiser?

In terms of donations, less than the fundraiser’s first milestone was reached. Therefore, instead of working full-time and burning through the money in a matter of a few months, Thibault and Mathieu decided to work at a slower rate while simultaneously providing professional multimedia consulting services to put food on the table.

Nonetheless, they eventually reached the point where they had worked through all the donated funds, and so they continued in their free time. The GTK+ Timeline and GtkSink work, for example, is one of the big architectural changes that Thibault had to do on his spare time, without monetary compensation whatsoever.

Now is still a good time to let others know and ask those around you to donate! We appreciate it.

A call for ruthless testing

As it is much more stable already, we recommend all users to upgrade to Pitivi 0.95 and help us find remaining issues, if any. Until this release trickles down into distributions, you can download our all-in-one bundle and try out 0.95, right here and now. Enjoy!

You’re in luck: I already spent a lot of my (very limited) spare time testing and investigating the most serious issues. In fact, one of the reasons why it’s been so long since the last release is that I have been Thibault’s worse nightmare for months (there’s a reason why my name strikes fear in the hearts of GStreamer developers):


Every two weeks or so, Thibault would come to me and say, “Hey look, I fixed all your bugs, how about we release now?”. I would then spend a day testing and return with ten more bugs. Then he would fix them all, and I would find ten other bugs in different areas. Then he would fix them, and I would find another batch that I couldn’t test last time. And so on and so forth, from spring to autumn. For example, these are the bugs I’ve found just for the GTK Timeline. Can’t believe I haven’t killed that poor guy.

Now that the blocker issues are solved, I’m quite impressed with how much more reliable this version of Pitivi is shaping out to be now. But hey, we’re not perfect, maybe there are bugs we’ve overlooked, so please grab 0.95 and try to break it as hard as you can, reporting the issues you find (especially freezes, crashes, incorrect output, etc.). We want it to be solid. Go wild.

office space printer

Thank you for reading, commenting and sharing! This blog post is part of a série of articles tracking progress made with work related to the 2014 Pitivi fundraiser. Researching and writing quality articles takes a lot of time, so please be patient and enjoy the ride! 😉
  1. An update from the 2014 summer battlefront
  2. The 0.94 release
  3. The War Against Deadlocks, part 1: The story of our new thread-safe mixing elements reimplementation
  4. The War Against Deadlocks, part 2: GNonLin's reincarnation
  5. The 0.95 release, the GTK+ timeline and sink
  6. Measuring quality/reliability through time (clarifying what gst-validate is)
  7. Our all-in-one binaries building infrastructure, and why it matters
  8. Samples, “scenario” files and you: how you can help us reproduce (almost) any bug very easily
  9. The 1.0 release and closure of the fundraiser

If it's not practical to redistribute free software, it's not free software in practice

I've previously written about Canonical's obnoxious IP policy and how Mark Shuttleworth admits it's deliberately vague. After spending some time discussing specific examples with Canonical, I've been explicitly told that while Canonical will gladly give me a cost-free trademark license permitting me to redistribute unmodified Ubuntu binaries, they will not tell me what Any redistribution of modified versions of Ubuntu must be approved, certified or provided by Canonical if you are going to associate it with the Trademarks. Otherwise you must remove and replace the Trademarks and will need to recompile the source code to create your own binaries actually means.

Why does this matter? The free software definition requires that you be able to redistribute software to other people in either unmodified or modified form without needing to ask for permission first. This makes it clear that Ubuntu itself isn't free software - distributing the individual binary packages without permission is forbidden, even if they wouldn't contain any infringing trademarks[1]. This is obnoxious, but not inherently toxic. The source packages for Ubuntu could still be free software, making it fairly straightforward to build a free software equivalent.

Unfortunately, while true in theory, this isn't true in practice. The issue here is the apparently simple phrase you must remove and replace the Trademarks and will need to recompile the source code. "Trademarks" is defined later as being the words "Ubuntu", "Kubuntu", "Juju", "Landscape", "Edubuntu" and "Xubuntu" in either textual or logo form. The naive interpretation of this is that you have to remove trademarks where they'd be infringing - for instance, shipping the Ubuntu bootsplash as part of a modified product would almost certainly be clear trademark infringement, so you shouldn't do that. But that's not what the policy actually says. It insists that all trademarks be removed, whether they would embody an infringement or not. If a README says "To build this software under Ubuntu, install the following packages", a literal reading of Canonical's policy would require you to remove or replace the word "Ubuntu" even though failing to do so wouldn't be a trademark infringement. If an email address is present in a changelog, you'd have to change it. You wouldn't be able to ship the juju-core package without renaming it and the application within. If this is what the policy means, it's so impractical to be able to rebuild Ubuntu that it's not free software in any meaningful way.

This seems like a pretty ludicrous interpretation, but it's one that Canonical refuse to explicitly rule out. Compare this to Red Hat's requirements around Fedora - if you replace the fedora-logos, fedora-release and fedora-release-notes packages with your own content, you're good. A policy like this satisfies the concerns that Dustin raised over people misrepresenting their products, but still makes it easy for users to distribute modified code to other users. There's nothing whatsoever stopping Canonical from adopting a similarly unambiguous policy.

Mark has repeatedly asserted that attempts to raise this issue are mere FUD, but he won't answer you if you ask him direct questions about this policy and will insist that it's necessary to protect Ubuntu's brand. The reality is that if Debian had had an identical policy in 2004, Ubuntu wouldn't exist. The effort required to strip all Debian trademarks from the source packages would have been immense[2], and this would have had to be repeated for every release. While this policy is in place, nobody's going to be able to take Ubuntu and build something better. It's grotesquely hypocritical, especially when the Ubuntu website still talks about their belief that people should be able to distribute modifications without licensing fees.

All that's required for Canonical to deal with this problem is to follow Fedora's lead and isolate their trademarks in a small set of packages, then tell users that those packages must be replaced if distributing a modified version of Ubuntu. If they're serious about this being a branding issue, they'll do it. And if I'm right that the policy is deliberately obfuscated so Canonical can encourage people to buy licenses, they won't. It's easy for them to prove me wrong, and I'll be delighted if they do. Let's see what happens.

[1] The policy is quite clear on this. If you want to distribute something other than an unmodified Ubuntu image, you have two choices:
  1. Gain approval or certification from Canonical
  2. Remove all trademarks and recompile the source code
Note that option 2 requires you to rebuild even if there are no trademarks to remove.

[2] Especially when every source package contains a directory called "debian"…

comment count unavailable comments

Red Hat Enterprise Linux 7.2 – A major desktop milestone

So many of you have probably seen that RHEL 7.2 is out today. There are many important updates in this release, some of them detailed in the official RHEL 7.2 press release.

One thing however which you would only discover if you start digging into the 7.2 update is that its the first time in RHEL history that we are doing a full scale desktop update in a point release. We shipped RHEL 7.0 with GNOME 3.8 and in RHEL 7.2 we are updating it to GNOME 3.14. This brings in a lot of new major features into RHEL, like the work we did on improved HiDPI support, improved touch and gesture support, it brings GNOME Software to RHEL, the improved system status area and so on. We plan on updating the desktop further in later RHEL 7.x point releases.

This change of policy is of course important to the many RHEL Workstation customers we have, but I also hope it will make RHEL Workstation and also the CentOS Workstation more attractive options to those in the community who have been looking for a LTS version of Fedora. This policy change gives you the rock solid foundation of RHEL and the RHEL kernel and combines it with a very well tested yet fairly new desktop release. So if you feel Fedora is moving to quickly, yet have felt that RHEL on the other hand has been moving to slowly, we hope that with this change to RHEL we have found a sweet compromise.

We will of course also keep doing regular applications updates in RHEL 7.x, just like we started with in RHEL 6.x. Giving you up to date versions of things like LibreOffice, Firefox, Thunderbird and more.

I’m joining Endless

Some of you may have asked why I’m so silent for the last 3 weeks. Well, the titles pretty much explain things: between university duties, moving to a new house and conducting my research, I’m also joining Endless!

I’m ridiculously happy with this opportunity, and Endless folks were extremely respectful of my schedule – I’m working remotely, and only as a part-time intern. This is a great chance for me to work with FOSS on something that directly changes people’s lives!

So, wait for 2 more weeks so I can settle again, and I’ll start sending patches and crafting new features on rampage – like the good ol’ times.

Hitori available as an xdg-app preview

I’ve been playing a bit with xdg-app recently, and just spent half an hour packaging Hitori as an xdg-app bundle. It was remarkably easy — in fact, it only took 7 commands.

If you want to install it, make sure you’ve got xdg-app installed, then:

# Install to /usr/share/ostree/trusted.gpg.d first
# Then set up the runtimes (if you haven’t already)
xdg-app add-remote --user gnome-sdk
xdg-app install-runtime --user gnome-sdk org.gnome.Platform 3.18
xdg-app install-runtime --user gnome-sdk org.freedesktop.Platform 1.2
# Add the repository for Hitori, then install it
xdg-app add-remote --user --no-gpg-verify pwithnall-hitori
xdg-app install-app --user pwithnall-hitori org.gnome.Hitori
# Play the game!
xdg-app run org.gnome.Hitori

What works?

  • Playing the game
  • Use of X11
  • Icons
  • Desktop file

What doesn’t work (yet)?

  • GSettings access (I may have misconfigured this, because it’s supposed to work)
  • Help manual

I don’t plan to support this repository especially well, since it was just for playing around, but it shows that xdg-app is coming along nicely!

Cockpit 0.83 and 0.84 Released

Cockpit releases every week. This week it was 0.84. I’ll also include notes from 0.83 here.

Building Cockpit on Debian

At systemd.conf Dominik worked with Michael Biebl one of the Debian systemd maintainers on packaging Cockpit for Debian. We’re still looking for a maintainer long term.

Here’s a blog post with more details.

Cross Distro Integration Tests

In Cockpit we run hundreds of tests on real operating systems for each pull request. Without running these tests on an OS it’s impossible to know that the features of Cockpit actually works. So far we’ve been running these tests on Fedora, Atomic, and RHEL. But we’d really like to run them on Debian as well. That’ll make Cockpit much more well rounded.

Marius worked on the first steps toward running the tests on Debian, by doing the Cockpit build inside of our test VM images. Hopefully we’ll see more progress on this.

SELinux certificate file type

The cockpit.service helpfully sets the appropriate user and group on the certificates that cockpit-ws will use for TLS. Now it also sets the SELinux file context type properly, so this is one less things to break for an admin.

Cockpit manual page

There is now a man cockpit overview manual page that links to the guide and elsewhere.

From the future

Marius has done work on an SOS reporting view. Needs some further backend work, but should be ready soon:

Peter has mostly completed the work to add machines with alternate users, and non-standard SSH ports. Among other things, this is useful for cloud instances. I’m looking forward to seeing this in Cockpit 0.85.

Machine Dialogs Wireframes

Try it out

Cockpit 0.84 is available now:

November 18, 2015

guv: Automatic scaling of Heroku workers

At The Grid we do a lot of computationally heavy work server-side, in order to produce websites from user-provided content. This includes image analytics (for understanding the content), constraint solving (for page layout) and image processing (optimization and filtering to achieve a particular look). Currently we serve some thousand sites, with some hundred thousands sites expected by the time we’ve completed beta – so scalability is a core concern.

All computationally intensive work is put as jobs in a AMQP/RabbitMQ message queue, which are consumed by Heroku workers. To make it easy to manage many queues and worker roles we also use MsgFlo.
This provides us with the required flexibility to scale: the queues buffer the in-progress work, broker distributes evenly between available workers, and with Heroku we can change number of workers with one command. But, it still leaves us with the decision on how much compute capacity to provision. And when load is dynamic, it is tedious & inefficient to do it manually – especially as Heroku bills workers used by the second.

RabbitMQ and Heroku dashboards

Monitoring RabbitMQ queues and scaling Heroku workers manually when demand changes; not fun.

If we would instead regulate this every 1-5 minute based on demand, we would reduce costs. Or alternatively, with a fixed budget, provide a better quality-of-service. And most importantly, let developers worry about other things.

Of course, there already exists a number of solutions for this. However, some used particular metrics providers which we were not using, some used metrics with unclear relationship to required workers (like number of users), or had unacceptable limitations (only one worker per service, only run as a service with pay-by-number-of-workers).


guv 0.1 implements a simple proportional scaling model. Based the current number of jobs in the queue, and an estimate of job processing time – it calculates the number of workers required for all work to be completed within a configured deadline.

guv system model

The deadline is the maximum time you allow for your users to wait for a completed job. The job processing time [average, deviation] can be calculated from metrics of previous jobs. And the number of jobs in queue is read directly from RabbitMQ.

# A simple guv config for one worker role.
# One guv instance typically manages many worker roles
  app: my-heroku-app
  queue: 'analyze.IN' # RabbitMQ queue name
  worker: analyzeworker # Heroku dyno role name
  process: 20
  deadline: 120.0
  min: 1 # keep something always running
  max: 15 # budget limits

Now there are a couple limitations of this model. Primarily, it is completely reactive; we do not attempt to predict how traffic will develop in the future. Prediction is after all terribly tricky business – better not go there if it can be avoided.
And since it takes a non-zero amount of time to spin up a new worker (about 45-60 seconds), on a sudden spike in demand may cause some jobs to miss a tight deadline, as the workers can’t spin up fast enough. To compensate for this, there is some simple hysteresis: scale up more aggressively, and scale down a bit reluctanctly – we might need the workers next couple of minutes.

As a bonus, guv includes some integration with common metrics services: The metrics about ‘jobs-in-flight’ on, come directly from guv. And using New Relic Insights, we can analyze how the scaling is performing.

Last 2 days of guv scaling history on some of the workers roles at The Grid.

If we had a manual scaling with a constant number over 48 hours period, workers=35 (Max), then we would have paid at least 3-4 times more than we did with autoscaling (difference in size of area under Max versus area under the 10 minute line). Alternatively we could have provisioned a lower number of workers, but then with spikes above that number – our users would have suffered because things would be taking longer than normal.

We’ve been running this in production since early June. Back then we had 25 users, where as now we have several thousand. Apart from updating the configuration to reflect service changes we do not deal with scaling – the minute to minute decisions are all done by guv. Not much is planned in terms of new features for guv, apart from some more tools to analyze configuration. For more info on using guv, see the README.

flattr this!

LinuxCon Europe – Day 1

attendee registration

The conference was opened by the LinuxFoundation’s Executive Jim Zemlin. He thanked the FSF for their 30 years of work. I was a little surprised to hear that, given the differences between OpenSource and Free Software. He continued by mentioning the 5 Billion Dollar report which calculates how much “value” the projects hosted at Linux Foundation have generated over the last five years. He said that a typical product contains 80%, 90%, or even more Free and Open Source Software. He also extended the list of projects by the Real Time Collaborative project which, as far as I understood, effectively means to hire Thomas Gleisxner to work on the Real Time Linux patches.

world without Linux

The next, very interesting, presentation was given by Sean Gourley, the founder of Quid, a business intelligence analytics company. He talked about the limits of human cognition and how algorithms help to exploit these limits. The limit is the speed of your thinking. He mentioned that studies measured the blood flow across the brain when making decisions which found differences depending on how proficient you are at a given task. They also found that you cannot be quicker than a certain limit, say, 650ms. He continued that the global financial market is dominated by algorithms and that a fibre cable from New York to London costs 300 million dollars to save 5 milliseconds. He then said that these algorithms make decisions at a speed we are unable to catch up with. In fact, the flash crash of 2:45 is inexplicable until today. Nobody knows what happened that caused a loss of trillions of dollars. Another example he gave was the crash of Knight Capital which caused a loss of 440 million dollars in 45 minutes only because they updated their trading algorithms. So algorithms are indeed controlling our lives which he underlined by saying that 61% of the traffic on the Internet is not generated by humans. He suggested that Bots would not only control the financial markets, but also news reading and even the writing of news. As an example he showed a Google patent for auto generating social status updates and how Mexican and Chinese propaganda bots would have higher volume tweets than humans. So the responsibilities are shifting and we’d be either working with an algorithm or for one. Quite interesting thought indeed.

man vs. machine

Next up was IBM on Transforming for the Digital Economy with Open Technology which was essentially a gigantic sales pitch for their new Power architecture. The most interesting bit of that presentation was that “IBM is committed to open”. This, she said, is visible through IBM’s portfolio and through its initiatives like the IBM Academic Initiative. OpenPower Foundation is another one of those. It takes the open development model of software and takes it further to everything related to the Power architecture (e.g. chip design), she said. They are so serious about being open, that they even trademarked “Open by Design“…

IBM sales pitch

Then, the drone code people presented on their drone project. They said that they’ve come a long way since 2008 and that the next years are going to fundamentally change the drone scene as many companies are involved now. Their project, DroneCode, is a stack from open hardware to flight control and the next bigger thing will be CAN support, which is already used in cards, planes, and other vehicles. The talk then moved to ROS, the robot operating system. It is the lingua franca for robotic in academia.


Matthew Garret talked on securing containers. He mentioned seccomp and what type of features you can deprive processes of. Nowadays, you can also reason about the arguments for the system call in question, so it might be more useful to people. Although, he said, writing a good seccomp policy is hard. So another mechanism to deprive processes of privileges is to set capabilities. It allows you to limit the privileges in a more coarse grained way and the behaviour is not very well defined. The combination of capabilities and seccomp might have surprising results. For example, you might be allowing the mknod() call, but you then don’t have the capability to actually execute it or vice versa. SELinux was next on his list as a mechanism to secure your containers. He said that writing SELinux policy is not the most fun thing in the world. Another option was to run your container in a virtual machine, but you then lose some benefits such as introspection of fine grained control over the processes. But you get the advantages of more isolation. Eventually, he asked the question of when to use what technology. The performance overhead of seccomp, SELinux, and capabilities are basically negligible, he said. Fully virtualising is usually more secure, he said, but the problem is that you have more complex infrastructure which tend to attract bugs. He also mentioned GRSecurity as a means of protecting your Linux kernel. Let’s hope it’ll be merged some day.


Canonical’s Daniel Watkins then talked on cloud-init. He said it runs in three stages. Init, config, and final in which init sets up networking, config does the actual configuration of your services, final is for the things that eventually need to be done. The clound-init architecture is apparently quite flexible and versatile. You can load your own configuration and user-data modules so that you can set up your cloud images as you like. cloud-init allows you get rid of custom images such that you can have confidence in your base image working as intended. In fact, it’s working not only with BSDs but also with Windows images. He said, it is somewhat similar to tools like Ansible, so if you are already happily using one of those, you’re good.


An entertaining talk was given by Florian Haas on LXC and containers. He talked about tricks managing your application containers and showed a problem when using a naive chroot which is that you get to see the host processes and networking information through the proc filesystem. With LXC, that problem is dealt with, he said. But then you have a problem when you update the host, i.e. you have to take down the container while the upgrade is running. With two nodes, he said, you can build a replication setup which takes care of failing over the node while it is upgrading. He argued that this is interesting for security reasons, because you can upgrade your software to not be vulnerable against “the latest SSL hack” without losing uptime. Or much of it, at least… But you’d need twice the infrastructure to run production. The future, he said, might be systemd with it’s nspawn tool. If you use systemd all the way, then you can use fleet to manage the instances. I didn’t take much away, personally, but I guess managing containers is all the rage right now.


Next up was Michael Hausenblas on Filesystems, SQL and NoSQL with Apache Mesos. I had briefly heard of Mesos, but I really didn’t know what it was. Not that I’m an expert now, but I guess I know that it’s a scheduler you can use for your infrastructure. Especially your Apache stack. Mesos addresses the problem of allocating resources to jobs. Imagine you have several different jobs to execute, e.g. a Web server, a caching layer, and some number crunching computation framework. Now suppose you want to increase the number crunching after hours when the Web traffic wears off. Then you can tell Mesos what type of resources you have and when you need that. Mesos would then go off and manage your machines. The alternative, he said, was to manually SSH into the machines and reprovision them. He explained some existing and upcoming features of Mesos. So again, a talk about managing containers, machines, or infrastructure in general.


The following Kernel panel didn’t provide much information to me. The moderation felt a bit stiff and the discussions weren’t really enganged. The topics mainly circled around maintainership, growth, and community.

Kernel Panel

SuSE’s Ralf was then talking on DevOps. He described his DevOps needs based on a cycle of planning, coding, building, testing, releasing, deploying, operating, monitoring, and then back to planning. When bringing together multiple projects, he said, they need to bring two independent integration loops together. When doing DevOps with a customer, he mentioned some companies who themselves provide services to their customers. In order to be successful when doing DevOps, you need, he said, Smart tools, Process automation, Open APIs, freedom of choice, and quality control are necessary. So I guess he was pitching for people to use “standards”, whatever that exactly means.

SuSE DevOps

I awaited the next talk on Patents and patent non aggression. Keith Bergelt, from OIN talked about ten years of the Open Invention Network. He said that ten years ago Microsoft sued Linux companies to hinder Linux distribution. Their network was founded to embrace patent non-aggression in the community. A snarky question would have been why it would not be simply enough to use GPLv3, but no questions were admitted. He said that the OIN has about 1750 licensees now with over a million patents being shared. That’s actually quite impressive and I hope that small companies are being protected from patent threats of big players…


That concluded the first day. It was a lot of talks and talking in the hallway. Video recordings are said to be made available in a couple of weeks. So keep watching the conference page.


IBM Booth

November 17, 2015

Announcing Changeset Consulting

I'm delighted to announce the launch of my new business. I am the founder of Changeset Consulting, LLC.

Changeset provides short-term project management services to free and open source software projects. Need to expedite the releases of new versions of software, write developer onboarding and user documentation, triage and respond to bugs, clean out the code review queue, or prioritize tasks for upcoming work? Changeset Consulting lightens the load on your maintainers.

Details about the services I offer, my past work, and useful resources I've made are at I'm seeking new clients and would love referrals.

For now the shop is just me, but I'm aiming to have enough income and work by summer 2016 to hire an intern or apprentice, and to eventually hire full-time staff. We'll see how it goes.

I highly recommend Galaxy Rise Consulting, the firm I hired to design my website. Much thanks to Shauna Gordon-McKeon, and to all the friends and family who encouraged me on my way here!

November 14, 2015

fwupd and DFU

For quite a long time fwupd has supported updating the system ‘BIOS’ using the UpdateCapsule UEFI mechanism. This open specification allows vendors provide a single update suitable for Windows and Linux, and the mechanism for applying it is basically the same for all vendors. Although there are only a few systems in the wild supporting capsule updates, a lot of vendors are planning new models next year, and a few of the major ones have been trialing the LVFS service for quite a while too. With capsule updates, fwupd and the LVFS we now have a compelling story for how to distribute and securely install system BIOS updates automatically.

It’s not such a rosy story for USB devices. In theory, everything should be using the DFU specification which has been endorsed by the USB consortium, but for a number of reasons quite a few vendors don’t use this. I’m guilty as charged for the ColorHug devices, as I didn’t know of the existance of DFU when designing the hardware. For ColorHug I just implemented a vendor-specific HID bootloader with a few custom commands as so many other vendors have done; it works well, but every vendor does things a slightly different way which needs having vendor specific update tools and fairly random firmware file formats.

With DFU, what’s supposed to happen is there are two modes for the device, a normal application runtime which is doing whatever the device is supposed to be doing, and another DFU mode which is really just an EEPROM programmer. By ‘detaching’ the application firmware using a special interface you can program the device and then return to normal operation.

So, what to do? For fwupd I want to ask vendors of removable hardware to implement DFU so that we don’t need to write code for each device type in fwupd. To make this a compelling prospect I’ve spent a good chunk of time of last week:

  • Creating a GObjectIntrospectable and cancellable host-side library called libdfu
  • Writing a reference GPLv3+ device-side implementation for a commonly used USB stack for PIC microcontrollers
  • Writing the interface code in fwupd to support DFU files wrapped in .cab files for automatic deployment

At the moment libdfu supports reading and writing raw, DFU and DfuSe file types, and supports reading and writing to DFU 1.1 devices. I’ve not yet implemented writing to ST devices (a special protocol extension invented by ST Microsystems) although that’s only because I’m waiting for someone to lend me a device with a STM32F107 included (e.g. DSO Nano). I’ve hopefully made the code flexible enough to make this possible without breaking API, although the libdfu library is currently private to fwupd until it’s had some proper review. You can of course use the dependable dfu-util tool to flash firmware, but this wasn’t suitable for use inside fwupd for various reasons.

Putting my money where my mouth is, I’ve converted the (not-yet-released) ColorHug+ bootloader and firmware to use DFU; excluding all the time I spent writing the m-stack patch and the libdfu support in fwupd it only took a couple of hours to build and test. Thanks to Christoph Brill, I’ll soon be getting some more hardware (a Neo FreeRunner) to verify this new firmware update mechanism on a real device with multiple implemented DFU interfaces. If anyone else has any DFU-capable hardware (especially Arduino-style devices) I’d be glad of any donations.

Once all this new code has settled down I’m going to be re-emailing a lot of the vendors who were unwilling to write vendor-specific code in fwupd. I’m trying to make the barrier to automatic updates on Linux as low as possible.

Comments welcome.

November 13, 2015

Coursera/Stanford course: Algorithms: Design and Analysis, Part 2

A few weeks ago I mentioned completing Part 1 of the online Coursera/Stanford “Algorithms: Design and Analysis” course. Part 2 of Algorithms: Design and Analysis isn’t due to start again until next year, but I didn’t want to wait, so I enrolled in the archived version of the course to watch the videos and do the assignments. I should be ready to just reuse my work when Part 2 starts again for real.

Part 2 was where things got really interesting. The assignments required implementing these algorithms, though the course covered others too:

  • A Greedy Algorithm for job scheduling.
  • Prim’s and Kruskal’s minimum spanning tree algorithms. (Both O(m log n) but Prims does better on dense graphs, with more edges.)
  • Modifying a minimum spanning tree to identify clusters.
  • The Knapsack problem (Dynamic Programming).
  • Shortest Path with the Belmann-Ford SSSP (Single-Source Shortest Path) algorithm (O(mn) and works with negative paths, but fails with negative cycles) as an alternative to Dijkstra’s Shortest Path algorithm (O(m log n) and works only with positive paths).
  • All Pairs Shortest Path with the Floyd Warshall algorithm (dynamic programming) (O(n3) and works with negative paths, though it fails when it detects negative cycles).
  • All Pairs Shortest path with Johnson’s algorithm, via one call to Belmann-Ford on a modified graph and repeated calls to Dijkstra on a reweighted graph. (O(mn log n) and works with negative paths, but fails with negative cycles.)
  • A dynamic programming algorithm for the Traveling Salesman Problem, both bottom up and recursive.
  • Local search with the 2Sat problem, using Papadimitriou’s Algorithm.

I particularly enjoyed exploring “dynamic programming”, which is really just avoiding unnecessary repeated work after you’ve identified the appropriate sub-problems. It’s identifying the sub problems that is really hard. I enjoyed playing with bottom-up dynamic programming (filling in an array as you go, often discarding the n-2th set of results as you go), and top-down dynamic programming, also known as memoization (usually recursing into sub problems and ideally not doing as many sub problems as you’d do working bottom up).

While implementing a dynamic programming solution for the Traveling Salesman Problem, I learned about Gosper’s Hack for iterating over subsets. It’s now a personal favorite in my toolbox.

As with part 1 of the course, I am not allowed to publish the code of my homework solutions. But I did create a public version of the knapsack problem for solving the Make Change problem without a canonical currency (not a real world set of coins), using dynamic programming, though you shouldn’t use that as a first way to understand the classic knapsack problem. I also implemented a simple greedy algorithm for the Make Change problem with a canonical currency (real world set of coins).

The Make Change problem was interesting because I’ve read that people can and should learn to recognize NP-Complete problems, such as the traveling salesman problem. However, it is not obvious which sets of coins would cause the Make Change problem to be solvable with a greedy algorithm and which would need dynamic programming, though it might at first seem like a minor detail. (I haven’t actually read that paper yet.)

I’ve also been reading through Steven Skiena’s The Algorithm Design Manual book, which I can highly recommend. It’s more practical and enjoyable than the classic Introduction to Algorithms book by Cormen et al.


November 12, 2015

The next big thing

GNOME 3 has come a long way since 3.0. When you compare what we have now to that initial 3.0 release, the difference is quite huge. Though it might look similar, the experience of using it has improved dramatically. This improvement is a major achievement, and has only happened thanks to the tireless efforts of the GNOME community.

With each release we’ve achieved a significant improvement to the overall experience. We’ve filled out the experience with major new features like system-wide search, the lock screen, online accounts, and a raft of new apps. Also, when necessary we’ve made major design changes, such as with notifications.

Much of this work has been a hard slog: reviewing every little bit of UI, and improving it one step at a time. We had round after round of Every Detail Matters to get the shell up to scratch, just as we’ve refined the apps with each subsequent release.

Not all of this work has been glamorous or exciting. Gradual refinement, release after release, isn’t something that makes the heart race. However, this kind of dedication has been absolutely essential, and it’s essential that it carries on: that we keep up the pace of improvement, so every version is better than the one before.

It’s important to remember the progress that we have made, because it is easy to forget how far we’ve come.

At the same time, when you are in the routine of making iterative improvements on a time-based schedule, it is easy to lose sight of the bigger changes that are happening around you. It is here that I want to draw attention to two significant developments that are currently happening in the GNOME project.

Application sandboxing

Alex’s incredible work with xdg-app is a subject that gets talked about a lot, but it really can’t be emphasised how significant it is for GNOME as a project.

The cross-platform and security aspects of application sandboxing are important, but there’s a lot more to it than that. It will enable us to improve the whole user experience, by enabling the OS to be more robust and stable, and by improving performance. It will also allow more flexible release cycles for applications, without having to wait for the next version of a specific distro, and it promises to transform the application developer experience.

This could have significant implications. Perhaps it will be the end of the six month release cycle for many GNOME applications?

GTK+ Scene Graph

Development of the GTK+ scene graph has been trundling along in the background for some time, thanks to Emmanuele. It is really starting to get somewhere now, and will allow interfaces that make greater use of animations.

There are some obvious uses for the GTK+ scene graph, but there are also many other possibilities and real scope for experimentation. The introduction of more widespread animations could have a revolutionary impact on how we design and create GNOME applications.

Putting it all together

Each of these initiatives has developed independently, and it’s easy to focus on them individually. However, I think that we should be viewing them as elements of something bigger: the next significant evolution of GNOME. We might just be on the verge of the next big thing, the next evolution of GNOME.

And since we are potentially at a major turning point for the platform, it is worth thinking about other changes that could form part of new vision for GNOME 3. For example, one thing I would particularly like to see is a greater emphasis on cloud integration, so it is possible to have a fully cloud-backed GNOME experience. I’m sure there are other changes that we should also be considering.

GNOME still needs to keep iterating: slowly and gradually improving every aspect of the user experience. At the same time, it is healthy to periodically think about the big picture and bigger changes. I think we have the basic ingredients for a new vision for GNOME 3: something new, exciting, and potentially revolutionary.

November 11, 2015

The Steam Machines has arrived! And you should get one!

So yesterday, the 10th of November, was the official launch day of the Steam Machines. The hardware are meant to be dedicated game machines for the living room taking advantage of the Steam ecosystem, to take on the Xbox One and PS4.

But for us in the Linux community these machines are more than that, they are an important part of helping us break into a broader market by paving the way for even more games and more big budget games coming to our platform. Playing computer games is not just a niche, it is a mainstream activity these days, and not having access to games on our platform has cost us quite a few users and potential contributors over the years. I have for instance met a lot of computer science students who ended up not using Linux as the main operating system during their studies simply due to the lack of games on the platform. Instead Linux got de-regulated to that thing in a VM only run when you needed it for an assignment.

Steam for Linux and SteamOS can and will be important pieces of breaking through that. SteamOS and the Steam Macines are also important for the Linux community for another reason. They can help funnel more resources from hardware companies into Linux drivers and support. I know for instance that all the 3 major GPU vendors have increased their Linux drivers investments due to SteamOS.

So I want to congratulate Valve on the launch of the first Steam Machines and strongly recommend everyone in the community to get a Steam machine for their home!

People who have had a good chance to test the hardware has recommended me to get one of the Alienware SteamOS systems, so I am passing that recommendation onwards.

As a sidenote we are also working on a few features in Fedora Workstation to make it a better host for Steam and Steam games. This includes our work on the GL Dispatch and Optimus support as covered in a previous blog and libratbag, our new library for handling gaming mice under Linux. And finally we are working on a few bug fixes in Fedora to make it an even better host for the Steam client related to C++ ABI issues.

November 09, 2015

systemd.conf 2015 Summary

systemd.conf 2015 is Over Now!

Last week our first systemd.conf conference took place at betahaus, in Berlin, Germany. With almost 100 attendees, a dense schedule of 23 high-quality talks stuffed into a single track on just two days, a productive hackfest and numerous consumed Club-Mates I believe it was quite a success!

If you couldn't attend the conference, you may watch all talks on our YouTube Channel. The slides are available online, too.

Many photos from the conference are available on the Google Events Page. Enjoy!

I'd specifically like to thank Daniel Mack, Chris Kühl and Nils Magnus for running the conference, and making sure that it worked out as smoothly as it did! Thank you very much, you did a fantastic job!

I'd also specifically like to thank the CCC Video Operation Center folks for the excellent video coverage of the conference. Not only did they implement a live-stream for the entire talks part of the conference, but also cut and uploaded videos of all talks to our YouTube Channel within the same day (in fact, within a few hours after the talks finished). That's quite an impressive feat!

The folks from LinuxTag e.V. put a lot of time and energy in the organization. It was great to see how well this all worked out! Excellent work!

(BTW, LinuxTag e.V. and the CCC Video Operation Center folks are willing to help with the organization of Free Software community events in Germany (and Europe?). Hence, if you need an entity that can do the financial work and other stuff for your Free Software project's conference, consider pinging LinuxTag, they might be willing to help. Similar, if you are organizing such an event and are thinking about providing video coverage, consider pinging the the CCC VOC folks! Both of them get our best recommendations!)

I'd also like to thank our conference sponsors! Specifically, we'd like to thank our Gold Sponsors Red Hat and CoreOS for their support. We'd also like to thank our Silver Sponsor Codethink, and our Bronze Sponsors Pengutronix, Pantheon, Collabora, Endocode, the Linux Foundation, Samsung and Travelping, as well as our Cooperation Partners LinuxTag and, and our Media Partner

Last but not least I'd really like to thank our speakers and attendees for presenting and participating in the conference. Of course, the conference we put together specifically for you, and we really hope you had as much fun at it as we did!

Thank you all for attending, supporting, and organizing systemd.conf 2015! We are looking forward to seeing you and working with you again at systemd.conf 2016!


embracing conway's law

Most of you have heard of "Conway's Law", the pithy observation that the structure of things that people build reflects the social structure of the people that build them. The extent to which there is coordination or cohesion in a system as a whole reflects the extent to which there is coordination or cohesion among the people that make the system. Interfaces between components made by different groups of people are the most fragile pieces. This division goes down to the inner life of programs, too; inside it's all just code, but when a program starts to interface with the outside world we start to see contracts, guarantees, types, documentation, fixed programming or binary interfaces, and indeed faults as well: how many bug reports end up in an accusation that team A was not using team B's API properly?

If you haven't heard of Conway's law before, well, welcome to the club. Inneresting, innit? And so thought I until now; a neat observation with explanatory power. But as aspiring engineers we should look at ways of using these laws to build systems that take advantage of their properties.

in praise of bundling

Most software projects depend on other projects. Using Conway's law, we can restate this to say that most people depend on things built by other people. The Chromium project, for example, depends on many different libraries produced by many different groups of people. But instead of requiring the user to install each of these dependencies, or even requiring the developer that works on Chrome to have them available when building Chrome, Chromium goes a step further and just includes its dependencies in its source repository. (The mechanism by which it does this isn't a direct inclusion, but since it specifies the version of all dependencies and hosts all code on Google-controlled servers, it might as well be.)

Downstream packagers like Fedora bemoan bundling, but they ignore the ways in which it can produce better software at lower cost.

One way bundling can improve software quality is by reducing the algorithmic complexity of product configurations, when expressed as a function of its code and of its dependencies. In Chromium, a project that bundles dependencies, the end product is guaranteed to work at all points in the development cycle because its dependency set is developed as a whole and thus uniquely specified. Any change to a dependency can be directly tested against the end product, and reverted if it causes regressions. This is only possible because dependencies have been pulled into the umbrella of "things the Chromium group is responsible for".

Some dependencies are automatically pulled into Chrome from their upstreams, like V8, and some aren't, like zlib. The difference is essentially social, not technical: the same organization controls V8 and Chrome and so can set the appropriate social expectations and even revert changes to upstream V8 as needed. Of course the goal of the project as a whole has technical components and technical considerations, but they can only be acted on to the extent they are socially reified: without a social organization of the zlib developers into the Chromium development team, Chromium has no business automatically importing zlib code, because the zlib developers aren't testing against Chromium when they make a release. Bundling zlib into Chromium lets the Chromium project buffer the technical artifacts of the zlib developers through the Chromium developers, thus transferring responsibility to Chromium developers as well.

Conway's law predicts that the interfaces between projects made by different groups of people are the gnarliest bits, and anyone that has ever had to maintain compatibility with a wide range of versions of upstream software has the scar tissue to prove it. The extent to which this pain is still present in Chromium is the extent to which Chromium, its dependencies, and the people that make them are not bound tightly enough. For example, making a change to V8 which results in a change to Blink unit tests is a three-step dance: first you commit a change to Blink giving Chromium a heads-up about new results being expected for the particular unit tests, then you commit your V8 change, then you commit a change to Blink marking the new test result as being the expected one. This process takes at least an hour of human interaction time, and about 4 hours of wall-clock time. This pain would go away if V8 were bundled directly into Chromium, as you could make the whole change at once.

forking considered fantastic

"Forking" sometimes gets a bad rap. Let's take the Chromium example again. Blink forked from WebKit a couple years ago, and things have been great in both projects since then. Before the split, the worst parts in WebKit were the abstraction layers that allowed Google and Apple to use the dependencies they wanted (V8 vs JSC, different process models models, some other things). These abstraction layers were the reified software artifacts of the social boundaries between Google and Apple engineers. Now that the social division is gone, the gnarly abstractions are gone too. Neither group of people has to consider whether the other will be OK with any particular change. This eliminates a heavy cognitive burden and allows both projects to move faster.

As a pedestrian counter-example, Guile uses the libltdl library to abstract over the dynamic loaders of different operating systems. (Already you are probably detecting the Conway's law keywords: uses, library, abstract, different.) For years this library has done the wrong thing while trying to do the right thing, ignoring .dylib's but loading .so's on Mac (or vice versa, I can't remember), not being able to specify soversions for dependencies, throwing a stat party every time you load a library because it grovels around for completely vestigial .la files, et cetera. We sent some patches some time ago but the upstream project is completely unmaintained; the patches haven't been accepted, users build with whatever they have on their systems, and though we could try to take over upstream it's a huge asynchronous burden for something that should be simple. There is a whole zoo of concepts we don't need here and Guile would have done better to include libltdl into its source tree, or even to have forgone libltdl and just written our own thing.

Though there are costs to maintaining your own copy of what started as someone else's work, people who yammer on against forks usually fail to recognize their benefits. I think they don't realize that for a project to be technically cohesive, it needs to be socially cohesive as well; anything else is magical thinking.

not-invented-here-syndrome considered swell

Likewise there is an undercurrent of smarmy holier-than-thou moralism in some parts of the programming world. These armchair hackers want you to believe that you are a bad person if you write something new instead of building on what has already been written by someone else. This too is magical thinking that comes from believing in the fictional existence of a first-person plural, that there is one "we" of "humanity" that is making linear progress towards the singularity. Garbage. Conway's law tells you that things made by different people will have different paces, goals, constraints, and idiosyncracies, and the impedance mismatch between you and them can be a real cost.

Sometimes these same armchair hackers will shake their heads and say "yeah, project Y had so much hubris and ignorance that they didn't want to bother understanding what X project does, and they went and implemented their own thing and made all their own mistakes." To which I say, so what? First of all, who are you to judge how other people spend their time? You're not in their shoes and it doesn't affect you, at least not in the way it affects them. An armchair hacker rarely understands the nature of value in an organization (commercial or no). People learn more when they write code than when they use it or even when they read it. When your product has a problem, where will you find the ability to fix it? Will you file a helpless bug report or will you be able to fix it directly? Assuming your software dependencies model some part of your domain, are you sure that their models are adequate for your purpose, with the minimum of useless abstraction? If the answer is "well, I'm sure they know what they're doing" then if your organization survives a few years you are certain to run into difficulties here.

One example. Some old-school Mozilla folks still gripe at Google having gone and created an entirely new JavaScript engine, back in 2008. This is incredibly naïve! Google derives immense value from having JS engine expertise in-house and not having to coordinate with anyone else. This control also gives them power to affect the kinds of JavaScript that gets written and what goes into the standard. They would not have this control if they decided to build on SpiderMonkey, and if they had built on SM, they would have forked by now.

As a much more minor, insignificant, first-person example, I am an OK compiler hacker now. I don't consider myself an expert but I do all right. I got here by making a bunch of mistakes in Guile's compiler. Of course it helps if you get up to speed using other projects like V8 or what-not, but building an organization's value via implementation shouldn't be discounted out-of-hand.

Another point is that when you build on someone else's work, especially if you plan on continuing to have a relationship with them, you are agreeing up-front to a communications tax. For programmers this cost is magnified by the degree to which asynchronous communication disrupts flow. This isn't to say that programmers can't or shouldn't communicate, of course, but it's a cost even in the best case, and a cost that can be avoided by building your own.

When you depend on a project made by a distinct group of people, you will also experience churn or lag drag, depending on whether the dependency changes faster or slower than your project. Depending on LLVM, for example, means devoting part of your team's resources to keeping up with the pace of LLVM development. On the other hand, depending on something more slow-moving can make it more difficult to work with upstream to ensure that the dependency actually suits your use case. Again, both of these drag costs are magnified by the asynchrony of communicating with people that probably don't share your goals.

Finally, for projects that aim to ship to end users, depending on people outside your organization exposes you to risk. When a security-sensitive bug is reported on some library that you use deep in your web stack, who is responsible for fixing it? If you are responsible for the security of a user-facing project, there are definite advantages for knowing who is on the hook for fixing your bug, and knowing that their priorities are your priorities. Though many free software people consider security to be an argument against bundling, I think the track record of consumer browsers like Chrome and Firefox is an argument in favor of giving power to the team that ships the product. (Of course browsers are terrifying security-sensitive piles of steaming C++! But that choice was made already. What I assert here is that they do well at getting security fixes out to users in a timely fashion.)

to use a thing, join its people

I'm not arguing that you as a software developer should never use code written by other people. That is silly and I would appreciate if commenters would refrain from this argument :)

Let's say you have looked at the costs and the benefits and you have decided to, say, build a browser on Chromium. Or re-use pieces of Chromium for your own ends. There are real costs to doing this, but those costs depend on your relationship with the people involved. To minimize your costs, you must somehow join the community of people that make your dependency. By joining yourself to the people that make your dependency, Conway's law predicts that the quality of your product as a whole will improve: there will be fewer abstraction layers as your needs are taken into account to a greater degree, your pace will align with the dependency's pace, and colleagues at Google will review for you because you are reviewing for them. In the case of Opera, for example, I know that they are deeply involved in Blink development, contributing significantly to important areas of the browser that are also used by Chromium. We at Igalia do this too; our most successful customers are those who are able to work the most closely with upstream.

On the other hand, if you don't become part of the community of people that makes something you depend on, don't be surprised when things break and you are left holding both pieces. How many times have you heard someone complain the "project A removed an API I was using"? Maybe upstream didn't know you were using it. Maybe they knew about it, but you were not a user group they cared about; to them, you had no skin in the game.

Foundations that govern software projects are an anti-pattern in many ways, but they are sometimes necessary, born from the need for mutually competing organizations to collaborate on a single project. Sometimes the answer for how to be able to depend on technical work from others is to codify your social relationship.

hi haters

One note before opening the comment flood: I know. You can't control everything. You can't be responsible for everything. One way out of the mess is just to give up, cross your fingers, and hope for the best. Sure. Fine. But know that there is no magical first-person-plural; Conway's law will apply to you and the things you build. Know what you're actually getting when you depend on other peoples' work, and know what you are paying for it. One way or another, pay for it you must.

November 08, 2015

Adding as a remote to a local git repository, a pushable remote repository residing on a sshable server

Many people ask me how do I manage my work repositories since it is advisable to do all development on your PC and a headache to trace any change, from local repository to the test machine using scp or rsync each time a change is made. Fortunately, git has a easier way! The alternative is to add as a remote to a local git repository, a pushable remote repository residing on a sshable server. The steps are as follows:

//At the server's ssh
mkdir some_project
cd some_project/
git init
git checkout -b test #as we cannot push to a checked out branch at the server from our laptops.

//On my pc
git clone git/url/to/some_project.git 
cd some_project/
#Add pushable repo. which we inited on the server, as a remote to this local repo.
git remote add remote_name user@remote.server:/path/to/GIT/repo/some_project/
#check if added
git remote -vv
#Clear firewall to remote server, if needed
#push desired branch(es)(say master) to the init-ed blank repo on the remote server.
git push remote_name master 

//At the server ssh prompt to see the changes
git checkout master 

Voila! There's all your work!
Next time, you make any changes, in a separate branch, you just need to perform the push step at your PC and checkout the new branch on your server :D

November 07, 2015

Importing include paths in Eclipse

First of all, let me be clear: no, I’m not trying to leave Emacs again, already got over that stage. Emacs is and will be my main editor for the foreseeable future, as it’s clear to me that there’s no other editor I feel more comfortable with, which is why I spent some time cleaning up my .emacs.d and making it more “manageable”.

But as much as like Emacs as my main “weapon”, I sometimes appreciate the advantages of using a different kind of beast for specific purposes. And, believe me or not, in the past 2 years I learned to love Eclipse/CDT as the best work-mate I know when I need some extra help to get deep inside of the two monster C++ projects that WebKit and Chromium are. And yes, I know Eclipse is resource hungry, slow, bloated… and whatnot; but I’m lucky enough to have fast SSDs and lots of RAM in my laptop & desktop machines, so that’s not really a big concern anymore for me (even though I reckon that indexing chromium in the laptop takes “quite some time”), so let’s move on :-)

However, there’s this one little thing that still bothers quite me a lot of Eclipse: you need to manually setup the include paths for the external dependencies not in a standard location that a C/C++ project uses, so that you can get certain features properly working such as code auto-completion, automatic error-checking features, call hierarchies… and so forth.

And yes, I know there is an Eclipse plugin adding support for pkg-config which should do the job quite well. But for some reason I can’t get it to work with Eclipse Mars, even though others apparently can use it there for some reason (and I remember using it with Eclipse Juno, so it’s definitely not a myth).

Anyway, I did not feel like fighting with that (broken?) plugin, and in the other hand I was actually quite inclined to play a bit with Python so… my quick and dirty solution to get over this problem was to write a small script that takes a list of package names (as you would pass them to pkg-config) and generates the XML content that you can use to import in Eclipse. And surprisingly, that worked quite well for me, so I’m sharing it here in case someone else finds it useful.

Using frogr as an example, I generate the XML file for Eclipse doing this:

  $ pkg-config-to-eclipse glib-2.0 libsoup-2.4 libexif libxml-2.0 \
        json-glib-1.0 gtk+-3.0 gstreamer-1.0 > frogr-eclipse.xml

…and then I simply import frogr-eclipse.xml from the project’s properties, inside the C/C++ General > Paths and Symbols section.

After doing that I get rid of all the brokenness caused by so many missing symbols and header files, I get code auto-completion nicely working back again and all those perks you would expect from this little big IDE. And all that without having to go through the pain of defining all of them one by one from the settings dialog, thank goodness!

Now you can quickly see how it works in the video below:

VIDEO: Setting up a C/C++ project in Eclipse with pkg-config-to-eclipse

This has been very helpful for me, hope it will be helpful to someone else too!

November 06, 2015

Why improving kernel security is important

The Washington Post published an article today which describes the ongoing tension between the security community and Linux kernel developers. This has been roundly denounced as FUD, with Rob Graham going so far as to claim that nobody ever attacks the kernel.

Unfortunately he's entirely and demonstrably wrong, it's not FUD and the state of security in the kernel is currently far short of where it should be.

An example. Recent versions of Android use SELinux to confine applications. Even if you have full control over an application running on Android, the SELinux rules make it very difficult to do anything especially user-hostile. Hacking Team, the GPL-violating Italian company who sells surveillance software to human rights abusers, found that this impeded their ability to drop their spyware onto targets' devices. So they took advantage of the fact that many Android devices shipped a kernel with a flawed copy_from_user() implementation that allowed them to copy arbitrary userspace data over arbitrary kernel code, thus allowing them to disable SELinux.

If we could trust userspace applications, we wouldn't need SELinux. But we assume that userspace code may be buggy, misconfigured or actively hostile, and we use technologies such as SELinux or AppArmor to restrict its behaviour. There's simply too much userspace code for us to guarantee that it's all correct, so we do our best to prevent it from doing harm anyway.

This is significantly less true in the kernel. The model up until now has largely been "Fix security bugs as we find them", an approach that fails on two levels:

1) Once we find them and fix them, there's still a window between the fixed version being available and it actually being deployed
2) The forces of good may not be the first ones to find them

This reactive approach is fine for a world where it's possible to push out software updates without having to perform extensive testing first, a world where the only people hunting for interesting kernel vulnerabilities are nice people. This isn't that world, and this approach isn't fine.

Just as features like SELinux allow us to reduce the harm that can occur if a new userspace vulnerability is found, we can add features to the kernel that make it more difficult (or impossible) for attackers to turn a kernel bug into an exploitable vulnerability. The number of people using Linux systems is increasing every day, and many of these users depend on the security of these systems in critical ways. It's vital that we do what we can to avoid their trust being misplaced.

Many useful mitigation features already exist in the Grsecurity patchset, but a combination of technical disagreements around certain features, personality conflicts and an apparent lack of enthusiasm on the side of upstream kernel developers has resulted in almost none of it landing in the kernels that most people use. Kees Cook has proposed a new project to start making a more concerted effort to migrate components of Grsecurity to upstream. If you rely on the kernel being a secure component, either because you ship a product based on it or because you use it yourself, you should probably be doing what you can to support this.

Microsoft received entirely justifiable criticism for the terrible state of security on their platform. They responded by introducing cutting-edge security features across the OS, including the kernel. Accusing anyone who says we need to do the same of spreading FUD is risking free software being sidelined in favour of proprietary software providing more real-world security. That doesn't seem like a good outcome.

comment count unavailable comments

Gadget reviews

Not that I'm really running after more gadgets, but sometimes, there is a need that could only be soothed through new hardware.

Bluetooth UE roll

Got this for my wife, to play music when staying out on the quays of the Rhône, playing music in the kitchen (from a phone or computer), or when she's at the photo lab.

It works well with iOS, MacOS X and Linux. It's very easy to use, with whether it's paired, connected completely obvious, and the charging doesn't need specific cables (USB!).

I'll need to borrow it to add battery reporting for those devices though. You can find a full review on Ars Technica.

Sugru (!)

Not a gadget per se, but I bought some, used it to fix up a bunch of cables, repair some knickknacks, and do some DIY. Highly recommended, especially given the current price of their starter packs.

15-pin to USB Joystick adapter

It's apparently from Ckeyin, but you'll find the exact same box from other vendors. Made my old Gravis joystick work, in the hope that I can make it work with DOSBox and my 20-year old copy of X-Wing vs. Tie Fighter.

Microsoft Surface ARC Mouse

That one was given to me, for testing, works well with Linux. Again, we'll need to do some work to report the battery. I only ever use it when travelling, as the batteries last for absolute ages.

Logitech K750 keyboard

Bought this nearly two years ago, and this is one of my best buys. My desk is close to a window, so it's wireless but I never need to change the batteries or think about charging it. GNOME also supports showing the battery status in the Power panel.

Logitech T650 touchpad

Got this one in sale (17€), to replace my Logitech trackball (one of its buttons broke...). It works great, and can even get you shell gestures when run in Wayland. I'm certainly happy to have one less cable running across my desk, and reuses the same dongle as the keyboard above.

If you use more than one devices, you might be interested in this bug to make it easier to support multiple Logitech "Unifying" devices.

ClicLite charger

Got this from a design shop in Berlin. It should probably have been cheaper than what I paid for it, but it's certainly pretty useful. Charges up my phone by about 20%, it's small, and charges up at the same time as my keyboard (above).

Dell S2340T

Bought about 2 years ago, to replace the monitor I had in an all-in-one (Lenovo all-in-ones, never buy that junk).

Nowadays, the resolution would probably be considered a bit on the low side, and the touchscreen mesh would show for hardcore photography work. It's good enough for videos though and the speaker reaches my sitting position.

It's only been possible to use the USB cable for graphics for a couple of months, and it's probably not what you want to lower CPU usage on your machine, but it works for Fedora with this RPM I made. Talk to me if you can help get it into RPMFusion.

Shame about the huge power brick, but a little bonus for the builtin Ethernet adapter.

Surface 3

This is probably the biggest ticket item. Again, I didn't pay full price for it, thanks to coupons, rewards, and all. The work to getting Linux and GNOME to play well with it is still ongoing, and rather slow.

I won't comment too much on Windows either, but rather as what it should be like once Linux runs on it.

I really enjoy the industrial design, maybe even the slanted edges, but one as to wonder why they made the USB power adapter not sit flush with the edge when plugged in.

I've used it a couple of times (under Windows, sigh) to read Pocket as I do on my iPad 1 (yes, the first one), or stream videos to the TV using Flash, without the tablet getting hot, or too slow either. I also like the fact that there's a real USB(-A) port that's separate from the charging port. The micro SD card port is nicely placed under the kickstand, hard enough to reach to avoid it escaping the tablet when lugged around.

The keyboard, given the thickness of it, and the constraints of using it as a cover, is good enough for light use, when travelling for example, and the layout isn't as awful as on, say, a Thinkpad Carbon X1 2nd generation. The touchpad is a bit on the small side though it would have been hard to make it any bigger given the cover's dimensions.

I would however recommend getting a Surface Pro if you want things to work right now (or at least soon). The one-before-last version, the Surface Pro 3, is probably a good target.

November 05, 2015

Native file choosers in Gtk+

Recently I have been working on support for a filechooser portal for sandboxed applications (in xdg-app). The way these work is that the application triggers a file chooser, then the actual user interaction happens outside the sandbox, and the application only recieves the file data after the user finished the interaction.

Ideally something like this would be completely hidden by the toolkit, and the application would just use the regular file chooser APIs. However, the Gtk+ filechooser APIs expose too much details about the file chooser dialog, which means it has to be a regular in-process widget. Unfortunately this means we can’t replace it by an out-of-process dialog.

What we need is a Gtk+ API for the file chooser that hides the details of how the dialog works. At this point I realized that this is something that has been requested a lot in a different context. Such an API would allow us to plug in platform-native file chooser dialogs.

So, I got to work, and today I landed support for native Windows file choosers in Gtk:

win32 file chooser in gtkWe will also look at implementing an OSX version of this to ensure that the APIs work for the common cases.

Now that we have this people can start porting their applications (which is quite trivial in the common case, as the APIs is very similar to the old APIs). And once the apps are ported they will automatically get support for the filechooser portal to make them work in a sandboxed environment.

So, anyone maintaining a Gtk 3 application that want to work better on other platforms, or when sandboxed should take a look at the new GtkFileChooserNative API. Right now it only exists in git, but once there is a new release it should show up in the API docs.

attribute((cleanup)), mixed declarations and code, and goto.

One of the cool features of recent GLib is g_autoptr() and g_autofree. It’s liberating to be able to write:

g_autofree char *filename = g_strdup_printf("%s/%d.txt", dir, count);

And be sure that will be freed no matter how your function returns. But as I started to use it, I realized that I wasn’t very sure about some details about the behavior, especially when combined with mixing declarations and code as allowed by C99.

Internally g_autofree uses __attribute__((cleanup)), which is supported by GCC and clang. The definition of g_autofree is basically:

static inline void
g_autoptr_cleanup_generic_gfree (void *p)
  void **pp = (void**)p;
  g_free (*pp);

#define g_autofree __attribute__((cleanup(g_autoptr_cleanup_generic_gfree)))

Look at the following examples:

int count1(int arg)
  g_autofree char *str;

  if (arg < 0)
    return -1;

  str = g_strdup_printf("%d", arg);

  return strlen(str);

int count2(int arg)
  if (arg < 0)
    return -1;

  g_autofree char *str = g_strdup_printf("%d", arg);

  return strlen(str);

int count3(int arg)
  if (arg < 0)
    goto out;

  g_autofree char *str = g_strdup_printf("%d", arg);

  return strlen(str);

  return -1;

int count4(int arg)
  if (arg < 0)
    goto out;

    g_autofree char *str = g_strdup_printf("%d", arg);

    return strlen(str);

  return 0;

Which ones of these do you think work as intended, and which ones are buggy? (I’m not recommending this as a way counting the digits in a number – the example is artificial.)

count1() is pretty clearly buggy – the cleanup function will run in the error
path and try to free an uninitialized string. Slightly, more subtly, count3() is also buggy – because the goto jumps over the initialization. But count2() and count4() work as intended.

To understand why this is the case, it’s worth looking at how attribute((cleanup)) is described in the GCC manual – all it says is “the ‘cleanup’ attribute runs a function when the variable goes out of scope.” I first thought that this was a completely insufficient definition – not complete enough to allow figuring out what was supposed to happen in the above cases, but thinking about it a bit, it’s actually a precise definition.

To recall, the scope of a variable in C is from the point of the declaration of the variable to the end of the enclosing block. What the definition is saying is that any time a variable is in scope, and then goes out of scope, there is an implicit call to the cleanup function.

In the early return in count1() and at the return that is jumped to in count3(), the variable ‘str’ is in scope, so the cleanup function will be called, even though the variable is not initialized in either case. In the corresponding places in count2() and count4() the variable ‘str’ is not in scope, so the cleanup function will not be called.

The coding style takewaways from this are 1) Don’t use the g_auto* attributes on a variable that is not initialzed at the time of definition 2) be very careful if combining goto with g_auto*.

It should be noted that GCC is quite good at warning about it if you get it wrong, but it’s still better to understand the rules and get it right from the start.

November 04, 2015

World Usability Day on November 12

If you are in the Twin Cities area, I would like to share this special announcement about World Usability Day at the University of Minnesota, on November 12.
The University of Minnesota will host its tenth annual World Usability Day at the Saint Paul Student Center Theater on Nov. 12 from 9:30 a.m. - 4 p.m. This global event aims to unite communities around the premise that the services and products important to life should be easier to access and simpler to use.

The theme for 2015 is innovation, and the day’s agenda centers on the many ways that innovation can enhance the human experience. Panels and presentations will highlight innovation in usability and user experience. Scheduled break times will offer the opportunity for conversation and networking.

Discussion topics will include:
  • Is innovation just a buzzword for novelty?
  • How do you teach/train/credential/measure innovation?
  • Planning a Space for Innovation: how do we make an environment conducive to this thinking?
  • Design Thinking
  • Innovation in Usability; Combating Bias and Artificiality, Gaining Actionable Value
This is a free event to all members of the University community.

GNOME’s newcomer guide is improving

Me, Carlos and others has been working on improving the newcomer experience. GNOME is a 15 years old project. Our community conventions and infrastructure which newcomers have to learn all bear signs of this. Over time such project in the size of GNOME build up vast amount of information, historial baggage and complex navigation structure. To me this is all a natural healthy sign of a large project swiftly advancing the Linux desktop.

However, this creates a high barrier of entry for outsiders and ultimately new contributors are necessary to keep GNOME sustainable. The newcomer experience effort aims to simplify the complex navigation, concentrate the information and remove historical baggage. Ideally this effort should spread to every type of contributor, but for now I am focusing on developers.


Introducing Newcomers Guide

In the spirit of simplification, the GNOME Love effort have been renamed to the Newcomers effort. The effort does what it says on the tin. All documentation has been moved from /GnomeLove/ to /Newcomers/ on the GNOME wiki and the IRC channel for newcomers is #newcomers. It is also planned that the “gnome-love” tag should change “newcomers”.

Furthermore the GnomeLove documentation has been restructured. Getting started contributing has been divided into 4 steps: Choose a project, Build the project, Solve a task, Send a patch. The simplicity here is very necessary. Human attention space is small and simplicity makes a large project like GNOME less intimidating.


The first step, Choose a project is a revamped page of the previous project/mentor list. This section presents a small list of projects that newcomers can get involved with. It is important to realize that not all GNOME projects are newcomer-friendly, which is very natural. Ideally a project should only be listed here if it is actively maintained. Concretely by “actively maintained” I mean:

  • The project has at least one mentor who help newcomers in #newcomers and the project chatroom.
  • The project’s wiki page is welcoming and presents the information you need.
  • The project’s bug tracker contains newcomer bugs, so newcomers can find tasks easily.
  • The project can is easily buildable.

Currently the section lists 10 projects. The plan is to review the infrastructure of each of these projects and ensure the list is kept concise.


After choosing the page, the newcomers are led onto the second step: Building the project. This is our newcomer JHbuild guide. Then the third Step, Solve a task, is a completely new page aiming to describe a general workflow when solving bugs in a GNOME project. The page also introduces the developer tools we use in GNOME.


Finally the 4th step, Submit the patch, introduces GNOME’s use of Git and how to submit patches to a bug on GNOME’s bugzilla.

What’s up next

With the 4 described steps above, the core of /Newcomers/ is now in place for the new developer experience. The plan is to actively maintain this core and continue to concentrate its contents.

The next immediate step is to look at external links which the Newcomers guide link to. GNOME Developer Center in particular needs a serious refresh. For now, instead of designing the ideal, I started making practical mockups for revamping GNOME Developer Center, reusing as much existing infrastructure as much as possible.


This seemed plausible to me, but they might not be as easily achievable as I hoped. However Frederic Peters is looking into making library-web which GNOME Developer Center is generated from, Django based.

Another item on my wishlist is the ability to build and develop applications without all the dependency mess. Most newcomers get stuck at step 2, Build the project. My own hope is that XDG-app, an SDK and Builder can help us on the way here. It would also be great if we could spread the effort to cover Documentation, Design, etc but for now I’ll be focusing my own efforts into the developer experience.

Feel free to join the efforts, fx by coordinating with me or csoriano at #newcomers. There’s always newcomers visiting and there’s plenty of external documentation we need to revise. By participating you’re ensuring GNOME’s future.

November 03, 2015

two paths, one peak: a view from below on high-performance language implementations

Ohmigod it's November. Time flies amirite. Eck-setra. These are not actually my sentiments but sometimes I do feel like a sloth or a slow loris, grasping out at quarter-speed. Once I get a hold it's good times, but hoo boy. The tech world churns and throws up new languages and language implementations every year and how is it that in 2015, some 20 years after the project was started, Guile still doesn't do native compilation?

Though I've only been Guiling for the last 10 years or so, this article aims to plumb those depths; and more than being an apology or a splain I want to ponder the onward journey from the here and the now. I was going to write something like "looking out from this peak to the next higher peak" but besides being a cliché that's exactly what I don't mean to do. In Guile performance work I justify my slow loris grip by a mistrust of local maxima. I respect and appreciate the strategy of going for whatever gains you the most in the short term, especially in a commercial context, but with a long view maybe this approach is a near win but a long lose.

That's getting ahead of myself; let's get into this thing. We started byte-compiling Guile around 2008 or so. Guile is now near to native compilation. Where are we going with this thing?

short term: template jit

The obvious next thing to do for Guile would be to compile its bytecodes to machine code using a template JIT. This strategy just generates machine code for each bytecode instruction without regard to what comes before or after. It's dead simple. Guile's bytecode is quite well-suited to this technique, even, in the sense that an instruction doesn't correspond to much code. As Guile has a register-based VM, its instructions will also specialize well against their operands when compiled to native code. The only global state that needs to be carried around at runtime is the instruction pointer and the stack pointer, both of which you have already because of how modern processors work.

Incidentally I have long wondered why CPython doesn't have a template JIT. Spiritually I am much more in line with the PyPy project but if I were a CPython maintainer, I would use a template JIT on the bytecodes I already have. Using a template JIT preserves the semantics of bytecode, including debugging and introspection. CPython's bytecodes are at a higher level than Guile's though, with many implicit method/property lookups (at least the last time I looked at them), and so probably you would need to add inline caches as well; but no biggie. Why aren't the CPython people doing this? What is their long-term perf story anyway -- keep shovelling C into the extension furnace? Lose to PyPy?

In the case of Guile we are not yet grasping in this direction because we don't have (direct) competition from PyPy :) But also there are some problems with a template JIT. Once you internalize the short-term mentality of a template JIT you can get stuck optimizing bytecode, optimizing template JIT compilation, and building up a baroque structure that by its sheer mass may prevent you from ever building The Right Thing. You will have to consider how a bytecode-less compilation pipeline interacts with not only JITted code but also bytecode, because it's a lose to do a template JIT for code that is only executed once.

This sort of short-term thinking is what makes people also have to support on-stack replacement (OSR), also known as hot loop transfer. The basic idea is that code that executes often has to be JITted to go fast, but you can't JIT everything because that would be slow. So you wait to compile a function until it's been called a few times; fine. But with loops it could be that a function is called just once but a loop in the function executes many times. You need to be able to "tier up" to the template JIT from within a loop. This complexity is needed at the highest performance level, but if you choose to do a template JIT you're basically committing to implementing OSR early on.

Additionally the implementation of a template JIT compiler is usually a bunch of C or C++ code. It doesn't make sense to include a template JIT in a self-hosted system that compiles to bytecode, because it would be sad to have the JIT not be written in the source language (Guile Scheme in our case).

Finally in Scheme we have tail-call and delimited continuation considerations. Currently in Guile all calls happen in the Guile bytecode interpreter, which makes tail calls easy -- the machine frame stays the same and we just have to make a tail call on the Scheme frame. This is fine because we don't actually control the machine frame (the C frame) of the bytecode interpreter itself -- the C compiler just does whatever it does. But how to tail call between the bytecode interpreter and JIT-compiled code? You'd need to add a trampoline beneath both the C interpreter and any entry into compiled code that would trampoline to the other implementation, depending on how the callee "returns". And how would you capture stack slices with delimited continuations? It's possible (probably -- I don't know how to reinstate a delimited continuation with both native and interpreted frames), but something of a headache, and is it really necessary?

if you compile ahead-of-time anyway...

The funny thing about CPython is that, like Guile, it is actually an ahead-of-time compiler. While the short-term win would certainly be to add a template JIT, because the bytecode is produced the first time a script is run and cached thereafter, you might as well compile the bytecode to machine code ahead-of-time too and skip the time overhead of JIT compilation on every run. In a template JIT, the machine code is only a function of the bytecode (assuming the template JIT doesn't generate code that depends on the shape of the heap).

Compiling native code ahead of time also saves on memory usage, because you can use file-backed mappings that can be lazily paged in and shared between multiple processes, and when these pages are in cache that translates also to faster startup too.

But if you're compiling bytecode ahead of time to native code, what is the bytecode for?

(not) my beautiful house

At some point you reach a state where you have made logical short-term decisions all the way and you end up with vestigial organs of WTF in your language runtime. Bytecode, for example. A bytecode interpreter written in C. Object file formats for things you don't care about. Trampolines. It's time to back up and consider just what it is that we should be building.

The highest-performing language implementations will be able to compile together the regions of code in which a program spends most of its time. Ahead-of-time compilers can try to predict these regions, but you can't always know what the behavior of a program will be. A program's run-time depends on its inputs, and program inputs are late-bound.

Therefore these highest-performing systems will use some form of adaptive optimization to apply run-time JIT compilation power on whatever part of a program turns out to be hot. This is the peak performance architecture, and indeed in the climb to a performant language implementation, there is but one peak that I know of. The question becomes, how to get there? What path should I take, with the priorities I have and the resources available to me, which lets me climb the farthest up the hill while always leaving the way clear to the top?

guile's priorities

There are lots of options here, and instead of discussing the space as a whole I'll just frame the topic with some bullets. Here's what I want out of Guile:

  1. The project as a whole should be pleasing to hack on. As much of the system as possible should be written in Scheme, as little as possible in C or assembler, and dependencies on outside projects should be minimized.

  2. Guile users should be able to brag about startup speed to their colleagues. We are willing to trade away some peak throughput for faster startup, if need be.

  3. Debuggability is important -- a Guile hacker will always want to be able to get stack traces with actual arguments and local variable values, unless they stripped their compiled Guile binaries, which should be possible as well. But we are willing to give up some debuggability to improve performance and memory use. In the same way that a tail call replaces the current frame in its entirety, we're willing to lose values of dead variables in stack frames that are waiting on functions to return. We're also OK with other debuggability imprecisions if the performance gains are good enough. With macro expansion, Scheme hackers expect a compilation phase; spending time transforming a program via ahead-of-time compilation is acceptable.

Call it the Guile Implementor's Manifesto, or the manifesto of this implementor at least.

beaucoup bucks

Of course if you have megabucks and ace hackers, then you want to dial back on the compromises: excellent startup time but also source-level debugging! The user should be able to break on any source position: the compiler won't even fold 1 + 1 to 2. But to get decent performance you need to be able to tier up to an optimizing compiler soon, and soon in two senses: soon after starting the program, but also soon after starting your project. It's an intimidating thing to build when you are just starting on a language implementation. You need to be able to tier down too, at least for debugging and probably for other reasons too. This strategy goes in the right direction, performance-wise, but it's a steep ascent. You need experienced language implementors, and they are not cheap.

The usual strategy for this kind of implementation is to write it all in C++. The latency requirements are too strict to do otherwise. Once you start down this road, you never stop: your life as an implementor is that of a powerful, bitter C++ wizard.

The PyPy people have valiently resisted this trend, writing their Python implementation in Python itself, but they concede to latency by compiling their "translated interpreter" into C, which then obviously can't itself be debugged as Python code. It's self-hosting, but staged into C. Ah well. Still, a most valiant, respectable effort.

This kind of language implementation usually has bytecode, as it's a convenient reification of the source semantics, but it doesn't have to. V8 is a good counterexample, at least currently: it treats JavaScript source code as the canonical representation of program semantics, relying on its ability to re-parse source text to an AST in the same way every time as needed. V8's first-tier implementation is actually a simple native code compiler, generated from an AST walk. But things are moving in the bytecode direction in the V8 world, reluctantly, so we should consider bytecode as the backbone of the beaucoup-bucks language implementation.

shoestring slim

If you are willing to relax on source-level debugging, as I am in Guile, you can simplify things substantially. You don't need bytecode, and you don't need a template JIT; in the case of Guile, probably the next step in Guile's implementation is to replace the bytecode compiler and interpreter with a simple native code compiler. We can start with the equivalent of a template JIT, but without the bytecode, and without having to think about the relationship between compiled and (bytecode-)interpreted code. (Guile still has a traditional tree-oriented interpreter, but it is actually written in Scheme; that is a story for another day.)

There's no need to stop at a simple compiler, of course. Guile's bytecode compiler is already fairly advanced, with interprocedural optimizations like closure optimization, partial evaluation, and contification, as well as the usual loop-invariant code motion, common subexpression elimination, scalar replacement, unboxing, and so on. Add register allocation and you can have quite a fine native compiler, and you might even beat the fabled Scheme compilers on the odd benchmark. They'll call you plucky: high praise.

There's a danger in this strategy though, and it's endemic in the Scheme world. Our compilers are often able to do heroic things, but only on the kinds of programs they can fully understand. We as Schemers bend ourselves to the will of our compilers, writing only the kinds of programs our compilers handle well. Sometimes we're scared to fold, preferring instead to inline the named-let iteration manually to make sure the compiler can do its job. We fx+ when we should +; we use tagged vectors when we should use proper data structures. This is déformation professionelle, as the French would say. I gave a talk at last year's Scheme workshop on this topic. PyPy people largely don't have this problem, for example; their langauge implementation is able to see through abstractions at run-time to produce good code, but using adaptive optimization instead of ahead-of-time trickery.

So, an ahead-of-time compiler is perhaps a ridge, but it is not the peak. No amount of clever compilation will remove the need for an adaptive optimizer, and indeed too much cleverness will stunt the code of your users. The task becomes, how to progress from a decent AOT native compiler to a system with adaptive optimization?

Here, as far as I know, we have a research problem. In Guile we have mostly traced the paths of history, re-creating things that existed before. As Goethe said, quoted in the introduction to The Joy of Cooking: "That which thy forbears have bequeathed to thee, earn it anew if thou wouldst possess it." But finally we find here something new, or new-ish: I don't know of good examples of AOT compilers that later added adaptive optimization. Do you know of any, dear reader? I would be delighted to know.

In the absence of a blazed trail to the top, what I would like to do is to re-use the AOT compiler to do dynamic inlining. We might need to collect type feedback as well, though inlining is the more important optimization. I think we can serialize the compiler's intermediate representation into a special section in the ELF object files that Guile produces. A background thread or threads can monitor profiling information from main threads. If a JIT thread decides two functions should be inlined, it can deserialize compiler IR and run the standard AOT compiler. We'd need a bit of mutability in the main program in which to inject such an optimization; an inline cache would do. If we need type feedback, we can make inline caches do that job too.

All this is yet a ways off. The next step for Guile, after the 2.2 release, is a simple native compiler, then register allocation. Step by step.

but what about llvmmmmmmmmmmmmm

People always ask about LLVM. It is an excellent compiler backend. It's a bit big, and maybe you're OK with that, or maybe not; whatever. Using LLVM effectively depends on your ability to deal with churn and big projects. But if you can do that, swell, you have excellent code generation. But how does it help you get to the top? Here things are less clear. There are a few projects using LLVM effectively as a JIT compiler, but that is a very recent development. My hubris, desire for self-hosting, and lack of bandwidth for code churn makes it so that I won't use LLVM myself but I have no doubt that a similar strategy to that which I outline above could work well for LLVM. Serialize the bitcode into your object files, make it so that you can map all optimization points to labels in that bitcode, and you have the ability to do some basic dynamic inlining. Godspeed!


If you're interested, I gave a talk a year ago on the state of JavaScript implementations, and how they all ended up looking more or less the same. This common architecture was first introduced by Self; languages implementations in this category include HotSpot and any of the JavaScript implementations.

Some notes on how PyPy produces interpreters from RPython.

and so I bid you good night

Guile's compiler has grown slowly, in tow of my ballooning awareness of ignorance and more slowly inflating experience. Perhaps we could have done the native code compilation thing earlier, but I am happy with our steady progress over the last five years or so. We had to scrap one bytecode VM and one or two compiler intermediate representations, and though that was painful I think we've done pretty well as far as higher-order optimizations go. If we had done native compilation earlier, I can't but think the inevitably wrong decisions we would have made on the back-end would have prevented us from having the courage to get the middle-end right. As it is, I see the way to the top, through the pass of ahead-of-time compilation and thence to a dynamic inliner. It will be some time before we get there, but that's what I signed up for :) Onward!

Journal-brief now in Fedora

The journal-brief package is now available in Fedora (22+)!

Also, the 1.1.2 release adds a ‘systemd’ output format to report on failed systemd units:

$ journal-brief -o systemd

Failed systemd units:

 7 x NetworkManager.service
 33 x abrtd.service
 1 x configure-printer@usb-001-008.service
 1 x dnf-makecache.service

For more about journal-brief visit the GitHub page.

SDN/NFV DevRoom at FOSDEM 2016

We are pleased to announce the Call for Participation in the FOSDEM 2016 Software Defined Networking and Network Functions Virtualization DevRoom!

Important dates:

  • Nov 18: Deadline for submissions
  • Dec 1: Speakers notified of acceptance
  • Dec 5: Schedule published

This year the DevRoom topics will cover two distinct fields:

  • Software Defined Networking (SDN), covering virtual switching, open source SDN controllers, virtual routing
  • Network Functions Virtualization (NFV), covering open source network functions, NFV management and orchestration tools, and topics related to the creation of an open source NFV platform

We are now inviting proposals for talks about Free/Libre/Open Source Software on the topics of SDN and NFV. This is an exciting and growing field, and FOSDEM gives an opportunity to reach a unique audience of very knowledgeable and highly technical free and open source software activists.

Topics accepted include, but are not limited to:


  • SDN controllers – OpenDaylight, OpenContrail, ONOS, Midonet, OVN, OpenStack Neutron,Calico, IOvisor, …
  • Dataplane processing: DPDK, OpenDataplane, netdev, netfilter, ClickRouter
  • Virtual switches: Open vSwitch, Snabb Switch, VDE, Lagopus
  • Open network protocols: OpenFlow, NETCONF, OpenLISP, eBPF, P4, Quagga


  • Management and Orchestration (MANO): Deployment and management of network functions, policy enforcement, virtual network functions definition –, Cloudify, OpenMANO, Tacker, …
  • Open source network functions: Clearwater IMS, FreeSWITCH, OpenSIPS, …
  • NFV platform features: Service Function Chaining, fault management, dataplane acceleration, …

Talks should be aimed at a technical audience, but should not assume that attendees are already familiar with your project or how it solves a general problem. Talk proposals can be very specific solutions to a problem, or can be higher level project overviews for lesser known projects. Please include the following information when submitting a proposal:

  • Your name
  • The title of your talk (please be descriptive, as titles will be listed with around 250 from other projects)
  • Short abstract of one or two paragraphs
  • Short bio (with photo)

The deadline for submissions is November 18th, 2015. FOSDEM will be held on the weekend of January 30th-31st 2016 and the SDN/NFV DevRoom will take place on Sunday, January 31st 2016. Please use the following website to submit your proposals: (you do not need to create a new Pentabarf account if you already have one from past years). You can also join the devroom’s mailing list, which is the official communication channel for the DevRoom: network-devroom at (subscription page:

The Networking DevRoom 2016 Organization Team