May 25, 2019

Pango future directions

Pango is in clearly maintenance mode — Behdad and  I have a Pango work day once every few months, whenever we get together somewhere.

But thats about it. Things that we don’t get done in a day often sit unfinished for long times in git branches or issues. Examples for this are the color Emoji support that took many years to land, or the  subpixel positioning work that is still unfinished.

This doesn’t mean that text rendering is in decline. Far from it. In fact, Harfbuzz is more active than ever and has had unprecedented success: All major Web browsers, toolkits, and applications are using it.

We’ve discussed for a while how to best take advantage of Harfbuzz’ success for text rendering on the desktop. Our conclusion is that we have to keep Pango from getting in the way. It should be a thin and translucent layer, and not require us to plumb APIs for every new feature through several internal abstractions.

We have identified several steps that will let us make progress towards this goal.

Unicode APIs

Many libraries provide subsets of Unicode data and APIs: GLib has some. ICU has some, fribidi has some. Even Harfbuzz has some.

Pango really does not need to  provide its own wrappers for text direction or Unicode scripts. Not doing so means we don’t have to update Pango when there is a new version of Unicode.

Harfbuzz passthrough

New font features land regularly in Harfbuzz. By providing direct access to Harfbuzz objects, we can make these available to GTK and applications without adding APIs to Pango.

Stop using freetype

Freetypes FT_Face object has locking semantics that are broken and hard to work with; they are constantly getting in the way as we are juggling hb_fonts, FT_Face and cairo scaled font objects.

We’ve concluded that the best way forward is to stop using freetype for font loading or accessing font and glyph metrics. We can use Harfbuzz for all of these (a small gap will be closed soon).

Using Harfbuzz for font loading means that we will lose support for bitmap and type1 fonts. We think this is an acceptable trade-off, but others might disagree. Note that Harfbuzz does support loading bitmap-only OpenType fonts.

Unified shaping

Shaping is the process of turning a paragraph of text and fonts into a sequence of positioned glyphs for rendering. Historically,  Pango has used a different implementation on each platforms.

Going forward, we want to use Harfbuzz for shaping on all platforms. The web browsers already do this, and it works well. This will let us clean up a lot of old, unused shaping engine abstractions in Pango.

Unhinted rendering

As a general direction, we want to move Pango towards (horizontally) unhinted rendering, combined with subpixel positioning. Other platforms are already doing this. And it gives us resolution-independent layout that is better suited for scalable apis and OpenGL rendering.

 

EndlessOS dual boot with Fedora

I've a ThinkPad X1 Yoga, that's basically a ThinkPad X1 Carbon 4th but with a touch screen with a pencil that works like a wacom tablet.

I've this laptop since 2016. The first thing that I did when I received it was install a GNU/Linux operating system. I'm a GNU/Linux user since the year 2000 going through a lot of distributions, Debian, Ubuntu, Archlinux, etc.

When I received this computer I've a customized Archlinux there, and I wanted to dd my harddisk and boot, but I was unable to do that. I didn't know nothing about UEFI and I was unable to boot the Archlinux installer.

So I decided to go ahead and change my main distribution. I installed Fedora and almost all the hardware worked so I keep that one and was happy, until today :P

OStree, the new way of distribute GNU/Linux

With OStree and flatpak, there's a new way to distribute GNU/Linux, instead of use directly a package manager and update each package, we can use OStree and mount the root filesystem as read only and do full OS upgrades without broken packages and dependencies and so. The operating system go as is and the user should try hard to break it.

The other great thing about OStree is that it's like a git repository, so you can have different branches and a history, so you can easily go back and forward, it's really easy to test the beta and go back without breaking your system.

The main problem is that you can't install anything on your OS, you should use contained apps like flatpak or install by hand, you can't use apt-get. But that's okay for a day to day user, a power user always can unlock the OS and use it as a normal GNU/Linux distribution.

I wanted to try one of these distributions. The logical choice was Silverblue because it's a Fedora and I'm using it for three years now, but there's another option, EndlessOS is also OStree based, and I'm working with this OS, so I should give a try and use EndlessOS.

The EndlessOS install process

Like all the new modern GNU/Linux distros, EOS comes with a easy to use installer, you only need to boot from the USB and click next until it's done...

But here we've the first problem. I've more than one partition in my disk:

  • /dev/sda1: UEFI
  • /dev/sda3: Fedora /
  • /dev/sda8: Swap
  • /dev/sda4: Fedora /boot
  • /dev/sda6: /home

I want to keep my Fedora (who knows if something bad happens) and try to use the same home partition for my new EOS. But the installer only give me the option to erase all and have a clean system. But that won't stop me.

Let's go back and instead of reformat I will click on Try Endless OS:

What we need to do, Robert McQueen gave me some directions:

The constraints for booting Endless are 1) you use our grub, and 2) the root partition is labelled "ostree"

So if you have an EFI system, you can copy our EFI binaries into the ESP, and create a new partition for Endless, then dd the endless ostree filesystem into it

Then you should be able to boot, if you add a boot entry for the endless grub to your firmware, or make it the default (by providing the fallback boot64.efi or whatever it's called), or chain load it from another Linux loader

Install EOS with other linux (EFI system)

  1. Boot from USB, select try
  2. Launch the gnome-disk-utility and prepare a partition. I've not free space, but I was able resize my Fedora partition and I split in two of the same size, now I've a new ext4 partition /dev/sda7 with 25GB.

  3. Copy the endless ostree:

$ sudo su
# dd if=/dev/mapper/endless/image3 /dev/sda7
  1. Copy endless grub to EFI. I mounted all partitions in /tmp, the first partition in /tmp/sda1 and the EOS efi in the /tmp/EOS:
# mkdir /tmp/sda1 /tmp/EOS
# mount /dev/sda1 /tmp/sda1
# mount /dev/mapper/endless-image1 /tmp/EOS
# cp -rf /tmp/EOS/EFI/endless/ /tmp/sda1/EFI
  1. Add the new boot entry:
# efibootmgr -c -d /dev/sda -p 1 -L EOS -l \\EFI\\endless\\grubx64.efi
  1. Set as default boot:
# cp /tmp/EOS/EFI/endless/grubx64.efi /tmp/sda1/EFI/Boot/bootx64.efi
  1. Reboot and create my default user. Then I add my home partition to the /etc/fstab file:
UUID=c885e171-1a03-4afb-8519-f9fe26fe92b7 /sysroot/home ext4 defaults 1 2

And because the first user in EOS is the shared account, with UID 1000, I've to change the UID of my user editing the file in /etc/passwd. Then I rebooted again and all works. I've all my flatpak apps installed in the user space working.

So here we're, with a shiny new OS working like a charm:

The EFI and efibootmgr (who needs grub to select the OS?)

I didn't know much about UEFI and I was very impressed about how easy is to update this from GNU/Linux. There's a tool called efibootmgr that does all the work, and you can mount the partition, that's a FAT32, and put files there.

In my ThinkPad, I can boot directly to the UEFI boot menu pressing F12 during the boot, and that menu can be changed using the efibootmgr so it's not needed anymore to use the grub2 OS selection interface, I can use the UEFI menu for that!

This has some disadvantages, if you remove files from the UEFI partition, you can break the whole boot, so review all before any change.

New Background panel, Calendar search engine, GTK4 shortcut engine (Sprint 1)

This is a new series, coming out every 3 weeks or so, with various updates. My focus will be on the apps I maintain (Calendar, To Do, and Settings), but in the future it may also include other applications that I contribute to.

GNOME Settings: Redesigned Background panel

So let’s start with one of the most intense and well coordinated one: the new Background panel.

I could write at length about it, but I guess the following video expresses what it became much better:

The redesigned panel in action

The most proeminent feature is the ability to select new wallpapers. Selected wallpapers are copied to the local data directory, so even if the original image vanishes, the wallpaper will survive. Colors and Drag and Drop were removed. We have plans to improve it even further, but for this week, this is more than satisfactory!

This work was only possible due to the design team, and in specific Allan Day and Sam Hewitt, actively engaging and giving real-time feedback.

GNOME Calendar: UI refinements, and a search engine

After a somewhat long hiatus, I’m finally back working on Calendar! Working on a codebase that you built yourself is refreshing, and I have various points that would like to address for GNOME 3.34.

Because Calendar was the first application I started with when reorganizing my routine, not a lot managed to be coordinated. Nonetheless, thanks to Sam, we figured a few low hanging fruits that could be fixed in less than a week.

The first thing I worked on was a new search engine for Calendar. The UI side of it is handled by Christian Hergert’s amazing new search button in libdazzle. This is all really about standing on the shoulder of giants.

Better feedback for synchronization, a new “+” button in the headerbar, and reorganization of the menus are other areas that improved.

GNOME To Do: Keyboard Shortcuts in GTK4

GNOME To Do is full GTK4 these days. Which means it’s both a testbed for new GTK4 features, and also a way to give feedback as an app developer for the GTK team. Unfortunately, it also means To Do is blocked on various areas where GTK4 is lacking.

One of these areas is keyboard shortcut.

Last year, Benjamin wrote a major revamp for keyboard shortcuts. As part of this cycle, I decided to rebase and finish it; and also make To Do use the new API. Unfortunately, I failed to achieve what I set myself to.

Turns out, adding a shortcuts engine to GTK4 is more involving and requires way more context than I had when trying to get it up to speed. I failed to predict that one week would have not been enough to finish it all.

However, that does not mean all the efforts were wasted! The rebasing of the shortcuts engine was a non-trivial task successfully completed (see gtk!842), and I also fixed a few bugs while working on it. I also got a working prototype of GNOME To Do with the new APIs, and confirmed that it’s well suited — at least for a simpler application such as To Do.

In retrospect, I believe I should have been more realistic (and perhaps slightly pessimistic) about the length and requirements of this task.

What’s next?

After doing a retrospective today, we agreed on what will be part of the next cycle of work:

  • GNOME Calendar will see its calendar management dialog revamped;
  • GNOME To Do will receive support for archiving task lists;
  • GNOME Settings will go through a session of “Every Detail Matters” (with a list of minor annoyances yet to be filled)

Unless the plan changes, I’ll focus my efforts on getting those points fixed during the next three weeks.

Sprint?

Ever since I started contributing to GNOME, the projects I got involved and responsibilities I acquired grew substantially. I now maintain 3 different projects, and am involved with another 3 (Music, Mutter, and Shell.) On top of that, I have my own set of tasks for Endless that usually are not related to any of these projects.

It is quite a lot.

It happens that changing contexts is a difficult task. There is extensive research showing that we humans are limited when it comes to how many different projects we can handle. The more projects there are, the worse the performance is. Having to take care of such many projects myself means my productivity was grinding to a halt. And not being productive makes even the happiest of us sad.

After reaching a low and wanting to abbandon those projects, thanks mostly to the support of my Endless colleagues, I was fortunately able to recover with enough energy to find alternative solutions to the problem of having too much to do.

The Coincidence

I take Japanese classes on Saturday. My Saturday routine is essentially:

  • Start the day early, warm up the Japanese with some homework
  • Go to the class
  • After class, go to a nearby shopping mall’s bookstore and code/read/buy books

During that same week, after the class, one of the highlighted books was:

In Portuguese: “Scrum: The Art of Doing Twice the Work in Half the Time” by Jeff Sutherland & J. J. Sutherland

At the time, that sounded suspiciously interesting.

While reading, I decided to start incorporating some aspects of the proposed workflow into my daily routines — which includes my GNOME contributions. The work presented here was done as my initial experiment.

May 24, 2019

Dear Ubuntu: Please Stop Packaging Epiphany If You Won’t Do It Properly

Dear Ubuntu,

When users try Epiphany on Ubuntu, they receive a sub-par, broken browser. If you’re not willing to do this right, please just remove Epiphany from your repositories. We’d all be happier this way. You are the  most popular distributor of Epiphany by far, and your poor packaging is making the browser look bad.

Epiphany 3.28.1 Is Stupid Old

Currently you’ve packaged Epiphany 3.28.1 for Ubuntu 18.04, your current LTS release. 3.28.1 is a seriously broken version with an overaggressive adblock filters subscription that blocks legitimate resources on a wide variety of websites, resulting in broken page rendering. We obviously don’t want users to ever use 3.28.1. There is a 3.28.2, released on May 22, 2018, which fixes this problem, but after one year you have still not yet updated. Ideally you would update to 3.28.5, which has been available since September 21, 2018. It’s not like I’m expecting you to upgrade to 3.30 or to 3.32 (the current stable series). I’d be happy to release a 3.28.6, except I know that it’s pointless: you would not upgrade to it if I did.

In Ubuntu 19.04, you have packaged Epiphany 3.32.0. The current version is 3.32.2. There are a lot of bugs fixed in between. (Update: Exam has pointed out that your snap package takes precedence over the Debian package in GNOME Software, so most users will actually receive the snap instead. The snap is still using 3.30.4, because Epiphany 3.32 depends on GTK 3.24, and that is not available in snaps yet. All app menu items are unavailable because Ubuntu’s GNOME Shell 3.32 does not display Epiphany 3.30’s app menu, so there’s no way to edit preferences, view history, import/export bookmarks, etc. This is not good.)

Because Epiphany is in your universe repository, rather than main, I understand that Canonical does not provide updates. But this is really not OK. Do I really need to add an xscreensaver-style time bomb to protect the reputation of Epiphany?

You’ve Disabled the JPEG 2000 Support

WebKitGTK is in main and you have been updating it regularly and in a timely manner, which is good. Thanks for this!

But we also need WebKitGTK to be built with OpenJPEG support, so that it can display JPEG 2000 images. Because you build WebKitGTK without OpenJPEG support, lots of popular websites are broken, including all websites using Akamai Image Manager. Because we have “Safari” but not “Chromium” in our user agent, these websites send us lots of JPEG 2000 images, and we need to be prepared to handle them properly to avoid broken websites. (Changing our user agent to avoid receiving JPEG 2000 images is, unfortunately, not practical.)

Here we have a really difficult issue, because you admittedly have a good reason for disabling OpenJPEG use. OpenJPEG has failed your security review for inclusion in main. Seth Arnold from the Ubuntu Security Team has reported 24 issues in OpenJPEG, of which 21 still remain unfixed. (It’s probably too much to ask, but if any readers want to help tackle a couple of these, that would be really great.) WebKitGTK is only as secure as its least-secure image decoder, and it seems likely that that would be OpenJPEG. Exposing the low-quality OpenJPEG library to the entire web is risky.

And yet, a web browser that doesn’t display websites properly is simply not worth delivering to users. We need this image decoder for web compatibility. WebKitGTK 2.26 will (hopefully) ship with a sandbox to mitigate security risks. Perhaps future versions of Epiphany should refuse to start if OpenJPEG support is unavailable?

lightening run-time code generation

The upcoming Guile 3 release will have just-in-time native code generation. Finally, amirite? There's lots that I'd like to share about that and I need to start somewhere, so this article is about one piece of it: Lightening, a library to generate machine code.

on lightning

Lightening is a fork of GNU Lightning, adapted to suit the needs of Guile. In fact at first we chose to use GNU Lightning directly, "vendored" into the Guile source respository via the git subtree mechanism. (I see that in the meantime, git gained a kind of a subtree command; one day I will have to figure out what it's for.)

GNU Lightning has lots of things going for it. It has support for many architectures, even things like Itanium that I don't really care about but which a couple Guile users use. It abstracts the differences between e.g. x86 and ARMv7 behind a common API, so that in Guile I don't need to duplicate the JIT for each back-end. Such an abstraction can have a slight performance penalty, because maybe it missed the opportunity to generate optimal code, but this is acceptable to me: I was more concerned about the maintenance burden, and GNU Lightning seemed to solve that nicely.

GNU Lightning also has fantastic documentation. It's written in C and not C++, which is the right thing for Guile at this time, and it's also released under the LGPL, which is Guile's license. As it's a GNU project there's a good chance that GNU Guile's needs might be taken into account if any changes need be made.

I mentally associated Paolo Bonzini with the project, who I knew was a good no-nonsense hacker, as he used Lightning for a smalltalk implementation; and I knew also that Matthew Flatt used Lightning in Racket. Then I looked in the source code to see architecture support and was pleasantly surprised to see MIPS, POWER, and so on, so I went with GNU Lightning for Guile in our 2.9.1 release last October.

on lightening the lightning

When I chose GNU Lightning, I had in mind that it was a very simple library to cheaply write machine code into buffers. (Incidentally, if you have never worked with this stuff, I remember a time when I was pleasantly surprised to realize that an assembler could be a library and not just a program that processes text. A CPU interprets machine code. Machine code is just bytes, and you can just write C (or Scheme, or whatever) functions that write bytes into buffers, and pass those buffers off to the CPU. Now you know!)

Anyway indeed GNU Lightning 1.4 or so was that very simple library that I had in my head. I needed simple because I would need to debug any problems that came up, and I didn't want to add more complexity to the C side of Guile -- eventually I should be migrating this code over to Scheme anyway. And, of course, simple can mean fast, and I needed fast code generation.

However, GNU Lightning has a new release series, the 2.x series. This series is a rewrite in a way of the old version. On the plus side, this new series adds all of the weird architectures that I was pleasantly surprised to see. The old 1.4 didn't even have much x86-64 support, much less AArch64.

This new GNU Lightning 2.x series fundamentally changes the way the library works: instead of having a jit_ldr_f function that directly emits code to load a float from memory into a floating-point register, the jit_ldr_f function now creates a node in a graph. Before code is emitted, that graph is optimized, some register allocation happens around call sites and for temporary values, dead code is elided, and so on, then the graph is traversed and code emitted.

Unfortunately this wasn't really what I was looking for. The optimizations were a bit opaque to me and I just wanted something simple. Building the graph took more time than just emitting bytes into a buffer, and it takes more memory as well. When I found bugs, I couldn't tell whether they were related to my usage or in the library itself.

In the end, the node structure wasn't paying its way for me. But I couldn't just go back to the 1.4 series that I remembered -- it didn't have the architecture support that I needed. Faced with the choice between changing GNU Lightning 2.x in ways that went counter to its upstream direction, switching libraries, or refactoring GNU Lightning to be something that I needed, I chose the latter.

in which our protagonist cannot help himself

Friends, I regret to admit: I named the new thing "Lightening". True, it is a lightened Lightning, yes, but I am aware that it's horribly confusing. Pronounced like almost the same, visually almost identical -- I am a bad person. Oh well!!

I ported some of the existing GNU Lightning backends over to Lightening: ia32, x86-64, ARMv7, and AArch64. I deleted the backends for Itanium, HPPA, Alpha, and SPARC; they have no Debian ports and there is no situation in which I can afford to do QA on them. I would gladly accept contributions for PPC64, MIPS, RISC-V, and maybe S/390. At this point I reckon it takes around 20 hours to port an additional backend from GNU Lightning to Lightening.

Incidentally, if you need a code generation library, consider your choices wisely. It is likely that Lightening is not right for you. If you can afford platform-specific code and you need C, Lua's DynASM is probably the right thing for you. If you are in C++, copy the assemblers from a JavaScript engine -- C++ offers much more type safety, capabilities for optimization, and ergonomics.

But if you can only afford one emitter of JIT code for all architectures, you need simple C, you don't need register allocation, you want a simple library to just include in your source code, and you are good with the LGPL, then Lightening could be a thing for you. Check the gitlab page for info on how to test Lightening and how to include it into your project.

giving it a spin

Yesterday's Guile 2.9.2 release includes Lightening, so you can give it a spin. The switch to Lightening allowed us to lower our JIT optimization threshold by a factor of 50, letting us generate fast code sooner. If you try it out, let #guile on freenode know how it went. In any case, happy hacking!

May 23, 2019

bigint shipping in firefox!

I am delighted to share with folks the results of a project I have been helping out on for the last few months: implementation of "BigInt" in Firefox, which is finally shipping in Firefox 68 (beta).

what's a bigint?

BigInts are a new kind of JavaScript primitive value, like numbers or strings. A BigInt is a true integer: it can take on the value of any finite integer (subject to some arbitrarily large implementation-defined limits, such as the amount of memory in your machine). This contrasts with JavaScript number values, which have the well-known property of only being able to precisely represent integers between -253 and 253.

BigInts are written like "normal" integers, but with an n suffix:

var a = 1n;
var b = a + 42n;
b << 64n
// result: 793209995169510719488n

With the bigint proposal, the usual mathematical operations (+, -, *, /, %, <<, >>, **, and the comparison operators) are extended to operate on bigint values. As a new kind of primitive value, bigint values have their own typeof:

typeof 1n
// result: 'bigint'

Besides allowing for more kinds of math to be easily and efficiently expressed, BigInt also allows for better interoperability with systems that use 64-bit numbers, such as "inodes" in file systems, WebAssembly i64 values, high-precision timers, and so on.

You can read more about the BigInt feature over on MDN, as usual. You might also like this short article on BigInt basics that V8 engineer Mathias Bynens wrote when Chrome shipped support for BigInt last year. There is an accompanying language implementation article as well, for those of y'all that enjoy the nitties and the gritties.

can i ship it?

To try out BigInt in Firefox, simply download a copy of Firefox Beta. This version of Firefox will be fully released to the public in a few weeks, on July 9th. If you're reading this in the future, I'm talking about Firefox 68.

BigInt is also shipping already in V8 and Chrome, and my colleague Caio Lima has an project in progress to implement it in JavaScriptCore / WebKit / Safari. Depending on your target audience, BigInt might be deployable already!

thanks

I must mention that my role in the BigInt work was relatively small; my Igalia colleague Robin Templeton did the bulk of the BigInt implementation work in Firefox, so large ups to them. Hearty thanks also to Mozilla's Jan de Mooij and Jeff Walden for their patient and detailed code reviews.

Thanks as well to the V8 engineers for their open source implementation of BigInt fundamental algorithms, as we used many of them in Firefox.

Finally, I need to make one big thank-you, and I hope that you will join me in expressing it. The road to ship anything in a web browser is long; besides the "simple matter of programming" that it is to implement a feature, you need a specification with buy-in from implementors and web standards people, you need a good working relationship with a browser vendor, you need willing technical reviewers, you need to follow up on the inevitable security bugs that any browser change causes, and all of this takes time. It's all predicated on having the backing of an organization that's foresighted enough to invest in this kind of long-term, high-reward platform engineering.

In that regard I think all people that work on the web platform should send a big shout-out to Tech at Bloomberg for making BigInt possible by underwriting all of Igalia's work in this area. Thank you, Bloomberg, and happy hacking!

May 22, 2019

Better support for running games under Wayland (with GNOME3/mutter as compositor)

First of all I do not want people to get their hopes up about $subject of this blogpost. Improving gaming support is a subjects which holds my personal interest and it is an issue I plan to spend time on trying to improve. But this will take a lot of time (think months for simple things, years for more complex things).

As I see it there are currently 2 big issues when running games under Wayland:

1. Many games show as a smal centered image with a black border (letterbox) around the image when running fullscreen.

For 2D games this is fixed by switching to SDL2 which will transparently scale the pixmap the game renders to the desktop resolution. This assumes that 2D games in general do not demand a lot of performance and thus will not run into performance issues when introducing an extra   scaling step. A problem here is that many games still use SDL1.2 (and some games do not use SDL at all).

I plan to look into the recently announced SDL1.2 compatibility wrapper around SDL2. If this works well this should fix this issue for all SDL1.2 2D games, by making them use SDL2 under the hood.

For 3D games this can be fixed by rendering at the desktop resolution, but this might be slow and rendering at a lower resolution leads to the letterbox issue.

Recently mutter has has grown support for the WPviewport extension, which allows Wayland apps to tell the compositor to scale the pixmap the app gives to the compositor before presenting it. If we add support to SDL2's Wayland backend for this then, this can be used to allow rendering 3D apps at a lower resolution and still have them fill the entire screen.

Unfortunately there are 2 problems with this plan:

  1. SDL2 by default uses its x11 backend, not its wayland backend. I'm not sure what fixes need to be done to change this, at a minimum we need a fix at either the SDL or mutter side for this issue, which is going to be tricky.

  2. This only helps for SDL2 apps, again hopefully the SDL1.2 compatibility wrapper for SDL2 can help here, at least for games using SDL.

2. Fullscreen performance is bad with many games.

Since under Wayland games cannot change the monitor resolution, they need to either render at the full desktop resolution, which can be very slow; or they render at a lower resolution and then need to do an extra scaling step each frame.

If we manage to make SDL2's Wayland backend the default and then add WPviewport support to it then this should help by reducing an extra memcpy/blit of a desktop-sized pixmap. Currently what apps which use scaling do is:

  1. render lower-res-pixmap;

  2. scale lower-res-pixmap to desktop-res-pixmap

  3. give desktop-res-pixmap to the compositor;

  4. compositor does a hardware blit of the desktop-res-pixmap to the framebuffer.

With viewport support this becomes:

  1. render lower-res-pixmap;

  2. give low-res-pixmap to the compositor;

  3. compositor uses hardware to do a scaling blit from the low-res-pixmap to the desktop-res framebuffer

Also with viewport support, the compositor could in the case of there only being the one fullscreen app even keep the framebuffer in lowres and use a hardware scaling drm-plane to send the low-res framebuffer scaled to desktop-res to the output while only reading the low-res framebuffer from memory saving a ton of memory bandwidth. But this optimization is going to be a challenge to pull off.

Wayland itches summary

Thank you all for the large amount of feedback I have received after my previous Wayland Itches blog post. I've received over 40 mails, below is an attempt at summarizing all the mails.

Highlights

1. Middle click on title / header bar to lower the Window does not work for native apps. Multiple people have reported this issue to me. A similar issue was fixed for not being able to raise Windows. It should be easy to apply a similar fix for the lowering problem. There are bugs open for this here, here and here.

2. Running graphical apps via sudo or pxexec does not work. There are numerous examples of apps breaking because of this, such as lshw-gui and usbivew. At least for X11 apps this is not that hard to fix. But sofar this has deliberately not been fixed. The reasoning behind this is described in this bug. I agree with the reasoning behind this, but I think it is not pragmatic to immediately disallow all GUI apps to connect when run as root starting today.

We need some sort of transition period. So when I find some time for this, I plan to submit a merge-requests which optionally makes gnome-shell/mutter start Xwayland with an xauth file, like how it is done when running in GNOME on Xorg mode. This will be controlled by a gsettings option, which will probably default to off upstream and then distros can choice to override this for now, giving us a transition period

Requests for features implemented as external programs on X11

There are various features which can be implemented as external programs
on X11, but because of the tighter security need to be integrated into the
compositor with Wayland:

  • Hiding of the mouse-cursor when not used à la unclutter-xfixes, xbanish.

  • Rotating screen 90 / 270 degrees à la "xrandr -o [left|right]" mostly used through custom hotkeys, possible fix is defining bindable actions for this in gsd-media-keys.

  • Mapping actions to mouse buttons à la easystroke

  • Some touchscreen's, e.g. so called smart-screens for education, need manual calibration. Under X11 there are some tools to get the callibration matrix for the touchscreen, after which this can be manually applied through xinput. Even under X11 this currently is far from ideal but at least it is possible there.

  • Keys Indicator gnome-shell extension. This still works when using Wayland, but only works for apps using Xwayland, it does not work for native apps.

  • Some sort of xkill and xdotool utility equivalents would be nice

  • The GNOME on screen keyboard is not really suitable for use with apps which are not touch-enabled, as it lacks a way to send ctrl + key, etc. Because of this some users have reported that it is impossible to use alternative on screen keyboards with Wayland. Not being able to use alternative on screen keyboards is by design and IMHO the proper fix here is to improve GNOME's on screen keyboard.

App specific problems


  • Citrix ICA Client does not work well with Xwayland

  • Eclipse does not work well with Xwayland

  • Teamviewer does not work with Wayland. It needs to be updated to use pipewire for screencapturing and the RemoteDesktop portal to inject keyboard and mouse events.

  • Various apps lack screenrecording / capture support due to the app not having support for pipewire: gImageReader, green-recorder, OBS studio, peek, screenrecorder, slack

  • For apps which do support pipewire, there is not an option to share the contents of a window, other then the window making the request. On Xorg it is possible to share a random window and since pipewire allows sharing the whole desktop I see no security reason why we would not allow sharing another window.

  • guake window has incorrect size when using HiDPI scaling, see this issue

Miscellaneous problems


  • Mouse cursor is slow / lags

  • Drag and drop sometimes does not work, e.g. dragging files into file roller to compress or out of file roller to extract.

  • Per keyboard layouts. On X11 after plugging in a keyboard, the layout/keymap for just that one keyboard can be updated manually using xinput, allowing different keyboard layouts for different keyboards when multiple keyboards are connected

  • No-title-bar shell extension, X button can be hit unintentionally, see this issue

  • Various issues with keyboard layout switching

Hard to fix issues


  • Alt-F2, r equivalent (restart the gnome-shell)

  • X11 apps running on top of Xwayland do not work well on HiDPI screens

  • Push-to-talk (passive key grab on space) does not work in Mumble when using native Wayland apps, see this issue

Problems with other compositors then GNOME3 / mutter

I've also received several reports about issues when using another Wayland compositor as GNOME / mutter (Weston, KDE, Sway). I'm sorry but I have not looked very closely into these reports. I believe that it is great that Linux users have multiple Desktop Environments to choose from and I wish for the other DEs to thrive. But there are only so many hours in a day so I've chosen to mainly focus on GNOME.

May 21, 2019

2019-05-21 Tuesday

  • Mail chew, M. ill at home; picked up H. and Charlotte after an exam; built slides, poked projections. Lunch. Monthly mgmt call. Admin.

May 20, 2019

2019-05-20 Monday

  • Out for a run, mail chew, pleased to see Lubos' improved script to render perf output in kcachegrind - mmerging a couple of cool tools. Picked up babes from school, customer calls until late.

Do you know what IVBP, ROMB or UTOK are?

First, thanks to everyone for the all the help with UEFI modules. There are a ton left, but also a lot done, so we’re getting there. If anyone is intimately familiar with ME firmware, I’d also really like some help with this Intel Management Engine document too. Thanks!

May 17, 2019

How to map objects to databases

GDA stands for GNOME Data Access and is a library to wrap database connections and its data using GObject classes, you can execute queries and much more.

VDA stands for Vala Data Access, is a library providing a heavily object oriented API to access databases’s data.

The API developed for VDA is going to be:

  • Heavily Object Oriented
  • Asynchronous almost by default
  • Provides GObject to database mapping

Object Oriented

GDA uses lot of structures, they are hard to be introspectable, so hard to be used outside C and Vala.

Providers now are Connection objects, so you have to instantiate them and then call open and catch opened signal to know if you are up with the connection.

SQL statements now are Query objects, created from string and now from structured objects with simple API, they can use execute on itself to get a TableModel, AffectedRows or any other Result object. An object for almost all kind of SQL commands will be added, with a simple easy to use API to provides the data it needs to be executed over the Connection.

There are models for tables, rows and columns, some of them implementing GLib.ListModel, so can iterate over their members, like rows in a table or columns in a row.

Asynchronous

Database operations can take time to be executed on servers, locally or remote, so you now have all Query execution as async methods, so for opening connections.

API

As you can notice, some API are still in development for VDA, so you can use the one already working or access GDA’s Connection objects if you are using Providers from it.

Eventually all API will be implemented by native Connection objects, without bind GDA. Searching to provide an easy and fast way to access introspection data from databases.

Easy API for New Database Servers

Currently GDA implementation for Providers is hard to implement for new database servers. VDA provides a new easy to implement interfaces, for new database servers, so is possible to extend VDA by creating new libraries, without depend on plugins.

Map objects to databases

Recently, VDA gains Vda.DataObject interface, it provides mapping your object to database’s table’s row, where the row’s columns are mapped to object’s properties and back.

Vda.DataObject supports:

  • Gets data from the database, through a SELECT query execution
  • Sets data in a table’s row using an UPDATE command
  • Creates new rows in the tables using  an INSERT command
  • Removes a row in the table using a DELETE command

Best of all you just need:

  • Implement Vda.DataObject with just 3 properties
  • Mark your object’s properties you want to map to the database’s table’s row’s column, using its nick with a text like: @Property Name::id, this is: your field’s in the data base can have any supported name, including spaces, we use @ to mark a property as to be mapped and ::id to mark it as an ID property used to query from the database.

All queries to read/write data to the database will be calculated automatically for you. Your class should set a Vda.Connection and the table’s name, through Vda.DataObject.database_connection and Vda.DataObject.database_table properties, this last one just at construction time.

This an example on how your code could be seen. Pay attention at initialization() method, was added here to show you how the table is created in the database and how the data is mapped using compatible types, in this case string to variant. In the near feature, could be possible to add automatic table creation if it doesn’t exits yet.

public class Client : Object, Vda.DataObject {

    // Database mapping
    [Description (nick="@id::id")]
    public string id { get; set; }
    [Description (nick="@name")]
    public string name { get; set; }
    [Description (nick="@description")]
    public string description { get; set; }
    [Description (nick="@phone")]
    public string phone { get; set; }

    construct {
      database_table_name = "clients";
      id = GLib.Uuid.string_random ();
    }

    public async string initialization () throws GLib.Error {
      var qct = database_connection.parse_string ("CREATE TABLE IF NOT EXISTS clients (id varchar(50), name varchar(50), description varchar(50), phone varchar(50))");
      yield qct.execute (null);
      var qi = database_connection.parse_string ("INSERT INTO clients (id, name, description, phone) VALUES ('"+id+"','"+name+"','"+description+"','"+phone+"')");
      yield qi.execute (null);
      return id;
    }

    // DataObject
    public string database_table_name { get; construct set; }
    public Vda.Connection database_connection { get; set; }
    public Cancellable cancellable { get; set; }
  }


May 13, 2019

Donating 5 minutes of your time to help the LVFS

For about every 250 bug reports I recieve I get an email offering to help. Most of the time the person offering help isn’t capable of diving right in the trickiest parts of the code and just wanted to make my life easier. Now I have a task that almost anyone can help with…

For the next version of the LVFS we deploy we’re going to be showing what was changed between each firmware version. Rather than just stating the firmware has changed from SHA1:DEAD to SHA1:BEEF and some high level update description provided by the vendor, we can show the interested user the UEFI modules that changed. I’m still working on the feature and without more data it’s kinda, well, dull. Before I can make the feature actually useful to anyone except a BIOS engineer, I need some help finding out information about the various modules.

In most cases it’s simply googling the name of the module and writing 1-2 lines of a summary about the module. In some cases the module isn’t documented at all, and that’s fine — I can go back to the vendors and ask them for more details about the few we can’t do ourselves. What I can’t do is ask them to explain all 150 modules in a specific firmware release, and I don’t scale to ~2000 Google queries. With the help of EDK2 I’ve already done 213 myself but now I’ve run out of puff.

So, if you have a spare 5 minutes I’d really appreciate some help. The shared spreadsheet is here, and any input is useful. Thanks!

May 12, 2019

Emulating rpath on Windows via binary patching

A nice feature provided by almost every Unix system is rpath. Put simply it is a way to tell the dynamic linker to look up shared libraries in custom directories. Build systems use it to be able to run programs directly from the build directory without needing to fiddle with file copying or environment variables. As is often the case, Windows does things completely differently and does not have a concept of rpath.

In Windows shared libraries are always looked up in directories that are in the current PATH. The only way to make the dynamic linker look up shared libraries in other directories is to add them to the PATH before running the program. There is also a way to create a manifest file that tells the loader to look up libraries in a special place but it is always a specially named subdirectory in the same directory as the executable. You can't specify an arbitrary path in the manifest, so the libraries need to be copied there. This makes Windows development even more inconvenient because you need to either fiddle with paths, copy shared libraries around or statically link everything (which is slooooow).

If you look at Windows executables with a hex editor, you find that they behave much the same way as their unixy counterparts. Each executable contains a list of dependency libraries that it needs, such as helper.dll. Presumably what happens is that at runtime the dynamic linker will parse the exe file and pass the library names to some lookup function that finds the actual libraries given the current PATH value. This raises the obvious question: what would happen if, somehow, the executable file would have an absolute path written in it rather than just the filename?

It turns out that it works and does what you would expect it to. The backend code accepts absolute paths and resolves them to the correct file without PATH lookups. With this we have a working rpath simulacrum. It's not really workable, though, since the VS toolchain does not support writing absolute paths to dependencies in output files. Editing the result files by hand is also a bit suspicious because there are many things that depend on offsets inside the file. Adding or removing even one byte will probably break something. The only thing we can really do is to replace one string with a different one with the same length.

This turns out to be the same problem that rpath entries have on Unix and the solution is also the same. We need to get a long enough string inside the output file and then we can replace it with a different string. If the replacement string is shorter, it can be padded with null bytes because the strings are treated as C strings. I have written a simple test repository doing this, which can be downloaded from Github.

On unix rpath is specified with a command line argument so it can be padded to arbitrary size. Windows does not support this so we need to fake it. The basic idea is simple. Instead of creating a library helper.dll we create a temporary library called aaaaaaaaaaaaaaaaaaaaaaaa.dll and link the program against that. When viewed in a hex editor the executable looks like this.


Now we can copy the library to its real name in a subdirectory and patch the executable. The result looks like this.


The final name was shorter than what we reserved so there are a bunch of zero bytes in the executable. This program can now be run and it will always resolve to the library that we specified. When the program is installed the entry can be changed to just plain helper.dll in the same way making it indistinguishable from libraries built without this trick (apart from the few extra null bytes).

Rpath on Windows: achieved.

Is this practical?

It's hard to say. I have not tested this on anything except toy programs but it does seem to work. It's unclear if this was the intended behaviour, but Microsoft does take backwards compatibility fairly seriously so one would expect it to keep working. The bigger problem is that the VS toolchain creates many other files, such as pdb debug info files, that probably don't like being renamed like this. These files are mostly undocumented so it's difficult to estimate how much work it would take to make binary hotpatching work reliably.

The best solution would be for Microsoft to add a new linker argument to their toolchain that would write dependency info to the files as absolute paths and to provide a program to rewrite those entries as discussed above. Apple already provides all of this functionality in their core toolchain. It would be nice for MS to do the same. This would simplify cross platform development because it would make all the major platforms behave in the same way.
It would be nice to get the same tools for Linux, too, but it's not that urgent since build systems already can do this reliably on their own.

May 10, 2019

Delta Airlines Crosses One Line Too Far in Union Busting

We create, develop, document and collaborate as users of Free and Open Source Software (FOSS) from around the globe, usually by working remotely on the Internet. However, human beings have many millennia of evolution that makes us predisposed to communicate most effectively via in-person interaction. We don't just rely on the content of communication, but its manner of expression, the body language of the communicator, and thousands of different non-verbal cues and subtle communication mechanisms. In fact, I believe something that's quite radical for a software freedom activist to believe: meeting in person to discuss something is always better than some form of online communication. And this belief is why I attend so many FOSS events, and encourage (and work in my day job to support) programs and policies that financially assist others in FOSS to attend such events.

When I travel, Delta Airlines often works out to be the best option for my travel: they have many international flights from my home airport (PDX), including a daily one to AMS in Europe — and since many FOSS events are in Europe, this has worked out well.

Admittedly, most for-profit companies that I patronize regularly engage in some activity that I find abhorrent. One of the biggest challenges of modern middle-class life in an industrialized soceity is figuring out (absent becoming a Thoreau-inspired recluse) how to navigate one's comfort level with patronizing companies that engage in bad behaviors. We all have to pick our own boycotts and what vendors we're going to avoid.

I realize that all the commercial airlines are some of the worst environmental polluters in the world. I realize that they all hire union-busting law firms to help them mistreat their workers. But, Delta Airlines recent PR campaign to frighten their workers about unions was one dirty trick too far.

I know unions can be inconvenient for organizational leadership; I actually have been a manager of a workforce who unionized while I was an executive. I personally negotiated that union contract with staff. The process is admittedly annoying and complicated. But I fundamentally believe it's deeply necessary, because workers' rights to collectively organize and negotiate with their employers is a cornerstone of equality — not just in the USA but around the entire world.

Furthermore, the Delta posters are particularly offensive because they reach into the basest problematic instinct in humans that often becomes our downfall: the belief that one's own short-term personal convenience and comfort should be valued higher than the long-term good of our larger communityf. It's that instinct that causes us to litter, or to shun public transit and favor driving a car and/or calling a ride service.

We won't be perfect in our efforts to serve the greater good, and sometimes we're going to selfishly (say) buy a video game system with money that could go to a better cause. What's truly offensive, and downright nefarious here, is that Delta Airlines — surely in full knowledge of the worst parts of some human instincts — attempted to exploit that for their own profit and future ability to oppress their workforce.

As a regular Delta customer (both personally, and through my employer when they reimburse my travel), I had to decide how to respond to this act that's beyond the pale. I've decided on the following steps:

  • I've written the following statement via Delta's complaint form:

    I am a Diamond Medallion (since 2016) on Delta, and I've flown more than 975,000 miles on Delta since 2000. I am also a (admittedly small) shareholder in Delta myself (via my retirement savings accounts).

    I realize that it is common practice for your company (and indeed likely every other airline) to negotiate hard with unions to get the best deal for your company and its shareholders. However, taking the step to launch what appears to be a well-funded and planned PR campaign to convince your workers to reject the union and instead spend union dues funds on frivolous purchases instead is a despicable, nefarious strategy. Your fiduciary duty to your shareholders does not mandate the use of unethical and immoral strategies with your unionizing labor force — only that you negotiate in good faith to get the best deal with them for the company.

    I demand that Delta issue a public apology for the posters. Ideally, such an apology should include a statement by Delta indicating that you believe your workers have the right to unionize and should take seriously the counter-arguments put forward by the union in favor of union dues and each employee should decide for themselves what is right.

    I've already booked my primary travel through the rest of the year, so I cannot easily pivot away from Delta quickly. This gives you some time to do the right thing. If Delta does not apologize publicly for this incident by November 1st, 2019, I plan to begin avoiding Delta as a carrier and will seek a status match on another airline.

    I realize that this complaint email will likely primarily be read by labor, not by management. I thus also encourage you to do two things: (a) I hope you'll share this message, to the extent you are permitted under your employment agreement, with your coworkers. Know that there are Diamond Medallions out here in the Delta system who support your right to unionize. (b) I hope you escalate this matter up to management decision-makers so they know that regular customers are unhappy at their actions.

  • Given that I'm already booked on many non-refundable Delta flights in the coming months, I would like to make business-card-sized flyers that say something like: I'm a Delta frequent flyer & I support a unionizing workforce. and maybe on the other side: Delta should apologize for the posters. It would be great if these had some good graphics or otherwise be eye-catching in some way. The idea would be to give them out to travelers and leave them in seat pockets on flights for others to find. If anyone is interested in this project and would like to help, email me — I have no graphic design skills and would appreciate help.
  • I'm encouraging everyone to visit Delta's complaint form and complain about this. If you've flown Delta before with a frequent flyer account, make sure you're logged into that account when you fill out the form — I know from experience their system prioritizes how seriously they take the complaint based on your past travel.
  • I plan to keep my DAL stock shares until the next annual meeting, and (schedule-permitting), I plan to attend the annual meeting and attempt to speak about the issue (or at least give out the aforementioned business cards there). I'll also look in to whether shareholders can attend earnings calls to ask questions, so maybe I can do something of this nature before the next annual meeting.

Overall, there is one positive outcome of this for me personally: I am renewed in my appreciation for having spent most of my career working for charities. Charities in the software freedom community have our problems, but nearly everyone I've worked with at software freedom charities (including management) have always been staunchly pro-union. Workers have a right to negotiate on equal terms with their employers and be treated as equals to come to equitable arrangements about working conditions and workplace issues. Unions aren't perfect, but they are the only way to effectively do that when a workforce is larger than a few people.

Call for testing the Pitivi 1.0 RC

Pitivi 1.0 is scheduled to be released on Monday, May 20th. All the important bugs we were aware of have been fixed.

To fix one of the last issues, Thibault very recently rewrote the timeline/layers/clips management in GES, and this might have introduced new bugs. While we have lots of tests which all pass, they don’t cover everything.

We ask you to test the 1.0 RC! Grab a bunch of video files shot with your phone or camera and make some video out of them, trying various features. Tell us if you notice problems, and if you can reproduce them please file an issue describing the steps to reproduce the bug.

You can find us in the #pitivi IRC channel on FreeNode, or in this bridged Pitivi Matrix room.

How to install

To install the Pitivi 1.0 RC, simply run:

flatpak install http://flatpak.pitivi.org/pitivi.flatpakref

If there are conflicts, you can uninstall the conflicting installation.

How to test

Start Pitivi from the console, and keep it in view to notice any warnings or errors which might show up:

flatpak run org.pitivi.Pitivi//stable

You should be able to use any video file supported by GStreamer, but we officially support only a limited set of formats. If the file appears in the media library with a warning sign, right-click it and select proxy to create an optimized media file used automatically instead of the original.

Most useful would be to test the timeline editing, which should be responsive and precise. Try for example splitting and trimming clips, moving them around. Try ungrouping audio-video clips to make the respective video shorter than the audio. Overlap clips to create cross-fade transitions.

If time allows, you can also try adding effects, for example adding a background to a green screen. Mix diverse footage formats in the same timeline. Make a slideshow and include title clips and background music.

For reference, see the user manual. At the end, please render it and show us what you did!

May 09, 2019

Developing GNOME: The Basics

I’ve been working in the GNOME community for a little under a year and a half now. During my time contributing to the project, I’ve seen a lot of misunderstandings from the community about how we work. So, I’ve decided to write a multi-part series on how development on GNOME works.

This first post will cover the basics. In future I’ll explain our tooling, how apps are created, and how changes happen across the ecosystem.

The Release Cycle

At the center of GNOME development is the release cycle. Every 6 months we put out a major release, with patch releases in-between. Major releases typically happen in September and March, and are named after the city of the most recent conference. GNOME 3.30 was named after the city we held GUADEC in, and 3.32 is named after where GNOME.Asia took place.

At different intervals in the cycle we have freeze periods, after which no changes can be made to certain parts of the code. Core apps, such as Web, Files, and Settings all strictly follow the release cycle and freezes. Apps outside of the core set like Polari, Builder, and Fractal can follow their own schedules or versioning schemes, but tend to put out releases alongside the GNOME core.

The freeze deadlines often determine what contributions make it into a release. For example, if a UI change is submitted after the UI freeze, maintainers need to seek approval from the Design and Release teams before the change can be merged. Freezes are staggered, and come in the following order:

  • UI, Feature, and API/ABI Freezes
  • String Freeze
  • Hard Code Freeze

The hard code freeze ends once the major release for the cycle is out. If you want to apply a change that violates the other freezes, you need to create a branch for the latest stable release. So, if I need to merge a UI change after the 3.32 release is out, I need to first create a gnome-3-32 branch before I accept the change onto master. This branch will then be used to cherry-pick changes for the 3.32.X releases.

How Apps Are Maintained

Each project within GNOME has its own developers. The Music developers aren’t necessarily the same people working on the Shell, and the Shell developers generally aren’t the same people working on GTK. While many developers work across the ecosystem on different projects, there is no one team of developers. This is why “GNOME decided such and such” is often inaccurate.

The maintainers of a project have full say over what contributions are or are not accepted. While certain things can be proposed, maintainers have the right to reject proposals. This is, for example, is why Terminal did not have a HeaderBar until 3.32 and doesn’t enable it by default. Nobody is forced to do anything, but often developers and designers will agree on a direction for an app, or the ecosystem at large.

Contrary to popular belief, most maintainers are not paid by Red Hat although some core components like Files and GNOME Shell do have Red Hat employees employed to work on them. Other companies such as Canonical, Purism, and Endless employ developers to work on the parts of the stack that matter to them. That said, most contributors are not on company time even if they are employed by the likes of Red Hat. And of course those that are employed to work on GNOME still aren’t paid for all of their work on GNOME. Most of our work is done on our own time, as limited by the rest of our lives.

It’s also worth noting that GNOME is built with a wide range of technologies; while GTK is written exclusively in C, Music is a Python project and Calculator is implemented with Vala. The Python developers working on Music are great with Python and GStreamer, but they aren’t going to be much help fixing a rounding error in Calculator as a general rule, and as volunteers it wouldn’t be fair to expect them to be, either.

tl;dr: GNOME is a community of individuals each with their own motivations and vision for their own part of the project doing their best to build a great platform for users.

May 08, 2019

Why crowdfunding freely licensed documentation is illegal in Finland

On the Meson manual crowdfunding page it is mentioned that the end result can not be put under a fully free license. Several people have said that they "don't believe such a law could exist" or words to that effect. This blog post is an attempt to to explain the issue in English as all available text about the case is in Finnish. As a disclaimer: I'm not a lawyer, the following is not legal advice, there is no guarantee, even that any of the information below is factual.

To get started we need to go back in time a fair bit and look at disaster relief funds. In Finland you must obtain a permit from the police in order to gather money for general charitable causes. This permit has strict requirements. The idea is that you can't just start a fundraising, take people's money and pocket it, instead the money must provably go to the cause it was raised for. The way the law is written is that a donation to charity is done without getting "something tangible" in return. Roughly if you give someone money and get a physical item in return, it is considered a sales transaction. If you give money to someone and in return get a general feeling of making the world better in some way, that is considered a donation. The former is governed by laws of commerce, the latter by laws of charity fundraising.

A few years ago there was a project to create a book to teach people Swedish. The project is page is here, but it is all in Finnish so it's probably not useful to most readers. They had a crowdfunding project to finish the project with all the usual perks. One of the goals of the crowdfunding was to make the book freely distributable after publishing. This is not unlike funding feature work on FOSS projects works.

What happened next is that the police stepped in and declared this illegal (news story, in Finnish). Their interpretation was that participating in this campaign without getting something tangible in return (i.e. paying less than the amount needed to get the book) was a "charitable donation". Thus it needs a charity permit as explained above. Running a crowdfunding campaign is still legal if it is strictly about pre-sales. That is, every person buys "something" and that something needs to have "independent value" of some sort. If the outcome of a project is a PDF and that PDF becomes freely available, it can be argued that people who participated did not get any "tangible value" in exchange for their money.

Because of this the outcome of the Meson manual crowdfunding campaign can not be made freely available. This may seem a bit stupid, but sadly that's the law. The law is undergoing changes (see here, in Finnish), but those changes will not take effect for quite some time and even when they do it is unclear how those changes would affect these kinds of projects.

GDA 5.2.9 Released

GDA is the GNOME Data Access library, able you to use GObject like API to access database provides like PostgreSQL, SQLite and MySQL, among others like JDBC, running queries and get data in data models for your application use.

This new release is an effort to fix bugs in 5.2 stable branch, including:

  • Fix Sun JRE 1.8 detection
  • Fix JDK 11.0 detection
  • Drop unneeded JAVAH variable check
  • Fix build for System Installed SQLite libs
  • Non-Dates and Timestamps values returns ‘NULL’ string when converted
  • Fix –with-ui, now UI is buildable when enable

Expect more fixes as this version is available in LTS distribution versions and current master targeting 6.0 release is getting more and more fixed on memory leaks, than ever.

GTK 3.96.0

This week, we released GTK 3.96.0. Again, it has been a while since the last release, so it is worth summarizing whats new in this release. There is really too much here to cover it all, so this post will only highlight the most important changes.

This release is another milestone on our way towards GTK 4. And while there are still some unfinished things, this release is much closer to we hope to achieve with GTK 4.

GSK

GSK has seen a number of bug fixes and new tests that are made much easier using a new debug tool, gtk4-node-editor. It can load and display serialized render node trees, such as this one that was saved from the GTK inspector, and compare the output of different renderers.

The 3D transformation support has been brought up to the level where we can do animated transitions like the cube spin below.

GDK

The trend to move toward Wayland inspired APIs has continued, with more X11-only apis being moved to the X11 backend or just removed. Use of child surfaces and global coordinates has been greatly reduced, but this work remains incomplete.

The refactoring of Drag-and-Drop has also continued, with the introduction of GdkDrag and GdkDrop objects. The GTK part of this refactoring is still incomplete.

Events have been simplified and are now used just for input. Other event have been replaced by signals and properties on GdkSurface. In detail, expose events have been replaced by the ::render signal, configure events have been replaced by the ::size-changed signal. Map events have been replaced by the :mapped property, and gdk_event_handler_set() has been replaced by the ::event signal.

The Wayland backend has gained support for the Settings portal for GtkSettings, and uses the text-input-unstable-v3 protocol for its input method support.

GTK

Widgets

One big change for custom widgets is the introduction of GtkLayoutManager, which is a new delegate object that takes over size allocation. Layout managers can optionally use layout children for holding layout properties. This replaces the layout-related child properties in GTK containers such as GtkBox or GtkGrid.

A number of layout managers are available:

  • GtkBinLayout, for simple single-child containers
  • GtkBoxLayout, for children that are arranged linearly
  • GtkGridLayout, for children that are arranged in a grid
  • GtkFixedLayout, for freely positioned and transformed children
  • GtkCustomLayout, as a quick way to turn traditional measure and size_allocate vfuncs into a layout manager

More layout manager implementations will appear in the future. Most prominently, work is underway on a constraints-based layout manager.

GtkAssistant, GtkStack and GtkNotebook have publicly
accessible page objects for their children. The page objects
are also exposed via a list model. They non-layout related child properties of these containers have been converted into regular properties on these page objects.

Since all existing child properties have been converted to regular properties, moved to layout properties or moved to such page objects, support for child properties has been dropped from GtkContainer.

The core GtkEntry functionality has been moved into a new GtkText widget, which also implements an expanded GtkEditable interface. All existing entry subclasses in GTK have been turned into GtkEditable implementations wrapping a GtkText widget. This also includes a new GtkPasswordEntry.

Other Changes

GTK widgets can transform their children using projective linear
transformations. This functionality is available in CSS and
as a GskTransform argument to gtk_widget_allocate. GtkFixed is
the first container that exposes this functionality. For further examples,
see the swing transition of GtkRevealer, the rotate transitions
of GtkStack or the Fixed Layout example in gtk4-demo.

A number of list models have been introduced, for internal use
and as public API: GtkMapListModel, GtkSliceListModel, GtkSortListModel, GtkSelectionModel, GtkSingleSelection. These will become more widely used when we introduce a list model-based GtkListView.

GtkBuilder can specify object-valued properties inline, instead of referring to them by ID, and the simplify command of gtk4-builder-tool has gained an option to automatically convert GTK 3 UI definition files to GTK 4.

Coming soon

For more information on the things that are still still coming for GTK 4, find us on Discourse, IRC, or look here.

May 07, 2019

Restricting users

Using a computer is mostly about executing apps, reading, writing and doing. But it can also be about not doing.

Confusing? Bear with me.

Imagine for a second that you are in an elementary school. The leadership is optimistic on exposing students to technology. They have set up big rooms with rows and rows of computers ready for their students to use.

Would you give complete permissions to these teenagers using the computers? Would you allow them to install and uninstall programs as they wish, access any website they feel like, use for as much time they want?

An elementary question

This is an intriguing situation to think about. At the same time that we want to restrict what the user can do — in this case, the student — we still want them to be able to get stuff done.

Thanks to Cassidy Blaede from the elementary OS team, this school-wants-to-limit-students situation was brought to the roundtable during the Parental Controls & Metered Data hackfest that happened during the second half of April.

The event itself was extensively covered already by Cassidy, and by Philip’s and Iain’s daily reports, and that is not going to be the focus of this article. I’m personally more interested in the design and technological aspects of what was discussed.

Apps or not apps

Naturally, when talking about user restriction, apps are probably the very first aspect of it that comes to our minds. Followed by browsers.

There is not a lot of controversy to the idea of assuming that administrators are likely to be the “supervisors”, and users without administrator privilages are the “supervised”.

At first, the use case we thought we were dealing was mostly about guardians or parents restricting what kind of contents, applications, or websites their kids would be able to access.

This is natural on systems that are heavily app-based. Smartphones have interesting implementations that are not only sane, but also technically feasible. But the traditional way of distributing apps via distro packages, and executing them without restrictions, makes that task incredibly hard, impossible dare I say. Unfortunately, the Linux desktop in general is not quite there yet.

Flatpak to the rescue!

Implementing app restrictions generically may be close to impossible on the traditional desktop, but it is actually very much feasible on newer technologies such as Flatpak.

If you have an OSTree-based immutable filesystem where all user-managed apps are installed via Flatpak, such as Endless OS or SilverBlue, app restrictions suddenly becomes even more realistic.

In fact, we at Endless have implemented something like that already, even if a bit rudimentary.

Endless OS implementation of app restrictions

For the purpose of continuing the hackfest discussions, Allan managed to do a quick sketch about it:

Disclaimer: this is nowhere near being final.

An interesting aspect of the mockups is the “Supervisor Password”; a password that does not give full root permissions, and yet allows whoever is managing the system to assume the role of supervisor. Think of a teacher, without the administrator password, but still able to supervise students’ accounts.

But browsers…

Restricting websites, however, is remarkably tricky to accomplish. It is essentially impossible to whitelist websites, at least when implenting it outside the browsers themselves. In general, blacklisting is easier(ish) than whitelisting because we don’t have to deal with the case of websites accessing contents outside their domains.

I’m pretty sure this is a reasonably studied field though, and I simply lack the pre-existing knowledge.

What about me?

While discussing those aspects of restricting users, it quickly became clear that applying restrictions on yourself is perfectly valid. It is actually branded as “Digital Wellbeing” and similar on various mobile OSes.

Those who work in front of computers and have more control over their schedule will appreciate being able to restrict themselves and improve their health, focus, and productivity.

More mockups (and again, quick sketches, not final)

It is also interesting to see per-app usage:

It is not clear yet how a potential new Focus mode would interact with notifications.

Metrics

You might notice that those mockups require some sort of a metrics system that would monitor which app the user is using. This could range from a simple event logger to a sophisticated daemon collecting much more. Nothing is settled in this front, no decision was made yet.

Of course, privacy concerns were taken into account. There is absolutely no room for sharing this data. All data would be stored locally.


My attendance to the Parental Controls & Metered Data hackfest was proudly sponsored by the GNOME Foundation.

May 06, 2019

Things to Love About Ducktype

I spent a lot of time making Ducktype into a lightweight syntax that I would really enjoy using. I had a list of design goals, and I feel like I hit them pretty well. In this post, I want to outline some of the things I hope you’ll love about the syntax.

1. Ducktype has a spec

I know not everybody nerds out over reading a spec the way I do. But whether or not you like reading them, specs are important for setting expectations and ensuring interoperable implementations. Probably my biggest gripe with Markdown is that there are just so many different flavors. Couple that with the multiple wiki flavors I regularly deal with, and my days become a game of guess-and-check.

Even in languages that haven’t seen flavor proliferation, you can still run into issues with new versions of the language. In a sense, every version of a parser that adds features creates a new flavor. Without a way to specify the version or do any sort of introspection or conditionals, you can run into situations where what you type is silently formatted very differently on different systems.

In Ducktype, you can declare the language version and any extensions (more on those later) right at the top of the file.

@ducktype/1.0
= Header

This is a paragraph.

2. Ducktype keeps semantics

I’ve been a strong proponent of semantic markup for a long time, since I started working with LaTeX over 20 years ago. Presentational markup has short-term gains in ease-of-use, but the long-term benefits of semantic markup are clear. Not only can you more easily adapt presentation over time, but you can do a lot with semantic markup other than display it. Semantic markup is real data.

Different lightweight languages support semantic markup to different degrees. Markdown is entirely presentational, and if you want to do anything even remotely sophisticated, you have to break out to HTML+CSS. AsciiDoc and reStructuredText both give you some semantic markup at the block level, and give you the tools to create inline semantic markup, but out of the box they still encourage doing semantic-free bold, italic, and monospace.

My goal was to completely capture the semantics of Mallard with less markup, not to just create yet another syntax. Ducktype does that. What I found along the way is that it can fairly nicely capture the semantics of other formats like DocBook too, but that’s a story for another day.

Ducktype uses a square bracket syntax to introduce semantic block elements, inspired by the group syntax in key files. So if you want a note, you type:

[note]
This text is an implicit paragraph in the note.

If you want a plain old bullet list, you type it just like in any other lightweight syntax:

* First item
* Second item
* Third item

But if you want a semantic steps list, you add the block declaration:

[steps]
* First step
* Second step
* Third step

Ducktype keeps the semantics in inline markup too. Rather than try to introduce special characters for each semantic inline element, Ducktype just lets you use the element name, but with a syntax that’s much less verbose than XML. For example:

Click $gui(File).
Call the $code(frobnicate) function.
Name the file $file(index.duck).

Both AsciiDoc and reStructuredText let you do something similar, although sometimes you have to cook up the elements yourself. With Ducktype, you just automatically get everything Mallard can do. You can even include attributes:

Press the $key[xref=superkey](Super) key.
Call the $code[style=function](frobnicate) function.

3. Ducktype keeps metadata

In designing Ducktype, it was absolutely critical that we have a consistent way to provide all metadata. Not only is metadata important for the maintenance of large document sets, but linking metadata is how documents get structure in Mallard. Without metadata, you can’t really write more than a single page.

You can do very good page-level metadata in both AsciiDoc and reStructuredText, but for Mallard we need to do metadata at the section and even block levels as well. Markdown, once again, is the poor format here with no support for any metadata at all. Some tools support a YAML header in Markdown, which would be pretty great if it were any sort of standard.

Ducktype uses a single, consistent syntax for metadata, always referencing the element name, and coming after the page header, section header, or block declaration that it’s providing info for.

= My First Topic
@link[type=guide xref=index]
@desc This is the first topic page I ever wrote.

== My First Section
@keywords first section, initial section

Do you like my first topic page with a section?

We can even do this with block elements:

[listing]
  @credit
    @name Shaun McCance
  . A code listing by Shaun
  [code]
    here_is_some_code()

There’s a lot going on in that little example, from nesting to shorthand block titles. The important part is that we can provide metadata for the code listing.

(Side note: I thought a lot about a shorthand for credits, and I have ideas on an extension to provide one. But in the end, I decided that the core syntax should have less magic. Explicit is better than implicit.)

4. Ducktype allows nesting

I do quite a bit of semi-manual format conversion fairly frequently. It’s not something I enjoy, and it’s not the best use of my time, but it just keep having to be done, and I’m kind of good at it. If you do that a lot, you’ll often run into cases where something just can’t be represented in another format, and that usually comes down to nesting. Can you put lists inside table cells? Can you put tables inside list items? Can lists nest? Can list items have multiple paragraphs?

Both reStructuredText and (shockingly) most flavors of Markdown allow some amount of nesting using indentation. This sometimes breaks down when tables or code blocks are involved, but it’s not terrible. This is the one place where AsciiDoc is my least favorite. In fact, this is my single least favorite thing about AsciiDoc. It uses a plus on its own line to indicate a continuation. It’s hard to visualize, and it has severely limited functionality.

Ducktype very explicitly allows nesting with indentation, whether you’re nesting lists, tables, code blocks, notes, or anything else. It allows you to skip some indentation in some common cases, but you can always add indentation to stay in a block. Ducktype specifies the exact nesting behavior.

[note style=important]
  This is very important note about these three things:

  * The first thing is hard to explain.

    It takes two paragraphs.

  * The second thing can really be broken in two:

    * First subthing
    * Second subthing

  * The third thing involves some code:

    [code]
      here_is_some_code()

You can use any number of spaces you like. I prefer two, in part because it fits in with the shorthand syntax for things like lists. Using indentation and block declarations, there is absolutely no limit on what or how deeply you can nest.

5. Ducktype tables are sane

I have never met a lightweight table syntax I liked. Trying to represent real tables with some sort of pseudo-ascii-art works ok for very simple tables, but it falls apart real fast with anything remotely non-trivial. Add to this the extra syntax required for for all the non-trivial stuff, and a substantial portion of your parser is devoted to tables. As a user, I have a hard time keeping all those workarounds in my head.

As I said above, the ability to nest things was very important, so most of these tables syntaxes were just a non-starter. How do you add extra blocks in a table cell when the whole table row is expected to fit on one line? In the end, I decided not to have a table syntax. Or more accurately, to treat tables as a sort of list.

Without any special treatment, tables, rows, and columns can already be written in Ducktype using the block declarations you’ve already seen:

[table]
  [tr]
    [td]
      One
    [td]
      Two
  [tr]
    [td]
      Three
    [td]
      Four

There’s no magic happening here. This is just standard block nesting. But it’s pretty verbose, not as verbose as XML, but still. Can’t we make it a bit easier, like we do for lists? Well yes, we make it easier exactly like we do for lists.

[table]
[tr]
* One
* Two
[tr]
* Three
* Four

There are a few things going on here. Most importantly, the asterisk is shorthand for a table cell, not a list item. What that shorthand does depends on context. But also, we’ve removed quite a lot of indentation. Earlier I wrote that there are some special rules that allow you to trim indentation in certain common cases. This is one of them.

Using a vertical table syntax means that you can do the same kind of nesting you can do with other elements. Here’s a list in a table cell:

[table]
[tr]
* Here's a list:
  * First
  * Second
* Another table cell

Ducktype isn’t the first format to allow a vertical table syntax. MediaWiki allows tables to be written vertically, and it works really well. But Ducktype is the only one to my knowledge not to introduce any new syntactical constructs at all. I have a hard time keeping all those dots and dashes in my head. Using a single, consistent syntax makes it easier to remember, and it reduces the number of special rules you need.

6. Ducktype supports extensions

I’ve mentioned extensions a couple of times already in this post, but I haven’t elaborated on them. In addition to being able to use Mallard extensions (which you can do without any extra syntax), I wanted  a way to add and experiment with syntax without shoving everything in the core. And, importantly, I wanted pages to have to declare what extensions they use, so we don’t have a guess-and-check mess of incompatible flavors. I live in a world where files are frequently cherry-picked and pushed through different tool chains. Being explicit is important to me.

So in Ducktype, you declare your extensions at the top, just like you do with the version of Ducktype you’re using:

@ducktype/1.0 if/experimental
= Page with Experimental Extension

What kinds of things can an extension do? Well, according to the spec, extensions can do literally anything. Realistically, what extensions can do depends on the extension points in the implementation. The reference implementation has two rough classes of extensions. There are the methods in the ParserExtension class, which let you plug in at the line level to affect how things are parsed. And there’s the NodeFactory, which lets you affect how parsed things are interpreted.

All of the ParserExtension methods are implemented and pretty well commented in the _test extension. One interesting use of this is the experimental csv extension. Remember how tables are always written vertically with a list-like syntax. That’s very powerful, but sometimes you really do just want something super simple. The csv extension lets you do simple comma-separated tables, like this:

@ducktype/1.0 csv/experimental
= CSV Table Example
[csv:table]
one,two,three
four,five,six
seven,eight,nine

That’s in about 40 lines of extension code. Another example of line parser extensions is a special syntax for conditionals, which I blogged about before.

The NodeFactory extension point, on the other hand, lets you control what the various bits of special syntax do. If you want to want to use Ducktype to write DocBook, you would use DocBook element names in explicit block declarations and inline notation, but what about the elements that get implicitly created for paragraphs, headers, list items, etc? That’s what NodeFactory does.

Et cetera

That’s six things to love about Ducktype. I’ve been really happy with Mallard’s unique approach to structuring docs. I hope Ducktype can make the power of Mallard available without all the extra typing that comes along with XML. If you want to learn more, get in touch.

 

 

 

Intellectual property registries

Past Monday we Carlos J. Vives and me gave a talk about Creative Commons and open content in a local education center:

The talk is part of the ccALM Almería Creative Commons Festival.

The only goal of this entry is to collect some links to registration services for IP useful for any digital creator in Internet, particularly for open culture works. As far I remember:

In the past there were this Digital Media Rights service, but seems broken now: http://dmrights.com/

Limited to Spain there is two public managed services:

Some of this services are thought to be used as a legal resource in case of litigation. Others are just an historical record for websites. If you want to use any of it study carefully their features and advantages of your interest.

May 05, 2019

Kung-fu Master, Blindfolded debugging

Recently I've been working in the EOS update. The change is really big because EOS has some modifications over the upstream code so there are a lot of commits applied after the last stable upstream commit.

EOS 3.5 was based on gnome 3.26 and we are updating to use the gnome 3.32 so all the downstream changes done since the last release should be rebased on top of the new code and during this process we refactor the code and commits using new tools and remove what's now in gnome upstream.

I've been working in the Hack computer custom functionality that's on top of the EOS desktop and basically I've been rebasing the code in the shell to do the Flip to Hack, the Clubhouse, a side component and notification override to propse hack quests and the wobbly windows effects.

I've been working mainly with gnome shell and updating javascript code to the new gjs version, but this was a big change and some rare bugs.

Here comes the blindfolded debugging. I've a development system with a lot of changes and a functionality in the gnome shell that depends on different projects, applications and technologies.

I don't know the code a lot, because I'm not the one that wrote all of this, I'm working here since February, so I know a bit how things works, but I'm not the one who knows everything, there are a lot of rare cases that I don't know about.

I've found and fixes several bugs in different projects, that I don't know about a lot, during this process. How can I do that? If you are a developer that write code since a few years maybe you've experienced something similar, I'm calling this the blindfolded debugging technique:

Start to change code without knowing exactly what you're doing, but with a small feeling that maybe that line is the problem.

This technique is only for experienced programmers, as a kungfu master that put a blindfold in their eyes to fight against an opponent, the developer that will be brave enough to try this should have a lot of experience or he will fail.

You're an experienced developer, but you don't know the software that you're debugging. It doesn't matter. The same way that in a kungfu fight you don't know your opponent, but almost every fighter has two arms and two legs, so more or less you'll know that he'll try to punch you or kick you, you've a clue. As a programmer, every software has an structure, functions, loops...

No matters who wrote that or how old that code is, if you're an experienced programmer you'll feel the code and without knowing exactly why or how, you will be able to look at one line and says: Here you're little buggy friend.

Maybe I'm only trying to justify my lucky with a story to feel better and say that I'm not a dumb changing random lines to try to find a bug, but I'm sure that other developers has this feeling too, that feeling that guides you to the exact problem but that you're not able to rationalize.

I think that this is the experience, the expertise is your body and your inner brain doing the work meanwhile you don't need to think about it or know exactly what you are doing.

May 04, 2019

Evaluating Survival Models

Evaluating Survival Models

The most frequently used evaluation metric of survival models is the concordance index (c index, c statistic). It is a measure of rank correlation between predicted risk scores $\hat{f}$ and observed time points $y$ that is closely related to Kendall’s τ. It is defined as the ratio of correctly ordered (concordant) pairs to comparable pairs. Two samples $i$ and $j$ are comparable if the sample with lower observed time $y$ experienced an event, i.e., if $y_j > y_i$ and $\delta_i = 1$, where $\delta_i$ is a binary event indicator. A comparable pair $(i, j)$ is concordant if the estimated risk $\hat{f}$ by a survival model is higher for subjects with lower survival time, i.e., $\hat{f}_i >\hat{f}_j \land y_j > y_i$, otherwise the pair is discordant. Harrell's estimator of the c index is implemented in concordance_index_censored.

While Harrell's concordance index is easy to interpret and compute, it has some shortcomings:

  1. it has been shown that it is too optimistic with increasing amount of censoring [1],
  2. it is not a useful measure of performance if a specific time range is of primary interest (e.g. predicting death within 2 years).

Since version 0.8, scikit-survival supports an alternative estimator of the concordance index from right-censored survival data, implemented in concordance_index_ipcw, that addresses the first issue.

The second point can be addressed by extending the well known receiver operating characteristic curve (ROC curve) to possibly censored survival times. Given a time point $t$, we can estimate how well a predictive model can distinguishing subjects who will experience an event by time $t$ (sensitivity) from those who will not (specificity). The function cumulative_dynamic_auc implements an estimator of the cumulative/dynamic area under the ROC for a given list of time points.

The first part of this post will illustrate the first issue with simulated survival data, while the second part will focus on the time-dependent area under the ROC applied to data from a real study.

To see the full source code for producing the figures in this post, please see this notebook.

Bias of Harrell's Concordance Index

Harrell's concordance index is known to be biased upwards if the amount of censoring in the test data is high [1]. Uno et al. proposed an alternative estimator of the concordance index that behaves better in such situations. In this section, we are going to apply concordance_index_censored and concordance_index_ipcw to synthetic survival data and compare their results.

Simulation Study

We are generating a synthetic biomarker by sampling from a standard normal distribution. For a given hazard ratio, we compute the associated (actual) survival time by drawing from an exponential distribution. The censoring times were generated from an uniform independent distribution $\textrm{Uniform}(0,\gamma)$, where we choose $\gamma$ to produce different amounts of censoring.

Since Uno's estimator is based on inverse probability of censoring weighting, we need to estimate the probability of being censored at a given time point. This probability needs to be non-zero for all observed time points. Therefore, we restrict the test data to all samples with observed time lower than the maximum event time $\tau$. Usually, one would use the tau argument of concordance_index_ipcw for this, but we apply the selection before to pass identical inputs to concordance_index_censored and concordance_index_ipcw. The estimates of the concordance index are therefore restricted to the interval $[0, \tau]$.

Let us assume a moderate hazard ratio of 2 and generate a small synthetic dataset of 100 samples from which we estimate the concordance index. We repeat this experiment 200 times and plot mean and standard deviation of the difference between the actual (in the absence of censoring) and estimated concordance index.

Since the hazard ratio remains constant and only the amount of censoring changes, we would want an estimator for which the difference between the actual and estimated c to remain approximately constant across simulations.

We can observe that estimates are on average below the actual value, except for the highest amount of censoring, where Harrell's c begins overestimating the performance (on average).

With such a small dataset, the variance of differences is quite big, so let us increase the amount of data to 1000 and repeat the simulation.

Now we can observe that Harrell's c begins to overestimate performance starting with approximately 49% censoring while Uno's c is still underestimating the performance, but is on average very close to the actual performance for large amounts of censoring.

For the final experiment, we double the size of the dataset to 2000 and repeat the analysis.

The trend we observed in the previous simulation is now even more pronounced. Harrell's c is becoming more and more overconfident in the performance of the synthetic marker with increasing amount of censoring, while Uno's c remains stable.

In summary, while the difference between concordance_index_ipcw and concordance_index_censored is negligible for small amounts of censoring, when analyzing survival data with moderate to high amounts of censoring, you might want to consider estimating the performance using concordance_index_ipcw instead of concordance_index_censored.

Time-dependent Area under the ROC

The area under the receiver operating characteristics curve (ROC curve) is a popular performance measure for binary classification task. In the medical domain, it is often used to determine how well estimated risk scores can separate diseased patients (cases) from healthy patients (controls). Given a predicted risk score $\hat{f}$, the ROC curve compares the false positive rate (1 - specificity) against the true positive rate (sensitivity) for each possible value of $\hat{f}$.

When extending the ROC curve to continuous outcomes, in particular survival time, a patient’s disease status is typically not fixed and changes over time: at enrollment a subject is usually healthy, but may be diseased at some later time point. Consequently, sensitivity and specificity become time-dependent measures. Here, we consider cumulative cases and dynamic controls at a given time point $t$, which gives rise to the time-dependent cumulative/dynamic ROC at time $t$. Cumulative cases are all individuals that experienced an event prior to or at time $t$ ($t_i \leq t$), whereas dynamic controls are those with $t_i>t$. By computing the area under the cumulative/dynamic ROC at time $t$, we can determine how well a model can distinguish subjects who fail by a given time ($t_i \leq t$) from subjects who fail after this time ($t_i>t$). Hence, it is most relevant if one wants to predict the occurrence of an event in a period up to time $t$ rather than at a specific time point $t$.

The cumulative_dynamic_auc function implements an estimator of the cumulative/dynamic area under the ROC at a given list of time points. To illustrate its use, we are going to use data from a study that investigated to which extent the serum immunoglobulin free light chain (FLC) assay can be used predict overall survival. The dataset has 7874 subjects and 9 features; the endpoint is death, which occurred for 2169 subjects (27.5%).

First, we are loading the data and split it into train and test set to evaluate how well markers generalize.

x, y = load_flchain()
 
(x_train, x_test,
 y_train, y_test) = train_test_split(x, y, test_size=0.2, random_state=0)

Serum creatinine measurements are missing for some patients, therefore we are just going to impute these values with the mean using scikit-learn's SimpleImputer.

num_columns = ['age', 'creatinine', 'kappa', 'lambda']
 
imputer = SimpleImputer().fit(x_train.loc[:, num_columns])
x_train = imputer.transform(x_train.loc[:, num_columns])
x_test = imputer.transform(x_test.loc[:, num_columns])

Similar to Uno's estimator of the concordance index described above, we need to be a little bit careful when selecting the test data and time points we want to evaluate the ROC at, due to the estimator's dependence on inverse probability of censoring weighting. First, we are going to check whether the observed time of the test data lies within the observed time range of the training data.

y_events = y_train[y_train['death']]
train_min, train_max = y_events["futime"].min(), y_events["futime"].max()
 
y_events = y_test[y_test['death']]
test_min, test_max = y_events["futime"].min(), y_events["futime"].max()
 
assert train_min <= test_min < test_max < train_max, \
    "time range or test data is not within time range of training data."

When choosing the time points to evaluate the ROC at, it is important to remember to choose the last time point such that the probability of being censored after the last time point is non-zero. In the simulation study above, we set the upper bound to the maximum event time, here we use a more conservative approach by setting the upper bound to the 80% percentile of observed time points, because the censoring rate is quite large at 72.5%. Note that this approach would be appropriate for choosing tau of concordance_index_ipcw too.

times = np.percentile(y["futime"], np.linspace(5, 81, 15))
print(times)

[ 470.3        1259.         1998.         2464.82428571 2979.
 3401.         3787.99857143 4051.         4249.         4410.17285714
 4543.         4631.         4695.         4781.         4844.        ]

We begin by considering individual real-valued features as risk scores without actually fitting a survival model. Hence, we obtain an estimate of how well age, creatinine, kappa FLC, and lambda FLC are able to distinguish cases from controls at each time point.

The plot shows the estimated area under the time-dependent ROC at each time point and the average across all time points as dashed line.

We can see that age is overall the most discriminative feature, followed by $\kappa$ and $\lambda$ FLC. That fact that age is the strongest predictor of overall survival in the general population is hardly surprising (we have to die at some point after all). More differences become evident when considering time: the discriminative power of FLC decreases at later time points, while that of age increases. The observation for age again follows common sense. In contrast, FLC seems to be a good predictor of death in the near future, but not so much if it occurs decades later.

Next, we will fit an actual survial model to predict the risk of death from the Veterans' Administration Lung Cancer Trial. After fitting a Cox proportional hazards model, we want to assess how well the model can distinguish survivors from deceased in weekly intervals, up to 6 months after enrollment.

The plot shows that the model is doing quite well on average with an AUC of ~0.82 (dashed line). However, there is a clear difference in performance between the first and second half of the time range. Performance increases up to about 100 days from enrollment, but quickly drops thereafter. Thus, we can conclude that the model is less effective in predicting death past 100 days.

Conclusion

I hope this post helped you to understand some of the pitfalls when estimating the performance of markers and models from right-censored survival data. We illustrated that Harrell's estimator of the concordance index is biased when the amount of censoring is high, and that Uno's estimator is more appropriate in this situation. Finally, we demonstrated that the time-dependent area under the ROC is a very useful tool when we want to predict the occurrence of an event in a period up to time $t$ rather than at a specific time point $t$.

sebp Sat, 05/04/2019 - 13:12

Comments

Design Tool Hackfest 2019

Last month I was in Berlin for the “Design Tools Hackfest 2019”. This whole thing started during last year’s GUADEC in Almeria, when I started playing around with cairo and librsvg. Because of that I got pulled into some discussions around improving tooling for GNOME designers, especially around icons.

During this hackfest we worked on three main issues:

– How to extract multiple icons contained in a single SVG file as separate SVGs (e.g. the stencil file in adwaita icon theme)
– How to generate Nightly hicolor app icons automatically
– How to export optimized icons directly from Icon Preview

The third point comes more or less for free once the first is done, so we focused on the former two.

Gnome Stencils

Splitting the stencils into different files is currently done with a script which uses Inkscape. This is very very slow because it has to open Inkscape for every single icon.

As far I could find there are no libraries which would allow us to manipulate SVG files in the way we need. Therefore I’m using cario and librsvg to generate the files. This approach may sound crazy, but it works quite well. The basic idea is to render an SVG file into a cairo surface with librsvg and then export the surface via cairo as a new SVG which contains only the part we’re interested in.

Nightly Icons, generated with svago-export

This way we can even use cario masks to automatically render nightly icons, and I started integrating this into Zander’s Icon Preview.

 

Sadly I’m currently quite busy with university, so I didn’t get around to finish and clean up svago-export so far, but if you want to have a look at the experiments I did feel free to do so. Luckily, the semester will soon be over and I will have more free time \o/

Special thanks to Tobias for hosting the event and thanks to the GNOME Foundation for sponsoring my travel.

May 03, 2019

Lessons when creating a C API from Rust

I have recently created a C API for a library dealing with Boot Loader Spec files as well as the GRUB environment file. In the process I have learnt a few things that, coming from a C background, were not obvious to me at all.

Box to control unsafe ownership

Say we have this simple Rust API:

pub struct A {
  counter: u8
}

impl A {
  pub fn new(count: u8) -> A {
    A { counter: count }
  }
}

Let’s start with the new method wrapper:

#[no_mangle]
pub extern "C" fn a_new(count: u8) -> *mut A {
  let boxed_a = Box::new(A {counter: count});
  Box::into_raw(boxed_a)
}

A Box is basically a smart pointer, it allows us to control the lifetime of the data outside of the boundaries of Rust’s borrow checker. Box::into_raw returns a pointer to the allocated A instance. Let’s see how to access that data again:

#[no_mangle]
pub extern "C" fn a_get_counter(a: *mut A) -> u8 {
  let a = unsafe { Box::from_raw(a) };
  let count = a.counter;
  Box::into_raw(a);
  count
}

Box::from_raw is an unsafe method that turns a pointer into an owned Box, this allows us to access the pointer data safely from Rust. Note that Box is automatically dereferenced.

UPDATE: Sebastian Dröge has rightly pointed out that the above method is wrong, note that this is how I found most StackOverflow and other people explain how I should use the data again, but if Sebastian says it is wrong then I know it to be wrong ;-).

Turns out that casting the pointer as a reference inside an unsafe block is enough:

#[no_mangle]
pub unsafe extern "C" fn a_get_counter(a: *mut A) -> u8 {
  let a = &*a;
  a.counter
}

 

Now we need to give the C user a deallocator for instances of A, this is relatively straightforward, we wrap the object around a Box and since we don’t call into_raw again, as soon as the Box is out of scope the inner contents are dropped too:

#[no_mangle]
pub unsafe extern "C" fn a_drop(a: *mut A) {
  Box::from_raw(a);
}

Strings

In Rust there are two standard ways to interact with strings, the String type, a dynamic utf-8 string that can be modified and resized, and &str, which basically is a bare pointer to an existing String. It took me a while to realize that internally a String is not null terminated and can contain many null characters. This means that the internal represenation of String is not compatible with C strings.

To address this, Rust provides another two types and referenced counterparts:

  • OsString and &OsStr: a native string tied to the runtime platform encoding and sizing
  • CString and &CStr: a null terminated string

My main grudge with this model when creating a C API is that it creates friction with the C boundary for a couple of reasons:

  • Internal Rust APIs often expect String or &str, meaning that at the C API boundary you need to allocate a CString if you use String as the internal representation, or the other way around if you use CString as the internal representation
  • You can’t “transparently” pass ownership of a CString to C without a exposing a deallocator specific to CString, more on this on the next section.

This means that compared to a C implementation of the API your code will liekly use more allocations which might or might not be critical depending on the use case, but this is something that struck me as a drawback for Rustification.

UPDATE: I am talking solely about the C API boundary, Rust is _great_ at giving you tools to avoid extra allocations (yay slices!), you can create a parser of a large chunk of text without allocating any extra strings to chunk the source text around.

Allocator mismatch

Something else I stumbled upon was that Rust does not use malloc/free, and that mismatch has annoying side effects when you are trying to rustify an API. Say you have this C code:

char* get_name() {
  const char* STATIC_NAME = "John";
  char* name = (char*)malloc(sizeof(STATIC_NAME));
  memcpy(name, STATIC_NAME, sizeof(STATIC_NAME));
}

int main () {
  char * name = get_name();
  printf("%s\n", name);
  free(name);
  return 0;
}

Now if you want to Rustify that C function, the naive way (taking into account the String vs. CString stuff I mentioned before) would be to do this:

#[no_mangle]
pub extern "C" fn get_name() -> *mut std::os::raw::c_char {
  const STATIC_NAME: &str = "John";
  let name = std::ffi::CString::new(STATIC_NAME)
               .expect("Multiple null characters in string");
  name.into_raw()
}

But this is not exactly the same as before, note that in the C example we call free() in order to drop the memory. In this case we would have to create a new method that calls CString::from_raw() but that won’t be compatible with the original C API.

This is the best I was able to came up with:

/* You can use the libc crate as well */
extern {
  fn malloc(size: usize) -> *mut u8;
  fn memcpy(dest: *mut u8, src: *const u8, size: usize) -> *mut u8;
}

#[no_mangle]
pub extern "C" fn get_name() -> *mut u8 {
  const STATIC_NAME: &str = "John";
  let name = std::ffi::CString::new(STATIC_NAME)
               .expect("Multiple null characters in string");
  let length = name.as_bytes_with_nul().len();
  let cname = unsafe { malloc(length) };
  unsafe { memcpy(cname, name.as_bytes_with_nul().as_ptr(), length) };
  cname
}

Note that STATIC_NAME is just an example, usually the data comes from a String/&str in your Rust API. The problem here is that we allocated an extra CString to then copy its contents using malloc/memcpy and then drop it immediately.

However, later, while working on creating UEFI binaries from Rust, I learned that Rust allows you to override its own allocator and use a custom one or the native system one. This would be another way to achieve the same and save the malloc/memcpy step, but don’t trust me 100% here as I am not sure whether this is entirely safe (if you know, let me know in the comments).

UPDATE: Many people have pointed out that overriding the allocator to use the system is absolutely fine:

use std::alloc::System;

#[global_allocator]
static GLOBAL: System = System;

#[no_mangle]
pub extern "C" fn get_name() -> *mut u8 {
  const STATIC_NAME: &str = "John";
  let name = std::ffi::CString::new(STATIC_NAME).expect("Multiple null characters in string");
  name.into_raw() as *mut u8
}

Traits as fat pointers

Let’s say we have the following API with two types and a trait implemented by both:

pub struct A {}
pub struct B {}

impl A {
  pub fn new () -> A { A{} }
}

impl B {
  pub fn new () -> B { B{} }
}

pub trait T {
  fn get_name(&self) -> std::ffi::CString;
}

impl T for A {
  fn get_name(&self) -> std::ffi::CString {
    std::ffi::CString::new("I am A").expect("CString error")
  }
}

impl T for B {
  fn get_name(&self) -> std::ffi::CString {
    std::ffi::CString::new("I am B").expect("CString error")
  }
}

Now the problem is, if we want a single wrapper for T::get_name() to avoid having to wrap each trait implementation family of functions, what do we do? I banged my head on this trying to Box a reference to a trait and other things until I read about this in more detail. Basically, the internal representation of a trait is a fat pointer (or rather, a struct of two pointers, one to the data and another to the trait vtable).

So we can transmute a reference to a trait as a C struct of two pointers, the end result for type A would be like this (for B you just need another constructor and cast function):

#[repr(C)]
pub struct CTrait {
  data: *mut std::os::raw::c_void,
  vtable: *mut std::os::raw::c_void
}

#[no_mangle]
pub extern "C" fn a_new() -> *mut A {
  Box::into_raw(Box::new(A::new()))
}

#[no_mangle]
pub extern "C" fn a_drop(a: *mut A) {
  unsafe{ Box::from_raw(a) };
}

#[no_mangle]
pub extern "C" fn a_as_t (a: *mut A) -> CTrait {
  let mut boxed_a = unsafe { Box::from_raw(a) };
  let ret: CTrait = {
    let t: &mut dyn T = &mut *boxed_a;
    unsafe { std::mem::transmute::<&mut dyn T,CTrait> (t) }
  };
  Box::into_raw(boxed_a);
  ret
}

#[no_mangle]
pub extern "C" fn t_get_name(t: CTrait) -> *mut u8 {
  let t = unsafe { std::mem::transmute::<CTrait, &mut dyn T> (t) };
  t.get_name().into_raw() as *mut u8
}

The C code to consume this API would look like this:

typedef struct {} A;
typedef struct {
  void* _d;
  void* _v;
} CTrait;

A*      a_new();
void    a_drop(A* a);
CTrait  a_as_t(A* a);
char*   t_get_name(CTrait);

int main () {
  A* a = a_new();
  CTrait t = a_as_t(a);
  char* name = t_get_name(t);
  printf("%s\n", name);
  free(name);
  a_drop(a);
  return 0;
}

Error reporting

Another hurdle has been dealing with Result<> in general, however this is more of a shortcoming of C’s lack of standard error reporting mechanism. In general I tend to return NULL to C API calls that expect a pointer and let C handle it, but of course data is lost in the way as the C end has no way to know what exactly went wrong as there is no error type to query. I am tempted to mimick GLib’s error handling. I think that if I was trying to replace an existing C library with its own error reporting mapping things would become easier.

Conclusions

I am in love with Rust and its ability to impersonate C is very powerful, however it is note entirely 0 cost, for me, the mismatch between string formats is the biggest hurdle as it imposes extra allocations, something that could become really expensive when rustifying C code that passes strings back and forth from/to the API caller. The other things I mentioned were things that took me quite some time to realize and by writing it here I hope I help other people that are writing Rust code to expose it as a C API. Any feedback on my examples is welcome.

Desktop Icons New Maintainer & Release

Desktop icons just had a new release! Plenty of good things went in:

Sergio fixed the “Open in Terminal” menu item not opening in the desktop folder, and also made the files appear from top to bottom, left to right, that more users were used to. Also, we discovered that the settings panel wasn’t getting translated, so Sergio fixed it too and now all translations are in place.

We had another developer contributing too, Andrea Azzarone. Andrea fixed the rubberband going over windows, quite a nice touch. Also added a way to group consecutive rubberbands with and , which is definitively useful!

You can grab the new version in tarball format or in extensions.gnome.org.

A new Maintainer

And I have good news… Sergio has just being apointed as co-maintainer of the project!

Sergio has been working on the project constantly, with a high sense of responsability, for more than 9 months. In fact, I have been struggling to catch up with all the work he has been doing. Whenever I reviewed a merge request, another two were opened :-)

Sergio deserves the maintainer position, having more decision making responsabilities and to take over important tasks in the project.

Congrats Sergio!

Improving the Container Workflow

As I mentioned in my talk at Scale 17x, if you aren’t using containers for building your application yet, it’s likely you will in the not-so-distant future. Moving towards an immutable base OS is a very likely future because the security advantages are so compelling. With that, comes a need for a mutable playground, and containers generally fit the bill. This is something I saw coming when I started making Builder so we have had a number of container abstractions for some time.

With growing distributions, like Fedora’s freshly released Silverblue, it becomes even more necessary to push those container boundaries soon.

This week I started playing with some new ideas for a terminal workspace in Builder. The goal is to be a bit of a swiss-army knife for container oriented development. I think there is a lot we can offer by building on the rest of Builder, even if you’re primary programming workhorse is a terminal and the IDE experience is not for you. Plumbing is plumbing is plumbing.

I’m interested in getting feedback on how developers are using containers to do their development. If that is something that you’re interested in sharing with me, send me an email (details here) that is as concise as possible. It will help me find the common denominators for good abstractions.

What I find neat, is that using the abstractions in Builder, you can make a container-focused terminal in a couple of hours of tinkering.

I also started looking into Vagrant integration and now the basics work. But we’ll have to introduce a few hooks into the build pipeline based on how projects want to be compiled. In most cases, it seems the VMs are used to push the app (and less about compiling) with dynamic languages, but I have no idea how pervasive that is. I’m curious how projects that are compiling in the VM/container deal with synchronizing code.

Another refactor that needs to be done is to give plugins insight into whether or not they can pass file-descriptors to the target. Passing a FD over SSH is not possible (although in some cases can be emulated) so plugins like gdb will have to come up with alternate mechanisms in that scenario.

I will say that trying out vagrant has been a fairly disappointing experience compared to our Flatpak workflow in Builder. The number of ways it can break is underwhelming and the performance reminds me of my days working on virtualization technology. It makes me feel even more confident in the Flatpak architecture for desktop developer tooling.

Some Maps news

I guess it's time for some news on goin ons in GNOME Maps again since been a couple of months since last time.

In the development branch (leading up to 3.34 this autumn) I have re-written the code used for producing printouts of route searches to not use the GtkOffscreen “pseudo widget“ to render instructions, since this will not be supported in GTK 4. Instead it now uses Cairo directly to do the rendering. The appearance of the printouts should however not really be any different, so in that respect it is not that exciting :-)

On the surface though there's been some concerns regarding the quality of the turn-by-turn-based route search results provided by GraphHopper, that in some cases gives quite a lot of unneeded “continue on” instructions on the same road, as shown in the following screenshot from a 3.30.x version:


This was something that there were upstream reports about in GraphHopper. But I thought that it might be worth a shot to try to do some client-side clean-ups of the results by merging consecutive instructions continuing on with a common part of the road name, and this is the after result:


Which makes a bit more condensed and easier to use. This has also been backported to the stable branch and will be available in next week's 3.32.2 release.

The other concern was that the search function is not that great at intelligently finding relevant, nearby results. And it's also not too forgiving to spelling mistakes and matching on partial strings. I have done some researching on alternatives for this as well. Will get back on the subject rather soon…

So, until next time!

Get notified of new upstream releases

If you maintain a distribution package, or app bundle, or container image, it’s handy to know when components you use in it have new releases.

bell-1096280_640

There is a really useful service Anitya that resides on release-monitoring.org. It watches almost 20 thousand projects for new releases and notify about them.

I maintain several packages in Fedora and the Fedora Project already makes it really convenient for me. It uses Anitya and opens a new bug against your package every time there is a new upstream release which I close once I update the package. But not every project gives you this service.

I also maintain several apps on Flathub which doesn’t provide such a service (yet). And it’s even more important to know about new upstream releases because besides the apps themselves I also have to maintain their dependencies which are not available in runtimes. Especially Evolution has quite a few of them.

Anitya gives you an API, so you can write your own service that will be checking with Anitya. But I’m a lazy person and why to write something that already exists, right? The Fedora Project has a web app called Fedora Notifications which can send notifications of all sorts of events to your email or IRC. You don’t have to be a Fedora package maintainer or anyhow involved in the Project. All you need is a Fedora account (FAS) to log in.

You pick Email Notifications (or IRC, whichever you prefer), click Create New Filter, then you pick Anything regarding a particular “upstream project” rule, and add projects you want to monitor separated by commas. Then click Add this rule and you’re good to go. This is what my rule for components I maitain in Flathub looks like:

Snímek z 2019-05-03 16-47-04

You’ll be getting notifications of new upstream releases to your email and it’s up to you how you’ll act on them. I typically check what kind of release it is, if there are any security or important bug fixes. I try to release those ASAP. Otherwise I plan them for future app releases and test with them beforehand.

I can imagine that there can be a nice automation built on the top of the Anitya API. If it detects a new release, it updates the manifest, triggers a build, and sends you info how it went. Maybe one day 🙂

May 01, 2019

scikit-survival 0.8 released

scikit-survival 0.8 released

This release of scikit-survival 0.8 adds some nice enhancements for validating survival models.

Previously, scikit-survival only supported Harrell's concordance index to assess the performance of survival models. While it is easy to interpret and compute, it has some shortcomings:

  1. it has been shown that it is too optimistic with increasing amount of censoring1,
  2. it is not a useful measure of performance if a specific time point is of primary interest (e.g. predicting 2 year survival).

The first issue is addressed by the new concordance_index_ipcw function, which implements an alternative estimator of the concordance index.1,2

The second point can be addressed by extending the well known receiver operating characteristic curve (ROC curve) to possibly censored survival times. Given a time point t, we can estimate how well a predictive model can distinguishing subjects who will experience an event by time t (sensitivity) from those who will not (specificity). The newly added function cumulative_dynamic_auc implements an estimator of the cumulative/dynamic area under the ROC for a given list of time points3.

Both estimators rely on inverse probability of censoring weighting, which means they require access to training data to estimate the censoring distribution from. Therefore, if the amount of censoring is high, some care must be taken in selecting a suitable time range for evaluation.

For a complete list of changes see the release notes.

Download

Pre-built conda packages are available for Linux, OSX and Windows:

conda install -c sebp scikit-survival

Alternatively, scikit-survival can be installed from source via pip:

pip install -U scikit-survival

References

  1. Uno, H., Cai, T., Pencina, M. J., D’Agostino, R. B., & Wei, L. J. (2011). On the C-statistics for evaluating overall adequacy of risk prediction procedures with censored survival data. Statistics in Medicine, 30(10), 1105–1117.
  2. H. Hung and C. T. Chiang, Estimation methods for time-dependent AUC models with survival data, Canadian Journal of Statistics, vol. 38, no. 1, pp. 8–26, 2010.
  3. H. Uno, T. Cai, L. Tian, and L. J. Wei, Evaluating prediction rules for t-year survivors with censored regression models, Journal of the American Statistical Association, vol. 102, pp. 527–537, 2007.
sebp Wed, 05/01/2019 - 17:50

Comments

April 30, 2019

Living inside the box

A few weeks ago, I wrote a post outlining the major improvements in Silverblue that are coming in Fedora 30. Now that Fedora 30 is released, you should go and try them out!

Today, I’ll dive a bit deeper into one particular item from that post which is not Silverblue-specific: the toolbox.

The toolbox experience

I’ll be honest: my path towards using the toolbox has not been 100% smooth.

I’m using the Rawhide version of Silverblue on my laptop, and my ability to use the toolbox has been affected by a number of regressions in the underlying tooling (unprivileged containers, podman and buildah) – which is to be expected for such young technology.

Thankfully, we got it all to work in time for the release, so your experience in Fedora 30 should be much smoother.

First steps

I’m doing all of my GTK and Flatpak development in a toolbox now. A small, but maybe interesting detail: While my host system is bleeding edge rawhide, the toolbox I’m using runs Fedora 30. I created it using the command:

$ toolbox create --release 30

After creating a toolbox, you can enter it using

$ toolbox enter

You will find yourself in another shell. If you pay close attention, you’ll notice that we show a colorful glyph in the prompt as a hint that you are in a different environment.

A minimal base

Entering the toolbox for the first time and looking around, it does not look much like a development environment at all:

$ rpm -q gcc gdb make
package gcc is not installed
package gdb is not installed
package make is not installed

But thankfully, dnf is available, so we can take advantage of the Fedora package universe to install what we need.  Thus, the first steps in working in a toolbox are easy, but a bit repetitive – install things.

I went into my GTK checkout and tried to build it in this environment:

[gtk+] $ meson --prefix=/usr build
bash: meson: command not found...

[gtk+] $ sudo dnf install meson
...

After a few rounds of installing missing dependencies, I eventually got to the point where GTK buit successfully.

From then on, everything basically worked as normal. Running tests and examples that I’ve build works just fine since the toolbox takes care of setting up DISPLAY, etc. I still had to install the occasional missing tool, such as vim or gdb, but development works just as it did before.

Some things are different, though.

A different perspective

In the past, I’ve done GTK development on a Fedora Workstation system, which means that many things are always installed, whether I think of them or not: icon themes, cursor themes, 3rd party pixbuf loaders, mime handlers, and many other things that make up a full desktop.

Working in a toolbox gives a somewhat different perspective. As explained earlier, the toolbox is created from a pretty minimal base (the fedora-toolbox image). None of the things I listed above are in it. Since they are not build dependencies, I didn’t get around to installing them at first either.

And I got to see how gtk-widget-factory and other demos look when they are missing – that has let to several bug fixes and improvements in the way GTK handles such situations.

A maybe unexpected beneficiary: other platforms, such as Windows, where it is much more likely that GTK will be installed without some of these extras.

Overall, I’m quite happy with this way of doing development.

Trying it out

I hope you are now eager to try it out yourself. You can do that on a traditional Fedora 30 system just as easily as on a Silverblue installation.

One thing I will recommend is that you should install the 0-day updates for the toolbox and gnome-terminal – we’ve worked hard during the freeze to make the toolbox as polished and welcoming as we can, and the result is in these updates.

Enjoy! ❤

April 26, 2019

Naming Your App

So you’ve decided to make a new app for GNOME, and designed a great interface for it. Now you want to start building it, so you open Gitlab to create a new repository. But then, oh no — it wants a name for the repository. Existential dread sets in.

Naming things is hard, and naming user-facing things is even more so. App names are read, pronounced, heard, and remembered by lots of people. The name is, along with the icon, the most important identifier for your project. With this tutorial I hope to make finding a great name for your app a bit easier.

General Guidance

As the GNOME Human Interface Guidelines put it:

“An application’s name is vital. It is what users will be first exposed to, and will help them decide whether they want to use an application or not. It is a major part of your application’s public face.”

Finding a good name is not always easy, but putting in a bit of effort up-front is worth it, because renaming the app once it’s established is much harder and messier.

A good name should:

  • Consist of one or two simple nouns
  • Be related to the app’s domain (e.g. Celluloid for a video app)
  • Be short (less than 15 characters)
  • Be easy to pronounce
  • Make it easy to come up with a good icon (e.g. by referencing a physical object that could be used as the icon)
  • Use title case (e.g. Icon Preview instead of iconPreview)

A good name should avoid:

  • Using trademarks or names of other projects (e.g. GNOME MPV)
  • Having a “G” prefix (e.g. GParted)
  • Overly complicated names and acronyms (e.g. GIMP)
  • Puns and inside jokes (e.g. D-Feet)
  • Non-standard punctuation or whitespace (e.g. UberWriter)
  • Made-up words or word combinations (e.g. Inkscape)

The Process

Over the years I’ve been involved with naming a lot of projects, and at this point I have a process which consistently produces pretty good results. It more or less goes like this:

  1. Write down all the words related to the app’s domain you can think of
  2. Do a thesaurus search for some of those words to find even more related words
  3. When you have about 15, start picking out some of the best-sounding ones, and look at their other qualities. Are they too long? Easy enough to pronounce? Do they have any unintended negative connotations?
  4. Once you have picked a handful of favorites, do a quick check to see if the names are already taken
  5. Among the ones not taken, pick your favorite one

Of course, when naming an app which is part of GNOME itself, the rules are a little different because these apps have completely generic names describing their function or the type content they show (e.g. Files, Image Viewer, Fonts, Music). That’s a much more rare (and usually easier) case though, so in this tutorial I’ll focus on independent third-party apps.

Let’s look at a real-world example of this process. A few months ago I was involved in renaming an internet radio app. At the time it was called Gradio, which was a bad name for many of the reasons mentioned above, and we wanted a nicer name for the new, completely rewritten version of the app.

1. Brainstorm

So, internet radio. What words immediately come to mind?

  • Radio
  • Transmission
  • Stations

These are pretty generic, so let’s branch out a bit. As with most digital technologies it’s hard to find nice metaphors, but we can make use of the analog predecessor to it (i.e. analog radio). Are there physical objects related to that which we could use?

  • Receiver
  • Headphones
  • Antenna

Maybe something related to analog radio technology?

  • Transistor
  • Frequencies

What about names of people who worked on the technology?

  • Marconi
  • Hertz

2. Thesaurus

Now that we have a few words to start with, let’s plug them into a thesaurus and see if there are any good related words. This is usually pretty hit or miss, as most related words you’ll find will not be relevant to the domain or make sense as names. However, after a few searches I always find a few good options that I didn’t think of before.

Here are a few additional words from thesaurus searches:

  • Transmission
  • Shortwave
  • Wireless
  • Decibel

In this particular case we also had a branstorming session on Matrix with a group of people from the community, which produced a few more options:

  • Longwave
  • Shortrange
  • Hzzzzz
  • Spectrum
  • Waves

3. Pick the best ones

Now we have about 20 words, which is a good place to stop brainstorming and start looking at which ones would make good names.

This process is not very scientific, just look at each word and imagine it as the name of the app. In particular, pay attention to the length of the name, ease of pronunciation, and whether it sounds nice.

In this case, some of my favorites are:

  • Transistor
  • Hertz
  • Spectrum
  • Shortwave

They’re all relatively short, easy to pronounce, and sound good as app names. The main question at this point is whether we can use them or if they’re already taken.

4. Check if they’re taken

I usually start off by searching directly on Github, to see if there are other FOSS already projects using the name. If I don’t find anything there, I search for the name on on Duckduckgo, adding “app” or “open source”.

In many cases you’ll find something somewhere using the name already. That doesn’t necessarily have to be a problem if it’s an app/project/company in a different domain, but it’s good to avoid large projects and companies.

In this case, it turns out “Transistor” is already a radio app for Android. Since it’s an app doing something very similar, people might think our radio app is affiliated with this project, which we want to avoid.

Hertz” is the name of a car rental service. It’s a pretty big company, so best to stay away from that as well.

Spectrum” is already the name of a forum software (which looks really cool btw). The potential for confusion is low, but the project looks well-established with 6000+ stars on Github, so not a great option.

Shortwave” is used by a bookmarking app, and there are some search results related to actual analog radio software, but nothing that looks big or problematic. This seems like a viable option.

5. Pick a winner

At this point you probably already know which of the viable options you like best, so just go ahead and make it official. In our example,”Shortwave” won because it was short but distinct-sounding, related to the domain, a pronounceable English word, and not taken by any major projects or companies.

The Shortwave Gitlab repo with the new name

If you find that all your favorites are taken, go back to the first steps and do some more brainstorming. The perfect name for your app is out there, and you’ll find it!

Bonus: Good Examples

For inspiration, here are some examples of well-named third party apps in the GNOME ecosystem, and what makes their names great:

Fragments — A torrent app. The name is great because it’s unique among torrent app names (which usually reference water, e.g. Deluge), yet still clearly connected to the domain (BitTorrent splits up files into lots of tiny parts and sends them in random order).

Peek — A GIF screen recorder. The name is very appropriate since the app is for making short recordings of small parts of the screen. The app feels small, quick, and frictionless, which the name perfectly encapsulates.

Teleport — Sends files across the local network.  The idea behind Teleport is to make sending a file across the local network effortless compared to other methods, such as using web services or USB drives. Using a sci-fi metaphor in the name lends itself well to that.

April 25, 2019

Twitter without Infinite Scroll

I like reading stuff on twitter.com because a lot of interesting people write things there which they don’t write anywhere else.

But Twitter is designed to be addictive, and a key mechanism they use is the “infinite scroll” design. Infinite scroll has been called the Web’s slot machine because of the way it exploits our minds to make us keep reading. It’s an unethical design.

In an essay entitled “If the internet is addictive, why don’t we regulate it?”, the writer Michael Schulson says:

… infinite scroll has no clear benefit for users. It exists almost entirely to circumvent self-control.

Hopefully Twitter will one day consider the ethics of their design. Until then, I made a Firefox extension to remove the infinite scroll feature and replace it with a ‘Load older tweets’ link at the bottom of the page, like this:

example

The Firefox extension is called Twitter Without Infinite Scroll. It works by injecting some JavaScript code into the Twitter website which disconnects the ‘uiNearTheBottom’ event that would otherwise automatically fetch new data.

Quoting Michael Shulson’s article again:

Giving users a chance to pause and make a choice at the end of each discrete page or session tips the balance of power back in the individual’s direction.

So, if you are a Twitter user, enjoy your new-found power!

Translucent Completion

Sometimes completion windows get in the way of reading the surrounding code. With Builder Nightly, you can press and release Left Control to toggle translucency of the completion window.

April 24, 2019

After GNOME Security Internship - Update 7

Here you can find the introduction, the update 1, the update 2, the update 3, the update 4, the update 5 and the update 6.

Part 1, protection from unwanted new USB devices

I received a few code reviews in the GNOME Settings Daemon MR that I’ll try to address in the next days.

Also I’m going to widen the requirements for allowing keyboards when the screen is locked. Right now if the lock screen is active we authorize a keyboard only if it is the only available keyboard in the system. It was a good idea in theory but not that much in practice.

For example let’s assume that you use an hardware USB switch hub between your desktop and your laptop with a mouse and a keyboard attached. If you have a “gaming” mouse with extra keys it is not only a mouse but also a keyboard. That means that when you want to switch from your laptop the the desktop, the mouse and the keyboard will be connected nearly simultaneously and if the mouse goes first the real keyboard will not the authorized. So you’ll be locked out from your system.

The gaming mouse is also only an example. If you have a yubikey shared in this USB hub there will be the same problem explained above.

For this reason in the next days I’ll edit the current implementation so that every USB keyboards will be authorized even if the lock screen is active. However we will still show a notification to explain that we authorized a new keyboard while the screen was locked.

Security at the expense of usability comes at the expense of security.

-AviD’s Rule of Usability

Part 2, limit untrusted USB keyboards

At the beginning of this month I started to work for Collabora and they allowed me to continue this GNOME project in my R&D time!

So thanks to that I made some interesting progress:

  • I added the required new GSetting to enable/disable the keyboard protection in gsettings-desktop-schemas

  • I experimented with the kernel key masking EVIOCSMASK. It worked very well with just a couple of issues still to address. Anyway after these tests I came to the conclusion that maybe the key filtering in the userspace is a better solution for this usecase. For example in the future it will be easier to add extra functionalities like showing a system notification when a not authorized keyboard sends a dangerous key.

  • In Mutter previously when we needed to check if a key was in the dangerous keys list we used a binary search. But the dangerous keys list is static so I removed the binary search in favor of using an hashset. Even if the performance gain is negligible, we can now perform a search in O(1) instead of O(log n).

  • Last time I did a benchmark it was when we used a device hash with name, vendor and product id instead of using the udev property as we do now. So I redid it and I’m glad to see that in the worst case scenario the delay of a key press halved.

  • Instead of using hwdb we store the device authorization in a local db and we use an udev helper to import the rules from the db. In this way the udev rule will always be static.

  • I completed the integration with GNOME Control Center. Now there is a working lock/unlock button to gain admin privileges via polkit. When the user changes the authorization level of a device it gets stored in a local db and we trigger an udev reload echoing “change” into the “uevent” sysfs attribute of the device. This new panel is also hidden if there isn’t a wayland session (the mutter key filtering works only on wayland).

  • I opened the necessary MRs upstream to gather feedback from the community. GNOME Control Center MR, Mutter MR, GSettings Desktop Schemas MR.

Benchmark v2

Now that we simply check if a device has the udev property GNOME_AUTHORIZED I did another benchmark:

  • Mutter stock: 1-2 usec to compute the pressed key.

  • With keyboard security off: 1-2 usec. It’s just an extra if check.

  • With keyboard security on: 1-2 usec. An extra if and a search in the hashset of “dangerous” keys.

  • Keyboard security on and you press a “dangerous” key: 4-6 usec. In this case we also need to check the udev properties of the device.

Compared to last time, the worst case scenario execution time dropped from 7-12 usec to 4-6 usec.

April 23, 2019

GStreamer Editing Services OpenTimelineIO support

GStreamer Editing Services OpenTimelineIO support

OpenTimelineIO is an Open Source API and interchange format for editorial timeline information, it basically allows some form of interoperability between the different post production Video Editing tools. It is being developed by Pixar and several other studios are contributing to the project allowing it to evolve quickly.

We, at Igalia, recently landed support for the GStreamer Editing Services (GES) serialization format in OpenTimelineIO, making it possible to convert GES timelines to any format supported by the library. This is extremely useful to integrate GES into existing Post production workflow as it allows projects in any format supported by OpentTimelineIO to be used in the GStreamer Editing Services and vice versa.

On top of that we are building a GESFormatter that allows us to transparently handle any file format supported by OpenTimelineIO. In practice it will be possible to use cuts produced by other video editing tools in any project using GES, for instance Pitivi:

At Igalia we are aiming at making GStreamer ready to be used in existing Video post production pipelines and this work is one step in that direction. We are working on additional features in GES to fill the gaps toward that goal, for instance we are now implementing nested timeline support and framerate based timestamps in GES. Once we implement them, those features will enhance compatibility of Video Editing projects created from other NLE softwares through OpenTimelineIO. Stay tuned for more information!

WebKitGTK and WPE gain WebRTC support back!

WebRTC is a w3c draft protocol that “enables rich, high-quality RTP applications to be developed for the browser, mobile platforms, and IoT devices, and allow them all to communicate via a common set of protocols”. The protocol is mainly used to provide video conferencing systems from within web browsers.

https://appr.tc running in WebKitGTKhttps://appr.tc running in WebKitGTK

A brief history

At the very beginning of the WebRTC protocol, before 2013, Google was still using WebKit in chrome and they started to implement support using LibWebRTC but when they started the blink fork the implementation stopped in WebKit.

Around 2015/2016 Ericsson and Igalia (later sponsored by Metrological) implemented WebRTC support into WebKit, but instead of using LibWebRTC from google, OpenWebRTC was used. This had the advantage of being implemented on top of the GStreamer framework which happens to be used for the Multimedia processing inside WebKitGTK and WebKitWPE. At that point in time, the standardization of the WebRTC protocol was still moving fast, mostly pushed by Google itself, and it was hard to be interoperable with the rest of the world. Despite of that, the WebKit/GTK/WPE WebRTC implementation started to be usable with website like appr.tc at the end of 2016.

Meanwhile, in late 2016, Apple decided to implement WebRTC support on top of google LibWebRTC in their ports of WebKit which led to WebRTC support in WebKit announcement in June 2017.

Later in 2017 the OpenWebRTC project lost momentum and as it was considered unmaintained, we, at Igalia, decided to use LibWebRTC for WebKitGTK and WebKitWPE too. At that point, the OpenWebRTC backend was completely removed.

GStreamer/LibWebRTC implementation

Given that Apple had implemented a LibWebRTC based backend in WebKit, and because this library is being used by the two main web browsers (Chrome and Firefox), we decided to reuse Apple’s work to implement support in our ports based on LibWebRTC at the end of 2017. A that point, the two main APIs required to allow video conferencing with WebRTC needed to be implemented:

  • MediaDevices.GetUserMedia and MediaStream: Allows to retrieve Audio and Video streams from the user Cameras and Microphones (potentially more than that but those are the main use cases we cared about).
  • RTCPeerConnection: Represents a WebRTC connection between the local computer and a remote peer.

As WeKit/GTK/WPE heavily relies on GStreamer for the multimedia processing, and given its flexibility, we made sure that our implementation of those APIs leverage the power of the framework and the existing integration of GStreamer in our WebKit ports.

Note that the whole implementation is reusing (after refactoring) big parts of the infrastructure introduced during the previously described history of WebRTC support in WebKit.

GetUserMedia/MediaStream

To implement that part of the API the following main components were developed:

  • RealtimeMediaSourceCenterLibWebRTC: Main entry point for our GStreamer based LibWebRTC backend.
  • GStreamerCaptureDeviceManager: A class to list and manage local Video/Audio devices using the GstDeviceMonitor API.
  • GStreamerCaptureDevice: Implementation of WebKit abstraction for capture devices, basically wrapping GstDevices.
  • GStreamerMediaStreamSource: A GStreamer Source element which wraps WebKit abstraction of MediaStreams to be used directly in a playbin3 pipeline (through a custom mediastream:// protocol). This implementation leverages latest GstStream APIs so it is already one foot into the future.

The main commit can be found here

RTCPeerConnection

Enabling the PeerConnection API meant bridging previously implemented APIs and the LibWebRTC backend developed by Apple:

  • RealtimeOutgoing/Video/Audio/SourceLibWebRTC: Passing local stream (basically from microphone or camera) to LibWebRTC to be sent to the peer.
  • RealtimeIncoming/Video/Audio/SourceLibWebRTC: Passing remote stream (from a remote peer) to the MediaStream object and in turn to the GStreamerMediaStreamSource element.

On top of that and to leverage GStreamer Memory management and negotiation capabilities we implemented encoders and decoder for LibWebRTC (namely GStreamerVideoEncoder and GStreamerVideoDecoders). This brings us a huge number of Hardware accelerated encoders and decoders implementations, especially on embedded devices, which is a big advantage in particular for WPE which is tuned for those platforms.

The main commit can be found here

WebKitWebRTC dataflow diagram

Conclusion

While we were able to make GStreamer and LibWebRTC work well together in that implementation, using the new GstWebRTC component (that is now in upstream GStreamer) as a WebRTC backend would be cleaner. Many pieces of the current implementation could be reused and it would allow us to have a simpler infrastructure and avoid having several RTP stack in the WebKitGTK and WebKitWPE ports.

Most of the required APIs and features have been implemented, but a few are still under development (namely MediaDevices.enumerateDevices, canvas captureStream and WebAudio and MediaStream bridging) meaning that many Web applications using WebRTC already work, but some don’t yet, we are working on those!

A big thanks to my employer Igalia and Metrological for sponsoring our work on that!

April 22, 2019

Announcing my Contract with Purism for an Adaptive Fractal UI

Over the past year or so I have been a regular contributor to Fractal, a Matrix chat client for GNOME. My contributions have allowed me to take on a bigger role in the GNOME community, including the maintainership of a few apps. I am pleased to announce that over the next week I will be working to make Fractal’s UI adaptive for the Librem 5’s launch. This contract began last week, and I already have some results to show off.

The Main View

Fractal has used a split window pattern since it’s inception. With HdyLeaflet apps that use this pattern can adapt their size to fit different widths. Once !363 is merged, Fractal will use HdyLeaflet to adapt it’s main view.

The Dialogs

In combination with HdyLeaflet, HdyDialog allows us to have dialogs that grow to the full size of the window at smaller sizes. I changed most of the instances of GtkDialog into HdyDialog and made sure each one would fit into the Librem 5’s constraints.

The Room Details

The room details view was something started a long while ago by Julian Sparber, made before HdyColumn was a thing. HdyColumn allows widgets to expand up to a certain width, while still being able to size appropriately on smaller screens.

Login

Our on-boarding process is in need of a redesign in general, so while redesigning it we can design for mobile as well.

Mapping Clicks To Touches

Currently we have a context menu that is accessed by right clicking on a message. This will need to work for long press on the phone, since there’s no way to right click on a touchscreen.

The Image Viewer

The controls of the Image Viewer’s headerbar are too large for it to shrink to the Librem 5’s constraints. We’ll need to make sure these controls are re-arranged in order to allow the viewer to fit correctly.

The Room Directory

The room directory is another view with controls that will need to adapt to small sizes. It has a searchbar, a back button, the items in the list don’t shrink enough for the rest of the window.

Overall, I’m very excited that Purism accepted my proposal and that I get to work on this. I have been looking forward to the day where I can run Fractal on my phone, and I’m glad to be bringing that closer.