GNOME.ORG

24 hours a day, 7 days a week, 365 days per year...

September 24, 2018

Getting the team together to revolutionize Linux audio

So anyone reading my blog posts would probably have picked up on my excitement for the PipeWire project, the effort to unify the world of Linux audio, add an equivalent video bit and provide multimedia handling capabilities to containerized applications. The video part as I have mentioned before was the critical first step and that is starting to look really good with the screen sharing functionality in GNOME shell already using PipeWire and equivalent PipeWire support being added to KDE by Jan Grulich. We have internal patches for both Firefox and Chrome(ium) which we are polishing up to propose them upstream, but we will in the meantime offer them as downstream patches in Fedora as soon as they are ready for primetime. Once those patches are deployed you should have any browser based desktop sharing software, like Google Hangouts, working fully under Wayland (and X).

With the video part of PipeWire already in production we decided the time has come to try to accelerate the development of the audio bits. So PipeWire creator Wim Taymans, PulseAudio developer Arun Raghavan and myself decided to try to host a PipeWire hackfest this fall to bring together many of the core Linux audio developers to try to hash out a plan and a roadmap. So I am very happy to say that at the end of October we will have a gathering in Edinburgh to work on this and the critical people we where hoping to have there are coming. Filipe Coelho who is the current lead developer on Jack will be there alongside Arun Raghavan, Colin Guthrie and Tanu Kaskinen from PulseAudio, Bastien Nocera from the GNOME project and Jan Grulich from KDE will be there representing desktop integration and finally Nirbheek Chauhan, Nicolas Dufresne and George Kiagiadakis from the GStreamer project. I think we have about the right amount of people for this to be productive and at the same time have representation from everyone who needs to be there, so I am feeling very optimistic that we can come out of this event with both a plan for what we want to do and the right people involved to make it happen. The idea that we can have a shared infrastructure for consumer level audio and pro-audio under Linux really excites me and I do believe that if we do this right Linux will take a huge step forward as a natural home for pro-audio desktop users.

A big thanks you to the GNOME Foundation for sponsoring this event and allow us to bring all this people together!

Wacom's graphic tablet sizes

For some reasons I’ve been looking for second hand Wacom graphic tablets. To me has been annoying to find out which size is for each model. So I’m writing here the list of the models I gathered.

The reason for looking only for Wacoms is because these days seem to be very well supported in Linux, at least the old models you can get second hand.

model active area size
CTL460 147,2 x 92,0 mm
CTL 420 127.6 x 92.8 mm
CTE-430 Graphire 3 127 x 101 mm
CTF-430 127.6 x 92.8 mm
CTL 460 147,2 x 92,0 mm
CTH-460 147,2 x 92,0 mm
CTH-461 147,2 x 92,0 mm
CTH-470 147,2 x 92,0 mm
CTL-470 147,2 x 92,0 mm
CTL-480 Intuos 152 x 95 mm
CTE-640 208.8 x 150.8 mm
CTE-650 216.5 x 135.3 mm
CTH-661 215.9 x 137.16 mm
CTH-670 217 x 137 mm
ET-0405A-U 127 x 106 mm
Graphire 2 127.6 x 92.8 mm
Intuos 2 127.6 x 92.8 mm (probably)
Volito 2 127.6 x 92.8 mm


As a reference these are the standards DIN sizes comparable with those models:

DIN type   size
A4 210 x 297 mm
A5 148 x 210 mm
A6 105 x 148 mm

If you find any typo or want to add other models feel free to comment.

Recently in Geoclue

After I started working for Collabora in April, I've finally been able to put some time on maintenance and development of Geoclue again. While I've fixed quite a few issues on the backlog, there has been some significant changes as of late, that I felt deserves some highlighting. Hence this blog post.

Leaving security to where it belongs

 


Since people's location is a very sensitive piece of information, security of this information had been the core part of Geoclue2 design. The idea was (and still is) to only allow apps access to user's location with their explicit permission (that they could easily revoke later). When Geoclue2 was designed and then developed, we didn't have Flatpak. Surely, people were talking about the need for something like Flatpak but even with those ideas, it wasn't clear how location access will be handled.

Hence we decided for geoclue to handle this itself, through an external app authorizing agent and implemented such an agent in GNOME Shell. Since there is no reliable way to identify an app on Linux, there were mixed reactions to this approach. While some thought it's good to have something rather than nothing, others thought it's better to wait for the time when we've the infrastructure that allows us to reliably identify apps.

Fast forward to an year or so ago, when Flatpak portals became a thing, I had a long discussion with Matthias Clasen and Bastien Nocera about how geoclocation should work in Flatpak. We disagreed on our approach and we forgot about the whole thing then.

Some months ago, we had to make app authorizing agent compulsory to plug some security holes and that made a lot of people who don't use GNOME, unhappy. We had to start installing the demo agent for non-GNOME as a workaround. This forced me to rethink the whole approach and after some more long discussions with Matthias and a lot of thinking, the plan is to:
  • Create a Flatpak geolocation portal. Matthias already has a work-in-progress implementation. I really wanted the portal API to be as identical to the Geoclue API but I failed to convince Matthias on that. This is not that big an issue though, as at least the apps using GeoclueSimple API will not need to change anything for accessing location from inside the Flatpak sandbox.
  • Drop all authorization from Geoclue and leave that to the geolocation portal. I've already dropped authorization for non-flatpak (i-e system) apps in git master. Once the portal is in place and GNOME shell and control-center have been modified to talk to it, we can drop all app authorizing code from Geoclue.

    Note
    that we have been able to reliably identify Flatpak apps and it's only the system apps that can lie about their identity.

A modern build system

 


Like many Free Software projects, Geoclue is also now using Meson for its builds. After it started to work reliably, I also dropped autotools-based build completely. The faster build makes development a much more pleasant experience.

And a modern issue tracker to go with it

 

Bugzilla served us well but patches in Bugzilla are no fun, even though git-bz makes it much much better. So when Daniel Stone setup gitlab on freedesktop.org, Geoclue was one of the first few projects to move to gitlab. Now it's much easier and simpler to contribute to Geoclue.

Minimize GeoIP use

 

While GeoIP is a nice backup if you have neither WiFi hardware nor a cellular modem, Geoclue would also use (only) that if an app only asked for city-level accuracy. Apps like GNOME Weather and GNOME Clocks ask for only that since that's the info they need and don't need to know which street you're currently on. This would be perfect if only the GeoIP database being used would be correct or accurate for at least 90% of the IP addresses but unfortunately the reality is far from that. This meant, a significant number of people getting annoyed with these apps showing them time and weather of a different town than their current one.

On the other hand, we couldn't just use a more accurate geolocation source (WiFi) since an app should not get more accurate location it asked for and it was authorized for by the user. While currently we don't have the UI in GNOME (or any other platform) that allows users to control the location accuracy, the infrastructure has always been in place to do that.

Recently one person decided to not only report this but had a good suggestion that I recently implemented: Use WiFi geolocation for city-level accuracy as well but randomize the location enough to mitigate the privacy concerns. It should be noted that while this solution ensures that apps don't get more accurate location then they should, it still means sending out the current WiFi data to the Mozilla Location Service (MLS) and Geoclue getting a very accurate (street-level) location in response. It's all over HTTPS so it's not as bad as it sounds.

The future of Mozilla Location Service


When Mozilla announced their location service in late 2013, Geoclue became one of it's first users as it was our only hope for a reliable WiFi-geolocation source. We couldn't use Google's service as their ToC don't allow it to be used in an open source project (I recall some clause that it can only be used with Google Maps and not any other Map software). MLS was a huge success in terms of people contributing WiFi data to it. I've been to quite a few places around Europe and North America in the last few years and I haven't been to any location, that is not already covered by MLS.

Mozilla's own interest in this service was tied to their Firefox OS project. Unfortunately Firefox OS project was abandoned two years ago and Mozilla lost its interest in MLS as a result. Mozilla folks are the good guys so they have kept the service running and users can still contribute data but it's no longer developed or maintained.

Since this is a very important service for all users of geoclue, I feel very uneasy about this uncertain future of MLS. So consider this a call for help. If your company relies on MLS (directly or through Geoclue) and you'd want to secure the future of Open Source geolocation, please do get in touch and we can discuss how we could possibly achieve that.

JavaScript news from GNOME 3.30

JavaScript news from GNOME 3.30

Welcome back to the latest news on GJS, the Javascript engine that powers GNOME Shell, Endless OS, and many GNOME apps.

I haven’t done one of these posts for several versions now, but I think it’s a good tradition to continue. GNOME 3.30 has been released for several weeks now, and while writing this post I just released the first bugfix update, GJS 1.54.1. Here’s what’s new!

If you prefer to watch videos rather than read, see my GUADEC talk on the subject.

JavaScript upgrade!

GJS is based on SpiderMonkey, which is the name of the JavaScript engine from Mozilla Firefox. We now use the version of SpiderMonkey from Firefox 60. (The way it goes is that we upgrade whenever Firefox makes an extended support release (ESR), which happens about once a year.)

This brings a few language improvements: not as many as in 2017 when we zipped through a backlog of four ESRs in one year, but here’s a short list:

  • Asynchronous iterators (for await (... in ...))
  • Rest operator in object destructuring (var {a, b, ...cd} = ...)
  • Spread operator in object literals (obj3 = {...obj1, ...obj2})
  • Anonymous catch (catch {...} instead of catch (e) {...})
  • Promise.prototype.finally()

There are also some removals from the language, of Mozilla-specific extensions that never made it into the web standards.

  • Conditional catch (catch (e if ...))
  • For-each-in loops (for each (... in ...))
  • Legacy lambda syntax (function (x) x * x)
  • Legacy iterator protocol
  • Array and generator comprehensions ([for (x of iterable) expr(x)])

Hopefully you weren’t using any of these, because they will not even parse anymore! I wrote a tool called moz60tool that will scan your source files and hopefully flag any uses of the removed syntax. It’s also available as a shell extension by Andy Holmes.’

person using black blood pressure monitor

Time for your code to get a checkup… Photo by rawpixel.com on Pexels.com

ByteArray

A special note about ByteArray: the SpiderMonkey upgrade made it necessary to rewrite the ByteArray class, since support for intercepting property accesses in C++-native JS objects was removed, and that was what ByteArray used internally to implement expressions like bytearray[5].

The replacement API I think would have made performance worse, and ByteArray is pretty performance critical; so I took the opportunity to replace ByteArray with JavaScript’s built-in Uint8Array. (Uint8Array didn’t exist when GJS was invented.) For this, I implemented a feature in SpiderMonkey that allows you to store a GBytes inside a JavaScript ArrayBuffer object.

The result is not 100% backwards compatible. Some functions now return a Uint8Array object instead of a ByteArray and there’s not really a way around that. The two are not really unifiable; Uint8Array’s length is immutable, for one thing. If you want the old behaviour back, you can call new ByteArray.ByteArray() on the returned Uint8Array and all the rest of your code should work as before. However, the legacy ByteArray will have worse performance than the Uint8Array, so instead you should port your code.

Technical Preview: Async Operations

The subject of Avi Zajac’s summer internship was integrating Promises and async functions with GIO’s asynchronous operations. That is, instead of this,

file.load_contents_async(null, (obj, res) => {
    const [, contentsBytes, etag] = obj.load_contents_finish(res);
    print(ByteArray.toString(contentsBytes));
});

you should be able to do this:

file.load_contents_async(null)
.then((contentsBytes, etag) => print(ByteArray.toString(contentsBytes)));

or this:

const [contentsBytes, etag] = await file.load_contents_async(null);
print(ByteArray.toString(contentsBytes));

If you don’t pass in a callback to the operation, it assumes you want a Promise instead of a callback, and will return one so that you can call .then() on it, or use it in an await expression.

This feature is a technology preview in GNOME 3.30 meaning, you must opt in for each method that you want to use it with. Opt in by executing this code at the startup of your program:

Gio._promisify(classPrototype, asyncMethodName, finishMethodName);

This is made a bit extra complicated for file operations, because Gio.File is actually an interface, not a class, and because of a bug where JS methods on interface prototypes are ignored. We also provide a workaround API for this, which unfortunately only works on local (disk) files. So the call to enable the above load_contents_async() code would look like this:

Gio._promisify(Gio._LocalFilePrototype, 'load_contents_async', 'load_contents_finish');

And, of course, if you are using an older GNOME version than 3.30 but you still want to use this feature, you can just copy the Promisify code into your own program, if the license is suitable. I’ve already been writing some code for Endless Hack in this way and it is so convenient that I never want to go back.

Debugger

At long last, there is a debugger. Run it with gjs -d yourscript.js!

The debugger commands should be familiar if you’ve ever used GDB. It is a bit bare-bones right now; if you want to help improve it, I’ve opened issues #207 and #208 for some improvements that shouldn’t be too hard to do.

The debugger is based on Jorendb, a toy debugger by Jason Orendorff which is included in the SpiderMonkey source repository as an example of the Debugger API.

Performance improvements

We’ve made some good improvements in performance, which should be especially apparent in GNOME Shell. The biggest improvement is the Big Hammer patch by Georges Stavracas, which should stop your GNOME Shell session from holding on to hundreds of megabytes at a time. It’s a mitigation of the Tardy Sweep problem which is explained in detail by Georges here. Unfortunately, it makes a tradeoff of worse CPU usage in exchange for better memory usage. We are still trying to find a more permanent solution. Carlos Garnacho also made some further improvements to this patch during the 3.30 cycle.

The other prominent improvement is better memory usage for GObjects in general. A typical GNOME Shell run contains thousands or maybe ten-thousands of GObjects, so shaving even a few bytes off per object has a noticeable effect. Carlos Garnacho started some work in this direction and I continued it. In the end we went from 128 bytes per GObject to 88 bytes. In both cases there is an extra 32 byte penalty if the object has any state set on it from JavaScript code. With these changes, GNOME Shell uses several tens of megabytes less memory before you even do anything.

I have opened two issues for further investigation, #175 and #176. These are two likely avenues to reduce the memory usage even more, and it would be great if someone were interested to work on them. If they are successful, it’s likely we could get the memory usage down to 56 bytes per GObject, and eliminate the extra 32 byte penalty.

celine-nadon-694931-unsplash.jpg

Eventually we will get to that “well-oiled machine” state… Photo by Celine Nadon on Unsplash

Developer Experience

I keep insisting it’s no coincidence, that as soon as we switched to GitLab we started seeing an uptick in contributors whom we hadn’t seen before. This trend has continued: we merged patches from 22 active contributors to GJS in this cycle, up from 13 last time.

Claudio André landed many improvements to the GitLab CI. For one thing, the program is now built and tested on more platforms and using more compile options. He also spent a lot of effort ensuring that the most common failures will fail quickly, so that developers get feedback quickly.

From my side, the maintainer tasks have gotten a lot simpler with GitLab. When I review a merge request, I can leave the questions of “does it build?” and “are all the semicolons there?” to the CI, and concentrate on the more important questions of “is this a feature we want?” and “is it implemented in the best way?” The thumbs-up votey things on issues and merge requests also provide a bit of an indication of what people would most like to see worked on, although I am not really using these systematically yet.

We have some improvements soon to be deployed to DevDocs, and GJS Guide, a site explaining some of the more basic GJS concepts. Both of these were the subject of Evan Welsh’s summer internship. Evan did a lot of work in upstream DevDocs, porting it from the current unsupported CoffeeScript version to a more modern web development stack, which will hopefully be merged upstream eventually.

mountains nature arrow guide

It’s about time we had a signpost to point the way in GJS. Photo by Jens Johnsson on Pexels.com

We also have an auto formatter for C++ code, so if you contribute code, it’s easier to avoid your branches failing CI due to linter errors. You can set it up so that it will correct your code style every time you commit; there are instructions in the Hacking file. It uses Marco Barisione’s clang-format-hooks. The process isn’t infallible, though: the CI job uses cpplint and the auto formatter uses clang-format, and the two are not 100% compatible.

There are a few miscellaneous nice things that Claudio made. The test coverage report for the master branch is automatically published on every push. And if you want to try out the latest GJS interpreter in a Flatpak, you can manually trigger the “flatpak” CI job and download one.

What’s coming in 3.32

There are a number of efforts already underway in the 3.32 cycle.

ES6 modules should be able to land! This is an often requested feature and John Renner has a mostly-working implementation already. You can follow along on the merge request.

Avi Zajac is working on the full version of the async Promises feature, both the gobject-introspection and GJS parts, which will make it no longer opt-in; Promises will “just work” with all GIO-based async operations.

Also related to async and promises, Florian Müllner is working on a new API that will simplify calling DBus interfaces using some of the new ES6 features we have gained in recent releases.

I hope to land Giovanni Campagna’s old “argument cache” patch set, which looks like it will speed up calls from JS into C by quite a lot. Apparently there is a similar argument cache in PyGObject.

Finally, and this will be the subject of a separate blog post coming soon, I think we have a plausible solution to the Tardy Sweep problem! I’m really excited to write about this, as the solution is really ingenious (I can say that, because I did not think of it myself…)

Contributors

Thanks to everyone who participated to bring GJS to GNOME 3.30: Andy Holmes, Avi Zajac, Carlos Garnacho, Christopher Wheeldon, Claudio André, Cosimo Cecchi, Emmanuele Bassi, Evan Welsh, Florian Müllner, Georges Basile Stavracas Neto, James Cowgill, Jason Hicks, Karen Medina, Ole Jørgen Brønner, pixunil, Seth Woodworth, Simon McVittie, Tomasz Miąsko, and William Barath.

As well, this release incorporated some old patches that people contributed in the past, even up to 10 years ago, that were never merged because they needed some tweaks here or there. Thanks to those people for participating in the past, and I’m glad we were finally able to land your contributions: Giovanni Campagna, Jesus Bermudez Velazquez, Sam Spilsbury, and Tommi Komulainen.

LAS GNOME

The 2018 edition of the LAS GNOME conference happened two weeks ago. I arrived in time for the second day of talks, and left early Sunday.

The conference was small but the group was energized and the talks were engaging. The group was made up of local GNOMErs, developers and designers from the US free software community, developers from KDE, and local students, among others. I was very impressed by the hard work of the volunteers. The weather in Denver was very nice. The venue was a beautiful old mansion situated close to downtown.

 A few of my favorite talks:

It was interesting to hear Aleix Pol's presentation on KDE's approach to integrating Flatpak, Snap, and Packagekit backends into their software center.

Britt Yazel's talk on Research Science and Libre Computing was very thought provoking. He talked about the enormous cost of using proprietary software and the lack of reproducibility of research outcomes due to bugs in software and unknown testing environments. It was fascinating to see the parallels between challenges software engineers themselves face in setting up production and test environments, and those faced by research scientists.

Heidi Ellis and Gregory Hislop's talk, "How Can You Make Your Open Source Project Attractive to Students?" outlined the challenges university professors face in trying to teach open source in the classroom, and how projects can make it easier. It was nice to see that GNOME's newcomers' initiatives already provides many of the necessary things: contact information for mentors, places for newcomers to ask questions, documentation on how to get started, etc.

Amisha Singla's talk on "Guarding the Maps from Vandals" explored the evolution of MapBox's approaches to detecting vandalism. They started with a rules-based approach and human review, and eventually re-wrote their system to use natural language processing and machine learning approaches.

Thanks to all the volunteers whose hard work made the event possible! Hope to see you all again next year.


September 23, 2018

Developer Center Initiative – Meeting Summary 21st September

Since last blog post there’s been two Developer Center meetings held in coordination with LAS GNOME Sunday the 9th September and again Friday the 21st September. Unfortunately I couldn’t attend the LAS GNOME meeting, but I’ll cover the general progress made here.

Test Instances Status

In the previous meetings we have been evaluating 4 possible technologies namely Sphinx, Django, Vuepress and HotDoc. Since then, the progress made in the development of these proposals has varied considerably. We got feedback from Christoph Reiter on the feasibility of using Sphinx and currently there are no efforts going towards making a test instance here. Michael was unsure he could commit the time to the Django proposal and suggests instead either Vuepress or HotDoc. For this reason the Sphinx and Django proposals have been closed off for now.

HotDoc has lately seen a lot of development by Matthieu and Thiblahute. A rough port of the Mallard-based gnome-devel-docs was demonstrated at the LAS GNOME call, so you can now for example find the Human Interface Guidelines in Markdown. Of course, there is still a long way to go, but this is a good first milestone to reach and HotDoc is the first of all the test instances to reach it. Matthieu also gave answers to criterias formulated in my previous blog post.

The main concern of HotDoc has been maintainability and the general small scale of the community surrounding it. On the other hand, Evan appears to be busy and Vuepress haven’t received attention since its initial proposal. As the choice narrows, we intend to give the test instances a last small window of time to gain activity. Simultaneously we have started to focus the short-term future efforts on improving the HotDoc test instance with Matthieu and Thiblahute.

Initial Content-plan

The second item discussed at the meeting was an initial content plan. Prior to the meeting I worked out how this content plan could look like based on Allan’s initial design. This is a summary of the proposed short-term plan:

  • The API Reference will explorable through the current gtk-doc static HTML and External API references will be linked where relevant.
  • The HIG will be ported to Markdown and maintenance from there continues in Markdown, see next bullet.
  • The tutorials section would consist of hand-ported GNOME Wiki HowDoIs and auto-ported GNOME Devel Docs. The GNOME Devel Docs repository would be ported at once to Markdown and reviewed. An announcement to the GNOME Docs mailing list when this happens. From that point on, documentation writers would be encouraged to continue edits directly through the new test instance.
  • The Distribute section will initially link to Flatpak’s Developer Documentation.
  • The Technologies overview will link to the corresponding GNOME.org page.
  • The Get Involved page will link to the GNOME Newcomer Guide on the GNOME Wiki.
  • Finally there is the GNOME Development Guide section, but this I would personally rather propose to merge with Tutorials.

There are a lot more question marks and wish thinking concerning the long-term plan, but you can read and comment on both short-term and long-term content plans in the Gitlab issue.

Next Steps

I will soon open a new framadate for a Developer Center meeting. For those interested in helping with the HotDoc test instance, feel free to file issues against it or join the discussion in the HotDoc Instance proposal. Personally I will try to get HotDoc running locally on my machine and review the current site structure so it matches closer to Allan’s proposal. I will also try to help Thiblahute with writing a migration guide from GtkDoc to HotDoc.

Reviewing the ported GNOME Devel Docs material itself is still too early, but if you would like to contribute in other ways, let us know!

LAS 2018

This month I was at my second Libre Application Summit in Denver. A smaller event than GUADEC but personally was my favorite conference so far.

One of the main goals of LAS has been to be a place for multiple platforms to discuss the desktop space and not just be a GNOME event. This year two KDE members, @aleixpol and Albert Astals Cid, who spoke about release cycle of KDE Applications, Plasma, and the history of Qt. It is always interesting to see how another project solves the same problems and where there is overlap.

The elementary folks were there since this is @cassidyjames home turf who had a great “It’s Not Always Techincal” talk as well as a talk with @danrabbit about AppCenter which are both very important areas the GNOME Project needs to improve in. I also enjoyed meeting a few other community members such as @Philip-Scott and talk about their use of elementary’s platform.

Heather from Purism spoke about the Librem 5 status which I’m excited for but has a way to go. It was great to get an opportunity to meet her since we’ve spoken online about their interest in Flatpak and GNOME-Builder.

There were some fantastic talks discussing FOSS usage at a broader level:

As always there was a big Flatpak presense and throughout we had the opportunity to discuss things like adding Qt to fdo, tracking runtime CVEs, sandboxing WebKitGTK, etc. We also had a Flatpak BoF on the last day discussing things like possibilty of selling apps and infrastructure improvements.

I really enjoyed the event overall and look forward to future LASes. Next week I will be in A Coruña, Spain, for the webengine hackfest.

GNOME Foundation Sponsored Badge

September 21, 2018

GUADEC 2018 Reminiscences

This year’s GUADEC in Almería, Spain, was over two months ago, and so here is a long overdue post about it. It was so long ago that I might as well call it a reminiscence! This will be a different kind of post than the ones I’ve done in past years, as plenty of other bloggers have already posted summaries about the talks.

Board of Directors

I didn’t even get to see that many talks anyway, as this was my first GUADEC after being elected to the GNOME Foundation board of directors and I found myself doing a lot of running around to complete things. The board has to prepare for a number of meetings including the GNOME Foundation’s Annual General Meeting that’s always held at GUADEC, and so there was plenty of preparation to be done.

So, except for a few sessions, I mainly followed the “hallway track” this year.

It’s an exciting time to be on the board; it’s been in the news recently that the GNOME Foundation has received two substantial donations, and is hiring some new roles. If you want more information and background, Rosanna Yuen, director of operations, explains all about it in this GUADEC talk.

Interns

Somehow I found myself mentoring two interns this summer, Avi Zajac and Evan Welsh, and both of them were able to attend GUADEC. (I co-mentored Evan with my coworker Manuel Quiñones, who unfortunately could not be there.) I had not done a good job introducing them to each other, but they connected with each other and realized that they were both working with me! If you haven’t already, make sure to read Avi’s blog post and Evan’s blog post for their perspectives on how it went. I was glad to have both of them there and really enjoyed meeting up in person.

Both internships have finished up in the meantime. Evan’s website is viewable here, as well as some improvements to DevDocs which I hope to deploy soon. Avi’s project was released as a technical preview in GNOME 3.30 and will be covered in my next blog post.

JavaScript Talk

I gave one scheduled talk, on GJS and JavaScript, and one unscheduled talk, on Endless Code.

I will cover the material from the JavaScript talk in my next post about the new features in GJS, but for now I wanted to post the slides for everyone’s reference. The video of the talk is here.

Endless Hack Talk

I was voted into one of the conference’s open talk slots with a proposal to talk about Endless Code (since then, renamed to Endless Hack). This is a new (well, it was new at the time of the conference) project at Endless. It’s a continuation of this feature which (I didn’t work on, but) my coworkers demoed about two years ago:

The Endless Hack product generalizes this idea to the whole desktop. The idea is that you should be able to tinker with everything, and there’s a narrative that guides you along the way and teaches you programming concepts. It’s aimed at children and young teenagers. Although this product hasn’t been released yet, and although some of the source code is currently open it’s not in a finished or usable state yet, I did want to talk about it at GUADEC because I think the ability to learn by tinkering is an important part of the free software experience and a direct consequence of one of the Four Software Freedoms, and it’s something the GNOME community should be aware of.

We also made a survey asking people about their experience learning how to program, or not learning how to program, and it’s still open, because I did not do a very good job in the talk of publicizing the link! You can fill it out here.

I haven’t dared to watch the video of me talking completely unrehearsed, but you can watch it here if you want.

Unconference

I had high hopes for organizing a GJS unconference session like last year, but after a certain point I was just completely tired out. We did eventually have a GJS session that consisted of people hacking on their favourite thing. Happily, I was able to convince Georges Stavracas to fix a regression that was preventing GNOME Shell from starting. I got a chance to work with Meg Ford on testing with JavaScript, and I also got some work done on the GJS debugger, a new feature in GNOME 3.30. I will talk about all this and more in my next post!

We also used some of the unconference time for a kickoff session for the GNOME Developer Center. Bastian Ilsø is leading this initiative and has a lot of material for you to read on what’s happened in the meantime.

Acknowledgements

I’d like to thank the GNOME Foundation for making it possible for me to attend the conference and the board meetings.

Thank you to my coworker Lisette Silva for convincing me to submit the open talk and giving some last-minute feedback beforehand.

On the topic of being part of a large and diverse community, including people whose identities you might not be able to personally understand

Adventure Time GIF: [Princess Bubblegum] People get built different. We don't need to figure it out. We just need to respect it

September 20, 2018

Speeding up AppStream: mmap’ing XML using libxmlb

AppStream and the related AppData are XML formats that have been adopted by thousands of upstream projects and are being used in about a dozen different client programs. The AppStream metadata shipped in Fedora is currently a huge 13Mb XML file, which with gzip compresses down to a more reasonable 3.6Mb. AppStream is awesome; it provides translations of lots of useful data into basically all languages and includes screenshots for almost everything. GNOME Software is built around AppStream, and we even use a slightly extended version of the same XML format to ship firmware update metadata from the LVFS to fwupd.

XML does have two giant weaknesses. The first is that you have to decompress and then parse the files – which might include all the ~300 tiny AppData files as well as the distro-provided AppStream files, if you want to list installed applications not provided by the distro. Seeking lots of small files isn’t so slow on a SSD, and loading+decompressing a small file is actually quicker than loading an uncompressed larger file. Parsing an XML file typically means you set up some callbacks, which then get called for every start tag, text section, then end tag – so for a 13Mb XML document that’s nested very deeply you have to do a lot of callbacks. This means you have to process the description of GIMP in every language before you can even see if Shotwell exists at all.

The typical way parsing XML involves creating a “node tree” when parsing the XML. This allows you treat the XML document as a Document Object Model (DOM) which allows you to navigate the tree and parse the contents in an object oriented way. This means you typically allocate on the heap the nodes themselves, plus copies of all the string data. AsNode in libappstream-glib has a few tricks to reduce RSS usage after parsing, which includes:

  • Interning common element names like description, p, ul, li
  • Freeing all the nodes, but retaining all the node data
  • Ignoring node data for languages you don’t understand
  • Reference counting the strings from the nodes into the various appstream-glib GObjects

This still has a both drawbacks; we need to store in hot memory all the screenshot URLs of all the apps you’re never going to search for, and we also need to parse all these long translated descriptions data just to find out if gimp.desktop is actually installable. Deduplicating strings at runtime takes nontrivial amounts of CPU and means we build a huge hash table that uses nearly as much RSS as we save by deduplicating.

On a modern system, parsing ~300 files takes less than a second, and the total RSS is only a few tens of Mb – which is fine, right? Except on resource constrained machines it takes 20+ seconds to start, and 40Mb is nearly 10% of the total memory available on the system. We have exactly the same problem with fwupd, where we get one giant file from the LVFS, all of which gets stored in RSS even though you never have the hardware that it matches against. Slow starting of fwupd and gnome-software is one of the reasons they stay resident, and don’t shutdown on idle and restart when required.

We can do better.

We do need to keep the source format, but that doesn’t mean we can’t create a managed cache to do some clever things. Traditionally I’ve been quite vocal against squashing structured XML data into databases like sqlite and Xapian as it’s like pushing a square peg into a round hole, and forces you to think like a database doing 10 level nested joins to query some simple thing. What we want to use is something like XPath, where you can query data using the XML structure itself.

We also want to be able to navigate the XML document as if it was a DOM, i.e. be able to jump from one node to it’s sibling without parsing all the great, great, great, grandchild nodes to get there. This means storing the offset to the sibling in a binary file.

If we’re creating a cache, we might as well do the string deduplication at creation time once, rather than every time we load the data. This has the added benefit in that we’re converting the string data from variable length strings that you compare using strcmp() to quarks that you can compare just by checking two integers. This is much faster, as any SAT solver will tell you. If we’re storing a string table, we can also store the NUL byte. This seems wasteful at first, but has one huge advantage – you can mmap() the string table. In fact, you can mmap the entire cache. If you order the string table in a sensible way then you store all the related data in one block (e.g. the <id> values) so that you don’t jump all over the cache invalidating almost everything just for a common query. mmap’ing the strings means you can avoid strdup()ing every string just in case; in the case of memory pressure the kernel automatically reclaims the memory, and the next time automatically loads it from disk as required. It’s almost magic.

I’ve spent the last few days prototyping a library, which is called libxmlb until someone comes up with a better name. I’ve got a test branch of fwupd that I’ve ported from libappstream-glib and I’m happy to say that RSS has reduced from 3Mb (peak 3.61Mb) to 1Mb (peak 1.07Mb) and the startup time has gone from 280ms to 250ms. Unless I’ve missed something drastic I’m going to port gnome-software too, and will expect even bigger savings as the amount of XML is two orders of magnitude larger.

So, how do I use this thing. First, lets create a baseline doing things the old way:

$ time appstream-util search gimp.desktop
real	0m0.645s
user	0m0.800s
sys	0m0.184s

To create a binary cache:

$ time xb-tool compile appstream.xmlb /usr/share/app-info/xmls/* /usr/share/appdata/* /usr/share/metainfo/*
real	0m0.497s
user	0m0.453s
sys	0m0.028s

$ time xb-tool compile appstream.xmlb /usr/share/app-info/xmls/* /usr/share/appdata/* /usr/share/metainfo/*
real	0m0.016s
user	0m0.004s
sys	0m0.006s

Notice the second time it compiled nearly instantly, as none of the filename or modification timestamps of the sources changed. This is exactly what programs would do every time they are launched.

$ df -h appstream.xmlb
4.2M	appstream.xmlb

$ time xb-tool query appstream.xmlb "components/component[@type=desktop]/id[firefox.desktop]"
RESULT: <id>firefox.desktop</id>
RESULT: <id>firefox.desktop</id>
RESULT: <id>firefox.desktop</id>
real	0m0.008s
user	0m0.007s
sys	0m0.002s

8ms includes the time to load the file, search for all the components that match the query and the time to export the XML. You get three results as there’s one AppData file, one entry in the distro AppStream, and an extra one shipped by Fedora to make Firefox featured in gnome-software. You can see the whole XML component of each result by appending /.. to the query. Unlike appstream-glib, libxmlb doesn’t try to merge components – which makes it much less magic, and a whole lot simpler.

Some questions answered:

  • Why not just use a GVariant blob?: I did initially, and the cache was huge. The deeply nested structure was packed inefficiently as you have to assume everything is a hash table of a{sv}. It was also slow to load; not much faster than just parsing the XML. It also wasn’t possible to implement the zero-copy XPath queries this way.
  • Is this API and ABI stable?: Not yet, as soon as gnome-software is ported.
  • You implemented XPath in c‽: No, only a tiny subset. See the README.md

Comments welcome.

Glade in Libre Application Summit

First of all ISponsored by GNOME Foundation would like to thanks the GNOME foundation for sponsoring my trip to Denver to attend Libre Application Summit

As usual, it was a great opportunity to catch up with old friends and make new ones specially outside the GNOME community.

This opportunity I talked about the plans I have to integrate Glade with Gnome Builder and other IDEs

You can find the slides of my talk as PDF here, and the sources here!

Sources you say?

Yes sources, since my talk about custom UI interfaces in 2013 I been making all my slides with Glade and using glade-previewer to present them live

glade-previewer is a tiny application shipped with Glade used mainly to preview UI

It’s relevant options are:

-f, --filename=FILENAME  Name of the file to preview
--screenshot             File name to save a screenshot
--css                    CSS file to use
--slideshow              Make a slideshow of every toplevel widget
                         by adding them in a GtkStack

Normally to preview a glade file with a custom css file you would use a command like

$ glade-previewer -f talk.glade --css talk.css

Now if you instead want to make a presentation with it all you need to do is add –slideshow option and glade-previewer will pack every toplevel widget in a GtkStack and switch between pages with PageUp/PageDown buttons

As a bonus I extended –screenshot option so that when used in conjunction with –slideshow it will take a screenshot of every toplevel and save them as multiple pages if the format supports it!

September 19, 2018

GNOME.Asia 2018

GNOME.Asia 2018 was co-hosted with COSCUP and openSUSE Asia this year in Taipei, Taiwan. It was a good success and I enjoyed it a lot. Besides, meeting old friends and making new ones are always great.

Talks

  • Desktop applications: life inside a sandbox — David King

Flatpak, as you all know, is quite popular and useful these days. So it’s good to know some implementation details from this talk.

  • Introducing Team Silverblue — Matthias Clasen

As Matthias mentioned in his talk, it’s his first time to give this talk. And I think it was quite a success. It’s a new variant of Fedora Workstation and it provides the excellent support to container-based workflows.

  • The future of the computer classrooms – GNOME inside — Eric Sun

I have to say I enjoyed this talk the most in the conference. Eric used ezgo Linux as an example and explained some interesting ideas, which blows my mind. When we were young and in the computer class, we were taught to use Microsoft Office, Photoshop, etc. All these are . And these software are gonna be the the top options when you are thinking about choosing a office/picture editing software.

  • There are more, but I can’t include them all

Socials

  • The welcom party was held at a bar near Taipei 101. The bar has a open platform, where you have a great view, which is pretty good. You can find beer, food and of course friends there.
  • We had a one-day tour to Taipei Palace Museum and Taipei 101 the day after the conference. We had various dumplings and some delicious food for lunch at Din Tai Fung(A top restaurant at 101 where you need queue around 90 mins even at weekday). It was well organized. And a big thank you for Max :)

Finally

Thanks Max for all the effort making this conference happen. Thanks GNOME Foundation for Sponsorship my trip to the conference.

September 18, 2018

Moving from Wordpress

Wordpress

wordpress blog

I have been using Wordpress for long. Basically since I joined GNOME because it was required for GSoC. Wordpress works, it’s okay. There are themes, it has a wysiwyg editor and you can embed images and videos quite easily. It kinda does the job…

The bad

Now, one of the biggest problems are ads. Not that they exist, which is completely understandable, but rather that they are kinda crazy. I got reports that sometimes they were about guns in USA, or they lead to scam sites.

I also missed a way to link to my personal twitter/linkdin/gitlab. That’s quite useful for people to check what other things I write about and how to get more info about what I do.

The ugly

More on the technical side, one of the issues was that I coulnd’t create a draft post and share it in private with other people so I could get some review. This was specially required for Nautilus anouncements. And in case I could, how could we collaborate in editing the post itself? That was simply not there.

Most importantly, I couldn’t give personality to the blog. In the same way I want the software I use being neutral because for it’s just a tool, my personal blog should be more repsentative on how I am and how I express myself. With Wordpress this was not possible. Hell, I couldn’t even put some colors here and there, or take the bloat out from some of the widgets provided by the Wordpress themes.

The enlightenment…

didier's blog

Once upon a time, on a stormy day, I wrote the blog post about the removal and future plans of the desktop icons as a guest writter on Didier’s Roche blog. And here came the enligthening, he was using some magic PR workflow in GitHub where I could just fix stuff, request to merge those changes, review happens, then gets accepted and published all automatically with CI.

Finally you could review, share drafts, and collaborate much easily than with Wordpress!

Not only that, but also his blog was much more personalized, closer to how he wanted to express himself.

The decision

One thing I had have in the back of my mind for some time is that I need to improve my skills with non-GNOME stuff, specially web and cloud technologies. So what a better oportunity than trying to set up a static web generator for blogs (and other stuff) to mimic the features of Didier’s blog?

And decided was it, got some free time this weekend and decided to learn this magic stuff.

The technology

I decided to go with Hugo, because back when I was experimenting with GitLab pages and a project page for Nautilus Hugo seemed to be the easiest and most convenient tech to create a static website that just works. Overall, it seems that Hugo is the most used static website generator for blogs.

I also decided that I would get a theme based in the well known and well maintained bootstrap. Most themes had some custom CSS, all delicately and manually crafted. But let’s be honest, I don’t want to maintain this, I wanted something simple I can go with.

So I chose the minimal theme wich is based in bootstrap and then applied my own changes such as a second accent (the red in the titles), support for centered images, the Ubuntu font (which I use it everywhere I can), some navbar changes, full content rss support, lot of spacing adjustments, softer main text coloring, etc.

Also, I could finally put a decent comment section by using DisQus, which is added to Hugo with a single line. Eventually I would like to go with a free software solution such as Talkyard, so far I didn’t have luck to make it work.

The nicest thing of Hugo is that adding a new post is a matter of dropping a MarkDown file in the post folder. And that’s it. That easy.

The result

The result

So here’s the result. I think it’s quite an improvement versus what I had in Wordpress, although Antonio says that the page looks like Nautilus… I guess I cannot help myself. Let’s see how it works in the future, specially since there is no wysiwyg editor. To be honest, using MarkDown is so easy that I don’t see that as a problem so far.

I can even embed some code with highlighting for free:

def do_measure(self, orientation: Gtk.Orientation,
               for_size: int) -> Tuple[int, int, int, int]:
    child = self.get_child()
    if not child:
        return -1, -1, -1, -1

    minimum, natural, _x, _x = child.measure(orientation, for_size)
    if self._max_width is not None and orientation == Gtk.Orientation.HORIZONTAL:
        natural = min(self._max_width, natural)

    return minimum, natural, -1, -1

And use Builder with Vim emulation to write this. That is already a big win!

If you want to take a look at the code or do something similar, feel free to take a look and use anything from the code in GNOME’s GitLab. I also added an example post in order to see all formats for headings, bullet lists, images, code blocks, etc. Any feedback about the looks, functionality, content, etc. is welcome; finally I would be able to do something about it 😏.

Hasta otra amigos 🍹

September 17, 2018

Flatpak on windows

As I teased about last week I recently played around with WSL, which lets you run Linux applications on Windows. This isn’t necessarily very useful, as there isn’t really a lack of native applications on Windows, but it is still interesting from a technical viewpoint.

I created a wip/WSL branch of flatpak that has some workarounds needed for flatpak to work, and wrote some simple docs on how to build and test it.

There are some really big problems with this port. For example, WSL doesn’t support seccomp or network namespaces which removes some of the utility of the sandbox. There is also a bad bug that makes read-only bind-mounts not work for flatpak, which is really unsafe as apps can modify themselves (or the runtime). There were also various other bugs that I reported. Additionally, some apps rely on things on the linux host that don’t exist in the WSL environment (such as pulseaudio, or various dbus services).

Still, its amazing that it works as well as it does. I was able to run various games, gnome and kde apps, and even the linux versions of telegram. Massive kudos to the Microsoft developers who worked on this!

I know you crave more screenshots, so here is one:

September 16, 2018

Gavi's Song sheet music with TuxGuitar and LilyPond

A year or two ago I bought Lindsey Stirling’s Album Brave Enough. It’s wonderful all around, but I really fell in love with Gavi’s Song. Three weeks ago I took a stab at playing this on my guitar. It’s technically not actually that difficult – After listening to the original and trying to repeat it for several days, I can now actually play through it without too many hiccups (still far from being YouTube’able, though).

Free software projects

My own projects/main developer python-dbusmock: Mock D-Bus objects for creating integration tests umockdev: record/mock hardware devices for bug reports and regression tests Apport: create rich crash and bug information on Linux clients, and send them to bug trackers like Launchpad python-distutils-extra: Extensions and “automagic” wrapper for Python’s distutils class postgresql-common: Provide a multi-version/multi-instance management for the PostgreSQL RDBMS Other projects I’m contributing to cockpit: Administer Linux servers via a web browser (tech lead, full-time developer) udev/systemd: Maintaining keymaps, various bug fixes Debian: The universal operating system (developer) GNOME desktop (committer, foundation member, bug fixing and reporting) PyGObject: Use GObject based C libraries from Python through gobject-introspection udisks: occasional bug fixes upower: occasional bug fixes

September 15, 2018

Ducktype parser extensions

When designing Ducktype, I wanted people to be able to extend the syntax, but I wanted extensions to be declared and defined, so we don’t end up with something like the mess of Markdown flavors. So a Ducktype file can start with a @ducktype/ declaration that declares the version of the Ducktype syntax and any extensions in use. For example:

@ducktype/1.0 if/1.0

This declares that we’re using version 1.0 of the Ducktype syntax,that we want an extension called if, and that we want version 1.0 of that extension.

Up until last week, extensions were just theoretical. I’ve now added two extension points to the Ducktype parser, and I plan to add three or four more. Both of these are exercised in the _test extension, which is fairly well commented so you can learn from it.

Let’s look at the extensions we have, plus the ones I plan to add.

Block line parser

This extension is implemented. It allows extensions to handle really any sort of line in block context, adding any sort of new syntax. Extensions only get to access to lines after headings, comments, fences, and a few other things are handled. This is a limitation, but it’s one that makes writing extensions much easier.

Let’s look at an actual example that uses this extension: Mallard Conditionals. You can use Mallard Conditionals in Ducktype just fine without any syntax extension. Just declare the namespace and use the elements like any other block element:

@ducktype/1.0
@namespace if http://projectmallard.org/if/1.0/

= Conditional Example

[if:if test=target:html]
  This is a paragraph only shown in HTML.

But with the if/1.0 Ducktype syntax extension, we can skip the namespace declaration and use a shorthand for tests:

@ducktype/1.0 if/1.0

= Conditional Example

? target:html
  This is a paragraph only shown in HTML.

We even have special syntax for branching with if:choose elements:

@ducktype/1.0 if/1.0

= Conditional Branching Example

??
  ? platform:fedora
    This paragraph is only shown on Fedora.
  ? platform:ubuntu
    This paragraph is only shown on Ubuntu.
  ??
    This paragraph is shown on any other operating system.

(As of right now, you actually have to use if/experimental instead of if/1.0. But that extension is pretty solid, so I’ll change it to if/1.0 along with the 1.0 release of the parser.)

Directive handler

Ducktype files can have parser directives at the top. We’ve just seen the @namespace parser directive to declare a namespace. There is an implemented extension point for extensions to handle parser directives, but not yet a real-world extension that uses it.

Extensions only get to handle directives with a prefix matching the extension name. For example, the _test extension only gets to see directives that look like @_test:foo.

Block element handler

This extension is not yet implemented. I want extensions to be able to handle standard-looking block declarations with a prefix. For example, I want the _test extension to be able to do something with a block declaration that looks like this:

[_test:foo]

In principle, you could handle this with the current block line parser extension point, but you’d have to handle parsing the block declaration by yourself, and it might span multiple lines. That’s not ideal.

Importantly, I want both block line parsers and block element handlers to be able to register themselves to handle future lines, so they can have special syntax in following lines. Here is how an extension for CSV-formatted tables might look:

@ducktype/1.0 csv/1.0

= CSV Table Example

[csv:table frame=all rules=rows]
one, two, three
eins, zwei, drei
uno, dos, tres

Inline element handler

This extension is not yet implemented. Similar to block element handlers, I want extensions to be able to handle standard-looking inline markup. For example, I want the _test extension to be able to do something with inline markup that looks like this:

$_test:foo(here is the content)

For example, a gnome extension could make links to GitLab issue reports easier:

$gnome:issue(yelp#138)

Inline text parser

This extension is not yet implemented. I also want extensions to be able to handle arbitrary inline markup extensions, things that don’t even look like regular Ducktype markup. This is what you would need to create Markdown-like inline markup like *emphasis* and `monospace`.

This extension might have to come in two flavors: before standard parsing and after. And it may be tricky because you want each extension to get a crack at whatever text content was output by other extensions, except you probably also want extensions to be able to block further parsing in some cases.

All in all, I’m really happy with the Ducktype syntax and parser, and how easy it’s been to write extension points so far.

Release of Foundry (previously known as rlife) 0.2.0

These past weeks, I’ve been working a lot on my side project and I’ve made a new release of it. First of all, the project has been renamed “Foundry” (instead of “rlife”). I wanted to find a better name for this project and as this project is now actually based on Vulkan (that was my primary objective when I started it), I thought it would be a good idea to give a name related to it. Plus, there was no crates already named “Foundry”.

So the biggest change is that the computations for passing from a generation of a grid to the next one are not done with the CPU anymore but with the GPU via the Vulkan API. To add the Vulkan support, I’ve used vulkano. The grid is represented as a Vec where each cell is a u8. So instead of computing and writing sequentially the new states of the cells in another new grid, we have now two grids (one for the current generation and the other for the next one) contained inside images stored on the Vulkan device’s memory (ideally the graphic card’s memory when there is one on the machine) and the device will launch parallel computations for determining and writing the next states of the cells.

So what are the results in term of performances? It turned out that there are huge gains regarding the time which is taken to compute the next generations of grids. Especially for computing a lot of generations at once and/or for large grids. Here are the results I’ve got with my machine that have an Intel Core i7-6700 as a CPU and an AMD Radeon RX 480 as a GPU (I’ve first generated a grid filled randomly and with a certain requested size and then run the computations):

    • calculating 1000 generations of a toroidal grid with a width of 1024 cells and a height of 1024 cells with the CPU: 74.040 seconds
    • doing the same with the GPU: 0.754 seconds
    • calculating 1000 generations of a resizable grid with a width of 1024 cells and a height of 1024 cells with the CPU: 100.603 seconds
    • doing the same with the GPU: 1.968 seconds
    • calculating 1 generation of a toroidal grid with a width of 16384 cells and a height of 16384 cells with the CPU: 18.917 seconds
    • doing the same with the GPU: 0.083 seconds
    • calculating 1 generation of a resizable grid with a width of 16384 cells and a height of 16384 cells with the CPU: 25.903 seconds
    • doing the same with the GPU: 0.243 seconds

I will soon write proper benchmarks for Foundry for better measurements.

Obviously, this is the very first implementation of the Vulkan API support so there are a lot of optimizations left to do.

See the PR that brings the Vulkan support for more details.

The next goal is to find a way for rendering the grid using Vulkan, so that Foundry can be used within GUI applications.

Behind the GNOME 3.30 Release Video

The GNOME 3.30 release video was announced earlier this week on Youtube. Additionally we now have a GNOME channel on Peertube by suggestion of Greg.

This marks the 10th release video for me which I find super exciting. The videos has been excellent platforms for me to learn and allowed me to reach a consistent level of production quality. For instance, check out the GNOME 3.12 release video which was my first video production.

Production moved to Gitlab

With each video I experiment with new workflows. Traditionally I have been involved in every step of the production apart from the voice-over with very few opportunities for others to step in and contribute. With Gitlab’s powerful issue tracking system, this no longer needs to be the case. This has meant that I can spend more time on production in Blender and spread out the other aspects of production to the GNOME community. The work done ithisn the GNOME 3.30 Release video is covered by the following Gitlab issues:

  • The 3.30 Release Video Main Issue provides an overview of the 10 main milestones in producing the release video (11 participants) with regular updates from me on how production was progressing.
  • The 3.30 Content Gathering Issue collected information from GNOME developers on what had changed in apps and organized how those changes should be screen recorded (25 participants / 16 tasks).
  • The 3.30 Video Manuscript Issue used the collected information to create a narrative around the changes resulting in a manuscript Karen and Mike could produce a voice-over from (6 participants / 3 tasks).
  • The 3.30 Animation Issue gave an overview of the animations which needed to be produced to match the manuscript and screen recordings (3 participants / 12 tasks).
  • The 3.30 Music Issue provided general guidelines to Simon on production of the music and in the future it could provide opportunity for community members to provide additional input.

The sheer number of participants should give you an idea how much big a relief opening the production has been for me. I still account for managing the production, animating the majority of the video and having a finger in most tasks, but it helps me focus my efforts on providing a higher quality result.

Future Challenges

There are still aspects of the production which are still not public. For example, I am not sharing drafts of the video prior to production which creates less feedback in the animation process and makes it harder for translation teams to provide adequate thistranslations in some cases. This is simply to avoid leaks of the production prior to release which has happened in the past and is a big pain point, considering the sheer effort everyone is putting into this. For next time I will experiment with releasing early stage work (animatics and sketches) to see if this could be a meaningful replacement.

For developers, there were many questions and issues regarding the process of screen recording which is documented in this issue. The production sets requirements to resolution and FPS which is not always possible for developers to meet due to hardware and software limitations. I would like feedback from you on how you think this went and how we might be able to improve the tooling. Let me know!

Finally, we have had two unforeseen events which caused the video to be delayed by 5 days. First, we are currently unable to convert subtitles for the release video due to a bug in po2sub. Secondly, the delay was caused by me and Simon having busy schedules close to the release date which is always hard to predict. However, the general policy here is to rather release late than release something unfinished. I am confident that as the Gitlab workflow matures and we improve how work is scheduled, these delays can be minimized.

Conclusion

I hope you enjoyed this insight into the release video production. If you are interested in participating or following GNOME 3.32 Release Video production, subscribe to the 3.32 Release Video Gitlab issue. Thanks to everyone who contributed this cycle!

September 14, 2018

2018-09-14 Friday

  • Pleased to my chat with Randal & Dan at FLOSS Weekly published.
  • Booked travel to DINAcon 2018 will be available to do some LibreOffice hackfest'y stuff, and a talk on "How best to migrate to LibreOffice".

September 13, 2018

2018-09-13 Thursday

  • Mail; distracted by a bit of hacking for a customer. Marketing & Sales call, ESC call, early dinner, interview.
  • Pleased to see Jona publish our latest LibreOffice Infographic giving an update on what we're doing.

Fractal contribution report: improvements for the context menu

These past weeks, I’ve been mainly working on my side project (rlife) but I’ve also done some small improvements for the context menu in Fractal.

First, I found out that there were problems regarding the line returns when inserting a quote: when clicking on the button “Reply”, there was a quote inserted at the beginning of the message input but there were an empty lines between each lines of the quote so for instance, we had this:

> A first line

> A second line

Answer

instead of

> A first line
> A second line

Answer

It was because there was an extra new line character appended at the end of each lines of the quote when inserting it in the message input. See this MR for more details.

Next, there were a problem with the name completion in the message input: if after doing a line return, you tried to write a name and complete it, it wouldn’t work; you absolutely had to have a space inserted just before the name you wanted to complete to get it work. Here is an example:

Capture du 2018-09-13 18-24-07.png

If we tried to complete “Rio” to have “Riot-bot”, Fractal would search the matches for “line\nRio” among the room’s user names instead of searching for “Rio”. We would have to add at least a space before “Rio” to have the good match. So to fix it, we had to split the strings to search with all white spaces instead of just ” “. Here is my MR to fix it.

I also have the redacted messages simply hidden instead of having them displayed as “Deleted” in this MR.

Finally, I’ve added appropriate actions for images in the context menu. I added buttons like “Open With…”, “Save Image As…” and “Copy Image” and removed the “Copy Text” button for messages that are of the type “m.image”. It looks like this:

Capture du 2018-09-13 18-32-03.png

Here is the MR that implemented it.

I also have an open MR for hiding the option to delete messages in the context menu when the user doesn’t have the right to do so (i.e. for the user’s own messages or when it has the right to do so in the room (e.g. for moderators or owners)). It’s pending for now because there are work done to reliably calculate the power level of a user given a certain room.

Google Code-in 2018 and Wikimedia: Mentors and smaller tasks wanted!

Google Code-in will take place again soon (from October 23 to December 13). GCI is an annual contest for 13-17 year old students to start contributing to free and open projects. It is not only about coding: We also need tasks about design, documentation, outreach/research, and quality assurance. And you can mentor them!

Last year, 300 students worked on 760 Wikimedia tasks, supported by 51 mentors from our community.

  • Your gadget code uses some deprecated API calls?
  • You’d enjoy helping someone port your template to Lua?
  • You’d welcome some translation help (which cannot be performed by machines)?
  • Your documentation needs specific improvements?
  • Your user interface has some smaller design issues?
  • Your Outreachy/Summer of Code project welcomes small tweaks?
  • You have tasks in mind that welcome some research?

Note that “beginner tasks” (e.g. “Set up Vagrant”) and generic tasks are very welcome (like “Choose and fix 2 PHP7 issues from the list in this task” style).

If you have tasks in mind which would take an experienced contributor 2-3 hours, become a mentor and add your name to our list!

Thank you in advance, as we cannot run this without your help.

September 11, 2018

The Commons Clause doesn't help the commons

The Commons Clause was announced recently, along with several projects moving portions of their codebase under it. It's an additional restriction intended to be applied to existing open source licenses with the effect of preventing the work from being sold[1], where the definition of being sold includes being used as a component of an online pay-for service. As described in the FAQ, this changes the effective license of the work from an open source license to a source-available license. However, the site doesn't go into a great deal of detail as to why you'd want to do that.

Fortunately one of the VCs behind this move wrote an opinion article that goes into more detail. The central argument is that Amazon make use of a great deal of open source software and integrate it into commercial products that are incredibly lucrative, but give little back to the community in return. By adopting the commons clause, Amazon will be forced to negotiate with the projects before being able to use covered versions of the software. This will, apparently, prevent behaviour that is not conducive to sustainable open-source communities.

But this is where things get somewhat confusing. The author continues:

Our view is that open-source software was never intended for cloud infrastructure companies to take and sell. That is not the original ethos of open source.

which is a pretty astonishingly unsupported argument. Open source code has been incorporated into proprietary applications without giving back to the originating community since before the term open source even existed. MIT-licensed X11 became part of not only multiple Unixes, but also a variety of proprietary commercial products for non-Unix platforms. Large portions of BSD ended up in a whole range of proprietary operating systems (including older versions of Windows). The only argument in favour of this assertion is that cloud infrastructure companies didn't exist at that point in time, so they weren't taken into consideration[2] - but no argument is made as to why cloud infrastructure companies are fundamentally different to proprietary operating system companies in this respect. Both took open source code, incorporated it into other products and sold them on without (in most cases) giving anything back.

There's one counter-argument. When companies sold products based on open source code, they distributed it. Copyleft licenses like the GPL trigger on distribution, and as a result selling products based on copyleft code meant that the community would gain access to any modifications the vendor had made - improvements could be incorporated back into the original work, and everyone benefited. Incorporating open source code into a cloud product generally doesn't count as distribution, and so the source code disclosure requirements don't trigger. So perhaps that's the distinction being made?

Well, no. The GNU Affero GPL has a clause that covers this case - if you provide a network service based on AGPLed code then you must provide the source code in a similar way to if you distributed it under a more traditional copyleft license. But the article's author goes on to say:

AGPL makes it inconvenient but does not prevent cloud infrastructure providers from engaging in the abusive behavior described above. It simply says that they must release any modifications they make while engaging in such behavior.

IE, the problem isn't that cloud providers aren't giving back code, it's that they're using the code without contributing financially. There's no difference between what cloud providers are doing now and what proprietary operating system vendors were doing 30 years ago. The argument that "open source" was never intended to permit this sort of behaviour is simply untrue. The use of permissive licenses has always allowed large companies to benefit disproportionately when compared to the authors of said code. There's nothing new to see here.

But that doesn't mean that the status quo is good - the argument for why the commons clause is required may be specious, but that doesn't mean it's bad. We've seen multiple cases of open source projects struggling to obtain the resources required to make a project sustainable, even as many large companies make significant amounts of money off that work. Does the commons clause help us here?

As hinted at in the title, the answer's no. The commons clause attempts to change the power dynamic of the author/user role, but it does so in a way that's fundamentally tied to a business model and in a way that prevents many of the things that make open source software interesting to begin with. Let's talk about some problems.

The power dynamic still doesn't favour contributors

The commons clause only really works if there's a single copyright holder - if not, selling the code requires you to get permission from multiple people. But the clause does nothing to guarantee that the people who actually write the code benefit, merely that whoever holds the copyright does. If I rewrite a large part of a covered work and that code is merged (presumably after I've signed a CLA that assigns a copyright grant to the project owners), I have no power in any negotiations with any cloud providers. There's no guarantee that the project stewards will choose to reward me in any way. I contribute to them but get nothing back in return - instead, my improved code allows the project owners to charge more and provide stronger returns for the VCs. The inequity has shifted, but individual contributors still lose out.

It discourages use of covered projects

One of the benefits of being able to use open source software is that you don't need to fill out purchase orders or start commercial negotiations before you're able to deploy. Turns out the project doesn't actually fill your needs? Revert it, and all you've lost is some development time. Adding additional barriers is going to reduce uptake of covered projects, and that does nothing to benefit the contributors.

You can no longer meaningfully fork a project

One of the strengths of open source projects is that if the original project stewards turn out to violate the trust of their community, someone can fork it and provide a reasonable alternative. But if the project is released with the commons clause, it's impossible to sell any forked versions - anyone who wishes to do so would still need the permission of the original copyright holder, and they can refuse that in order to prevent a fork from gaining any significant uptake.

It doesn't inherently benefit the commons

The entire argument here is that the cloud providers are exploiting the commons, and by forcing them to pay for a license that allows them to make use of that software the commons will benefit. But there's no obvious link between these things. Maybe extra money will result in more development work being done and the commons benefiting, but maybe extra money will instead just result in greater payout to shareholders. Forcing cloud providers to release their modifications to the wider world would be of benefit to the commons, but this is explicitly ruled out as a goal. The clause isn't inherently incompatible with this - the negotiations between a vendor and a project to obtain a license to be permitted to sell the code could include a commitment to provide patches rather money, for instance, but the focus on money makes it clear that this wasn't the authors' priority.

What we're left with is a license condition that does nothing to benefit individual contributors or other users, and costs us the opportunity to fork projects in response to disagreements over design decisions or governance. What it does is ensure that a range of VC-backed projects are in a better position to improve their returns, without any guarantee that the commons will be left better off. It's an attempt to solve a problem that's existed since before the term "open source" was even coined, by simply layering on a business model that's also existed since before the term "open source" was even coined[3]. It's not anything new, and open source derives from an explicit rejection of this sort of business model.

That's not to say we're in a good place at the moment. It's clear that there is a giant level of power disparity between many projects and the consumers of those projects. But we're not going to fix that by simply discarding many of the benefits of open source and going back to an older way of doing things. Companies like Tidelift[4] are trying to identify ways of making this sustainable without losing the things that make open source a better way of doing software development in the first place, and that's what we should be focusing on rather than just admitting defeat to satisfy a small number of VC-backed firms that have otherwise failed to develop a sustainable business model.

[1] It is unclear how this interacts with licenses that include clauses that assert you can remove any additional restrictions that have been applied
[2] Although companies like Hotmail were making money from running open source software before the open source definition existed, so this still seems like a reach
[3] "Source available" predates my existence, let alone any existing open source licenses
[4] Disclosure: I know several people involved in Tidelift, but have no financial involvement in the company

comment count unavailable comments

ASG! 2018 Tickets

All Systems Go! 2018 Tickets Selling Out Quickly!

Buy your tickets for All Systems Go! 2018 soon, they are quickly selling out! The conference takes place on September 28-30, in Berlin, Germany, in a bit over two weeks.

Why should you attend? If you are interested in low-level Linux userspace, then All Systems Go! is the right conference for you. It covers all topics relevant to foundational open-source Linux technologies. For details on the covered topics see our schedule for day #1 and for day #2.

For more information please visit our conference website!

See you in Berlin!

GNOME.Asia Summit 2018

COSCUP 2018, GNOME.Asia Summit 2018 and openSUSE.Asia Summit 2018 by Tooth

Last year I’d been COSCUP 2017 at first time, it gave a great impression of COSCUP. It’s open, freedom and very energetic. It’s very nice this year GNOME.Asia Summit joint with COSCUP and openSUSE.Asia.

In the first day, I participated several sessions, Benjamin’s “Supporting Miracast on GNOME”, David’s “Desktop application: life inside a sandbox”, Kat’s “Plan your testing” and ZengZhengJia’s “GNOME translation”.

Children workshop by daisuke1230

At afternoon there are a workshop for children to hack a “Flying paper cup”, my daughter very liked this lovely workshop.

GNOME.Asia BoF on Saturday

And at night we had a GNOME.Asia BoF to review the Good vs. Bad, we collected a lot of ideas to make the  GNOME.Asia better in future.

In the second day, I made a topic about “flatpak vs. snap”, introduced some concepts and basic usages. And I also listened Max’s “Community experience”, Kukuh’s “GNOME Recipes”, Shobha’s “Humanitarian FOSS projects” and Wen’s “GNOME.Asia experience”.

As before, the great part of conference is that I enjoy meeting a lot of old friends and making new friends in these two days.

GNOME.Asia group photo by ArvinWu
GNOME.Asia and openSUSE.Asia group photo by ArvinWu

Finally, thanks GNOME Foundation’s sponsorship for my trip to GNOME.Asia Summit 2018.

More pictures from below links.

https://www.flickr.com/groups/gnomeasia2018/

https://www.flickr.com/photos/coscup/albums

 

 

September 10, 2018

Gtranslator Resurrection

The last week I received a telegram message about Gtranslator, that was unmaintained for a long time. GNOME translators uses different tools to translate .po files, Gtranslator is a tool for translator that is integrated with the GNOME desktop, but with the time, Gtranslator is getting old and there are several known bugs that never get fixed.

So I decided to go ahead and become the maintainer of Gtranslator with the main idea of update the interface and fix mayor bugs.

Why we should care about Gtranslator

PO files are plain text files that can be edited with any text editor, but there're a lot of tools for translator to simplify the edition of .po files, these way the program will avoid formatting problems and can also provide some tools to help in the translation.

There's a lot of tools to edit .po files, so, why we need another application? why translators can't pick another useful tool if Gtranslator is unmaintained?

The main reason is the same as we've a text editor or the gnome-builder or other developer tools, because all users want to use applications integrated in the desktop, so it always will be better for a GNOME user a native GNOME app to edit .po files than a Qt one or a webpage.

The other reason is that we need to simplify the translators life to have better and more translations and that can only be done if we're developping the tool that translator will use, because in the future we can integrate Gtranslator with the gnome gitlab, or with Damned Lies, to download or upload translation files.

What's the plan

Gtranslator has a GNOME 2.0 interface, with toolbar, menus, statusbar and dockable widgets:

But in the git repo, the main branch was broken. Someone starts a big refactor, in 2015, to try to update the interface to look more like a modern GNOME 3 application but that refactor never ended so the master branch compiles, but doesn't do anything. There was a lot of widgets that was half ported to a modern code.

There's also some designs to modernize the interface, but the work wasn't finished.

My plan is to continue that work, modernize the UI, remove all deprecated code and warnings and then continue the development from there.

I don't want to spend a lot of time reworking the whole app to have a new one in two months, I want to have a functional app as soon as possible, and continue with a continuous development, improving the UI and the functionality needed by the translator team.

So the roadmap that I've in mind is:

  1. Fix the current master state
  2. Fix the known bugs
  3. Redesign the interface
  4. Enhance linking with Damned Lies

What I've done in one week

I started updating the build system, from autotools to meson build, and then I've added a flatpak manifest file, so now gnome-builder is able to build Gtranslator using meson and flatpak and it's easier to develop this way.

This build tools modernization is great because with the new gitlab we've now continuous integration, and that means that the gitlab compiles and generates a flatpak bundle for each commit, making it easier for testers to test new functionality or Merge Requests, you only need to download the .flatpak and install locally, no need to build at all.

Then I started to fix the master branch. There was a project selector view and then you go to the translation view. There was only the .ui files and not working yet, so I reworked that a bit and convert the project selector to a file selector for now and then when you select a file, you've almost the same old interface, but without the menu, toolbar and statusbar.

I've been moving the toolbar icons to the headerbar so we can have the main functionality working. We've now a functional application.

Plugins

Gtranslator has a plugin system and has eight plugins, but that plugins was disabled some time ago. So I've started to recover that functionality but instead of as plugins, I've started to move the plugins to the core app code.

The only plugin ported currently is the translation memory. I'll take a look to the other plugins and view if it's desirable to reimplement that or remove that code.

What's next

All the statusbar functionality is missing right now, I need to find some place in the new UI to show that.

I wan't to start as soon as possible to work in a new widget to show messages, instead of the current table.

The project selection part is a bit hard because I need to think about what's a project in this context. Currently Gtranslator is a .po file editor, but if we change that to manage projects like a group of .po files, I need to think about how to store that information and the easier way for translator to use the interface.

So if you've time and want to contribute in this project, go ahead, you can ask in the #gtranslator IRC channel or talk to me directly. Currently there's no a lot of issues in the gitlab, but as soon as I get a functional version in gnome-nightly flatpak repo and some translators starts to use it, we'll have a lot of things to do.

Sidebars in yelp-xsl 3.30

It’s now easier to add sidebars to HTML output in yelp-xsl 3.30. When I finally landed the HTML modernization changes in 3.28, I added the ability to have sidebars without completely mucking up the layout. There’s a main element that’s a horizontal flexbox, and you could implement html.sidebar.custom to put stuff in main. But you’d have to do sizing and styling yourself, and you’d have to implement all the stuff you want in there.

I wanted to make sidebars easy in Pintail, so I started implementing some stock sidebars there. Then I realized I could move most of that work into yelp-xsl.

With 3.30, you can now set what you want to see in the sidebars using the html.sidebar.left and html.sidebar.right parameters. These are both space-separated lists of words, where each word is a sidebar component. In yelp-xsl, we have two components out of the box: contents to give you a table of contents for the whole document, and sections to give you a list of sections on the current page. You can also use the special blank token to force a sidebar to appear without actually adding anything to it.

More importantly, you can add your own components. So you can still make fully custom sidebar content while letting yelp-xsl do the rest of the work. For example, let’s say you wanted a left sidebar with a table of contents followed by a Google ad. You could set html.sidebar.left to "contents googlead". Then add a template to your extension stylesheet like this:

<xsl:template
     mode="html.sidebar.mode"
     match="token[. = 'googlead']">
  <!-- Put Google's stuff here -->
</xsl:template>

Pintail will add further sidebar components, such as one to switch between versions of a document, a language selector, and a search bar.

If you use yelp-build, you can pass the extension stylesheet with -x. The extension styleshet is also where you’d set the parameters when using yelp-build.

With Pintail, you’ll be able to set the parameters in your pintail.cfg file.

[pintail]
sidebar_left = contents
sidebar_right = search languages

What’s more, you’ll be able to set these on a per-directory basis. So, for example, on nightly builds you could add a warning message. In your extension stylesheet:

<xsl:template
     mode="html.sidebar.mode"
     match="token[. = 'nightly']">
  <p style="color: red">NIGHTLY BUILD!</p>
</xsl:template>

Then your pintail.cfg would look like this:

[pintail]
sidebar_left = contents
sidebar_right = search languages

[/path/to/nightly/]
sidebar_left = contents nightly

GStreamer Rust bindings 0.12 and GStreamer Plugin 0.3 release

After almost 6 months, a new release of the GStreamer Rust bindings and the GStreamer plugin writing infrastructure for Rust is out. As usual this was coinciding with the release of all the gtk-rs crates to make use of all the new features they contain.

Thanks to all the contributors of both gtk-rs and the GStreamer bindings for all the nice changes that happened over the last 6 months!

And as usual, if you find any bugs please report them and if you have any questions let me know.

GStreamer Bindings

For the full changelog check here.

Most changes this time were internally, especially because many user-facing changes (like Debug impls for various types) were already backported to the minor releases in the 0.11 release series.

WebRTC

The biggest change this time is probably the inclusion of bindings for the GStreamer WebRTC library.

This allows using building all kinds of WebRTC applications outside the browser (or providing a WebRTC implementation for a browser), and while not as full-featured as Google’s own implementation, this interoperates well with the various browsers and generally works much better on embedded devices.

A small example application in Rust is available here.

Serde

Optionally, serde trait implementations for the Serialize and Deserialize trait can be enabled for various fundamental GStreamer types, including caps, buffers, events, messages and tag lists. This allows serializing them into any format that can be handled by serde (which are many!), and deserializing them back to normal Rust structs.

Generic Tag API

Previously only a strongly-typed tag API was exposed that made it impossible to use the wrong data type for a specific tag, e.g. code that tries to store a string for the track number or an integer for the title would simply not compile:

let mut tags = gst::TagList::new();
{
    let tags = tags.get_mut().unwrap();
    tags.add::<Title>(&"some title", gst::TagMergeMode::Append);
    tags.add::<TrackNumber>(&12, gst::TagMergeMode::Append);
}

While this is convenient, it made it rather complicated to work with tag lists if you only wanted to handle them in a generic way. For example by iterating over the tag list and simply checking what kind of tags are available. To solve that, a new generic API was added in addition. This works on glib::Values, which can store any kind of type, and using the wrong type for a specific tag would simply cause an error at runtime instead of compile-time.

let mut tags = gst::TagList::new();
{
    let tags = tags.get_mut().unwrap();
    tags.add_generic(&gst::tags::TAG_TITLE, &"some title", gst::TagMergeMode::Append)
.expect("wrong type for title tag");
    tags.add_generic(&gst::tags::TAG_TRACK_NUMBER, &12, gst::TagMergeMode::Append)
.expect("wrong type for track number tag");
}

This also greatly simplified the serde serialization/deserialization for tag lists.

GStreamer Plugins

For the full changelog check here.

gobject-subclass

The main change this time is that all the generic GObject subclassing infrastructure was moved out of the gst-plugin crate and moved to its own gobject-subclass crate as part of the gtk-rs organization.

As part of this, some major refactoring has happened that allows subclassing more different types but also makes it simpler to add new types. There are also experimental crates for adding some subclassing support to gio and gtk, and a PR for autogenerating part of the code via the gir code generator.

More classes!

The other big addition this time is that it’s now possible to subclass GStreamer Pads and GhostPads, to implement the ChildProxy interface and to subclass the Aggregator and AggregatorPad class.

This now allows to write custom mixer/muxer-style elements (or generally elements that have multiple sink pads) in Rust via the Aggregator base class, and to have custom pad types for elements to allow for setting custom properties on the pads (e.g. to control the opacity of a single video mixer input).

There is currently no example for such an element, but I’ll add a very simple video mixer to the repository some time in the next weeks and will also write a blog post about it for explaining all the steps.

September 09, 2018

The compiler as a shared library

Since times immemorial, compilers have been run as standalone batch processes. If you have 50 files to compile, then you invoke the compiler 50 times, once on each file. Since each compilation is independent of all others, the work can be parallelised perfectly. This seems like a simple and optimal solution.

But, as is commonly the case, this is not the whole truth. When compiling code, there are many subtasks that are common to each individual compilation and this causes a lot of duplication of effort. Perhaps the best known case of this are C++ templates. They are parsed and codegenerated for each file that uses them yielding in the same code in dozens of files. Then the linker comes along and throws all but one of them away. There are a bunch of other issues which are discussed in this video from LLVM developer's conference:

A problem of state preservation

One of the best known solution to this problem are precompiled headers. They work roughly like this:
  1. Parse the contents of headers
  2. Dump compiler internal state to a file
  3. Load the file on each compiler invocation
The two main problems with this is that it requires someone to design and implement a full serialisation format for the compiler-internal data. That is a lot of tedious work that very few people will volunteer to do. The other downside is that the files need to be loaded explicitly from disk in every compilation process, which takes time, and that the build system needs to tell the compiler how to get this done. The granularity is also fairly coarse.

Ideally we would like to preserve as much data between two compiler invocations as possible without needing to serialise it to disk. As discussed in the above video, one solution is to have a "compiler plugin".

Almost every build system currently works roughly like this:
  1. Read build definition (such as a Ninja file)
  2. For each compilation, spawn a new compiler process and invoke the compiler executable
  3. Shutdown
The proposed new model would go like this (no build system currently supports this, but adding it to e.g. Ninja is not a massive undertaking):
  1. Read build definition
  2. dlopen the compiler shared library file
  3. For each compilation, create a new compiler object and invoke compilation using e.g. a thread pool
  4. Destroy compiler objects and dclose the file
  5. Shutdown
In this model all compilation jobs live in the same process, thus they can coordinate work behind the scenes however they wish. This requires some tricky code with thread safe caches and the like but it all internal to the compiler and never exposed. Even without caching this makes a difference on platforms such as Windows where process spawning is slow.

The big question remaining here is the API to use. It should have the following requirements:
  1. Must be ABI stable in the C sense
  2. Must be supportable on all compilers for all languages
  3. Must expose the full functionality of the compiler
  4. Must support an arbitrary number of compiler tasks within a single process

An API proposal for compiler invocation

On the face of it this seems like an impossible task. The API surface of a compiler is enormous and differs from compiler to compiler. However all of them already expose a stable ABI: the command line argument arrays. Exploiting this allows us to create an API supporting all of the requirements above with only six functions.

First we initialise the library:

CompilerService* compiler_init_service();

Here CompilerService is an opaque struct to a state object. There is one of these per process and it holds (internally) all the cached state and related things. Then we create a compiler object, one per compilation task:

Compiler* compiler_create_compiler(CompilerService *service);

Now we can invoke the compilation:

CompilationResult* compiler_compile(Compiler *c, int argc, const char **argv);

This invocation matches the signature of the main function. Since we are not going through the shell/kernel we can pass an arbitrary number of arguments without needing to use response files, quote shell characters or any other nastiness. The return value contains the return code and the strings for stdout and stderr. The standalone compiler executable such as cl.exe could (in theory ;-) be implemented by just calling these functions and returning the results to the calling process.

The last thing we need are the deallocation functions:

void compiler_free_compilation_result(CompilationResult *r);
void compiler_free_compiler(Compiler *c);
void compiler_free_service(CompilerService *s);

When will this be available in <my favorite compiler>?

Probably not soon, this is all slideware. There is no actual code to implement this (that I know of at least). The big problem here is that most compilers have not been written with this sort of usage in mind. The have global variables and other things hostile to usage as a shared library. Fixing all that to be thread safe and isolated is a lot of work. LLVM is probably the compiler that could most easily get this done since it has been designed to be used as a library from the beginning.

September 07, 2018

On Flatpak dependencies

Package managers have to deal with dependencies – too many of them. Over time things have gotten complicated: there are now soft dependencies, reverse dependencies and boolean conditions. So complicated that you can probably do general computation in the dependency solver now.

Thankfully flatpak is a lot simpler: there’s apps, and there’s runtimes, and every app depends on exactly one runtime. Simple and beautiful.

Of course, thats not the whole story.  Lets take a look.

Dependencies

The only hard dependencies in Flatpak are between an app and the runtime that it uses.

Every app uses exactly one runtime. When installing the app, Flatpak will automatically install the runtime it needs, and it will refuse to uninstall the runtime as long as there are apps using it. You can override this using the –force-remove option.

Related refs

The common term Flatpak uses for apps and runtimes is refs. Apart from dependencies, Flatpak has a notion of related refs. These are what other packaging systems might classify as soft dependencies, and they come in various forms and shapes.

The first group are standard pieces that get split off at build time, like .Locale and .Debug refs. The first contain translations and are similar to what other packaging systems call langpacks. The second contain debug information for binaries, and are the equivalent of debuginfo packages in other systems.

The next group are extensions. Both applications and runtimes can declare extension points,  and flatpak will look for refs that match those when setting up a sandbox, and mounts the ones it finds inside the sandbox.

Some extensions are a bit more special, in that they are conditional on some condition of the system. For example, they might only be relevant if the system has an Nvidia GPU, or apply for a certain GTK+ theme.

Another kind of relationship exists between a runtime and its matching sdk.

Whats used ?

The Flatpak uninstall command has a –unused option that makes an educated guess about what refs are no longer needed on your system. For each ref, it looks if it is an application, or the runtime of an installed application, or related to one of these. In each of these cases, it considers the ref used and skips it. Whatever is left afterwards get removed.

There are some heuristics involved when looking at related refs, and we are still fine-tuning these. One change that we’re recently made is to consider sdks used, to make –unused usable for developers.

There is still more fine-tuning to be done. For example, Flatpak will happily use a runtime from the system installation to run an application from the user installation. But the uninstall command works only on a single installation, so it does not see these dependencies, and might remove the runtime. Thankfully, it is easy to recover, should this happen to you: just install the runtime again.

Offline software updates in Endless OS

Motivation

A lot of the software most of us use day-to-day is built with the assumption that Internet access is fast, cheap, unlimited*, and ubiquitous.

For a lot of Endless’ current and target users, most of those assumptions are not true. Internet connectivity is often capped to low throughput and monthly quotas and may be relatively expensive or unreliable. Or it may be inaccessible entirely due to cost or lack of infrastructure.

When these assumptions fail, a lot of modern software fails.

This includes software updates.

A solution

Endless OS is built on top of OSTree for OS management and Flatpak for app management. OS updates and apps are presented to the user in Gnome Software.

Offline app and OS updates have been some of our longest-running development projects and it’s been quite a challenge. There are a handful of complex components involved and there’s a lot of plumbing and API work that needed to be designed, discussed, and tested in real-world scenarios. And because there was some but not total overlap between USB app updates, USB OS updates, LAN app updates, and LAN OS updates (see below), we had to do a lot of work to not paint ourselves into a corner.

If we first focused narrowly on LAN app updates, we might have deemed limited Internet connectivity a fair assumption. But this can’t be assumed for USB updates where a computer may literally never reach the Internet in its lifetime. And, in fact, we had a bug in development where an app install from a USB disk would fail if the computer were offline. Flatpak assumed it required Internet access to download a very small amount of app data. (We’ve fixed this).

So, we’ve worked in multiple directions at once and now the first of those is complete: USB app installs.

USB app installs

We’ve worked closely with upstream to solve the various problems that crop up when you break assumptions of “always-on” Internet. We sign app updates so you can get a USB disk full of apps and know they came from us. Or Flathub. Or an independent app vendor (which you may, for good reason, not automatically trust).

The majority of our work will be invisible to users but we also implemented the UI to copy apps to an inserted USB disk and to install from a USB disk. Some of that, as Matthias described, includes command line tools. But we also implemented UI in Gnome Software to be accessible to everyone. We also automatically open Gnome Software to a new “USB” category when you insert the disk so it’s clear they can be installed right now. No Internet required.

Gnome Software showing the new “Copy to USB” button which appears when a USB disk is inserted.
When an app can be installed locally, Gnome Software’s “Download” button instead says “Install”.

Caveats

When we copy apps to a USB disk, we also copy all their dependencies. If you copied an app to a friend’s computer, it wouldn’t be very useful if it were missing its content or translations they required. And if they didn’t have the app’s runtime, it wouldn’t run at all.

Say computer A installs app Z, copies it to USB, and goes offline. New versions of the OS get released which include a newer version of app Z’s runtime. Computer B gets this newer version of the OS clean-installed and is offline. If you plug the USB disk into computer B, it will fail to install app Z because it thinks it’s already got app Z’s runtime installed but app Z actually requires the older version of the runtime. The one that’s conveniently sharing the USB disk with it. We’ll be fixing this soon.

If that seems complicated, realize that it’s just one of the many (and not the hardest) corner cases my colleagues and upstream maintainers have solved in developing this feature.

Credits

I’m really proud of all the work that’s been done by my colleagues at Endless and upstream in Gnome to make these features a reality. I want to make it clear that I’ve only done a small part of the work and want to make sure I give credit where it’s due.

Matthew Leeds and Philip Withnall did most of the hard work in Flatpak and OSTree to figure out appropriate plumbing and API. Joaquim Rocha implemented Gnome Software changes to detect and display apps on USB disks. Alex Larsson, Matthias Clasen, and Colin Walters did a lot of work helping us integrate our OSTree and Flatpak changes upstream. Richard Hughes helped with questions about Gnome Software. I created the UI in Gnome Software to copy apps to USB disks.

What’s next

Most of our changes to OSTree and Flatpak are already upstream. Most of our changes to Gnome Software are downstream-only but we definitely want to upstream as much of the UI and Flatpak-related changes as possible. I’ll be analyzing the work required there and working with the maintainers and Gnome UX designers to make sure we’ve got designs suitable for everyone.

Apps can be copied to and from USB disks but we still need to finalize the work for OS updates as well. We’re fairly close but it needs UI in Gnome Software. (And I’m sure we’ll find some exciting bugs with wider testing).

LAN updates

USB updates are essential for users without Internet connections but they’re also useful for users with “partial” Internet connections which are limited by quotas/throughput/reliability. Another update solution for people with “partial” Internet access is what we’re calling “LAN updates” and which we usually describe in terms of a classroom. In this scenario, a “teacher” computer has Internet access to receive OS and app updates normally. But it also serves those updates to other “student” computers on its LAN. These “student” computers are intentionally blocked from the Internet to avoid them contributing toward a monthly data cap, using some of the limited bandwidth, etc.

Imagine a 200-MiB app contains a number of books (as PDFs). A newer version of the app adds a book that adds 1 MiB to the total size. For a classroom of 30 computers and a naive app distribution system that does a full download per update, 201 MiB/computer * 30 computers = 5.9 GiB. This single update would blow past the monthly data caps for some of our customers (which can go as low as 350 MiB).

Flatpak is really fine-grained with its updates so this would drop down to a 30 MiB. That’s far more efficient but it’s already a non-trivial portion of a 350 MiB cap and it’s just for a single app update. All these numbers need to be scaled up when you factor in many app updates and an OS update or two.

With LAN updates, this app update would drop down to 1 MiB for the entire classroom. And Endless OS already detects data-metered Internet connections and turns off automatic updates to further reduce data usage in extra-constrained environments. (This setting can be overridden in the system settings at any time).

LAN updates are also nearly ready for deployment in Endless OS.

3 Million Firmware Files and Counting…

In the last two years the LVFS has supplied over 3 million firmware files to end users. We now have about a two dozen companies uploading firmware, of which 9 are multi-billion dollar companies.

Every month about 200,000 more devices get upgraded and from the reports so far the number of failed updates is less than 0.01% averaged over all firmware types. The number of downloads is going up month-on-month, although we’re no longer growing exponentially, thank goodness. The server load average is 0.18, and we’ve made two changes recently to scale even more for less money: signing files in a 30 minute cron job rather than immediately, and switching from Amazon to BunnyCDN.

The LVFS is mainly run by just one person (me!) and my time is sponsored by the ever-awesome Red Hat. The hardware costs, which recently included random development tools for testing the dfu and nvme plugins, and the server and bandwidth costs are being paid from charitable donations from the community. We’re even cost positive now, so I’m building up a little pot for the next server or CDN upgrade. By pretty much any metric, the LVFS is a huge success, and I’m super grateful to all the people that helped the project grow.

The LVFS does have one weakness, that it has a bus factor of one. In other words, if I got splattered by a bus, the LVFS would probably cease to exist in the current form. To further grow the project, and to reduce the dependence on me, we’re going to be moving various parts of the LVFS to the Linux Foundation. This means that they’ll be sysadmins who don’t have to google basic server things, a proper community charter, and access to an actual legal team. From a OEM point of view, nothing much should change, including the most important thing that it’ll continue to be free to use for everyone. The existing server and all the content will be migrated to the LVFS infrastructure. From a users point of view, new metadata and firmware will be signed by the Linux Foundation key, rather than my key, although we haven’t decided on a date for the switch-over yet. The LF key has been trusted by fwupd for firmware since 1.0.8 and it’s trivial to backport to older branches if required.

Before anyone gets too excited and starts pointing me at all my existing bugs for my other stuff: I’ll probably still be the person “onboarding” vendors onto the LVFS, and I’m fully expecting to remain the maintainer and core contributor to the lvfs-website code itself — but I certainly should have a bit more time for GNOME Software and color stuff.

In related news, even more vendors are jumping on the LVFS. No more public announcements yet, but hopefully soon. For a lot of hardware companies they’ll be comfortable “going public” when the new hardware currently in development is on shelves in stores. So, please raise a glass to the next 3 million downloads!

September 06, 2018

My gdk-pixbuf braindump

I want to write a braindump on the stuff that I remember from gdk-pixbuf's history. There is some talk about replacing it with something newer; hopefully this history will show some things that worked, some that didn't, and why.

The beginnings

Gdk-pixbuf started as a replacement for Imlib, the image loading and rendering library that GNOME used in its earliest versions. Imlib came from the Enlightenment project; it provided an easy API around the idiosyncratic libungif, libjpeg, libpng, etc., and it maintained decoded images in memory with a uniform representation. Imlib also worked as an image cache for the Enlightenment window manager, which made memory management very inconvenient for GNOME.

Imlib worked well as a "just load me an image" library. It showed that a small, uniform API to load various image formats into a common representation was desirable. And in those days, hiding all the complexities of displaying images in X was very important indeed.

The initial API

Gdk-pixbuf replaced Imlib, and added two important features: reference counting for image data, and support for an alpha channel.

Gdk-pixbuf appeared with support for RGB(A) images. And although in theory it was possible to grow the API to support other representations, GdkColorspace never acquired anything other than GDK_COLORSPACE_RGB, and the bits_per_sample argument to some functions only ever supported being 8. The presence or absence of an alpha channel was done with a gboolean argument in conjunction with that single GDK_COLORSPACE_RGB value; we didn't have something like cairo_format_t which actually specifies the pixel format in single enum values.

While all the code in gdk-pixbuf carefully checks that those conditions are met — RGBA at 8 bits per channel —, some applications inadvertently assume that that is the only possible case, and would get into trouble really fast if gdk-pixbuf ever started returning pixbufs with different color spaces or depths.

One can still see the battle between bilevel-alpha vs. continuous-alpha in this enum:

typedef enum
{
        GDK_PIXBUF_ALPHA_BILEVEL,
        GDK_PIXBUF_ALPHA_FULL
} GdkPixbufAlphaMode;

Fortunately, only the "render this pixbuf with alpha to an Xlib drawable" functions take values of this type: before the Xrender days, it was a Big Deal to draw an image with alpha to an X window, and applications often opted to use a bitmask instead, even if they had jagged edges as a result.

Pixel formats

The only pixel format that ever got implemented was unpremultiplied RGBA on all platforms. Back then I didn't understand premultiplied alpha! Also, the GIMP followed that scheme, and copying it seemed like the easiest thing.

After gdk-pixbuf, libart also copied that pixel format, I think.

But later we got Cairo, Pixman, and all the Xrender stack. These prefer premultiplied ARGB. Moreover, Cairo prefers it if each pixel is actually a 32-bit value, with the ARGB values inside it in platform-endian order. So if you look at a memory dump, a Cairo pixel looks like BGRA on a little-endian box, while it looks like ARGB on a big-endian box.

Every time we paint a GdkPixbuf to a cairo_t, there is a conversion from unpremultiplied RGBA to premultiplied, platform-endian ARGB. I talked a bit about this in Reducing the number of image copies in GNOME.

The loading API

The public loading API in gdk-pixbuf, and its relationship to loader plug-ins, evolved in interesting ways.

At first the public API and loaders only implemented load_from_file: you gave the library a FILE * and it gave you back a GdkPixbuf. Back then we didn't have a robust MIME sniffing framework in the form of a library, so gdk-pixbuf got its own. This lives in the mostly-obsolete GdkPixbufFormat machinery; it even has its own little language for sniffing file headers! Nowadays we do most MIME sniffing with GIO.

After the intial load_from_file API... I think we got progressive loading first, and animation support aftewards.

Progressive loading

This where the calling program feeds chunks of bytes to the library, and at the end a fully-formed GdkPixbuf comes out, instead of having a single "read a whole file" operation.

We conflated this with a way to get updates on how the image area gets modified as the data gets parsed. I think we wanted to support the case of a web browser, which downloads images slowly over the network, and gradually displays them as they are downloaded. In 1998, images downloading slowly over the network was a real concern!

It took a lot of very careful work to convert the image loaders, which parsed a whole file at a time, into loaders that could maintain some state between each time that they got handed an extra bit of buffer.

It also sounded easy to implement the progressive updating API by simply emitting a signal that said, "this rectangular area got updated from the last read". It could handle the case of reading whole scanlines, or a few pixels, or even area-based updates for progressive JPEGs and PNGs.

The internal API for the image format loaders still keeps a distinction between the "load a whole file" API and the "load an image in chunks". Not all loaders got redone to simply just use the second one: io-jpeg.c still implements loading whole files by calling the corresponding libjpeg functions. I think it could remove that code and use the progressive loading functions instead.

Animations

Animations: we followed the GIF model for animations, in which each frame overlays the previous one, and there's a delay set between each frame. This is not a video file; it's a hacky flipbook.

However, animations presented the problem that the whole gdk-pixbuf API was meant for static images, and now we needed to support multi-frame images as well.

We defined the "correct" way to use the gdk-pixbuf library as to actually try to load an animation, and then see if it is a single-frame image, in which case you can just get a GdkPixbuf for the only frame and use it.

Or, if you got an animation, that would be a GdkPixbufAnimation object, from which you could ask for an iterator to get each frame as a separate GdkPixbuf.

However, the progressive updating API never got extended to really support animations. So, we have awkward functions like gdk_pixbuf_animation_iter_on_currently_loading_frame() instead.

Necessary accretion

Gdk-pixbuf got support for saving just a few formats: JPEG, PNG, TIFF, ICO, and some of the formats that are implemented with the Windows-native loaders.

Over time gdk-pixbuf got support for preserving some metadata-ish chunks from formats that provide it: DPI, color profiles, image comments, hotspots for cursors/icons...

While an image is being loaded with the progressive loaders, there is a clunky way to specify that one doesn't want the actual size of the image, but another size instead. The loader can handle that situation itself, hopefully if an image format actually embeds different sizes in it. Or if not, the main loading code will rescale the full loaded image into the size specified by the application.

Historical cruft

GdkPixdata - a way to embed binary image data in executables, with a funky encoding. Nowadays it's just easier to directly store a PNG or JPEG or whatever in a GResource.

contrib/gdk-pixbuf-xlib - to deal with old-style X drawables. Hopefully mostly unused now, but there's a good number of mostly old, third-party software that still uses gdk-pixbuf as an image loader and renderer to X drawables.

gdk-pixbuf-transform.h - Gdk-pixbuf had some very high-quality scaling functions, which the original versions of EOG used for the core of the image viewer. Nowadays Cairo is the preferred way of doing this, since it not only does scaling, but general affine transformations as well. Did you know that gdk_pixbuf_composite_color takes 17 arguments, and it can composite an image with alpha on top of a checkerboard? Yes, that used to be the core of EOG.

Debatable historical cruft

gdk_pixbuf_get_pixels(). This lets the program look into the actual pixels of a loaded pixbuf, and modify them. Gdk-pixbuf just did not have a concept of immutability.

Back in GNOME 1.x / 2.x, when it was fashionable to put icons beside menu items, or in toolbar buttons, applications would load their icon images, and modify them in various ways before setting them onto the corresponding widgets. Some things they did: load a colorful icon, desaturate it for "insensitive" command buttons or menu items, or simulate desaturation by compositing a 1x1-pixel checkerboard on the icon image. Or lighten the icon and set it as the "prelight" one onto widgets.

The concept of "decode an image and just give me the pixels" is of course useful. Image viewers, image processing programs, and all those, of course need this functionality.

However, these days GTK would prefer to have a way to decode an image, and ship it as fast as possible ot the GPU, without intermediaries. There is all sorts of awkward machinery in the GTK widgets that can consume either an icon from an icon theme, or a user-supplied image, or one of the various schemes for providing icons that GTK has acquired over the years.

It is interesting to note that gdk_pixbuf_get_pixels() was available pretty much since the beginning, but it was only until much later that we got gdk_pixbuf_get_pixels_with_length(), the "give me the guchar * buffer and also its length" function, so that calling code has a chance of actually checking for buffer overruns. (... and it is one of the broken "give me a length" functions that returns a guint rather than a gsize. There is a better gdk_pixbuf_get_byte_length() which actually returns a gsize, though.)

Problems with mutable pixbufs

The main problem is that as things are right now, we have no flexibility in changing the internal representation of image data to make it better for current idioms: GPU-specific pixel formats may not be unpremultiplied RGBA data.

We have no API to say, "this pixbuf has been modified", akin to cairo_surface_mark_dirty(): once an application calls gdk_pixbuf_get_pixels(), gdk-pixbuf or GTK have to assume that the data will be changed and they have to re-run the pipeline to send the image to the GPU (format conversions? caching? creating a texture?).

Also, ever since the beginnings of the gdk-pixbuf API, we had a way to create pixbufs from arbitrary user-supplied RGBA buffers: the gdk_pixbuf_new_from_data functions. One problem with this scheme is that memory management of the buffer is up to the calling application, so the resulting pixbuf isn't free to handle those resources as it pleases.

A relatively recent addition is gdk_pixbuf_new_from_bytes(), which takes a GBytes buffer instead of a random guchar *. When a pixbuf is created that way, it is assumed to be immutable, since a GBytes is basically a shared reference into a byte buffer, and it's just easier to think of it as immutable. (Nothing in C actually enforces immutability, but the API indicates that convention.)

Internally, GdkPixbuf actually prefers to be created from a GBytes. It will downgrade itself to a guchar * buffer if something calls the old gdk_pixbuf_get_pixels(); in the best case, that will just take ownership of the internal buffer from the GBytes (if the GBytes has a single reference count); in the worst case, it will copy the buffer from the GBytes and retain ownership of that copy. In either case, when the pixbuf downgrades itself to pixels, it is assumed that the calling application will modify the pixel data.

What would immutable pixbufs look like?

I mentioned this a bit in "Reducing Copies". The loaders in gdk-pixbuf would create immutable pixbufs, with an internal representation that is friendly to GPUs. In the proposed scheme, that internal representation would be a Cairo image surface; it can be something else if GTK/GDK eventually prefer a different way of shipping image data into the toolkit.

Those pixbufs would be immutable. In true C fashion we can call it undefined behavior to change the pixel data (say, an app could request gimme_the_cairo_surface and tweak it, but that would not be supported).

I think we could also have a "just give me the pixels" API, and a "create a pixbuf from these pixels" one, but those would be one-time conversions at the edge of the API. Internally, the pixel data that actually lives inside a GdkPixbuf would remain immutable, in some preferred representation, which is not necessarily what the application sees.

What worked well

A small API to load multiple image formats, and paint the images easily to the screen, while handling most of the X awkwardness semi-automatically, was very useful!

A way to get and modify pixel data: applications clearly like doing this. We can formalize it as an application-side thing only, and keep the internal representation immutable and in a format that can evolve according to the needs of the internal API.

Pluggable loaders, up to a point. Gdk-pixbuf doesn't support all the image formats in the world out of the box, but it is relatively easy for third-parties to provide loaders that, once installed, are automatically usable for all applications.

What didn't work well

Having effectively two pixel formats supported, and nothing else: gdk-pixbuf does packed RGB and unpremultiplied RGBA, and that's it. This isn't completely terrible: applications which really want to know about indexed or grayscale images, or high bit-depth ones, are probably specialized enough that they can afford to have their own custom loaders with all the functionality they need.

Pluggable loaders, up to a point. While it is relatively easy to create third-party loaders, installation is awkward from a system's perspective: one has to run the script to regenerate the loader cache, there are more shared libraries running around, and the loaders are not sandboxed by default.

I'm not sure if it's worthwhile to let any application read "any" image format if gdk-pixbuf supports it. If your word processor lets you paste an image into the document... do you want it to use gdk-pixbuf's limited view of things and include a high bit-depth image with its probably inadequate conversions? Or would you rather do some processing by hand to ensure that the image looks as good as it can, in the format that your word processor actually supports? I don't know.

The API for animations is very awkward. We don't even support APNG... but honestly I don't recall actually seeing one of those in the wild.

The progressive loading API is awkward. The "feed some bytes into the loader" part is mostly okay; the "notify me about changes to the pixel data" is questionable nowadays. Web browsers don't use it; they implement their own loaders. Even EOG doesn't use it.

I think most code that actually connects to GdkPixbufLoader's signals only uses the size-prepared signal — the one that gets emitted soon after reading the image headers, when the loader gets to know the dimensions of the image. Apps sometimes use this to say, "this image is W*H pixels in size", but don't actually decode the rest of the image.

The gdk-pixbuf model of static images, or GIF animations, doesn't work well for multi-page TIFFs. I'm not sure if this is actualy a problem. Again, applications with actual needs for multi-page TIFFs are probably specialized enough that they will want a full-featured TIFF loader of their own.

Awkward architectures

Thumbnailers

The thumbnailing system has slowly been moving towards a model where we actually have thumbnailers specific to each file format, instead of just assuming that we can dump any image into a gdk-pixbuf loader.

If we take this all the way, we would be able to remove some weird code in, for example, the JPEG pixbuf loader. Right now it supports loading images at a size that the calling code requests, not only at the "natural" size of the JPEG. The thumbnailer can say, "I want to load this JPEG at 128x128 pixels" or whatever, and in theory the JPEG loader will do the minimal amount of work required to do that. It's not 100% clear to me if this is actually working as intended, or if we downscale the whole image anyway.

We had a distinction between in-process and out-of-process thumbnailers, and it had to do with the way pixbuf loaders are used; I'm not sure if they are all out-of-process and sandboxed now.

Non-raster data

There is a gdk-pixbuf loader for SVG images which uses librsvg internally, but only in a very basic way: it simply loads the SVG at its preferred size. Librsvg jumps through some hoops to compute a "preferred size" for SVGs, as not all of them actually indicate one. The SVG model would rather have the renderer say that the SVG is to be inserted into a rectangle of certain width/height, and scaled/positioned inside the rectangle according to some other parameters (i.e. like one would put it inside an HTML document, with a preserveAspectRatio attribute and all that). GNOME applications historically operated with a different model, one of "load me an image, I'll scale it to whatever size, and paint it".

This gdk-pixbuf loader for SVG files gets used for the SVG thumbnailer, or more accurately, the "throw random images into a gdk-pixbuf loader" thumbnailer. It may be better/cleaner to have a specific thumbnailer for SVGs instead.

Even EOG, our by-default image viewer, doesn't use the gdk-pixbuf loader for SVGs: it actually special-cases them and uses librsvg directly, to be able to load an SVG once and re-render it at different sizes if one changes the zoom factor, for example.

GTK reads its SVG icons... without using librsvg... by assuming that librsvg installed its gdk-pixbuf loader, so it loads them as any normal raster image. This kind of dirty, but I can't quite pinpoint why. I'm sure it would be convenient for icon themes to ship a single SVG with tons of icons, and some metadata on their ids, so that GTK could pick them out of the SVG file with rsvg_render_cairo_sub() or something. Right now icon theme authors are responsible for splitting out those huge SVGs into many little ones, one for each icon, and I don't think that's their favorite thing in the world to do :)

Exotic raster data

High bit-depth images... would you expect EOG to be able to load them? Certainly; maybe not with all the fancy conversions from a real RAW photo editor. But maybe this can be done as EOG-specific plugins, rather than as low in the platform as the gdk-pixbuf loaders?

(Same thing for thumbnailing high bit-depth images: the loading code should just provide its own thumbnailer program for those.)

Non-image metadata

The gdk_pixbuf_set_option / gdk_pixbuf_get_option family of functions is so that pixbuf loaders can set key/value pairs of strings onto a pixbuf. Loaders use this for comment blocks, or ICC profiles for color calibration, or DPI information for images that have it, or EXIF data from photos. It is up to applications to actually use this information.

It's a bit uncomfortable that gdk-pixbuf makes no promises about the kind of raster data it gives to the caller: right now it is raw RGB(A) data that is not gamma-corrected nor in any particular color space. It is up to the caller to see if the pixbuf has an ICC profile attached to it as an option. Effectively, this means that applications don't know if they are getting SRGB, or linear RGB, or what... unless they specifically care to look.

The gdk-pixbuf API could probably make promises: if you call this function you will get SRGB data; if you call this other function, you'll get the raw RGBA data and we'll tell you its colorspace/gamma/etc.

The various set_option / get_option pairs are also usable by the gdk-pixbuf saving code (up to now we have just talked about loaders). I don't know enough about how applications use the saving code in gdk-pixbuf... the thumbnailers use it to save PNGs or JPEGs, but other apps? No idea.

What I would like to see

Immutable pixbufs in a useful format. I've started work on this in a merge request; the internal code is now ready to take in different internal representations of pixel data. My goal is to make Cairo image surfaces the preferred, immutable, internal representation. This would give us a gdk_pixbuf_get_cairo_surface(), which pretty much everything that needs one reimplements by hand.

Find places that assume mutable pixbufs. To gradually deprecate mutable pixbufs, I think we would need to audit applications and libraries to find places that cause GdkPixbuf structures to degrade into mutable ones: basically, find callers of gdk_pixbuf_get_pixels() and related functions, see what they do, and reimplement them differently. Maybe they don't need to tint icons by hand anymore? Maybe they don't need icons anymore, given our changing UI paradigms? Maybe they are using gdk-pixbuf as an image loader only?

Reconsider the loading-updates API. Do we need the GdkPixbufLoader::area-updated signal at all? Does anything break if we just... not emit it, or just emit it once at the end of the loading process? (Caveat: keeping it unchanged more or less means that "immutable pixbufs" as loaded by gdk-pixbuf actually mutate while being loaded, and this mutation is exposed to applications.)

Sandboxed loaders. While these days gdk-pixbuf loaders prefer the progressive feed-it-bytes API, sandboxed loaders would maybe prefer a read-a-whole-file approach. I don't know enough about memfd or how sandboxes pass data around to know how either would work.

Move loaders to Rust. Yes, really. Loaders are security-sensitive, and while we do need to sandbox them, it would certainly be better to do them in a memory-safe language. There are already pure Rust-based image loaders: JPEG, PNG, TIFF, GIF, ICO. I have no idea how featureful they are. We can certainly try them with gdk-pixbuf's own suite of test images. We can modify them to add hooks for things like a size-prepared notification, if they don't already have a way to read "just the image headers".

Rust makes it very easy to plug in micro-benchmarks, fuzz testing, and other modern amenities. These would be perfect for improving the loaders.

I started sketching a Rust backend for gdk-pixbuf loaders some months ago, but there's nothing useful yet. One mismatch between gdk-pixbuf's model for loaders, and the existing Rust codecs, is that Rust codecs generally take something that implements the Read trait: a blocking API to read bytes from abstract sources; it's a pull API. The gdk-pixbuf model is a push API: the calling code creates a loader object, and then pushes bytes into it. The gdk-pixbuf convenience functions that take a GInputStream basically do this:

loader = gdk_pixbuf_loader_new (...);

while (more_bytes) {
    n_read = g_input_stream_read (stream, buffer, ...);
    gdk_pixbuf_loader_write(loader, buffer, n_read, ...);
}

gdk_pixbuf_loader_close (loader);

However, this cannot be flipped around easily. We could probably use a second thread (easy, safe to do in Rust) to make the reader/decoder thread block while the main thread pushes bytes into it.

Also, I don't know how the Rust bindings for GIO present things like GInputStream and friends, with our nice async cancellables and all that.

Deprecate animations? Move that code to EOG, just so one can look at memes in it? Do any "real apps" actually use GIF animations for their UI?

Formalize promises around returned color profiles, gamma, etc. As mentioned above: have an "easy API" that returns SRGB, and a "raw API" that returns the ARGB data from the image, plus info on its ICC profile, gamma, or any other info needed to turn this into a "good enough to be universal" representation. (I think all the Apple APIs that pass colors around do so with an ICC profile attached, which seems... pretty much necessary for correctness.)

Remove the internal MIME-sniffing machinery. And just use GIO's.

Deprecate the crufty/old APIs in gdk-pixbuf. Scaling/transformation, compositing, GdkPixdata, gdk-pixbuf-csource, all those. Pixel crunching can be done by Cairo; the others are better done with GResource these days.

Figure out if we want blessed codecs; fix thumbnailers. Link those loaders statically, unconditionally. Exotic formats can go in their own custom thumbnailers. Figure out if we want sandboxed loaders for everything, or just for user-side images (not ones read from the trusted system installation).

Have GTK4 communicate clearly about its drawing model. I think we are having a disconnect between the GUI chrome, which is CSS/GPU friendly, and graphical content generated by applications, which by default right now is done via Cairo. And having Cairo as a to-screen and to-printer API is certainly very convenient! You Wouldn't Print a GUI, but certainly you would print a displayed document.

It would also be useful for GTK4 to actually define what its preferred image format is if it wants to ship it to the GPU with as little work as possible. Maybe it's a Cairo image surface? Maybe something else?

Conclusion

We seem to change imaging models every ten years or so. Xlib, then Xrender with Cairo, then GPUs and CSS-based drawing for widgets. We've gone from trusted data on your local machine, to potentially malicious data that rains from the Internet. Gdk-pixbuf has spanned all of these periods so far, and it is due for a big change.

September 04, 2018

Reference counting in modern GLib

Reference counting is a fairly common garbage collection technique used in many projects; the core GNOME platform uses pretty much all the time, from container data types, to GObject.

Implementing reference counting in C is typically fairly easy. Add a field to your data type:

typedef struct {
  int ref_count;

  char *name;
  char *address;
  char *city;

  int age;
} Person;

Then initialise it when creating a new instance:

Person *
person_new (void)
{
  Person *res = g_new0 (Person, 1);

  res->ref_count = 1;

  return res;
}

Instead of a person_copy() and a person_free() pair of functions, you need to write a person_ref() and a person_unref() pair:

Person *
person_ref (Person *p)
{
  // Acquire a reference
  p->ref_count++;

  return p;
}

static void
person_free (Person *p)
{
  // Free the data
  g_free (p->name);
  g_free (p->address);
  g_free (p->city);

  // Free the instance
  g_free (p);
}

void
person_unref (Person *p)
{
  // Release a reference
  p->ref_count--;

  // If this was the last reference, free the data
  // associated to the instance and then the instance
  // itself
  if (p->ref_count == 0)
    {
      person_free (p);
    }
}

Of course, trivial doesn’t mean correct. For instance, the code above assumes that all reference acquisition and release operations will happen from the same thread; if that’s not the case, you’ll have to use atomic integer operations to increase and decrease the reference count. Additionally, the code above does not check for overflows in the reference counting, which means that the value could saturate and lead to leaks.

Reimplementing all the checks and behaviours is not only boring, but it inevitably leads to mistakes along the way. For this reason, GLib 2.58 introduced two new types:

  • grefcount, to implement simple reference counting
  • gatomicrefcount, to implement atomic reference counting

Both come with their own API, which leads to this code:

typedef struct {
  grefcount ref_count;
  // or: gatomicrefcount ref_count;

  // same as above
} Person;

Person *
person_new (void)
{
  Person *res = g_new0 (Person, 1);

  g_ref_count_init (&res->ref_count);
  // or: g_atomic_ref_count_init (&res->ref_count);

  return res;
}

Person *
person_ref (Person *p)
{
  g_ref_count_inc (&p->ref_count);
  // or: g_atomic_ref_count_inc (&p->ref_count);

  return p;
}

void
person_unref (Person *p)
{
  if (g_ref_count_dec (&p->ref_count))
  // or: if (g_atomic_ref_count_dec (&p->ref_count))
    {
      person_free (p);
    }
}

The grefcount and gatomicrefcount make it immediately obvious that the field is used to implement reference counting semantics; the API checks for saturation of the counters, and will emit a warning; the atomic operations are verified. Additionally, the API checks if you’re trying to pass an atomic reference count to the grefcount API, and vice versa, so you have a layer of protection there during eventual refactoring, even if the types are both integer aliases.

We did not stop there, though.

Adding a reference count field to a structure is not always possible; for instance, if you have ABI compatibility restrictions, or if the type defition is public, adding a field may just not be something you can do within the same API version. By way of an example: you may have a type meant to be typically placed on the stack, so it needs a public, complete declaration, in order to have a well-defined size at compile time. You may also want to pass around an instance of the type as the argument for a GObject signal, or as a property — but it may be expensive to copy the data around, so you really want to have an optional reference counting mechanism that is invisible to the vast majority of the use cases.

This is why we also added an allocator function that adds reference counting semantics to the memory areas it returns, called GRcBox:

typedef struct {
  char *name;
  char *address;
  char *city;

  int age;
} Person;

Person *
person_new (void)
{
  return g_rc_box_new0 (Person);
}

Person *
person_ref (Person *p)
{
  return g_rc_box_acquire (p);
}

void
person_unref (Person *p)
{
  // person_free() is copied from the code above; we use
  // g_rc_box_release_full() because we have data to free
  // as well, but there's a variant for structures that
  // do not have internal allocations
  g_rc_box_release_full (p, person_free);
}

As you can see, this cuts down the boilerplate and repetition considerably.

The data returned to you by g_rc_box_new() and friends is exactly the same as you’d get from g_new(), so you can pass it around to your API exactly the same — but it is transparently augmented with reference counting semantics. You acquire and release references without having an explicit reference counter. The only restriction is that you cannot reallocate the data, so calling realloc() is not allowed; and, of course, you cannot free the memory directly with free() — you need to release the last reference.

Similar to GRcBox, there’s a GAtomicRcBox, which provides atomic reference counting semantics, with a similar API.

Both GRcBox and GAtomicRcBox are Valgrind-safe, so you won’t get false positives or unreachable memory if run your code under Valgrind.

You don’t even need to have a structure to allocate: you can use GRcBox to allocate any memory area with a non-zero size. Incidentally, this is how we implemented a oft-requested feature: reference counted strings, which are also available in GLib 2.58.

Reference counted strings work exactly like any other C string, but instead of copying them and freeing them, you acquire and release references to them:

char *s = g_ref_string_new ("hello, world!");

// "s" is just like any other C string
g_print ("s['%s'] length = %d\n", s, strlen (s));

g_ref_string_release (s);

Reference counted strings can also be interned, which means that calling g_ref_string_new_intern() with the same string twice will give you a new reference the second time around, instead of allocating a new string; once the last reference to the interned string is dropped, the string is un-interned, and the resource allocated for it are freed.

Since you may be wary of passing around char * for reference counted strings, there’s a handy GRefString C type you can use to improve readability, like we have GStrv for char **:

GRefString *s = g_ref_string_new ("hello");

// ...

g_ref_string_release (s);

GRefString also has an autocleanup function, so you can do:

{
  g_autoptr(GRefString) s = g_ref_string_new ("hello!");

  // ...
}

and the string will automatically be released once it goes out of scope.

GUADEC 2018 Videos: All Done

All the editing & uploading for the GUADEC videos is now finished. The videos were all uploaded to YouTube some time ago, and they are all now available on http://videos.guadec.org/2018 as well.

Thanks to everyone who helped with the editing: Alexis Diavatis, Bin Li, Garrett LeSage, Alexandre Franke (who also did a lot of the work of uploading to YouTube), and Hubert Figuiere (who managed to edit so many that I’m suspicious he might be some kind of robot in disguise).

edit: If you are hungry for more videos to edit, some footage from GUADEC 2002 has been unearthed. It’d be great to have some of this history from fifteen years ago up on YouTube! If you’re interested, reply to the mail or speak up in #guadec on GIMPnet and we can coordinate efforts.

What's new in libinput 1.12

libinput 1.12 was a massive development effort (over 300 patchsets) with a bunch of new features being merged. It'll be released next week or so, so it's worth taking a step back and looking at what actually changed.

The device quirks files replace the previously used hwdb-based udev properties. I've written about this in more detail here but the gist is: we have our own .ini style file format that can match on devices and apply the various quirks devices need. This simplifies debugging a lot, we can now reliably tell users why a quirks file applies or doesn't apply, historically a problem with the hwdb.

The sphinx-based documentation was merged, fixed and added to. We switched to sphinx for the docs and the result is much more user-friendly. Which was the point, it was a switch from a developer-oriented documentation to a user-oriented one. Not that documentation is ever finished.

The usual set of touchpad improvements went in, e.g. the slight motion on finger up is now ignored. We have size-based thumb detection now (useful for Apple touchpads!). And of course various quirks for better pressure ranges, etc. Tripletap on some synaptics touchpads had a tendency to cause multiple taps because of some weird event sequence. Movement in the software button now generates events, the buttons are not just a dead zone anymore. Pointer jump detection is more adaptive now and catches and discards smaller jumps that previously slipped through the cracks. A particularly quirky behaviour was seen on Dell XPS i2c touchpads that exhibit a huge pointer jump, courtesy of the trackpoint controller going to sleep and taking its time to wake up. The delay is still there but the pointer at least lands in the correct location.

We now have improved direction-locking for two-finger scrolling on touchpads. Scrolling up/down should not generate horizontal scroll events anymore as long as the movement is close enough to vertical. This feature is transparent, a diagonal or horizontal movement will immediately disable the direction lock and produce horizontal scroll events as expected.

The trackpoint acceleration has been re-done, see this post for more details and links to the background articles. I've only received one bug report for the new acceleration so it seems to work quite well now. Trackpoints that send events in bursts (e.g. bluetooth ones) are smoothened now to avoid jerky movement.

Velocity averaging was dropped to increase pointer accuracy. Previously we averaged the velocity across multiple events which makes the motion smoother on jittery devices but less accurate on good devices.

We build on FreeBSD now. Presumably this also means it works on FreeBSD :)

libinput now supports palm detection on touchscreens, at least where the ABS_MT_TOOL_TYPE evdev bit is provided.

I think that's about it. Busy days...

September 03, 2018

Natural Language Processing

This month I have been thinking about good English sentence and paragraph structure. Non-native English speakers who are learning write in English will often think of what they want to say in their first language and then translate it. This generally results in a mess. The precise structure of the mess will depend on the rules of the student’s first language. The important thing is to teach the conventions of good English writing; but how?

Visualizing a problem helps to solve it. However there doesn’t seem to be a tool available today that can clearly visualize the various concerns writers have to deal with. A paragraph might contain 100 words, each of which relate to each other in some way. How do you visualize that clearly… not like this, anyway.

I did find some useful resources though. I discovered the Paramedic Method, through this blog post from helpscout.net. The Paramedic Method was devised by Richard Lanham and consists of these 6 steps:

  1. Highlight the prepositions.
  2. Highlight the “is” verb forms.
  3. Find the action. (Who is kicking whom?)
  4. Change the action into a simple active verb.
  5. Start fast—no slow windups.
  6. Read the passage out loud with emphasis and feeling.

This is good advice for anyone writing English. It’ll be particularly helpful in my classes in Spain where we need to clean up long strings of relative clauses. (For example, a sentence such as “On the way I met one of the workers from the company where I was going to do the interview that my friend got for me”. I would rewrite this as: “On the way I met a person from Company X, where my friend had recently got me an interview.”

I found a tool called Write Music which I like a lot. The idea is simple: to illustrate and visualize the rule that varying sentence length is important when writing. The creator of the tool, Titus Wormer, seems to be doing some interesting and well documented research.

I looked at a variety of open source tools for natural language processing. These provide good ways to tokenize a text and to identify the “part of speech” (noun, verb, adjective, adverb, etc.) but I didn’t yet find one that could analyze the types of clauses that are used. Which is a shame. My understanding of this is an area of English grammar is still quite weak and I was hoping my laptop might be able teach me by example but it seems not.

I found some surprisingly polished libraries that I’m keen to use for … something. One day I’ll know what. The compromise library for JavaScript can do all kinds of parsing and wordplay and is refreshingly honest about its limitations, and spaCy for Python also looks exciting. People like to interact with a computer through text. We hide the UNIX commandline. But one of the most popular user interfaces in the world is the Google search engine, which is a text box that accepts any kind of natural language and gives the impression of understanding it. In many cases this works brilliantly — I check spellings and convert measurements all the time using this “search engine” interface. Did you realize GNOME Shell can also do unit conversions? Try typing “50lb in kg” into the GNOME Shell search box and look at the result. Very useful! More apps should do helpful things like this.

I found some other amazing natural language technologies too. Inform 7 continues to blow my mind whenever I look at it. Commercial services like IBM Watson can promise incredible things like analysing the sentiments and emotions expressed in a text, and even the relationships expressed between the subjects and objects. It’s been an interesting day of research!

September 02, 2018

GNOME Tweaks 3.30

GNOME 3.30 will be released within a few days. That makes this a good time to showcase the improvements in GNOME Tweaks 3.30.

One problem with moving power settings from Tweaks into Settings a year ago was that the Power panel only had one setting. GNOME Designer Allan Day suggested we use a new General panel to include the remaining power switch, the sound Over-Amplification switch, and the Animations switch.

(Note that if you’re using Ubuntu’s default session, the Over-Amplification switch is in the Settings app, not in Tweaks.)

Also, the volume indicators in GNOME 3.30 now indicate when over-amplification is in use:

There is only one new setting here: a sound theme selector. While sound themes aren’t very popular yet, maybe this will help them become a bit more noticeable. Ubuntu 18.10’s new Yaru theme (not pictured here) includes a sound theme named Yaru.

The Top Bar page now has a toggle for the weekday in the top bar clock. (The Activities Overview Hot Corner switch will only show if your distro includes the gnome-shell patch for that feature.)

Since we now have so many windows settings available, I split Window Titlebars into its own page. The Edge Tiling and Center New Windows switches are new in 3.30.

Ubuntu 18.10 now enables Center New Windows by default. I think it’s a nice touch and it would be nice if it were the default in GNOME too. Try it out for yourself!

For more details about what’s changed in 3.30 and who’s done the changing, see the project NEWS file.

September 01, 2018

GNOME Keysign 0.9.9

tl;dr: We have a new Keysign release with support for exchanging keys via the Internet.

I am very proud to announce this version of GNOME Keysign, because it marks an important step towards a famous “1.0”. In fact, it might be just that. But given the potentially complicated new dependencies, I thought it’d be nice to make sort of an rc release.

The main feature is a transport via the Internet. In fact, the code has been lurking around since last summer thanks to Ludovico’s great work. I felt it needed some massaging and more gentle introduction to the code base before finally enabling it.

For the transport we use Magic Wormhole, an amazing package for transferring files securely. If you don’t know it yet, give it a try. It is a very convenient tool for sending files across the Internet. They have a rendezvous server so that it works in NATted environments, too. Great.

You may wonder why we need an Internet transport, given that we have local network and Bluetooth already. And the question is good, because initially I didn’t think that we’d expose ourselves to the Internet. Simply because the attack surface is just so much larger and also because I think that it’s so weird to go all the way through the Internet when all we need is to transfer a few bytes between two physically close machines. It doesn’t sound very clever to connect to the Internet when all we need is to bridge 20 centimetres.

Anyway, as it turns out, WiFi access points don’t allow clients to connect to each other :( Then we have Bluetooth, but it’s still a bit awkward to use. My impression is that people are not satisfied with the quality of Bluetooth connections. Also, the Internet is comparatively easy to use, both as a programmer and a user.

Of course, we now also have the option to exchange keys when not being physically close. I do not recommend that, though, because our security assumes the visual channel to be present and, in fact, secure. In other words: Scan the barcode for a secure key signing experience. Be aware that if you transfer the “security code” manually via other means, you may be compromised.

With this change, the UX changes a bit for the non-Internet transports, too. For example, we have a final page now which indicates success or failure. We can use this as a base for accompanying the signing process further, e.g. sign the key again with a non-exportable short-term signature s.t. the user can send an email right away. Or exchange the keys again after the email has been received. Exciting times ahead.

Now, after the wall of text, you may wonder how to get hold of this release. It should show up on Flathub soon.

Feeds