PDF's drawing model inherits from PostScript, which was originally designed in the early 80s. It works and is really nice but the one glaring hole it has is missing transparency support. The original PDF spec from 1993 has no transparency support either, it was added in version 1.4 that came out mere 8 years later in 2001.
Setting drawing operation colours in PDF is fairly straightforward. You list the values you want and call the color setting command, like this:
The property name for alpha in PDF is CA and ca for stroking and nonstroking operations, respectively. This would imply that you should be able to do this:
0.9 ca % Sets nonstroke alpha? [no] 0.2 CA % Sets stroke alpha? [no]
But of course you can't. Instead the alpha value can only be set via a Graphics State dictionary, which is a top level PDF object that you then need to create, use, link to the current page's used resources and all that jazz.
/Alpha01State gs % Set graphics state to the dictionary with the given name
This works fine for use cases such as compositing layers that do not themselves use transparency. In that case you render the layer to a separate XObject, create a graphics dictionary with the given composition parameters, use it and then render the layer XObject. It does not work at all in pretty much any other case.
The more observant among you might have already noticed that you need to create a separate graphics state dictionary for every transparency value used in your document. If you are, say, a vector graphics application and allow artists to adjust the transparency for each object separately using a slider (as you should), then it means that you may easily have hundreds or thousands of different alpha values in your document. Which means that you need to create hundreds or thousands of different transparency graphics state dictionaries if you want to preserve document fidelity.
Why is it implemented like this?
As is common, we don't really know (or at least I don't). The Finger of Speculation points towards the most usual of suspects: backwards compatibility.
Suppose the alpha command had been added to the PDF spec. It would mean that trying to open documents with those commands in older PDF renderers would cause crashes due to unknown commands. Yes, even if the document had specified a newer minimum PDF version, because PDF renderers would try to open and show them anyway rather than popping an error message saying "Document version is too new". OTOH adding new entries to a graphics state dictionary is backwards compatible because the renderer would just ignore keys it does not understand. This renders the document as if no transparency had ever been defined. The output is not correct, but at least mostly correct. Sadly the same can't be done in the command stream because it is a structureless stream of tokens following The One True Way of RPN.
If the hypothesis above is true, then the inescapable conclusion is that no new commands can be added to the PDF command stream. Ever.
In the last few months, I have been trying to improve the default UI shipped by QEMU. As you might not know, QEMU ships with various UI backends: GTK, SDL, Cocoa and recently a DBus one.
I first started trying to port the GTK 3 backend to GTK 4 but faced some issues where I couldn't use GtkApplication as it starts its own GMainLoop which interferes with some god knows what internal GMainLoop started by QEMU itself. My intention was not to only do a simple port but also to see how we could optimize the rendering path as well.
At that time, I also learned that Christian Hergert started working on libmks, a new client-side C library of the DBus backend as he has the intention of using it in GNOME Builder. Marc-André Lureau, one of the upstream QEMU maintainers, is also working on something similar, with a larger scope and using Rust called RDW, a Remote Desktop Widget to rule them all.
Damage areas
From a rendering perspective, the major difference between libmks and RDW at that time is that libmks used a custom GdkPaintable doing a tiled rendering of a GLTexture imported from a DMABuf as GTK had no API to set the damaged region of a GLTexture and it was doing a pointer comparison for a GSK_TEXTURE_NODE render nodes, hence the usage of a tiled rendering. RDW on the other hand was using a plain GtkGLArea.
Christian Hergert also shared with Javier Martinez Canillas and me about his findings while working on libmks:
So now that we have fixed the VirtIO GPU driver and QEMU, we should be able to get proper damage reporting information from the guest, right??? Well, that is what Javier & I were hoping for. A few days into debugging what was going on without much luck, I met Robert Mader during Linux App Summit and discussed the issue with him and he mentioned a certain deny list of drivers from using the atomic KMS API in Mutter. Interesting.
The Monday after LAS, I shared the info with Javier and we found out the root cause of our pain. As the atomic KMS API has no properties for setting the cursor hotspot, which is useful for the virtualized use case, it was put on a deny list until such properties are added to the kernel. Javier also found that Zack Rusin from VMware is already on it, he submitted a patch for the kernel and had a local branch for Mutter as well. Albert Esteve tested the Kernel/Mutter patches and confirmed they were good. He also wrote an IGT test, which will hopefully help get the patch merged soon.
Around that time, Benjamin Otte was working on adding a new builder-like API to simplify the creation of a GdkGLTexture, with a very neat feature, allowing to set the exact damage region instead of doing a pointer comparison. I offered to test the API and ported libmks to use it.
Other improvements
Sergio Lopez Pascual worked on adding multi-touch support to virtio-input/QEMU which I then exposed on the QEMU DBus interface & implemented the libmks side of it.
For the future
We still don't have all the required features to make libmks a drop-in replacement for Spice usage in GNOME Boxes like USB redirection, Clipboard sharing. I have WIP branches for both, one day I will finish them.
Porting GNOME Boxes to GTK 4
The Kernel / Mutter patches from Zack being merged
Once all these pieces land, you will hopefully be able to enjoy a slightly better experience when running a virtualized OS that properly propagates the damaged regions.
For now, you can try libmks either by using the debugging tool shipped with the library or by using this very small client I wrote on top of it, Snowglobe.
Acknowledgement
As you can tell, a bunch of components and people were involved in this few months trip. I would like to thank everyone for taking the time to explain basic things to me as well as helping out to get this done! You can now enjoy a well deserved screen recording showing the implication of these changes
I am nailing down the release process for this, but my intention is
that starting with GNOME 45 / librsvg-2.57.0, the Rust crate will be
in sync with the GNOME release schedule: a stable minor release every
six months, and micro releases regularly and as needed.
For now, these releases are equivalent and contain the same code:
2.56.92 (the usual tarball, beta release before GNOME 45)
2.57.0-beta.2 (release for crates.io)
Both git tags exist in
gitlab.gnome.org's repository and point to the same commit.
While GNOME uses .9x micro version numbers to identify alpha or beta
releases, Rust uses Semantic Versioning, which
allows for the -beta.2 suffixes. I'll keep the corresponding
schemes and add both git tags to the release commits.
As part of the same process outlined in Matthias Clasen's "LibreOffice packages" email, my management chain has made the decision to stop all
upstream and downstream work on desktop Bluetooth, multimedia
applications (namely totem, rhythmbox and sound-juicer) and
libfprint/fprintd. The rest of my upstream and downstream work will be
reassigned depending on Red Hat's own priorities (see below), as I am
transferred to another team that deals with one of a list of Red Hat’s
priority projects.
I'm
very disappointed, because those particular projects were already
starved for resources: I spent less than 10% of my work time on them in
the past year, with other projects and responsibilities taking most of
my time.
This means that, in the medium-term at least, all those GNOME projects will go without a maintainer, reviewer, or triager:
- gnome-bluetooth (including Settings panel and gnome-shell integration)
- totem, totem-pl-parser, gom
- libgnome-volume-control
- libgudev
- geocode-glib
- gvfs AFC backend
Those freedesktop projects will be archived until further notice:
- power-profiles-daemon
- switcheroo-control
- iio-sensor-proxy
- low-memory-monitor
I will not be available for reviewing libfprint/fprintd, upower, grilo/grilo-plugins, gnome-desktop thumbnailer sandboxing patches, or any work related to XDG specifications.
Kernel
work, reviews and maintenance, including recent work on SteelSeries
headset and Logitech devices kernel drivers, USB revoke for Flatpak
Portal support, or core USB is suspended until further notice.
All my Fedora packages
were orphaned about a month and a half ago, it's likely that there are
still some that are orphaned, if there are takers. RHEL packages were
unassigned about 3 weeks ago, they've been reassigned
since then, so I cannot point to the new maintainer(s).
If
you are a partner, or a customer, I would recommend that you get in
touch with your Red Hat contacts to figure out what the plan is going
forward for the projects you might be involved with.
If
you are a colleague that will take on all or part of the 90% of the
work that's not being stopped, or a community member that was relying on
my work to further advance your own projects, get in touch, I'll do my
best to accommodate your queries, time permitting.
I'll try to make sure to update this post, or create a new one if and when any of the above changes.
I was lucky enough to attend the 2023 edition of GUADEC in Riga, Latvia.
In a way GUADEC is more fun each year because each time I know more people, have more things to talk about, and also it’s 4 years since my last in-person GNOME event so this one was particularly special.
Where else can you have in-depth conversations about the future of GObject, for example? (The future sounds interesting, by the way).
My main focus was getting everyone interested in end-to-end testing using openQA, which met with quite some success, although there is still some way to go before we can declare the end-to-end testing “production ready”. I will keep working on the remaining tasks in my 1-2 hrs a week that I am sometimes able to spend on it; as always, things will go much faster if and when more folk get involved in the effort. We are in #gnome-os if you want to join in.
You can watch my talk about end-to-end testing with openQA here.
We had a BoF session about desktop search, which was one of the most useful discussions I remember having on this topic, we covered topics all across the GNOME desktop rather than focusing on specific components, breaking free for once from Conway’s Law. I wrote up some of the outcomes here. It’s a good time to get involved in search!
After GUADEC was done I sailed from Liepaja to Lübeck, then had some vacation time around Germany and Italy, including a memorable gig watching the Slackers near lake Garda, finishing at midnight and somehow arriving back to Santiago about 10 hours later via various car, plane, train, etc. Luckily we missed most of the heatwave during the trip. I am only now getting back to work.
Many thanks to the local team who did a great job organising the event (I know first-hand how difficult it is :-), and good luck to Cassidy and the team organising GUADEC 2024 in Denver, USA.
It’s too early to know if I’ll make it to Denver but it’s only fair that us Europeans sometimes do a bit of long haul travelling and dealing with hostile border control authorities, stuff that some GUADEC attendees have to do every single year
We have landed the two last significant changes for GNOME 45. The image loading via glycin is now fully sandboxed, including SVGs. And the print dialog got updated to a new design by Allan Day.
GTK
Cross-platform widget toolkit for creating graphical user interfaces.
finally, focusing on list views is more accurate and reliable
tons of fixes for the accessibility support, with better name and description computation following the ARIA specification; improved handling of composite button widgets; and a whole accessibility overlay in the GTK inspector, showing missing names and descriptions in your UI
the Vulkan renderer has seen major improvements; Vulkan support is still marked as experimental, but it’s definitely less of a science project
multiple fixes in the GL renderer when it comes to mipmaps and texture filtering
Wayland, Windows, and macOS improvements and bug fixes
there are a few new deprecations, mainly related to GdkPixbuf
The GTK team is already hard at work on the next GTK 4.14 development cycle, so stay tuned!
Maps
Maps gives you quick access to maps all across the world.
Maps now has an experimental vector layer, using the new vector map support in libshumate. Currently it is using the OSM Liberty style. Longer-term the plan is to develop GNOME style style sheets supporting light and dark variant, using icons for the markers based on the GNOME Icon Library. At this time this showcases horizontally aligned labels (regardless of map rotation) and sharp rendering, also at fractional zoom levels. Some things that are not yet implemented includes being able to directly click on labels and markers, and localized names.
GNOME Contacts 45.beta is out, now powered by the GtkListView/GtkSectionModel APIs. This decreased memory usage of Contacts locally with about 20% and helps save some precious CPU cycles! It should also enable some nice-to-have behaviors like being able to select multiple contacts at once with shift-click.
GJS
Use the GNOME platform libraries in your JavaScript programs. GJS powers GNOME Shell, Polari, GNOME Documents, and many other apps.
GJS released a new version for GNOME 45 beta. There are several bug fixes and documentation improvements. Highlights of the release are performance improvements by Marco Trevisan, and the upgrade of the JS engine to SpiderMonkey 115 thanks to Xi Ruoyao.
New things that you will be able to use in your JS code in GNOME 45 are the new findLast(), findLastIndex(), with(), toReversed(), toSorted(), and toSpliced() array methods, and the ability to create an array from an async iterable with Array.fromAsync().
NewsFlash got a new application id on flathub. It moved from com.gitlab.newsflash to io.gitlab.news_flash.NewsFlash. The flatpak CLI and Gnome software should ask you if you want to migrate to the new id.
This was done to be able to verify the app on flathub and in preparation for a big feature update. So stay tuned.
Gaphor development is continuing at full pace. This week Dan Yeaw released Gaphor 2.20.0. Support for SysML modeling has been improved with new elements and the ability to assign types to model elements.
Phosh and phoc have been released a couple of days ago. Besides not downscaling screenshots any longer phosh switched to newer versions of libgnome-volume-control and gmobile. The later was updated to support more notches (those of the Fairphone 4 and Poco F1) which were added to the 0.0.2 release of gmobile (also tagged last week).
Phoc got updated to newer wlroots allowing us to add support for newer xdg-shell versions but most importantly it fixes a long standing issues with GTK4 popups and on screen keyboard interaction.
The screenshot shows phoc reneding the rounded corners and the bounding box of a notch while phosh uses the same information to move the clock out of the center.
Parabolic V2023.8.1 is here! This release is packed with new features, improvements, and bug fixes. Read about them below :)
Changelog:
Added the option to prevent suspend while using Parabolic
Added the ability to skip the password dialog when unlocking Keyring
Added the ability to reset the Keyring when locked
Improved bitrate used for audio-only downloads with the best quality
Parabolic will now prefer videos with the h264 codec when downloading in the mp4 format. If space is a concern, users are recommended to download in the webm format which uses the vp9/vp8/av1 codecs
If overwriting files is off and a download’s file name exists, a numbered suffix will be attached to the end of the file name to prevent error
Fixed an issue where downloads with specific timeframes would download incorrectly
Fixed an issue where opus downloads would sometimes fail
Fixed an issue where Parabolic would be unusable on systems without NetworkManager installed
Fixed an issue where Parabolic would say no active internet connection even though there was a connection established
Updated translations (Thanks everyone on Weblate!)
Last week, I announced Daikhan media player without mentioning any features of the app. Well, one of its features is that it remembers what selections the user has made (e.g. selected audio/subtitle languages, etc.) for each opened file and when the user opens any of those files again, it restores those selections.
This week, the “Remember Selections” feature received superpowers. Now, the player can recognize files by their content. So, it can restore selections made by the user even if files have been renamed, moved to a new location or both.
Additionally,
Hardware acceleration support was added to the flatpak
Screenshots were updated
Access to files is only granted when you open them. Previously, the app had full access to the filesystem.
NOTE: The app is still in early access. So, expect some bugs and beware that big changes (even backward-incompatible ones) might occur in the app’s interface and behavior.
Cavalier V2023.8.1 is here! Tired of existing 6 drawing modes? Well, now there are 11, because every mode except Splitter got Circle variant! We also fixed the issue that caused incorrect drawing with display scaling enabled, and added a cute easter egg 👀
Here’s the full changelog:
All drawing modes except Splitter now have Circle variants
Added an easter egg (run the program with –help to find how to activate it)
Fixed an issue where the app wasn’t drawing correctly with >100% display scaling
Added welcome screen that is shown on start until any sound gets detected
Added Cairo backend that can be used in case of problems with OpenGL. To activate, run the program with environment variable CAVALIER_RENDERER=cairo
CAVA was updated to 0.9.0
Pipewire is now used as default input method, you can still change back to Pulse Audio using environment variable CAVALIER_INPUT_METHOD=pulse
Updated translations (Thanks everyone on Weblate!)
August is traditionally a slow month for the Foundation, as staff tends to take a small break after another successful GUADEC. Somehow, this week was chock full of meetings jammed in between folks who just got back from holidays and those about to leave for the same. Of particular note were:
the first regular Board meeting of the new term — we covered a lot of topics and I am looking forward to working with this new group for the next year
code of conduct committee meeting — amongst other things, we are working on our processes and transparency reports
executive director search committee — interviews are ongoing and we met to touch base and make sure there aren’t any hiccups
In addition, I have been catching up on the bills that accumulated on my virtual desk during GUADEC, doing various HR tasks (including digging up paperwork to give references), finishing the filing of our 2022 tax forms, and making sure the staff get the support they need. I see travel committee requests starting to come in… that will be on my todo list next week!
I like to keep myself honest by using slower computers regularly to do my job. When things become obnoxious, it reminds me to take a closer look at what’s going on.
Today, I did some more profiling of 45.beta with a not-too-old-but-still-a-bit-old laptop. It’s the first laptop I received at Red Hat in 2015. X1 Carbon gen3 (5th gen i7), with 8gb RAM. Not amazing by today’s standards, but still pretty good! Surely things will be fine.
Okay, so first up, after boot, I ssh in from my workstation so I can run sysprof-cli --session-bus --system-bus capture.syscap. Immediately afterwards, I type my login password and hit Enter.
Things are hanging for quite some time, what does Sysprof say?
Looks like we are spending most of our time decoding JXL images. Given that it took about 17 seconds to login, something is clearly going wrong here.
gdk-pixbuf-thumbnailer -s 256 adwaita-l.jxl test.png only takes about 3 seconds. So clearly there is more.
But first, we file an issue because we want to be logged in in about 500 milliseconds, not 15 seconds.
Up next we seem to be spending a lot of time in subdivide_infos() in … GTK CSS parsing. But more specifically, GTK 3 CSS parsing. Hrmm strange, what is GTK 3 that is loading at login? Apparently it’s all the gsd-* tools. And they’re GTK 3 still presumably from libcanberra-gtk or some other requirement. Cool.
Now here is the kicker, I remember putting in code years ago to avoid CSS parsing in these to specifically avoid this issue by creating a fake (and empty) GTK theme. I wonder why it’s not working. Maybe the transition from autotools to meson broke it.
After a bit of re-familiarizing myself with the code it looks like GTK will fallback to internal theme parsing if it can’t find the specified them from $GTK_THEME. Next we pop open gresources list gsd-media-keys to check that the theme resources are there, and indeed they are not. Quick patch to ensure linkage across static libraries and we’re off to the races. A bunch of time wasted in login shaved off and a couple dozen MB of memory reclaimed.
As you know, even with a “simple” language like Python, porting a desktop application to a new version of GTK can be a pretty significant amount of work; doubly so when it is accompanied by major refactoring of the app itself at the same time.
We need your help to determine when/if it would be “safe” to merge our long-running core rewrite & GTK 4 port branch to GTG‘s main branch and repository. We think it mostly works, but we can’t be sure until you try it out (preferrably while using a copy of your real world data file).
If you have an up-to-date Linux distro with recent versions of GTK (ex: 4.10.4+), and would like to help test & fix the upcoming GTK 4 version of GTG, check out this branch / PR and give it a try in the coming days. You can then help by:
Finding and reporting remaining showstopper issues: problems specific to that branch should be reported here (in Diego’s repository) for now.
We’re looking for problems that would be considered unacceptable/non-functional for the sake of merging to the main repository. Especially, anything related to data or dangerously broken behavior, if any.
Broken plugins are acceptable at this point.
If you can’t find any unreported issues after extensive testing… let us know too, it’s nice to know our code is perfect 😉
Sending merge requests to Diego’s repository (check the issues and existing PRs, if any, to avoid duplicate work) would help a ton, of course, because we’ve been stretched very thin lately.
Being able to confidently merge this branch would make it much easier for others in the GTG community to have a stable base to work on, and provide the remaining fixes to bring version 0.7 to the finish line.
P.s.: remember that we also have a Matrix discussion channel (see the GTG wiki page for details) if you’d like to coordinate some aspects in a more informal and interactive manner before posting to GitHub.
Up early; packed & bid 'bye to the lovely
American family, T. dropped me at Alewife & bus
onwards to a customer visit in New York.
Slowly started trying to catch up with E-mail
in the bus; lots of things to chew on - an hour per
day of mail to triage? Team been doing excellent things
in my absence (I should go away more), including our
latest COOL
23.05 enterprise release, packed with new goodness.
Up earlyish, helped the babes pack - somewhat
traumatic - normally it's just one of me reluctantly
leaving them, rather than five of them leaving me.
Watched How to train your Dragon with
the babes. B. cooked a fine chicken noodle lunch, and
went to see her school together, ice-cream on the way
home. B. taught the babes how to play basketball a bit.
Back, caught up with Thomas, flight delayed, and
then a tearful goodbye from misc. cousins to each other,
and off they went. Used flightradar24 to track their
progress in the night.
Helped T. to get many things back into their
normal good order in their house and chatted until late.
Update: I am delighted to have been wrong! See the end.
Briefly, an interesting negative result: consider benchmarks b1, b2,
b3 and so on, with associated .c and .h files. Consider libraries
p and q, with their .c and .h files. You want to run each
benchmark against each library.
P and Q implement the same API, but they have different ABI: you need to
separately compile each benchmark for each library. You also need to
separate compile each library for each benchmark, because p.c also
uses an abstract API implemented by b1.h, b2.h, and so on.
The problem: can you implement a short GNU Makefile that produces
executables b1.p, b1.q, b2.p, b2.q, and so on?
The answer would appear to be "no".
You might think that with call and all the other functions available
to you, that surely this could be done, and indeed it's easy to take the
cross product of two lists. But what we need are new rules, not just
new text or variables, and you can't programmatically create rules. So
we have to look at rules to see what facilities are available.
With pattern
rules,
you can easily modify these rules to parameterize either over
benchmark or over library, but not both. What you want is something
like:
But that doesn't work: you can't have a wildcard (*) in the pattern
rule. (Really you would like to be able to match multiple patterns, but
the above is the closest thing I can think of to what make has.)
Static pattern
rules
don't help: they are like pattern rules, but more precise as they apply
only to a specific set of targets.
You might think that you could use $* or other special variables on
the right-hand side of a pattern rule, but that's not the case.
You might think that secondary
expansion
might help you, but then you open the door to an annoying set of
problems: sure, you can mix variable uses that are intended to be
expanded once with those to be expanded twice, but the former set better
be idempotent upon second expansion, or things will go weird!
Perhaps the best chance for a make-only solution would be to recurse
on generated makefiles, but that seems to be quite beyond the pale.
To be concrete, I run into this case when benchmarking
Whippet: there are some number of
benchmarks, and some number of collector configurations. Benchmark code
will inline code from collectors, from their header files; and
collectors will inline code from benchmarks, to implement the
trace-all-the-edges functionality.
So, with Whippet I am left with the strange conclusion that the only
reasonable thing is to generate the
Makefile
with a little custom generator, or at least generate the part of it to
do this benchmark-library cross product. It's hard to be certain about
negative results with make; perhaps there is a trick. If so, do let
me know!
See, I thought that make functions were only really good in variables and rules and the like, and couldn't be usefully invoked "at the top level", to define new rules. But that's not the case! eval in particular can define new rules.
So a solution with eval might look something like this:
I dug out a computer running Fedora 28, which was released 2018-04-01 - over 5 years ago. Backing up the data and re-installing seemed tedious, but the current version of Fedora is 38, and while Fedora supports updates from N to N+2 that was still going to be 5 separate upgrades. That seemed tedious, so I figured I'd just try to do an update from 28 directly to 38. This is, obviously, extremely unsupported, but what could possibly go wrong?
Running sudo dnf system-upgrade download --releasever=38 didn't successfully resolve dependencies, but sudo dnf system-upgrade download --releasever=38 --allowerasing passed and dnf started downloading 6GB of packages. And then promptly failed, since I didn't have any of the relevant signing keys. So I downloaded the fedora-gpg-keys package from F38 by hand and tried to install it, and got a signature hdr data: BAD, no. of bytes(88084) out of range error. It turns out that rpm doesn't handle cases where the signature header is larger than a few K, and RPMs from modern versions of Fedora. The obvious fix would be to install a newer version of rpm, but that wouldn't be easy without upgrading the rest of the system as well - or, alternatively, downloading a bunch of build depends and building it. Given that I'm already doing all of this in the worst way possible, let's do something different.
The relevant code in the hdrblobRead function of rpm's lib/header.c is: int32_t il_max = HEADER_TAGS_MAX; int32_t dl_max = HEADER_DATA_MAX;
which indicates that if the header in question is RPMTAG_HEADERSIGNATURES, it sets more restrictive limits on the size (no, I don't know why). So I installed rpm-libs-debuginfo, ran gdb against librpm.so.8, loaded the symbol file, and then did disassemble hdrblobRead. The relevant chunk ends up being: 0x000000000001bc81 <+81>: cmp $0x3e,%ebx 0x000000000001bc84 <+84>: mov $0xfffffff,%ecx 0x000000000001bc89 <+89>: mov $0x2000,%eax 0x000000000001bc8e <+94>: mov %r12,%rdi 0x000000000001bc91 <+97>: cmovne %ecx,%eax
which is basically "If ebx is not 0x3e, set eax to 0xffffffff - otherwise, set it to 0x2000". RPMTAG_HEADERSIGNATURES is 62, which is 0x3e, so I just opened librpm.so.8 in hexedit, went to byte 0x1bc81, and replaced 0x3e with 0xfe (an arbitrary invalid value). This has the effect of skipping the if (regionTag == RPMTAG_HEADERSIGNATURES) code and so using the default limits even if the header section in question is the signatures. And with that one byte modification, rpm from F28 would suddenly install the fedora-gpg-keys package from F38. Success!
But short-lived. dnf now believed packages had valid signatures, but sadly there were still issues. A bunch of packages in F38 had files that conflicted with packages in F28. These were largely Python 3 packages that conflicted with Python 2 packages from F28 - jumping this many releases meant that a bunch of explicit replaces and the like no longer existed. The easiest way to solve this was simply to uninstall python 2 before upgrading, and avoiding the entire transition. Another issue was that some data files had moved from libxcrypt-common to libxcrypt, and removing libxcrypt-common would remove libxcrypt and a bunch of important things that depended on it (like, for instance, systemd). So I built a fake empty package that provided libxcrypt-common and removed the actual package. Surely everything would work now?
Ha no. The final obstacle was that several packages depended on rpmlib(CaretInVersions), and building another fake package that provided that didn't work. I shouted into the void and Bill Nottingham answered - rpmlib dependencies are synthesised by rpm itself, indicating that it has the ability to handle extensions that specific packages are making use of. This made things harder, since the list is hard-coded in the binary. But since I'm already committing crimes against humanity with a hex editor, why not go further? Back to editing librpm.so.8 and finding the list of rpmlib() dependencies it provides. There were a bunch, but I couldn't really extend the list. What I could do is overwrite existing entries. I tried this a few times but (unsurprisingly) broke other things since packages depended on the feature I'd overwritten. Finally, I rewrote rpmlib(ExplicitPackageProvide) to rpmlib(CaretInVersions) (adding an extra '\0' at the end of it to deal with it being shorter than the original string) and apparently nothing I wanted to install depended on rpmlib(ExplicitPackageProvide) because dnf finished its transaction checks and prompted me to reboot to perform the update. So, I did.
And about an hour later, it rebooted and gave me a whole bunch of errors due to the fact that dbus never got started. A bit of digging revealed that I had no /etc/systemd/system/dbus.service, a symlink that was presumably introduced at some point between F28 and F38 but which didn't get automatically added in my case because well who knows. That was literally the only thing I needed to fix up after the upgrade, and on the next reboot I was presented with a gdm prompt and had a fully functional F38 machine.
You should not do this. I should not do this. This was a terrible idea. Any situation where you're binary patching your package manager to get it to let you do something is obviously a bad situation. And with hindsight performing 5 independent upgrades might have been faster. But that would have just involved me typing the same thing 5 times, while this way I learned something. And what I learned is "Terrible ideas sometimes work and so you should definitely act upon them rather than doing the sensible thing", so like I said, you should not do this in case you learn the same lesson.
I attended GUADEC 2023 this year in-person in Rīga. It was nice to be able to see more people in person than last year’s mini-GUADEC in Berlin, and nice to be able to travel overland/oversea to Rīga, avoiding flights and the associated huge carbon emissions. Thank you to the GNOME Foundation and my employer, Endless, for covering the travel. And a big thank you to the local event organising team, the AV team, the volunteers and the Foundation staff for making it all happen.
The quality of the talks this year was really high. I don’t think there was a single talk slot I skipped. As a result, I didn’t get much hacking done! But there were good hallway conversations and catch ups.
I gave two talks, one on some simple improvements people can make to their apps to reduce internet data use and power use when doing so would be beneficial to the user (when on a metered network or in power-saver mode). The aim was to remind people how easy it is to do this, and provide some examples of how different apps present these states/events in the UI, since the best way to do that can differ between apps.
The talk has resulted in three GNOME initiatives. Each corresponds to a different state your app can pay attention to — some apps should pay attention to all of them, some apps only some of them. Please take 10 minutes to check your app against the initiatives and make updates if necessary. I’m in the gnome-hackers and gnome-circle Matrix rooms if you have any questions about them.
My second talk was an overview of work I’ve been doing on-and-off over the past couple of years (based on work by others) to allow apps to save and restore their state when the user logs out or shuts down the computer. The idea is that the user can restore the set of apps they were using, in the state and configuration they were left in, when next starting the computer or logging in. It’s a big project, and there’s a long way to go, but it felt like the right time to present what we’ve got so far, let more people know how the project is structured, and get feedback from toolkit developers and app authors about the whole idea.
Thankfully there was 5 minutes for questions at the end of the talk, and people used them well to raise some good points and ask some good questions. I’m still in the process of factoring all that feedback into the plan, but should hopefully have an update to give on the project in a future blog post soon.
Interesting talks I attended included Peter Hutterer’s talk about sending and capturing input in Wayland, which I think distilled his blogposts about the topic very clearly. Allan Day’s talk about communication, which was an excellent summary of a rather long list of books on the subject, and it was very clearly presented. I feel like I could do with a cheatsheet list of his recommendations to stick next to my computer sometimes. Evan Welsh, Philip Chimento, Nasah Kuma and Sonny Piers’s talks about JavaScript, TypeScript and the latest changes in GJS. These provided a good update on a lot that has been happening in GJS, a useful overview of TypeScript, and were really clearly presented. Jussi Pakkanen’s talk about circles, or was it about settings? It was a good example of a lightning talk which draws you in then spins you around.
In the hallway track, I had some interesting chats with Bartłomiej about the infrastructure of ODRS, which is used to provide ratings/review data to gnome-software. I also had some positive conversation with Cassidy about plans for GUADEC 2024 in Denver. And at the event dinner, a really energising chat with Scott about canyoning, hiking, skiing and caving in the US and the UK.
I’m back from Rīga, Latvia where I attended GUADEC 2023—the annual GNOME conference. It was my first time attending since 2019 due to COVID-19 plus having had another child in 2022. It simultaneously felt like it had been ages since seeing everyone, and like we picked back up after no time at all.
For me, attending GUADEC in person is exceptionally important: the face-to-face conversations, exploration of the host city, eating and drinking together, and spontaneous hacking all serve as powerful bonding experiences and help develop mutual understandings of the actual humans behind online usernames and avatars. We come away from GUADEC with fresh ideas, new perspectives, stronger friendships, and renewed energy to make GNOME and the open technologies we all care about better than before. You can get a small fraction of that from remote attendance—and remote participation is important to make the event as accessible and inclusive as possible—but I greatly prefer in-person participation if at all possible.
While traveling from the US was as long and grueling as ever (and meant nearly two weeks away from my family), it was ultimately worth it to be in person. I just hope we can spread GUADEC out a bit more in coming years so it’s not this hard every year. 😅 More on that in a bit…
I was given the opportunity to attend the GNOME Advisory Board meeting for the first time, representing Endless. The Advisory Board is a group of organizations involved in and with a vested interest in the success of GNOME that meets twice annually—at GUADEC and FOSDEM. There is no decision making authority, but it serves as a time for people from these organizations to get together with one another and the elected GNOME Foundation Board of Directors to discuss their ongoing and upcoming plans as they relate to GNOME.
There is a shared understanding of both collaboration and confidentiality—we share with one another what we can to coordinate our efforts, ultimately multiplying our individual organizations’ impact. Because of the understanding of confidentiality, I won’t go into detail; but here were common threads and conversations:
Accessibility
Multiple people mentioned investing efforts into improving accessibility of GNOME technologies. Platform modernization with Wayland and GTK4 brought troves of technical improvements, but the resources for tackling accessibility to ensure it remains as good as or better than in the past have been lacking in recent years. This has been recognized and I’m confident it will be well-addressed.
In addition to resources provided by organizations, there may be funding from grants to help in these areas in the coming years. If you are someone who is aware of grant application processes or knows of specific grant programs, this is a huge way you can contribute to GNOME; reach out to the Foundation to help coordinate securing additional funding in this area. Or contact me, and I can point you in the right direction—my contact details are in the footer!
Member Participation
Due to a variety of reasons, the Advisory Board meeting was less-attended this year than in the past. We discussed this a bit—particularly over the lunch hour—and brainstormed ways to ensure everyone can be represented. One area identified was that the move from mailing lists to Discourse and GitLab may have left some folks behind, and this will be addressed. We also identified outdated contact details and planned to speak to specific individuals to help ensure member organizations knew of the meeting and knew their attendance was welcome and encouraged.
We also chatted a bit—both at the meeting and throughout the week—about how we could encourage other organizations to join the Advisory Board to represent diverse interests. This inspired me to sign up for a “What is the GNOME Advisory Board?” lightning talk as a first step towards demystifying the Advisory Board—and encouraging more organizations to join.
Travel Sponsorship & Hackfest Locations
A member of one organization reported that they were able to help GNOME by providing physical space at their organization’s offices for hackfests, which is another great way organizations can contribute to GNOME; travel is one aspect of organizing meetups, but having a well-resourced location is important as well. If you work for an organization that could help out in this area as well, let the Foundation know—or contact me via the details in the footer and I can forward it on.
There was also feedback about the travel sponsorship process; this was an area the Foundation knew was a bit painful this year as things moved to GitLab, and some outdated information persisted in other places. But it’s being addressed.
Reproducibility of Flatpaks
We heard an update from Flathub as well as at least one member organization that there are mutual interests in getting reproducible Flatpak builds integrated with Flathub.
I’m not deeply into the enterprise world, so forgive me if I don’t quite get this right, but; my understanding of one example was that if an enterprise distribution wants to encourage their customers to use Flathub because it’s where the apps are, they should be able to sign off on a hash of the app build built on their own infrastructure—even if the customer ends up downloading that build from Flathub’s infrastructure. If the apps are built reproducibly, the hashes would match, so it wouldn’t matter where the bits themselves came from.
I know this is an important step in the overall trust model of a neutral infrastructure like Flathub, anyway; you should be able to trust the middleware not because of the people who are involved (and they’re good people!), but because it’s entirely transparent and auditable.
Upstreaming More than Code
Something raised was: how can the Foundation and Advisory Board organizations better support upstreaming more than just code contributions—that is, things like metrics, survey results, user study results, and rationale behind design decisions? For example, some downstream distributions make design decisions that aren’t upstreamed into GNOME (think: adding a dock, changing stylesheets, etc.). While downstream differentiation is important to those organizations, are there ways we can better support that or learn from those changes upstream to make GNOME a better product for all the downstreams?
This was more of a “get the gears turning” discussion without any specific immediate actions or outcomes, but it was an interesting conversation nonetheless.
My Talks
I originally planned to give one talk this year: an updated and more GNOME-specific version of a talk I gave last year at the Ubuntu Summit. Since I was involved with the GNOME Design team over the past year, I also ended up volunteering to co-present the annual GNOME Design State of the Union talk, then I signed up for the aforementioned What is the GNOME Advisory Board lightning talk—and then I was almost roped into co-presenting another talk but ended up just helping write and prepare slides for it. 😅️
I actually really enjoyed being so involved this year, and I regret not getting signed up to help with introducing speakers, A/V, or some other volunteer position. Maybe I’ll make up for it next year…
How to Make a Delightful App Listing
This was a fun talk, but I committed one of the conference sins I always try to avoid: I rewrote much of it the night before. 😬️ It was honestly not that bad since I’d presented a version of it last year, but it turns out that Flathub had since published really great AppData guidelines—and I only learned of this at a bar the night before my talk. So… I rewrote the last section of my talk incorporating some of that advice.
In the end, I think it made the talk far better. It was a good taster of the guidelines and I hope it encouraged app developers—both of core GNOME apps, GNOME Circle apps, and the broader ecosystem—to revisit their app listings to present them in the best light possible.
The annual update from the GNOME Design team was fun to participate in. While I wasn’t deeply involved in making mockups, icons, or major design decisions, I was involved nearly every week over the past year in providing feedback and insight. We divided the talk into a six sections plus an intro and summary, with us each presenting two of those. I took the Intro and Core Apps; while they were brief, it was great to be able to share the stage with (and show off the work of) talented folks constantly pushing GNOME design forward.
This was the lightning talk I decided to put on the schedule after attending my first GNOME Advisory Board meeting. It was pretty rapid-fire, and most of the actual content came straight from the GNOME wiki, but I got a lot of feedback afterwards that it helped demystify the Advisory Board—and even inspired a few people to reach out to organizations to encourage them to join. 🙌️
This was a talk by Robert McQueen, CEO of the Endless OS Foundation and President of the GNOME Foundation Board of Directors—but Will Thompson and I did help out a bit in preparing notes and some slides. 😁️
Rob pulled it all together into great talk sharing insights, knowledge, and ways forward learned by our work at Endless—particularly around how first-time computer users (including both kids and adults) are an impactful target audience for the work we do on GNOME.
A huge part of the GUADEC experience for me is the “hallway” chats often happening literally in hallways, but also over lunch, drinks, or late nights in someone’s accommodation.
GUADEC 2024: Denver, CO, USA 🎉️
A persistent focus of mine was on GUADEC itself: how it was organized, how people “did” GUADEC, and what their feedback was. This was because—as was announced in the closing ceremony—GUADEC is coming to my home town of Denver, Colorado in 2024! I think I kept the lid on it until the closing ceremony, but I made sure to pay attention people to and listen to their input.
One tiny example of feedback: yoga mats! Someone suggested that with all the travel, walking, lugging backpacks, and late nights, it would be nice to have a quiet corner somewhere with half a dozen or more yoga mats and sanitizing spray. Attendees could take a moment to stretch out or just center themselves during a break. I thought this was a great idea, and I’ll be pushing for this in Denver—maybe it will become a recurring part of GUADEC?
We also talked a lot about how to document things clearly on the GUADEC website, reducing the duplicated GUADEC communication channels (I think I counted five different channels on Matrix… and most people only used one), encouraging inclusive eating options like vegetarian, vegan, and gluten-free restaurants, and of course: all the fun things people would want to do in and around Denver (Casa Bonita? Meow Wolf? The mountains?).
Another area we focused on was climate impact: while flying a bunch of Europeans to the US obviously has its impacts, there will also be significantly reduced travel for many of us within North, Central, and South America. Even so, I’d like to encourage lower-carbon transit options as much as possible, e.g. rail and bus if possible within the US, trying to keep everything for the week of GUADEC itself along convenient public transit or walkable, and reducing our impact in smaller but meaningful ways like encouraging reusing lanyards and avoiding other single-use waste. Philip Withnall provided some great tips for me, and I’ll be taking his advice for how to approach some of these areas.
The Can of Worms: Metrics
Against my better judgment (and the advice of friends), I broached the topic of privacy-preserving metrics once or twice… In all honesty, GUADEC hallways/bars are the perfect sort of place to have these conversations without the pressure of forming an Internet Opinion™️ etched forever into the collective memory. We’ve regularly been having these sorts of conversations since at least 2018, anyway, and while recent Fedora Discourse threads of doom have made it more of “a thing” than in years past, I felt it was important to speak candidly about it in person.
I came away with a few insights.
First, there are real areas where GNOME maintainers literally don’t know if complex code paths are ever touched in the real world, and this is an area a privacy-preserving tally-counter could provide real value to users; for example, GNOME Document Viewer (Evince) apparently handles something like ten more file formats than just PDFs, but nobody we talked to at a GNOME developer conference even knew that. Document Viewer is also one of the handful of apps that hasn’t been ported to GTK4 and Libadwaita, in part due to the complexity of maintaining those codepaths. It sure would be beneficial to the maintainers to know if anyone actually uses those other formats in Document Viewer, and if not, it could greatly speed up the modernization of the app—bringing both consistent design and improved performance—by simplifying what it supports. At the same time, if it turns out that one or two other format than PDF is actually widely used in the wild, then that could inform the design of Document Viewer as well, making it better at handling the tasks it’s actually used for.
Second, people still don’t really know the details of the open source metrics system developed by Endless and proposed to be used by Fedora. For example, the system protects privacy by reducing precision on the client side while still answering broad topics, e.g. by filing off the last several digits of latitude and longitude in estimated geolocations. It can aggregate data points locally and only submit a summary of events—e.g. an aggregate count of how many times a specific feature was used over the past week rather than submitting a timestamped event every time the feature was used. This was well received, but it was also clear that it could be better communicated since people didn’t know it.
Third, the idea of further “fuzzing” data via differential privacy or randomized response (techniques from at least as far back as the 1960s to collect aggregate information while protecting individual privacy) was very popular. While the aforementioned reduction in precision is already implemented, this sort of randomized response to ensure individual event deniability is not yet—this is a clear area that could improve the system for all users.
Lastly, people seem to be scared to raise their (or affiliated organizations’) heads over the parapet to become a target of The Internet Mob. It’s understandable considering the difficulty in communicating nuance plus the knee-jerk backlash from people around the Fedora proposal. However, I think this is actually a downstream issue from the two above: if people don’t know how this works and it doesn’t handle privacy preservation in some ways as thoroughly as say, Apple (even if it does so in a much more transparent and auditable way), then it sounds easier to just avoid it. Perhaps more energy could be spent around addressing those areas—in the end, GNOME contributors are some of the most privacy-conscious people and are exactly the type of people I want designing and developing anything to do with metrics, but that’s a point that seems to be lost on the Internet at large.
Guerrilla User Testing
An entertaining and at least mildly insightful task at GUADEC was cornering unsuspecting locals and getting them to try GNOME. Okay, it wasn’t quite like that: we did find friendly faces at coffee shops and around town who were willing to take a few moments to run through some impromptu user testing around the Activities button and the new proposed design. This included locals, staff of venues, and attendees’ partners who are not currently GNOME users.
I think there will be more shared on this process in the future, but my take-away was that the informal nature of this served as a sort of vibe check more than anything; we weren’t disciplined in how we took notes, what leading questions we asked or avoided, and how we presented it from person to person. Allan Day did write up a great more formal process that I’d like us to follow to get better results from, though.
Stars & Thumbs: Ratings & Reviews
The topic of ratings and reviews (e.g. in GNOME Software) came up a few times, inspired from feedback from the Bottles developers, attendees of my App Listing talk, and other hallway chats. I’ve summarized my thoughts in a blog post published last week; give it a read!
Summary & Thanks
I originally set out to detail every talk I attended, each social venue, etc. but there was so much to talk about around the “hallway” track that I wanted to detail that, instead. I would highly recommend taking a look at the YouTube Playlist of talks: they were all professionally recorded and streamed, and a gracious commentor has provided timestamps to each talk. 🙏️
I hope this write-up provided some insight into the goings-on of GUADEC and my experience there.
I’d like to thank my employer, the Endless OS Foundation, for allowing me to spend an entire week of work at GUADEC; it’s an incredibly valuable aspect of GNOME, and I love that Endless sees that. I’d also like to thank the GNOME Foundation for partially sponsoring my attendance as a member of the GNOME Foundation; ensuring people can get to these events is crucial for community building and all of the high-bandwidth communication that can really only happen in person.
This was my first time ever traveling outside of my country. I remember landing in Riga, and on my way to the hotel just being in awe looking at the various buildings and roads, everything was so different from what I’m used to and even though I have seen stuff in movies, seeing it in person is a very different experience. I video-called my parents that day, showing them around the city and I found it very surprising that it was around 9 pm at that time but there was broad daylight everywhere. I’m pretty sure I slept at around 3 am that night because of my excitement for the event.
The next day, I got late for the opening remarks and ended up missing the first talk because my smart ass went to the wrong venue and I had to quickly go back to the hotel and get a taxi to the actual venue.
I first attended the talk given by Regina Nkemchor, where she talked about introducing Africans to the GNOME community and the kinds of challenges she faces. It was a very informative talk, and the insights she shared can be very useful for other people as well who are trying to take on a similar task. After the talk, we had a 10-minute coffee break where I finally got a chance to meet my GSoC mentor Sonny Piers.
A photo I took with Sonny Piers, who happens to be rather tall and is trying to squat down for the photo
Another talk that stood out for me, which I was also very excited about was “GNOME Design: State of the Union” where the GNOME design team talked about not only some notable design changes they made over the past couple of years but also some new and upcoming design changes we might expect to see which all seem very promising for example the new Activities button and the window tiling system.
After the first day of talks got over, me and the other GSoC interns took a walk around Old Town Riga where we got to see some really cool stuff.
On the second day, while on my way to the venue, I found Federico Mena Quintero (one of the co-founders of GNOME) also on his way to GUADEC. He was very friendly and I had a nice conversation with him about the history of GNOME and GTK while we were on our way, which was enlightening.
A photo I took with Federico later that day
I attended several interesting talks that day, like “How to make a delightful app listing” by Cassidy James which even though was aimed at app developers, I also found it useful to know if I ever plan to create a GTK app of my own. Another talk that stood out for me was “A GTK status update” by Matthias Clasen where he answered some important questions surrounding GTK, and also showed us some new additions to GTK we can expect to see in the upcoming updates. There was also a little discussion about GTK 5.
Then there was also a talk about GNOME Javascript, which was given by Evan Welsh, Philip Chimento, and my mentor Sonny Piers where they talked about various things surrounding GJS and its development. Sonny also decided to mention the GSoC work we do for Workbench, which I did not expect but I’d like to thank him for.
The third day was the last day of the conference, where after some rounds of talks, there was an Annual General Meeting and then a group photo following it. We finally ended the day with the GNOME Intern Lightning Talks where GSoC and Outreachy students get around 5 minutes to talk about the work they are doing and its progress. It was a fun exercise for me to talk in front of a crowd. You can watch our entire presentation here.
After the conference ended, the day wasn’t over for us yet because GNOME had arranged and sponsored a dinner for everyone. This was the most fun I had ever had, I got to interact with some cool people and it was nice to see everyone get along together and forming little groups, having conversations.
A photo I took with Julian Hofer (the author of “GUI Development with Rust and GTK4”). I became good friends with him during the conference.
The following 2 days we had BoFs (Birds of Feathers) which are sort of like interactive workshops/discussions. We had a workshop for Workbench as well where Sonny talked about creating apps using Workbench, and I also tried helping some people during the workshop. There was also a discussion about GSoC and Outreachy internships, how to make the internship experience better, and discussing challenges faced by participants which turned out to be very productive and we came up with some good ideas.
The last few days I also just spent some time sightseeing the beautiful city.
I also have to mention this one other thing, one of the days while coming back from dinner I stumbled across a bunch of GNOME developers, one of them being Sonny on their way back to the hotel and he told me that they were going to do a bit of “hacking” together and invited me to join them as well. And so I did, and I really liked it. Everyone was congregated in a Airbnb doing their own work and I found it very cool. I got some GSoC work done there and also got to have a good chat with Sonny and Julian again. I went back to the hotel at 1 am later that night but it was fun.
On the last day of my stay there, GNOME had planned a day trip for us where they took us to Sigulda, a town 50kms away from Riga. We visited several places in Sigulda and we also ate lunch at a palace which was very posh.
I can’t even begin to express my gratitude towards GNOME for sponsoring my travel and accommodation to attend the event. I wouldn’t have imagined in my wildest of dreams that I would ever get an opportunity like this.
openSUSE Tumbleweed is a rolling release distribution, so it's
ideal for developers and users that like to have the bleeding edge
software.
It's also really "stable" to be a rolling release, from time
to time you can find a broken package because of one package is
updated and another one is not compatible yet, but it's something that
doesn't happen too often thanks to openqa tests, so you don't
need to worry about a breaking system.
You can find the Python interpreter and a lot of python modules
in every Linux distribution, but Tumbleweed does an interesting thing
for Python.
Default Python version (python3 -> python-3.11)
If you don't worry about the python version, you can just rely on
python3. In Tumbleweed python3 is not a real package, but the
default python version provides python3, so depending on when you
install python3 you will get a different package, if you install
it today (August 2023) you'll get python311.
In your system you'll have the /usr/bin/python3 binary that points
to the default python, so you don't need to worry about the current
version, you'll have there the default Python version for the
operating system.
In addition to the Python interpreter, you can find in the
distribution a lot of python modules, but again you can use the
default version, so, for example, if you want to install poetry, you
just use zypper install python3-poetry and that will install
the real package python311-poetry.
Multiple python versions (3.8, 3.9, 3.10, 3.11)
Besides the default Python, in Tumbleweed you can also find other
supported Python interpreters. Right now you can find all the Python
versions currently supported by the Python Foundation, from 3.8 to
3.11.
For Python 3.8 you'll only find the interpreter, because the python
modules are not built anymore, but for all the other versions you can
find almost the same modules.
All python modules that provide binaries uses the
update-alternatives, so you can configure in your system the version
that you want to use as default. For example, if you want to use the
3.9 version of poetry, having installed different versions you can
decide what /usr/bin/poetry points to:
/usr/bin/python3 is a link provided by the default python package,
so you can't modify with update-alternatives, so if you want to use
a different python version, make sure to do the correct call with the
full name python3.9, and use the correct shebang in your python
scripts, for example #!/usr/bin/env python3.9.
What happens when default changes?
This way of distributing Python interpreter and modules is useful,
because you don't need to update your software to work with the latest
Python version, if your software requires another version, you can just
install and continue using it, even on a bleeding-edge distribution
like Tumbleweed.
But this method has some problems. When the default Python interpreter
is changed in the distribution, all packages that depends on python3
will be updated, and that works correctly. But if you have installed
some python module using the python3 prefix, that package, and all
dependencies, is not updated automatically.
For example, if you installed python3-poetry when python3.10 was
the default system, and then the distribution updates the system
Python to python3.11, you don't get the python311-poetry package
by default, you'll need to install it again. This could break you
software, because if you use python3 and the dependencies are not
updated, you'll find that some dependencies are not installed after
updating.
For that reason, when the default python is changed in Tumbleweed, if
you've software that rely on python3 you should make sure to install
dependencies again by hand. And also you can do a cleanup and remove
all old version packages that you don't need anymore.
If you just want the latest version you can just do all at once with a
simple script like this:
#!/bin/bash# up-py-tw.shFROM=$1TO=$2if[$#-lt2]thenecho"Usage: up-py-tw.sh FROM TO"echo"example: up-py-tw.sh 310 311"exit0fiecho"Updating python packages from $FROM to $TO"OLDP=$(zypper-qsearch-ipython${FROM}-*|tail--lines+4|cut--delimiter\|--fields2)NEWP=$(echo$OLDP|sed"s/python${FROM}/python${TO}/g")
sudozypperin$NEWP
sudozypperrm$OLDP
You should verify what will be installed and what will be removed to
make sure that just the python related packages are removed. It's
possible that other packages depend on a python version that's older
than the system one and that's okay.
Beta version (3.12.0b4)
In Tumbleweed, right now you can find all the supported Python
versions, but it's not just that. At this moment you can also find the
beta version of the next Python interpreter.
$sudozypperinpython312
So if you are adventurous and want to test some new feature in the
next Python release, you can do it with the version provided there.
This version could also be used by openSUSE packagers to test the
packages before the Python version is released so it allow us to
prepare everything for the release and it could be in the distribution
earlier and more tested.
Development
All this multiple python versions is done at distribution level and
usually there's only one source package that produces the
python39-foo, python310-foo and python311-foo. The source
package is usually called python-foo and can be found in the
Python devel project.
These packages spec uses the python-rpm-macros to generate all the
versions from one source. If you're a packager you can find more
information on the openSUSE wiki about Python Packaging.
This is the first post in a series about tools used to develop GNOME and GNOME OS. Part two coming soon.
In the old age, developing the desktop was simple. You only had to install a handful of toolchains, development headers, and tools from the distribution packages, run make install, execute say_prayer.sh and if you had not eaten meat on Friday you had a 25% chance for your system to work after a reboot.
But how do you develop software in the brave new world of Image-Based systems and containerization?
Well, if you are an application developer Flatpak makes this very simple. Applications run against a containerized runtimes, with their own userspace. The only thing they need from the host system is a working Desktop Environment, Flatpak, and Portals. Builder and flatpak-builder along with all the integration we built into the Desktop make sure that it will be a breeze.
But what if you are developing a system component, like a daemon or perhaps GNOME Shell?
Till now the goto solution has been “Grab a fedora container and open enough sandbox holes until things work”. This is what Fedora Toolbox does, and it works great for a lot of things. However the sandbox makes things way more difficult than they need to be, and its rather limiting on what parts of the system you can test. But there is another way.
In GNOME OS we provide two images. The first one is how we envision GNOME to be, with all the applications and services we develop. The second one is complimentary, and it adds all the development tools, headers and debugging information we need to develop GNOME. Now the popular image based OSes don’t provide you with something like that, and you have to layer everything on your own. This makes it much harder to do things like running gdb against the host system. But in GNOME OS its easy, and by switching to the Development Edition/Image you get access to all the tools required to build any GNOME component and even GNOME OS itself.
Alright, I have a compiler and all the dependencies I need to build my project but /usr is still immutable. How do I install and run my modified build?
I am glad you asked! Enter systemd-sysext.
If you are familiar with ostree-based operating systems, you probably used ostree admin unlock at some point. systemd-sysext is another take on the same concept. You build your software as usual with a /usr (or /opt) prefix and install it in a special folder along with some metadata. Then upon running systemd-sysext merge, systemd creates an ovelayfs with the merged contents of /usr and your directory, and replaces the existing /usr mount atomically.
The format is very simple, there are 2 directories we care about for our usecase:
/run/extensions
/var/lib/extensions
/run/extensions is a temporary directory that’s wiped on reboot, so it’s excellent for experimental changes that might take down the system. After a power cycle you will boot back in a clean slate.
/var/lib/extensions is for persistent changes. Say you are experimenting with UI changes or want to thoroughly test a patch set for a couple days. Or you might simply want to have a local FFmpeg build with extra codecs, cause god lawyer forbid we have working video playback.
If you are installing a build in /run/extensions only thing you need to do is the following two commands:
This installs our tree into the custom_install directory and tells systext to refresh, which means that it will look at all extensions we might have, unmerge (unmount) them and then merge the updated contents again. Congrats, you can launch your new binaries now.
Normally sysext will check if the extension is compatible with the host operating system. This is done by using a metadata file that includes the ID= field you’d find in /etc/os-release (see more at the systemd-sysext manpage). The --force argument ignores this, as we build the project on host and we can be reasonably sure things will work.
If we want to have our build to be persistent and available across reboots, we have to install it in /var/lib/extensions/ and create the metadata file so systemd-sysext will mount it automatically upon boot. It’s rather simple to do but gets repetitive after a while and hunting in your console history is never fun. Here is a simple script that will take care of it.
#! /bin/bash
set-euEXTENSION_NAME="custom_install"# DESTDIR="/var/lib/extensions/$EXTENSION_NAME"DESTDIR="/run/extensions/$EXTENSION_NAME"VERSION_FILE="$DESTDIR/usr/lib/extension-release.d/extension-release.$EXTENSION_NAME"sudo mkdir -p$DESTDIR# rm -rf _build # meson setup _build --prefix=/usr# meson compile -C _build sudo meson install --destdir=$DESTDIR-C _build --no-rebuildsudo mkdir -p$DESTDIR/usr/lib/extension-release.d/# ID=_any to ignore it completlyecho ID=org.gnome.gnomeos |sudo tee $VERSION_FILEsudo systemd-sysext refreshsudo systemd-sysext list
Here’s a demo, I used sysprof as the example since it’s more visible change than my gnome-session MR. You can also test gnome-shell the same way by installing, refreshing and then logging out and login in again.
Anothe example was from today, where I was bisecting gdm. Before systemd-sysext, I’d be building complete images with different commits of gdm in order to bisect. It was still fast, at ~25m per build for the whole OS, but that’s still 24 minutes more after it becomes annoying.
Now, I switched to the gdm checkout, started a bisect, compiled, installed and then run systemctl restart gdm.service. The login greeter would either come up and I’d continue the bisect, or it would be blank at which point I’d ssh in, switch to a tty or even hit the power button and continue knowing it’s a bad commit. Repeat. Bisect done in 10 minutes.
And the best is that we can keep updating the operating system image uninterrupted, and on next boot the trees will get merged again. Want to go back? Simply systemd-sysext unmerge or remove the extension directories!
One caveat when using systemd-sysext is that you might occasionally need to systemctl daemon-reload. Another one when using custom DESTDIRs, is that meson won’t run post-install/integration commands for you (nor would it work), so if you need to recompile glib schemas, you will have to first systemd-sysext refresh, compile the schemas, place the new binary in the extension or make a new extension, and systemd-sysext refresh again.
Another use case I plan on exploring in the near future, is generating systemd-sysext images in the CI for Merge Requests, same way we generate Flatpak Bundles for applications. This proved to be really useful for people wanting to tests apps in an easier way. Begone shall be the days where we had to teach designers how to setup JHBuild in order to test UI changes in the Shell. Just grab the disk image, drop it in GNOME OS, refresh and you are done!
And that’s not all, none of this is specific to GNOME OS, other than having bleeding edge versions of all the gnome components that is! You can use systemd-sysext the same way in Fedora Workstation, Arch, Elementary OS etc. The only requirement is having recent enough systemd and a merged /usr tree. Next time you are about to meson install on your host system, give systemd-sysext a try!
This whole post is basically a retelling of Lennart’s blogpost about systemd-sysext, It has more details and you should check it out. This is also how I initially found out about this awesome tool! I tried to get people hooked on it in the past but it didn’t bear fruit, so here’s one post specific to GNOME development!
Last time I wrote we talked about a new search index for libgweather. In the end I decided to take another route so that we can improve application performance without any changes. Instead, I added a kdtree to do nearest neighbor search when deserializing GWeatherLocation. That code path was looking for the nearest city from a latitude/longitude in degrees.
The merge request indexes some 10,000 lat/lon points in radians at startup into a kd-tree. When deserializing it can find nearest city without the need for a linear scan. Maybe this is enough to allow significantly more data into the database someday so my small hometown can be represented.
Nautilus
I found a peculiarity in that I was seeing a lot of gtk_init() calls while profiling search. That means processes are being spawned. Since I have D-Bus session capture in Sysprof now, I was able to find this being caused by Nautilus sending an org.freedesktop.DBus.Peer.Ping() RPC to kgx and gnome-disks.
Seems like a reasonable way to find out if a program exists, but it does result in those applications being spawned. gtk_init() can take about 2% CPU alone and combined this is now 4-5% that is unnecessary. Nautilus itself needing to initialize GTK on startup plus these two as a casualty puts us combined over 6%.
D-Bus has an org.freedesktop.DBus.ListActivatableNames() RPC which will tell you what names are activatable. Same outcome, less work.
I was hesitant to look at gnome-photos because I think it’s falling out of core this next cycle. But since it’s on my system now, I want to take a look.
First off, I was seeing about 10 instances of gnome-photos in a single Sysprof capture. That means that the process was spawned 10 times in response to search queries. In other words, it either must have crashed or exited after each search request I typed into GNOME Shell. Thankfully, it was the later. But since we know gtk_init() alone is about 2%, and combined this looks like it’s in the 30-40% range there must be more to it.
So, easy fix. Call g_application_set_inactivity_timeout() with something reasonable. The search provider was already doing the right thing in calling g_application_hold() and g_application_release() to extend the process lifetime. But without inactivity-timeout set, it doesn’t help much.
After that, we’re down to just one instance. Cool.
Next capture we see we’re still at a few percent, which means something beyond just gtk_init() is getting called. Looks like it’s spending a bunch of time in gegl_init(). Surely we don’t need GEGL to provide search results (which come from tracker anyway), so make a quick patch to defer that until the first window is created. That generally won’t happen when just doing Shell queries, so it disappears from profiles now.
Calculator
Rarely do I have gnome-calculator actually running when performing a Shell search despite using it a lot. That means it too needs to be spawned to perform the search.
It’s already doing the right thing in having a dedicated gnome-calculator-search-provider binary that is separate from the application (so you can reduce start-up time and memory usage) so why is it showing up on profiles? Looks like it’s initializing GTK even though that isn’t used at all in providing search results. Probably a vestige of yesteryear.
Easy fix by just removing gtk_init(). Save time connecting to the display server, setting up seats, icon themes, and most importantly, parsing unused CSS.
Another couple percent saved.
Measure, Measure, Measure
Anyway, in conclusion, I might leave you with this tidbit. Nobody gets code right on the first try, especially me. If you don’t take a look and observe it, my guess is that it looks a lot different at run-time than it does in your head.
Sysprof is my attempt to make that a painless process.
Workbench gained the ability to save and load on-disk projects. In addition, we’ve made it easier to contribute to the library of demos and examples, checkout the contributing guide.
I just finished rewriting entirely my Markdown editor, MibiMdEditor, and released the first beta version of it (since I’ve rewritten it), but it is not just a random Markdown editor, it works by adding scripts that will be called to generate the HTML code, so you can use it to write MarkDown, BBcode, Asciidoc, just by coding a simple script! I’ll make a Flatpak soon!
Stay tuned!
In this release we’ve added select and deselect all buttons to a download’s playlist page and added a proxy url setting in Preferences. We’ve also added a network connection monitor to Parabolic. On the side of bugs, we fixed an issue where docs were not available when running via snap and an issue where History wasn’t saved correctly in json.
Here’s the full changelog:
Added buttons to select or deselect all items in a playlist
Added a proxy setting in Preferences
Parabolic will now check for an active network connection
Fixed an issue where docs were not available when running Parabolic via snap
Fixed an issue where History wasn’t saved correctly in json
Updated translations (Thanks everyone on Weblate!)
I have been working on a new media player named Daikhan (previously Envision Media Player) for some time. I have now released an early access version as a flatpak on Flathub.
Unfortunately, the flatpak does not yet support hardware accelerated playback. The player itself however does support it if GStreamer is set up properly.
Due to the early access nature of the app, there may be some other issues and there will be significant changes occurring in the future.
This release is packed with new features! We added a new drawing mode and the ability to set a background image in the window. We’ve also added a CLI to Cavalier in which you can change its settings right from the terminal!
Here’s the full changelog:
Cavalier can now be controlled from command line. Run the app with –help option to see full list of available options.
Reverse mirror option is now available with full mirror.
It’s now possible to set frames per second to 144 or other custom value.
Added anti-aliasing, so rounded items now look less pixelated.
Added ability to set background image.
New drawing mode - Splitter.
Bars limit was increased to 100.
Updated translations (Thanks everyone on Weblate!)
I released Solaar Extension for GNOME https://extensions.gnome.org/extension/6162/solaar-extension/ , this allows Solaar to support features on Wayland desktops which were previously unsupported, namely the process and mouse process conditions, which allow specific actions to happen on events from Logi keyboards and mice based on which application is currently in focus or currently has the mouse over it respectively. These should start working automatically on Wayland desktops with this extension installed with the next update of Solaar, or you can try the building the latest code from Solaar’s repository or from git or nightly releases if available for your distro.
At this year’s GUADEC, I made a proposal: that the GNOME project should migrate its documentation from wiki.gnome.org to other locations, and then disable its wiki. This blog post describes the proposal in more detail, in order to receive feedback from the wider community.
Background
For many years, the GNOME wiki (a MoinMoin instance) has been the main source of internal project documentation. It contains key project information, including guidance for newcomers, the release schedule, pages for the various teams, and much more.
Today, much of the content on the wiki is out of date, and a large portion of it isn’t actively updated. Many activities have moved from the wiki to other platforms (particularly Gitlab), leaving the related wiki content to rot. There have also been problems with spam, which have meant that edit rights have to be manually granted.
This situation inevitably causes issues, particularly for new contributors who are trying to learn how to get involved.
The proposal
The proposal is for us to replace the wiki with a combination of alternative platforms. The main alternatives would be:
gnome.org, for small amounts of important high-level information
In-tree documentation, for individual project docs
Gitlab issues in special-purpose projects, for more transient documentation
A new project handbook site, for general project documentation (contribution guide, code of conduct, governance, release process, and so on)
The new handbook site would be hosted using Gitlab pages and generated with Sphinx. We already do this for developer.gnome.org and the HIG, so the new site would not be unique in GNOME.
We started using Sphinx for docs in GNOME after evaluating a variety of alternative solutions. It’s not the newest or trendiest docs tool, but it has some key features which made it a good choice for us:
It’s a drop in tool which is both mature and well-maintained
It generates static HTML, which makes deployment and hosting easy with CI and Gitlab pages
Docs content is written using a light weight markup language
We can easily restyle the docs sites to match GNOME branding
Our experience so far has been positive: Sphnix-based sites are easy to set up, deploy, host and edit.
In terms of editing experience, you can make quick changes using Gitlab’s web-based editing tools or, for a richer experience, VS Code will give you live previews and error checking.
Transition process
My suggestion for retiring the wiki would be to have a managed transition, which would look something like this:
Create a migration guide for how and where to move existing wiki content
Announce the transition period (including an end date)
Transition! I’d finish the handbook, and other teams and individuals would be responsible for migrating their own content
Finish the transition: archive the wiki, and update any links to point to the new docs locations
Archiving the wiki could take various forms depending on what is most practical. We could put it in read-only mode and add a banner, like we did with Bugzilla, or we could create a static archive and take it offline completely. I’ll leave that question to our sysadmins.
Pros and cons
I’m definitely not suggesting that this proposal is perfect. Nor am I suggesting that there aren’t other alternatives to consider (it would be great to hear any other ideas)! So, in the interests of transparency, here are the main pros and cons of the proposal.
Advantages of the new handbook site:
It’s a nice website with good styling, clear navigation, and search
Having a clear scope and purpose as the “GNOME Project Handbook” is very helpful for both documentation users and authors
Having the content be maintained would (hopefully) result in higher overall quality
The site would use a contribution workflow that is consistent with other GNOME Gitlab projects
And the disadvantages of the general proposal:
There is more overhead required to make changes to docs stored in Git, compared to a wiki
Making changes via Git requires more technical knowledge (though we can potentially minimize this with the Gitlab web IDE and good documentation)
The new handbook site would run into problems if it were to become unmaintained
Docs that aren’t hosted on the handbook site could be fragmented and hard to find
Feedback wanted!
This proposal is just that: a proposal. We can and should consider alternatives. It also needs to be elaborated to cover all the requirements GNOME has today. If you have any concerns, suggestions, or uncertainties, please let me know.
Thanks!
¹ With restructured text, you get more powerful linking features, as well as internal link validation.
² All you really need to know is the format for an internal and external link, and you’re good to go.
A lot of things happened these past few weeks. First, the MR to add address and system DBus recording has already been merged. Though, before that was merged, a few revisions have been made: it now parses from messages bytes directly, instead of a PCAP stream, to prevent overhead. Additionally, it now properly propagates errors, such as when the dbus-monitor binary is not found.
Second, when recording, the sequence diagram now sticks at the bottom because, in common use cases, it is mostly wanted to observe the new messages as they come.
Lastly, the signals and method call arrows and arcs are now (finally!) drawn in the diagram. While it looks simple, it took me days to get it right. However, there is no merge request yet since it requires more bug fixing and code cleanup.
GUADEC
Aside from working on the code, I also attended GUADEC remotely. I had a lightning talk about a bit about myself and an overview of this project.
Plans for the Following Weeks
For the next weeks, the diagram still needs bug fixing and polishing, as mentioned. For instance, it currently does not draw arcs on method calls when either the call or return row is not visible. The header is also only visible when scrolled all the way to the top, and it only shows the unique name instead of the well-known name.
After the diagram is completed, we will go on with the implementation of service filtering and adding a more convenient way to have multiple instances of recording or opened files via tabs.
Where we left, the workspace tree view didn't display the hierarchy.
First, I discovered bug in the SQL triggers for folder path update. I
was missing the AFTER keyword to have it run after the update.
Workspace
Second, I need to specify a parent when adding items to the model,
then locate the parent, add to it. The problem is that we get a list
of folders in some order (I can order by any of the SQL columns). The
problem is adding an item to a parent that is not yet int the
model. There are few ways to go about it.
Sort the folder list in order of a flattened tree. To be fair my
SQL skills are not good enough, and I'm not even sure I can do that
in sqlite. This would solve most of the problems.
Sort the folder list once received. But this might not always work.
Handle the case in the model: add with a placeholder parent and
re-parent as needed when the parent is added.
For now I chose 3. The risk is that if the parent doesn't exist then
the tree will stay lingering. It's just the view though.
Here is the result:
Tree view cleanup
Back in November, I re-implemented the workspace tree view using
gtk4::ListView. But it has
some issues, due to limitations in the widget implementation, with the
way the expander (the arrow to expand a sub-tree) is handled. The
ListView expect to know whether it can be expanded when it is
created in the factory. Either there is a model and it can be
expanded, or there is no model, and it can't be. The problem is that
if you don't want to have an expander because there is no children,
you can't. Not with the API.
Can we hide the expander? The answer is yes. Simply in CSS set the
icon source to none.
This CSS selector grab just any expander in a list view. Now we need to
do it as needed.
First give the list view a name: workspace. Then we'll
programatically set a style for the row item when it doesn't have any
children. We'll use nochildren, and add/remove the CSS class as
needed. The bind and unbind are a good place.
I suspect there might be a few update problems, likely when adding or
removing items. We'll see. I will need to implement some testing at
one point, but that's a topic for another day.
Also I fixed the display of item counts that didn't work with deeper
levels of the tree. This was a long standing issue logic issue
introduced with the move to GtkListView.
Build
Using the GNOME templates, I added CI so that we get builds and check
automatically,
I also removed some no longer used C++ code.
Other
I then diverted my efforts into libopenraw trying to wrap up the Rust
port by finishing the C API. I'll have a detailed post about that
soon.
Due to my GSoC internship, I was given the opportunity to attend GUADEC 2023 in Riga, Latvia. First of all, I am very grateful for giving me this possibility and want to give my sincere thanks.
GUADEC 2023 allowed me first and foremost connect with other members of the GNOME and FOSS community. It was very nice meeting everyone and having the opportunity to exchange opinions and thoughts, as well as getting to know the people behind their online persona.
I've attended GUADEC alongside my GSoC Mentor, Cogitri, and we did manage a lot of progress on our GSoC project, FlatSync, but more on that later.
I do want to encourage all of you to check out the talks if you haven't done so already, the live streams are all up on YouTube, basically all of them have timestamps if you're only interested in a couple of talks. Personally, my highlight was the GNOME Design talk, and more specifically the window management part. I'm very interested in how the mixture of tiling and floating modes will come out in the end, and the improvements to edge tiling look very nice as well (imo. Windows currently has the edge over GNOME on this, and these changes look to be very promising in overcoming this gap). Furthermore, I like the idea of putting maximized windows onto a new Workspace a lot, as this is what I'm currently doing manually most of the time anyway. Also, at the end of GUADEC, we had GNOME intern lightning talks, you can check those out here. I also got a few responses regarding FlatSync, and so far the feedback has been positive, so I'm glad others are looking forward to FlatSync's release and growth as well! : )
Lastly, if I piqued your interest in the GUADEC, next year will be in Denver, Colorado, though if the US is a bit too far for you, there will also be a "mini-GUADEC" in Berlin, Germany.
My last FlatSync dates back quite a while, this is mainly due to me and my mentor being stuck fighting a nasty limitation with manipulating Flatpak installations from within a Flatpak sandbox (which is where the FlatSync daemon lives in). Basically, this is currently not possible without workarounds. Luckily though, there's been a discussion around this here, and we were able to use some of the workarounds from the Souk project for use with FlatSync. We were thus able to continue development and installing FlatSync's daemon as a Flatpak as well. Otherwise, we'd have had to require users to either install all sub-parts of the project (CLI, GUI, daemon) or the daemon exclusively natively to the system instead of getting it bundled as a Flatpak.
Currently, FlatSync is now able to differentiate between local and remote states and to properly diff and prioritize between the two by using timestamps indicating when the installation list was last changed.
If the remote state differs from the local state and is newer, missing applications are installed on the target systems (same goes for remotes), and locally installed applications no longer residing within the remote are respectively uninstalled.
There's also been a lot of work behind the scenes, cleaning up the codebase and preparing it for easy adaptation of additional API providers.
Though the progress is summed up pretty easily, it's a rather big milestone for the project. The daemon's (very) basic functionality is now complete, and you can test the project already if you're interested (and not intimidated by CLI), though please be aware that it's still very much in early development and stable or ready for everyday use at all.
As for the next steps, we want to move from polling for local changes to using libflatpak's monitoring capabilities and listen for local changes that way. Furthermore, we plan on respecting monitored networking, thus holding back remote synchronization until the user is connected to a non-metered connection again.
Furthermore, we plan on getting a basic GUI ready to accompany the CLI for the setup and configuration of the daemon.
We do have some issues open on the project page, both within and after GSoC's timespan, and we also got some inspiration for extended functionality, most notably from Flatseal's maintainer to also synchronize Flatpak overrides.
Other than that, we plan on making the synchronization process more verbose and interactive (e.g. pausing synchronization, only respecting user- or system-wide installations, splitting between hostnames).
I’m on my way back from GUADEC in Rīga, Latvia and one recurring discussion (of many) was centered around how ratings and reviews are presented in GNOME software and other app stores.
I’ve been building open app ecosystems for over a decade now. Ratings (quantitative feedback) and reviews (qualitative feedback) of apps can serve useful purposes: a powerful signal for ranking, a way to provide feedback to a developer, a way to demonstrate “social proof” (like testimonials or positive social media posts on a website), and a heads-up to other users for positive or negative experiences. But the widely-used five-star and review system has its problems.
Since I’m sitting in an airport killing time while waiting to my flight back to Denver (where GUADEC 2024 is happening, if you haven’t heard!), let’s dig into this.
What’s wrong with five-star ratings?
The meaning of the five-star rating system changes depending on culture, and even at its best, is a skewed system. If you’ve ever ridden in a ride-share like Lyft, Uber, or Bolt, you’re expected to give a five-star review unless something went wrong—and at least as of 2017, drivers with an average rating of less than 4.6 stars can lose their job. This is a circular problem: as more people are pressured to give drivers in general five stars, the average rating raises, which in turn raises the threshold at which ride-sharing companies will fire the drivers.
There’s also the ambiguity: ask ten people what they would rate a product from an online retailer that they found “just fine,” and you’re likely to get at least three different answers: 2, 3, or 4 stars. If the spread is that wide, is it really a useful system? We’ve seen popular apps and platforms learn this over the years, too—and many have already abandoned the five-star rating system.
YouTube used five-star reviews until 2009, when they pointed out the issues across their growing platform: most people gave five stars, some gave one star, and basically nobody gave two or three stars. As a result, they switched to a like/dislike system.
Netflix used five stars until 2017, and replaced them with thumbs-up and thumbs-down—though for slightly different reasoning; they use the ratings as a signal for recommendations rather than showing those ratings to other users. But by lowering the friction of providing feedback, they increased the amount of feedback given by over 200%.
Specifically in the case of the Open Desktop Ratings Service (ODRS) which is used to display ratings and reviews in app stores like GNOME Software, I’ve personally heard from app developers that a five-star rating system seems ill-suited when there are few total ratings; if just a few people give the app a low rating, it looks like the app is low quality—and it’s hard to get people to update those ratings, even after any raised issues are fixed.
What’s wrong with written reviews?
Five-star ratings often come with free-form written reviews to explain why a specific rating was given, but these have their own problems, too.
Accepting written reviews can enable review bombing and harassment of developers—even for reasons unrelated to their app. This happens pretty regularly on Steam and app stores like Google Play. Think about it: soliciting written reviews is basically making a comments section. All of the problems we’ve learned with comments sections over the years apply here.
One solution could be having the app developer moderate their apps comments, but this too has problems: first, if a developer is being harassed, it’s not good for their mental health to make them be the one required to read every review. At the same time, if an app developer is ultimately in charge of what reviews are displayed, it creates an incentive to hide any negative reviews—potentially defeating the purpose of them in the first place. As a result, it should be a requirement to have a neutral third party screen reviews—and that’s very hard to scale properly, especially as an ecosystem grows.
After a review is screened by some neutral party, you very much do want the developer to be notified about it—otherwise it’s a post to other potential users without giving the developer the opportunity to improve. If people are going to spend their precious time writing a review, it should be as easy as possible for the developer of that app to actually receive the feedback!
Especially if you’re thinking from the context of a relatively small or largely volunteer-driven app ecosystem, it seems like reviews are a very difficult problem that is unsolved by some of the largest platforms at best, and a net negative on the ecosystem at worst.
What would be better?
I’ve long felt that the issues with the five-star rating plus review system could be remedied by using a simpler setup. So what’s the alternative?
My first suggestion is to use a simpler signal for ratings: something like a thumbs-up or heart icon to signal that someone likes the app. Combined with a tally of how many downloads there have been, this can give you relative rankings of apps with very little interaction overhead. If many people install an app and nobody gives it a thumbs-up, it probably shouldn’t rank as highly as an app that has a medium number of installs but a very high thumbs-up:installs ratio.
If you want to get more granular feedback, I still feel like something we discussed back in 2017 at an elementary sprint could work: quantitative feedback in predetermined categories, e.g. binary options for “fast,” “useful,” or “works well.”
I personally would avoid written reviews completely. Each time an update is released, it could render all reviews irrelevant, or it could be an entirely non-visible change—any there’s no reliable way to know that from an app store point of view. A specific bug behind a one-star rating could have been fixed, yet that one-star rating will likely stay on the app’s listing in perpetuity—especially if it’s difficult or impossible for a user to update that rating if they’ve reinstalled their OS or just never visit the app’s listing again. Encouraging users to revisit old ratings could be possible in the context of the same OS installation or if there is an account system, but that introduces a ton of complexity to a concept (written reviews) that even at its best has issues of review bombing, moderation, and localization.
An idea I’ve floated with mixed response has been to wire a thumbs-down to actually just open the AppStream MetaInfo issue reporting URL as a way to encourage people to directly file issues rather than leaving feedback somewhere simultaneously more public to users and less-viewed by the developers. However, this could seem a bit awkward or technical, so I’m not sure it’s a great idea—and there is already a “report an issue” item in GNOME Software today if that metadata is provided by the developer.
What about the ODRS?
To be clear, I’m immensely impressed by and grateful for all the work that has gone into ODRS so far: Richard Hughes has done an incredibly thoughtful and admirable job at building something nobody else was doing in the open desktop space—that’s kind of his thing! At the same time, it seems like the traction behind ODRS—especially when it comes to scaling moderation and inspiring outside contributions—has been lower than would be sustainable. So… let’s iterate and improve!
We have several interested parties in the ecosystem: GNOME Core and Circle apps, KDE apps, Flathub as an app store, cross-platform apps targeting Flatpak, elementary AppCenter and third-party apps, and downstream distros benefiting from the app ecosystem and repackaging apps for their own repos like Fedora, Debian, Ubuntu, Pop!_OS, and many more.
Perhaps it’s time to reinvigorate ODRS with the kind of cross-desktop collaboration that makes our open ecosystem as a whole stronger: the beautiful coopertition of working together on shared interests and technologies while striving to make what we each think is the best experience possible. I’m chatting with Richard and some of those folks mentioned to see how we can best direct this energy, and am looking forward to helping however I can—and if you’re interested in joining the effort, let me know!
Thank you for coming to the the 1st Contribution Hackathon at GUADEC 2023. As the name implies, in this event, we’ll teach you how to make your first code contribution to GNOME. Buckle up!
Setup
There are three components that we need for making the contribution: flatpak, flathub, and GNOME Builder. Here’s how to get each of them:
Flatpak
Flatpak is a packaging format for graphical applications that works on any GNU/Linux distribution. It is the most widely embraced packaging format for GNOME applications. To get flatpak on your distro of choice, run:
Flatpak é um formato de empacotamento de aplicações gráficas, que funciona em qualquer distribuição de GNU/Linux (Fedora, Ubuntu, Arch, etc). Esse é o formato escolhido pelo Projeto GNOME para publicar seus pacotes, então o usaremos para instalar os aplicativos e suas dependências.
Ubuntu:
sudo apt install flatpak
Arch:
sudo pacman -S flatpak
Fedora:
# already installed
Flathub
Flathub is an app store for the flatpak packaging format. It’s one of the biggest software catalogs in the Linux world! You can get it on your computer by running:
Finally, GNOME Builder is an integrated development environment (IDE) for developing applications in the GNOME ecosystem. Install it with:
flatpak install org.gnome.Builder
And you’re all set!
What is GNOME?
GNOME is a software platform. That means GNOME provides an integrated stack for software development, distribution, and consumption.
An important part of this ecosystem are the GNOME apps, to which we’ll contribute in this event! These apps are built with GTK, using GNOME Human Interface Guidelines (HIG).
What is GTK?
In programmer’s lingo: it is an object oriented widgets-based framework for developing graphical applications. Let’s break that down!
Object oriented:
Object Oriented Programming is a programming paradigm that seeks to improve code organization, reduce code duplication, among many other things. Object Oriented Programming is very “concept heavy”, so teaching everything about it is out of scope for this event. Still, we hope to spark your curiosity enough for you to look it up on your own later :)
Widgets-based:
In GTK, every graphical element is a “widget”. These widgets are organized hierarchically, for example Window → Header bar → Menu button. Here are some of the widgets and their names, in the wonderful Frog application:
Widgets’ behaviur and appearance can be configured with a programming language (C, Python, Rust, Javascript, Vala, etc) and/or a markup language (XML, Blueprint, etc).
Graphical applications:
Graphical applications are applications that show a window to the user. Some applications are not graphical, for example, that ones that run directly on the terminal.
Access Keys
What?
GTK has a feature called Access Keys (also known as “mnemonics”). Here’s how it works:
Hold the Alt key until some characters are underlined
Press an underlined characters while you hold Alt
The action associated with that widget will be triggered
Why?
Access Keys are an important for two use cases:
Offering a keyboard-centric workflow for power users
Improving the accessibility of the GNU/Linux desktop
How?
To implement an Access Key for a widget within a application, you need to prepend an underline (_) to whichever character you want to highlighted when the user hold the Alt key. For example, to highlight the C in button whose label is "Continue", go with "_Continue".
Then, activate the Access Key feature for that widget. To do that you need to set the property “user-underline” to True.
This can sometimes be done with programming or markup languages. For programming languages, you’ll find some examples in the official documentation. Luckly for you, we have also made some additional examples, which you can checkout in the Projects Sheet. Here are some highlights:
Example in XML
Contribution by Gustavo Pechta to the app “Komikku” (link).
Example in Python
Contribution by Tárik Sá to the app “What IP” (link).
Example in Javascript
Contribution by Pedro Sader Azevedo to the extension “Space Bar” (link).
GNOME Human Interface Guidelines
Now you might asking yourself: How to I decide which widgets to add mnemonics to? Which characters should I choose to underline?
The answer to all these questions (and plenty more) lie in the GNOME Human Interface Guidelines (HIG). This document describes, in a very detailed manner, directives for building graphical user interfaces that comply with the design standards of GNOME.
The HIG contains a section on Access Keys, that you should read thoroughly (it’s very short). When you’re back, here’s summary on how to implement Access Keys in GNOME apps:
As many as you can
Prioritize most used widgets
Prefer the first letter
Avoid thin letters (i)
Avoid ambiguous letters (I, l)
Avoid letters with descenders (g, y, q)
Avoid vowels (a, e, i, o, u)
Tip: view switchers are usually good candidates for Access Keys! They look like this:
Git
The most popular version control system is git. We’ll need to use its command line interface (CLI) to make our contribution but, unfortunately, git is a very complex and convoluted tool. Teaching all about it would take a very long (probably a workshop of its own). If want a deeper introduction to the concepts of git you can read “The tip of the git iceberg”, by yours truly 😉.
That being said, we’ll only give you what you need for this workshop:
Download the source code of the application
git clone https://...
Create a new branch for your contribution
git checkout -b add-access-keys
See commit history to check for conventions
git log
Create a new version of the application
git commit
Send your branch to your GitHub/GitLab account
git push origin add-access-keys
Warning: We’ll contribute to projects that are hosted on GitHub and GitLab. Although there’s only link for GitHub (github.com), there are multiple links to GitLab (gitlab.com, gitlab.gnome.org, etc). Make sure you create your account using the same GitLab link as your chosen project is hosted at.
After you’ve pushed your changes to your GitHub/GitLab account, you have to request that the maintainers of the project accept your contribution. This is known as a “merge request” (MR) or a “pull request” (PR).
When either of these git platforms detect a git push coming from you, they will offer you to create an MR/PR. Accept the offer and write a detailed description of what your contribution does. Again, you can check the examples in the Project Sheet.
Hackathon
Finally, you’re all set! Follow the steps below to make your first code contribution to GNOME and don’t hesitate to ask any questions.
Pick a project and write down your name in the sheet
Find the project’s repository in its Flathub page
Create an account and access token at GitHub/GitLab
For the the project and clone your fork
Build once without any modifications
Implement the Access Keys
Check the commit conventions
Commit, push, and PR/MR
Put the PR/MR link in the sheet
Wait for feedback
Attribution
This blog post is a slightly re-written translation of this blog post by LKCAMP, which was written by myself and Tárik Sá.
tl;dr: Passim is a local caching server that uses mDNS to advertise files by their SHA-256 hash. Named after the Latin word for “here, there and everywhere” it might save a lot of people a lot of money.
Introduction
Much of the software running on your computer that connects to other systems over the Internet needs to periodically download metadata or other information needed to perform other requests.
As part of running the passim/LVFS projects I’ve seen how download this “small” file once per 24h turns into tens of millions of requests per day — which is about ~10TB of bandwidth! Everybody downloads the same file from a CDN, and although a CDN is not super-expensive, it’s certainly not free. Everybody on your local network (perhaps dozens of users in an office) has to download the same 1MB blob of metadata from a CDN over a perhaps-non-free shared internet link.
What if we could download the file from the Internet CDN on one machine, and the next machine on the local network that needs it instead downloads it from the first machine? We could put a limit on the number of times it can be shared, and the maximum age so that we don’t store yesterdays metadata forever, and so that we don’t turn a ThinkPad X220 into a machine distributing 1Gb/s to every other machine in the office. We could cut the CDN traffic by at least one order of magnitude, but possibly much more. This is better for the person paying the cloud bill, the person paying for the internet connection, and the planet as a whole.
This is what Passim might be. You add automatically or manually add files to the daemon which stores them in /var/lib/passim/data with xattrs set on each file for the max-age and share-limit. When the file has been shared more than the share limit number of times, or is older than the max age it is deleted and not advertised to other clients.
The daemon then advertises the availability of the file as a mDNS service subtype and provides a tiny single-threaded HTTP v1.0 server that supplies the file over HTTP. Note, I said HTTP there – not HTTPS – i.e. so there’s no SSL or encryption involved at all. It’s also not HTTP v1.1, so there’s no pipelining or compression and the server can be super simple.
The file is sent when requested from a URL like http://192.168.1.1:27500/filename.xml.gz?sha256=the_hash_value – any file requested without the checksum will not be supplied. Although this is a chicken-and-egg problem where you don’t know the payload checksum until you’ve checked the remote server, this is solved using a tiny <100 byte request to the CDN for the payload checksum (or a .jcat file) and then the multi-megabyte (or multi-gigabyte!) payload can be found using mDNS. Using a Jcat file also means you know the PKCS#7/GPG signature of the thing you’re trying to request. Using a Metalink request would work as well I think.
Sharing Considerations
Here we’ve assuming your local network (aka LAN) is a nice and friendly place, without evil people trying to overwhelm your system or feed you fake files. Although we request files by their hash (and thus can detect tampering) and we hopefully also use a signature, it still uses resources to send a file over the network.
We’ll assume that any network with working mDNS (as implemented in Avahi) is good enough to get metadata from other peers. If Avahi is not running, or mDNS is turned off on the firewall then no files will be shared.
The cached index is shared with all users without any kind of authentication — both over mDNS and as a HTTP page on http://localhost:27500/.
Currently only processes running as UID 0 (a.k.a. root) can publish content to Passim. Before sharing everything, the effects of sharing can be subtle; if you download a security update for a Lenovo P1 Gen 3 laptop and share it with other laptops on your LAN — it also tells any attacker [with a list of all possible firmware updates] on your local network your laptop model and also that you’re running a system firmware that isn’t currently patched against the latest firmware bug.
My recommendation here is only to advertise files that are common to all machines. For instance:
AdBlocker metadata
Firmware update metadata
Remote metadata for update frameworks, e.g. apt-get/dnf etc.
Implementation Considerations
Any client MUST calculate the checksum of the supplied file and verify that it matches. There is no authentication or signing verification done so this step is non-optional. A malicious server could advertise the hash of firmware.xml.gz but actually supply evil-payload.exe — and you do not want that.
Comparisons
The obvious comparison to make is IPFS. I’ll try to make this as fair as possible, although I’m obviously somewhat biased.
IPFS
Existing project that’s existed for many years tested by many people
Allows sharing with other users not on your local network
Not packaged in any distributions and not trivial to install correctly
Requires a significant time to find resources
Does not prioritize local clients over remote clients
Requires a internet-to-IPFS “gateway” which cost me a lot of $$$ for a large number of files
Passim
New project that’s not even finished, with zero security review
Only allowed sharing with computers on your local network
Returns results within 2s
One concern we had specifically with IPFS for firmware were ITAR/EAR legal considerations. e.g. we couldn’t share firmware containing strong encryption with users in some countries — which is actually most of the firmware the LVFS distributes. From an ITAR/EAR point of view Passim would be compliant (as it only shares locally, presumably in the same country) and IPFS certainly is not.
There’s a longer README in the git repo. There’s also a test patch that wires up fwupd with libpassim although it’s not ready for merging. For instance, I think it’s perfectly safe to share metadata but not firmware or distro package payloads – but for some people downloading payloads on a cellular link might be exactly what they want – so it’ll be configurable. For reference Windows Update also shares content (not just metadata) so maybe I’m worrying about nothing, and doing a distro upgrade from the computer next to them is exactly what people need. Small steps perhaps.
TL;DR: The gnome-shell-sass repository is no longer getting updated upstream.
Background
As gnome-shell’s CSS grew more complex, designers needed something more expressive, so they started compiling the stylesheet from SASS. The sources were moved to a subproject, so they could be shared between the regular stylesheet and the GNOME Classic stylesheet.
Fast-forward to the present day, gnome-shell now includes a light variant to support the global color-scheme option.
GNOME Classic has been updated to use that built-in support instead of compiling a separate stylesheet on its own.
That means that the gnome-shell-sass repository is no longer needed by the gnome-shell-extensions project, and will therefore no longer be updated upstream.
If you are building your own theme from those sources, you will either have to get them from the gnome-shell repository yourself, or coordinate with other 3rd party theme authors to keep the subproject updated.
Window management is one of those areas I’m fascinated with because even after 50 years, nobody’s fully cracked it yet. Ever since the dawn of time we’ve relied on the window metaphor as the primary way of multitasking on the desktop. In this metaphor, each app can spawn one or more rectangular windows, which are stacked by most recently used, and moved or resized manually.
Overlapping windows can get messy quickly
The traditional windowing system works well as long as you only have a handful of small windows, but issues emerge as soon the number and size of the windows grows. As new windows are opened, existing ones are obscured, sometimes completely hiding them from view. Or, when you open a maximized window, suddenly every other window is hidden.
Over the decades, different OSes have added different tools and workflows to deal with these issues, including workspaces, taskbars, and switchers. However, the basic primitives have not changed since the 70s and, as a result, the issues have never gone away.
While most of us are used to this system and its quirks, that doesn’t mean it’s without problems. This is especially apparent when you do user research with people who are new to computing, including children and older people. Manually placing and sizing windows can be fiddly work, and requires close attention and precise motor control. It’s also what we jokingly refer to as shit work: it is work that the user has to do, which is generated by the system itself, and has no other purpose.
Most of the time you don’t care about exact window sizes and positions and just want to see the windows that you need for your current task. Often that’s just a single, maximized window. Sometimes it’s two or three windows next to each other. It’s incredibly rare that you need a dozen different overlapping windows. Yet this is what you end up with by default today, when you simply use the computer, opening apps as you need them. Messy is the default, and it’s up to you to clean it up.
What about tiling?
Traditional tiling window managers solve the hidden window problem by preventing windows from overlapping. While this works well in some cases, it falls short as a general replacement for stacked, floating windows. The first reason for this is that tiling window managers size windows according to the amount of available screen space, yet most apps are designed to be used at a certain size and aspect ratio. For example, chat apps are inherently narrow and end up having large amounts of empty space at large sizes. Similarly, reading a PDF in a tiny window is not fun.
GNOME 44 with the “Forge” tiling extension. Just because windows can be tall and narrow doesn’t mean they should be :)
Another issue with tiling window manager is that they place new windows in seemingly arbitrary positions. This is a consequence of them not having knowledge about the content of a window or the context in which it is being used, and leads to having to manually move or resize windows after the fact, which is exactly the kind of fiddling we want to avoid in the first place.
More constrained tiling window managers such as on iPadOS are interesting in that they’re more purposeful (you always intentionally create the tiling groups). However, this approach only allows tiling two windows side-by-side, and does not scale well to larger screens.
History
This topic has been of interest to the design team for a very long time. I remember discussing it with Jakub at my first GUADEC in 2017, and there have been countless discussions, ideas, and concepts since. Some particular milestones in our thinking were the concept work leading up to GNOME 40 in 2019 and 2020, and the design sessions at the Berlin Mini GUADEC in 2022 and the Brno hackfest in 2023.
Tiling BoF in Brno during the HDR hackfest. Left to right: Robert Mader, Marco Trevisan, Georges Stavracase, Jakub Steiner and Allan Day (remote), Florian Müllner, Jonas Dreßler
I personally have a bit of a tradition working on this problem for at least a few weeks per year. For example, during the first lockdown in 2020 I spent quite a bit of time trying to envision a tiling-first version of GNOME Shell.
GNOME has had basic tiling functionality since early in the GNOME 3 series. While this is nice to have, it has obvious limitations:
It’s completely manual
Only 2 windows are supported, and the current implementation is not extensible to more complex layouts
Tiled windows are not grouped in the window stack, so both windows are not raised simultaneously and other windows get in the way
Workspaces are manual, and not integrated into the workflow
Because tiled windows are currently mixed with overlapping floating windows they’re not really helping make things less messy in practice.
We’ve wanted more powerful tilingfor years, but there has not been much progress due to the huge amount of work involved on the technical side and the lack of a clear design direction we were happy with. We now finally feel like the design is at a stage where we can take concrete next steps towards making it happen, which is very exciting!
Get out of my way
The key point we keep coming back to with this work is that, if we do add a new kind of window management to GNOME, it needs to be good enough to be the default. We don’t want to add yet another manual opt-in tool that doesn’t solve the problems the majority of people face.
To do this we landed on a number of high level ideas:
Automatically do what people probably want, allow adjusting if needed
Make use of workspaces as a fully integrated part of the workflow
Richer metadata from apps to allow for better integration
Our current concept imagines windows having three potential layout states:
Mosaic, a new window management mode which combines the best parts of tiling and floating
Edge Tiling, i.e. windows splitting the screen edge-to-edge
Floating, the classic stacked windows model
Mosaic is the default behavior. You open a window, it opens centered on the screen at a size that makes the most sense for the app. For a web browser that might be maximized, for a weather app maybe only 700×500 pixels.
As you open more windows, the existing windows move aside to make room for the new ones. If a new window doesn’t fit (e.g. because it wants to be maximized) it moves to its own workspace. If the window layout comes close to filling the screen, the windows are automatically tiled.
You can also manually tile windows. If there’s enough space, other windows are left in a mosaic layout. However, if there’s not enough space for this mosaic layout, you’re prompted to pick another window to tile alongside.
You’re not limited to tiling just two windows side by side. Any tile (or the remaining space) can be split by dragging another window over it, and freely resized as the window minimum sizes allow.
There are always going to be cases that require placing a window in a specific position on the screen. The new system allows windows to be used with the classic floating behavior, on a layer above the mosaic/tiling windows. However, we think that this floating behaviour is going to be a relatively uncommon, similar to the existing “always on top” behavior that we have today.
There’s of course much more to this, but hopefully this gives an idea of what we have in mind in terms of behavior.
New window metadata
As mentioned above, to avoid the pitfalls of traditional tiling window managers we need more information from windows about their content. Windows can already set a fixed size and they have an implicit minimum size, but to build a great tiling experience we need more.
Some apps should probably never be maximized/tiled on a 4K monitor…
One important missing piece is having information on the maximum desired size of a window. This is the size beyond which the window content stops looking good. Not having this information is one of the reasons that traditional tiling window managers have issues, especially on larger screens. This maximum size would not be a hard limit and manual resizing would still be possible. Instead, the system would use the maximum size as one factor when it calculates an optimal window layout. For example, when tiling to the side of the screen, a window would only grow as wide as its maximum width rather than filling exactly half of the screen.
In addition, it’d be helpful to know the range of ideal sizes where an app works best. While an app may technically work at mobile sizes that’s probably not the best way to use that app if you have a large display. To stay with our chat example, you probably want to avoid folding the sidebar if it can be avoided, so the range of ideal sizes would be between the point where it becomes single pane and its maximum usable size.
Ideally these properties could be set dynamically depending on the window content. For example, a spreadsheet with a lot of columns but few rows could have a wider ideal size than one with lots of rows.
Depending on apps using new system APIs can be challenging and slow — it’s not easy to move the entire ecosystem! However, we think there’s a good chance of success in this case, due to the simplicity and universal usefulness of the API.
Next steps
At the Brno hackfest in April we had an initial discussion with GNOME Shell developers about many of the technical details. There is tentative agreement that we want to move in the direction outlined in this post, but there’s still a lot of work ahead.
On the design side, the biggest uncertainty is the mosaic behavior — it’s a novel approach to window management without much prior art. That’s exciting, but also makes it a bit risky to jump head-first into implementation. We’d like to do user research to validate some of our assumptions on different aspects of this, but it’s the kind of project that’s very difficult to test outside of an actual prototype that’s usable day to day.
If you’d like to get involved with this initiative, one great way to help out would be to work on an extension that implements (parts of) the mosaic behavior for testing and refining the interactions. If you’re interested in this, please reach out :)
There’s no timeline or roadmap at this stage, but it’s definitely 46+ material and likely to take multiple cycles. There are individual parts of this that could be worked on independently ahead of the more contingent pieces, for example tiling groups or new window metadata. Help in any of these areas would be appreciated.
This post is summarizing collaborative work over the past years by the entire design team (Allan Day, Jakub Steiner, Sam Hewitt, et al). In particular, thanks to Jakub for the awesome animations bringing the behaviors to life!
If you’re new to WebRTC, you must’ve heard that it’s a way to do video calls in your browser without needing to install an app. It’s pretty great!
However, it uses a bunch of really arcane terminology because it builds upon older technologies such as RTP, RTCP, SDP, ICE, STUN, etc. To understand what WebRTC Signalling is, you must first understand these foundational technologies.
Readers who are well-versed in this subject might find some of the explanations annoyingly simplistic to read. They will also notice that I am omitting a lot of of detail, leading to potentially misleading statements.
I apologize in advance to these people. I am merely trying to avoid turning this post into a book. If you find a sub-heading too simplistic, please feel free to skip it. :-)
RTP
Real-time Transport Protocol is a standardized way of taking video or audio data (media) and chopping it up into “packets” (you can literally think of them as packets / parcels) that are sent over the internet using UDP. The purpose is to try and deliver them to the destination as quickly as possible.
UDP (user datagram protocol) is a packet-based alternative to TCP (transmission control protocol), which is connection-based. So when you send something to a destination (IP address + port number), it will be delivered if possible but you have no protocol-level mechanism for finding out if it was received, unlike, say, TCP ACKs.
You can think of this as chucking parcels over a wall towards someone whom you can’t see or hear. A bunch of them will probably be lost, and you have no straightforward way to know how many were actually received.
UDP is used instead of TCP for a number of reasons, but the most important ones are:
TCP is designed for perfect delivery of all data, so networks will often try too hard to do that and use ridiculous amounts of buffering (sometimes 30 seconds or more!), which leads to latencies that are too large for two people to be able to talk over a call.
UDP doesn’t have that problem, but the trade-off is that it gives no guarantees of delivery at all!
You’d be right to wonder why nothing new has been created to be a mid-way point between these two extremes. The reason is that new transport protocols don’t get any uptake because existing systems on the Internet (operating systems, routers, switches, etc) don’t (want to) support them. See this list on Wikipedia.
Due to this, new protocols are just built on top of UDP and try to add mechanisms to detect packet loss and such. One such mechanism is…
RTCP
RTP Control Protocol refers to standardized messages sent by a media sender to all receivers, and also messages that are sent back by the receiver to the sender (feedback). As you might imagine, this message-passing system has been extended to do a lot of things, but the most important are:
Receivers use this to send feedback to the sender about how many packets were actually received, what the latency was, etc.
Senders send information about the stream to receivers using this, for instance to synchronize audio and video streams (also known as lipsync), to tell receivers that the stream has ended (a BYE message), etc.
Similar to RTP, these messages are also sent over UDP. You might ask “what if these are lost too”? Good question!
RTCP packets are sent at regular intervals, so you’d know if you missed one, and network routers and switches will prioritize RTCP packets over other data, so you’re unlikely to lose too many in a row unless there was a complete loss of connectivity.
Peer
WebRTC is often called a “peer-to-peer” (P2P) protocol. You might’ve heard that phrase in a different context: P2P file transfer, such as Bittorrent.
The word “peer” contrasts with “server-client” architectures, in which “client” computers can only talk to (or via) “server” computers, not directly to each other.
We can contrast server-client architecture with peer-to-peer using a real-world example:
If you send a letter to your friend using a postal service, that’s a server-client architecture.
If you leave the letter in your friend’s mailbox yourself, that’s peer-to-peer.
But what if you don’t know what kind of messages the recipient can receive or understand? For that we have…
SDP
Stands for Session Description Protocol which is a standardized message format to tell the other side the following:
Whether you want to send and/or receive, audio and/or video
How many streams of audio and/or video you want to send / receive
What formats you can send or receive, for audio and/or video
This is called an “offer”. Then the other peer uses the same message format to reply with the same information, which is called an “answer”.
This constitutes media “negotiation”, also called “SDP exchange”. One side sends an “offer” SDP, the other side replies with an “answer” SDP, and now both sides know what to do.
As you might expect, there’s a bunch of other technical details here, and you can know all about them at this excellent page that explains every little detail. It even explains the format for ICE messages! Which is…
ICE
Interactive Connectivity Establishment is a standardized mechanism for peers to tell each other how to transmit and receive UDP packets. The simplest way to think of it is that it’s just a list of IP address and port pairs.
Once both sides have successfully sent each other (“exchanged”) ICE messages, both sides know how to send RTP and RTCP packets to each other.
Why do we need IP address + port pairs to know how to send and receive packets? For that you need to understand…
How The Internet Works
If you’re connected to the internet, you always have an IP address. That’s usually something like 192.168.1.150 – a private address that is specific to your local (home) network and has no meaning outside of that. Having someone’s private IP address is basically like having just their house number but no other parts of their address, like the street or the city.
Most personal devices (computer or phone or whatever) with access to the Internet don’t actually have a public IP address. Picking up the analogy from earlier, a public IP address is the internet equivalent of a full address with a house number, street address, pin code, country.
When you want to connect to (visit) a website, your device actually talks to an ISP (internet service provider) router, which will then talk to the web server on your behalf and ask it for the data (website in this case) that you requested. This process of packet-hopping is called “routing” of network packets.
This ISP router with a public address is called a NAT (Network Address Translator). Like the name suggests, its job is to translate the addresses embedded in packets sent to it from public to private and vice-versa.
Let’s say you want to send a UDP packet to www.google.com. Your browser will resolve that domain to an IP address, say 142.250.205.228. Next, it needs a port to send that packet to, and both sides have to pre-agree on that port. Let’s pick 16789 for now.
Your device will then allocate a port on your device from which to send this packet, let’s say 11111. So the packet header looks a bit like this:
From
To
192.168.1.150:11111
142.250.205.228:16789
Your ISP’s NAT will intercept this packet, and it will replace your private address and port in the From field in the packet header to its own public address, say 169.13.42.111, and it will allocate a new sender port, say 22222:
From
To
169.13.42.111:22222
142.250.205.228:16789
Due to this, the web server never sees your private address, and all it can see is the public address of the NAT.
When the server wants to reply, it can send data back to the From address, and it can use the same port that it received the packet on:
From
To
142.250.205.228:16789
169.13.42.111:22222
The NAT remembers that this port 22222 was recently used for yourFrom address, and it will do the reverse of what it did before:
From
To
142.250.205.228:16789
192.168.1.150:11111
And that’s how packets are send and received by your phone, computer, tablet, whatever when talking to a server.
Since at least one side needs to have a public IP address for this to work, how can your phone send messages to your friend’s phone? Both only have private addresses.
Solution 1: Just Use A Server As A Relay
The simplest solution is to have a server in the middle that relays your messages. This is how all text messaging apps such as iMessage, WhatsApp, Instagram, Telegram, etc work.
You will need to buy a server with a public address, but that’s relatively cheap if you want to send small messages.
Bandwidth can get expensive very quickly, so you don’t want to always use a TURN server. But this is a fool-proof method to transmit data, so it’s used a backup.
Previously we saw how the NAT will remember the mapping between a “port on its public address” and “your device’s private address and port”. With many NATs, this actually works for any packet sent on that public port by anyone.
This means if a public server can be used to create such mappings on the NATs of both peers, then the two can send messages to each other from NAT-to-NAT without a relay server!
Let’s dig into this, and let’s substitute hard-to-follow IP addresses with simple names: AlicePhone, AliceNAT, BobPhone, BobNAT, and finally STUNServer:19302.
First, AlicePhone follows this sequence:
AlicePhone sends a STUN packet intended for STUNServer:19302 using UDP
From
To
AlicePhone:11111
STUNServer:19302
AliceNAT will intercept this and convert it to:
From
To
AliceNAT:22222
STUNServer:19302
When STUNServer receives this packet, it will know that if someone wants to send a packet to AlicePhone:11111, they could use AliceNAT:22222 as the To address. This is an example of an ICE candidate.
STUNServer will then send a packet back to AlicePhone with this information.
Next, BobPhone does the same sequence and discovers that if someone wants to send a packet to BobPhone:33333 they can use BobNAT:44444 as the To address. This is BobPhone’s ICE candidate.
Now, AlicePhone and BobPhone must exchange these ICE candidates.
How do they do this? They have no idea how to talk to each other yet.
The answer is… they Just Use A Server As A Relay! The server used for this purpose is called a Signalling Server.
Note that these called “candidates” because this mechanism won’t work if one of the two NATs changes the public port also based on the public To address, not just the private From address. This is called a Symmetric NAT, and in these (and other) cases, you have to fallback to TURN.
Signalling Server
Signalling is a technical term that simply means: “a way to pass small messages between peers”. In this case, it’s a way for peers to exchange SDP and ICE candidates.
Once these small messages have been exchanged, the peers know how to send data to each other over the internet without needing a relay.
Heck, Alice and Bob can copy/paste these messages into iMessage on both ends. In theory, you can even use carrier pigeons — it’ll just take a very long time to exchange messages 😉
That’s it, this is what Signalling means in a WebRTC context, and why it’s necessary for a successful connection!
What a Signalling Server gives you on top of this is state management: checking whether a peer is allowed to send messages to another peer, whether a peer is allowed to join a call, can be invited to a call, which peers are in a call right now, etc.
Based on your use-case, this part can be really easy to implement or really difficult and heavy in corner-cases. Most people can get away with a really simple protocol, just by adding authorization to this multi-party protocol I wrote for the GStreamer WebRTC multiparty send-receive examples. More complex setups require a more bespoke solution, where all peers aren’t equal.
GUADEC 2023 is underway in Riga. It’s great to see people face to face after a long time. Kicked off the first day with an ADHD trip of a talk. Rather than putting links in my slides, where nobody has the chance of actually follow, I’ve assembled them here. Enjoy!
Color Pallette. Useful tool to get GNOME colors across your toolchain.
Good evening, comrades. This evening, words on reading.
obtaining readables
Since the pandemic, or maybe a bit before, I picked up a new habit: if I hear of a book which is liked by someone I find interesting, I buy a copy.
Getting to this point required some mental changes. When I was younger, buying lots of books wasn’t really thinkable; as a student I was well-served by libraries in my mother tongue, and never picked up the habit of allocating my meager disposable income to books. I did like bookstores, especially used bookstores, but since leaving university I emigrated to countries that spoke different languages; the crate-digging avenue was less accessible to me.
I also had to make peace with capitalism, or as the comic says it, “participating in society”. We all know that Amazon imposes a dialectic on the written word: it makes it astonishingly easy and cheap to obtain anything you want, but also makes it hard to earn a living from writing, producing, and distributing books; consuming words via Amazon undermines the production of the words you want to read.
So I didn’t. Participate, I mean; didn’t order off Amazon. Didn’t read much either. At this point in my life, though, I see it everywhere: purity is the opposite of praxis. I still try to avoid Amazon, quixotically ordering mostly off AbeBooks, a now-fully-owned subsidiary of Amazon which fulfills via independent bookstores, but not avoiding Amazon entirely.
If you have access to and a habit of borrowing via a library in your language, you are almost certainly a better reader than I. But for the rest of us, and if you have the cash, maybe you too can grant yourself permission: when you hear of an interesting book, just buy it.
the ministry for the future
It was surely in this way that I heard of Kim Stanley Robinson’s « The Ministry for the Future ». My copy sat around for a year or so before I got around to it. We all hear the doom and gloom about the climate, and it’s all true; at least, everything about the present. But so much of it comes to us in a way that is fundamentally disempowering, that there is nothing to be done. I find this to be painful, viscerally; I prefer no news rather than bad news that I can’t act on, or at least incorporate into a longer narrative of action.
This may be a personal failing, or a defect. I have juvenile cultural tastes: I don’t like bad things to happen to good people. Jane Eyre was
excruciating: to reach an ultimately heavenly resolution, as a reader I had to suffer with every step. Probably this is recovering-Catholic scar damage.
In any case, climate fiction tends to be more hell than heaven, and I hate it all the more for another reason, that we are here in this moment and we can do things: why tell us that nothing matters? Why would you write a book like that? What part of praxis is that?
Anyway it must have been the premise that made Robinson’s book pass my pre-filters: that the COP series of meetings established a new UN agency that was responsible for the interests of future generations, notably as regards climate. The book charts the story of this administration from its beginnings over a period of some 40 years or so. It’s a wonderfully human, gripping story, and we win.
We don’t win inevitably; the book isn’t a prediction, but more of a prophecy in a sense: an exercise in narrative genesis, a precursor to history, a seed: one not guaranteed to germinate, but which might.
At this point I find the idea of actually participating in a narrative to be almost foreign, or rather alienated. When I was younger, the talk was “after the revolution” this, or “in the future” that, but with enough time and deracination, I lost my vision of how to get there from here. I don’t have it back, not yet anyway, but I think it’s coming. Story precedes history; storytellers are the ones that can breathe life into the airships of the mind. Robinson has both the airships and the gift to speak them into existence. Check it out, comrades, it is required reading!
The original motivation for creating CapyPDF was understanding how fully color managed and print-ready PDF files are actually put together. A reasonable measure of this would be being able to generate fully spec conforming PDF/X-3 files. Most of the building blocks were already there so this was mostly a question of exposing all that in Python and adding the missing bits.
I have now done this and it seems to work. The script itself can be found in CapyPDF's Git repo and the generated PDF document can be downloaded using this link. It looks like this in Acrobat Pro.
It is not graphically the most flashy, but it requires quite a lot of functionality:
The document is A4 size but it is defined on a larger canvas. After printing it needs to be cut into the final shape. This is needed as the document has a bleed.
To obtain this the PDF has a bleed box (the blue box) as well as a trim box (the green box). These are not painted in the document itself, Acrobat Pro just displays them for visualisation purposes.
All colors in graphical operations are specified in CMYK.
The image is a color managed CMYK TIFF. It has an embedded ICC profile that is different from the one used in final printing. This is due to laziness as I had this image laying around already. You don't want to do this in a real print job.
The heading has both a stroke and a fill
The printer marks are just something I threw on the page at random.
Here is a screenshot of the document passing PDF-X/3 validation in Acrobat Pro.
You probably all know that you can easily access files on connected devices using the sidebar items in the Files application. Most of you also know that the Other Locations view provides access to internal storage and some other discovered services. But I have recently noticed that some people are completely unaware of the ability to connect to locations that are not propagated by our volume monitors or discovery backends. And even if people are aware of this option, they often don’t know which locations are supported. So I decided to write this blog spot to summarize what all URI schemes are handled by the GVfs project for use by the Files and other GNOME applications.
Connecting to locations
For locations that are not automatically propagated, there is the following bar in the Other Locations view of the Files application. You can enter a URI and press the Connect button to mount some additional locations. The question mark icon suggests some of the URI schemes you can use.
Network protocols
To connect to devices on your network, cloud providers, or web hostings, there is a list of network protocols that can be used:
SMB – Uses the smb:// scheme, so e.g. smb://host/share.
FTP – There are 3 different protocol schemes for the FTP protocol:
ftp:// for unencrypted connection
ftps:// for explicit SSL/TLS encryption
ftpis:// for implicit SSL/TLS encryption
SFTP – Uses the sftp:// scheme (the ssh:// alias can also be used).
WebDAV – There are the dav:// and davs:// schemes, corresponding to the HTTP and HTTPS variants.
NFS – Uses the nfs:// scheme. Only NFSv2 and NFSv3 protocol versions are supported.
AFP – Uses the afp:// scheme.
Just a note that there is also support for the HTTP protocol over the http:// and https:// schemes, although this is not intended to be used through the Files application.
Network discovery
Available locations on your local network can be automatically listed in the Other Locations view thanks to the network backend (the network:// scheme). The network backend merges results from other backends that implement the DNS-SD/mDNS and NetBIOS protocols (these use the dns-sd:// and smb:// schemes).
NetBIOS-based discovery has been deprecated along with the NT1 protocol, which is why the Windows Network folder in the Other Location view is empty in most cases. Instead, Microsoft relies on the WS-Discovery protocol. This is not yet supported by the GVfs project but it is in development.
Connected devices
Connected devices are always propagated by volume monitors and are not expected to be mounted manually. Cell phones, cameras, and audio players typically use the MTP, PTP, and AFC protocols (with the mtp://, gphoto2://, and afc:// schemes). Mass storage devices (e.g. portable flash disks and hard disks) and optical disks are announced via the UDisks2 volume monitor and use the file:// scheme. Only audio disks are accessible via the cdda:// scheme.
There is also a special burn:// scheme for blank optical media, but Brasero integration is currently broken.
Online Accounts
The Online Accounts panel in the Settings application allows users to configure their Nextcloud and Google accounts, among others. You can then easily access your files from these providers via the sidebar items in the Files application. These are propagated through the GOA volume monitor and use the davs:// and google-drive:// schemes.
The GVfs project also provides some special locations. The recent:// scheme allows you to browse recent files. The admin:// scheme allows you to access files with elevated privileges. The archive:// scheme can be used to browse archive files without extracting them. It was supposed to be integrated into the Files application, but that never happened. Not sure if anyone uses it today. The computer:// scheme is used to provide a list of connected devices. It is no longer used by the Files application, but some other file managers still use it.
For completeness, there are also the test:// and localtest:// schemes for testing purposes.
The specific list of supported locations is always dependent on the installed packages, but mainstream distributions should have most of them installed by default. I hope this gives you a better idea of the locations provided by the GVfs project that are available in GNOME. Which protocols do you use most?
Furthermore, my mentor, Bilal, started implementing session monitoring and saving to a PCAP file while I continued their work and implemented system and address recording. However, it is not yet merged as there are some things to be figured out and more work to do. Specifically, it currently requires a dbus-monitor binary installed on the host, which is not great for distributions that do not package it.
Aside from those mentioned above, I have also been working on drawing the diagram. I am still studying the older Haskell codebase to prevent reinventing the wheel on some implementations. Unfortunately, the current implementation is still not ready enough for a screenshot.
To easily track the missing features, we also set up an issue board.
Plans for the Following Weeks
In the following weeks, I will get the PR to implement and address recording merged and work on more parts of the diagram drawing.
You know the feeling of actually coding something, a new library which doesn’t exist yet but you know exactly how it should look and you just need to chip away at the problem space until its final form appears ?
I was really lacking any creative flow recently and decided to flesh out the libgnomesearch library that I proposed a while ago, which my colleague Kaspar Matas already made a start at developing.
The purpose of this library is to consolidate existing search code that currently lives in Nautilus, GTK, GNOME Shell and various core apps. There are some interesting designs for improving GNOME’s desktop search functionality, and I suspect it will be easier to prototype and implement these if there is a single place to make the changes.
I dedicated some evenings and weekends to the fun part of building an API and example app, and this is the result. Some code, some documentation, and a demo app:
To state the obvious : it’s not finished and it has bugs!
I can’t spend work time on this and probably won’t have much more volunteer time for it either, so let’s see what happens next. If this sounds like a project you are interested in continuing, get in touch – and if you are at GUADEC 2023 then we can chat there!
I have been increasingly disillusioned and disappointed by corporate technology. The absolute dumpster fire of Twitter and the issues present at Reddit has also reminded me how absolutely horrible these massive tech companies are. I do not want to be another underpaid cog in a massive contraption that is only making our world worse.
I hope that one day in the future I could make a living as an independent developer. I envision a career where I am not just building software, but creating tools that respect users' freedom and privacy while contributing to the common good. I want my work to reflect my values, and to stand for something more than just profit.
My adventures in developing for GNOME during this internship with Outreachy, have proven to be an invaluable stepping stone towards this goal. It's an experience I'm profoundly grateful for and one that has reinforced my commitment to a future aligned with FOSS values.
It is highly likely that this is all just a pipe dream and paying my rent via patreon will not be my reality. Thanks for reading.
Telemetry is one of the biggest controversial topics in the Linux community. Many people believe that telemetry is entirely meaningless, because developers can “just” ask their users. Some people also argue that users can opt into telemetry if they want to participate, but most of these users are in consensus that opt-out telemetry shouldn’t be there in the first place.
However, I don’t believe that asking users or explicitly opting into telemetry helps to a degree where developers and designers can form educated conclusions, as both methods share many issues regarding gathering data accurately. In this article, we’re going to explore the issues around asking users and opting into telemetry, and then I will explain why opt-out telemetry is a better approach to gather accurate data and forming educated conclusions.
Definition and Methods of Gathering User Data
Per Wikipedia, telemetry “is used to gather data on the use and performance of applications and application components, e.g. how often certain features are used, measurements of start-up time and processing time, hardware, application crashes, and general usage statistics and/or user behavior. In some cases, very detailed data are reported like individual window metrics, counts of used features, and individual function timings.”
There are three main methods to gather user data and they will be defined as such:
Asking users: when developers explicitly ask users for feedback through surveys or similar.
Opt-in telemetry: when users explicitly enable telemetry, meaning telemetry is disabled by default.
Opt-out telemetry: when users explicitly disable telemetry, meaning telemetry is enabled by default.
Problems With Opt-in Telemetry and Asking Users
There are several problems with opt-in telemetry and asking users: they are prone to inaccurate and misleading data, they’re undiscoverable for less technical users, and if they are discoverable, then they are usually displayed in such a way they aggressively shift users’ focus.
Prone to Inaccurate Data
The main (and harmful) problem with asking users and opt-in telemetry is inaccurate and biased data provided by enthusiasts.
I’m going to take “gnome-info-collect: What we learned” as an example. Recently, GNOME conducted a research to gather data from their users. Users were encouraged to participate, but the research was entirely opt-in. They needed to install a package named “gnome-info-collect” and run it. With all the data collected by GNOME, we got the following result for distributions used:
Distro
Number of responses
% of responses
Fedora
1376
54.69%
Arch
469
18.64%
Ubuntu
267
10.61%
Manjaro
140
5.56%
Other
78
3.10%
EndeavourOS
66
2.62%
Debian
44
1.75%
openSUSE
38
1.51%
Pop!
38
1.51%
Total
2516
100.00%
Notice that Fedora Linux users represent the highest amount of participants — literally more than half of the vote count, followed by Arch Linux and Ubuntu. However the consensus seems to be that Ubuntu is the most used Linux distribution (excluding Android). Ubuntu Desktop (the main Ubuntu in the lineup) ships with GNOME, just like Fedora Workstation, which means that the majority of GNOME users are actually Ubuntu users, not Fedora Linux users (or even Arch Linux users). In this case, Fedora Linux and Arch Linux users would be considered as vocal minorities, as there are more participants than Ubuntu’s, despite having less overall users.
Vocal minorities and enthusiasts are not representative of the entire userbase, especially in the context of software designed for less technical users. These data can be harmful, as they are often inaccurate and unrepresentative. Especially in user-interface (UI) and user-experience (UX) research, these data can skew the opinions and conclusions of developers and designers, which could exclusively benefit vocal minorities but harm their target audience(s).
This could also make it difficult to judge which statistics are accurate or even form a moderate conclusion, because we can’t tell which one is representative, unless there is a consensus on the matter.
In the above example, since there is a consensus that Ubuntu is the most used distribution, we have a reference that the data provided is not representative. However, in topics that lack consensus, our conclusions may be misguided.
Undiscoverable for Less Technical Users
Asking users and opt-in telemetry may be undiscoverable to users, as they must be made aware of said method(s) in some way.
For opt-in telemetry, they are often obscured that users will rarely discover them. Once again, this is typically enabled by vocal minorities and enthusiasts, like myself, which may result in biased results. For example, on Mozilla Thunderbird, to enable telemetry, we go through the hamburger menu → Settings → Privacy & Security, then scroll all the way down to “Thunderbird Data Collection and Use”. I only discovered it because I felt like exploring in the Settings.
As for asking users, they need a way to be reached out. In GNOME’s telemetry, I quote:
The people who provided their data with gnome-info-collect were primarily recruited via GNOME’s media channels, including Discourse and Twitter. This means that the data we collected was on a fairly particular subset of GNOME’s users: namely, people who follow our social media channels, are interested enough to respond to our call for help […]
Typically, enthusiasts will participate in surveys and/or opt into telemetry, as they’re active in chat rooms, follow the project on social media and/or blog, participate in forums, etc. However, there won’t be an easy way to reach out to less technical users without implementing annoyances. This makes it very difficult to gain accurate data of the entire userbase and making it representative.
Aggressively Shifting Focus to Ask for Feedback
A common tactic to ask users for feedback and the most “effective” one I know of is by displaying popups, banners, notifications, or other forms of behavior that will shift users’ focus.
For example, LibreOffice sometimes asks for feedback using a banner:
Banner written “ℹ️ Help us make LibreOffice even better!” with “Get involved” and close buttons.
If you’ve ever seen this banner on LibreOffice or other apps, then you have probably clicked the close button, which is a totally normal thing to do. When you open LibreOffice, you don’t open it to participate in surveys; you typically open it to actually use LibreOffice.
When you’re in the middle of using an app and are prompted to shift your focus onto something that is completely unrelated to your intended goal, then you will likely do everything you can to get rid of it. If there is a “do not show again” button, then you will likely press that alongside, so it doesn’t annoy you again.
There are some exceptions, though. Some might want to participate if they have the time. However, this brings back to data that may be inaccurate and unrepresentative.
Time-consuming
Many users may not have the time or mood to participate in surveys for whatever reason, myself included. As pointed out previously, when you open an app, you open it to use it, not to participate in a project. I maintain and contribute to several projects, play video games with friends, spend time with family, etc. Many of us have jobs and chores to do additionally. We may not always feel like spending time filling up a survey, as we’re already exhausted and have very little time to spare. At that time, I was creating a resumé.
At the same time, personally, even if it takes 5 minutes, I prefer to participate properly, in the right mood; provide useful information and answer questions correctly, but this kind of mood rarely happens to me, so I rarely participate in surveys.
Opt-out Telemetry
Opt-out telemetry, despite its controversies, addresses those problems by including everybody by default, including less technical users, as opposed to opt-in telemetry and asking users. This means that less technical users’ data have more potential to influence the data into results that projects can use as a reference and form educated conclusions, and tailor their software around their target audience(s) and the majority of the userbase.
In addition, since opt-out telemetry is enabled by default, there is the benefit that there will be more participants as a result. Of course, users can always opt-out if they want, but at least these data get us much closer to something that is representative and accurate.
Conclusion
Opt-in telemetry and asking users for feedback may result in data that is unrepresentative and inaccurate, which may cause the collected data to be misleading and lead to misguided conclusions and incorrect decisions. Furthermore, these methods may be unreachable to less technical users, as they don’t follow the projects’ social media, forums, chat, etc. Asking users for feedback can also be obnoxious, because they are designed to shift users’ focus to something that they did not anticipate, which can be distracting. They can also be time-consuming, given the circumstances we may live in.
This is not to say that asking users and opt-in telemetry are a bad thing. I personally support asking users specifically for feature proposals — in fact, this is what issue trackers are for. However, I do not support asking users for general feedback, because general feedback struggle with representativeness.
In my opinion, inaccurate data are more harmful than no data.
This will be a short one but it took significant time to get things done right.
Renaming PlayAnswer to ClueGrid
I made an initial change by renaming “PlayAnswer” to “ClueGrid” since we utilize a similar grid in the crossword editor and it sounds more appropriate. The “EditClueDetails” widget specifically uses a similar grid.
Mapping ClueGrid cell selection onto the MainGrid
Now it took a significant amount of time to do this task because I was relatively new to g_signals. Let me explain how it works:
When we perform an action on the puzzle, this is how the update flows and finally, the state is pushed onto the PuzzleStack:
play_xword_update_state gets called after an action is performed
Then, play_clue_list_update
Next, we execute play_clue_row_update
Finally, we perform clue_grid_update
So when we do a CLUE_CELL_SELECT on the ClueGrid, we would catch and emit the signal in a similar manner but in reverse order.
To achieve this, we maintain acells_map that maps each index on the ClueGrid to the corresponding cell on the main grid.
Related Merge Requests:
Mapping ClueGrid cell selection onto the MainGrid !114
Next, my plan is to add a GUESS_AT_CLUE_CELLsignal and focus on switching navigation based on the “focus”.
I can’t belive its been more than a year since my last composefs blog. So much has changed, yet the goal is the same. However, finally things are starting to settling down, so I think it is time to do an update on the current state.
Background
First some background, like what even is Composefs?
The initial version of Composefs was an actual linux kernel filesystem module. But during development and upstream discussions it has become clear that a better approach is to use existing kernel features, with some minor feature additions rather than a completely new filesystem. However, from a user perspective, it still looks like a filesystem.
Composefs is not a regular filesystem like ext4 though. It is not meant to be stored on a disk partition, but instead as a file that gets downloaded and mounted. The typical usecase is that you have some directory structure, call it an “image”, and you want to package up this image so it can easily be used elsewhere without needing to unpack it. For example, this could be a container image, or a rootfs image.
The closest thing that we have today is a loopback mount of a read-only filesystem (such as squashfs or erofs). However, composefs has two big advantages over this: file sharing and integrity validation.
A composefs image is made up of two things, the filesystem image and a directory of backing files. For example, suppose you have this directory:
$ tree example-dir/
example-dir/
├── data.txt
└── subdir
└── other.txt
With the mkcomposefs tool you can create an image from this:
$ mkcomposefs --digest-store=objects /the/example-dir example.cfs
$ tree
├── example.cfs
└── objects
├── 9e
│ └── 3ba51c3a07352a7c32a27a17604d8118724a16...
└── a2
└── 447bfab34972328b1f67e3d06e82c990c64f12...
The example.cfs image has all the information about the files and directories, like filenames, permissions, mtimes, xattrs, etc. However, the actual file data is stored in the individual backing files in the objects subdirectory, and are only accessed when needed (i.e. when respective file is opened).
We can mount it like this (using the mount.composefs helper):
$ sudo mount -t composefs -o basedir=objects example.cfs /mnt
$ ls -l /mnt/
-rw-r--r--. 1 alex alex 18 11 jul 14.25 data.txt
drwxr-xr-x. 2 alex alex 48 11 jul 14.26 subdir
Note that the backing files are named by the checksum of their content. This means that if you create multiple images with a shared objects directory, then the same backing file will be used for any file that is shared between images.
Not only does this mean that images that share files can be stored more efficiently on disk, it also means that any such shared files will be stored only once in page-cache (i.e. ram). A container system using this would allow more containers to run on the same hardware, because libraries shared between unrelated images can be shared.
Additionally, composefs supports using fs-verity for both the image file, and all the backing files. This means that if you specify the expected digest when you mount the composefs image, it will be validated before use. Then the image itself contains the expected fs-verity digests of the backing files, and these will be also be verified at use time. Using this we get the integrity benefit of something like dm-verity, while still allowing fine-grained disk and memory sharing.
Composefs status
Composefs itself consists of a few pieces:
Userspace support, including mkcomposefs and mount.composefs
The image format, which is based on an erofs image with overlayfs xattrs
New overlayfs kernel feature for supporting the composefs usecase
The userspace is available at https://github.com/containers/composefs and is now in a pretty good state. The current releases are marked pre-release, because we don’t fully want to mark it stable until all the overlayfs changes are in a kernel release so we can fully rely on the format being long-term stable.
On the erofs side, all the features we need are in kernel 5.15 or later.
For overlayfs there are two features the we rely on, the new “data-only” lower directory support, which has landed in 6.5-rc1, and the support for storing fs-verity digests in the overlay.metacopy xattr, which is queued for 6.6 in the overlayfs-next branch. However, these kernel changes are only needed for the integrity checking, so if you don’t need those, then current kernels work.
OSTree status
One of the main usecases for composefs is OSTree. Initial work for composefs integration landed in OSTree 2023.4, and further work is ongoing to support validation of composefs images using ed25519 signatures. This will allow secure boot to extend the chain of trust from the kernel/initrd to the whole of userspace.
This integration with OSTree is intentionally done with minimal changes, because the goal is to allow current users of OSTree to easily benefit from the advantages of composefs. However, in parallel there are long term plans to radically redo and simplify some parts of OSTree on top of composefs. See this upstream issue for more discussions about this.
Container backend
Another important usecase for composefs has always been OCI containers. Recently the initial work on this landed in containers/storage, which is the first step in making podman use composefs for images, thus allowing higher density and integrity for container images.
Going forward
These are only the initial steps for composefs. I think the future holds even more interesting ideas. For example, given that both ostree and podman are converging on the same storage model, there is hope that we will be able to share backing files between ostree and podman, such that files identical in the host OS and a container image are stored only once.
We also hope that as composefs starts being deployed more widely people will come up with new exciting use cases.
The phrase "Root of Trust" turns up at various points in discussions about verified boot and measured boot, and to a first approximation nobody is able to give you a coherent explanation of what it means[1]. The Trusted Computing Group has a fairly wordy definition, but (a) it's a lot of words and (b) I don't like it, so instead I'm going to start by defining a root of trust as "A thing that has to be trustworthy for anything else on your computer to be trustworthy".
(An aside: when I say "trustworthy", it is very easy to interpret this in a cynical manner and assume that "trust" means "trusted by someone I do not necessarily trust to act in my best interest". I want to be absolutely clear that when I say "trustworthy" I mean "trusted by the owner of the computer", and that as far as I'm concerned selling devices that do not allow the owner to define what's trusted is an extremely bad thing in the general case)
Let's take an example. In verified boot, a cryptographic signature of a component is verified before it's allowed to boot. A straightforward implementation of a verified boot implementation has the firmware verify the signature on the bootloader or kernel before executing it. In this scenario, the firmware is the root of trust - it's the first thing that makes a determination about whether something should be allowed to run or not[2]. As long as the firmware behaves correctly, and as long as there aren't any vulnerabilities in our boot chain, we know that we booted an OS that was signed with a key we trust.
But what guarantees that the firmware behaves correctly? What if someone replaces our firmware with firmware that trusts different keys, or hot-patches the OS as it's booting it? We can't just ask the firmware whether it's trustworthy - trustworthy firmware will say yes, but the thing about malicious firmware is that it can just lie to us (either directly, or by modifying the OS components it boots to lie instead). This is probably not sufficiently trustworthy!
Ok, so let's have the firmware be verified before it's executed. On Intel this is "Boot Guard", on AMD this is "Platform Secure Boot", everywhere else it's just "Secure Boot". Code on the CPU (either in ROM or signed with a key controlled by the CPU vendor) verifies the firmware[3] before executing it. Now the CPU itself is the root of trust, and, well, that seems reasonable - we have to place trust in the CPU, otherwise we can't actually do computing. We can now say with a reasonable degree of confidence (again, in the absence of vulnerabilities) that we booted an OS that we trusted. Hurrah!
Except. How do we know that the CPU actually did that verification? CPUs are generally manufactured without verification being enabled - different system vendors use different signing keys, so those keys can't be installed in the CPU at CPU manufacture time, and vendors need to do code development without signing everything so you can't require that keys be installed before a CPU will work. So, out of the box, a new CPU will boot anything without doing verification[4], and development units will frequently have no verification.
As a device owner, how do you tell whether or not your CPU has this verification enabled? Well, you could ask the CPU, but if you're doing that on a device that booted a compromised OS then maybe it's just hotpatching your OS so when you do that you just get RET_TRUST_ME_BRO even if the CPU is desperately waving its arms around trying to warn you it's a trap. This is, unfortunately, a problem that's basically impossible to solve using verified boot alone - if any component in the chain fails to enforce verification, the trust you're placing in the chain is misplaced and you are going to have a bad day.
So how do we solve it? The answer is that we can't simply ask the OS, we need a mechanism to query the root of trust itself. There's a few ways to do that, but fundamentally they depend on the ability of the root of trust to provide proof of what happened. This requires that the root of trust be able to sign (or cause to be signed) an "attestation" of the system state, a cryptographically verifiable representation of the security-critical configuration and code. The most common form of this is called "measured boot" or "trusted boot", and involves generating a "measurement" of each boot component or configuration (generally a cryptographic hash of it), and storing that measurement somewhere. The important thing is that it must not be possible for the running OS (or any pre-OS component) to arbitrarily modify these measurements, since otherwise a compromised environment could simply go back and rewrite history. One frequently used solution to this is to segregate the storage of the measurements (and the attestation of them) into a separate hardware component that can't be directly manipulated by the OS, such as a Trusted Platform Module. Each part of the boot chain measures relevant security configuration and the next component before executing it and sends that measurement to the TPM, and later the TPM can provide a signed attestation of the measurements it was given. So, an SoC that implements verified boot should create a measurement telling us whether verification is enabled - and, critically, should also create a measurement if it isn't. This is important because failing to measure the disabled state leaves us with the same problem as before; someone can replace the mutable firmware code with code that creates a fake measurement asserting that verified boot was enabled, and if we trust that we're going to have a bad time.
(Of course, simply measuring the fact that verified boot was enabled isn't enough - what if someone replaces the CPU with one that has verified boot enabled, but trusts keys under their control? We also need to measure the keys that were used in order to ensure that the device trusted only the keys we expected, otherwise again we're going to have a bad time)
So, an effective root of trust needs to:
1) Create a measurement of its verified boot policy before running any mutable code 2) Include the trusted signing key in that measurement 3) Actually perform that verification before executing any mutable code
and from then on we're in the hands of the verified code actually being trustworthy, and it's probably written in C so that's almost certainly false, but let's not try to solve every problem today.
Does anything do this today? As far as I can tell, Intel's Boot Guard implementation does. Based on publicly available documentation I can't find any evidence that AMD's Platform Secure Boot does (it does the verification, but it doesn't measure the policy beforehand, so it seems spoofable), but I could be wrong there. I haven't found any general purpose non-x86 parts that do, but this is in the realm of things that SoC vendors seem to believe is some sort of value-add that can only be documented under NDAs, so please do prove me wrong. And then there are add-on solutions like Titan, where we delegate the initial measurement and validation to a separate piece of hardware that measures the firmware as the CPU reads it, rather than requiring that the CPU do it.
But, overall, the situation isn't great. On many platforms there's simply no way to prove that you booted the code you expected to boot. People have designed elaborate security implementations that can be bypassed in a number of ways.
[1] In this respect it is extremely similar to "Zero Trust" [2] This is a bit of an oversimplification - once we get into dynamic roots of trust like Intel's TXT this story gets more complicated, but let's stick to the simple case today [3] I'm kind of using "firmware" in an x86ish manner here, so for embedded devices just think of "firmware" as "the first code executed out of flash and signed by someone other than the SoC vendor" [4] In the Intel case this isn't strictly true, since the keys are stored in the motherboard chipset rather than the CPU, and so taking a board with Boot Guard enabled and swapping out the CPU won't disable Boot Guard because the CPU reads the configuration from the chipset. But many mobile Intel parts have the chipset in the same package as the CPU, so in theory swapping out that entire package would disable Boot Guard. I am not good enough at soldering to demonstrate that.
. You only had to install a handful of toolchains, development headers, and tools from the distribution packages, run make install, execute 
