March 21, 2023

WebKitGTK API for GTK 4 Is Now Stable

With the release of WebKitGTK 2.40.0, WebKitGTK now finally provides a stable API and ABI for GTK 4 applications. The following API versions are provided:

  • webkit2gtk-4.0: this API version uses GTK 3 and libsoup 2. It is obsolete and users should immediately port to webkit2gtk-4.1. To get this with WebKitGTK 2.40, build with -DPORT=GTK -DUSE_SOUP2=ON.
  • webkit2gtk-4.1: this API version uses GTK 3 and libsoup 3. It contains no other changes from webkit2gtk-4.0 besides the libsoup version. With WebKitGTK 2.40, this is the default API version that you get when you build with -DPORT=GTK. (In 2.42, this might require a different flag, e.g. -DUSE_GTK3=ON, which does not exist yet.)
  • webkitgtk-6.0: this API version uses GTK 4 and libsoup 3. To get this with WebKitGTK 2.40, build with -DPORT=GTK -DUSE_GTK4=ON. (In 2.42, this might become the default API version.)

WebKitGTK 2.38 had a different GTK 4 API version, webkit2gtk-5.0. This was an unstable/development API version and it is gone in 2.40, so applications using it will break. Fortunately, that should be very few applications. If your operating system ships GNOME 42, or any older version, or the new GNOME 44, then no applications use webkit2gtk-5.0 and you have no extra work to do. But for operating systems that ship GNOME 43, webkit2gtk-5.0 is used by gnome-builder, gnome-initial-setup, and evolution-data-server:

  • For evolution-data-server 3.46, use this patch which applies on evolution-data-server 3.46.4.
  • For gnome-initial-setup 43, use this patch which applies on gnome-initial-setup 43.2.
  • For gnome-builder 43, all required changes are present in version 43.7.

Remember, patching is only needed for GNOME 43. Other versions of GNOME will have no problems with WebKitGTK 2.40.

There is no proper online documentation yet, but in the meantime you can view the markdown source for the migration guide to help you with porting your applications. Although the API is now stable and it is close to feature parity with the GTK 3 version, there are some problems to be aware of:

Big thanks to everyone who helped make this possible.

March 20, 2023

The joy of font debugging

Remember how in the previous blog post it was said that creating text in PDF would be "just a matter of setting some parameters"?

Well let's start by creating the main text in two justified columns.

Ok, nice. Next we add an author name.

Whoopsies. After some debugging one can find out that this only happens if you only use fewer than 32 characters from the font you are subsetting. Obviously. But no matter, after fixing this minor niggle we only need to display the title, author names and finally an email address. This is more of the same so nothing can go wrong.

After some hours of debugging it becomes clear that the values of left side bearings are sometimes read from the source file using incorrect offsets (while still being 100% memory safe, all accesses are inside the source data). Good Now that that's fully fix...

This is where things get extremely weird. No matter where you look or how deeply you peruse the binary data files, nothing seems to be incorrect. Maybe this is a bug in the Noto Mono font used here? So you try Liberation Mono. It fails too. And then, just to be sure, you try Ubuntu Mono. It works correctly. As does Free Mono.

Hmmmmmhmhm.

Opening the file in Fontforge says that the width of all characters is 1228 font units. That is also what Freetype reports. Which is comforting because in the TrueType file format struct fields that are designated as 32 bit integers might be either a) 32 bit integer b) 26.6. fixed point or c) 16.16 fixed point. You can't ever really be sure which, though, because it depends on values of bitfields far and away from the actual structs themselves.

Things get even weirder if you test exporting a PDF that uses those broken fonts either with Cairo or LibreOffice. In the generated PDF files the width of characters in this font to 600. Not 1228. Trying to read their source to find out how and why they do this is problematic, because they support many different font formats and thus convert the input data to their internal representation and then generate the output from those. Trying to understand how the input data correlates with the output data can give you a major headache without even trying too hard.

The actual solution is even weirder than all of the above. TrueType fonts store the horizontal metrics of glyphs in a table called hmtx. It stores both the glyph advance and left side bearings. As a special case you can only specify the latter and use a common value for the former. This provides space savings of UP TO 2 BYTES PER CHARACTER but the downside is more complex parsing. Further, going through Freetype's public structs reveals that they contain a field called x_scale. After a lot of trial error you can eventually decipher the actual algorithm needed:

If the character has both glyph advance and left side bearings defined than you use them as-is but if it only has left side bearings defined then you must divide the default width value with the scale.

Then finally.

Addendum: Freetype

Freetype has many flags and knobs to specify whether you want metrics in original font coordinates or "output coordinates". I could not come up with a combination that would have provided consistent values, some postprocessing seems to always be needed. This might be a bug, it might be a limitation of the TrueType format, it might be something completely different. I don't really know, and I don't have the energy to dig further to uncover the underlying issue.

a world to win: webassembly for the rest of us

Good day, comrades!

Today I'd like to share the good news that WebAssembly is finally coming for the rest of us weirdos.

A world to win

WebAssembly for the rest of us

17 Mar 2023 – BOB 2023

Andy Wingo

Igalia, S.L.

This is a transcript-alike of a talk that I gave last week at BOB 2023, a gathering in Berlin of people that are using "technologies beyond the mainstream" to get things done: Haskell, Clojure, Elixir, and so on. PDF slides here, and I'll link the video too when it becomes available.

WebAssembly, the story

WebAssembly is an exciting new universal compute platform

WebAssembly: what even is it? Not a programming language that you would write software in, but rather a compilation target: a sort of assembly language, if you will.

WebAssembly, the pitch

Predictable portable performance

  • Low-level
  • Within 10% of native

Reliable composition via isolation

  • Modules share nothing by default
  • No nasal demons
  • Memory sandboxing

Compile your code to WebAssembly for easier distribution and composition

If you look at what the characteristics of WebAssembly are as an abstract machine, to me there are two main areas in which it is an advance over the alternatives.

Firstly it's "close to the metal" -- if you compile for example an image-processing library to WebAssembly and run it, you'll get similar performance when compared to compiling it to x86-64 or ARMv8 or what have you. (For image processing in particular, native still generally wins because the SIMD primitives in WebAssembly are more narrow and because getting the image into and out of WebAssembly may imply a copy, but the general point remains.) WebAssembly's instruction set covers a broad range of low-level operations that allows compilers to produce efficient code.

The novelty here is that WebAssembly is both portable while also being successful. We language weirdos know that it's not enough to do something technically better: you have to also succeed in getting traction for your alternative.

The second interesting characteristic is that WebAssembly is (generally speaking) a principle-of-least-authority architecture: a WebAssembly module starts with access to nothing but itself. Any capabilities that an instance of a module has must be explicitly shared with it by the host at instantiation-time. This is unlike DLLs which have access to all of main memory, or JavaScript libraries which can mutate global objects. This characteristic allows WebAssembly modules to be reliably composed into larger systems.

WebAssembly, the hype

It’s in all browsers! Serve your code to anyone in the world!

It’s on the edge! Run code from your web site close to your users!

Compose a library (eg: Expat) into your program (eg: Firefox), without risk!

It’s the new lightweight virtualization: Wasm is what containers were to VMs! Give me that Kubernetes cash!!!

Again, the remarkable thing about WebAssembly is that it is succeeding! It's on all of your phones, all your desktop web browsers, all of the content distribution networks, and in some cases it seems set to replace containers in the cloud. Launch the rocket emojis!

WebAssembly, the reality

WebAssembly is a weird backend for a C compiler

Only some source languages are having success on WebAssembly

What about Haskell, Ocaml, Scheme, F#, and so on – what about us?

Are we just lazy? (Well...)

So why aren't we there? Where is Clojure-on-WebAssembly? Where are the F#, the Elixir, the Haskell compilers? Some early efforts exist, but they aren't really succeeding. Why is that? Are we just not putting in the effort? Why is it that Rust gets to ride on the rocket ship but Scheme does not?

WebAssembly, the reality (2)

WebAssembly (1.0, 2.0) is not well-suited to garbage-collected languages

Let’s look into why

As it turns out, there is a reason that there is no good Scheme implementation on WebAssembly: the initial version of WebAssembly is a terrible target if your language relies on the presence of a garbage collector. There have been some advances but this observation still applies to the current standardized and deployed versions of WebAssembly. To better understand this issue, let's dig into the guts of the system to see what the limitations are.

GC and WebAssembly 1.0

Where do garbage-collected values live?

For WebAssembly 1.0, only possible answer: linear memory

(module
  (global $hp (mut i32) (i32.const 0))
  (memory $mem 10)) ;; 640 kB

The primitive that WebAssembly 1.0 gives you to represent your data is what is called linear memory: just a buffer of bytes to which you can read and write. It's pretty much like what you get when compiling natively, except that the memory layout is more simple. You can obtain this memory in units of 64-kilobyte pages. In the example above we're going to request 10 pages, for 640 kB. Should be enough, right? We'll just use it all for the garbage collector, with a bump-pointer allocator. The heap pointer / allocation pointer is kept in the mutable global variable $hp.

(func $alloc (param $size i32) (result i32)
  (local $ret i32)
  (loop $retry
    (local.set $ret (global.get $hp))
    (global.set $hp
      (i32.add (local.get $size) (local.get $ret)))

    (br_if 1
      (i32.lt_u (i32.shr_u (global.get $hp) 16)
                (memory.size))
      (local.get $ret))

    (call $gc)
    (br $retry)))

Here's what an allocation function might look like. The allocation function $alloc is like malloc: it takes a number of bytes and returns a pointer. In WebAssembly, a pointer to memory is just an offset, which is a 32-bit integer (i32). (Having the option of a 64-bit address space is planned but not yet standard.)

If this is your first time seeing the text representation of a WebAssembly function, you're in for a treat, but that's not the point of the presentation :) What I'd like to focus on is the (call $gc) -- what happens when the allocation pointer reaches the end of the region?

GC and WebAssembly 1.0 (2)

What hides behind (call $gc) ?

Ship a GC over linear memory

Stop-the-world, not parallel, not concurrent

But... roots.

The first thing to note is that you have to provide the $gc yourself. Of course, this is doable -- this is what we do when compiling to a native target.

Unfortunately though the multithreading support in WebAssembly is somewhat underpowered; it lets you share memory and use atomic operations but you have to create the threads outside WebAssembly. In practice probably the GC that you ship will not take advantage of threads and so it will be rather primitive, deferring all collection work to a stop-the-world phase.

GC and WebAssembly 1.0 (3)

Live objects are

  • the roots
  • any object referenced by a live object

Roots are globals and locals in active stack frames

No way to visit active stack frames

What's worse though is that you have no access to roots on the stack. A GC has to keep live objects, as defined circularly as any object referenced by a root, or any object referenced by a live object. It starts with the roots: global variables and any GC-managed object referenced by an active stack frame.

But there we run into problems, because in WebAssembly (any version, not just 1.0) you can't iterate over the stack, so you can't find active stack frames, so you can't find the stack roots. (Sometimes people want to support this as a low-level capability but generally speaking the consensus would appear to be that overall performance will be better if the engine is the one that is responsible for implementing the GC; but that is foreshadowing!)

GC and WebAssembly 1.0 (3)

Workarounds

  • handle stack for precise roots
  • spill all possibly-pointer values to linear memory and collect conservatively

Handle book-keeping a drag for compiled code

Given the noniterability of the stack, there are basically two work-arounds. One is to have the compiler and run-time maintain an explicit stack of object roots, which the garbage collector can know for sure are pointers. This is nice because it lets you move objects. But, maintaining the stack is overhead; the state of the art solution is rather to create a side table (a "stack map") associating each potential point at which GC can be called with instructions on how to find the roots.

The other workaround is to spill the whole stack to memory. Or, possibly just pointer-like values; anyway, you conservatively scan all words for things that might be roots. But instead of having access to the memory to which the WebAssembly implementation would spill your stack, you have to do it yourself. This can be OK but it's sub-optimal; see my recent post on the Whippet garbage collector for a deeper discussion of the implications of conservative root-finding.

GC and WebAssembly 1.0 (4)

Cycles with external objects (e.g. JavaScript) uncollectable

A pointer to a GC-managed object is an offset to linear memory, need capability over linear memory to read/write object from outside world

No way to give back memory to the OS

Gut check: gut says no

If that were all, it would already be not so great, but it gets worse! Another problem with linear-memory GC is that it limits the potential for composing a number of modules and the host together, because the garbage collector that manages JavaScript objects in a web browser knows nothing about your garbage collector over your linear memory. You can easily create memory leaks in a system like that.

Also, it's pretty gross that a reference to an object in linear memory requires arbitrary read-write access over all of linear memory in order to read or write object fields. How do you build a reliable system without invariants?

Finally, once you collect garbage, and maybe you manage to compact memory, you can't give anything back to the OS. There are proposals in the works but they are not there yet.

If the BOB audience had to choose between Worse is Better and The Right Thing, I think the BOB audience is much closer to the Right Thing. People like that feel instinctual revulsion to ugly systems and I think GC over linear memory describes an ugly system.

GC and WebAssembly 1.0 (5)

There is already a high-performance concurrent parallel compacting GC in the browser

Halftime: C++ N – Altlangs 0

The kicker is that WebAssembly 1.0 requires you to write and deliver a terrible GC when there is already probably a great GC just sitting there in the host, one that has hundreds of person-years of effort invested in it, one that will surely do a better job than you could ever do. WebAssembly as hosted in a web browser should have access to the browser's garbage collector!

I have the feeling that while those of us with a soft spot for languages with garbage collection have been standing on the sidelines, Rust and C++ people have been busy on the playing field scoring goals. Tripping over the ball, yes, but eventually they do manage to make within striking distance.

Change is coming!

Support for built-in GC set to ship in Q4 2023

With GC, the material conditions are now in place

Let’s compile our languages to WebAssembly

But to continue the sportsball metaphor, I think in the second half our players will finally be able to get out on the pitch and give it the proverbial 110%. Support for garbage collection is coming to WebAssembly users, and I think even by the end of the year it will be shipping in major browsers. This is going to be big! We have a chance and we need to sieze it.

Scheme to Wasm

Spritely + Igalia working on Scheme to WebAssembly

Avoid truncating language to platform; bring whole self

  • Value representation
  • Varargs
  • Tail calls
  • Delimited continuations
  • Numeric tower

Even with GC, though, WebAssembly is still a weird machine. It would help to see the concrete approaches that some languages of interest manage to take when compiling to WebAssembly.

In that spirit, the rest of this article/presentation is a walkthough of the approach that I am taking as I work on a WebAssembly compiler for Scheme. (Thanks to Spritely for supporting this work!)

Before diving in, a meta-note: when you go to compile a language to, say, JavaScript, you are mightily tempted to cut corners. For example you might implement numbers as JavaScript numbers, or you might omit implementing continuations. In this work I am trying to not cut corners, and instead to implement the language faithfully. Sometimes this means I have to work around weirdness in WebAssembly, and that's OK.

When thinking about Scheme, I'd like to highlight a few specific areas that have interesting translations. We'll start with value representation, which stays in the GC theme from the introduction.

Scheme to Wasm: Values

;;       any  extern  func
;;        |
;;        eq
;;     /  |   \
;; i31 struct  array

The unitype: (ref eq)

Immediate values in (ref i31)

  • fixnums with 30-bit range
  • chars, bools, etc

Explicit nullability: (ref null eq) vs (ref eq)

The GC extensions for WebAssembly are phrased in terms of a type system. Oddly, there are three top types; as far as I understand it, this is the result of a compromise about how WebAssembly engines might want to represent these different kinds of values. For example, an opaque JavaScript value flowing into a WebAssembly program would have type (ref extern). On a system with NaN boxing, you would need 64 bits to represent a JS value. On the other hand a native WebAssembly object would be a subtype of (ref any), and might be representable in 32 bits, either because it's a 32-bit system or because of pointer compression.

Anyway, three top types. The user can define subtypes of struct and array, instantiate values of those types, and access their fields. The life cycle of reference-typed objects is automatically managed by the run-time, which is just another way of saying they are garbage-collected.

For Scheme, we need a common supertype for all values: the unitype, in Bob Harper's memorable formulation. We can use (ref any), but actually we'll use (ref eq) -- this is the supertype of values that can be compared by (pointer) identity. So now we can code up eq?:

(func $eq? (param (ref eq) (ref eq))
           (result i32)
  (ref.eq (local.get a) (local.get b)))

Generally speaking in a Scheme implementation there are immediates and heap objects. Immediates can be encoded in the bits of a value, whereas for heap object the bits of a value encode a reference (pointer) to an object on the garbage-collected heap. We usually represent small integers as immediates, as well as booleans and other oddball values.

Happily, WebAssembly gives us an immediate value type, i31. We'll encode our immediates there, and otherwise represent heap objects as instances of struct subtypes.

Scheme to Wasm: Values (2)

Heap objects subtypes of struct; concretely:

(struct $heap-object
  (struct (field $tag-and-hash i32)))
(struct $pair
  (sub $heap-object
    (struct i32 (ref eq) (ref eq))))

GC proposal allows subtyping on structs, functions, arrays

Structural type equivalance: explicit tag useful

We actually need to have a common struct supertype as well, for two reasons. One is that we need to be able to hash Scheme values by identity, but for this we need an embedded lazily-initialized hash code. It's a bit annoying to take the per-object memory hit but it's a reality, and the JVM does it this way, so it must not be so terrible.

The other reason is more subtle: WebAssembly's type system is built in such a way that types that are "structurally" equivalent are indistinguishable. So a pair has two fields, besides the hash, but there might be a number of other fundamental object types that have the same shape; you can't fully rely on WebAssembly's dynamic type checks (ref.test et al) to be able to query the type of a value. Instead we re-use the low bits of the hash word to include a type tag, which might be 1 for pairs, 2 for vectors, 3 for closures, and so on.

Scheme to Wasm: Values (3)

(func $cons (param (ref eq)
                   (ref eq))
            (result (ref $pair))
  (struct.new_canon $pair
    ;; Assume heap tag for pairs is 1.
    (i32.const 1)
    ;; Car and cdr.
    (local.get 0)
    (local.get 1)))

(func $%car (param (ref $pair))
            (result (ref eq))
  (struct.get $pair 1 (local.get 0)))

With this knowledge we can define cons, as a simple call to struct.new_canon pair.

I didn't have time for this in the talk, but there is a ghost haunting this code: the ghost of nominal typing. See, in a web browser at least, every heap object will have its first word point to its "hidden class" / "structure" / "map" word. If the engine ever needs to check that a value is of a specific shape, it can do a quick check on the map word's value; if it needs to do deeper introspection, it can dereference that word to get more details.

Under the hood, testing whether a (ref eq) is a pair or not should be a simple check that it's a (ref struct) (and not a fixnum), and then a comparison of its map word to the run-time type corresponding to $pair. If subtyping of $pair is allowed, we start to want inline caches to handle polymorphism, but the checking the map word is still the basic mechanism.

However, as I mentioned, we only have structural equality of types; two (struct (ref eq)) type definitions will define the same type and have the same map word (run-time type / RTT). Hence the _canon in the name of struct.new_canon $pair: we create an instance of $pair, with the canonical run-time-type for objects having $pair-shape.

In earlier drafts of the WebAssembly GC extensions, users could define their own RTTs, which effectively amounts to nominal typing: not only does this object have the right structure, but was it created with respect to this particular RTT. But, this facility was cut from the first release, and it left ghosts in the form of these _canon suffixes on type constructor instructions.

For the Scheme-to-WebAssembly effort, we effectively add back in a degree of nominal typing via type tags. For better or for worse this results in a so-called "open-world" system: you can instantiate a separately-compiled WebAssembly module that happens to define the same types and use the same type tags and it will be able to happily access the contents of Scheme values from another module. If you were to use nominal types, you would't be able to do so, unless there were some common base module that defined and exported the types of interests, and which any extension module would need to import.

(func $car (param (ref eq)) (result (ref eq))
  (local (ref $pair))
  (block $not-pair
    (br_if $not-pair
      (i32.eqz (ref.test $pair (local.get 0))))
    (local.set 1 (ref.cast $pair) (local.get 0))
    (br_if $not-pair
      (i32.ne
        (i32.const 1)
        (i32.and
          (i32.const 0xff)
          (struct.get $heap-object 0 (local.get 1)))))
    (return_call $%car (local.get 1)))

  (call $type-error)
  (unreachable))

In the previous example we had $%car, with a funny % in the name, taking a (ref $pair) as an argument. But in the general case (barring compiler heroics) car will take an instance of the unitype (ref eq). To know that it's actually a pair we have to make two checks: one, that it is a struct and has the $pair shape, and two, that it has the right tag. Oh well!

Scheme to Wasm

  • Value representation
  • Varargs
  • Tail calls
  • Delimited continuations
  • Numeric tower

But with all of that I think we have a solid story on how to represent values. I went through all of the basic value types in Guile and checked that they could all be represented using GC types, and it seems that all is good. Now on to the next point: varargs.

Scheme to Wasm: Varargs

(list 'hey)      ;; => (hey)
(list 'hey 'bob) ;; => (hey bob)

Problem: Wasm functions strongly typed

(func $list (param ???) (result (ref eq))
  ???)

Solution: Virtualize calling convention

In WebAssembly, you define functions with a type, and it is impossible to call them in an unsound way. You must call $car exactly 2 arguments or it will not compile, and those arguments have to be of specific types, and so on. But Scheme doesn't enforce these restrictions on the language level, bless its little miscreant heart. You can call car with 5 arguments, and you'll get a run-time error. There are some functions that can take a variable number of arguments, doing different things depending on incoming argument count.

How do we square these two approaches to function types?

;; "Registers" for args 0 to 3
(global $arg0 (mut (ref eq)) (i31.new (i32.const 0)))
(global $arg1 (mut (ref eq)) (i31.new (i32.const 0)))
(global $arg2 (mut (ref eq)) (i31.new (i32.const 0)))
(global $arg3 (mut (ref eq)) (i31.new (i32.const 0)))

;; "Memory" for the rest
(type $argv (array (ref eq)))
(global $argN (ref $argv)
        (array.new_canon_default
          $argv (i31.const 42) (i31.new (i32.const 0))))

Uniform function type: argument count as sole parameter

Callee moves args to locals, possibly clearing roots

The approach we are taking is to virtualize the calling convention. In the same way that when calling an x86-64 function, you pass the first argument in $rdi, then $rsi, and eventually if you run out of registers you put arguments in memory, in the same way we'll pass the first argument in the $arg0 global, then $arg1, and eventually in memory if needed. The function will receive the number of incoming arguments as its sole parameter; in fact, all functions will be of type (func (param i32)).

The expectation is that after checking argument count, the callee will load its arguments from globals / memory to locals, which the compiler can do a better job on than globals. We might not even emit code to null out the argument globals; might leak a little memory but probably would be a win.

You can imagine a world in which $arg0 actually gets globally allocated to $rdi, because it is only live during the call sequence; but I don't think that world is this one :)

Scheme to Wasm

  • Value representation
  • Varargs
  • Tail calls
  • Delimited continuations
  • Numeric tower

Great, two points out of the way! Next up, tail calls.

Scheme to Wasm: Tail calls

;; Call known function
(return_call $f arg ...)

;; Call function by value
(return_call_ref $type callee arg ...)

Friends -- I almost cried making this slide. We Schemers are used to working around the lack of tail calls, and I could have done so here, but it's just such a relief that these functions are just going to be there and I don't have to think much more about them. Technically speaking the proposal isn't merged yet; checking the phases document it's at the last station before headed to the great depot in the sky. But, soon soon it will be present and enabled in all WebAssembly implementations, and we should build systems now that rely on it.

Scheme to Wasm

  • Value representation
  • Varargs
  • Tail calls
  • Delimited continuations
  • Numeric tower

Next up, my favorite favorite topic: delimited continuations.

Scheme to Wasm: Prompts (1)

Problem: Lightweight threads/fibers, exceptions

Possible solutions

  • Eventually, built-in coroutines
  • binaryen’s asyncify (not yet ready for GC); see Julia
  • Delimited continuations

“Bring your whole self”

Before diving in though, one might wonder why bother. Delimited continuations are a building-block that one can use to build other, more useful things, notably exceptions and light-weight threading / fibers. Could there be another way of achieving these end goals without having to implement this relatively uncommon primitive?

For fibers, it is possible to implement them in terms of a built-in coroutine facility. The standards body seems willing to include a coroutine primitive, but it seems far off to me; not within the next 3-4 years I would say. So let's put that to one side.

There is a more near-term solution, to use asyncify to implement coroutines somehow; but my understanding is that asyncify is not ready for GC yet.

For the Guile flavor of Scheme at least, delimited continuations are table stakes of their own right, so given that we will have them on WebAssembly, we might as well use them to implement fibers and exceptions in the same way as we do on native targets. Why compromise if you don't have to?

Scheme to Wasm: Prompts (2)

Prompts delimit continuations

(define k
  (call-with-prompt ’foo
    ; body
    (lambda ()
      (+ 34 (abort-to-prompt 'foo)))
    ; handler
    (lambda (continuation)
      continuation)))

(k 10)       ;; ⇒ 44
(- (k 10) 2) ;; ⇒ 42

k is the _ in (lambda () (+ 34 _))

There are a few ways to implement delimited continuations, but my usual way of thinking about them is that a delimited continuation is a slice of the stack. One end of the slice is the prompt established by call-with-prompt, and the other by the continuation of the call to abort-to-prompt. Capturing a slice pops it off the stack, copying it out to the heap as a callable function. Calling that function splats the captured slice back on the stack and resumes it where it left off.

Scheme to Wasm: Prompts (3)

Delimited continuations are stack slices

Make stack explicit via minimal continuation-passing-style conversion

  • Turn all calls into tail calls
  • Allocate return continuations on explicit stack
  • Breaks functions into pieces at non-tail calls

This low-level intuition of what a delimited continuation is leads naturally to an implementation; the only problem is that we can't slice the WebAssembly call stack. The workaround here is similar to the varargs case: we virtualize the stack.

The mechanism to do so is a continuation-passing-style (CPS) transformation of each function. Functions that make no calls, such as leaf functions, don't need to change at all. The same goes for functions that make only tail calls. For functions that make non-tail calls, we split them into pieces that preserve the only-tail-calls property.

Scheme to Wasm: Prompts (4)

Before a non-tail-call:

  • Push live-out vars on stacks (one stack per top type)
  • Push continuation as funcref
  • Tail-call callee

Return from call via pop and tail call:

(return_call_ref (call $pop-return)
                 (i32.const 0))

After return, continuation pops state from stacks

Consider a simple function:

(define (f x y)
  (+ x (g y))

Before making a non-tail call, a "tailified" function will instead push all live data onto an explicitly-managed stack and tail-call the callee. It also pushes on the return continuation. Returning from the callee pops the return continuation and tail-calls it. The return continuation pops the previously-saved live data and continues.

In this concrete case, tailification would split f into two pieces:

(define (f x y)
  (push! x)
  (push-return! f-return-continuation-0)
  (g y))

(define (f-return-continuation-0 g-of-y)
  (define k (pop-return!))
  (define x (pop! x))
  (k (+ x g-of-y)))

Now there are no non-tail calls, besides calls to run-time routines like push! and + and so on. This transformation is implemented by tailify.scm.

Scheme to Wasm: Prompts (5)

abort-to-prompt:

  • Pop stack slice to reified continuation object
  • Tail-call new top of stack: prompt handler

Calling a reified continuation:

  • Push stack slice
  • Tail-call new top of stack

No need to wait for effect handlers proposal; you can have it all now!

The salient point is that the stack on which push! operates (in reality, probably four or five stacks: one in linear memory or an array for types like i32 or f64, three for each of the managed top types any, extern, and func, and one for the stack of return continuations) are managed by us, so we can slice them.

Someone asked in the talk about whether the explicit memory traffic and avoiding the return-address-buffer branch prediction is a source of inefficiency in the transformation and I have to say, yes, but I don't know by how much. I guess we'll find out soon.

Scheme to Wasm

  • Value representation
  • Varargs
  • Tail calls
  • Delimited continuations
  • Numeric tower

Okeydokes, last point!

Scheme to Wasm: Numbers

Numbers can be immediate: fixnums

Or on the heap: bignums, fractions, flonums, complex

Supertype is still ref eq

Consider imports to implement bignums

  • On web: BigInt
  • On edge: Wasm support module (mini-gmp?)

Dynamic dispatch for polymorphic ops, as usual

First, I would note that sometimes the compiler can unbox numeric operations. For example if it infers that a result will be an inexact real, it can use unboxed f64 instead of library routines working on heap flonums ((struct i32 f64); the initial i32 is for the hash and tag). But we still need a story for the general case that involves dynamic type checks.

The basic idea is that we get to have fixnums and heap numbers. Fixnums will handle most of the integer arithmetic that we need, and will avoid allocation. We'll inline most fixnum operations as a fast path and call out to library routines otherwise. Of course fixnum inputs may produce a bignum output as well, so the fast path sometimes includes another slow-path callout.

We want to minimize binary module size. In an ideal compile-to-WebAssembly situation, a small program will have a small module size, down to a minimum of a kilobyte or so; larger programs can be megabytes, if the user experience allows for the download delay. Binary module size will be dominated by code, so that means we need to plan for aggressive dead-code elimination, minimize the size of fast paths, and also minimize the size of the standard library.

For numbers, we try to keep module size down by leaning on the platform. In the case of bignums, we can punt some of this work to the host; on a JavaScript host, we would use BigInt, and on a WASI host we'd compile an external bignum library. So that's the general story: inlined fixnum fast paths with dynamic checks, and otherwise library routine callouts, combined with aggressive whole-program dead-code elimination.

Scheme to Wasm

  • Value representation
  • Varargs
  • Tail calls
  • Delimited continuations
  • Numeric tower

Hey I think we did it! Always before when I thought about compiling Scheme or Guile to the web, I got stuck on some point or another, was tempted down the corner-cutting alleys, and eventually gave up before starting. But finally it would seem that the stars are aligned: we get to have our Scheme and run it too.

Miscellenea

Debugging: The wild west of DWARF; prompts

Strings: stringref host strings spark joy

JS interop: Export accessors; Wasm objects opaque to JS. externref.

JIT: A whole ’nother talk!

AOT: wasm2c

Of course, like I said, WebAssembly is still a weird machine: as a compilation target but also at run-time. Debugging is a right proper mess; perhaps some other article on that some time.

How to represent strings is a surprisingly gnarly question; there is tension within the WebAssembly standards community between those that think that it's possible for JavaScript and WebAssembly to share an underlying string representation, and those that think that it's a fool's errand and that copying is the only way to go. I don't know which side will prevail; perhaps more on that as well later on.

Similarly the whole interoperation with JavaScript question is very much in its early stages, with the current situation choosing to err on the side of nothing rather than the wrong thing. You can pass a WebAssembly (ref eq) to JavaScript, but JavaScript can't do anything with it: it has no prototype. The state of the art is to also ship a JS run-time that wraps each wasm object, proxying exported functions from the wasm module as object methods.

Finally, some language implementations really need JIT support, like PyPy. There, that's a whole 'nother talk!

WebAssembly for the rest of us

With GC, WebAssembly is now ready for us

Getting our languages on WebAssembly now a S.M.O.P.

Let’s score some goals in the second half!

(visit-links
 "gitlab.com/spritely/guile-hoot-updates"
 "wingolog.org"
 "wingo@igalia.com"
 "igalia.com"
 "mastodon.social/@wingo")

WebAssembly has proven to have some great wins for C, C++, Rust, and so on -- but now it's our turn to get in the game. GC is coming and we as a community need to be getting our compilers and language run-times ready. Let's put on the coffee and bang some bytes together; it's still early days and there's a world to win out there for the language community with the best WebAssembly experience. The game is afoot: happy consing!

Writing elsewhere; the pain of moving platforms

I’ve been doing a lot of writing elsewhere of late. Some links:

  • I’ve written a fair amount in the past year for the Tidelift blog, most recently on the EU’s Cyber Resiliency Act and what it might mean for open source.
  • I wrote last week at opensource.com; the latest in a now multi-year series on board candidates in elections for the Open Source Initiative.
  • I have a newsletter on the intersection of open and machine learning at openml.fyi. It is fun!
  • I’ve moved to the fediverse for most of my social media—I’m social.coop/@luis_in_brief (and you can subscribe to this blog via the fediverse at @lu.is/@admin).

I don’t love (mostly) leaving Twitter; as I’ve said a few times, the exposure to different people there helped make me a better person. But one of my primary political concerns is the rise of fascism in the US, and that absolutely includes Elon and the people who enable him. I can’t quit cold-turkey; unfortunately, too many things I care about (or need to follow for work) haven’t left. But I can at least sleep well.

March 18, 2023

Integrating the RawTherapee engine

RawTherapee is one of the two major open source RAW photo processing applications, the other is Darktable.

Can I leverage RawTherapee RAW processing code for use in Niepce? Yes I can.

So let's review of I did it.

Preamble

License-wise GPL-3.0 is a match.

In term of tech stack, there are a few complexities.

  1. RawTherapee is written in C++, while Niepce is being converted to Rust. Fortunately it's not really an issue, it require just a bit of work, even at the expense of writing a bit of C++.
  2. It is not designed to be used as a library: it's an application. Fortunately there is a separation between the engine (rtengine) and the UI (rtgui) which will make our life easier. There are a couple of places where this separation blurs, but nothing that can't be fixed.
  3. The UI toolkit is gtkmm 3.0. This is a little point of friction here as Niepce uses GTK4 with some leftovers C++ code using gtkmm 4.0. We are porting the engine, so it shouldn't matter except that the versions of neither glibmm nor cairomm match (ie they are incompatible) and the engine relies on them heavily.
  4. Build system: it uses CMake. Given that RawTherapee is not meant to be built as a library, changes will be required. I will take a different approach though.

Organization

The code will not be imported in the repository and instead will be used as a git submodule. I already have cxx that way for the code generator. Given that some code needs to be changed, it will reference my own fork of RawTherapee, based on 5.9, with as much as I can upstreamed.

The Rust wrappers will live in their own crate: Niepce application code is setup as a workspace with 4 crates: npc-fwk, npc-engine, npc-craw and niepce. This would be the fifth: rtengine.

The rtengine crate will provide the API for the Rust code. No C++ will be exposed.

npc-craw (Niepce Camera Raw1), as it is meant to implement the whole image processing pipeline, will use this crate. We'll create a trait for the pipeline and implement it for both the ncr pipeline and rtengine.

Integrating

Build system

Niepce wrap everything into a meson build. So to build rtengine we will build a static library and install the supporting file. We have to bring in a lot of explicit dependencies, which bring a certain amount bloat, but we can see later if there is a way to reduce this. It's tedious to assemble everything.

The first build didn't include everything needed. I had to fix this as I was writing the wrappers.

Dependencies

glibmm and cairomm: the version used for gtkmm-3.0 and gtkmm-4.0 differs. glibmm changed a few things like some enum are now C++ enum class (better namespacing), and Glib::RefPtr<> is now a std::shared_ptr<>. The biggest hurdle is the dependency on the concurrency features of glibmm (Glib::Mutex) that got completely removed in glibmm-2.68 (gtkmm-3.0 uses glibmm-2.4). I did a rough port to use the C++ library, and upstream has a languishing work in progress pull request. Other changes include adding explicit includes. I also need to remove gtkmm dependencies leaking into the engine.

Rust wrapper

I recommend heavily to make sure you can build your code with the address sanitizer. In the case of Niepce, I have had it for a long time, and made sure it still worked when I inverted the build order to link the main binary with Rust instead of C++.

Using cxx I created a minimum interface to the C++ code. The problem was to understand how it works. Fortunately the command line interface for RawTherapee does exactly that. This is the logic we'll follow in the Rust code.

Lets create the bridge module. We need to bridge the following types:

  • InitialImage which represents the image to process.
  • ProcParams which represents the parameters for processing the the image.
  • PartialProfile which is used to populate the ProcParams from a processing profile.
  • ProcessingJob which represents the job of processing the image.
  • ImageIO which is one of the classes the processed image data inherit from, the one that implement getting the scanlines.

Ownership is a bit complicated you should pay attention how these types get cleaned up. For example a ProcessingJob ownership get transfered to the processImage() function, unless there is an error, in which case there is a destroy() function (it's a static method) to call. While PartialProfile needs deleteInstance() to be called before being destroyed, or it will leak.

Example:

let mut proc_params = ffi::proc_params_new();
let mut raw_params = unsafe {
    ffi::profile_store_load_dynamic_profile(image.pin_mut().get_meta_data())
};
ffi::partial_profile_apply_to(&raw_params, proc_params.pin_mut(), false);

We have created proc_params as a UniquePtr<ProcParams>. We obtain a raw_params as a UniquePtr<PartialProfile>. UniquePtr<> is like a Box<> but for use when coming from a C++ std::unique_ptr<>.

raw_params.pin_mut().delete_instance();

raw_params will be freed when getting out of scope, but if you don't call delete_instance() (the function is renamed in the bridge to follow Rust conventions), memory will leak. The pin_mut() is necessary to obtain a Pin<> of the pointer for a mutable pointer required as the instance.

let job = ffi::processing_job_create(
    image.pin_mut(),
    proc_params.as_ref().unwrap(),
    false,
);
let mut error = 0_i32;
// Warning: unless there is an error, process_image will consume it.
let job = job.into_raw();
let imagefloat = unsafe { ffi::process_image(job, &mut error, false) };
if imagefloat.is_null() {
    // Only in case of error.
    unsafe { ffi::processing_job_destroy(job) };
    return Err(Error::from(error));
}

This last bit, we create the job as a UniquePtr<ProcessingJob> but then we have to obtain the raw pointer to sink either with process_image(), or in case of error, sink with processing_job_destroy(). into_raw() do consume the UniquePtr<>.

image is also is a UniquePtr<InitialImage> and InitialImage has a decreaseRef() to unref the object that must be called to destroy the object. It would be called like this:

unsafe { ffi::decrease_ref(image.into_raw()) };

Most issues got detected with libasan, either as memory errors or as memory leaks. There is a lot of pointer manipulations, but let's limit this to the bridge and not expose it ; at least unlike in C++, cxx::UniquePtr<> consume the smart pointer when turning it into a raw pointer, there is no risk to use it again, at least in the Rust code.

Also, some glue code needed to be written as some function take Glib::ustring instead of std::string, constructors needs to be wrapped to return UniquePtr<>. Multiple inheritence make some direct method call not possible, and static methods are still work in progress with cxx.

One good way to test this was to write a simple command line program. As the code shown above, it's tricky to use correctly, so I wrote a safe API to use the engine, one that is more in line with Niepce "architecture".

At that point rendering an image is the following code:

use rtengine::RtEngine;

let engine = RtEngine::new();
if engine.set_file(filename, true /* is_raw */).is_err() {
    std::process::exit(3);
}

match engine.process() {
    Err(error) => {
        println!("Error, couldn't render image: {error}");
        std::process::exit(2);
    }
    Ok(image) => {
        image.save_png("image.png").expect("Couldn't save image");
    }
}

Results

I have integrated it in the app. For now switching rendering engine needs a code change, there is a bit more work to integrate rendering parameters to the app logic.

Here is how a picture from my Canon G7X MkII looked with the basic pipeline from ncr:

ncr rendering

Here is how it looks with the RawTherapee engine:

RawTherapee engine rendering

As you can notice, lens correction is applied.

1

there is an unrelated ncr crate on crates.io, so I decided to not use that crate name, and didn't want to use npc-ncr, even though the crate is private to the application and not intended to be published separately.

March 17, 2023

Mushroom Gardiane Recipe

Gardiane is a traditional recipe from Camargue, this vegan variant replaces streaked bull meat by mushrooms. I tried it with seitan too but it ends up a bit too soggy and mushy to my liking, especially after some time in the fridge. This vegan variant also has the advantage of requiring virtually no set up — in the original recipe you have to marinate the meat for 24–48 hours — and of taking less time to cook, as you only need 1–2 hours instead of 2–4 hours.

Regarding olives, the most traditional would be black Picholines, though I prefer to use black Lucques. Please please please do not use greek style olives as they contain way too much salt, they would spoil the gardiane. I strongly recommend using olives with stones as it helps keeping them in good shape while everything cooks, and removing the stones with your mouth is part of the experience, like cherry stones in clafoutis.

For 6–8 servings.

Ingredients

  • 2 kg of large white or brown common mushrooms
  • 200 g of firm tofu
  • 200 g of smoked firm tofu
  • 500 g of black olives, preferably stoned Picholines or Lucques
  • 2 large yellow onions
  • 75 cl of red wine, preferably Côte du Rhône or Languedoc
  • 6 garlic cloves
  • ⅓ tsp of ground cloves
  • 2 tsp of thyme
  • 2 tsp of rosemary
  • 2 large bay leaves
  • a bit of ground black pepper
  • 1 tsp of salt
  • 3 tbsp of starch or flower
  • 500 g long grain white rice, preferably Camargue
  • some olive oil
  • some water

This day I couldn’t find Picholines or Lucques, and I forgot to invite the tofu to the familly photo

Instructions

  • Remove the stems from the mushrooms. Remove dirty bits from the stems.
  • Peel the mushroom caps, this will help the wine soak in.
  • Cut the caps into 3–4 cm wide pieces, in pratice that means you don’t cut the small ones, you cut the medium-sized ones in two, and you cut the larger ones in three or four.
  • Wash the caps and stems in a salad spinner.

The mushrooms, peeled, cut and washed

  • Cut the tofu in 2 cm long and 5 mm thick batonnets.

The cut smoked tofu

  • Peel the onions. Cut the onions in 8–10 vertically.
  • Peel the garlic cloves. Mince the garlic cloves.
  • If the olives come in water, remove the water.
  • Put a large pot on medium heat.
  • In the pot, put a bit of olive oil and lightly brown the onions with the garlic, the thyme and the rosemary. Reserve.

The browned onions

  • In the pot, put a bit of olive oil and lightly brown the tofu. Reserve.

The browned tofu

  • In the pot, put a bit of olive oil and lightly brown the mushrooms. Do it in several batches so they don’t cook too much in their own water, the goal is to get somw brown bits.

A batch of browned mushrooms, these cooked in their own water a bit too much, but it’s fine as they still have some brown bits

  • Remove the pot from the fire, and pour everything in it but the starch or flour, the black pepper and the rice. This means you add the wine, the mushrooms, the onions, the tofu, the olives, the bay leaves, the ground cloves, and the salt.
  • Add water in the pot until everything is covered.

The pot is filled with all the ingredients, the gardiane is ready to be cooked

  • Cover the pot, put it on medium-low heat and let it cook for 2 hours.
  • While the gardiane cooks, prepare and cook the rice.
  • Remove the pot from the stove.
  • Put some sauce from the gardiane in a glass with the starch or flour. Mix them with a spoon and put the liquid back in the pot.
  • Add the black pepper in the pot and stir, the starch or flour will very lightly thicken the sauce.

A served plate

  • Serve the gardiane either aside the rice or on top of it, as you prefer. Serve with another bottle of a similar wine.

This recipe is approved by Tobias Bernard

Libadwaita 1.3

Another cycle, another release. Let’s take a look at what’s new.

Banners

Screenshot of AdwBanner

AdwBanner is a brand new widget that replaces GtkInfoBar.

Jamie started implementing it before 1.2 was released, but we were already in the API freeze so it was delayed to this cycle instead.

While it looks more or less the same as GtkInfoBar, it’s not a direct replacement. AdwBanner has a title and optionally one button. That’s it. It does not have a close button, it cannot have multiple buttons or arbitrary children. In exchange, it’s easier to use, behaves consistently and has an adaptive layout:

Wide screenshot of AdwBanner. The button is on the right, the title is centered.

Medium-width screenshot of AdwBanner. The button is on the right, the title is left-aligned as it wouldn't fit centered.

Narrow screenshot of AdwBanner. The title is center-aligned, the button is centered below it.

Like GtkInfoBar, AdwBanner has a built-in revealer and can be shown and hidden with an animation.

There are situations where it cannot be used, but in most of those cases GtkInfobar was already the wrong choice and they should be redesigned. For example, Epiphany was using them for the “save password” prompt, and is using a popover for that now.

Tab Overview

A work-in-progress grid-based tab overview A work-in-progress carousel-based tab overview

AdwTabOverview is a new tab overview widget for AdwTabView that finally makes it possible to use tabs on mobile devices without implementing a mobile switcher manually.

Back when I wrote HdyTabView, I mentioned a tab overview widget in the works, and even had demo screenshots. Of course, basically everything from that demo got rewritten since then, and the carousel for narrow mode got scrapped completely, but now we’re getting into The Ship of Theseus territory.

This required a pretty big rework of AdwTabView to allow tabs to have thumbnails when they are not visible, and in particular it does not use a GtkStack internally anymore.

By default the selected tab has a live thumbnail and other thumbnails are static, but apps can opt into using live thumbnails for specific pages. They can also control the thumbnail alignment in case the thumbnail gets clipped. Thumbnails themselves are currently not public, but it might be interesting to use them for e.g. tooltips at some point.

Overview is not currently used very widely – it’s available in Console, and there is a merge request adding it to Epiphany, but I didn’t have the energy to finish the latter this cycle.

Tab Button

Screenshot of AdwTabButton. It's showing two tab buttons. One displays 3 open tabs, the other one 15 and has an attention indicator

AdwTabButton is much less interesting, it’s just a button that shows the number of open tabs in a given AdwTabView. It’s intended to be used as the button that opens the tab overview on mobile devices.

Unlike tab overview, this widget is more or less a direct port of what Epiphany has been using since 3.34. It does have one new feature though – it can display an indicator if a tab needs attention.

Accessibility

You might have noticed that widgets like AdwViewStack, AdwTabView or AdwEntryRow were not accessible in libadwaita 1.2.x. The reason for that is that GTK didn’t provide public API to make that possible. While GtkAccessible existed, it wasn’t possible to implement it outside GTK itself.

This cycle, Lukáš Tyrychtr has implemented the missing pieces, as well as fixed a few other issues. And so libadwaita widgets are now properly accessible.

Animation Additions

One of AdwAnimation features is that it automatically follows the system setting for disabling animations. While this is the expected behavior in most cases, there are a few where it gets in the way instead.

One of these cases is in apps where the animations are the app’s primary content, such as Elastic, and so I added a property that allows a specific animation to ignore the setting.

The animation in the demo is now using it as well, for the same reason as Elastic, and in the future it will allow us to have working spinners while animations are disabled system-wide as well.

A screenshot of an animation graph from Elastic

Elastic also draws an animation graph before running the actual animation, and while for timed animations it’s relatively easy since easing functions are available through the public API, it wasn’t possible for spring animations. It is now, via calculate_value() and calculate_velocity().

All of this uncovered a few bugs with spring animations – for example, the velocity was completely wrong for overdamped springs, and Manuel Genovés fixed them.

Unrelated to the above, a common complaint about AdwPropertyAnimationTarget was that it prints a critical if the object containing the property it’s animating is finalized before the target. While it was easy to avoid in C, it was nearly impossible from bindings. And so we don’t print that critical anymore.

Other Changes

  • Christopher Davis added a way to make AdwActionRow subtitle selectable.
  • Matt Jakeman added title-lines and subtitle-lines properties to AdwExpanderRow, matching AdwActionRow.
  • AdwEntryRow now has a grab_focus_without_selecting() method, matching GtkEntry.
  • The API to set an icon on AdwActionRow and AdwExpanderRow are now deprecated, since they were mostly unused. Apps that need icons can add a GtkImage as a prefix widget instead.
  • AdwMessageDialog now has the async choose() method, matching the new GTK dialogs like GtkAlertDialog. The response signal is still there and is not deprecated, but in some cases the new method may be more convenient, particularly from bindings:

    [GtkCallback]
    private async void clicked_cb () {
        var dialog = new Adw.MessageDialog (
            this,
            "Replace File?",
            "A file named “example.png” already exists. Do you want to replace it?"
        );
    
        dialog.add_response ("cancel", "_Cancel");
        dialog.add_response ("replace", "_Replace");
    
        dialog.set_response_appearance (
            "replace",
            DESTRUCTIVE
        );
    
        var response = yield dialog.choose (null);
    
        if (response == "replace") {
            // handle replacing
        }
    }
    
  • Corey Berla added missing drag-n-drop related API to AdwTabBar to make it work properly in Nautilus.
  • Since GTK now allows to change texture filtering, AdwAvatar properly scales custom images, so they don’t appear pixelated when downscaled or blurry when upscaled. This only works if the custom image is a GdkTexture – if your app is using a different GdkPaintable, you will need to do the equivalent change yourself.
  • Jason Francis implemented dark style and high contrast support when running on Windows.
  • Selected items in lists and grids are now using accent color instead of grey, same as Nautilus in 43. Sidebars and menus still look the same as before.

New Dependencies

The accessibility additions, scaled texture render nodes in AdwAvatar, as well as mask render nodes (that I didn’t mention because it’s an internal change) and deprecation fixes mean that libadwaita 1.3 requires GTK 4.10 instead of 4.6.


As always, thanks to all the contributors, and thanks to my employer, Purism, for letting me work on libadwaita and GTK to make this release happen.

Status update 17/03/2023

Hello from my parents place, sitting on the border of Wales & England, listening to this excellent Victor Rice album, thinking of this time last year when I actually got to watch him play at Freedom Sounds Festival, which was one of my first adventures of the post-lockdown 2020s.

I have many distractions at the moment, many being work/life admin but here are some of the more interesting ones:

  • Playing in a new band in Santiago – Killo Karallo – recording some initial music which is to come out next week
  • Preparing new Vladimir Chicken music, also cooked and ready for release in April
  • Figuring out how we can grow the GNOME OpenQA tests while keeping them fun to work with. Here’s an experimental commandline tool which might help with that.
  • Learning about marketing, analytics, and search engine optimization.
  • Trying out the new LLaMA language model and generally trying to keep up with the ongoing revolution in content generation technology.

Also I got to see real snow for the first time in a few years! Thanks Buxton!



#87 Editable Shortcuts

Update on what happened across the GNOME project in the week from March 10 to March 17.

GNOME Development Tools

GNOME Builder

IDE for writing GNOME-based software.

hergertme says

Just in time for GNOME 44, Builder has gained support for editable shortcuts. You can individually override a number of shortcuts from the Preferences dialog. Popover menus and the Keyboard Shortcuts window will reflect changes made by the user.

GNOME Core Apps and Libraries

Libadwaita

Building blocks for modern GNOME apps using GTK4.

Alex (they/them) says

libadwaita 1.3 is out: https://blogs.gnome.org/alexm/2023/03/17/libadwaita-1-3/

GNOME Circle Apps and Libraries

Sophie 🏳️‍🌈 🏳️‍⚧️ ✊ reports

This week, Clairvoyant joined GNOME Circle. Clairvoyant gives you psychic answers to your questions. For free. Congratulations!

Third Party Projects

Casper Meijn reports

This week I released version 0.3.0 of Read It Later. This is a client for Wallabag, which allows you to save web articles and read them later. The significant changes are the upgrade to GTK 4, some fixed bugs, and new translations. Download on Flathub.

Tube Converter

An easy-to-use video downloader (yt-dlp frontend).

Nick announces

Tube Converter V2023.3.0 is here! Introducing the all-new Tube Converter completely rewritten in C#. First off I must thank @fsobolev and @DaPigGuy because this release would not be possible without them ❤️ Besides being rewritten in C#, this release includes many new features compared to the previous C++ version: Users can now download playlists, there is a new queue system to more easily manage downloads, plus a redesigned interface with support for smaller screens. We can’t wait for all of you to try this release 🥳

Portfolio

A minimalist file manager for those who want to use Linux mobile devices.

Martín Abente Lahaye says

After a long hiatus, a new release of Portfolio is out 📱🤓. This new release comes with important bug fixes, small-detail additions and a few visual improvements. Download on Flathub.

Fractal

Matrix messaging app for GNOME written in Rust.

Kévin Commaille says

As predicted 3 weeks ago, Fractal 4.4.2 has been released and is available on Flathub.

As a reminder, there are no new features, but it makes Fractal compatible with newer versions of our dependencies. Which means our Flatpak has been rebased on top of the GNOME 43 runtime just in time for the release of GNOME 44! Big thanks to everyone who tested the beta version and feel free to uninstall it.

In another news, let’s talk a bit about Fractal 5…

We have finally implemented one of the most annoying missing features: sending read receipts and updating the fully-read marker. What is doubly great about this, is that we are now only one feature away from being regression-free compared to our stable release (and a merge request is opened for the last one)!

This means that a beta release is around the corner, but we also have serious performance issues to resolve first. We hope that switching to the new store backend currently developped in the Matrix Rust SDK will fix some of it, but we still need to investigate properly how we can improve the situation.

That’s all for this week, don’t hesitate to come say hi in #fractal:gnome.org. If you would like to help us, take a look at our newcomers issues (and don’t forget to read our CONTRIBUTING.md first).

Denaro

A personal finance manager.

Nick announces

Denaro V2023.3.0-beta2 is here! This release features more customization for an account’s custom currency, the ability to add a password to an exported PDF file, and many fixes when importing QIF and OFX files :) Here’s the full changelog across betas:

  • Added the ability to customize the decimal and group separators used per account
  • Added the ability to password protect a PDF file
  • Fixed an issue where OFX files with security could not be imported
  • Fixed an issue where QIF files could not be imported on non-English systems
  • Fixed an issue where editing a transaction with a receipt would crash the application
  • The UI is now built using blueprint

Shell Extensions

sereneblue announces

Colosseum and krypto extensions have received support for GNOME Shell 44.

GNOME Foundation

Kristi Progri announces

I am Kristi Progri, the Program Manager at the GNOME Foundation, where I am in charge of conferences, events and ensuring that everything is run efficiently in our community.

The Linux App Summit is approaching and we have just finished organising the conference schedule, sending emails to all speakers, opening the call for volunteers and gathering the supplies we will need to make the conference run smoothly, and made sure we have enough people present to support us. Talking and communicating with sponsors is another aspect of my duties. If you or your business is interested in sponsoring the Linux App Summit monetarily, please contact us by email at kprogri@gnome.org.

After Linux App Summit the other conference that we have is GUADEC. GUADEC is the GNOME community’s largest conference and this year is happening in Riga, Latvia from 26th-31st of July. The call for papers is open until March 27th, so here’s your chance to submit your proposals.

Here are the tasks we have been working on as a team:

  • Reviewing the papers submission page
  • Updating the website with the latest information about the city, attractions and transportation
  • Constantly talking with the venue people making sure we have the rooms we want to rent for the event
  • Forming the papers team, etc.

In addition to these activities, I am working on the Annual Report, for which I would appreciate some help. These are the tasks that we need help with:

  • Contacting and sending reminders to everyone who is writing the articles;
  • Collecting all these materials and creating a document;
  • Update the Gitlab issues with the latest update on annual report progress

If you are interested in volunteering to help me with that, please ping me either in matrix @kristiprogri:gnome.org or by email.

That’s all for this week!

See you next week, and be sure to stop by #thisweek:gnome.org with updates on your own projects!

March 16, 2023

The PDF text model is quite nice, actually

As was discussed earlier, the way PDF handles fonts and glyphs is arcane and tedious. It takes a lot of boilerplate and hitting your shins against sharp stones to get working. However once you do and can turn to the higher level text functionality, things become a lot nicer. (Right-to-left, vertical and calligraphic scripts might be more difficult, but I don't know any of those.)

The PDF text drawing model provides a fairly wide selection of text operations.

If you, for example, want to produce a paragraph of justified text, first you need to calculate how the text should be split in lines and the additional word and character scaling parameters needed. Then the text can be rendered with the following pseudocode:

  • Create a text object with the correct font and position.
  • Set spacing values for the current line.
  • Output the current line of text (add kerning manually if it is in a format Freetype does not handle)
  • Repeat the above two steps until the paragraph is done
  • Close the text object
This shifts the burden of having to compute each letter's exact location from you to the PDF renderer. If you need more precision than this, then you need to dig out Harfbuzz, and draw the glyphs one by one to precomputed coordinates.

Register for Linux App Summit 2023!

LAS 2023 is happening next month and registrations are open!

You can  check the schedule in https://conf.linuxappsummit.org/event/5/timetable/#20230422

We are excited to have you visiting us in Brno, Czech Republic. The conference starts on Friday, April 21st, with a pre-registration social event. Saturday and Sunday are full of interesting talks, panels, workshops, and more!

March 15, 2023

Portfolio 0.9.15

After a long hiatus, a new release of Portfolio is out 📱🤓. This new release comes with important bug fixes, small-detail additions and a few visual improvements.

In terms of visuals, by popular demand, the most notable change is the use of regular icons for the files browser view. It should be easier now to quickly catch what each file is about. Thanks to @AngelTomkins for the initial implementation, @Exalm for helping with the reviews, and to the GNOME design team for such lovely new icons.

Another addition is support for system-wide style management. This is specially useful now that desktops like GNOME provide quick toggle buttons to switch between dark and light modes. Thanks to @pabloyoyoista for the initial implementation.

One small-detail change to the properties view is the addition of file permissions.  Plus, the properties view was broken down into three different sections to reduce the visual load, and labels can now be selected which is useful for copying locations or ellipsized values.

Moving on to bug fixes, two important changes landed. The first one solves an issue which prevented opening files with special characters 🤦. Thanks to @jwaataja for detecting and fixing this issue. The second one solves an issue with Portfolio not properly detecting mount points under some specific conditions. Thanks to mo2mo for reaching out and sharing his system details, so I could figure this out.

Last but never least, many thanks to @carlosgonz0, @Vistaus, @rffontenelle, @AsciiWolf, and @eson57 for keeping translations up to date, and  to @rene-coty for the new French translation.

March 11, 2023

2023-03-11 Saturday

  • Up lateish, J. to AllSaints, provided on-hand debugging advice for N. much of the day.
  • Tried to get memory trimming patch pushed; fell over amusing CI fails due to day-of-week dependent code in unit test at the weekend issue up-stream. Makes a pleasant change from random breakage introduced by me. Always bad when you have no recollection of writing the code, but the style shows it must have been you.

March 10, 2023

2023-03-10 Friday

  • 8am sales call, partner & customer calls through the day, calmed down a runaway logging problem in a corner-case. Implemented a memory / cache trimming feature for inactive documents.

pre-initialization of garbage-collected webassembly heaps

Hey comrades, I just had an idea that I won't be able to work on in the next couple months and wanted to release it into the wild. They say if you love your ideas, you should let them go and see if they come back to you, right? In that spirit I abandon this idea to the woods.

Basically the idea is Wizer-like pre-initialization of WebAssembly modules, but for modules that store their data on the GC-managed heap instead of just in linear memory.

Say you have a WebAssembly module with GC types. It might look like this:

(module
  (type $t0 (struct (ref eq)))
  (type $t1 (struct (ref $t0) i32))
  (type $t2 (array (mut (ref $t1))))
  ...
  (global $g0 (ref null eq)
    (ref.null eq))
  (global $g1 (ref $t1)
    (array.new_canon $t0 (i31.new (i32.const 42))))
  ...
  (function $f0 ...)
  ...)

You define some struct and array types, there are some global variables, and some functions to actually do the work. (There are probably also tables and other things but I am simplifying.)

If you consider the object graph of an instantiated module, you will have some set of roots R that point to GC-managed objects. The live objects in the heap are the roots and any object referenced by a live object.

Let us assume a standalone WebAssembly module. In that case the set of types T of all objects in the heap is closed: it can only be one of the types $t0, $t1, and so on that are defined in the module. These types have a partial order and can thus be sorted from most to least specific. Let's assume that this sort order is just the reverse of the definition order, for now. Therefore we can write a general type introspection function for any object in the graph:

(func $introspect (param $obj anyref)
  (block $L2 (ref $t2)
    (block $L1 (ref $t1)
      (block $L0 (ref $t0)
        (br_on_cast $L2 $t2 (local.get $obj))
        (br_on_cast $L1 $t1 (local.get $obj))
        (br_on_cast $L0 $t0 (local.get $obj))
        (unreachable))
      ;; Do $t0 things...
      (return))
    ;; Do $t1 things...
    (return))
  ;; Do $t2 things...
  (return))

In particular, given a WebAssembly module, we can generate a function to trace edges in an object graph of its types. Using this, we can identify all live objects, and what's more, we can take a snapshot of those objects:

(func $snapshot (result (ref (array (mut anyref))))
  ;; Start from roots, use introspect to find concrete types
  ;; and trace edges, use a worklist, return an array of
  ;; all live objects in topological sort order
  )

Having a heap snapshot is interesting for introspection purposes, but my interest is in having fast start-up. Many programs have a kind of "initialization" phase where they get the system up and running, and only then proceed to actually work on the problem at hand. For example, when you run python3 foo.py, Python will first spend some time parsing and byte-compiling foo.py, importing the modules it uses and so on, and then will actually run foo.py's code. Wizer lets you snapshot the state of a module after initialization but before the real work begins, which can save on startup time.

For a GC heap, we actually have similar possibilities, but the mechanism is different. Instead of generating an array of all live objects, we could generate a serialized state of the heap as bytecode, and another function to read the bytecode and reload the heap:

(func $pickle (result (ref (array (mut i8))))
  ;; Return an array of bytecode which, when interpreted,
  ;; can reconstruct the object graph and set the roots
  )
(func $unpickle (param (ref (array (mut i8))))
  ;; Interpret the bytecode, building object graph in
  ;; topological order
  )

The unpickler is module-dependent: it will need one case to construct each concrete type $tN in the module. Therefore the bytecode grammar would be module-dependent too.

What you would get with a bytecode-based $pickle/$unpickle pair would be the ability to serialize and reload heap state many times. But for the pre-initialization case, probably that's not precisely what you want: you want to residualize a new WebAssembly module that, when loaded, will rehydrate the heap. In that case you want a function like:

(func $make-init (result (ref (array (mut i8))))
  ;; Return an array of WebAssembly code which, when
  ;; added to the module as a function and invoked, 
  ;; can reconstruct the object graph and set the roots.
  )

Then you would use binary tools to add that newly generated function to the module.

In short, there is a space open for a tool which takes a WebAssembly+GC module M and produces M', a module which contains a $make-init function. Then you use a WebAssembly+GC host to load the module and call the $make-init function, resulting in a WebAssembly function $init which you then patch in to the original M to make M'', which is M pre-initialized for a given task.

Optimizations

Some of the object graph is constant; for example, an instance of a struct type that has no mutable fields. These objects don't have to be created in the init function; they can be declared as new constant global variables, which an engine may be able to initialize more efficiently.

The pre-initialized module will still have an initialization phase in which it builds the heap. This is a constant function and it would be nice to avoid it. Some WebAssembly hosts will be able to run pre-initialization and then snapshot the GC heap using lower-level facilities (copy-on-write mappings, pointer compression and relocatable cages, pre-initialization on an internal level...). This would potentially decrease latency and may allow for cross-instance memory sharing.

Limitations

There are five preconditions to be able to pickle and unpickle the GC heap:

  1. The set of concrete types in a module must be closed.

  2. The roots of the GC graph must be enumerable.

  3. The object-graph edges from each live object must be enumerable.

  4. To prevent cycles, we have to know when an object has been visited: objects must have identity.

  5. We must be able to create each type in a module.

I think there are three limitations to this pre-initialization idea in practice.

One is externref; these values come from the host and are by definition not introspectable by WebAssembly. Let's keep the closed-world assumption and consider the case where the set of external reference types is closed also. In that case if a module allows for external references, we can perhaps make its pickling routines call out to the host to (2) provide any external roots (3) identify edges on externref values (4) compare externref values for identity and (5) indicate some imported functions which can be called to re-create exernal objects.

Another limitation is funcref. In practice in the current state of WebAssembly and GC, you will only have a funcref which is created by ref.func, and which (3) therefore has no edges and (5) can be re-created by ref.func. However neither WebAssembly nor the JS API has no way of knowing which function index corresponds to a given funcref. Including function references in the graph would therefore require some sort of host-specific API. Relatedly, function references are not comparable for equality (func is not a subtype of eq), which is a little annoying but not so bad considering that function references can't participate in a cycle. Perhaps a solution though would be to assume (!) that the host representation of a funcref is constant: the JavaScript (e.g.) representations of (ref.func 0) and (ref.func 0) are the same value (in terms of ===). Then you could compare a given function reference against a set of known values to determine its index. Note, when function references are expanded to include closures, we will have more problems in this area.

Finally, there is the question of roots. Given a module, we can generate a function to read the values of all reference-typed globals and of all entries in all tables. What we can't get at are any references from the stack, so our object graph may be incomplete. Perhaps this is not a problem though, because when we unpickle the graph we won't be able to re-create the stack anyway.

OK, that's my idea. Have at it, hackers!

Maps and GNOME 44

 So it's that time that occurs twice a year when we're approaching a new GNOME release, this time 44.



In Maps there's been some polish touching up some of the rough edges in 43.x (from the GTK 4 port).

For example keyboard navigation of the search results is back.


Also, the focus handling of the entry is fixed so that the quirky flickering (re-popping up the the results popover when typing a space into the search entry) is now gone.

Also thanks to Christopher Davis for fixing up the styling of the results!

We also had contributions from an anonymous user improving the style of some labels, among others the tooltips of the headerbar buttons.


Adrien Plazas has worked on making the „Export as Image“ dialog fit on phones.


Unfortunately an issue has come to our attention with the way the „go to“ animations works, resulting some times when covering large distances (for example when selecting a search result far away) that results in throttling from the tile server (HTTP 429, „too many requests”). This results in empty tiles showing up at the destination (and one needs to sometimes wait a while before Maps gets the view refreshed with new tiles (after restarting to force re-trying).

 As this results in a really bad user experience as a work-around we have disabled the animations for now (the animations when zooming is still in place, and of course panning the map view still works as usual).

I also cherry-picked this for the 43 branch, and will probably also cut a 43.4 with this change (versions prior to 43.0 is not affected, as the animations in libchamplain uses another algorithm, first zooming out „high“ (a low zoom level) before moving, resulting in less intermediate tiles compared to the “fly to“-like animations now used in libshumate.

 We will try to come up with a better solution in libshumate soon. Either reverting to an approach like that in libchamplain, or try to implement our own rate limiting of some sort during animations to avoid firing off too many requests.

 Meanwhile at the libshumate side James Westman has been busy working on the vector tile support implementing label support among other things.

He is also working on a vector tile style using the GNOME color palette in light and dark variants.


This can be trying out on https://maps.jwestman.net/

 There will also be some other things coming for the next development cycle for GNOME 45, but let's save that for next time!

#86 New Decoding

Update on what happened across the GNOME project in the week from March 03 to March 10.

GNOME Incubating Apps

Loupe

A simple and modern image viewer.

Sophie 🏳️‍🌈 🏳️‍⚧️ ✊ says

The new image decoding code has finally landed in Loupe. This change will be the basis for many features to come, like supporting color profiles, animated images, and potentially sandboxed decoding. Already landed has the support for viewing SVGs. Thanks to GTK 4 and tiled rendering, you can view large SVGs without the UI seeing any performance penalties. However, a few known rendering bugs still have to be fixed.

The GDK-PixBuf support has, for now, been removed from Loupe. Loupe already supports many image formats directly, making it unclear if support for GDK-PixBuf plugins is required. Please let us know if an image format vital to you is missing from the list of supported formats.

Apart from this, we have:

  • Fixed some memory leaks,
  • redone the scroll-wheel logic to support high-resolution scroll wheels,
  • made rotation and pan gestures work correctly on touch screens, and
  • merged many more bug fixes, tweaks, code improvements, and documentation.

Finally, a small optical change: Drag and drop now features a newly designed thumbnail that provides more contrast to the content behind it.

GNOME Circle Apps and Libraries

Workbench

A sandbox to learn and prototype with GNOME technologies.

Sonny announces

In just 2 weeks, Workbench gained 11 new GNOME Platform demos and examples. More on the way. A huge thanks to all the contributors! Library examples and demos have 3 functions

  • Showcase the capabilities of the platform
  • Teach how to use the APIs, patterns, and widgets
  • Provide functional snippets ready to use

The plan for now is to focus on a wide set of UI / JavaScript entries. In the long run, we want to support more languages into Workbench and port the examples.

I have written a guide on how to get started contributing to Workbench https://github.com/sonnyp/Workbench/blob/main/CONTRIBUTING.md For the time being I will be focusing on stability, mentoring and preparing Workbench 44

Tobias Bernard Shared their experience using Workbench as a design / prototype tool https://mastodon.social/@tbernard/109972593064382462

James Westman and I discussed reducing the scope of Blueprint 1.0 in order to drop its experimental status and make it the default syntax in Workbench

I’d like to give kudos and thanks to GSoC and Outreachy early applicants for their contributions ✨

Share Preview

Test social media cards locally.

Rafael Mardojai CM reports

Share Preview has a new “logs” feature. The app will now give better hints about errors, missing metadata, and limits set by social platforms like image size.

This new feature has added a lot of new strings. Everyone is welcome to contribute or update translations for their language on Weblate.

Pika Backup

Keep your data safe.

Sophie 🏳️‍🌈 🏳️‍⚧️ ✊ announces

After the release of Pika Backup 0.5, Fina has taken care of many outstanding smaller and more significant issues. Pika Backup now also uses the new background status for apps that will be available with GNOME Shell 44. That way, you can always check what Pika Backup is doing in the background.

Some of the other changes that happened:

  • Fix compact is not run after pruning
  • Fix potential crash after deleting archives
  • Fix spurious ‘Pika Backup crashed’ messages
  • Change secret service error messages to include specific instructions how to resolve the issue
  • Change to explain checkpoint creation when aborting backups
  • Change to restart backup after SSH connection timeout
  • Change reconnection to be abortable and count down seconds remaining
  • Add ability to answer questions from borg process

Third Party Projects

Alessandro Iepure announces

I released Dev Toolbox! If you are tired of using random websites for conversions or simple checks during your coding, give it a try. Simple to use with local processing.
Includes encoders and decoders, formatters for various languages, image converters, text and hash generators, and much more.
Available on Flathub: https://beta.flathub.org/apps/me.iepure.devtoolbox

Can Lehmann announces

This week we released owlkettle 2.2.0, a declarative GUI framework based on GTK. Owlkettle is a library for the Nim programming language. One major focus for this release was improving the documentation. There are 15 new examples as well as a new document explaining how owlkettle works internally. We also improved the support for libadwaita by adding bindings for the EntryRow, ComboRow, ExpanderRow, WindowSurface, Flap and SplitButton widgets. Other improvements include support for multithreading and sending notifications. You can find a short tutorial for getting started with owlkettle here.

Mazhar Hussain says

nautilus-code received support for translations and has already been translated in Hungarian and Italian.

nxyz reports

This week I released Chromatic, a simple tuner app written in Rust. Chromatic detects the frequency of audio input, converts it to the correct musical note and octave, and displays the cents error. Cents are displayed on an analog gauge to make tuning more visually intuitive. Requires PulseAudio or PipeWire.

GH repo: https://github.com/nate-xyz/chromatic Flathub page: https://beta.flathub.org/apps/io.github.nate_xyz.Chromatic

Telegrand

A Telegram client optimized for the GNOME desktop.

Marco Melorio announces

It’s finally time for a new update about Telegrand development! This is a brief look at the new biggest changes from the last update:

  • Added support for file messages by yuraiz
  • Added support for GIF messages
  • Added support for more event-like messages by Alisson Lauffer
  • Added support for viewing message replies by karl0d
  • Added ability to edit and reply to messages
  • Added a Christmast easter egg animation by yuraiz (sorry, we’re late!)
  • Added markdown support for composing messages by karl0d
  • Added more information in the chat info window, like group description, usernames and phone number
  • Added a contacts window to view saved contacts
  • Improved the chat view for channels, by adding a mute/unmute button by karl0d
  • Improved the style of the chat view by yuraiz
  • Big performance improvements to chat view scrolling
  • Ground work for future support of chat folders and archived chats

Flare

An unofficial Signal GTK client.

schmiddi reports

Flare 0.7.0-beta.1 was released. It does not provide any big new features, but updates many dependencies which lead to pretty big changes in the codebase. To check everything is working as expected, this beta was released for anyone to try out if they want. Note that due to changes in the storage, a relink will be required at the first start of the application.

Furthermore, Flare has gained experimental support for integrating with feedbackd to provide vibration or audio feedback on message receive.

Blurble

Word guessing game

Vojtěch Perník reports

After a longer time I started working on Blurble again and the app has moved a few steps closer to version 1.0.0:

  • Keyboard navigation has been improved. The currently active cell is highlighted and it is possible to navigate the UI using the Tab key.
  • The buttons on the keyboard are now also colored. For better playability are now keyboard buttons also colored based on if and where the character is in the word.
  • The app has been redesigned. A welcome page, help and nicer game result information have been added.

You can get the latest version from Flathub.

Also, a big thanks to the GNOME design team, namely Tobias Bernard and Allan Day, for helping come up with better solutions.

Shell Extensions

oae reports

I have updated the Pano - Clipboard Manager with new features and fixes

  • Gnome Shell 44 support
  • Marking items as a favorite is now possible.
  • New Emoji type
  • Many customization options are added (Item styles, Pano height…).
  • Links can be now opened in the default browser
  • History can be filtered based on the item type
  • Content-aware notifications
  • Many navigation improvements
  • And many more changes.
  • You can read more on GitHub.

Just Perfection announces

Extension download count is now available to the public on extensions.gnome.org.

Currently, Argos extension (not developed since 3.32) holding the records for more than 13.8M downloads following by Dash to Dock extension with 6.2M downloads.

GNOME Foundation

mwu reports

My name is Melissa Wu and I am a Project Coordinator with the GNOME Foundation. I’m in charge of special projects, fundraising and assist Caroline Henriksen with marketing and branding and Kristi Progri with events and conferences. I’m also the resident travel coordinator for hotels and other conference travel.

Right now I have two focuses – Raising sponsorships for GUADEC 2023 in Riga, Lativa (https://events.gnome.org/event/101/) and organizing our volunteers. I need help with both!! If you are interested, please feel free to reach out to me directly. This is a great time to learn something new or share your talents with us!

This week I am preparing for SCaLE where I will manning the booth for the next three days – fundraising, selling shirts, and sharing what’s new with GNOME. If you are in the area, come see us at the booth or stop by our GNOME Beers event on Sat (March 11) at 6pm at Craft by Smoke and Fire (30 W. Green Street. Pasadena, CA)

That’s all for this week!

See you next week, and be sure to stop by #thisweek:gnome.org with updates on your own projects!

March 07, 2023

Flathub in 2023

It’s been quite a few months since the most recent updates about Flathub last year. We’ve been busy behind the scenes, so I’d like to share what we’ve been up to at Flathub and why—and what’s coming up from us this year. I want to focus on:

  • Where Flathub is today as a strong ecosystem with 2,000 apps
  • Our progress on evolving Flathub from a build service to an app store
  • The economic barrier to growing the ecosystem, and its consequences
  • What’s next to overcome our challenges with focused initiatives

Today

Flathub is going strong: we offer 2,000 apps from over 1,500 collaborators on GitHub. We’re averaging 700,000 app downloads a day, with 898 million HTTP requests totalling 88.3 TB served by our CDN each day (thank you Fastly!). Flatpak has, in my opinion, solved the largest technical issue which has held back the mainstream growth and acceptance of Linux on the desktop (or other personal computing devices) for the past 25 years: namely, the difficulty for app developers to publish their work in a way that makes it easy for people to discover, download (or sideload, for people in challenging connectivity environments), install and use. Flathub builds on that to help users discover the work of app developers and helps that work reach users in a timely manner.

Initial results of this disintermediation are promising: even with its modest size so far, Flathub has hundreds of apps that I have never, ever heard of before—and that’s even considering I’ve been working in the Linux desktop space for nearly 20 years and spent many of those staring at the contents of dselect (showing my age a little) or GNOME Software, attending conferences, and reading blog posts, news articles, and forums. I am also heartened to see that many of our OS distributor partners have recognised that this model is hugely complementary and additive to the indispensable work they are doing to bring the Linux desktop to end users, and that “having more apps available to your users” is a value-add allowing you to focus on your core offering and not a zero-sum game that should motivate infighting.

Ongoing Progress

Getting Flathub into its current state has been a long ongoing process. Here’s what we’ve been up to behind the scenes:

Development

Last year, we concluded our first engagement with Codethink to build features into the Flathub web app to move from a build service to an app store. That includes accounts for users and developers, payment processing via Stripe, and the ability for developers to manage upload tokens for the apps they control. In parallel, James Westman has been working on app verification and the corresponding features in flat-manager to ensure app metadata accurately reflects verification and pricing, and to provide authentication for paying users for app downloads when the developer enables it. Only verified developers will be able to make direct uploads or access payment settings for their apps.

Legal

So far, the GNOME Foundation has acted as an incubator and legal host for Flathub even though it’s not purely a GNOME product or initiative. Distributing software to end users along with processing and forwarding payments and donations also has a different legal profile in terms of risk exposure and nonprofit compliance than the current activities of the GNOME Foundation. Consequently, we plan to establish an independent legal entity to own and operate Flathub which reduces risk for the GNOME Foundation, better reflects the independent and cross-desktop interests of Flathub, and provides flexibility in the future should we need to change the structure.

We’re currently in the process of reviewing legal advice to ensure we have the right structure in place before moving forward.

Governance

As Flathub is something we want to set outside of the existing Linux desktop and distribution space—and ensure we represent and serve the widest community of Linux users and developers—we’ve been working on a governance model that ensures that there is transparency and trust in who is making decisions, and why. We have set up a working group with myself and Martín Abente Lahaye from GNOME, Aleix Pol Gonzalez, Neofytos Kolokotronis, and Timothée Ravier from KDE, and Jorge Castro flying the flag for the Flathub community. Thanks also to Neil McGovern and Nick Richards who were also more involved in the process earlier on.

We don’t want to get held up here creating something complex with memberships and elections, so at first we’re going to come up with a simple/balanced way to appoint people into a board that makes key decisions about Flathub and iterate from there.

Funding

We have received one grant for 2023 of $100K from Endless Network which will go towards the infrastructure, legal, and operations costs of running Flathub and setting up the structure described above. (Full disclosure: Endless Network is the umbrella organisation which also funds my employer, Endless OS Foundation.) I am hoping to grow the available funding to $250K for this year in order to cover the next round of development on the software, prepare for higher operations costs (e.g., accounting gets more complex), and bring in a second full-time staff member in addition to Bartłomiej Piotrowski to handle enquiries, reviews, documentation, and partner outreach.

We’re currently in discussions with NLnet about funding further software development, but have been unfortunately turned down for a grant from the Plaintext Group for this year; this Schmidt Futures project around OSS sustainability is not currently issuing grants in 2023. However, we continue to work on other funding opportunities.

Remaining Barriers

My personal hypothesis is that our largest remaining barrier to Linux desktop scale and impact is economic. On competing platforms—mobile or desktop—a developer can offer their work for sale via an app store or direct download with payment or subscription within hours of making a release. While we have taken the “time to first download” time down from months to days with Flathub, as a community we continue to have a challenging relationship with money. Some creators are lucky enough to have a full-time job within the FLOSS space, while a few “superstar” developers are able to nurture some level of financial support by investing time in building a following through streaming, Patreon, Kickstarter, or similar. However, a large proportion of us have to make do with the main payback from our labours being a stream of bug reports on GitHub interspersed with occasional conciliatory beers at FOSDEM (other beverages and events are available).

The first and most obvious consequence is that if there is no financial payback for participating in developing apps for the free and open source desktop, we will lose many people in the process—despite the amazing achievements of those who have brought us to where we are today. As a result, we’ll have far fewer developers and apps. If we can’t offer access to a growing base of users or the opportunity to offer something of monetary value to them, the reward in terms of adoption and possible payment will be very small. Developers would be forgiven for taking their time and attention elsewhere. With fewer apps, our platform has less to entice and retain prospective users.

The second consequence is that this also represents a significant hurdle for diverse and inclusive participation. We essentially require that somebody is in a position of privilege and comfort that they have internet, power, time, and income—not to mention childcare, etc.—to spare so that they can take part. If that’s not the case for somebody, we are leaving them shut out from our community before they even have a chance to start. My belief is that free and open source software represents a better way for people to access computing, and there are billions of people in the world we should hope to reach with our work. But if the mechanism for participation ensures their voices and needs are never represented in our community of creators, we are significantly less likely to understand and meet those needs.

While these are my thoughts, you’ll notice a strong theme to this year will be leading a consultation process to ensure that we are including, understanding and reflecting the needs of our different communities—app creators, OS distributors and Linux users—as I don’t believe that our initiative will be successful without ensuring mutual benefit and shared success. Ultimately, no matter how beautiful, performant, or featureful the latest versions of the Plasma or GNOME desktops are, or how slick the newly rewritten installer is from your favourite distribution, all of the projects making up the Linux desktop ecosystem are subdividing between ourselves an absolutely tiny market share of the global market of personal computers. To make a bigger mark on the world, as a community, we need to get out more.

What’s Next?

After identifying our major barriers to overcome, we’ve planned a number of focused initiatives and restructuring this year:

Phased Deployment

We’re working on deploying the work we have been doing over the past year, starting first with launching the new Flathub web experience as well as the rebrand that Jakub has been talking about on his blog. This also will finally launch the verification features so we can distinguish those apps which are uploaded by their developers.

In parallel, we’ll also be able to turn on the Flatpak repo subsets that enable users to select only verified and/or FLOSS apps in the Flatpak CLI or their desktop’s app center UI.

Consultation

We would like to make sure that the voices of app creators, OS distributors, and Linux users are reflected in our plans for 2023 and beyond. We will be launching this in the form of Flathub Focus Groups at the Linux App Summit in Brno in May 2023, followed up with surveys and other opportunities for online participation. We see our role as interconnecting communities and want to be sure that we remain transparent and accountable to those we are seeking to empower with our work.

Whilst we are being bold and ambitious with what we are trying to create for the Linux desktop community, we also want to make sure we provide the right forums to listen to the FLOSS community and prioritise our work accordingly.

Advisory Board

As we build the Flathub organisation up in 2023, we’re also planning to expand its governance by creating an Advisory Board. We will establish an ongoing forum with different stakeholders around Flathub: OS vendors, hardware integrators, app developers and user representatives to help us create the Flathub that supports and promotes our mutually shared interests in a strong and healthy Linux desktop community.

Direct Uploads

Direct app uploads are close to ready, and they enable exciting stuff like allowing Electron apps to be built outside of flatpak-builder, or driving automatic Flathub uploads from GitHub actions or GitLab CI flows; however, we need to think a little about how we encourage these to be used. Even with its frustrations, our current Buildbot ensures that the build logs and source versions of each app on Flathub are captured, and that the apps are built on all supported architectures. (Is 2023 when we add RISC-V? Reach out if you’d like to help!). If we hand upload tokens out to any developer, even if the majority of apps are open source, we will go from this relatively structured situation to something a lot more unstructured—and we fear many apps will be available on only 64-bit Intel/AMD machines.

My sketch here is that we need to establish some best practices around how to integrate Flathub uploads into popular CI systems, encouraging best practices so that we promote the properties of transparency and reproducibility that we don’t want to lose. If anyone is a CI wizard and would like to work with us as a thought partner about how we can achieve this—make it more flexible where and how build tasks can be hosted, but not lose these cross-platform and inspectability properties—we’d love to hear from you.

Donations and Payments

Once the work around legal and governance reaches a decent point, we will be in the position to move ahead with our Stripe setup and switch on the third big new feature in the Flathub web app. At present, we have already implemented support for one-off payments either as donations or a required purchase. We would like to go further than that, in line with what we were describing earlier about helping developers sustainably work on apps for our ecosystem: we would also like to enable developers to offer subscriptions. This will allow us to create a relationship between users and creators that funds ongoing work rather than what we already have.

Security

For Flathub to succeed, we need to make sure that as we grow, we continue to be a platform that can give users confidence in the quality and security of the apps we offer. To that end, we are planning to set up infrastructure to help ensure developers are shipping the best products they possibly can to users. For example, we’d like to set up automated linting and security scanning on the Flathub back-end to help developers avoid bad practices, unnecessary sandbox permissions, outdated dependencies, etc. and to keep users informed and as secure as possible.

Sponsorship

Fundraising is a forever task—as is running such a big and growing service. We hope that one day, we can cover our costs through some modest fees built into our payments—but until we reach that point, we’re going to be seeking a combination of grant funding and sponsorship to keep our roadmap moving. Our hope is very much that we can encourage different organisations that buy into our vision and will benefit from Flathub to help us support it and ensure we can deliver on our goals. If you have any suggestions of who might like to support Flathub, we would be very appreciative if you could reach out and get us in touch.

Finally, Thank You!

Thanks to you all for reading this far and supporting the work of Flathub, and also to our major sponsors and donors without whom Flathub could not exist: GNOME Foundation, KDE e.V., Mythic Beasts, Endless Network, Fastly, and Equinix Metal via the CNCF Community Cluster. Thanks also to the tireless work of the Freedesktop SDK community to give us the runtime platform most Flatpaks depend on, particularly Seppo Yli-Olli, Codethink and others.

I wanted to also give my personal thanks to a handful of dedicated people who keep Flathub working as a service and as a community: Bartłomiej Piotrowski is keeping the infrastructure working essentially single-handedly (in his spare time from keeping everything running at GNOME); Kolja Lampe and Bart built the new web app and backend API for Flathub which all of the new functionality has been built on, and Filippe LeMarchand maintains the checker bot which helps keeps all of the Flatpaks up to date.

And finally, all of the submissions to Flathub are reviewed to ensure quality, consistency and security by a small dedicated team of reviewers, with a huge amount of work from Hubert Figuière and Bart to keep the submissions flowing. Thanks to everyone­—named or unnamed—for building this vision of the future of the Linux desktop together with us.

(originally posted to Flathub Discourse, head there if you have any questions or comments)

3D Printing on Endless OS

Screenshot

I recently got a 3D printer for my birthday, and I have been absolutely loving tinkering, printing, and learning with it. There is such a wealth of openly-shared and -licensed models out there, I’ve been having fun finding things to print for myself, my friends, and especially my toddler. The best part for me, though? The fact that I can dive deep into this whole new world of 3D modeling, slicing, and printing using the computer I already have with freely-available software right in the app store I already use on Endless OS: App Center with Flathub.

I got a Creality Ender 3 v2 printer, which is to say: a popular and well-reviewed printer that didn’t break the bank. For a super quick (and over-simplified) look at the 3D printing process, there are a few steps to get from nothing to a physical 3D object in your hands:

  1. Design a thing; this can be on a whiteboard, in a 2D design app, or whatever. The point is, you have to start with some sort of specification/measurements.

  2. Model that thing; you have to take the idea you have and turn it into a 3D shape on the computer. You can use a modeling program like Blender if you know what you’re doing or want to use your model in different contexts, or you can use a computer-aided drawing (CAD) tool to define the shapes for your object.

  3. Slice the model; this is the process of taking a defined 3D object and turning it into actual instructions for your specific printer, filament, etc. 3D printer companies usually offer free slicing programs designed with their own printers in mind, but it’s mostly all inter-compatible and based on the same few open source projects.

  4. Print the model; slicing outputs a GCODE file, which is very literal instructions for your printer to follow. You have to get that GCODE file onto your printer, either with an SD card, a USB connection, or a network connection. Then it can be printed!

With all the open source software that powers 3D printing combined with the open sharing culture, you can accomplish each of these steps on Endless OS or any other Linux-based OS that has access to Flathub. Let’s look at each step and the options you have:

1. Design a thing

I love whiteboards, so I usually start there if I’m designing a new thing from scratch. I’ll sketch a few angles out until I’m confident the measurements make sense, then I’ll move on to modeling.

However, if you’re trying to convert a 2D object into 3D, or you just want to play around in a 2D space first, I recommend Inkscape! It’s a great vector image editor, and it’s available on Flathub. You can lay things out and make sure they make sense in 2D space before moving onto the more complex 3D modeling.

If you already live in Blender, you can of course use it to design your object from the start, as well. It’s also available on Flathub.

2. Model that thing

Modeling is my weakest point personally, as I haven’t taken the time to really learn Blender or the open source CAD tools. So for me, I only tackle simple 3D models—and I use the web-based Autodesk Tinkercad to do it.

Screenshot of Tinkercad in Chromium

Luckily on Endless OS you can trivially install web apps like Tinkercad from the included Chromium web browser; just hit the little “Install” button that pops up on the address bar when you’re on the site. Then Tinkercad’s icon will be added to the app grid on your home screen and you can add it to your favorites on the dash.

GNOME Web also supports web apps, Tinkercad works there as well. If you use Google Chrome or any other browser based on Chromium, it should also have an easy to use install button. If you use Firefox, you can always use Tinkercad from in the browser itself.

If you’re more experienced (or adventurous than I am now), you can also try something like FreeCAD or OpenSCAD from Flathub—available in App Center right out of the box on Endless OS.

3. Slice the model

Slicing is the most exciting step of the process to me, if only because people keep innovating in 3D printing purely on the software side to get more speed, performance, etc. out of existing hardware—and that all happens in the slicers.

Popular 3D printer companies like UltiMaker/MakerBot, Prusa Labs, and FlashForge have their slicers available on Flathub. While others might offer downloads for their software on their website, my experience has been it’s less integrated over all; I’d stick to the ones on Flathub.

If whoever makes your printer doesn’t have a slicer on Flathub, fret not! They’re basically all inter-compatible, as the slicers are all based on the same handful of open source projects. Personally, I prefer UltiMaker Cura, and I could even add my Creality-brand Ender 3 v2 from in Cura’s UI with all the right preset values. Some people prefer PrusaSlicer, and it’s a similar story there.

4. Print the model

The simplest/most old-school way of printing is to save your sliced GCODE file out to a microSD card, eject it, then stick it in your printer, and use your printer’s physical screen and wheel to select the print file. And that will work just fine! You can also connect your printer over USB and print directly from your slicing software.

Screenshot of Prusa's GCODE viewer inspecting a model

If you want to learn more about how your model will print before actually printing, you can use your slicer or a separate program to preview the GCODE file. While Cura has a preview built in, sometimes I like using PrusaSlicer’s included GCODE viewer app (installed alongside PrusaSlicer) to inspect a specific GCODE file, too.

Screenshot of the Raspberry Pi imager with OctoPi selected

If you really want to go all out and have a spare Raspberry Pi, you can install OctoPrint with the OctoPi OS, and print to, monitor, and control your printer remotely. I won’t get into all the details, but you can get started with the official Raspberry Pi Imager by navigating to install an “other specific-purpose OS,” “3D printing,” and then “OctoPi.”


Hopefully these apps and tips are useful to you! I’ve been super happy with the experience of designing, modeling, slicing, and printing all entirely from Endless OS—and if you have access to Flathub on your OS of choice, you can do it, too!

March 06, 2023

Building a big-endian Arm system in Yocto

For reasons I won't bore anyone with I needed to build a 32-bit big-endian system with the Yocto Project to test a package, and I thought I'd write the steps down in case I ever need to do it again (or, even more unlikely, someone else needs to do it).

For unsurprising reasons I thought I'd do a big-endian Arm build. So we start by picking the qemuarm machine, which is a Armv7-A processor (Cortex-A15, specifically) in little-endian mode by default.

MACHINE = "qemuarm"

qemumarm.conf requires tune-cortexa15.inc which then requires arch-armv7ve.inc, and this file defines the base tunes. The default tune is armv7ve, we can make it big-endian by simply adding a b:

DEFAULTTUNE:qemuarm = "armv7veb"

And now we just build an image:

$ MACHINE=qemuarm bitbake core-image-minimal
...
Summary: 4 tasks failed:
  .../poky/meta/recipes-kernel/linux/linux-yocto_6.1.bb:do_package_qa
  .../poky/meta/recipes-graphics/xorg-proto/xorgproto_2022.2.bb:do_configure
  .../poky/meta/recipes-core/glib-2.0/glib-2.0_2.74.5.bb:do_configure
  .../poky/meta/recipes-graphics/wayland/wayland_1.21.0.bb:do_configure

Or not.

There are two failure cases here. First, the kernel:

ERROR: linux-yocto-6.1.9+gitAUTOINC+d7393c5752_ccd3b20fb5-r0 do_package_qa: QA Issue: Endiannes did not match (1, expected 0) in /lib/modules/6.1.9-yocto-standard/kernel/net/ipv4/ah4.ko [arch]

It turns out the kernel needs to be configured specifically to be big or little endian, and the default configuration is, predictably, little endian. There is a bug open to make this automatic, but big-endian really is dead because it has been open since 2016. The solution is a quick kernel configuration fragment added to the kernel's SRC_URI:

CONFIG_CPU_BIG_ENDIAN=y
CONFIG_CPU_LITTLE_ENDIAN=n

With this, the kernel builds as expected. The second set of failures are all from Meson, failing to execute a target binary:

../xorgproto-2022.2/meson.build:22:0: ERROR: Executables created by c compiler armeb-poky-linux-gnueabi-gcc [...] are not runnable.

Meson is trying to run the target binaries in a qemu-user that we set up, but the problem here is to save build time we only build the qemu targets that are typically used. This doesn't include usermode big-endian 32-bit Arm, so this target needs enabling:

QEMU_TARGETS:append = " armeb"

Now the image builds successfully, and we discover that indeed gdbm refuses to open a database which was generated on a system with a different endian.

March 05, 2023

David Revoy (Comic Artist): “At First, Publishing Under a Free License Scared Me”

In this article I use the words “librism” and “librist” as direct borrows from the French “librisme” and “libriste”. You can define librism as a conjunction of the free software and free culture movements. I prefer to define it as the fringe of the anticapitalist struggles that strives to destroy intellectual property by creating new intangible commons and protecting them.

The following is a translation of an interview published in the February 2023 issue of Alternative Libertaire. I conducted this interview with the goal of offering to its readers a glimpse of what the free culture movement is. The magazine doesn’t target free software or free culture enthusiasts, and its printed edition constrains the size of its content. Please excuse the resulting briefness and simplifications.

Thanks to Nina for proofreading the translation.

Making comics, deciding to give up copyright and hoping to make a living from them in capitalist lands, what an idea! Yet that’s what the comic artist and librist activist David Revoy decided to do. Here’s a look at his unusual journey into the commons.

Alternative Libertaire: Hi David, your main work follows the adventures of the young witch Pepper and her cat Carrot, and you distribute your works under free licenses. What led you to make your work part of the commons?

David Revoy : At first, I was afraid of free licenses, I was afraid that one of my characters would be copied, that my style would be imitated, that I would have no control over it.

In the years 2009–2010, I worked for the Blender Foundation, the Creative Commons was mandatory and under their wing, it reassured me. I made concept-arts and it went well, I saw beautiful derivations, fan-arts and I didn’t see any unpleasant derivation, it reassured me a lot. I even saw the beneficial effects on propagation, audience and interest. There was an added value, it became a common good, a sandbox that other creators could use.

When I did my own comic book project, Pepper & Carrot, the commons imposed itslef to me as evident.

David Revoy is a French comic book author, publishing all its work under a free license

A community collaborates in the creation of you universes, how is it integrated into the creative process?

It’s not completely integrated, I’m using a software forge to create my collaborative space which is abused as a forum, and we have an instant messenging tool on the side. I made a special scenario forge, where people can publish their Pepper & Carrot scenarios under a free license, to check if it can inspire me to make an episode. But you can’t improvise yourself a scenarist, and I have a lot of scripts that start out with very good ideas but fail to come up with a ending or a plot-twist. So there’s a lot of fragments of ideas that I sometimes pick up, some published episodes are inspired by things that have been suggested to me.

What really happens in terms of collaboration is feedback. I share my story-board and the community will show me things that could be avoided, like a proofreading before publication. For example, two panels didn’t flow well and we couldn’t understand why the character comes from the right and then from the left, or why there was a bubble that said that at this particular moment. It allows me to anticipate these questions and to know if they are desired or if it is uncontrolled, if something is not understandable.

After that, the collaborative space is mostly around translation, there are now sixty languages and about a hundred people gravitating around the project, around twenty of which are active.

How do you manage to make a living from your art while giving up on the usual models?

Since all my work is under a free license and accessible without any pay wall, to make an economic model around it, there are not 3 000 solutions, there is only donation, voluntary patronage of the readers. Only a small percentage of the readership will contribute to the author’s livelihood by making a donation for the next episode that I publish every two months.

Home page of www.peppercarrot.com

When I made Pepper & Carrot in 2014, asking money like that was really frowned upon, for web-comics it was donations on Paypal and e-shops, one-time donations, but not recurring donations. Now it’s much more commonplace and almost every starting artist has a system for recurring donations, this system has spread a lot.

Since 2016, Pepper & Carrot is published by Glénat in print edition. Did you approach them or did they come to you after the online success of Pepper & Carrot?

Glénat clearly came to me after the online success of the comic, the publishing house was looking for web-comics authors and they contacted me to edit Pepper & Carrot. I was offered the classic contract following their process, with the standard contract from their legal department. They are the ones who have leverage on the author; unless you already make a lot of sales, when you are a very small author, you don’t tend to negotiate too much.

Except I told them that this would not be the concept at all, that there was the free tool, the free culture, and that I had no interest in the exclusivity system. They were convinced by the unusual concept that now serves as an experiment. We just signed the Creative Commons together. It bothered the legal department a bit because it’s not their usual process and they know that a competing publisher like Delcourt could publish Pepper & Carrot, it would be completely legal.

Glénat pays me when I release a new comic, as patronage, and I have no copyright on the books. When I do signings, whether I sell a million copies or zero, I don’t make any money, but I help Glénat to help me in return. Besides, I love signing books and I can’t sign web-comics, there’s a kind of synergy, we have a relationship based on trust.

Last October, you shared your irritation when you discovered the Bulgarian edition removed a lesbian relationship from the story. If the license gives them this right, how do you reconcile the desire to create commons and to preserve your story?

We had a full range of legal resources at our disposal, the Creative Commons doesn’t remove moral right, so if this edition put a red cross on the lesbian kiss panel, it would have been an opinion expressed to say “this is bad” and there I could have expressed my moral right and sued over it, because it is not what I want to express as an author. The fact that it removes the panel from the story doesn’t mean that the author is against it, because it doesn’t take anything away from the story. There’s an omission because the editor didn’t want to be persona non grata in the media for defending an opinion too progressive for his country.

I couldn’t use my moral rights on it, it’s annoying but there was a movement of readers who organized themselves to print the panel and glue it back into their album, it became a sign of militancy to patch the printed book.

You also promote the freely licensed tools you use, like the Krita drawing software. What do you gain by using these instead of proprietary tools like Photoshop?

Mostly freedom. Not the freedom to not pay because I have a support subscription on Krita equal to what Photoshop costs, but I can install it on as many machines as I want, use it in a computer room, and tell everyone “we’re doing a Krita workshop, install it”.

I’ve been helping the Krita project since 2010 by giving feedback, and I had a lot of adjustments made for my needs. Krita has become tailored to my drawing technique, I have all my reflexes catered for, it is the ultimate comfort. Secondly, I’m sure there is no telemetry. I can’t read Krita’s code, but I know that there’s a whole community who does and can assure me that at no moment in time there can be a line of code able to find out which documents I’m using, what tablet, how much time I’m spending on my drawing. It guarantees my privacy and ensures that I don’t have any ad.

Illustration taken from the 37th episode of “Pepper & Carrot”, David Revoy, 2022

If tomorrow Adobe is bought by a billionaire, people who use Photoshop have no recourse. On Krita, there’s the Krita Foundation that develops it, but if they stop, its free license requires that the code has to be published somewhere and accessible. There will certainly be another group, or even myself, who will continue to build Krita and use it. This ensures that I can still benefit from my level and practice of drawing in the years to come, a guarantee that a Photoshp user would never get.

You also participate in the promotion of free software and free culture by illustrating many Framasoft campaigns. What political significance do you see in librism?

I see it as an online sandbox where you can try your hand at collaboration. You can try out a pyramidal structure with a leader, or a horizontal structure with each person being here on their own initiative, wanting to develop the project and to know how it works. I think this sandbox will allow us to develop reflexes of appreciation for certain models that will make us adopt certain policies instead retrospectively.

I don’t really believe in someone who will come up with a ready-made policy. School is not made like that, hobbies in France are not made like that, we always have a master on a podium, and now, all of a sudden, we’re going to be able to really have a space for experimentation of what collaboration is. I’ve already seen in librist festivals, when you have to pack up, how people organize themselves. That’s where free software has a great political force: to train people to collaborate.

March 02, 2023

Niepce February 2023 updates

This is the February 2023 update.

Not as much direct progress as I wished.

One big change is that now the binary is linked from Rust. And at the same time the autotools buid system is gone. The latter came first, as to not have to ignore it when doing the former.

Several challenges presented themselves with linked the app from Rust.

  1. We have to invert the dependency. We'd build the Rust code as a library, generate the headers for the cxx bridge and then build the C++ code, and link it. Instead we generate the headers, build the C++ code as libraries and then the Rust code.
  2. We have to figure some tricks: the Rust executable needs to link to some system libraries called directly from the C++ code, and also need to link to asan if needed.

Otherwise, for the two previous topics, namely the importer and the previewer.

The importer

The importer hasn't really moved much, sadly.

The previewer

For the previewer, I implemented a cache backed by a database. Currently it only handles the embedded thumbnails, but I'm working on reworking the image renderer (a simple pipeline based in GEGL) to work aynchronously, its design being hopefully suited for the use case.

UI toolkit and dependencies

I wrote a couple of macros to help with some of the UI patterns, like declaring actions to send an event. And the higher order action groups with all the actions. Less code to write, less possible errors.

gtk4-rs

An update of the bindings came out so I had to upgrade. Some code changes, but on overall it was eventless. Congrats to the team.

libpanel

Investigating how libpanel can help me implement the UI designs I have in mind. Straight from gnome-builder, libpanel offers a bunch of widgets to implement dockable panels, in a way I imagined earlier ideas for Niepce UI. While it has Rust bindings, it lacked documentation, so I've addressed this, at least enough for further contributions.

Also addressed some issues with improper annotations that caused floating references to have improper ownership and being dropped too early by the Rust binding. The annotations are part of gir and used by the bindings generation tools.

I think it should be in good shape now to use it here ; I tested it in compiano. I need to come up with a UI design to change the few things I want to change ; but I need more feature first.

Other side quest

Like in any good RPG campaigns there are side quests that will help you fullfilling the main goals

libopenraw

Did some work on libopenraw, which is used for thumbnailing. This include vendoring the crates for the mp4parse crate (forked from Mozilla's), when generating the tarball. (Release to come)

Also did some work on the Rust port, including speed ups (BufReader is a simple boost as by default I/O are not buffered in Rust), with benchmarking to measure performance gains, and a few others.

gpsami

Another side project, needed to download GPS logging data from my GPS logger after an excursion. It happens that the current code for GPSAmi was broken, so I had to fix it. I have to admit I carelessly changed including porting to gtk4 without testing.

Also another issue I want to fix is to no longer use gpsbabel. Two reasons for that:

  1. it's a command line tool, so interfacing with it is what it is.
  2. it requires Qt5. That latter part it really a bummer with Flatpak because I have to either use the KDE SDK and build gtk4 or use the GNOME SDK and build Qt. It's 2023 you you still can't build QtCore separately it seems.

So I started writing a crate (in Rust) to support downloading from my GPS logger, which in turn make me realize that serial port management is complicated with permissions. And that's putting aside bluetooth serial: it doesn't work on my laptop but work on the desktop.

I think that's what we call full stack. More on that later.

Thank you reading.

February 27, 2023

GSoC 2023: GNOME Foundation has been accepted as a mentoring org!

We are glad to announce that once again the GNOME Foundation will be part of Google Summer of Code. We are interested in onboarding new contributors that are passionate about GNOME and motivated to become long term GNOME developers!

@Contributors interested in participating in GSoC with GNOME should visit https://gsoc.gnome.org for more information.

@Mentors interested in mentoring projects this year should file a gitlab issue in Teams / Engagement / Internship Project Ideas · GitLab 2

Project ideas

Our ideas list is available in GNOME + GSoC | 2023 Project Ideas 2 and is the result of the discussions in Teams / Engagement / Internship Project Ideas · GitLab 2.

You can still submit project ideas until March 19th, when GSoC applicants are expected to submit their final proposals.

Important upcoming dates:

  • Now – March 19: Proactive GSoC contributors will reach out asking questions about your ideas list and receive feedback from us so they can start crafting their project proposals.

    @Contributors make sure you research enough about the project and work towards making a small contribution. You should consider the proposals available in GNOME + GSoC | 2023 Project Ideas or propose your own project ideas as soon as possible in Teams / Engagement / Internship Project Ideas · GitLab 2
    Make sure you approach potential mentors to move your idea towards an internship.

    @Mentors, point contributors to gsoc.gnome.org for more information and be patient with their questions. Contributors are open to suggest new project proposals and you should indicate whether you’d be interested in mentoring those proposals and help them compose a project proposal that is realistic and benefits the project.

  • March 20 – April 4 18:00 UTC: GSoC contributors will submit their proposals through the program website.
  • April 4 – 26: We (admins and mentors) will review all submitted GSoC Contributor proposals and consider how many we want to select (based on how many committed mentors we have available). Admins and mentors will rank the contribution proposals.
  • April 27 18:00 UTC: Deadline to submit ranked slot requests (Org Admins enter requests)
  • April 27 – May 2: Google Program Admins review and assign org slots
  • May 3: Organizations receive notification of their accepted GSoC 2023 Contributors
  • May 4: Accepted GSoC 2023 GSoC Contributor projects are announced
  • May 4 – 28: Community Bonding Period
  • May 27: Deadline to notify Google Admins of an inactive GSoC Contributor that you wish to remove from the program
  • May 29: Coding begins

For more information on the timeline, visit Google Summer of Code 2023 Timeline  |  Google Developers

If you have any doubts or questions, please reply to this message on Discourse.

February 25, 2023

Reducing code size in librsvg by removing an unnecessary generic struct

Someone mentioned cargo-bloat the other day and it reminded me that I have been wanting to measure the code size for generic functions in librsvg, and see if there are improvements to be made.

Cargo-bloat can give you a rough estimate of the code size for each Rust crate in a compiled binary, and also a more detailed view of the amount of code generated for individual functions. It needs a [bin] target to work on; if you have just a [lib], it will not do anything. So, for librsvg's purposes, I ran cargo-bloat on the rsvg-convert binary.

$ cargo bloat --release --crates
    Finished release [optimized] target(s) in 0.23s
    Analyzing target/release/rsvg-bench

 File  .text     Size Crate
10.0%  38.7%   1.0MiB librsvg
 4.8%  18.8% 505.5KiB std
 2.5%   9.8% 262.8KiB clap
 1.8%   7.1% 191.3KiB regex
 ... lines omitted ...
25.8% 100.0%   2.6MiB .text section size, the file size is 10.2MiB

Note: numbers above are a result of guesswork. They are not 100% correct and never will be.

The output above is for cargo bloat --release --crates. The --release option is to generate an optimized binary, and --crates tells cargo-bloat to just print a summary of crate sizes. The numbers are not completely accurate since, for example, inlined functions may affect callers of a particular crate. Still, this is good enough to start getting an idea of the sizes of things.

In this case, the librsvg crate's code is about 1.0 MB.

Now, let's find what generic functions we may be able to condense. When cargo-bloat is run without --crates, it prints the size of individual functions. After some experimentation, I ended up with cargo bloat --release -n 0 --filter librsvg. The -n 0 option tells cargo-bloat to print all functions, not just the top N biggest ones, and --filter librsvg is to make it print functions only in that crate, not for example in std or regex.

$ cargo bloat --release -n 0 --filter librsvg

File .text    Size   Crate Name
0.0%  0.0%  1.2KiB librsvg librsvg::element::ElementInner<T>::new
0.0%  0.0%  1.2KiB librsvg librsvg::element::ElementInner<T>::new
0.0%  0.0%  1.2KiB librsvg librsvg::element::ElementInner<T>::new
... output omitted ...
0.0%  0.0%    825B librsvg librsvg::element::ElementInner<T>::set_style_attribute
0.0%  0.0%    825B librsvg librsvg::element::ElementInner<T>::set_style_attribute
0.0%  0.0%    825B librsvg librsvg::element::ElementInner<T>::set_style_attribute
... output omitted ...
0.0%  0.0%    358B librsvg librsvg::element::ElementInner<T>::get_cond
0.0%  0.0%    358B librsvg librsvg::element::ElementInner<T>::get_cond
0.0%  0.0%    358B librsvg librsvg::element::ElementInner<T>::get_cond
... etc ...

After looking a bit at the output, I found the "duplicated" functions I wanted to find. What is happening here is that ElementInner<T> is a type with generics, and rustc is generating one copy of each of its methods for every type instance. So, there is one copy of each method for ElementInner<Circle>, one for ElementInner<Rect>, and so on for all the SVG element types.

The code around that is a bit convoluted; it's in a part of the library that hasn't gotten much cleanup after the C to Rust port and initial refactoring. Let's see what it is like.

The initial code

Librsvg parses the XML in an SVG document and builds something that resembles a DOM tree. The tree itself uses the rctree crate; it has reference-counted nodes and functions like first_child or next_sibling. Nodes can represent XML elements, or character content inside XML tags. Here we are interested in elements only.

Consider an element like this:

<path d="M0,0 L10,10 L0,10 Z" fill="black"/>

Let's look at how librsvg represents that. Inside each reference-counted node in an rctree, librsvg keeps a NodeData enum that can differentiate between elements and character content:

enum NodeData {
    Element(Element),
    Text(Chars),
}

Then, Element is an enum that can distinguish between all the elements in the svg namespace that librsvg supports:

enum Element {
    Circle(Box<ElementInner<Circle>>),
    Ellipse(Box<ElementInner<Ellipse>>),
    Path(Box<ElementInner<Path>>),
    // ... about 50 others omitted ...
}

Inside each of those enum's variants there is an ElementInner<T>, a struct with a generic type parameter. ElementInner holds the data for the DOM-like element:

struct ElementInner<T: ElementTrait> {
    element_name: QualName,
    attributes: Attributes,
    // ... other fields omitted
    element_impl: T,
}

For the <path> element above, this struct would contain the following:

  • element_name: a qualified name path with an svg namespace.
  • attributes: an array of (name, value) pairs, in this case (d, "M0,0 L10,10 L0,10 Z"), (fill, "black").
  • element_impl: A concrete type, Path in this case.

The specifics of the Path type are not terribly interesting here; it's just the internal representation for Bézier paths.

struct Path {
    path: Rc<SvgPath>,
}

Let's look at the details of the memory layout for all of this.

Initial memory layout

Here is how the enums and structs above are laid out in memory, in terms of allocations, without taking into account the rctree:Node that wraps a NodeData.

NodeData enum, and ElementInner<T> (description in text)

There is one allocated block for the NodeData enum, and that block holds the enum's discriminant and the embedded Element enum. In turn, the Element enum has its own discriminant and space for a Box (i.e. a pointer), since each of its variants just holds a single box.

That box points to an allocation for an ElementInner<T>, which itself contains a Path struct.

It is awkward that the fields to hold XML-isms like an element's name and its attributes are in ElementInner<T>, not in Element. But more importantly, ElementInner<T> has a little bunch of methods:

impl<T: ElementTrait> ElementInner<T> {
    fn new(...) -> ElementInner<T> {
        // lots of construction
    }

    fn element_name(&self) -> &QualName {
        ...
    }

    fn get_attributes(&self) -> &Attributes {
        ...
    }

    // A bunch of other methods
}

However, none but one of these methods actually use the element_impl: T field! That is, all of them do things that are common to all element types. The only method that really deals with the element_impl field is the ::draw() method, and the only thing it does is to delegate down to the concrete type's implementation of ::draw().

Removing that generic type

So, let's shuffle things around. I did this:

  • Turn enum Element into a struct Element, with the fields common to all element types.

  • Have an Element.element_data field...

  • ... that is of type ElementData, an enum that actually knows about all supported element types.

There are no types with generics in here:

struct Element {
    element_name: QualName,
    attributes: Attributes,
    // ... other fields omitted
    element_data: ElementData,
}

enum ElementData {
    Circle(Box<Circle>),
    Ellipse(Box<Ellipse>),
    Path(Box<Path>),
    // ...
}

Now the memory layout looks like this:

NodeData enum with boxes, Element, and ElementData (description in text)

One extra allocation, but let's see if this changes the code size.

Code size

We want to know the size of the .text section in the ELF file.

# old
$ objdump --section-headers ./target/release/rsvg-bench
Idx Name          Size      VMA               LMA               File off  Algn
 15 .text         0029fa17  000000000008a060  000000000008a060  0008a060  2**4
(2750999 bytes)

# new
Idx Name          Size      VMA               LMA               File off  Algn
 15 .text         00271ff7  000000000008b060  000000000008b060  0008b060  2**4
(2564087 bytes)

The new code is is 186912 bytes smaller. Not earth-shattering, but cargo-bloat no longer shows duplicated functions which have no reason to be monomorphized, since they don't touch the varying data.

old:

$ cargo bloat --release --crates
 File  .text     Size Crate
10.0%  38.7%   1.0MiB librsvg
# lines omitted
25.8% 100.0%   2.6MiB .text section size, the file size is 10.2MiB

new:

$ cargo bloat --release --crates
 File  .text     Size Crate
 9.2%  37.5% 939.5KiB librsvg
24.6% 100.0%   2.4MiB .text section size, the file size is 10.0MiB

Less code should help a bit with cache locality, but the functions involved are not in hot loops. Practically all of librsvg's time is spent in Cairo for rasterization, and Pixman for compositing.

Dynamic dispatch

All the concrete types (Circle, ClipPath, etc.) implement ElementTrait, which has things like a draw() method, although that is not visible in the types above. This is what is most convenient for librsvg; using Box<ElementTrait> for type erasure would be a little awkward there — we used it a long time ago, but not anymore.

Eventually the code needs to find the ElementTrait vtable that corresponds to each of ElementData's variants:

let data: &dyn ElementTrait = match self {
    ElementData::Circle(d) =>   &**d,
    ElementData::ClipPath(d) => &**d,
    ElementData::Ellipse(d) =>  &**d,
    // ...
};

data.some_method_in_the_trait(...);

The ugly &**d is to arrive at the &dyn ElementTrait that each variant implements. It will get less ugly when pattern matching for boxes gets stabilized in the Rust compiler.

This is not the only way of doing things. For librsvg it is convenient to actually know the type of an element, that is, to keep an enum of the known element types. Other kinds of code may be perfectly happy with the type erasure that happens when you have a Box<SomeTrait>. If that code needs to go back to the concrete type, an alternative is to use something like the downcast-rs crate, which lets you recover the concrete type inside the box.

Heap usage actually changed

You may notice in the diagrams below that the original NodeData didn't box its variants, but now it does.

Old:

enum NodeData {
    Element(Element),
    Text(Chars),
}

New:

enum NodeData {
    Element(Box<Element>),
    Text(Box<Chars>),
}

One thing I didn't notice during the first round of memory reduction is that the NodeData::Text(Chars) variant is not boxed. That is, the size of NodeData enum is the size of the biggest of Element and Chars, plus space for the enum's discriminant. I wanted to make both variants the same size, and by boxing them they occupy only a pointer each.

I measured heap usage for a reasonably large SVG:

India Roadway Map, from Wikimedia Commons

I used Valgrind's Massif to measure peak memory consumption during loading:

valgrind --tool=massif --massif-out-file=massif.out ./target/release/rsvg-bench --num-load 1 --num-render 0 India_roadway_map.svg
ms_print massif.out

The first thing that ms_print shows is an overview of the program's memory usage over time, and the list of snapshots it created. The following is an extract of its output for the new version of the code, where snapshot 36 is the one with peak memory usage:

MB
14.22^                                                                      : 
     |                                                @#::::::::::::::::::::: 
     |                                              @@@#:      :::: :: ::: :: 
     |                                            @@@@@#:      :::: :: ::: :: 
     |                                          @@@ @@@#:      :::: :: ::: :: 
     |                                        @@@ @ @@@#:      :::: :: ::: :: 
     |                                       @@@@ @ @@@#:      :::: :: ::: :: 
     |                                    @@@@@@@ @ @@@#:      :::: :: ::: :: 
     |                                  @@@@ @@@@ @ @@@#:      :::: :: ::: :: 
     |                                 @@ @@ @@@@ @ @@@#:      :::: :: ::: :::
     |                               @@@@ @@ @@@@ @ @@@#:      :::: :: ::: :::
     |                              @@ @@ @@ @@@@ @ @@@#:      :::: :: ::: :::
     |                             @@@ @@ @@ @@@@ @ @@@#:      :::: :: ::: :::
     |                          @@@@@@ @@ @@ @@@@ @ @@@#:      :::: :: ::: :::
     |                        :@@@ @@@ @@ @@ @@@@ @ @@@#:      :::: :: ::: ::@
     |                     @@@:@@@ @@@ @@ @@ @@@@ @ @@@#:      :::: :: ::: ::@
     |                 @@@@@ @:@@@ @@@ @@ @@ @@@@ @ @@@#:      :::: :: ::: ::@
     |              :::@ @ @ @:@@@ @@@ @@ @@ @@@@ @ @@@#:      :::: :: ::: ::@
     |            :@:: @ @ @ @:@@@ @@@ @@ @@ @@@@ @ @@@#:      :::: :: ::: ::@
     |     @@@@::::@:: @ @ @ @:@@@ @@@ @@ @@ @@@@ @ @@@#:      :::: :: ::: ::@
   0 +----------------------------------------------------------------------->Mi
     0                                                                   380.9

Number of snapshots: 51
 Detailed snapshots: [3, 4, 5, 9, 12, 13, 14, 15, 16, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36 (peak), 50]

Since we are just measuring memory consumption during loading, the chart shows that memory usage climbs steadily until it peaks when the complete SVG is loaded, and then it stays more or less constant while librsvg does the initial CSS cascade.

The version of librsvg without changes shows this (note how the massif snapshot with peak usage is number 39 in this one):

--------------------------------------------------------------------------------
  n        time(i)         total(B)   useful-heap(B) extra-heap(B)    stacks(B)
--------------------------------------------------------------------------------
 39    277,635,004       15,090,640       14,174,848       915,792            0

That is, 15,090,640 bytes.

And after making the changes in memory layout, we get this:

--------------------------------------------------------------------------------
  n        time(i)         total(B)   useful-heap(B) extra-heap(B)    stacks(B)
--------------------------------------------------------------------------------
 36    276,041,810       14,845,456       13,935,702       909,754            0

I.e. after the changes, the peak usage of heap memory when the whole file is loaded is 14,845,456 bytes. So the changes above not only reduced the code size, but also slightly lowered memory consumption at runtime. Nice!

Wall-clock performance

This file is not huge — say, 15 MB when loaded — so whatever we gained in memory consumption is a negligible win. It's nice to know that code size can be reduced, but it is not a problem for librsvg either way.

I did several measurements of the time used by the old and new versions to render the same file, and there was no significant difference. This is because although we may get better cache locality and everything, the time spent executing the element-related code is much smaller than the rendering code. That is, Cairo takes up most of the runtime of rsvg-convert, and librsvg itself takes relatively little of it.

Conclusion

At least for this case, it was feasible to reduce the amount of code emitted for generics, since this is a case where we definitely didn't need generics! The code size in the ELF file's .text section shrank by 186912 bytes, out of 2.6 MB.

For code that does need generics, one can take different approaches. For example, a function that take arguments of type AsRef<Path> can first actually obtain the &Path, and then pass that to a function that does the real work. For example, from the standard library:

impl PathBuf {
    pub fn push<P: AsRef<Path>>(&mut self, path: P) {
        self._push(path.as_ref())
    }

    fn _push(&mut self, path: &Path) {
        // lots of code here
    }
}

The push function will be monomorphized into very tiny functions that call _push after converting what you passed to a &Path reference, but the big _push function is only emitted once.

There is also the momo crate, which helps doing similar things automatically. I have not used it yet, so I can't comment further on it.

You can see the patches for librsvg in the merge request.

February 24, 2023

Introducing Elastic

Elastic is a new spring animation editor app.

Screenshot of Elastic

Ever since 1.0, libadwaita has had spring animations. These animations aren’t controlled with a duration and an easing function, but instead with physical properties: damping ratio (or optionally damping), mass, stiffness and initial velocity, as well as epsilon. While this allows for a lot of control over the animation, it can be pretty hard to understand if you’re not familiar with physics involved, and to be truly useful it needs an editor.

So, Elastic is that editor. It provides a way to tweak each parameter, explains what they do, allows to preview the animation in various ways, and generates the code to create that animation.

Screenshot of the animation graph in Elastic

Backstory

This app has a pretty convoluted history.

At one point of time, before 1.0, me and Tobias wanted to rework libadwaita demo and Tobias made new mockups for it. At about the same time, I was experimenting to see how feasible spring animations would be and needed some playground. Since it was understood that this will eventually be in libadwaita, Tobias made this design a part of the new demo:

Spring animation demo mockups

So I implemented it about as closely as I could at the time, but since then a few things happened:

  • We decided that the new demo was too design-focused to work well as, well, a demo (one of the purposes of which is using it as a test case for libadwaita itself). So, instead, we decided to make the new design a separate app called Patterns. That app is stalled for other reasons as it needs features that libadwaita doesn’t provide at the moment. Hopefully, next cycle we should have everything needed to finish it.
  • We decided that the spring animation page was too involved for a demo and was better off separately. So I split that code out and ported it to Vala instead of C.
  • As part of his GSoC project, Manuel Genovés implemented proper spring animations instead of a hacky prototype, and I ported the editor to that.

And then… everyone kinda forgot about it – Patterns was still blocked on other things, Manuel implemented a simpler animation demo for libadwaita and this one was left out.

Fast forward two years, Tobias was reviewing an app in Circle and needed a way to play with spring animations. We realized we never did anything with the editor, so I turned it into a proper app, fixed the remaining bugs and made it generate code instead of just previewing animations. We named it Spring Editor (imaginative, I know) and my impression was that design tools were covered by the new app inclusion process, so I submitted it to incubator.

It turned out more complicated than expected, so I decided to ignore it and try and get it into GNOME Circle instead. So now the Spring Editor name is too generic, so the app got renamed again, this time to Elastic (thanks Christopher Davis for that name) and it’s on Flathub now.


Anyway, the app is finally out, it’s on Flathub, so have fun. :)

Download on Flathub

UPDATE: It’s in Circle now.

February 23, 2023

Tip for debugging refcounting issues: change ref calls to copy

Over the last couple of days I’ve been looking at a refcounting issue in GLib’s D-Bus implementation.

As with many things in GLib, the allocations in this code use refcounting, rather than new/free, to manage object lifecycles. This makes it quite hard to debug lifecycle issues, because the location of a bug (a ref or unref call which isn’t correct) can be quite far removed (in time and code) from where the effects of that bug become visible. This is because the effects of refcounting problems only become visible when an object’s refcount reaches zero, or when the program ends and its refcount still hasn’t reached zero.

While debugging this code, I tried an approach I haven’t before: changing some of the ref calls on the buggy object to be copy calls instead. (Specifically, changing g_object_ref() to g_dbus_message_copy().) That split up the lifecycle of the object into smaller pieces, narrowing down the sets of ref/unref calls which could be buggy. Ultimately, this allowed me to find some bugs in the code, and hopefully those are the bugs causing the refcounting issue. Since the issue is intermittent, it’s a bit hard to be sure.

This debugging approach was possible in this case because the object I was debugging is immutable, so passing around copies of it doesn’t affect the behaviour of other bits of code vs passing around the original. Hence this approach is only applicable in some situations. But it’s another good reason why using immutable objects is quite helpful when writing code, and it’s certainly an approach I’m going to be using again when I can.

February 22, 2023

Flathub Brand Refresh

While Flatpak itself is an architecturally decentralized way to distribute apps, Flathub embraces the convenience for people to come to one place to install apps. There’s some fairly big changes coming to Flathub that I’m extremely excited about, that will make sure the new wave of Linux apps doesn’t fizzle out and we’ll maintain a sustainable application ecosystem.

Thus there is no better time to refresh what has become a little dated, Flathub’s visual identity. It centers around the core of what Flathub is — the apps themselves. The identity isn’t very loud and shouty. The flashy colors and emphasis remains on the apps themselves, Flathub is the neutral platform for the apps to shine.

Brand Guidelines

Take a peek at the brand guidelines to learn more about the new Flathub brand or download the logos in scalable format. The download buttons for app websites are also available.

Flathub Light WIP Flathub Dark WIP

Building a brand on neutral greys isn’t quite an easy sell, but precisely targets the main point of the Flathub brand. It creates the stage for apps to shine. Flathub isn’t flashy by itself. It allows the apps to be at the center of your attention.

If you’ve read until this point, you deserve a treat! Here’s some eye candy from the early stages of the process. Wallpapers derived from the elemental shapes of the logomark. Clearly off limits now, but can’t just throw them away, can I?

Flathub Patterns Flathub Candy

Big shoutout to razze for his ongoing work on the website and implementing the brand so quickly. Many thank yous to Tobias Bernard for significant involvment in this during the Berlin Mini GUADEC and Václav Vančura for sensible feedback.

Previously

February 20, 2023

Writing Bindable API, 2023 Edition

First of all, you should go on the gobject-introspection website and read the page on how to write bindable API. What I’m going to write here is going to build upon what’s already documented, or will update the best practices, so if you maintain a GObject/C library, or you’re writing one, you must be familiar with the basics of gobject-introspection. It’s 2023: it’s already too bad we’re still writing C libraries, we should at the very least be responsible about it.

A specific note for people maintaining an existing GObject/C library with an API designed before the mainstream establishment of gobject-introspection (basically, anything written prior to 2011): you should really consider writing all new types and entry points with gobject-introspection in mind, and you should also consider phasing out older API and replacing it piecemeal with a bindable one. You should have done this 10 years ago, and I can already hear the objections, but: too bad. Just because you made an effort 10 years ago it doesn’t mean things are frozen in time, and you don’t get to fix things. Maintenance means constantly tending to your code, and that doubly applies if you’re exposing an API to other people.


Let’s take the “how to write bindable API” recommendations, and elaborate them a bit.

Structures with custom memory management

The recommendation is to use GBoxed as a way to specify a copy and a free function, in order to clearly define the memory management semantics of a type.

The important caveat is that boxed types are necessary for:

  • opaque types that can only be heap allocated
  • using a type as a GObject property
  • using a type as an argument or return value for a GObject signal

You don’t need a boxed type for the following cases:

  • your type is an argument or return value for a method, function, or virtual function
  • your type can be placed on the stack, or can be allocated with malloc()/free()

Additionally, starting with gobject-introspection 1.76, you can specify the copy and free function of a type without necessarily registering a boxed type, which leaves boxed types for the thing they were created: signals and properties.

Addendum: object types

Boxed types should only ever be used for plain old data types; if you need inheritance, then the strong recommendation is to use GObject. You can use GTypeInstance, but only if you know what you’re doing; for more information on that, see my old blog post about typed instances.

Functionality only accessible through a C macro

This ought to be fairly uncontroversial. C pre-processor symbols don’t exist at the ABI level, and gobject-introspection is a mechanism to describe a C ABI. Never, ever expose API only through C macros; those are for C developers. C macros can be used to create convenience wrappers, but remember that anything they call must be public API, and that other people will need to re-implement the convenience wrappers themselves, so don’t overdo it. C developers deserve some convenience, but not at the expense of everyone else.

Addendum: inline functions

Static inline functions are also not part of the introspectable ABI of a library, because they cannot be used with dlsym(); you can provide inlined functions for performance reasons, but remember to always provide their non-inlined equivalent.

Direct C structure access for objects

Again, another fairly uncontroversial rule. You shouldn’t be putting anything into an instance structure, as it makes your API harder to future-proof, and direct access cannot do things like change notification, or memoization.

Always provide accessor functions.

va_list

Variadic argument functions are mainly C convenience. Yes, some languages can support them, but it’s a bad idea to have this kind of API exposed as the only way to do things.

Any variadic argument function should have two additional variants:

  • a vector based version, using C arrays (zero terminated, or with an explicit length)
  • a va_list version, to be used when creating wrappers with variadic arguments themselves

The va_list variant is kind of optional, since not many people go around writing variadic argument C wrappers, these days, but at the end of the day you might be going to write an internal function that takes a va_list anyway, so it’s not particularly strange to expose it as part of your public API.

The vector-based variant, on the other hand, is fundamental.

Incidentally, if you’re using variadic arguments as a way to collect similarly typed values, e.g.:

// void
// some_object_method (SomeObject *self,
//                     ...) G_GNUC_NULL_TERMINATED

some_object_method (obj, "foo", "bar", "baz", NULL);

there’s very little difference to using a vector and C99’s compound literals:

// void
// some_object_method (SomeObject *self,
//                     const char *args[])

some_object_method (obj, (const char *[]) {
                      "foo",
                      "bar",
                      "baz",
                      NULL,
                    });

Except that now the compiler will be able to do some basic type check and scream at you if you’re doing something egregiously bad.

Compound literals and designated initialisers also help when dealing with key/value pairs:

typedef struct {
  int column;
  union {
    const char *v_str;
    int v_int;
  } value;
} ColumnValue;

enum {
  COLUMN_NAME,
  COLUMN_AGE,
  N_COLUMNS
};

// void
// some_object_method (SomeObject *self,
//                     size_t n_columns,
//                     const ColumnValue values[])

some_object_method (obj, 2,
  (ColumnValue []) {
    { .column = COLUMN_NAME, .data = { .v_str = "Emmanuele" } },
    { .column = COLUMN_AGE, .data = { .v_int = 42 } },
  });

So you should seriously reconsider the amount of variadic arguments convenience functions you expose.

Multiple out parameters

Using a structured type with a out direction is a good recommendation as a way to both limit the amount of out arguments and provide some future-proofing for your API. It’s easy to expand an opaque pointer type with accessors, whereas adding more out arguments requires an ABI break.

Addendum: inout arguments

Don’t use in-out arguments. Just don’t.

Pass an in argument to the callable for its input, and take an out argument or a return value for the output.

Memory management and ownership of inout arguments is incredibly hard to capture with static annotations; it mainly works for scalar values, so:

void
some_object_update_matrix (SomeObject *self,
                           double *xx,
                           double *yy,
                           double *xy,
                           double *yx)

can work with xx, yy, xy, yx as inout arguments, because there’s no ownership transfer; but as soon as you start throwing things in like pointers to structures, or vectors of string, you open yourself to questions like:

  • who allocates the argument when it goes in?
  • who is responsible for freeing the argument when it comes out?
  • what happens if the function frees the argument in the in direction and then re-allocates the out?
  • what happens if the function uses a different allocator than the one used by the caller?
  • what happens if the function has to allocate more memory?
  • what happens if the function modifies the argument and frees memory?

Even if gobject-introspection nailed down the rules, they could not be enforced, or validated, and could lead to leaks or, worse, crashes.

So, once again: don’t use inout arguments. If your API already exposes inout arguments, especially for non-scalar types, consider deprecations and adding new entry points.

Addendum: GValue

Sadly, GValue is one of the most notable cases of inout abuse. The oldest parts of the GNOME stack use GValue in a way that requires inout annotations because they expect the caller to:

  • initialise a GValue with the desired type
  • pass the address of the value
  • let the function fill the value

The caller is then left with calling g_value_unset() in order to free the resources associated with a GValue. This means that you’re passing an initialised value to a callable, the callable will do something to it (which may or may not even entail re-allocating the value) and then you’re going to get it back at the same address.

It would be a lot easier if the API left the job of initialising the GValue to the callee; then functions could annotate the GValue argument with out and caller-allocates=1. This would leave the ownership to the caller, and remove a whole lot of uncertainty.

Various new (comparatively speaking) API allow the caller to pass an unitialised GValue, and will leave initialisation to the callee, which is how it should be, but this kind of change isn’t always possible in a backward compatible way.

Arrays

You can use three types of C arrays in your API:

  • zero-terminated arrays, which are the easiest to use, especially for pointers and strings
  • fixed-size arrays
  • arrays with length arguments

Addendum: strings and byte arrays

A const char* argument for C strings with a length argument is not an array:

/**
 * some_object_load_data:
 * @self: ...
 * @str: the data to load
 * @len: length of @str in bytes, or -1
 *
 * ...
 */
void
some_object_load_data (SomeObject *self,
                       const char *str,
                       ssize_t len)

Never annotate the str argument with array length=len. Ideally, this kind of function should not exist in the first place. You should always use const char* for NUL-terminated strings, possibly UTF-8 encoded; if you allow embedded NUL characters then use a bytes array:

/**
 * some_object_load_data:
 * @self: ...
 * @data: (array length=len) (element-type uint8): the data to load
 * @len: the length of the data in bytes
 *
 * ...
 */
void
some_object_load_data (SomeObject *self,
                       const unsigned char *data,
                       size_t len)

Instead of unsigned char you can also use uint8_t, just to drive the point home.

Yes, it’s slightly nicer to have a single entry point for strings and byte arrays, but that’s just a C convenience: decent languages will have a proper string type, which always comes with a length; and string types are not binary data.

Addendum: GArray, GPtrArray, GByteArray

Whatever you do, however low you feel on the day, whatever particular tragedy befell your family at some point, please: never use GLib array types in your API. Nothing good will ever come of it, and you’ll just spend your days regretting this choice.

Yes: gobject-introspection transparently converts between GLib array types and C types, to the point of allowing you to annotate the contents of the array. The problem is that that information is static, and only exists at the introspection level. There’s nothing that prevents you from putting other random data into a GPtrArray, as long as it’s pointer-sized. There’s nothing that prevents a version of a library from saying that you own the data inside a GArray, and have the next version assign a clear function to the array to avoid leaking it all over the place on error conditions, or when using g_autoptr.

Adding support for GLib array types in the introspection was a well-intentioned mistake that worked in very specific cases—for instance, in a library that is private to an application. Any well-behaved, well-designed general purpose library should not expose this kind of API to its consumers.

You should use GArray, GPtrArray, and GByteArray internally; they are good types, and remove a lot of the pain of dealing with C arrays. Those types should never be exposed at the API boundary: always convert them to C arrays, or wrap them into your own data types, with proper argument validation and ownership rules.

Addendum: GHashTable

What’s worse than a type that contains data with unclear ownership rules decided at run time? A type that contains twice the amount of data with unclear ownership rules decided at run time.

Just like the GLib array types, hash tables should be used but never directly exposed to consumers of an API.

Addendum: GList, GSList, GQueue

See above, re: pain and misery. On top of that, linked lists are a terrible data type that people should rarely, if ever, use in the first place.

Callbacks

Your callbacks should always be in the form of a simple callable with a data argument:

typedef void (* SomeCallback) (SomeObject *obj,
                               gpointer data);

Any function that takes a callback should also take a “user data” argument that will be passed as is to the callback:

// scope: call; the callback data is valid until the
// function returns
void
some_object_do_stuff_immediately (SomeObject *self,
                                  SomeCallback callback,
                                  gpointer data);

// scope: notify; the callback data is valid until the
// notify function gets called
void
some_object_do_stuff_with_a_delay (SomeObject *self,
                                   SomeCallback callback,
                                   gpointer data,
                                   GDestroyNotify notify);

// scope: async; the callback data is valid until the async
// callback is called
void
some_object_do_stuff_but_async (SomeObject *self,
                                GCancellable *cancellable,
                                GAsyncReadyCallback callback,
                                gpointer data);

// not pictured here: scope forever; the data is valid fori
// the entirety of the process lifetime

If your function takes more than one callback argument, you should make sure that it also takes a different user data for each callback, and that the lifetime of the callbacks are well defined. The alternative is to use GClosure instead of a simple C function pointer—but that comes at a cost of GValue marshalling, so the recommendation is to stick with one callback per function.

Addendum: the closure annotation

It seems that many people are unclear about the closure annotation.

Whenever you’re describing a function that takes a callback, you should always annotate the callback argument with the argument that contains the user data using the (closure argument) annotation, e.g.

/**
 * some_object_do_stuff_immediately:
 * @self: ...
 * @callback: (scope call) (closure data): the callback
 * @data: the data to be passed to the @callback
 *
 * ...
 */

You should not annotate the data argument with a unary (closure).

The unary (closure) is meant to be used when annotating the callback type:

/**
 * SomeCallback:
 * @self: ...
 * @data: (closure): ...
 *
 * ...
 */
typedef void (* SomeCallback) (SomeObject *self,
                               gpointer data);

Yes, it’s confusing, I know.

Sadly, the introspection parser isn’t very clear about this, but in the future it will emit a warning if it finds a unary closure on anything that isn’t a callback type.

Ideally, you don’t really need to annotate anything when you call your argument user_data, but it does not hurt to be explicit.


A cleaned up version of this blog post will go up on the gobject-introspection website, and we should really have a proper set of best API design practices on the Developer Documentation website by now; nevertheless, I do hope people will actually follow these recommendations at some point, and that they will be prepared for new recommendations in the future. Only dead and unmaintained projects don’t change, after all, and I expect the GNOME stack to last a bit longer than the 25 years it already spans today.

February 19, 2023

Dilemma’s in Rust Land: porting a GNOME library to Rust

It has been a while since my last post, so I figured I just picked up a topic that has been around my mind lately.

After I ported the RSVG Pixbuf Loader to Rust (although I gave up the meson-fu to Federico and Bilal) I decided that maybe I should give a try at porting the WebP Pixbuf Loader.

webp-pixbuf-loader is probably the only FOSS project I have started on my own that I have managed to maintain in the long run without orphaning or giving it up to someone else. I wrote it out of curiosity and it turns out plenty of people find it rather useful as webp is pretty popular for storing comic book backups and other media.

The WebP Pixbuf Loader is relatively small although ever since animation support was contributed it grew quite a bit. I’ve been handling a couple of issues ranging from endianness to memory leaks, I thought it was probably worth the while to give it some Rusty love.

Porting the static image support was relatively quick, there’s, but it took me a while to understand how animation works in GDK-Pixbuf as the original animation support in C was contributed by alanhaw.

I suspect I am prolly the first person to use the GdkPixbufLoader APIs to implement a new pixbuf loader, I had request a few fixes upstream, kudos to Sebastian Dröge and Bilal for handling those swiftly and shipping them in the last release.

Anyhow, last month I finally made it all work:

Hesitations

Now comes the hesitation part, regardless of integrating the former tests in the build system (Cargo is great at embedded unit testing but meson is better at running an integration test plan), my main gripe is that it turns out that there’s quite a few people packaging this, not just for Linux distros but also BSDs, Illumos, Windows and brew/macOS…

I really don’t know what the impact would be for anyone packaging outside of the Linux world, I have a few CI pipelines for Windows but obviously I am about to break them if I switch.

I am pondering the idea of releasing a bunch of beta releases and hoping package maintainers will start taking notice that I’m moving on, but I am trying to be mindful of how much time they need to sink for the Rust move to happen and weight that against the net benefit.

The other part that makes me hesitate over flipping the switch is the measure of the overall benefit. Sure Rust is nicer to maintain, but it still is a small codebase, Rust adds a bunch of runtime dependencies (bindings) and it is not clear to me how healthy the webp bindings are going to be long term, there are two similarly named bindings one has more frequent releases and the other is more complete which is annoying. These bindings bring an issue of size: the Rust port is 4MB in size vs. 76KB for the C implementation.

Not sure what to do, feel free to add your thoughts in the comment section.

UPDATE: It looks like the size issue can be addressed and total size drops down to ~300KB by using the right flags.

February 17, 2023

How is Linux used by FIT BUT students

The Faculty of Information Technology of Brno University of Technology is one of the two top computer science schools in Brno, Czech Republic. Our development office of Red Hat has intensive cooperation with them including educating students about Linux and open source software. To find out more about how they use Linux, we ran a survey that collected answers from 176 students which is a pretty good sample. I promised to share results publicly, so here they are:

The following chart shows the distribution of responders by year of school. The survey was primarily targeting students in the first year which is why they make up over 50% of the responses.

The following chart shows how many students had experience with a Linux distribution prior their studies at the university. 46% did which shows a pretty good exposure to Linux at high schools.

And now what desktop OS students use primarily. Windows are dominating, but Linux is used as a primary OS by roughly one third of students. macOS is only at 10%. Although we gave responders an option to specify other OSes, no one submitted, for example, BSD.

The following chart shows in what form students use Linux primarily (as either a primary or secondary OS). 44% of students have it installed on their desktop/laptop. 31% use Windows Subsystem for Linux. School programming assignments have to run on Linux, so if they want to stick with Windows, WSL is the easiest way for them. Virtualization is at 9% and remote server at 13% (I suspect it’s mostly uni servers where students can test their assignments before submission).

And here come shares of Linux distributions. Responders could pick multiple options, so the total is over 100%. Basically the only relevant distributions among FIT BUT students are Ubuntu, Fedora, Arch Linux and Debian.

Ubuntu has a clear lead. It’s the default option for WSL where it is on vast majority of installations, so I wondered what the share would be without WSL.

Without WSL the gap between Ubuntu and the rest of the pack is smaller. And since I’m from the Red Hat desktop team I also wondered what are the shares among students who indicated they use Linux primarily on desktop/laptop.

When it comes to desktop computers and laptops the shares of Fedora and Ubuntu are almost the same. That shows two things: 1. Fedora is strong on the desktop among local students, 2. being the default option in WSL gives Ubuntu an advantage in mindshare. Fedora is not even officially available for WSL, but even if it was, it wouldn’t probably change much because other distros are available in the Microsoft Store and only one student of out 50+ who primarily use WSL responded that they use something else than Ubuntu. WSL is probably used by users who want some Linux in their Windows and don’t care much which one it is, so they stay with the default.

We also asked students what prevents them from using Linux primarily. By far the most frequent answer (80%) was “Software I use is not available for Linux”, followed by “I don’t like the UX and logic of the OS” (28%) and “Compatibility with my hardware” (11%). Some students also responded that they simply hadn’t had enough time to get familiar with Linux and are staying with what they know. Other reasons were marginal.

Status update, 17/02/2023

This month I attended FOSDEM for the first time since 2017. In addition to eating 4 delicious waffles, I had the honour of presenting two talks, the first in the Testing & Automation devroom on Setting up OpenQA testing for GNOME.

GNOME’s initial OpenQA testing is mostly implemented now and it’s already found its first real bug. The next step is getting more folk interested within GNOME, so we can ensure ongoing maintenance of the tests and infra, and ensure a bus factor of > 1. If you see me at GUADEC then I will probably talk to you about OpenQA, be prepared!! 🙂

My second talk was in the Python devroom, on DIY music recommendations. I intermittently develop a set of playlist generation tools named Calliope, and this talk was mostly aiming to inspire people to start similar fun & small projects, using simple AI techniques that you can learn in a weekend, and taking advantage of the amazing resource that is Musicbrainz. It seemed to indeed inspire some of the audience and led to an interesting chat with Rob Kaye of the Metabrainz Foundation – there is more cool stuff on the way from them.

Here’s a fantastic sketch of the talk by Jeroen Heijmans:

Talk summary sketch, CC BY-SA 4.0

I didn’t link to this in the talk, but apropos of nothing here’s an interesting video entitled Why Spotify Will Eventually Fail.

On the Saturday I met up with Carlos Garnacho and gatecrashed the GNOME docs hackfest, discussing various improvements around search in GNOME. Most of these are now waiting for developer time as they are too large to be done in occasional moments of evening and weekend downtime, get in touch if you want to find out more!

I must also shout out Marco Trevisan for showing me where to get a decent meal near Madrid Chamartín station on the way home.

Meanwhile at Codethink I have been getting more involved in marketing. Its a company that exists in two worlds, commercial software services on one side and community-driven open source software on the other, often trying our best to build bridges between the two. There aren’t many marketing graduates who are experts in open source, and neither many experienced software developers who want to work fulltime on managing social media, so we are still figuring out the details…

Anyway, the initial outcome is that Codethink is now on the Fediverse – follow us here! @codethink@social.codethink.co.uk

February 16, 2023

WebRTC in WebKitGTK and WPE, status updates, part I

Some time ago we at Igalia embarked on the journey to ship a GStreamer-powered WebRTC backend. This is a long journey, it is not over, but we made some progress. This post is the first of a series providing some insights of the challenges we are facing and the plans for the next release cycle(s).

Most web-engines nowadays bundle a version of LibWebRTC, it is indeed a pragmatic approach. WebRTC is a huge spec, spanning across many protocols, RFCs and codecs. LibWebRTC is in fact a multimedia framework on its own, and it’s a very big code-base. I still remember the suprised face of Emilio at the 2022 WebEngines conference when I told him we had unusual plans regarding WebRTC support in GStreamer WebKit ports. There are several reasons for this plan, explained in the WPE FAQ. We worked on a LibWebRTC backend for the WebKit GStreamer ports, my colleague Thibault Saunier blogged about it but unfortunately this backend has remained disabled by default and not shipped in the tarballs, for the reasons explained in the WPE FAQ.

The GStreamer project nowadays provides a library and a plugin allowing applications to interact with third-party WebRTC actors. This is in my opinion a paradigm shift, because it enables new ways of interoperability between the so-called Web and traditional native applications. Since the GstWebRTC announcement back in late 2017 I’ve been experimenting with the idea of shipping an alternative to LibWebRTC in WebKitGTK and WPE. The initial GstWebRTC WebKit backend was merged upstream on March 18, 2022.

As you might already know, before any audio/video call, your browser might ask permission to access your webcam/microphone and even during the call you can now share your screen. From the WebKitGTK/WPE perspective the procedure is the same of course. Let’s dive in.

WebCam/Microphone capture

Back in 2018 for the LibWebRTC backend, Thibault added support for GStreamer-powered media capture to WebKit, meaning that capture devices such as microphones and webcams would be accessible from WebKit applications using the getUserMedia spec. Under the hood, a GStreamer source element is created, using the GstDevice API. This implementation is now re-used for the GstWebRTC backend, it works fine, still has room for improvements but that’s a topic for a follow-up post.

A MediaStream can be rendered in a <audio> or <video> element, through a custom GStreamer source element that we also provide in WebKit, this is all internally wired up so that the following JS code will trigger the WebView in natively capturing and rendering a WebCam device using GStreamer:

<html>
  <head>
    <script>
    navigator.mediaDevices.getUserMedia({video: true, audio: false }).then((mediaStream) => {
        const video = document.querySelector('video');
        video.srcObject = mediaStream;
        video.onloadedmetadata = () => {
            video.play();
        };
    });
    </script>
  </head>
  <body>
    <video/>
  </body>
</html>

When this WebPage is rendered and after the user has granted access to capture devices, the GStreamer backends will create not one, but two pipelines.

flowchart LR
    pipewiresrc-->videoscale-->videoconvert-->videorate-->valve-->appsink
flowchart LR
    subgraph mediastreamsrc
    appsrc-->srcghost[src]
    end
    subgraph playbin3
       subgraph decodebin3
       end
       subgraph webkitglsink
       end
       decodebin3-->webkitglsink
    end
    srcghost-->decodebin3

The first pipeline routes video frames from the capture device using pipewiresrc to an appsink. From the appsink our capturer leveraging the Observer design pattern notifies its observers. In this case there is only one observer which is a GStreamer source element internal to WebKit called mediastreamsrc. The playback pipeline shown above is heavily simplified, in reality more elements are involved, but what matters most is that thanks to the flexibility of Gstreamer, we can leverage the existing MediaPlayer backend that we at Igalia have been maintaining for more than 10 years, to render MediaStreams. All we needed was a custom source element, the rest of our MediaPlayer didn’t need much changes to support this use-case.

One notable change we did since the initial implementation though is that for us a MediaStream can be either raw, encoded or even encapsulated in a RTP payload. So depending on which component is going to render the MediaStream, we have enough flexibility to allow zero-copy, in most scenarios. In the example above, typically the stream will be raw from source to renderer. However, some webcams can provide encoded streams. WPE and WebKitGTK will be able to internally leverage these and in some cases allow for direct streaming from hardware device to outgoing PeerConnection without third-party encoding.

Desktop capture

There is another JS API, allowing to capture from your screen or a window, called getDisplayMedia, and yes, we also support it! Thanks to these recent years ground-breaking progress of the Linux Desktop such as PipeWire and xdg-desktop-portal we can now stream your favorite desktop environment over WebRTC. Under the hood when the WebView is granted access to the desktop capture through the portal, our backend creates a pipewiresrc GStreamer element, configured to source from the file descriptor provided by the portal, and we have a healthy raw video stream.

Here’s a demo:

WebAudio capture

What more, yes you can also create a MediaStream from a WebAudio node. On the backend side, the GStreamerMediaStreamAudioSource fills GstBuffers from the audio bus channels and notifies third-parties internally observing the MediaStream, such as outgoing media sources, or simply an <audio> element that was configured to source from the given MediaStream. I have no demo for this, you’ll have to take my word.

Canvas capture

But wait there is more. Did I hear canvas? Yes we can feed your favorite <canvas> to a MediaStream. The JS API is called captureStream, its code is actually cross-platform but defers to the HTMLCanvasElement::toVideoFrame() method which has a GStreamer implementation. The code is not the most optimal yet though due to shortcomings of our current graphics pipeline implementation. Here is a demo of Canvas to WebRTC running in the WebKitGTK MiniBrowser:

Wrap-up

So we’ve got MediaStream support covered. This is only one part of the puzzle though. We are facing challenges now on the PeerConnection implementation. MediaStreams are cool but it’s even better when you can share them with your friends on the fancy A/V conferencing websites, but we’re not entirely ready for this yet in WebKitGTK and WPE. For this reason, WebRTC is not yet enabled by default in the upcoming WebKitGTK and WPE 2.40 releases. We’re just not there yet. In the next part of these series I’ll tackle the PeerConnection backend on which we’re working hard on these days, both in WebKit and in GStreamer.

Happy hacking and as always, all my gratitude goes to my fellow Igalia comrades for allowing me to keep working on these domains and to Metrological for funding some of this work. Is your organization or company interested in leveraging modern WebRTC APIs from WebKitGTK and/or WPE? If so please get in touch with us in order to help us speed-up the implementation work.

February 13, 2023

Los cajones - Dos mesitas de noche, parte 3

Disculpen la pausa tan larga en escribir este blog. Comencé a poner mis hilos de carpintería en Mastodon y dejé olvidado este lugar. Pero bien, prosigamos.

Dos mesitas de noche:

En la parte 2 hicimos las patas para una de las mesitas. Ahora va el cajón.

Estoy viendo que no tengo tantas fotos del proceso como quisiera, pero ojalá lo siguiente sirva.

Las cuatro tablas del cuerpo del cajón tienen que quedar muy bien escuadradas. Aquí me fui un poco chueco con el serrucho, según la marca del cuchillo:

Corte del extremo de la tabla un poco chueco

La arreglé con la chancla, el cepillo pequeñito. Primero se lubrica (yo uso grasa de tocino clarificada), y se cepilla con cuidado hasta llegar a la línea:

Lubricando la chancla con grasa de tocino clarificada Cepillando el extremo con la chancla hasta la línea

Hasta que queda bien escuadrado.

Extremo ya bien escuadrado

Con el gramil se marca el largo de las colas de milano en uno de los extremos de la tabla que va a ser el frente del cajón, y con escuadra falsa se trazan las colas. Luego, se marca el largo de las colas con el gramil sobre las tablas laterales.

Se marca el largo de las colas con el gramil Se marca la misma distancia en las tablas laterales

Ahora, al revés. Se transfiere el grosor de las tablas laterales a la tabla del frente del cajón.

Transfiriendo el grosor de las tablas laterales a la tabla frontal

Ahora, con la escuadra, se marca el largo de las colas a partir de las diagonales marcadas en el extremo.

Marcando con la escuadra

Marcamos las partes que van a ser basura y podemos empezar a cortar. Como siempre, se empieza el corte del serrucho siguiendo las líneas en dos planos: ambas líneas se intersectan y determinan un plano en el espacio, que es el plano de la hoja del serrucho.

Las partes de desecho marcadas, para no confundirse Comenzando el corte con serrucho Los cortes ya hechos, y cortes intermedios para formonear

En la última foto, los cortes en las diagonales se pasan de la línea. No importa; no se van a ver, pues quedan por detrás de la parte frontal, y así se facilita después el corte con formón.

Ahora vamos a escarbar las colas de milano con un formón. Se hace poco a poco, desde el extremo hacia adentro, con cuidado para no romper la parte más estrecha de la tabla.

Comenzando el corte con formón vertical Sacamos un pedacito... Poco a poco se escarba la cola completa Espacio terminado para una cola Las tres colas escarbadas

Hay que transferir la forma de las colas a las tablas laterales. Para facilitar el trabajo, conviene tener un tope en las tablas laterales para acomodar la tabla frontal encima. Ese tope lo hago sacando un pequeñísimo escalón en el extremo de las tablas laterales. Se pone una tira de madera, se marca con cuchillo, se quitan las esquinas y se rebaja el escalón con un guillame.

Tira puesta en la posición del escalón Quitando las esquinas con formón para que no se astille Escalón rebajado con guillame

Aquí es donde me faltan fotos. No tengo imágenes de cómo se hace descansar la tabla frontal sobre ese escalón de las laterales, pero es ahí donde se marca con cuidado la forma de las colas de milano. Igual, se serruchan y luego se saca el desecho con formón.

Todas las tablas requieren una ranura para la tabla delgadita del fondo del dajón. Esa ranura se puede acomodar adentro de la cola de milano de hasta abajo.

Ranurando la tabla frontal:

Ranurando la tabla frontal, con la ranura dentro de la cola de milano

Ranurando una tabla lateral:

Ranurando una tabla lateral

Cómo sostener la tabla para ranurarla, con un tope y un clavito:

Un tope puesto a lo largo, y un clavito para sostener el extremo de la tabla

La tabla del fondo del cajón casi lista para entrar en la ranura:

En esa ranura va la tabla del fondo del cajón

Ya se puede armar el cajón, pero luego hay que escuadrarlo. Primero lo armamos y pegamos:

Uno de los ensambles ya armado

Y con las reglas de enderezar, se ve si el cajón está torcido. Lo está un poco; lo cepillamos con cuidado para no astillar las tablas transversales.

Uno de los ensambles ya armado

Ya podemos poner el cajón en sus rieles y probar la altura. Donde le sobre, se rebaja con el cepillo.

Probando la altura del cajón metido en su lugar

Al fondo de los rieles se pegan bloquecitos de madera que va a servir como tope. Cada tope se ajusta de forma independiente, rebajándolo poco a poco con formón, hasta que el frente del cajón quede alineado con el frente de la mesa.

Un bloquecito al fondo como tope

Fíjate en los rieles; abarcan el ancho interno de las patas y no le dejan nada de juego al cajón. El chiste es que entre lo más derecho posible, sin espacio para que entre sesgado — si sólo puede entrar derechito, no se va a atorar.

Sobre los topes... aquí está el cajón metido hasta el fondo, y sobresale del frente de la mesita:

Un bloquecito al fondo como tope

Rebajamos el bloquecito de cada lado...

Un bloquecito al fondo como tope Un bloquecito al fondo como tope

... hasta que todo el frente del cajón queda alineado y al ras.

Un bloquecito al fondo como tope

Ahora hay que preparar el fondo del cajón, para que quepa en las ranuras. Queremos que quede así:

El fondo ya preparado y metido en sus ranuras

Esos rebajes inclinados se hacen inclinando el cepillo: Se inclina el cepillo al rebajar

Con un poco de cuidado, queda derecho. O primero se puede marcar con lápiz paralelo al borde. No tiene mucha importancia, porque es un lado que no se ve.

Cepillando el borde en diagonal

La parte de atrás sobresale. La marcamos y cepillamos al ras. La parte de atrás sobresale de la parte trasera

Marcando con lápiz

February 09, 2023

Updates from inside GTK

Some of the core GTK developers recently got together for a few days to do some focused work and talk about current and future plans.

It is difficult to summarize three days of discussions in a blog post, but here are some of the highlights.

Icon themes

The GtkIconTheme code has been with us for a long time. It implements the icon theme spec, and comes from an era when we were shipping big sets of icons with the desktop and themes were expected to switch them out. That is not really how icons are made or used today.

We need a better solution for the workflow from a designer making icons a set of icons in a sheet to the developer copying individual icons into their app.

Inside GTK, this will need some form of “asset manager” to maintain the mapping from icon name to image / file / resource.

Languages

While we can’t get away from providing a C interface with gobject-introspection metadata for all our language bindings, it could be nice to use a more expressive language and a more powerful compiler than C has to offer.

Of course we can’t and won’t rewrite all of GTK in a different language. It would be nice to experiment with replacing smaller parts. Allowing new code to be written in different languages would also potentially bring in new contributors.

Input

We discussed the scroll speed question, and decided to write up an explanatory comment in the issue about what we consider the right solution:

  • treat wheel and touchpad scrolling separately
  • inject configuration from the control-center/compositor into libinput
  • gtk gets it via events

The other big input question we talked about is ‘asynchronous event handling’ and its problems. The two main cases where this comes up are webkit, with its ui<>web process communication, and IBus. In both cases, we think that there is no actual interest in reinjecting unhandled events into the GTK capture/bubble propagation. Instead, such leftover event should just be handled ‘locally’ (in the IBus case, adding/removing characters to the entry, or moving the cursor).

Platform libraries

With GTK4, we’ve made the intentional change to move away from having everything in GTK itself, and instead introduced the idea of  ‘platform libraries’ like libadwaita to carry the more platform-specific widgetry.

Overall, we are happy with how this has turned out, and we would like to continue with this approach. There is maybe room for moving some things that are more plumbing than widgetry back into GTK itself.

GTK5 ?

We need to open a .90 branch to do things that would break the APIs that we have deprecated now (like the file chooser, and more general the chooser dialog/widget split). Some of us have been itching to start on that work. But there’s also still a lot of work to be done in 4.x (GtkListView fixes, for example).

With an eye towards the color management work that is planned to land in 4.12, the suggestion is to open a 4.90 development branch after 4.12. That would put it towards the end of this year, and 3 years after the 4.0 release, which seems reasonable.

The End

On the last day, we had the pleasure of hosting both the documentation and the tracker teams at our place.

Three hackfests in a room!

We’d like to thank the GNOME foundation for supporting our meeting. ❤

February 08, 2023

Smoother Scrolling of Text Views

When working on GTK 4, special care was taken to ensure that most of a GtkTextView‘s content could be rendered without GL program changes and maximal use of glDrawArrays() with vertices from a VBO.

That goes a long way towards making smooth scrolling.

In recent releases, GTK gained support for scrolling using more precise scroll units. On macOS with Apple touchpads, for example, that might map physical distance to a certain number of logical pixels within the application.

If you’re at 2x scaling, you might get values from the input system with “half pixel” values (e.g. .5) since that would map just fine to the physical pixel boundary of the underlying display server.

That’s great for our future, but not everything from GTK’s early designs around X11 have been excised from the toolkit. Currently, widget allocations are still integer based, meaning at 2x scaling they will sit on 2x physical pixel boundaries even though 1x physical pixel boundaries would be just fine to keep lines sharp (assuming you aren’t fractionally scaling afterwards).

Furthermore, GtkTextView works in logical pixels as well. That means that if you want to smooth scroll a text view and you get those 0.5 logical pixel values (1x physical pixels) you wont scroll until you get to 1.0 which then jumps you 2x physical pixels.

That can create some unsightly jitter, most noticeable during kinetic deceleration.

To fix the widget allocation situation, a future ABI break in GTK would have to move to some sort of float/double for widget coordinates. Everything underneath GTK (like GSK/GDK/Graphene/etc) is already largely doing this as GDK went through substantial improvements and simplification for GTK 4.

But with a little creativity, abstraction, and willingness to completely break ABI for a prototype, you can get an idea of what that would be like.

I put together a quick branch today which makes GtkTextView use double for coordinates so that I could push it to snap to physical pixel boundaries.

The fun thing is finding all the ways it breaks stuff. Like text underline getting into situations where it looks different as you scroll or having to allocate GtkWidget embedded within the GtkTextView on logical pixels.

Like I said earlier, it completely breaks ABI of GtkTextView, so don’t expect to replace your system GTK with it or anything.

P.S. I’m on Mastodon now.

Profiling & optimizing GNOME Software

Over the past few months, the Endless OS Foundation has been putting focus on improving GNOME Software’s reliability and performance. Endless OS is an OSTree-based immutable OS, and applications are entirely distributed as Flatpaks. GNOME Software is the frontend to that, and since our target userbase is definitely not going to use a terminal, we need to make sure GNOME Software delivers a good experience.

This focus has been materializing with Philip’s long effort to switch GNOME Software to a different threading model, more suitable to what Software does these days; and more recently, I’ve been tasked to look at GNOME Software’s performance from a profiling perspective.

Profile, Profile, Profile

I’ve been looking at performance-related tasks for many years now. It’s a topic of investigation that I personally enjoy, even if it’s a bit annoying and frustrating at times. My first real, big performance investigation was some good six years ago, on GNOME Music. Back then, the strategy to profiling was: guess what’s wrong, change that, see if it works. Not exactly a robust strategy. Luckily, it worked for that particular case.

Then over the next year I started to contribute to Mutter and GNOME Shell, and one of my first bigger performance-related investigations in this area involved frame timings, GJS, and some other complicated stuff. That one was a tad bit more complicated, and it still relied on gut feelings, but luckily some good improvements came out of that too.

Those two instances of gut-based profiling are successful when put like this, but that’s just a trick with words. Those two efforts were successful due to a mix of ungodly amounts of human effort, and luck. I say this because just a few months after, the GNOME Performance Hackfest took place in Cambridge, UK, and that was the beginning of a massive tsunami that took the GNOME community.

This tsunami is called profiling.

In that hackfest, Christian and Jonas started working on integrating Sysprof and Mutter / GNOME Shell. Sysprof was already able to collect function calls without any assistance, but this integration work enabled, for example, measuring where in the paint routines was GNOME Shell taking time to render. Of course, we could have printed timestamps in the console and parsed that, but let’s be honest, that sucks. Visualizing data makes the whole process so much easier.

So much easier that even a fool like myself could contribute. After quickly picking up the work where Jonas left, GNOME Shell was able to produce profiling data like below:

GNOME Shell frame timing marks on Sysprof, as it was in 2018

That integration, alone, allowed us to abandon the Old Ways of profiling, and jump head first into a new era of data-based optimizations. Just looking at the data above, without even looking at function calls, we can extract a lot of information. We can see that the layout phase was taking 4ms, painting was taking 5ms, picking was taking ~2ms, and the whole process was taking ~12ms. Looking at this data in depth is what allowed, for example, reducing pick times by 92%, and rendering times by a considerable factor too.

The most important lesson I could derive from that is: optimizing is borderline useless if not backed by profiling data. Always profile your app, and only optimize based on profiling data. Even when you think you know which specific part of your system is problematic, profile that first, just to confirm.

With these lessons learned, of course, I used this approach to profile GNOME Software.

Profiling an App Store

Naturally, the problem domain that GNOME Software – an app store – operates into is vastly different from that of GNOME Shell – a compositor. GNOME Software is also in the middle of a major transition to a new threading model, and that adds to the challenge of profiling and optimizing it. How do you optimize something that’s about to receive profound changes?

Fortunately for us, GNOME Software already had a basic level of integration with Sysprof, which gave some introspection on how it was spending its time, but not enough. My initial reaction was to extend this Sysprof integration and cover more events, in a more hierarchical manner. The goal was to improve whatever was making navigation through categories slow.

This proved to be a successful strategy. Here’s what profiling GNOME Software gave us:

Zoom in in the image. Take a few seconds to analyse it. Can you spot where the issue is?

If not, let’s do this together.

This slice of profiling shows that the GsPluginJobListApps operation (first line) took 4 whole seconds to execute. That’s a long time! This operation is executed to list the apps of a particular category. This means people have to wait 4 entire seconds, just to see the apps of a category. Certainly a usability killer, exploring these apps is supposed to be fun and quick, and this issue hurts that.

You may notice that there are other sub-events beneath GsPluginJobListApps. The first line is the timing of the whole operation, and beneath that, we have the timings of each individual sub-operation it does to end up with a list of apps.

Skimming through the entire profiler window, the widest black bars beneath the first line are, in order of appearance:

  • GsPluginJobListApps:flatpak taking 600ms
  • GsPluginJobRefine taking 3.6 seconds
    • GsPluginJobRefine:flatpak taking 600ms
    • GsPluginJobRefine:icons taking about 3 seconds

What that tells us is that the GsPluginJobListApps operation runs a GsPluginJobRefine sub-operation, and the icons are what’s taking most of the time there. Icons!

Refining is GNOME Software terminology for gathering metadata about a certain app, such as the name of the author, origin of the application, description, screenshots, icons, etc.

Icons!

Contrary to all my expectations, what was clear in these profiling sessions is that loading icons was the worst offender to GNOME Software’s performance when navigating through categories. This required further investigation, since there’s a fair bit of code to make icon management fast and tight on memory.

It didn’t take long to figure out what was happening.

Some applications declare remote icons in their metadata, and GNOME Software needs to download these icons. Turns out, in Flathub, there are a couple of apps that declare such remote icons, but the icons don’t actually exist in the URL they point to! Uh oh. Obviously these apps need to fix their metadata, but we can’t let that make GNOME Software slow to a crawl.

Knowing exactly what was the problem, it wasn’t difficult to come up with potential solutions. We always have to download and cache app icons, but we can’t let that block loading categories, so the simplest solution is to queue all these downloads and continue loading the categories. The difference is quite noticeable when comparing with the current state:

Most of the time, GNOME Software already has access to a local icon, or a previously downloaded and cached icon, so in practice it’s hard to see them downloading.

Final Thoughts

I hope to have convinced you, the reader, that profiling your application is an important step when working on optimization. This article is focused on the timing marks, since they’re the easiest to understand on a quick glance, but these marks have little meaning when not accompanied with the function calls stack. There’s a lot to unpack on the subject, and sometimes this feels more like an art than simply a mechanical evaluation of numbers, but it sure it a fun activity.

Sadly, until very recently, even profiling was still a major pain – although much less painful than guess-based profiling – since you’d need to build at least your app and some of its dependencies with debug symbols to get a meaningful call stack. Most of the profiling I’ve mentioned above required building all dependencies up to GLib with debug symbols.

However, seems like a game changing decision has been made by the Fedora community to finally enable frame pointers on their packages by default. And that means that the setup overhead to perform profiling like the above is brutally reduced, it’s almost trivial even, and I’m hopeful with the prospects of democratizing profiling like this. At the very least, I can say that this is a massive improvement to desktop developers.

Buy Me a Coffee at ko-fi.com

February 02, 2023

Hackweek 2023

Hack Week is the time SUSE employees experiment, innovate & learn interruption-free for a whole week! Across teams or alone, but always without limits.

This year the Hack Week was this week, the last week of January and for my first SUSE hack week I decided to work in something funny, LILS.

Linux Immersive Learning System (LILS)

I don't think that this is a good name, but don't focus on it. The main idea of this project is to create some basic machinery to be able to write "interactive" tutorials or games using the INK language.

This is not an original idea, indeed all I've done is something that's currently working on EndlessOS, and was the main idea behind the dead project Hack Computer, you can even take a look to the Hack app in flathub. But I wanted to work around this, and create something simpler, from scratch.

I wanted to build something simple, with just Python, and make it simple enough to be able to build other tools on top. The design is simple, an INK parser, with a simple game runner. In the INK script you can define commands, to do something special, and wait for events with listeners, to wait for an event in the OS to continue.

With this basic functionality it's possible to build different user interfaces for different environments. And the original idea was to make the commands and listeners something extensible with a simple API, but that's something that I have not done yet, it's all Python functions without extension point.

The code can be found in github.

The INK parser

The most complex part of this project is the INK language parser. The Ink parser is free software and there's a Linux version that you can use to parse and compile to json, but I wanted to create my own parser with Python.

I've spent most of the Hack Week time fighting with the parser and indeed was the most challenging and fun part, because I've not worked a lot with parsers and it's not something easy as pie 😛️.

I remember creating a java compiler long time ago, when I was in the Seville University, for the Language Processors course. We did that with ANTLR, so starting from that, and looking for a Python lib, I found the Lark project. So if you like regular expressions, writing a grammar is a lot more FUN.

At the end I was able to support some basic INK language with support for:

  • Text
  • Tag support
  • Options, with suppress text support
  • Knots, Stitches and Diverts
  • Include other .ink files
  • Variable definition and basic operations
  • Knots and Stitches automatic visiting count variables
  • Conditional options using variables

It still fails in some cases, the comments and TODO placed in between text is not detected correctly and there's a lot of complex stuff that's not supported yet, but with what's supported right now it's possible to create complex scripts with loops and complex game graphs, so it's good enough to build games just with it.

GNOME shell extension

To integrate with the system I've done a simple GNOME shell extension. The extension just shows the text as bubbles and options as buttons, it's really simple and I've no time to make it something ready to be used, but I was able to make something usable.

To be able to run the LILS python library from gjs I've created a simple dbus service that exposes the basic InkScript class functionality as a dbus API.

I was thinking about being able to change the desktop background, depending of the value of a background variable in the script and do something similar to play music and sounds, so it could be a cool game engine with some additions.

SUSE Hack Week

So this Hack week was really fun and I learned a lot. It's really great that SUSE does things like this, letting us work in different projects for a week, to learn, to grow or to just explore different paths.

Blocking free API access to Twitter doesn't stop abuse

In one week from now, Twitter will block free API access. This prevents anyone who has written interesting bot accounts, integrations, or tooling from accessing Twitter without paying for it. A whole number of fascinating accounts will cease functioning, people will no longer be able to use tools that interact with Twitter, and anyone using a free service to do things like find Twitter mutuals who have moved to Mastodon or to cross-post between Twitter and other services will be blocked.

There's a cynical interpretation to this, which is that despite firing 75% of the workforce Twitter is still not profitable and Elon is desperate to not have Twitter go bust and also not to have to tank even more of his Tesla stock to achieve that. But let's go with the less cynical interpretation, which is that API access to Twitter is something that enables bot accounts that make things worse for everyone. Except, well, why would a hostile bot account do that?

To interact with an API you generally need to present some sort of authentication token to the API to prove that you're allowed to access it. It's easy enough to restrict issuance of those tokens to people who pay for the service. But, uh, how do the apps work? They need to be able to communicate with the service to tell it to post tweets, retrieve them, and so on. And the simple answer to that is that they use some hardcoded authentication tokens. And while registering for an API token yourself identifies that you're not using an official client, using the tokens embedded in the clients makes it look like you are. If you want to make it look like you're a human, you're already using tokens ripped out of the official clients.

The Twitter client API keys are widely known. Anyone who's pretending to be a human is using those already and will be unaffected by the shutdown of the free API tier. Services like movetodon.org do get blocked. This isn't an anti-abuse choice. It's one that makes it harder to move to other services. It's one that blocks a bunch of the integrations and accounts that bring value to the platform. It's one that hurts people who follow the rules, without hurting the ones who don't. This isn't an anti-abuse choice, it's about trying to consolidate control of the platform.

comment count unavailable comments

January 29, 2023

GNOME 44 Wallpapers

As we gear up for the release of GNOME 44, let’s take a moment to reflect on the visual design updates.

GNOME 44 Wallpapers

We’ve made big strides in visual consistency with the growing number of apps that have been ported to gtk4 and libadwaita, embracing the modern look. Sam has also given the high-contrast style in GNOME Shell some love, keeping it in line with gtk’s updates last cycle.

The default wallpaper stays true to our brand, but the supplementary set has undergone some bolder changes. From the popular simple shape blends to a nostalgic nod to the past with keypad and pixel art designs, there’s something for everyone. The pixelized icons made their debut in the last release, but this time we focus on GNOME Circle apps, rather than the core apps.

Another exciting development is the continued use of geometry nodes in Blender. Although the tool has a steep learning curve, I’m starting to enjoy my time with it. I gave a talk on geometry nodes and its use for GNOME wallpaper design at the Fedora Creative Freedom Summit. You can watch the stream archive recording here (and part2).

Previously, Previously, Previously

January 27, 2023

Further adventures in Apple PKCS#11 land

After my previous efforts, I wrote up a PKCS#11 module of my own that had no odd restrictions about using non-RSA keys and I tested it. And things looked much better - ssh successfully obtained the key, negotiated with the server to determine that it was present in authorized_keys, and then went to actually do the key verification step. At which point things went wrong - the Sign() method in my PKCS#11 module was never called, and a strange
debug1: identity_sign: sshkey_sign: error in libcrypto
sign_and_send_pubkey: signing failed for ECDSA "testkey": error in libcrypto"

error appeared in the ssh output. Odd. libcrypto was originally part of OpenSSL, but Apple ship the LibreSSL fork. Apple don't include the LibreSSL source in their public source repo, but do include OpenSSH. I grabbed the OpenSSH source and jumped through a whole bunch of hoops to make it build (it uses the macosx.internal SDK, which isn't publicly available, so I had to cobble together a bunch of headers from various places), and also installed upstream LibreSSL with a version number matching what Apple shipped. And everything worked - I logged into the server using a hardware-backed key.

Was the difference in OpenSSH or in LibreSSL? Telling my OpenSSH to use the system libcrypto resulted in the same failure, so it seemed pretty clear this was an issue with the Apple version of the library. The way all this works is that when OpenSSH has a challenge to sign, it calls ECDSA_do_sign(). This then calls ECDSA_do_sign_ex(), which in turn follows a function pointer to the actual signature method. By default this is a software implementation that expects to have the private key available, but you can also register your own callback that will be used instead. The OpenSSH PKCS#11 code does this by calling EC_KEY_set_method(), and as a result calling ECDSA_do_sign() ends up calling back into the PKCS#11 code that then calls into the module that communicates with the hardware and everything works.

Except it doesn't under macOS. Running under a debugger and setting a breakpoint on EC_do_sign(), I saw that we went down a code path with a function called ECDSA_do_sign_new(). This doesn't appear in any of the public source code, so seems to be an Apple-specific patch. I pushed Apple's libcrypto into Ghidra and looked at ECDSA_do_sign() and found something that approximates this:
nid = EC_GROUP_get_curve_name(curve);
if (nid == NID_X9_62_prime256v1) {
  return ECDSA_do_sign_new(dgst,dgst_len,eckey);
}
return ECDSA_do_sign_ex(dgst,dgst_len,NULL,NULL,eckey);
What this means is that if you ask ECDSA_do_sign() to sign something on a Mac, and if the key in question corresponds to the NIST P256 elliptic curve type, it goes down the ECDSA_do_sign_new() path and never calls the registered callback. This is the only key type supported by the Apple Secure Enclave, so I assume it's special-cased to do something with that. Unfortunately the consequence is that it's impossible to use a PKCS#11 module that uses Secure Enclave keys with the shipped version of OpenSSH under macOS. For now I'm working around this with an SSH agent built using Go's agent module, forwarding most requests through to the default session agent but appending hardware-backed keys and implementing signing with them, which is probably what I should have done in the first place.

comment count unavailable comments