poking at packaging

After hearing about the exciting new poke project a few months back, I got deep into the weeds of packaging again. At the time of writing the authors of the poke project have just tagged poke 0.90 as a pre-release of the tool.

For the past few months I’ve wanted to work with poke, but it has some extremely recent dependencies that have made building and using it difficult for me. I don’t really want to just dump newer versions of binaries into my filesystem and hope everything keeps working (though it might be okay to do so), but I still wanted to experiment with the Poke language and see what it is. Briefly though, poke is hot shit for decomposing and working with binary structured data files. I’m going to leave it at that, and say if you’ve ever been frustrated modifying any binary file manually with some sort of hex editor and some manually hacked C or Python, you should take a look at the magic of poke. The Applied Pokology blog is a good start, as are some of the cool talks (at Kernel Recipes 2019 and GNU Tools Cauldron 2019) the project’s lead author has given of late.

Why Snaps?

Canonical’s snaps have interested me precisely because I hoped they would let me sidestep the biggest pain points I have with standard packaging. I hate the magic build scripts that make up the debian packaging system, and I’m routinely confounded by attempts to dig out the documentation as to what they do and why they fail to reproduce packages that are written by others.¹

Secondly, I wanted a way to isolate my build system and the runtime requirements from my host system. Having tried and failed to build legacy versions of Corebird back when it required newer versions of GTK than those that shipped with Ubuntu, I don’t want to maintain parallel build trees and library paths ever again.²

Snaps advertize themselves as a solution to that problem. The tutorial suggests the build environment is fully contained, and the build commands are automated provided you don’t need any dramatic deviations from the typical “build from source” commands (e.g. ./configure, make, sudo make install). More to the point, a few Canonical employee’s have been preaching the glory of snaps for simple little projects on a few podcasts I picked up during COVID.

So here I am, I’ve got a fancy new project, that requires some fancy build deps, and hopefully another reason to look at snaps and see if they can let me use the tool.

Results and Observations

Simply put, I nearly got it. The poke snap has rudimentary functionality, but getting here was far from painless.

My biggest gripe about trying to learn how to build a deb is the magic build tools scripts that convert the magic debian metadata directory into the archived bundle. Snaps are somewhat better in this regard, but you wouldn’t realize that unless you wandered off to the Multipass web site to learn about the multipass tool. Multipass is a qemu VM manager focused on Ubuntu images. When you call snapcraft on a project, mutlipass builds a VM behind the scenes and runs through a series of build phases as specified by your snapcraft.yaml file.

Without knowing this the process of trying to build a snap is one of confounding trial and error when addressing anything remotely nonstandard in the build process. Worse still, until very recently every trial and error attempt would start back over from scratch. If you found you had some minor flag problem in the last phase of the build, you could spend hours rebuilding the entire project and pulling down source trees from the internet. Thankfully this is no longer the case with the core20 base image. All that is to say, major pain points in the build process existed in the past, but are much better today.

Recommendations for would-be snapcrafters

Dig into the build process details.
- Snaps are built of parts, and the parts are each built in isolated directories inside of the VM performing the build.
Wade through documentation like Wikipedia
- The documentation on snapcraft.io totally sidesteps the complexities of the build tree, and hides a few magic nuggets (like being able to call snapcraft --shell to inspect a snap build process) in seemly random places. The best solution to this is unfortunately to pour through links, and never close a tab you came from.
Inspect the github
- It’s here. All of the pain of not following the staging can be (slowly) dissected from the various build tools. It’s a far cry from formal documentation, but the various tooling scripts spit out bash commands at the bottom layer. Everything you battle in the build process usually comes down to one or two python scripts in the snapcraft repo.
- As an example, recently the autotools plugin gained support for ./bootstrap commands. It does not support the build flags I needed for poke, but I now know exactly what modifications are expected.
Standard Maintainer build rules apply.
- For example, if you’re building a library, install it into /usr rather than /usr/local. The later will build the tool but not let you use it as an internal dependency for other parts of the build. Both the bison and gettext dependancies in my poke snap are simple examples.
Confinement is painful.
- Most of my time in finishing the poke snap has been fighting confinement. I want all of the normal $HOME/.<tool>rc and $HOME/.config/<tool> resources to be exposed to the snap. That linkage is broken by default. Confined processes internally see $HOME as a common user location and files must be manually linked.

Final Thoughts

I found myself constantly asking basic questions about the build process: “Where does the multipass configuration live? Why do the various VMs appear to stomp on each other in the build process? Can I give a VM extra RAM or CPU horsepower? Can I reconfigure a VM post spin-up? Can I rename a VM?” Some of these I still don’t have a clue about. These paper cuts will continue to impede adoption of snaps even if they’re a great solution to a common set of problems. That said, snaps are quite close to being ready for primetime. The documentation leaves a lot to be desired, but the paper cuts these days are just that, paper cuts. They aren’t debilitating wounds.

Having never tried to bundle a Flatpak or Appimage I don’t have full insight, but I don’t think these particular pain points (or anything else I’ve mentioned) are unique to snaps. Flatpack has confinement problems like snaps, it has a confounded build system, and I suspect it has similarly sparse documentation.

I’m left in resignation that packaging will always suck to learn, but on the upside, soon a really cool binary hacking tool will be accessible to users of likely quite old Ubuntu releases.

SOAPBOX: has anyone who isn’t a kernel maintainer ever tried to rebuild the kernel package? How many build failures did you go through before you finished? Do you have any faith in the final output being “correct”? ↩
See Cawbird for the project fork since Corebird’s demise. ↩