This post is about how (and why) I recently produced my own Debian packages for a recent upstream release of Mesa to work around a bug on my own machine. Though the specifics are not specifically interesting, you may extrapolate my story to a more general guide of what to do when you need to run an upstream software release that's too recent to be packaged by Debian, and too complex to be easy to install by hand with make install.

I recently tried to figure out the source of weird crashes on my machine, which runs Debian testing. I understood from /var/log/syslog that I was probably affected by Bug #771045. From the linked discussions it seemed that a patch in Mesa had a good chance of fixing the problem. The patch was part of version 10.3.4 of Mesa, but Debian testing only had 10.3.2-1, judging from the output of:

dpkg-query -l | grep -i mesa

So, if I wanted the patch, I needed to install manually a newer version of Mesa.

A first thing to check was whether it was not packaged in Debian unstable. I have the unstable repository set up and pinned at a lower priority with the following in /etc/apt/preferences.d/prefs:

Package: *
Pin: release a=testing
Pin-Priority: 650

Package: *
Pin: release a=unstable
Pin-Priority: 600

I use this to manually install a recent version of iceweasel from unstable. Could it be the case that a recent Mesa was packaged in unstable? Alas, no, checking on a random mesa package among those at version 10.3.2-1:

apt-cache policy libegl1-mesa

libegl1-mesa:   
  Installed: 10.3.2-1
  Candidate: 10.3.2-1
  Version table:
 *** 10.3.2-1 0
        650 http://debian.proxad.net/debian/ testing/main amd64 Packages
        600 http://cdn.debian.net/debian/ unstable/main amd64 Packages
        100 /var/lib/dpkg/status

So here is how I manually compiled a more recent version. I was surprised to see how simple it turned out to be, in fact: precious little manual intervention is required.

Of course, a word of warning applies: make sure you will know how to fix your system if it gets broken by the home-made packages that you installed or by anything else you did.

I'm assuming that your machine's architecture is amd64, otherwise what I say later about compiling for i386 may make no sense.

You may need to start by installing some packages such as build-essential if you don't have them. Then, issue:

sudo apt-get build-dep libegl1-mesa

This will install¹ the packages needed to compile Mesa.

Then, go to a fresh folder and download the sources for Mesa:

mkdir -p ~/mesa
cd ~/mesa
apt-get source libegl1-mesa

The source for the packaged version (10.3.2-1) should be downloaded. You could then go in the source tree and run uscan from the devscripts package:

cd mesa-10.3.2
uscan

This would automatically search for a newer release on the Mesa website and download it. However, it would pick the latest version (10.4.1 as of this writing) for which the packaging wouldn't work as is. So instead, download directly Mesa 10.3.4, the version that we want, from the official website, and symlink it to the conventions used by the Debian tools.

cd ~/mesa
wget ftp://ftp.freedesktop.org/pub/mesa/older-versions/10.x/10.3.4/MesaLib-10.3.4.tar.gz
ln -s MesaLib-10.3.4.tar.gz mesa_10.3.4.orig.tar.gz

Now we can go back in the source tree for the old version, and use uupdate to unpack the new version and migrate the Debian-specific stuff. Then go to the new source tree thus created:

cd mesa-10.3.2
uupdate ../mesa_10.3.4.orig.tar.gz
cd ../mesa-10.3.4

Now we are ready to start the compilation process:

debuild -us -uc

This should compile everything and take 10-30 minutes. If a problem occurs, you should have copious output to try to understand what went wrong. Otherwise, if all goes well, all the deb packages for the new version will have been generated in the parent folder. That was easy!² :)

That's all nice and well, but then I realized that I needed the i386 version of the packages, because I use Skype, whose "multiarch" package is actually i386. How to cross-compile the packages? In fact, it's quite easy. We will use pbuilder to create a chroot where the compilation can take place, but everything is automatic:

sudo apt-get install pbuilder
sudo pbuilder --create --architecture i386

Now go in the source tree and issue³:

sudo pbuilder --build ../mesa_10.3.4-1.dsc

This should take care of everything. The packages will be produced in /var/cache/pbuilder/result.

Now you just have to figure out which packages you need to install (using dpkg-query -l) and install them with sudo dpkg -i, then correct any dependency problems with sudo apt-get -f install. Alternatively, you can just issue sudo debi -u from the source tree, which should do those steps automatically. However, I haven't tested it. Of course to install the i386 packages in this way you will need to move them, or let debi know where to find them.

Note that to avoid problems it looks like it is safer to install all packages at once. If you run into file conflicts between the i386 and amd64 packages, a solution may be to apt-get remove all the old versions and then install the new ones. Of course, this will probably remove a bunch of useful packages on your system that depend on Mesa; if you do this should make sure that you have kept a trace of the packages that you will want to reinstall afterwards, and also that you are confident using a tty or ssh if your graphical environment breaks.

And, well, that's it. If you have installed the packages successfully, you can just reboot. Then you can check in dpkg-query that you are indeed using the newer version of the packages, and double-check using:

glxinfo | grep -i version

Of course, if something went bad, maybe your system will not boot (or at least not boot graphically) and then it will be up to you to investigate. Hopefully removing the new packages and installing back the Debian ones should make things work again. :)

One last thing: hopefully, as the version numbers of your home-made Mesa packages correspond to the actual version number of the release, apt-get upgrade should move you to the more recent versions as soon as they start getting packaged by Debian.

I am a fond user of Wikipedia, so I was interested to find out how to use it when I have no Internet connection. On my mobile phone, I already use the free and open-source Aard Dictionary, but for space reasons I only have the French Wikipedia, without images, and with some formatting glitches. On my computers, where I have more space and where it is easier to set things up, I wanted to have more, so that I can have a usable Wikipedia when I'm on the train, in a plane, or in a place with a crappy Internet connection.

This blog post explains how to set up a local Wikipedia mirror using Kiwix. Start by downloading Kiwix and unpacking it. Kiwix ships with a special GUI to browse Wikipedia offline, but I prefer to use my usual Web browser. Fortunately, Kiwix also includes the kiwix-serve program that can serve dumps as a regular Web server.

Next, download the dumps for the projects that you want. Ideally I would be interested in generating my own dumps, but I haven't looked into this yet. I used Wikipedia and Wiktionary French and English, totalling to about 60 GB.

Next, to be able to perform full-text search, you must index the dumps. Maybe the pre-indexed dumps can be used, I haven't tried. I indexed them manually instead. For each file a.zim, I did kiwix-index -v a.zim a.zim.idx, where kiwix-index is provided by Kiwix. The process takes a lot of time (10-30 hours or so) but does not require any interaction. The indexes take another 30 GB.

To serve all dumps with kiwix-serve, you need to build a library. First, move the .zim and the .zim.idx files (or symlink them) to have shorter names; this will make the URLs shorter afterwards. I use wen.zim, wfr.zim, wtfr.zim and wten.zim. Now run, for each file a.zim in the working directory: kiwix-manage `pwd`/wiki.xml add `pwd`/a.zim --indexPath=`pwd`/a.zim.idx, where `pwd`/wiki.xml is the name of the library file that will be created. It is safer to use absolute paths throughout.

This should have created the library file wiki.xml. Now, to start the server, choose a port number (say 4242) and run kiwix-serve --port=4242 --library /where/you/put/wiki.xml. Test it by browsing to http://localhost:4242 and checking that it works. If it does, you probably want to arrange for this command to be run at startup.

Please note that Kiwix will also be available from other machines, not just localhost. I couldn't find a way to change this behavior. For now, I use iptables to filter incoming connections from other hosts:

sudo iptables -A INPUT -p tcp --dport 4242 -s 127.0.0.0/8 -j ACCEPT
sudo iptables -A INPUT -p tcp --dport 4242 -j REJECT
sudo ip6tables -A INPUT -p tcp --dport 4242 -s ::1 -j ACCEPT
sudo ip6tables -A INPUT -p tcp --dport 4242 -j REJECT
sudo iptables-save | sudo tee /etc/iptables/rules.v4
sudo ip6tables-save | sudo tee /etc/iptables/rules.v6

The last step, if you use Firefox, is to use Smart Keywords to be able to reach your local dump efficiently. To do this, for every of the dumps that you have on localhost:4242, right-click on the search text field and add a keyword for it. As I use Firefox Sync, those bookmarks are synchronized across my different machines.

These smart keywords work for full-text search in the dumps. If you want to have bookmarks that directly reach articles and will never perform full-text search (because it is faster), you can edit the bookmarks in Firefox to set "Location" to, e.g., "http://localhost:23552/wen/A/%s.html". However, this technique does not work with all dumps (it depends on the URL structure), and in this case you must remember that the first letter of the search term must be uppercase.

In terms of formatting, the Kiwix dumps are fairly OK. There are occasional glitches (e.g., with <span>) but not many of them, and certainly a lot less than with Aard Dictionary. Equations are supported, and images are there if you pick the right dumps. Most templates, formatting, etc., is fine. The HTML interface added by kiwix-serve is not perfect but it's mostly unobtrusive.

I have recently migrated my passwords to pass and so far I've been really happy about it. The previous system was to have them all in a huge text file, which wasn't especially convenient or secure¹, and wasn't shared between my various machines. Here is some info about pass.

pass has been packaged for Debian since Jessie, so installing it is as simple as sudo apt-get install pass. However, it's just a shell script just over 600 lines, so really easy to review, and install manually if you need to.

The way pass manages passwords is dead simple: a hierarchy of gpg-encrypted files. The assumption is that each file corresponds to a website, or machine, or other authentication realm, and contains the password. The use of gpg provides a layer of security, so that your gpg key and passphrase serve as a master password. Of course, it is nice to have a properly configured gpg-agent(1) to avoid having to enter the passphrase multiple times.

The basic commands of pass are pass init KEYID which sets up the store for gpg key KEYID (by default in ~/.password-store), pass FILE which decrypts and shows FILE, and pass edit FILE, which decrypts FILE to a secure temporary location in /dev/shm, edits it, and encrypts it back. You can also use pass ls (which shows a nice output using tree), pass find to search for files using find, pass grep to search in the decrypted password files using grep, and pass rm, pass mv, pass cp. Of course, you can also mess around in the password store by hand.

As pass has this very nice CLI interface, migrating my passwords from my custom system was very easy, although it seems like the Debian package also installs a bunch of scripts to migrate from other password managers.

Beyond the generic commands I have presented, pass obviously offers commands tailored for password management. You have pass insert FILE which creates FILE with the password you provide (and turns off echo and makes you enter it twice for confirmation). You have pass -c FILE which copies the password in FILE to the clipboard, so you can input the password where you need it, and automatically clears it after 45 seconds (which is a reasonable thing to do). You have pass generate FILE LENGTH which generates a password of LENGTH chars in FILE and displays it (or copies it to the clipboard with -c); what is very nice is that pass itself does not include password generation logic, but entrusts pwgen(1) with the task.

Icing on the cake: pass is designed to be used with git, and provides pass git to call git commands. If you use git, all the pass commands will automatically git commit what's needed. This makes it very easy to share passwords between different machines. Of course, as the files are encrypted, git cannot be expected to solve conflicts within files, but it can nicely merge changes across various files. You can also use this setup to share passwords between different people, as pass supports encrypting for multiple keys.

For once, I find it hard to find something to dislike about pass. Eventually I may want to tweak password generation so that it generates passwords the way I'm used to, but this would be easy to do. I'm also missing support for usernames, as I use different usernames on different websites, but pass allows you to store anything in the password file (and only the first line is taken into account for pass -c and others), so I can just add the username as the second line if needed, it's just that I will have to retrieve it by hand, or script something that does what I want. Other than that, I'm very happy to have a convenient, lightweight, and secure way to manage my passwords and share them across machines using git.

This post is about how I manage the apt packages that are installed on my Debian systems.

Left to myself, I tend to apt-get install various things every now and then: to try them out, or because I temporarily need them. In most cases it turns out I never really use them, but of course I will never remember to clean them up. Eventually my / partition fills up and I have to waste time tracking down which packages are useless. (Of course, accumulating useless packages is also a bad idea in terms of security, performance, etc.)

If I want to back up my current selection of packages, I could dump the output of apt-mark showmanual somewhere, but that's not really satisfactory; the list of packages that I use should be stored as a first-class citizen in my config, not just obtained from the current system state.

If I need to set up a new machine, I can install all the packages that were installed on the previous one, but this will end up downloading gigs of packages including lots that I don't really care about. Of course, as soon as I start using several different machines, it is necessary to install on all machines the new packages that I need, so the installed packages must be kept in sync somehow. Otherwise, I have to waste time watching apt-get installing stuff every time I run a command on a host and realize that a package is missing. (However, remember that most installed packages are not really important and probably don't need to be synchronized at all.) All of this is made worse by the fact that not all machines are equal (I don't want to install graphical stuff on servers, laptop tools should only go on laptops, etc.).

My current answer to this is to maintain in my public configuration repository a bunch of program lists for various kinds of uses. This list is synchronized between machines using git, as is the rest of my configuration. It contains only the packages that I really have some use for, rather than all the random crap I have ever installed and don't remember the point of¹. (This being said, I don't care about it being minimalistic, and I'm OK with including large tools that I use rarely, as long as there are reasonable odds I will use them again.)

I have a private file that lists, for my various hosts, which program selections it needs from that list, for instance:

my-laptop-1 minimal laptop server util
my-server minimal server

New packages that I install don't go to this list by default, but my crontab on each host mails me every month the diff between the currently manually selected packages on that host and the ones which should be installed according to the host's list. Here is the script: check-packages.sh.

From this monthly report, I can then update the list by removing the new installed packages that I don't need after all, putting the ones that I do need in the right selection, and installing the ones which should be installed. (And, of course, postponing the ones I haven't yet made up my mind about.)

The system is of course fairly rudimentary: there is no dependency between package selections, the dependencies of every piece of software that I compile myself have to be listed in a separate file, the sync process is manual, etc. Yet I am happy that I have, at last, one central list of the packages that I consider useful to have on my systems; as a bonus I can even share it with the world.

This blog used to be generated by Fugitive, a static blog engine¹ backed by git. Fugitive is very neat and minimalistic, but it is also written in pure sh, and its design makes it spawn a lot of processes, which makes it unacceptably slow for me.

Rather than rewriting a similar engine with a different design, I chose to migrate to Pelican, a more common static blog engine in Python. I hoped that this would be simpler than rolling my own. It still took several hours to set everything up correctly, rewrite the templates, understand the way it works, etc. However, everything should be fine now, and very little should have changed relative to the fugitive version. I hope nothing subtle broke in the process.

In terms of performance, pelican takes about 2 seconds to recompile the entire blog, as opposed to around 30 seconds for fugitive to recompile one page (most of which is spent compiling the archives page), and far more to recompile all pages when templates have changed. (This being on fairly recent machines.) It also looks like pelican's processing time comes from the constant overhead of loading its own files, rather than from the blog size; I guess time will tell. The new setup also separates neatly the git versioning of the blog source from the tools used to generate it, rather than relying on fugitive's hooks which I always found a bit too brittle for my liking.

As an added benefit, I have set up some elaborate things which I had never taken the time to configure with fugitive: using markdown to write blogposts, having syntax highlighting in code snippets with Pygments, math rendering with MathJax, footnotes, etc. Categories, tags and languages should also be easy to set up if needed.

Pelican does have a few downsides: it is a bit complex to understand and heavy, it requires separate plugins for various things, which you also have to install. Most of its complexity is because it is too generic for my needs, and its config tries to abstract away things which I'm OK with hardcoding (e.g., in the templates). Also, pelican does not interact with git at all, and it is a bit silly that article creation and modification times must be recorded as metadata in the articles whereas they are available from git. I intend to rely on simple scripts to save me from having to add this kind of info in files by hand.