a3nm's blog

Modern blockbusters: a dining metaphor

This is just a text I wrote to explain how I felt about most blockbuster movies. I didn't know what to do with it, so here it is.

There's this new restaurant in town that has posters and ads everywhere. Everyone's talking about it, and they all seem to have a pretty strong opinion, so you go with some friends to see what it's like.

The first impression is outstanding. The restaurant is lavishly decorated. The room, furniture, atmosphere, music, are all spectacular and have obviously been painstakingly designed for your enjoyment. The waiters have fancy, colorful, creative dresses, and they usher you to a comfortable seat on a magnificent table adorned with the finest dining ware.

You spend some time admiring the setting: the paintings on the wall, the patterns of the wallpaper, the carefully engineered lighting, and the complex ballet of the waiters. Soon enough, the first dish is served. It consists of various kinds of canapés, neatly arranged on a splendid plate. They look wonderful even if not particularly original. You have a bite, and the taste is good, not exceptional compared to your expectations, but certainly not bad either; just not especially remarkable.

You finish the plate, and a different waiter comes to the table after a while, with a new plate of other kinds of canapés. How formal, you think, how unbelievably fancy to have two rounds of appetizers before the meal has even started! The ingredients are different, but your opinion is essentially the same: excellent visual impression, classical recipes, enjoyable yet somewhat unsurprising taste.

A third plate of canapés comes in, and now you start to suspect that something is off. Why are they only serving such cocktail food? Worse, you can't figure out any logic in the contents of the plates: now some of the bites are sweet, but on the next plate everything is savory again. And the meal continues like this, with a series of plates of different kinds of hors d'oeuvres brought by various waiters.

It's not that the experience is really unpleasant. You can appreciate the setting, the lighting, and the subtle changes in atmosphere and music throughout the evening. You can also wonder about the seemingly random assortment of tastes, plates, and waiters, that comes to the table every now and then. You can also enjoy the food, which is acceptable even if not strikingly good. But after one hour and a half of this, your expectations have been building up to something more. Surely all of this has been leading to a proper dish of some kind? Alas, no: the series of appetizers continues for one more hour, you progressively realize that it's getting too late for your hopes to materialize, and then the check comes and confirms what you had feared. You feel somewhat queasy as you get up and leave the table, like when you have too many snacks in a row: you're no longer hungry, but you don't feel like you had a proper meal either. In fact, it's a bit as if you had been robbed of the opportunity of having one.

As it turns out, your friends are all thrilled about this incredible dinner experience, but it seems that you haven't been paying attention to the same things as them. For one thing, they really enjoyed the beauty of the setting, the music, and how everything was pleasing to the eye and ears. You readily concede that all of this was perfect, but you try to bring the discussion back to the food. "But wasn't the food pretty too", they ask? "Didn't it perfectly match the plates, the table and the decoration of the room?"

Your friends also loved that the restaurant staff was so varied. This is something that you had essentially missed, although you do remember that the plates were brought by many different waiters, with interesting costumes and ties and hairstyles. To your friends, the main point of the various canapés was the story that they were telling about the lives of the waiters and the relationships between them. They can discuss it for ages: "Did you understand why the short bearded guy brought the foie gras plate, although the chunks of duck magret had all been delivered by the tall blonde waitress until then?" "Oh, my interpretation is that the bearded guy has a secret duck side in him, but he's conflicted about his relationship with the bald guy who brought the veal liver."

You ask: "But why the hell did they serve chocolate mousse verrines between the foie gras and veal liver?" Of course, they reply, the reason why the skinny old waitress brought the chocolate mousse was to appease the tension between bearded guy and bald guy. "But what good did it do to the meal", you ask? And they answer: "It brings forward the side of the old waitress's character that feels guilty for the bearded guy's struggle."

You try to explain how you would have liked the meal to have a certain structure, with recognizable dishes arranged in a consistent order. Your friends pounce on this, and question you: why are you so attached to this traditional structure of a formal meal? Why should a good meal necessarily consist of a starter, a main course, and a dessert? "But the point is not the specific structure," you reply, "so much as having any kind of understandable connection between the successive dishes." Some of your friends then ask: "But don't you see how subversive it is to have served an anchovy paste toast just after a chocolate parfait? Don't you like this sort of strong political statement?" You still fail to see the radical appeal of this, given that the setting was otherwise rather consensual, and the food consisted of perfectly standard Western fare. To you, the meal didn't look like a satire of anything in particular, except maybe itself.

They ask, "but didn't you like how the meal was surprising and unpredictable?" And indeed, you have to agree that you couldn't anticipate anything, given that it appeared to be completely random. You explain how the lack of structure makes it impossible for you to summarize, or indeed to remember, the sequence of foods that you had. They disagree: to them, the meal was rich and complex, and anyway the main questions to examine are character-related, e.g., how the blonde waitress's disappearance at the middle of the meal could be linked to the increasingly important role of the bald guy in connection to the sweet and especially fruit-flavored foods.

Your friends are all eager to return to this place when they will start serving their new menu next year. To them, this meal has been building up to the great surprises that the next dinner will surely bring. "Think of all the new kinds of food that we will discover! And in particular I wonder whether we will see the blonde waitress again? I wonder whether she might bring us some scallops in a green plate, because remember that the only seafood so far had been brought by the old waitress, also in a green plate, so this could be some hint of a family relationship between them?" And when you express your lack of enthusiasm, they don't understand you: if you complained so much about the food, why aren't you hungry for more?

Debian on Raspberry Pi 3

— updated

I have a Raspberry Pi 3 and I wanted to install Debian on it. I know about Debian derivatives for the Raspberry Pi, such as Raspbian, but what I don't like about them is that I have to use a special APT repository, and have to trust images generated by these people. I already trust Debian, so why not install Debian on my Raspberry Pi as well?

Debian has a wiki page about the Raspberry Pi 3, but it looked pretty experimental. I tried it out, and I'm happy to report that I got it to work: generating the image myself, booting it up, and using the resulting system.

To generate the image, I just followed the instructions here. I fell into some traps, but @stapelberg just accepted my pull request to document them, so you can just follow the instructions and hopefully they should work.

Once the image is successfully generated in raspi3.img, you can simply write it to the SD card as explained in the instructions. For the last step, if your local network doesn't resolve the rpi3 hostname (mine didn't), you can simply use nmap to find its IP. Of course, don't do this if the administrator of your local network could be worried about a network scan, and adapt it to your IP range:

sudo nmap -p0 192.168.0.1-255

Then you can use the system. What I didn't test:

  • HDMI: there was no HDMI signal (i.e., no video display), I don't know whether this is a known limitation or a bug, and whether the system can be made to use the video. I didn't need it, so I didn't investigate. Testing again, there seems to be a video signal after all: you can see the TTY prompt. However, while booting, there is no information displayed about what happens during the boot process, so you can't hope to debug anything from the display if booting fails.
  • Bluetooth and Wifi: there are comments about it here, but I didn't investigate either.
  • CSI, DSI, GPIO, sound, composite, etc.

What I did test:

  • Booting the system, network, SD card
  • Moving to non-snapshotted repositories for buster, doing apt-get update, apt-get dist-upgrade and rebooting (it still works)
  • USB port: mounting an USB mass storage device (USB key).
  • Reading the CPU temperature: this does not work currently, I get an error when trying to cat /sys/devices/virtual/thermal/thermal_zone0/temp, but it seems that a patch was checked in so it should work eventually.
  • Running stress -c 4 -i 10 -m 2 -d 10 --timeout 300, which worked OK.
  • Running cryptsetup. The results of cryptsetup benchmark are below. They are not great (probably due to the lack of hardware crypto support on the Raspberry Pi?). They probably mean that the CPU will be the bottleneck when reading/writing to an encrypted hard drive.
PBKDF2-sha1        92564 iterations per second for 256-bit key
PBKDF2-sha256     138115 iterations per second for 256-bit key
PBKDF2-sha512      96946 iterations per second for 256-bit key
PBKDF2-ripemd160   75155 iterations per second for 256-bit key
PBKDF2-whirlpool   33505 iterations per second for 256-bit key
#     Algorithm | Key |  Encryption |  Decryption
        aes-cbc   128b    14.7 MiB/s    16.1 MiB/s
    serpent-cbc   128b    12.8 MiB/s    13.6 MiB/s
    twofish-cbc   128b    15.3 MiB/s    16.3 MiB/s
        aes-cbc   256b    11.1 MiB/s    12.3 MiB/s
    serpent-cbc   256b    12.8 MiB/s    13.6 MiB/s
    twofish-cbc   256b    15.2 MiB/s    16.3 MiB/s
        aes-xts   256b    15.7 MiB/s    16.3 MiB/s
    serpent-xts   256b    13.9 MiB/s    14.0 MiB/s
    twofish-xts   256b    16.9 MiB/s    17.1 MiB/s
        aes-xts   512b    11.6 MiB/s    12.4 MiB/s
    serpent-xts   512b    13.9 MiB/s    14.0 MiB/s
    twofish-xts   512b    16.9 MiB/s    17.1 MiB/s

I just noticed that there are some default iptables rules (v4, v6) which prevent remote SSH connections. Hence, if you want to connect to your Raspberry Pi remotely, once you have made sure that it is secure to do so (in particular, changed the default password), you can issue:

sudo iptables -D INPUT 6
sudo ip6tables -D INPUT 4

You should also update /etc/iptables/rules.v4 and /etc/iptables/rules.v6 accordingly (remove the line with REJECT in each file).

Automatic git conflict resolution on logs and sets

TL;DR: In this post, I describe how to configure git to use scripts that automatically resolve conflicts on files where they don't matter: log files, that are chronologically ordered, and set files, where only the set of lines matters and not the order.

I use git to version many things, from papers to code to scripts to configuration. For several of these projects, I am the only user, and I mostly use git to synchronize things across machines. Conflicts then become something of a nuisance; while they can be avoided by always pulling before editing, this is not always possible, e.g., when I'm offline, or forget to do this. However, there are files on which conflicts do not matter, and are easy to solve:

  • One example are log files, i.e., files that log timestamped events, one per line. For such files, we can reconcile conflicts by merging events, intuitively sorting the lines by timestamp and deleting all conflict markers. I use this for a log of personal notes, but the same should work if you want to version, e.g., your bash history.
  • Another example are set files, i.e., files that describe a set, each line being an item, and with irrelevant order between the lines (and no duplicates). One example are vim spellchecking additions, when synchronizing them across machines. For such files, intuitively, we can solve conflicts by discarding duplicate lines and dropping conflicts markers.

I used to solve these conflicts by hand, but this was tedious and error-prone. I then realized that I could use custom merge drivers with git to automate this away. I have been using the setup for some months now without issues.

Set files

Let me start with this case, because it is simpler, and let me present things top-down. We will first add a file .gitattributes to our repository to indicate that a custom merge strategy should be used for some files. For instance:

cd myreporoot/
cat > .gitattributes <<EOF
mysetfile1.txt merge=set
mysetfile2.txt merge=set
EOF

Of course, you should then version this file with git:

git add .gitattributes
git commit -m 'automatic merges' .gitattributes

Now, we have to tell git what we mean by the set merge strategy. This is explained in the section "Defining a custom merge driver" in the gitattributes documentation or the manpage gitattributes(5), but I will summarize it here. Edit your .gitconfig file to register the set merge strategy:

cat >> ~/.gitconfig <<EOF
[merge "set"]
  name = set merger
  driver = ~/bin/git-merge-set %O %A %B %L
EOF

Now we have to create the ~/bin/git-merge-set script. This is fairly easy to do, once you have understood the meaning of the arguments that git passes to the program. Here is for instance my git-merge-set script, which concatenates the files and deletes duplicates (in a stable way, i.e., it preserves the order in the input files). Note that it depends on sponge from moreutils.

You should now be able to commit in your repository, pull conflicting changes as you like, and never hear about the conflicts on the files for which you have defined custom strategies. When pulling, git will just tell you that it is merging the changes, and everything will work fine.

Log files

For log files, it works exactly the same way, replacing "set" by "log" in all steps above, except that you have to define a different merge strategy. The script to write depends on the format of your log entries. Mine have a numerical timestamp, a space, and the line contents, and I use this git-merge-log script. Feel free to adapt it.

My applications to computer science research positions in France

I have applied to computer science research and teaching positions in France over the year 2015-2016, and obtained my current associate professor position in Télécom ParisTech. These application processes are public competitive exams, but there is not so much information available about them online, and many of my friends who are applying for 2016-2017 have asked me for a copy of my application files.

I thought that it would be better for this information to be available online, so that it may benefit all candidates equitably. So I have written a description of the applications I did last year, along with all the documentation I have submitted. I am making this a separate page, because I will probably need to update it, even though I will not necessarily make the effort to keep it completely up to date as these contests evolve.

Here is the page: Application to research positions.

Setting up SVN and git shared hosting on Debian

This note explains how to setup a shared Git and a shared SVN repository on a Debian stable (jessie) server, incidentally a Raspberry Pi. Both repositories are served with Apache, using mod_dav_svn for SVN, and using the git-http-backend CGI for git. (Note that these aren't the only possible choices, but they are the ones I will present.)

The setup uses Apache Digest authentication (with a file common to both SVN and git): the SVN and git setups are otherwise independent. The setup uses SSL provided with Let's Encrypt, set up with certbot. For brevity I am omitting sudo in all commands of this tutorial.

Note that I am dumping this from memory after having done the setup, but I'm not guaranteeing that everything will work. If something does not work, please email me.

Set up the domain names

If your main domain is example.com, then you probably want to set up svn.example.com and git.example.com to point to the machine where you will be hosting the repositories. This will allow you to configure each service easily using virtual hosts, and makes it easier to move the hosting to a different machine if you want.

You probably want to wait for the new DNS entries to have propagated before doing what follows, in particular for the Let's Encrypt challenge to work.

Install apache2 and certbot

apt-get install apache2 libapache2-svn libapache2-mod-svn
apt-get install certbot python-certbot-apache

Side note: certbot is not yet available on jessie, you need to enable jessie-backports. As my Raspberry Pi uses Raspbian, this also means I had to set up the Debian signing keys: download the keys that apt-get update complains about and install them with apt-key add.

Obviously you should also make sure that SVN and git are installed on the server.

Start your apache2 configuration

Add to the beginning of /etc/apache2/apache2.conf the default hostname of your machine (of course, change it to something reasonable):

ServerName machine.example.com

While you are at it, you can add the following to the bottom to avoid disclosing details of your Apache version:

ServerTokens Prod
ServerSignature Off

Now, create a file /etc/apache2/sites-available/000-git.conf and add the following, changing the domain name and email address (don't worry about the DocumentRoot, it doesn't matter):

<VirtualHost *:80>
        ServerName git.example.com

        ServerAdmin you@example.com
        DocumentRoot /var/www/html/

        ErrorLog ${APACHE_LOG_DIR}/error.log
        CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>

Create /etc/apache2/sites-available/000-svn.conf analogously (replacing git by svn). Then, enable the right sites with:

a2dissite 000-default
a2ensite 000-git
a2ensite 000-svn

Set up SSL with certbot

Run certbot --apache: it should pick up the configuration files that you created and understand what you are doing. Say that you want to enable encryption for both subdomains, that you accept the license, and that you want the Secure setup where users are redirected to the SSL version (we don't want the version control traffic ever to happen in cleartext). Check that https://git.example.com/ and https://svn.example.com/ both work with SSL, and that http:// URLs redirect to the https:// variant. Yes, it's just a 301 redirect, it's not HSTS, but to me this is good enough.

Create the user file

Authentication will be performed by Apache Digest, so create a password file with your own user using the following command (note that the second argument, the realm, is important because it is included in the digest):

htdigest -c /etc/vcs_passwd "example.com version control" yourlogin

Then, when you want to add more users, issue:

htdigest /etc/vcs_passwd "example.com version control" theirlogin

Ensure that the file is owned by www-data and only readable by them:

chown www-data:www-data /etc/vcs_passwd
chmod og-rwx /etc/vcs_passwd

Setting up Subversion

Now that the common SSL setup and user setup is done, let's start by setting up Subversion. For reference, the part of the SVN Book about what we are trying to do is here. I prefer creating one global repository and use path-based authorization to give access to relevant subsets to various users.

Creating the Subversion repository and setting up authorization

Create the repository, e.g.:

mkdir -p /var/svn
svnadmin create /var/svn/repos

We will be using path-based authorization. Create a file /var/svn/paths with a content like the following (check the path-based authorization documentation for details). Caution, it appears that path specifications (between square brackets) do not work if you put a trailing slash, so do not put a trailing slash, as in the following:

[/]
yourlogin = rw
close_friend = r

[/project]
close_friend = rw
other_friend = rw

Give ownership of everything to Apache with:

chown -R www-data:www-data /var/svn

Configuring Apache

First, you should enable the necessary modules:

a2enmod dav_svn authz_svn auth_digest

Next, restart Apache with service apache2 restart.

Edit the file /etc/apache2/sites-enabled/000-svn-le-ssl.conf which was created by certbot, and add the following block in the VirtualHost block:

<Location />
        DAV svn
        SVNPath /var/svn/repos
        AuthType digest
        AuthName "example.com version control"
        AuthUserFile /etc/vcs_passwd
        AuthzSVNAccessFile /var/svn/paths
        Require valid-user
        SSLRequireSSL
</Location>

Reload the Apache configuration with service apache2 reload. Hopefully the config should be accepted, and you should be able to checkout on a remote machine:

svn co https://svn.example.com/ myrepos

You should then be able to commit, etc.

This guide does not cover the task of setting up a backup (e.g., a periodic rsync) of the repository, i.e., the /var/svn folder.

Setting up git

We will be using git-http-backend, and create multiple repositories with one user group per repository.

For now, create the structure:

mkdir -p /var/git/repos
touch /var/git/groups

You should then make sure that this is readable by www-data.

Creating each git repository and setting up access rights

To create a git repository for project proj1, do:

cd /var/git/repos
git init --bare proj1
chown -R www-data:www-data .

Now, add to the file /var/git/groups a line to indicate who has the right to access this project (this is both read and write access):

proj1: yourlogin close_friend other_friend

Configuring Apache (initial configuration)

This only needs to be done the first time.

First, you should enable the necessary modules:

a2enmod macro authz_groupfile cgi auth_digest

And restart Apache.

Next, edit the file /etc/apache2/sites-enabled/000-git-le-ssl.conf and add, inside the VirtualHost block:

DocumentRoot /var/git/repos/
SetEnv GIT_PROJECT_ROOT /var/git/repos/
SetEnv GIT_HTTP_EXPORT_ALL
ScriptAlias / /usr/lib/git-core/git-http-backend/

<Macro Project $repos>
        <Location /$repos>
                AuthType digest
                AuthName "example.com version control"
                AuthUserFile /etc/vcs_passwd
                AuthGroupFile /var/git/groups
                Require group $repos
                SSLRequireSSL
        </Location>
</Macro>

UndefMacro Project

Configuring Apache (for each project)

Now, each time you add a project, say proj1, you should add the following line just before the UndefMacro line:

Use Project proj1

And of course you should issue service apache2 reload. You should now be able to do the following:

git clone https://git.example.com/proj1

To avoid getting asked about your password each time (and storing it in plaintext, which is OK if you trust the security of your machine and you use disk encryption), you can issue the following (see the end of this answer):

git config --global credential.helper store

As with Subversion, you should then arrange to back up the /var/git folder.