a3nm's blog

Shortcomings of the real world

— updated

Here is a list of fundamental differences between reality and idealized models of the world. It can provide guidelines when designing virtual worlds, or serve as a checklist when trying to reason about the real world:

Irreversibility.
Some things are more easily done than undone (building something versus destroying it, cleaning something up rather than making it dirty, etc.), and some cannot be undone at all (killing people, losing information, wasting resources, etc.). This means that a small number of wrongdoers can have a disproportionate impact because undoing their mess takes so much time, and this implies that preventive measures are needed to limit the occurrence of irreversible bad things. This is in contrast to virtual places like Wikipedia where reverting edits isn't substantially harder than making them, and where you can benefit from the fact that vandals are a small minority.
Low dimensionality.
The world has a small number of spatial dimensions: only two are really usable, the third one is harder to use because of gravity. Because of this, the possibility of interaction is limited: you cannot have a high number of things acceptably close to each other. This holds both for groups of people (large groups of people cannot interact meaningfully in real life, which is an obstacle to large-scale collaboration) and for cities (to have everything close to everything, you need absurdly high density).
Imperfect coordination.
Even with arbitrarily good communication technology, large groups of people are harder to coordinate than small groups, because of cognitive limits. For this reason, whenever two groups have contrary interests and must hold one against the other, the larger group will be disadvantaged and have much higher risk of defection. This is a factor explaining why the masses have a hard time coordinating, even though they are numerous by definition.
Non-autonomy of children.
While the harm principle dictates that consenting adults in isolation can be simplified out of the moral equation, this does not work with children: adults in isolation can have children, and those children will not be able to legally consent to everything their parents might do to them. For this reason, society has to keep an eye on how parents raise their children, and find some compromise between the parents' rights and the child's.
Necessary infrastructure.
Long-distance communication is not a given but depends on artificial infrastructure which is not free, can fail or can be controlled by malicious parties. You cannot assume that everyone has access to the Internet in the same way that anyone has access to air.
Unbounded vital needs.
If the vital needs of people could be bounded, there would be some hope of managing to satisfy the needs of everyone and assuming that the survival of every human being is ensured. Sadly, people can have arbitrarily complex health problems and could need arbitrarily involved and expensive treatment. This can be dealt with through an insurance system, but complicates things because some people who have simple needs will want to opt out of such a system, making it unsustainable.
Critical mind.
To achieve the independent thought and critical spirit required to be a free, autonomous agent, education is required. People who are not given this education cannot be considered as individuals and it might not make sense to consider that they are responsible for their actions. Yet, they need to be dealt with in some way or other.
Inheritance.
Assume that money represents some measure of social utility, and that people who earned money should be allowed to use it as they like. In this setting, it is a major problem that most people will want to give their money to their offspring, because the money that the offspring will thus inherit is not linked to their social value. The problem is that the individual interests of the donor ("benefit my offspring") are at odds with the interests of society ("allocate money to people who produce value"). There are no solutions except restricting the freedom of people to use their money or increasing inequality at birth because of the parents' wealth.
Physical encounters.
It is not possible to assume that people live autonomously in isolation from each other and only communicate by exchanging of information. People desire friendships, close relationships, and physical relationships. For this reason, they have to meet in real life.
Public-private continuum.
You cannot divide the world in public places and private places and say that there should be no expectation of privacy in public places, because you need to go through public places to travel from one private place to another. Besides, private conversations will often take place in public space with some expectation of privacy between the speakers. I tried to think more about this point.
Repetitive work.
In reality, repetitive tasks have to be carried out. If you want to do the same things multiple times, you will have to do so, and it will usually be complicated to build a robot to perform the task for you. This is in contrast to the virtual universe where things are usually much easier to formalize and automatize, and where the effort required by a task is much closer to its Kolmogorov complexity.
No records.
Even if there is no expectation of privacy somewhere, there is usually no complete perpetual record of what took place there. Hence, there cannot always be an objective assessment of the truth of factual statements involving public data. Note that the problem is not that records are not reliable and can be tampered with, but the fact that they are not complete or numerous enough: the higher the number of independent records, the harder it gets to engineer consistent fabrications. This is in contrast to virtual space where there is usually abundant evidence available because recording something is often easier than not recording it.

[I just wrote this list quickly to dump some ideas I had in the back of my head, it might not make much sense.]

Recording all your terminal sessions

— updated

I love to log as much information as I can about what I do on my computer. (Of course, I never send those logs to third-party services.) I log all of my keystrokes, I religiously keep all of my command history, all of my email and IRC logs, and so on.

However, something that I didn't log so far is what appears in my terminals. This was a shame: since terminals display text, you would expect that you could log everything which appears on them without using up so much space after all. Logging this information could be useful to reconstitute what you were doing at a particular point in time, to understand how you ended up making a certain mistake or doing a certain thing, to show to someone how to do something, to recover the output of any particular command of your history, and so on.

There is a tool called ttyrec which can be used to log what happens in your terminals (including timing information), but I didn't use it systematically so far because of one simple issue: if you run cat large_file, then ttyrec will happily put all the content of large_file in its log, even though you probably didn't care about it. Just a few accidents like this and your log files can become huge.

The point of this post is to advertise ttyrex, a slight modification of ttyrec which adds an option to cap the quantity of data logged every second. The point of this is that when doing cat large_file, you can just log a small quantity of the file every second and skip the rest, and you will get a reasonable approximation of what you saw on the terminal without using up too much space.

I have been starting ttyrex systematically with urxvt for some time now, compressing logs that are older than two weeks (this saves a tremendous amount of space), and the last two week's worth of logs use up a quantity of disk space which I think is reasonable by today's standards (less than 1 GB). I have also tweaked zsh to store the start time and stop time of the recorded sessions, the start time of ongoing sessions, and the command history of each session: I have then written a command to replay what happened at any point in time (i.e., open one replay terminal for each terminal that was open at some timestamp, and jump at the correct position in each of the replays), and a command to take a line of the command history and open a replay of the right session at the right time to see when the command was entered and which results it gave.

I haven't found any use for all of this yet except playing around, but it's pretty fun (having terminals which replay what I did in the past feels a lot like time travel).

Installing CyanogenMod on a Galaxy Nexus (GSM)

— updated

I just installed CyanogenMod on my Galaxy Nexus phone. There is an official guide; here is my summary of what you need to do.

Backup your data
The process will reset your phone, so you need to back up all your data. An useful open-source program to take care of (part of) this is Slight backup.
Retrieve fastboot
Follow these instructions to install fastboot. I'm not sure that this was entirely straightforward, but I did this long ago so I'm not really sure. fastboot is now packaged for Debian so it is much simpler to install: apt-get install android-tools-fastboot.
Unlock the bootloader
Power down the device, and press the power, volume up and volume down buttons simultaneously for a few seconds. You will thus reach the bootloader. Connect the USB cable and run fastboot oem unlock, and confirm. This will reset the device and unlock the bootloader.
Retrieve and run ClockworkMod
I trust the official guide to have an up-to-date ClockworkMod download link. This being said, you don't need to keep ClockworkMod on your device: you can just boot it as needed. To do so, get to the bootloader like in the previous section, and run fastboot boot CLOCKWORK where CLOCKWORK is the ClockworkMod image file.
Perform a backup
Use ClockworkMod to back up the device before installing anything else, and use adb to retrieve the backup to your computer.
Format all partitions
This is an important step missing from the official guide: you should format /cache, /system and /data before installing CyanogenMod. Otherwise, in my case, CyanogenMod was stuck at the boot animation and adb logcat seemed to suggest that it had to do with a NullPointerException while reading the existing settings.
Retrieve CyanogenMod
Download a CyanogenMod image from this page. I first thought I'd go with a stable version, but I picked the latest nightly as of this writing (cm-10-20120923-NIGHTLY-maguro.zip) and had no problems with it yet.
Install CyanogenMod
Use adb to push the downloaded image to the sdcard folder on the device, and install the image using ClockworkMod.

irctk -- an IRC toolkit

— updated

There are a lot of language-specific libraries to interact with IRC and write bots, but if you want to do this from the shell, the only option I know of is to use ii. Sadly, ii is based on the idea of setting a connection up and interacting with filesystem objects, which is inconvenient if you just want to hack something together in one line like you do with netcat for TCP connections. This post presents irctk, a C program I wrote using libircclient. irctk connects to a server specified on the CLI and uses its standard input and output to read what it should say and write what it just heard. With irctk, you can do stuff like:

# output your server log events on irc
ssh server tail -f logfile.log | irctk example.com '#dashboard'
# timestamp and log irc messages to a file
irctk example.com '#chan' | awk '{ print strftime("%s"), $0; fflush() }' >file

You can also write programs which interact with their standard input and output and then just lift them to IRC with irctk. An example of this is wikifirc, a tool to filter irc.wikimedia.org on specific pages and users. The general scheme is just:

mkfifo fifo
cat fifo | irctk example.com '#chan' | program > fifo

Or you can just write simple programs directly in bash. As a convenience, irctk has options to filter incoming messages and only keep those which are specifically addressed to him, and it can reply automatically to the person who addressed him on the channel where it was addressed. For example, if you address the following bot like "fingerbot: foobar" or "/msg fingerbot foobar", it will reply with information about user foobar found with the finger command:

cat fifo | irctk -Fr fingerbot@example.com '#chat' |
  while read; do
    finger -s -- "$REPLY" 2>&1 | tail -1
  done >fifo

Here is a funnier (bash-specific) example: a bot to roll dice like "dmbot: 3d42" (thanks, p4bl0!): Fixed the code to avoid modulo bias

cat fifo | irctk -Fr dmbot@example.com '#chat' |
while read line; do
  if grep -E '^[0-9]{1,2}d[1-9][0-9]{0,2}$' <<<"$line" &>/dev/null; then
    D=(${line/d/ })
    for ((i = 0; i < ${D[0]}; i++)); do
      shuf -i1-${D[1]} -n1 | tr '\n' ' '
    done
    echo
  else
    echo "format error: must be NdM with N<100 and M<1000"
  fi
done >fifo

irctk has a few other features, like support for commands like "/join", "/nick", "/part", etc., to be able to script actions. Installing irctk should be as easy as installing libircclient (by hand, the version packaged in e.g. Debian is not recent enough as of this writing), and then typing:

git clone 'https://a3nm.net/git/irctk'
cd irctk
make

You can check the README for more information. Comments, suggestions, bug reports and feature requests are welcome at <a3nmNOSPAM@a3nm.net>.

XOR of integers from 1 to n

— updated

A funny (but probably useless) formula. The left-hand side is the XOR of integers from 1 to n, the operators are the usual C bitwise operators.

1 ^ ... ^ n = (n >> 1) & 1 ^ (n&1 ? 1 : n)

It's totally trivial to prove (just write the sequence of XOR values and look at the pattern), but, before you think about it, it's not that obvious that a closed formula exists.

Of course, this formula directly yields a closed-form expression of the XOR of integers from m to n, for any m and n: just compute the XOR of integers from 1 to m-1 and from 1 to n and XOR them together. Thanks to Nathan D. Ryan for suggesting this.