I love email. I really love email. It is federated, which means I can run it on my own server. It is flexible, which means I can bend it to unusual uses. It is pretty simple, which means I can automate stuff with it or even write my own client. It has been around forever, which means it'll probably stay around for some more time. And so, as for everything which I spend a lot of time dealing with (and don't feel guilty about), I find myself trying to manage email the right way.

This post is not about the technical details of my current email setup (which is not perfect yet), but a general description of what I consider to be desirable features for an email setup. I might blog about the technical side once I get it to work exactly like I'd like it to work. Also, this post isn't about what you shouldn't do when sending email to someone, it's about dealing with incoming mail.

Archiving

Most people seem to keep all their email in their inbox (with occasional mass-archiving of all the inbox in a "sort later" folder), and either manage incoming email immediately or hope that they will somehow think about it later. It amazes me that people find this usable and usually remember important things. I use my inbox for messages which require some action on my part (like reading it carefully, reading something they refer to, doing something, answering, etc., or maybe just archiving the message). Whatever is in the inbox has to be moved out at some point. My inbox is a todo list.

The reason why people never archive their mail is because they usually have complicated schemes to archive messages. For instance, I've seen people archive messages in folders labeled by sender (hey, I also used to do that back in the days), which is quite pointless because this information is in the message anyway. My solution is different: one big archive folder. I'll sort later if I need to. Most stuff which gets there is never needed again. Archives like that should be optimized for writing.

Actually, I'll probably never need to sort anything because I can just search my mail (using indexing to get decent speed). True, it can be useful to keep some tags in order to add information which is not easy to find in the message (say I want to find quickly all email about some project...), but I'll only do it if I think I could probably need those messages later. If this isn't the case, just archive, and search later if need be.

Speed

The system should be keyboard-driven, and fast. "keyboard-driven" isn't just about having keyboard shortcuts, it's about having an interface designed with the keyboard in mind. "fast" isn't just about optimizing the hell out of some operations, but about having no useless noticeable delays, which is surprisingly rare.

Integrity

Unless you're working under very tight space constraints or you're being mailbombed, I don't see why you would delete a message, ever. Seriously. Archive it, and have a tag for things that you don't want to see unless you explicitly search for them. There is no reason for this to be hard (just bind it to the key you would ordinarily use to delete stuff).

This section is called "integrity", because I feel that you should distinguish the pristine data that you received from the stuff you add on top of it. Deleting a message violates this rule; modifying a message also does. Mainstream email clients usually don't do that (except to toggle flags), but I've seen mutt plugins silently change the message headers when you did some thing or the other. There should be a pristine copy of the mail which arrives, augmented by a datastore of what you did to the mail, probably supplemented by a cache to speed up things like search. The point is that you should distinguish the base data (email), the user data (labels, annotations, whatever) and the cache (which is just there for speed and can be regenerated).

Threading

Email threading is an absolute must. I cannot imagine what my inbox would look like without threading. I remember I was very surprised to learn that threading isn't guesswork based on subject and citations but a standardized, elegant and simple mechanism (with the headers Message-ID, In-Reply-To and References) which is unexpectedly followed by almost all email clients, even the ones which do not use them for display.

By email threading, I mean regular threads, ie. trees. The Gmail linear approximation (put messages in a thread, but don't show the tree, order messages by date) is better than nothing, but not satisfactory.

Priority inbox and spam filtering

People seem to rely on spam filtering to get rid of unwanted messages and, for Gmail users, rely on Priority inbox to cherry-pick important messages. Those two tools are useful, but it is unacceptable to delegate the task of choosing what to read and what to ignore to some software outside your control which tries to be intelligent.

As it turns out, I don't need those two tools, because I don't get much spam and I am able to deal with individual messages manually. If I needed to, though, I would insist on rolling my own there, more than anywhere else, to be sure that the filtering is sensible and that no important message gets lost because someone (or rather something) thought they knew what was interesting to me.

Mailing-lists

Email is not necessarily designed for messages that you want to read. It's also a convenient way to receive stuff you might want to catch up on later. Think of it as a neat way to keep a mirror of a mailing-list archive and browse it at some point if needed. It's okay to subscribe to tons of mailing-lists, as long as you filter them and redirect them in a folder which is separated from your real mail.

There is very little motive to unsubscribe from anything except technical limitations. If a list used to interest you, it might interest you again later--just redirect it directly to the archive, and tag it so it won't pollute your search results unless you want to.

Tags

In addition to the "tags to help future searches" idea mentioned above, there is also the use case "tags to manage your workflow". Everyone probably has their optimal setup, but there are a few common traps:

• A message can be "unread" because you didn't read it completely and satisfactorily, or because it just arrived and you didn't triage it. The second one is far more urgent than the first--you want to eliminate useless stuff and see important stuff at once, but having a long non-urgent message identified as such sitting around "unread" is acceptable. It might make sense to open an "unread" message in the second sense, and mark it as "unread" in the first sense (aka. "todo: read").
• The "sent" messages fall broadly into three categories, you will probably have to distinguish them somehow:
  1. The messages you have to send but don't care about, like answering a request from someone else. Once you have sent the message, the thread can be archived, because you don't care about it unless the other guy replies.
  2. The messages you send and care about, like requesting something. Once you have sent the message, you have to keep it somewhere and make sure that you renew your request after some time until you get an answer or lose interest.
  3. Personal messages where you care about the other person, ie. giving news and receiving news every now and then. The right way to manage this would probably be a counter indicating the last time you sent a message to someone, to give news regularly. A decent approximation, though, is to manage the thread like for case 2.
• There are "blocked" messages for which you have to do something, but cannot do it right now. Examples are: you need some information which isn't available online, you need to ask someone IRL, you have to wait for someone else to do something, etc. You need to keep track of those messages, but not keep them in your inbox and spend your time bumping into them and remembering that you can't do anything about them yet. It might be a good idea to annotate such messages to remember what you are waiting for before you move them back to the inbox. The problem is that you still have to check those every now and then to see if what you were waiting for has taken place; it would perhaps make sense to indicate when you want to be reminded about the message's existence.

Choices

If the inbox is a todo list, then there is a tradeoff to find between guidance ("next step is: do X") and flexibility ("actually, I feel motivated about Y right now"). Presenting the list of messages in the inbox and asking the user to pick one is very flexible, but the high number of possible choices can be daunting. It's interesting to imagine what the opposite approach would feel like: the system would display threads one after the other, and requires you to either do what is to be done, or add a note explaining what you're waiting for...

I have a book collection which I would like to have an index of. Fortunately, most recent books have a barcode on their cover which indicates their ISBN, an identifier which is unique for the book and can be used (at least in theory) to retrieve details about the book (title, author, cover image sometimes...). Thus, to keep a list of what is in my bookshelves, I just have to scan the barcode of each book and save the list somewhere.

To do this, I need a tool which takes a video stream as input and outputs the list of recognized EAN-13 codes, one per line. I do not need a fancy app with a slick captive user interface which would try to be smart and store a database in some mysterious place while making it harder for me to script things. Do one thing, and do it well.

If someone else intends to do this, here are two possible ways, and ideas to use the ISBNs.

Barcode Scanner on Android

This is actually a pretty natural thing to do with an Android phone, except you have to find the right barcode scanning app. (Most turn out to be riddled with ads, or ridiculous limitations, or don't have the required feature.)

The ZXing library has an Android port called Barcode Scanner which was able to read almost all tested barcodes with the webcam of my HTC Desire. Activate the Bulk scan mode option in Settings, scan items, go to History, select Send history (beware, Clear history is right next to it and has no confirmation step), and send an email to yourself with the csv attachment. It is free as in freedom (Apache License 2.0).

zbarcam

The zbarcam command (packaged for Debian) is exactly what I needed, but my computer's webcam was too crappy to scan barcodes. This might work for you if you have a better webcam. For the record, before learning about Barcode Scanner, I tried to use my HTC Desire as a webcam over USB using AndroidUsbCamera. Sadly, it does not work on recent kernels because it requires V4L1 support which was dropped in kernel 2.6.38.

What to do with the ISBNs

I thought that an ISBN lookup program would be packaged for Debian, but apparently not. The simplest API to use seems to be that of Open Library (a nice project, by the way, which provides dumps which are apparently in the public domain): example for ISBN 9780517223628. 56 out of my 115 scanned books were found.

For better options, a good starting point seems to be this StackOverflow thread. It seems that the Google Books API is the best one, but interfacing with Google APIs seems to require API keys, registration, etc., so I didn't bother yet.

This is not rocket science, but maybe it can be useful to someone, or it can serve as a simple example of how you can query the Mediawiki API (or any JSON API) using Python 3. The two arguments are the name of the wiki (like "en.wikipedia.org") and the name of the category, and the output is the contents of the category, one item per line. Request continuations are handled, the script will pause for two seconds every request (ie. every 500 items).

For the code (public domain), see here: wikicateg

Here's an example:

$ lswikicateg.py en.wikipedia.org Experiments
Animal testing
Carryover effect
Checking whether a coin is fair
Control variable
Cranfield Experiments
Generality (psychology)
Inflatable Antenna Experiment
Jerry Klein's 2006 radio experiment
LOCAD
Long-term experiment
Lost in the mall technique
Natural experiment
Scientific control
The Third Wave
Category:Experimental archaeology
Category:Experimental economics
Category:Experimental film
Category:Experimental mathematics
Category:Experimental mechanics
Category:Experimental medical treatments
Category:Experimental methods of birth control
Category:Experimental missiles
Category:Experimental music
Category:Experimental physics
Category:Experimental psychology
Category:Experimental rockets
Category:Science experiments
Category:Experimental television stations
Category:Thought experiments
Category:Experimental vehicles

As part of my musings with irctk (which is not yet usable as of this writing), I have been looking for ways to run commands without bufferisation. This is annoyingly hard, because what should make you save time (one-line IRC bots) is actually a mess of cases where things fail to work because buffering happens somewhere in some program in the pipe.

If the program you're trying to pipe is one that you wrote, there are often (more or less obscure) ways to disable buffering or flush buffers by hand. Think about the "-u" option in Python 2 or the (non-POSIX) "fflush" command in awk. If you didn't write the program, maybe it already has an option to do what you want (GNU grep has "--lines-buffered") or maybe not (I'm not aware of such a trick for cut, for instance).

In the last case, you have to use an external tool to change the behaviour of the program. In the relevant StackOverflow question, the suggested tools are a simple but limited program called unbuffer:

unbuffer myprogram

or a complicated socat call:

socat EXEC:myprogram,pty,ctty,echo=0 STDIO

It turns out that there is a third option: the stdbuf program in (recent versions of) the GNU Coreutils does the job both powerfully and with a pleasant syntax. Here is my current recipe to get rid of buffer woes:

stdbuf -i0 -o0 -e0 myprogram

If you want to really know how all this mess works, here is a nice explanation. If you don't care, just remember that (1) buffering can cause data to get stuck in pipes and that (2) stdbuf is the program you want to use in this case.

There are basically two ways to receive information you care about. Either poll periodically the information sources (check your inbox, rss feeds, refresh pages, etc.), or get notified whenever a new item arrives.

Intuitively, notifications seem better. They ensure that no time is wasted in useless polling, and that you receive information in a timely manner. They enable you to get those information sources out of your head and get things done, and only get notified when action is required on your part.

Of course, this means that you get distracted from what you were doing if a notification arrives, which is dangerous because of the time it might take to get back to the right mind configuration once you're done dealing with a piece of information. Much has been said about this problem, and about the fact that you should avoid notifications if you want to be productive.

In my opinion, avoiding notifications entirely is a bad solution, because checking information sources compulsively (and more often than needed) is just too tempting a way to procrastinate. I think a better solution is to have notifications which you can mute (ie. enqueue incoming notifications) and unmute (ie. dequeue the backlog), so that you can either stay connected when you're not doing stuff which requires high concentration, or get in a state of flow and be sure that you won't have missed anything when you turn the notifications back on. Obviously, the right way wouldn't be to have "on" and "off" states, but to have an importance threshold (with different sources having different priorities, along with more elaborate filtering).