a3nm's blog

New GPG key

— updated

It's a bit of a pain, but it turns out that my old OpenPGP key was using suboptimal settings, and so I've regenerated a new one.

I did this after reading this fine best practices tutorial (for which I incidentally helped write a French translation). The gist of it is (1.) that you should set up GPG correctly to fetch keys from key servers (there's the parcimonie-related paranoia, but there's the very embarrassing fact that by default it seems that GPG never manages to talk to a key server); and (2.) that you should check that your key is secure by issuing the following and checking for things in red:

sudo apt-get install hopenpgp-tools
# FINGERPRINT is the actual fingerprint, not a key ID
hkt export-pubkeys "FINGERPRINT" | hokey lint

Generating a new key isn't especially hard but here is a reminder of what you have to do. You then have to re-sign the keys that you had signed with the old key, using the new one...

The formal transition statement signed by both keys is here, so that you can sign the new key if you had signed the old one.

SVG rendering in Gecko and Webkit

— updated

Did you know that Gecko and Webkit's implementation of SVG disagree on fairly basic points of the spec? For instance, this SVG image should be rendered differently depending on which engine you are using.

The trick is that one of both texts is hidden using a gradient calculated relative to the bounding box of a path with control points going outside of the path, which is computed differently by Webkit and Gecko. See the source of the image for a bit more detail.

It seems that Inkscape is also working like Gecko, and its way of computing bounding boxes makes more sense, so this is probably a Webkit bug, however a quick search didn't turn up the relevant bug (though this is probably well-known).

I devised this SVG with the help of Marc Jeanmougin who has had a good reason to investigate SVG implementations in browsers lately.

Apparently, this no longer works as before. The image now renders more or less the same in Chromium and recent versions of Firefox (it says "using webkit", give or take some text overflow), though it renders differently in Inkscape and older versions of Firefox ("not using webkit").

Review of the Lenovo IdeaPad Yoga 13 (59384174)

— updated

To replace my dead laptop, I just bought a Lenovo IdeaPad Yoga 13, often identified with the code 59384174 although the official model number seems to be 2191; the EAN13 code is 0887942530297. Here are my impressions about the device. I am using Debian Testing amd64, kernel version is 3.13.1-amd64 as of this writing.

Criteria
I wanted a laptop with at least 13 inches (feels too cramped otherwise), with an SSD (and no SSD/HDD hybrid, I dislike mechanical drives for laptops and would not trust the controller's caching policy), with a screen resolution better than the default WXGA 1366 by 768 resolution of cheap 13 inch laptop screens, and I wanted the device to be light and slim (the Yoga 13 is 17 mm thick and weighs 1.5 kgs). Oh, and I wanted reasonable battery life.
Windows refund
The machine came with a preinstalled Windows 8. I contacted Lenovo France to get it refunded. However, the only option that their customer service offer to get a Windows refund is to refund the entire laptop... this is really sad.
Price
I managed to buy the device for 590 euros: it was priced at 700 euros by Cdiscount (may not apply anymore as of this writing), 10 euros could be recouped if paying via Buyster, and 100 euros could be recouped from Lenovo France as of this writing through a special offer. From all the devices that I reviewed, this was the cheapest one to satisfy all my criteria, by a fairly large margin.
Charger
The charger is pleasantly slim and outputs 20V. The plug for the power slot in the device is a rectangular but reversible plug.
Power button
Note that if you need to power down the device during the Windows setup phase (because, say, you inadvertently entered it), long-pressing the power button will suspend the computer, not power it off. You need to continue pressing the button after that to turn the computer down. Also note that the power button is easy to hit by mistake, hence I configured it in /etc/systemd/logind.conf to not cause any event when pressed; you still have to be careful when the device is suspended so that you do not make it resume by mistake, e.g., when putting it in a bag (because of the risk of overheating).
Screen
The screen looks very nice, though a bit glossy. The 1600 by 900 resolution is pleasant. The viewing angle is very wide (no noticeable hue shift except when you start looking at it diagonally). Screen backlighting can be controlled (even disabled outright) through /sys/devices/pci0000:00/0000:00:02.0/drm/card0/card0-LVDS-1/intel_backlight/brightness. By default there is some tearing during video playback, I tried to fix it with this method, not sure how much it helps.
Tablet
The main selling point of the Yoga series is that the screen folds 360 degrees to a tablet-like configuration. I only wanted to buy a laptop and have little plans to use this functionality (also for the reason that I wouldn't know which operating system to use to work as a tablet yet give me a suitable environment for a regular laptop), so that's just a bonus. Trying out the mechanism out of curiosity, it makes a fairly heavy and strange tablet, and having the keyboard behind is fairly bizarre. The rotation is achieved by a double-hinge mechanism: a standard hinge system works up to 180 degrees, and then those hinges rotate in a second hinge to go all the way to 360. The mechanism doesn't look especially solid and I wouldn't trust it not to break if used regularly, however for less than 180 degrees the standard hinge mechanism seems OK... After a few months of use, the right hinge has become a bit loose compared to the left one, which worries me a bit. A related complaint is that some magnets are used to make the 0 degrees (closed lid) and 360 degrees (tablet configuration) stable, and for 0 degrees those magnets are a bit too strong, so that it is a bit hard to open the laptop. On Linux, of course, if you use the Yoga in trestle mode, almost fully folded and reversed, which can be convenient e.g. to watch movies without having the keyboard in the way, you will have to take care of reversing the display with xrandr but also of reversing the input if you want to use the touchscreen. Here's a script to do this for you.
Orientation sensor
From the existence of a hardware button to lock screen rotation (see below), I imagine that the device must include an accelerometer, but it doesn't seem to be supported by the Linux kernel. Nothing relevant shows up in /proc/bus/input/devices or in /dev/input. In any case it appears that the mechanism is causing problems, which I have also experienced, and I had to add the relevant commands to /etc/rc.local to make it stop.
Sound
Interestingly, speakers are on the rear and the integrated microphone is on the side, probably because they must be usable both in laptop and tablet configuration. Sound quality is not especially good but not especially bad either, given the form factor.
Connectivity
The device features one minijack port (combined microphone and headphones), one eSATA-USB hybrid port, one USB port, one HDMI port, the power supply connector, one USB port, one SDcard-MMC reader, and that's it. Of course, no Ethernet port because the device is too thin. I bought a cheap USB-Ethernet adapter which was detected out of the box by Linux and works fine except that it is USB 1.0 so it is limited to about 750 kB/s.
Keyboard
The keyboard is pleasant enough, the keys are fairly high and have a reasonable size. The Left control is at the bottom left of the keyboard with the Fn button is at its right, which is how it should be (some laptops have the reverse). There is only a Super_L modifier (Windows key), not Super_R, but that's OK. Keys are positioned reasonably except that PageUp and PageDown are not quite at the position where I would have expected them (I would have expected them to be one row higher). By default, the F1-F12 keys can only be reached with Fn and the special (Volume, etc) fonctionalities being the default, but fortunately this can be changed in BIOS. The F1-F12 keys have the following additional features:
  1. Mute, generating XF86AudioMute.
  2. Volume down, generating XF86AudioLowerVolume.
  3. Volume up, generating XF86AudioRaiseVolume.
  4. Cross key, generating an Alt-F4 sequence (Windows keyboard shortcut to close a window).
  5. Refresh key, generating F5 (so doubles with the default behavior of the key.
  6. Disable touchpad key, exposes no events in xev but reports the following in dmesg: atkbd serio0: Unknown key pressed (translated set 2, code 0xbe on isa0060/serio0). Use 'setkeycodes e03e <keycode>' to make it known.
  7. Plane mode key, generating XF86WLAN, and from the dmesg it seems like the Wifi driver is seeing this also and disassociates, but nothing seems to be seen in rfkill.
  8. Three rectangles key, generating a Ctrl_L-Alt_L-Tab sequence (probably a Windows keyboard shortcut to switch application or something).
  9. Screen off key, toggles backlighting on and off (preempts the status in intel_backlight, no xev or dmesg event).
  10. Screen whatever key, generates a Super_L press? (Nothing more shows up, maybe things would be different if I had an external screen plugged in, but I have none to test this hypothesis).
  11. Brightness down key, generates XF86MonBrightnessDown (my i3 config uses xbacklight to handle this)
  12. Brightness up key, generates XF86MonBrightnessUp (ditto)
Additional keys
The laptop comes with a bunch of strange additional buttons, in addition to the keyboard. There is a button with a Windows logo (annoying, that) below the screen, which must be intended to be used in tablet mode with Windows; from xev it seems to simulate a press on Super_L (the windows key on keyboards), but only when released (it generates both the KeyPress and KeyRelease events when released). On the left side are two volume keys generating XF86AudioLowerVolume and XF86AudioRaiseVolume. On the right side is a button to lock screen rotation, which seems to emit the sequence Super_L+o (the Windows hotkey to do this, it seems). Last comes the power button and a small button on the left that can only be pressed with a pen or such; the documentation refers to it as the Novo button. It generates XF86Launch2.
Secure boot
Secure boot can be disabled in BIOS, whatever that is. BIOS offers both UEFI and legacy BIOS boot, legacy boot is not too fast. Interestingly trying to boot a Debian install USB key with UEFI will work except that the screen display will be garbled, this is fixed using legacy boot.
Bluetooth
To get Bluetooth to work, install the driver and load the module. Note that it seems to be necessary to manually install the Bluetooth driver even though the Wifi driver is now integrated to the kernel. Now, here is how I get my Bose Mini SoundLink speakers to work. bluez-test-audio seems to have disappeared from the bluez package in Debian recently, I am not sure about how to fix these instructions. First, reset those speakers and put them in discoverable mode. Install the bluetooth daemon (Debian package blueetoth) and pulseaudio-module-bluetooth, try to issue pactl load-module module-bluetooth-discover. Ensure in rfkill that the hci0 device exists and isn't blocked. Then, retrieve their address by issuing sudo hcitool scan. Issue bluez-simple-agent hci0 ADDRESS and then bluez-test-audio connect ADDRESS. The speakers should indicate that pairing succeded. Now, check in pacmd list-sinks that a sink exists for the speakers, and use pavuctl to connect your music player to the right sink.
SD card reader
The reader appears as /dev/sdb, not /dev/mmcblk*, and inserting a card will not do anything until you try accessing it with fdisk -l /dev/sdb, at which point the partition files /dev/sdb* should be created. I haven't tried if booting off the SD card reader was possible.
Touchpad
Since I wrote these instructions, things seem to have improved, so I am not using mtrack anymore. The following should no longer apply. The touchpad has an anti-feature: the button press zones are tactile, meaning that, by default, your pointer will move when trying to just click, this is extremely annoying. You can use synclient to set AreaBottomEdge when using the synaptics driver, but this has another downside: it means you cannot click and drag because clicking locks the pointer. I got much better results by installing xserver-xorg-input-mtrack and enabling it in xorg.conf: clicking at the bottom locks the mouse pointer except past a certain move threshold, which is the right thing to do. It is fairly annoying, however, that I couldn't get mtrack's ButtonZonesEnable feature to work, so to do a right click I have to click with one other finger touching the pad (and when left-clicking I mustn't leave another finger on the pad). I tend to touch the pad while typing, the IgnorePalm option seems to help a bit to ignore that.
Webcam
Webcam works. Test it, e.g., with Cheese.
Ambient light sensor
The device has an ambient light sensor on the screen that can be made to work with the Zenbook driver (clone, make, insmod als.ko, requires linux-headers etc.); the value ranges from 0 in total darkness to 40000 in maximum light, with a value in the 1500-2000 range in normal indoor lighting conditions. I didn't try to write a script to adjust screen brightness automatically based on ambient light.
CPU and memory
/proc/meminfo reports 3938992 kB RAM. I haven't checked but there should be one RAM slot with non-soldered RAM, upgradeable by replacing by an 8 GB DDR3 module in SO-DIMM 204 pins packaging (costing about 75 euros as of this writing). /proc/cpuinfo reports the CPU as a Intel(R) Core(TM) i3-3227U CPU @ 1.90GHz. CPU frequency scaling is available, but cpufreq-info reports that it uses the intel_pstate driver, and that only the performance and powersave governors are available, with powersave being used by default. Loading the performance driver doesn't seem to make much of a difference. From cpufreq-aperf it looks like the CPU frequency increases up to the maximum 1.9 Ghz under load, even with the powersave governor (so it isn't locking the frequency to the lowest possible value).
Battery
Battery status can be read with ACPI as expected. From a quick test, during reasonable usage (browsing, investigating hotkeys, partitioning, etc.), and with phone plugged in (to tether and work around a Wifi glitch), battery life was around 4:30.
Screws
The entire case is closed with Torx-looking screws, and it seems that nothing is intended to be user-serviceable. In particular, you are probably not expected to replace the battery yourself. Of course, people have succeeded in opening their device, and a cursory search suggests that replacement batteries may still be available.
Fan and temperature
Two fans evacuate heat at the back of the unit. Four temperature sensors are detected by libsensors, one under acpitz-virtual-0 and two under coretemp-isa-0000. Temperatures are about 40 degrees in standard usage, with the fans spinning. Annoyingly I found no way to retrieve the fan RPM or to control the fans, and I find it sad that the fans seem to be always spinning even when temperature is low; they are not too noisy but total silence would be perfect, and that's just wasteful in terms of battery life. Temperatures reach 50 degrees and the fans speed up a bit under heavy load. There is code to control fan speed in software, that works fine. I recommend it heartily, because under normal loads the CPU temperature will stay well below 55 degrees with the fans powered down entirely.After trying to run with this script for a while, I no longer use it, because when the script is running it semes that the kernel occasionally sees critical temperature values and powers down. As those values are outliers I think they are just a glitch, but it seems that running the script triggers it, I don't understand why.
SSD
SSD is a Samsung MZMTD256, 238.47 GiB. Surprisingly it seems that if you are willing to open up your unit you can install a second mSATA SSD in a vacant slot. The default partition layout is surreal, with a whopping 7 partitions: WINRE_DRV, a 1000 MiB NTFS partition, SYSTEM_DRRRV, a 260 MiB bootable FAT32 partition, LRS_ESP, a 1000 MiB FAT32 partition, a 128 MiB msftres unknown partition, the main Windows partition (which is the only one that isn't hidden), and from memory a 4 GB partition and a 12 GB partition for obscure recovery purposes. I didn't touch the first four ones because they don't take up that much space and I'm afraid of making the device unbootable (that's probably an unrealistic fear, but who knows...), but I removed the Windows partition and the two last recovery partitions to install Debian instead. No problems in using GRUB (though I haven't tried booting anything except Debian).
Network
Wifi does not work out of the box, and the absence of an Ethernet port means that Debian installation may be painful. A possible workaround is to use an USB-Ethernet adapter (not tested), or to tether a mobile phone's Wifi connection (Debian install knows how to use such connections) The Wifi chip has USB ID 0bda:1724, and to make it work, for now, you should compile and install the rtl8723au driver. It is painless and works. However, it doesn't seem like changing the MAC address is supported; sudo ifconfig wlan0 hw ether whatever doesn't fail but doesn't change anything, even if the interface is down and even if it is rfkilled. I haven't tried very hard yet or investigated whether the driver had a specific option for this. It does not look like activating monitor mode for airodump-ng works either. Also, the driver was merged in Linux 3.15, so with sufficiently recent kernels the Wifi now works out of the box. Sadly, there seem to be some crashes related to the Wifi, see this bug. Fortunately, these crashes can be avoided by installing manually the rtl8723au driver instead of using the one shipped by the kernel.
Hibernate
pm-suspend seems to work, taking about 1 second to suspend and resume. pm-hibernate works but I had to configure the hibernate swap partition manually. Hibernating takes around 10 seconds, booting and resuming from hibernate takes about 16 seconds.

All in all, I am pretty satisfied with the device so far.

Propositions for EU copyright legislation

— updated

The European Commission has opened a public consultation on the review of the EU copyright rules, open to all stakeholders. Their questionnaire is a bit lengthy, but fortunately, according to the Quadrature du Net (a French advocacy group for Internet freedom), you can also answer with a free-form text. So I did just that, taking it as an opportunity to write down some things I would like to see change in the EU copyright legislation. My answer is now published among the 4213 (!) user replies.

What I propose here is my moderate program for copyright reform. My actual views are a bit more extreme, and, within an idealized model of society, I am not entirely sure that copyright is a good thing altogether, in the longer term. Yet this is not a program for an idealized world, but concrete action steps which I feel should be taken as soon as possible to cut down on current copyright legislation's most blatant excesses, so as to reestablish a minimum amount of equilibrum between the interest of right holders and those of the general public.

Yet, I have obviously no hope that this letter will have any impact, because my view are so distant from current EU copyright policies. I would nevertheless like to invite you, dear reader, to submit your opinion to the consultation, while there is time (the deadline is Wednesday, March 5th), even though it feels a bit like you are writing a letter to Father Christmas. Not telling the European Commission about your views, and not giving them the opportunity to ignore them, will limit your right to complain legitimately about what they are doing on copyright issues.

This being said, here are the concrete changes which I suggested. This Web version includes hyperlinks and corrections to the original text.

I understand copyright as a balance struck between society's desire to use, distribute, adapt upon and perform creative works, and the necessity to incentivize creators by granting them a temporary monopoly over some rights on their work.

Current European legislation protects the rights of authors until seventy years after their death. This is disproportionate: such a long duration is certainly not required to encourage or sustain any reasonable form of creation, whereas society is substantially missing out on the opportunity to benefit from many works of art over this long period.

The only explanation I can see to the current state of affairs is that it was achieved through organized lobbying by right holders, which faced little resistance: because the general public does not know about the public domain or realize its importance, and because the organizations which attempt to protect it have a comparatively small economic weight.

I would therefore be in favor of restricting the duration of copyright in European member states, in all circumstances, to no more than 25 years after publication, performance or broadcast.

Since the Berne convention, there are no formalities required to assert copyright over your work; so that, for works with no known or reachable right holder, one must assume by default that the work is protected. Hence, the public's use of the work is limited to protect the monopoly of an absentee right holder: one cannot republish, digitize, adapt upon the work, for fear that the right holder may later resurface and sue for damages.

I am not in favor of introducing mandatory registration formalities for copyright, as I fear they would penalize the less-organized forms of creation (street art, amateur artists, etc.) who would probably not follow them.

However, I believe that, once the creator of a work has ceased to distribute it (out of print books, unmaintained software, etc.), and it has been impossible to contact them for some period of time, copyright protections on the work should lapse. The rationale would be analogous to that of acquisitive prescription: the right holder has shown insufficient motivation to exercise their exclusive rights.

In particular, reappropriation of unavailable works in a for-profit fashion, such as the French ReLIRE, do not serve the public interest.

If a work has been created by employees of European Union member states or European institutions during the course of their duties, it should not enjoy any copyright protection. The reason for this is that there is no need to incentivize or finance such creation, and as this is the best regime to allow the public to benefit from such works created using public money.

This would have many positive effects. As a first example, this would encourage the dissemination of publicly financed research, and ensure that copyright cannot be leveraged by academic publishers to restrict the transmission of scientific knowledge to the general public who indirectly paid for it. Hence, this would contribute to the decline of closed-access scientific venues, which would help decrease the proportion of public research money spent on subscription fees.

As a second example, this would ensure that open data efforts from public organisms must remain really open. This would not not prohibit the sale of extended datasets so that the cost of publishing open data is covered by those who make use of it, but it would ensure that copyright cannot be used as a weapon to limit the further redistribution of such datasets, as could be attempted by short-sighted organisms in a foolish attempt to reap easy profits at the public's ultimate expense.

Digital rights management (DRM) are a set of technologies to limit the copying of digital content in an effort to enforce copyright. Works burdened with DRM usually require specific software, operating system, hardware (sometimes from a specific manufacturer), in order to be consumed. Further, some such technologies require the user to remain in contact with the right holder's server to authorize every use of content, so that the works become unusable once technical intermediaries discontinue the service or remain out of business.

DRM is present on a variety of digital contents, such as e-books (Amazon), video games (Steam), movies (most DVDs), etc. While I respect the right of manufacturers and right holders to offer such content under such conditions, it seems paramount to me that customers be aware of the nature of what they are buying. Right now, the public is not well-informed about the nature and existence of DRM, and the presence of DRM is seldom clearly advertised.

I would argue that any creative work on sale should indicate prominently to the purchaser which equipment is required to read it: software, hardware from a specific manufacturer, and more generally any technology that it is an open standard which can be re-implemented by anyone (i.e., not requiring the payment of any royalties).

Further, if a creative work is offered in a scrambled form designed to be unusable without additional keys provided every time the content is consumed, it should be mandatory to label such offers as "rental" rather than "sale", as continued enjoyment of the work by the customer depends on the provider's will to further allow it.

Current European directives, such as Directive 2001/29/EC, have overreaching provisions which prohibit circumventing DRM, providing technical means to enable it, etc. This often prohibits legitimate cases of DRM circumvention, such as those required for interoperability. Prohibiting copyright infringement is sufficient ; no additional protection should be granted to DRM technologies.

The "freedom of panorama" exemption restricts the copyright protection of works of art permanently located in a public place. This exemption exists in some member countries but not others. European legislation should be harmonized by providing freedom of panorama in all member states, in its strongest form: there should not be any copyright protection of works of art permanently located in public space, as such copyright protection substantially interferes with the public's right to photograph and film in public space.

A huge amount of creativity nowadays is expressed by producing works of art that current copyright legislation consider to be derivative: remixes of well-known songs, fan fictions, videos incorporating sequences of protected video games, etc. The legal status of such works is entirely unclear, and right holders can leverage this to discourage, or censor, such creative works.

In my opinion, the need to incentivize creation through copyright does not justify such restrictions on derivative art forms. Indeed, such transformation has always been an important factor of creativity (folk tales, reuse of musical themes, etc.). It is regrettable that copyright has crept from restrictions on "fixed-form works" to restrictions on certain derivative uses such as performance and translation, to eventually impose overbearing theoretical restrictions on any form of derivative use, which I believe does not reflect the interest of society.

I believe the scope of copyright protection should be changed so that it does not impede transformative uses that cannot harm the interests of the creator. For instance, fan fictions certainly cannot be read as a substitute to the original book, nor can most remixes replace the original song, nor playthrough videos act as a substitute to a video game.

Not-for-profit sharing of copyrighted works between private individuals is a widespread practice in European member states. It was essentially uncontroversial in the times where works of art were fixed to a physical medium (such as books, CDs, etc.), because this made copying unpractical so that sharing was essentially limited to lending.

For digital works, such practices raise the question of how creators should be financed. However, prohibiting it altogether, as is currently done, does not seem sensible: it is disconnected from current practice and from the public's perception of right and wrong. I am unsure about whether a middle ground could be found with solutions such as the French "licence globale" proposals which propose to finance artistic creation through taxation and redistribution to creators: indeed, I do not see how to design a scheme of this kind while ensuring that it finances all artists fairly, including independent artists not registered to any copyright collection society.

I am nevertheless confident that new ways to finance creation, such as crowdfunding, will emerge; that the public is aware of the fact that they need to pay the creators of works they enjoy, and that private artistic sponsorship will increase as technologies reduce the friction of online payment. Hence, I would be in favor of a copyright exemption covering the not-for-profit exchange of creative works between private individuals, as a solution preferable to the current state of affairs.

Copyright is usually held, not by artists, but by copyright collectives of which they are members. Some of these, such as the French SACEM, require that their members automatically assign copyright to them on all their future creative works; terminating one's contract with SACEM is a lengthy and complicated process. [Author's note: There seems to be no up-to-date and reliable information available online about this...]

Such agreements are nothing short of creative servitude; the signer agrees to the transfer of all the fruits of their creativity, with little limits, uncertain compensation, for an indeterminate amount of time. Contracts of this kind should be deemed unfair and considered null and void.

Transferring preemptively one's copyright on one's future creations should be impossible except within employment contracts, in which case this arrangement should be restricted to the hours worked by the employee; additionally, resignation should always be possible, and all relevant labour protection laws should apply.

CalDAV and CardDAV with Radicale and DAVdroid on Android

— updated

In this post, I present an entirely self-hosted solution to manage your calendar and contacts on an Android phone running CyanogenMod, synchronizing to your own server using the open CalDAV and CardDAV protocols, using the free as in free speech DAVdroid program as client (along with the stock calendar and contacts application that come with Android) and Radicale as server. As of this writing I am using radicale 0.8 and davdroid 0.5.1-alpha.

To my knowledge, as of this writing, this is the only way to manage your calendar and contacts on an Android phone which satisfies the following desiderata: no dependency on closed protocols (eliminates Exchange); no dependency on a third-party server (eliminates Google Calendar); synchronization with an external machine (eliminates "local" calendars and local contacts); no dependency on proprietary applications (eliminates all the proprietary DAV software in the Google Play Store); actually functional (eliminates all the DAV clients I tried before DAVdroid).

After a few weeks of usage, the setup described here seems to work fine for me, except for occasional crashes of DAVdroid (but it restarts by itself and I have not experienced any data loss to date due to this, this is just a minor inconvenience of having to acknowledge occasionally that DAVdroid had to exit). Note that I usually only edit data from the phone, not a different DAV client (though I tested this once with Evolution and it seemed to synchronize fine), and I did not test having more than one calendar (maybe I will do so eventually, just to have different colors for different kinds of events on my phone), so I can't testify that this works for me

I have changed the server configuration: see this newer post for explanations. This update only concerns some sections of this guide, which are pointed out explicitly.

Warning about migrating existing data

Before we start, a big warning. On Android (at least on the versions I have used), contacts can be local to the phone, but events can not be local. If you have been using the default calendar application with what you think are local events, they are actually associated to some account provider (Google Calendar, Exchange, or some other application -- in my case, it used to be one of the many other buggy calendar synchronization applications), and deleting the associated account (even though it might seem unused) will delete your entire calendar. I did this once, essentially "lost" my entire calendar except for a nigh-unusable SQLite backup, and then I lost most of the data on my phone by a misguided attempt to undo this mistake by hand that messed up file ownership. Don't do that.

This being said, I don't know what would be a better way to migrate your existing events in the setup that I describe if you have been using pseudo-local events (with a dummy provider) on the stock Android applications. You can dig out the SQLite database in /data/data/com.android.providers.calendar/databases/ on the phone but that's not really helpful because I don't know of any way to convert it to something that can be imported. So well, if you used that, like me, you are screwed, and should have investigated switching costs more carefully.

For contacts, however, according to feedback from F., you can export them using the "Import-Export" functionality of the stock Android contacts application, and reimport them back once the CardDAV account has been set up, and it should work as intended.

For calendar events, according to Thomas Bartosik, there might be a solution to mass-migrate your calendar events from a local calendar to the DAVdroid one, by retrieving the file /data/data/com.android.providers.calendar/databases/calendar.db and editing it to copy the contents of the events table to itself, changing the value of the calendar_id attribute from the ID of the local calendar to that of the DAVdroid calendar (those IDs being obtained by peeking in that table). (Sorry, I won't provide the SQL command to do this because it is long and depends on the exact schema of that file.) Note that you should of course do backups before trying something like this, and I can't guarantee that this solution works as I have not tested it myself. The calendar and calendarstore should be killed on the phone once the modified database has been installed, so that the changes are picked up.

Now that you are warned, let us start with the actual guide on a more cheerful note: once the CalDAV setup described here is operational, your calendar and contacts will be synchronized with your computer and stored in the open iCalendar format, so you should be able to migrate them to something else whatever happens.

Install CyanogenMod

As CyanogenMod is discontinued, you should install, e.g., LineageOS

Not strictly necessary, what I'm describing probably works on Android too with minor adaptations, but I prefer CyanogenMod because it is more customizable and makes it possible to avoid cleanly all the proprietary Google applications. I won't describe how to install it -- if you're interested, go to the official website and follow the instructions. I'm using CyanogenMod version 10.1.3-maguro, which means a Galaxy Nexus.

Install F-Droid

I manage applications on my phone using F-Droid, which only includes free as in free speech software, including DAVdroid. First download the F-Droid APK, and install it, allowing the installation of third-party applications if needed yet (Settings → Security → Device Administration → Unknown sources). Launch F-Droid, let it update the application list (or do it manually: menu → Update Repos), search for "davdroid", and install the application.

You can also install DAVdroid from the Google Play Store, in which case you will have to pay a small donation to the developers of DAVdroid. I do not encourage you to use the proprietary Google Play Store, but I do encourage you to donate directly to the developers.

Set up your server

This part of the guide is superseded by this newer post, in particular I'm no longer using port 5232.

I assume that you have a server (an always-on computer) running some Unix system, and that it is reachable from the outside by a domain name or static IP (say "example.com") where it can accept incoming connections on TCP port 5232 (i.e., if the server is behind a modem performing NAT, incoming connexions to TCP port 5232 is forwarded to this computer as identified by its local network address -- having set up if needed a static DHCP lease for the machine based on its MAC address). Of course I assume that your phone will be able to reach the server, namely, that it has a working Internet connection and that port 5232 is not filtered by your mobile operator, by your Wi-Fi access point, etc.

Install Radicale on your server -- for instance, on a Debian system, run apt-get install radicale. From here, we will not need to use administrative privileges on the server (no su, no sudo), as radicale is meant to be run by the user who wants to use it. (I do not present multi-user configuration in this guide.)

Generate the certificate

This part of the guide is superseded by this newer post where SSL is managed by Apache using Let's Encrypt certificates. With Let's Encrypt being available, free, and functional, it's probably a bad idea to generate your own certificates.

We will want to encrypt the connection between the phone and the server, because we do not want the details of your events and contacts leaking to anyone snooping on the link. To do so, we need to generate an X.509 certificate for the server. If you already have such a certificate that you use for other purposes (e.g., HTTPS), you may choose to reuse your existing certificate instead, but this may be complicated because the radicale daemon (which we will run in an unpriviledged way), will probably not have the rights to read the Web server certificate's key file. In this case you can probably arrange to run radicale as root, or with rights to access the certificate, or at least you could sign both certificates with the same CA, but I will not cover this. I will just assume that you use a fresh self-signed certificate for radicale. To generate a self-signed certificate with an RSA 2048-bit key, valid for 10 years, run:

  openssl req -x509 -nodes -days 3650 -newkey rsa:2048 \
    -keyout radicale.key -out radicale.crt

The exact values asked by OpenSSL do not really matter except for the Common Name, which should be the exact name that you will use to access your server from your phone (e.g., "example.com", with no trailing dot). According to F., providing an IP address rather than a domain name will also work. Two files will be generated: radicale.crt, which contains the public part of the certificate, and radicale.key, which contains the private key (and should remain private).

Install the certificate to your phone

This part of the guide is no longer necessary when following the newer post.

As this server certificate has not been signed, Android will refuse to connect to the server using SSL unless you specifically authorize the certificate. Transfer the radicale.crt file (not the .key file) to the /mnt/sdcard folder on your phone, i.e., copy it to your (micro-)SD card if your phone has one, or move it to the topmost folder available through MTP, or use adb (apt-get install android-tools-adb on Debian). Then, install the certificate: Settings → Security → Credential Storage → Install from storage, select the right file if prompted (or it will be automatically selected if there is only one), and give any name to the certificate.

Generate the auth file, logging file, and collections folder for Radicale

With the newer guide, part of the configuration here has changed, and there is no longer any authentication.

I will assume that security of the storage on your server is not a problem (e.g., your drive is encrypted). As the connection is also encrypted, we do not need to worry about authentication information being stored or transferred in plaintext. Hence, the simplest way to authenticate users is to put the following in a file (say passwd), adapting the values as you like:

myuser:mysecretpassword

We also need to tell radicale not to write logs at locations unavailable to unpriviledged users (e.g., not /var/log/radicale/radicale.log). To do this, you need to specify a logging policy, with a file of this sort (call it logging):

[loggers]
# Loggers names, main configuration slots
keys = root

[handlers]
# Logging handlers, defining logging output methods
keys = console

[formatters]
# Logging formatters
keys = simple,full

[logger_root]
# Root logger
level = DEBUG
handlers = console

[handler_console]
# Console handler
class = StreamHandler
level = INFO
args = (sys.stdout,)
formatter = simple

[formatter_simple]
# Simple output format
format = %(message)s

[formatter_full]
# Full output format
format = %(asctime)s - %(levelname)s: %(message)s

Finally, we also need to set up a location to store the calendar and address book data on your server. Once again, I am assuming that this location is safe (stored on an encrypted partition). Create an empty folder (call it collections), and run the following to initialize the collections (replacing the first path and "myuser" as needed -- you can also adapt the names "calendar" and "contacts" if you like):

cd /where/you/put/collections
mkdir myuser
touch myuser/calendar.ics
touch myuser/contacts.vcf

Configure Radicale

With the newer guide, part of the configuration here has changed.

Adapt the following configuration and write it to a file (say config). You can refer to the Radicale user documentation for more info. When writing file locations, always write them as absolute paths and do not use ~ as an abbreviation for your home folder:

[server]
# accept incoming connections and specify the port
hosts = 0.0.0.0:5232
# do not go to the background -- useful for debug
daemon = False
# use SSL to encrypt connections
ssl = True
# adapt the following to point to the certificate and key
certificate = /where/you/put/radicale.crt
key = /where/you/put/radicale.key
# displayed to request password on the client -- use any value
realm = radicale

[encoding]
request = utf-8
stock = utf-8

[auth]
type = htpasswd
# point to the authentication file
htpasswd_filename = /where/you/put/passwd
# no encryption on this file
htpasswd_encryption = plain
# adapt the following
private_users = myuser
public_users = myuser

[rights]
# only you have access to your connection
type = owner_only

[storage]
# store in flat files
type = filesystem
# point to the collections folder
filesystem_folder = /where/you/put/collections

[logging]
# more info for debug
debug = True
# specify the logging policy
config = /where/you/put/logging

You can now run radicale using radicale -C /where/you/put/config. If it does not work, copious debug output should be produced in the console. When it works, you need to arrange for this invocation to be run on the server whenever it is restarted -- remember it should be run unprivileged, not run by root. My current way of doing things is to run it by hand and keep it in a screen alongside offlineimap.

Yuval Levy wrote to mention that at this point you might need to create a calendar and a contact list in radicale for DAVdroid to pick them up in the next step. This can be done using, e.g., Evolution, and you should be careful when doing so to create the collections in a subfolder (e.g., "http://example.com:5232/user/addressbook/") rather than directly in the "user/" folder, otherwise DAVdroid will not see them. This should show up in your server as, e.g., "collections/user/calendar.ics" and "collections/user/contacts.vcf". According to Thomas Bartosik, a more lightweight solution is to use cadaver to create the collections. He also recommends that you put a trailing slash, like https://server:port/username/{calendar.ics,contacts.vcf}/, and also provide the trailing slash when configuring in Davdroid.

Configure DAVdroid

With the newer guide, specifying 5232 is no longer necessary.

Now you need to configure DAVdroid on your phone. Go to Settings → Accounts → Add account, select DAVdroid, and provide the following information: "https" as method, example.com:5232/myuser/ as root URL (replacing "example.com" with your server domain name and "myuser" with your user name), and providing "myuser" as user and "mysecretpassword" as password. Leave the checkbox as-is (i.e., checked), and go forward. If all goes well you should now be able to select "contacts" and "calendar" as the collections to synchronize, and confirm.

From there, all should work as intended: new contacts and calendar events created in the stock Android "Calendar" and "People" applications on your phone should be eventually synchronized to your server. You can go to Settings → Accounts → DAVdroid to synchronize manually. Check on your server that all information is indeed stored to collections/myuser/calendar.ics and collections/myuser/contacts.vcf.

Configure backups

As my contacts and calendar are important information, and as the software presented here is a bit young, I mitigate the potential risk of data loss by performing automated daily full backups of all the information stored on the server (in addition to my regular backup policy). To do this, you can use the following script (say backup.sh):

#!/bin/bash

# adapt this to point to your collections folder
COLLECTIONS="/where/you/put/collections"
# adapt to where you want to back up information
BACKUP="/where/you/want/backups"

mkdir -p "$BACKUP"
tar zcf "$BACKUP/dump-`date +%s`.tgz" "$COLLECTIONS"

Make the file executable (chmod +x backup.sh) and schedule it for daily execution at 5:00 AM by running crontab -e and adding the following line:

0 5 * * * /where/you/put/backup.sh

Run the backup.sh script to check that it works, and check that dumps are indeed generated regularly by cron.

This concludes this guide: you now have a working address book and calendar on your Android phone, synchronized to your server using free and open source software and open protocols and file formats.