In this post, I present an entirely self-hosted solution to manage your calendar and contacts on an Android phone running CyanogenMod, synchronizing to your own server using the open CalDAV and CardDAV protocols, using the free as in free speech DAVdroid program as client (along with the stock calendar and contacts application that come with Android) and Radicale as server. As of this writing I am using radicale 0.8 and davdroid 0.5.1-alpha.

To my knowledge, as of this writing, this is the only way to manage your calendar and contacts on an Android phone which satisfies the following desiderata: no dependency on closed protocols (eliminates Exchange); no dependency on a third-party server (eliminates Google Calendar); synchronization with an external machine (eliminates "local" calendars and local contacts); no dependency on proprietary applications (eliminates all the proprietary DAV software in the Google Play Store); actually functional (eliminates all the DAV clients I tried before DAVdroid).

After a few weeks of usage, the setup described here seems to work fine for me, except for occasional crashes of DAVdroid (but it restarts by itself and I have not experienced any data loss to date due to this, this is just a minor inconvenience of having to acknowledge occasionally that DAVdroid had to exit). Note that I usually only edit data from the phone, not a different DAV client (though I tested this once with Evolution and it seemed to synchronize fine), and I did not test having more than one calendar (maybe I will do so eventually, just to have different colors for different kinds of events on my phone), so I can't testify that this works for me

I have changed the server configuration: see this newer post for explanations. This update only concerns some sections of this guide, which are pointed out explicitly.

Warning about migrating existing data

Before we start, a big warning. On Android (at least on the versions I have used), contacts can be local to the phone, but events can not be local. If you have been using the default calendar application with what you think are local events, they are actually associated to some account provider (Google Calendar, Exchange, or some other application -- in my case, it used to be one of the many other buggy calendar synchronization applications), and deleting the associated account (even though it might seem unused) will delete your entire calendar. I did this once, essentially "lost" my entire calendar except for a nigh-unusable SQLite backup, and then I lost most of the data on my phone by a misguided attempt to undo this mistake by hand that messed up file ownership. Don't do that.

This being said, I don't know what would be a better way to migrate your existing events in the setup that I describe if you have been using pseudo-local events (with a dummy provider) on the stock Android applications. You can dig out the SQLite database in /data/data/com.android.providers.calendar/databases/ on the phone but that's not really helpful because I don't know of any way to convert it to something that can be imported. So well, if you used that, like me, you are screwed, and should have investigated switching costs more carefully.

For contacts, however, according to feedback from F., you can export them using the "Import-Export" functionality of the stock Android contacts application, and reimport them back once the CardDAV account has been set up, and it should work as intended.

For calendar events, according to Thomas Bartosik, there might be a solution to mass-migrate your calendar events from a local calendar to the DAVdroid one, by retrieving the file /data/data/com.android.providers.calendar/databases/calendar.db and editing it to copy the contents of the events table to itself, changing the value of the calendar_id attribute from the ID of the local calendar to that of the DAVdroid calendar (those IDs being obtained by peeking in that table). (Sorry, I won't provide the SQL command to do this because it is long and depends on the exact schema of that file.) Note that you should of course do backups before trying something like this, and I can't guarantee that this solution works as I have not tested it myself. The calendar and calendarstore should be killed on the phone once the modified database has been installed, so that the changes are picked up.

Now that you are warned, let us start with the actual guide on a more cheerful note: once the CalDAV setup described here is operational, your calendar and contacts will be synchronized with your computer and stored in the open iCalendar format, so you should be able to migrate them to something else whatever happens.

Install CyanogenMod

As CyanogenMod is discontinued, you should install, e.g., LineageOS

Not strictly necessary, what I'm describing probably works on Android too with minor adaptations, but I prefer CyanogenMod because it is more customizable and makes it possible to avoid cleanly all the proprietary Google applications. I won't describe how to install it -- if you're interested, go to the official website and follow the instructions. I'm using CyanogenMod version 10.1.3-maguro, which means a Galaxy Nexus.

Install F-Droid

I manage applications on my phone using F-Droid, which only includes free as in free speech software, including DAVdroid. First download the F-Droid APK, and install it, allowing the installation of third-party applications if needed yet (Settings → Security → Device Administration → Unknown sources). Launch F-Droid, let it update the application list (or do it manually: menu → Update Repos), search for "davdroid", and install the application.

You can also install DAVdroid from the Google Play Store, in which case you will have to pay a small donation to the developers of DAVdroid. I do not encourage you to use the proprietary Google Play Store, but I do encourage you to donate directly to the developers.

Set up your server

This part of the guide is superseded by this newer post, in particular I'm no longer using port 5232.

I assume that you have a server (an always-on computer) running some Unix system, and that it is reachable from the outside by a domain name or static IP (say "example.com") where it can accept incoming connections on TCP port 5232 (i.e., if the server is behind a modem performing NAT, incoming connexions to TCP port 5232 is forwarded to this computer as identified by its local network address -- having set up if needed a static DHCP lease for the machine based on its MAC address). Of course I assume that your phone will be able to reach the server, namely, that it has a working Internet connection and that port 5232 is not filtered by your mobile operator, by your Wi-Fi access point, etc.

Install Radicale on your server -- for instance, on a Debian system, run apt-get install radicale. From here, we will not need to use administrative privileges on the server (no su, no sudo), as radicale is meant to be run by the user who wants to use it. (I do not present multi-user configuration in this guide.)

Generate the certificate

This part of the guide is superseded by this newer post where SSL is managed by Apache using Let's Encrypt certificates. With Let's Encrypt being available, free, and functional, it's probably a bad idea to generate your own certificates.

We will want to encrypt the connection between the phone and the server, because we do not want the details of your events and contacts leaking to anyone snooping on the link. To do so, we need to generate an X.509 certificate for the server. If you already have such a certificate that you use for other purposes (e.g., HTTPS), you may choose to reuse your existing certificate instead, but this may be complicated because the radicale daemon (which we will run in an unpriviledged way), will probably not have the rights to read the Web server certificate's key file. In this case you can probably arrange to run radicale as root, or with rights to access the certificate, or at least you could sign both certificates with the same CA, but I will not cover this. I will just assume that you use a fresh self-signed certificate for radicale. To generate a self-signed certificate with an RSA 2048-bit key, valid for 10 years, run:

  openssl req -x509 -nodes -days 3650 -newkey rsa:2048 \
    -keyout radicale.key -out radicale.crt

The exact values asked by OpenSSL do not really matter except for the Common Name, which should be the exact name that you will use to access your server from your phone (e.g., "example.com", with no trailing dot). According to F., providing an IP address rather than a domain name will also work. Two files will be generated: radicale.crt, which contains the public part of the certificate, and radicale.key, which contains the private key (and should remain private).

Install the certificate to your phone

This part of the guide is no longer necessary when following the newer post.

As this server certificate has not been signed, Android will refuse to connect to the server using SSL unless you specifically authorize the certificate. Transfer the radicale.crt file (not the .key file) to the /mnt/sdcard folder on your phone, i.e., copy it to your (micro-)SD card if your phone has one, or move it to the topmost folder available through MTP, or use adb (apt-get install android-tools-adb on Debian). Then, install the certificate: Settings → Security → Credential Storage → Install from storage, select the right file if prompted (or it will be automatically selected if there is only one), and give any name to the certificate.

Generate the auth file, logging file, and collections folder for Radicale

With the newer guide, part of the configuration here has changed, and there is no longer any authentication.

I will assume that security of the storage on your server is not a problem (e.g., your drive is encrypted). As the connection is also encrypted, we do not need to worry about authentication information being stored or transferred in plaintext. Hence, the simplest way to authenticate users is to put the following in a file (say passwd), adapting the values as you like:

myuser:mysecretpassword

We also need to tell radicale not to write logs at locations unavailable to unpriviledged users (e.g., not /var/log/radicale/radicale.log). To do this, you need to specify a logging policy, with a file of this sort (call it logging):

[loggers]
# Loggers names, main configuration slots
keys = root

[handlers]
# Logging handlers, defining logging output methods
keys = console

[formatters]
# Logging formatters
keys = simple,full

[logger_root]
# Root logger
level = DEBUG
handlers = console

[handler_console]
# Console handler
class = StreamHandler
level = INFO
args = (sys.stdout,)
formatter = simple

[formatter_simple]
# Simple output format
format = %(message)s

[formatter_full]
# Full output format
format = %(asctime)s - %(levelname)s: %(message)s

Finally, we also need to set up a location to store the calendar and address book data on your server. Once again, I am assuming that this location is safe (stored on an encrypted partition). Create an empty folder (call it collections), and run the following to initialize the collections (replacing the first path and "myuser" as needed -- you can also adapt the names "calendar" and "contacts" if you like):

cd /where/you/put/collections
mkdir myuser
touch myuser/calendar.ics
touch myuser/contacts.vcf

Configure Radicale

With the newer guide, part of the configuration here has changed.

Adapt the following configuration and write it to a file (say config). You can refer to the Radicale user documentation for more info. When writing file locations, always write them as absolute paths and do not use ~ as an abbreviation for your home folder:

[server]
# accept incoming connections and specify the port
hosts = 0.0.0.0:5232
# do not go to the background -- useful for debug
daemon = False
# use SSL to encrypt connections
ssl = True
# adapt the following to point to the certificate and key
certificate = /where/you/put/radicale.crt
key = /where/you/put/radicale.key
# displayed to request password on the client -- use any value
realm = radicale

[encoding]
request = utf-8
stock = utf-8

[auth]
type = htpasswd
# point to the authentication file
htpasswd_filename = /where/you/put/passwd
# no encryption on this file
htpasswd_encryption = plain
# adapt the following
private_users = myuser
public_users = myuser

[rights]
# only you have access to your connection
type = owner_only

[storage]
# store in flat files
type = filesystem
# point to the collections folder
filesystem_folder = /where/you/put/collections

[logging]
# more info for debug
debug = True
# specify the logging policy
config = /where/you/put/logging

You can now run radicale using radicale -C /where/you/put/config. If it does not work, copious debug output should be produced in the console. When it works, you need to arrange for this invocation to be run on the server whenever it is restarted -- remember it should be run unprivileged, not run by root. My current way of doing things is to run it by hand and keep it in a screen alongside offlineimap.

Yuval Levy wrote to mention that at this point you might need to create a calendar and a contact list in radicale for DAVdroid to pick them up in the next step. This can be done using, e.g., Evolution, and you should be careful when doing so to create the collections in a subfolder (e.g., "http://example.com:5232/user/addressbook/") rather than directly in the "user/" folder, otherwise DAVdroid will not see them. This should show up in your server as, e.g., "collections/user/calendar.ics" and "collections/user/contacts.vcf". According to Thomas Bartosik, a more lightweight solution is to use cadaver to create the collections. He also recommends that you put a trailing slash, like https://server:port/username/{calendar.ics,contacts.vcf}/, and also provide the trailing slash when configuring in Davdroid.

Configure DAVdroid

With the newer guide, specifying 5232 is no longer necessary.

Now you need to configure DAVdroid on your phone. Go to Settings → Accounts → Add account, select DAVdroid, and provide the following information: "https" as method, example.com:5232/myuser/ as root URL (replacing "example.com" with your server domain name and "myuser" with your user name), and providing "myuser" as user and "mysecretpassword" as password. Leave the checkbox as-is (i.e., checked), and go forward. If all goes well you should now be able to select "contacts" and "calendar" as the collections to synchronize, and confirm.

From there, all should work as intended: new contacts and calendar events created in the stock Android "Calendar" and "People" applications on your phone should be eventually synchronized to your server. You can go to Settings → Accounts → DAVdroid to synchronize manually. Check on your server that all information is indeed stored to collections/myuser/calendar.ics and collections/myuser/contacts.vcf.

Configure backups

As my contacts and calendar are important information, and as the software presented here is a bit young, I mitigate the potential risk of data loss by performing automated daily full backups of all the information stored on the server (in addition to my regular backup policy). To do this, you can use the following script (say backup.sh):

#!/bin/bash

# adapt this to point to your collections folder
COLLECTIONS="/where/you/put/collections"
# adapt to where you want to back up information
BACKUP="/where/you/want/backups"

mkdir -p "$BACKUP"
tar zcf "$BACKUP/dump-`date +%s`.tgz" "$COLLECTIONS"

Make the file executable (chmod +x backup.sh) and schedule it for daily execution at 5:00 AM by running crontab -e and adding the following line:

0 5 * * * /where/you/put/backup.sh

Run the backup.sh script to check that it works, and check that dumps are indeed generated regularly by cron.

This concludes this guide: you now have a working address book and calendar on your Android phone, synchronized to your server using free and open source software and open protocols and file formats.