CalDAV and CardDAV with Radicale and DAVdroid on Android
In this post, I present an entirely self-hosted solution to manage your calendar and contacts on an Android phone running CyanogenMod, synchronizing to your own server using the open CalDAV and CardDAV protocols, using the free as in free speech DAVdroid program as client (along with the stock calendar and contacts application that come with Android) and Radicale as server. As of this writing I am using radicale 0.8 and davdroid 0.5.1-alpha.
To my knowledge, as of this writing, this is the only way to manage your calendar and contacts on an Android phone which satisfies the following desiderata: no dependency on closed protocols (eliminates Exchange); no dependency on a third-party server (eliminates Google Calendar); synchronization with an external machine (eliminates "local" calendars and local contacts); no dependency on proprietary applications (eliminates all the proprietary DAV software in the Google Play Store); actually functional (eliminates all the DAV clients I tried before DAVdroid).
After a few weeks of usage, the setup described here seems to work fine for me, except for occasional crashes of DAVdroid (but it restarts by itself and I have not experienced any data loss to date due to this, this is just a minor inconvenience of having to acknowledge occasionally that DAVdroid had to exit). Note that I usually only edit data from the phone, not a different DAV client (though I tested this once with Evolution and it seemed to synchronize fine), and I did not test having more than one calendar (maybe I will do so eventually, just to have different colors for different kinds of events on my phone), so I can't testify that this works for me
I have changed the server configuration: see this newer post for explanations. This update only concerns some sections of this guide, which are pointed out explicitly.
Warning about migrating existing data
Before we start, a big warning. On Android (at least on the versions I have used), contacts can be local to the phone, but events can not be local. If you have been using the default calendar application with what you think are local events, they are actually associated to some account provider (Google Calendar, Exchange, or some other application -- in my case, it used to be one of the many other buggy calendar synchronization applications), and deleting the associated account (even though it might seem unused) will delete your entire calendar. I did this once, essentially "lost" my entire calendar except for a nigh-unusable SQLite backup, and then I lost most of the data on my phone by a misguided attempt to undo this mistake by hand that messed up file ownership. Don't do that.
This being said, I don't know what would be a better way to migrate your
existing events in the setup that I describe if you have been using
pseudo-local events (with a dummy provider) on the stock
Android applications. You can dig out the SQLite database in
/data/data/com.android.providers.calendar/databases/ on the phone
but that's not really helpful because I don't know of any way to convert it to
something that can be imported. So well, if you used that, like me, you are
screwed, and should have investigated switching costs more
carefully.
For contacts, however, according to feedback from F., you can export them using the "Import-Export" functionality of the stock Android contacts application, and reimport them back once the CardDAV account has been set up, and it should work as intended.
For calendar events, according to Thomas Bartosik, there might be a
solution to mass-migrate your calendar events from a local calendar to the
DAVdroid one, by retrieving the file
/data/data/com.android.providers.calendar/databases/calendar.db and
editing it to copy the contents of the events table to itself,
changing the value of the calendar_id attribute from the ID of the
local calendar to that of the DAVdroid calendar (those IDs being obtained by
peeking in that table). (Sorry, I won't provide the SQL command to do this
because it is long and depends on the exact schema of that file.) Note that you
should of course do backups before trying something like this, and I can't
guarantee that this solution works as I have not tested it
myself. The calendar and calendarstore should be killed on the phone once the
modified database has been installed, so that the changes are picked up.
Now that you are warned, let us start with the actual guide on a more cheerful note: once the CalDAV setup described here is operational, your calendar and contacts will be synchronized with your computer and stored in the open iCalendar format, so you should be able to migrate them to something else whatever happens.
Install CyanogenMod
As CyanogenMod is discontinued, you should install, e.g., LineageOS
Not strictly necessary, what I'm describing probably works on Android too with minor adaptations, but I prefer CyanogenMod because it is more customizable and makes it possible to avoid cleanly all the proprietary Google applications. I won't describe how to install it -- if you're interested, go to the official website and follow the instructions. I'm using CyanogenMod version 10.1.3-maguro, which means a Galaxy Nexus.
Install F-Droid
I manage applications on my phone using F-Droid, which only includes free as in free speech software, including DAVdroid. First download the F-Droid APK, and install it, allowing the installation of third-party applications if needed yet (Settings → Security → Device Administration → Unknown sources). Launch F-Droid, let it update the application list (or do it manually: menu → Update Repos), search for "davdroid", and install the application.
You can also install DAVdroid from the Google Play Store, in which case you will have to pay a small donation to the developers of DAVdroid. I do not encourage you to use the proprietary Google Play Store, but I do encourage you to donate directly to the developers.
Set up your server
This part of the guide is superseded by this newer post, in particular I'm no longer using port 5232.
I assume that you have a server (an always-on computer) running some Unix system, and that it is reachable from the outside by a domain name or static IP (say "example.com") where it can accept incoming connections on TCP port 5232 (i.e., if the server is behind a modem performing NAT, incoming connexions to TCP port 5232 is forwarded to this computer as identified by its local network address -- having set up if needed a static DHCP lease for the machine based on its MAC address). Of course I assume that your phone will be able to reach the server, namely, that it has a working Internet connection and that port 5232 is not filtered by your mobile operator, by your Wi-Fi access point, etc.
Install Radicale on your server -- for
instance, on a Debian system, run apt-get
install radicale. From here, we will not need to use administrative
privileges on the server (no su, no sudo), as radicale
is meant to be run by the user who wants to use it. (I do not present multi-user
configuration in this guide.)
Generate the certificate
This part of the guide is superseded by this newer post where SSL is managed by Apache using Let's Encrypt certificates. With Let's Encrypt being available, free, and functional, it's probably a bad idea to generate your own certificates.
We will want to encrypt the connection between the phone and the server, because we do not want the details of your events and contacts leaking to anyone snooping on the link. To do so, we need to generate an X.509 certificate for the server. If you already have such a certificate that you use for other purposes (e.g., HTTPS), you may choose to reuse your existing certificate instead, but this may be complicated because the radicale daemon (which we will run in an unpriviledged way), will probably not have the rights to read the Web server certificate's key file. In this case you can probably arrange to run radicale as root, or with rights to access the certificate, or at least you could sign both certificates with the same CA, but I will not cover this. I will just assume that you use a fresh self-signed certificate for radicale. To generate a self-signed certificate with an RSA 2048-bit key, valid for 10 years, run:
openssl req -x509 -nodes -days 3650 -newkey rsa:2048 \
-keyout radicale.key -out radicale.crt
The exact values asked by OpenSSL do not really matter except for the Common
Name, which should be the exact name that you will use to access your
server from your phone (e.g., "example.com", with no trailing dot). According to F., providing
an IP address rather than a domain name will also work. Two files
will be generated: radicale.crt, which contains the public part of
the certificate, and radicale.key, which contains the private key
(and should remain private).
Install the certificate to your phone
This part of the guide is no longer necessary when following the newer post.
As this server certificate has not been signed, Android will refuse to
connect to the server using SSL unless you specifically authorize the
certificate. Transfer the radicale.crt file (not the
.key file) to the /mnt/sdcard folder on your phone,
i.e., copy it to your (micro-)SD card if your phone has one, or move it to the
topmost folder available through MTP, or use adb (apt-get
install android-tools-adb on Debian). Then, install the certificate:
Settings → Security → Credential Storage → Install from storage, select the
right file if prompted (or it will be automatically selected if there is only
one), and give any name to the certificate.
Generate the auth file, logging file, and collections folder for Radicale
With the newer guide, part of the configuration here has changed, and there is no longer any authentication.
I will assume that security of the storage on your server is not a problem
(e.g., your drive is encrypted). As the connection is also encrypted, we do not
need to worry about authentication information being stored or transferred in
plaintext. Hence, the simplest way to authenticate users is to put the following
in a file (say passwd), adapting the values as you like:
myuser:mysecretpassword
We also need to tell radicale not to write logs at locations unavailable to
unpriviledged users (e.g., not /var/log/radicale/radicale.log). To
do this, you need to specify a logging policy, with a file of this sort (call it
logging):
[loggers] # Loggers names, main configuration slots keys = root [handlers] # Logging handlers, defining logging output methods keys = console [formatters] # Logging formatters keys = simple,full [logger_root] # Root logger level = DEBUG handlers = console [handler_console] # Console handler class = StreamHandler level = INFO args = (sys.stdout,) formatter = simple [formatter_simple] # Simple output format format = %(message)s [formatter_full] # Full output format format = %(asctime)s - %(levelname)s: %(message)s
Finally, we also need to set up a location to store the calendar and
address book data on your server. Once again, I am assuming that this location
is safe (stored on an encrypted partition). Create an empty folder (call it
collections), and run the following to initialize the collections
(replacing the first path and "myuser" as needed -- you can also adapt the names
"calendar" and "contacts" if you like):
cd /where/you/put/collections mkdir myuser touch myuser/calendar.ics touch myuser/contacts.vcf
Configure Radicale
With the newer guide, part of the configuration here has changed.
Adapt the following configuration and write it to a file (say
config). You can refer to the Radicale user documentation
for more info. When writing file locations, always write them as absolute paths
and do not use ~ as an abbreviation for your home
folder:
[server] # accept incoming connections and specify the port hosts = 0.0.0.0:5232 # do not go to the background -- useful for debug daemon = False # use SSL to encrypt connections ssl = True # adapt the following to point to the certificate and key certificate = /where/you/put/radicale.crt key = /where/you/put/radicale.key # displayed to request password on the client -- use any value realm = radicale [encoding] request = utf-8 stock = utf-8 [auth] type = htpasswd # point to the authentication file htpasswd_filename = /where/you/put/passwd # no encryption on this file htpasswd_encryption = plain # adapt the following private_users = myuser public_users = myuser [rights] # only you have access to your connection type = owner_only [storage] # store in flat files type = filesystem # point to the collections folder filesystem_folder = /where/you/put/collections [logging] # more info for debug debug = True # specify the logging policy config = /where/you/put/logging
You can now run radicale using radicale -C
/where/you/put/config. If it does not work, copious debug output should
be produced in the console. When it works, you need to arrange for this
invocation to be
run on the server whenever it is restarted -- remember it should be run
unprivileged, not run by root. My current way of doing things is to run it by
hand and keep it in a screen
alongside offlineimap.
Yuval Levy wrote to mention that at this point you might need
to create a calendar and a contact list in radicale for DAVdroid to pick them up
in the next step. This can be done using, e.g., Evolution, and you should be
careful when doing so to create the collections in a subfolder (e.g.,
"http://example.com:5232/user/addressbook/") rather than directly in the "user/"
folder, otherwise DAVdroid will not see them. This should show up in your server
as, e.g., "collections/user/calendar.ics" and
"collections/user/contacts.vcf". According to Thomas Bartosik, a more lightweight solution is to use cadaver to create the collections. He also recommends that
you put a trailing slash, like
https://server:port/username/{calendar.ics,contacts.vcf}/, and also
provide the trailing slash when configuring in Davdroid.
Configure DAVdroid
With the newer guide, specifying 5232 is no longer necessary.
Now you need to configure DAVdroid on your phone. Go to Settings → Accounts →
Add account, select DAVdroid, and provide the following information: "https" as
method, example.com:5232/myuser/ as root URL (replacing
"example.com" with your server domain name and "myuser" with your user name),
and providing "myuser" as user and "mysecretpassword" as password. Leave the
checkbox as-is (i.e., checked), and go forward. If all goes well you should now
be able to select "contacts" and "calendar" as the collections to
synchronize, and confirm.
From there, all should work as intended: new contacts and calendar events
created in the stock Android "Calendar" and "People" applications on your phone
should be eventually synchronized to your server. You can go to Settings →
Accounts → DAVdroid to synchronize manually. Check on your server that all
information is indeed stored to collections/myuser/calendar.ics and
collections/myuser/contacts.vcf.
Configure backups
As my contacts and calendar are important information, and as the software
presented here is a bit young, I mitigate the potential risk of data loss by
performing automated daily full backups of all the information stored on the
server (in addition to my regular backup policy). To do this, you can use the
following script (say backup.sh):
#!/bin/bash # adapt this to point to your collections folder COLLECTIONS="/where/you/put/collections" # adapt to where you want to back up information BACKUP="/where/you/want/backups" mkdir -p "$BACKUP" tar zcf "$BACKUP/dump-`date +%s`.tgz" "$COLLECTIONS"
Make the file executable (chmod +x backup.sh) and schedule it
for daily execution at 5:00 AM by running crontab -e and adding the
following line:
0 5 * * * /where/you/put/backup.sh
Run the backup.sh script to check that it works, and check that
dumps are indeed generated regularly by cron.
This concludes this guide: you now have a working address book and calendar on your Android phone, synchronized to your server using free and open source software and open protocols and file formats.