Tutorial – CalDAV/CardDAV

If you keep a digital calendar or address book, you want your data to be stored in one central location and accessible from any device, wherever and whenever you need it. You also want ownership of your data and metadata. By using free and open source software (FOSS), you can create calendars and address books in a private cloud that allows you to synchronize and share that data, without being locked into some corporate, data-harvesting walled garden.

In this tutorial, I will explain the open standards, CalDAV and CardDAV, that make independent storing and sharing of calendar and address book data possible. Then, I will show you how to automatically import or export calendars and contacts, from any source, to a Nextcloud instance, process that data, and migrate it to another server. Finally, I will outline how to set up your own standalone calendar and address book.

CalDAV/CardDAV

CalDAV and CardDAV, the open standards that allow centralized storage and management of personal data, are both supersets of the Web Distributed Authoring and Versioning (WebDAV) system. WebDAV's specification describes how software programs can edit remote content over the Internet, using the same protocol (HTTP) that browsers use to load pages from websites. CalDav is the WebDAV extension for calendars, and CardDAV is the extension for personal contacts.

Both CalDAV and CardDAV are client-server protocols: They let many users, each with their own interfaces, simultaneously access the same set of events or contacts stored on a common server.

At the lower level, single events or whole calendars are stored inside files with the iCalendar extension .ics (also .ical or .icalender) [1]. Files that contain address books, instead, have the Virtual Contact File extension .vcf [2].

Both .ics (Listing 1) and .vcf (Listing 2) files use plain text formats with a relatively simple syntax that could be written manually with any text editor. The calendar event in Listing 1 shows an annual all staff meeting that must occur from February 12 to 13 (2008 in this example), as described in the RRULE (recurrence rule) field. The DESCRIPTION variable describes the meeting's purpose (a project status checkup) and the GEO variable gives the meeting's location. The whole event is enclosed by BEGIN:VEVENT and END:VEVENT tags, inside a VCALENDAR that may contain other events both before or after this event.

BEGIN:VCALENDAR
BEGIN:VEVENT
SUMMARY:All Staff Meeting
STATUS:CONFIRMED
RRULE:FREQ=YEARLY;INTERVAL=1;BYMONTH=2;BYMONTHDAY=12
DTSTART:20080212
DTEND:20080213
CATEGORIES:teamwork, project management
GEO:87.5739497;-45.7399606
DESCRIPTION: project status check-up
END:VEVENT
END:VCALENDAR

BEGIN:VCARD
VERSION:4.0
FN:Bruce Wayne
ADR;TYPE=WORK:;;;Gotham City;USA;;Planet Earth
EMAIL;TYPE=HOME:batman@example.com
TEL;TYPE="HOME,VOICE":555-5555
URL:https://batman.example.com
ORG:Billionaire
TITLE:Dark Knight
END:VCARD

Besides VEVENTs, an .ics calendar may also store to-do items (VTODO), journal entries (VJOURNAL), and time zone information (VTIMEZONE). Rich text in HTML format can be specified by the parameter X-ALT-DESC.

Both .ics and .vcf files also contain unique identifiers (UID) that allow the servers to index them. On the host computers, the files are stored in folders with a defined structure called vdir. Each vdir corresponds to a different calendar or address book. While it is possible to create, delete, or edit a vdir's contents, I recommend using a proper CalDAV or CardDAV client to prevent corruption of vdir's indexing.

Nextcloud

In this tutorial, I use Nextcloud [3], which is currently the most promising solution for open source, self-hosted cloud services. To install Nextcloud and understand its importance, please see my blog [4].

Under the hood, both the calendars and contacts of all of a Nextcloud instance's users are managed by an embedded CalDAV/CardDAV server.

Figures 1 and 2 show details of the address book and calendar interfaces that I created for a test Nextcloud account. In Figure 1, you can see the data that Nextcloud saved in Listing 2.

Figure 1: Nextcloud's Contacts app is the interface to its own CardDAV server.

I describe the Nextcloud GUIs shown in Figures 1 and 2 in more detail in my blog [4]. However, there are at least two instances when there is a better way to process this personal data: The first is mass migration to or from another server or bulk export to some other database; the second is whenever you want to simultaneously modify contacts or calendars. Especially in the second instance, it can be much faster to make local copies of all the records, modify them with a script, and then re-upload everything to the server. The tools described in this tutorial can be used for all these tasks.

vdirsyncer

Theoretically, the simplest way to process automatically a CalDAV or CardDAV server's contents would be a command-line client that can connect directly to that server. In practice, I have found an alternate approach that seems easier to implement with standard tools and even more flexible: Copy the server data on your computer, change as needed those copies, and then sync the vdirs on the server with those on your local archive. This approach also has the advantage of keeping contacts and events accessible even when the server is unreachable. I use vdirsyncer [5] to perform the synchronization.

In addition to installing manually, you can find vdirsyncer binary packages for several distributions, but not in the usual places. On my Ubuntu 19.10 desktop, for instance, I had to install from a custom repository with the sequence of commands shown in Listing 3, which I found in vdirsyncer's very complete manual [6].

01 curl -s https://packagecloud.io/install/repositories/pimutils/vdirsyncer/script.deb.sh | sudo bash
02 sudo apt-get update
03 sudo apt-get install vdirsyncer-latest
04 export PATH="$PATH:/opt/venvs/vdirsyncer-latest/bin"

The command in line 1 of Listing 3 downloads and executes a shell script that fetches data for the custom repository and adds them to the Ubuntu package database. Line 3 is what actually installs vdirsyncer on your system. Line 4 tells Ubuntu to add the vdirsyncer executable's folder (/opt/...) to the list of folders ($PATH) where programs are stored.

Once vdirsyncer is installed, it needs to know which vdirs it should sync and their location. You do this with configuration files like vdirsyncer-nextcloud.conf (Listing 4).

01 [general]
02 status_path = "~/.vdirsyncer/status/"
03 [pair contacts_nextcloud_to_local]
04 a = "my_contacts_local"
05 b = "my_contacts_nextcloud"
06 collections = ["from a", "from b"]
07 [storage my_contacts_local]
08 type = "filesystem"
09 path = "~/.contacts/"
10 fileext = ".vcf"
11 [storage my_contacts_nextcloud]
12 type = "carddav"
13 url = https://full-url-of-my-private-nextcloud-instance/"
14 username = "nextcloud_test_user"
15 password = "nextcloud_test_password"
16
17 [pair cal_nextcloud_to_local]
18 a = "my_cal_local"
19 b = "my_cal_nextcloud"
20 collections = ["from a", "from b"]
21 [storage my_cal_local]
22 type = "filesystem"
23 path = "~/.calendars/"
24 fileext = ".ics"
25 [storage my_cal_nextcloud]
26 type = "caldav"
27 url = https://full-url-of-my-private-nextcloud-instance/"
28 username = "nextcloud_test_user"
29 password = "nextcloud_test_password"

Lines 1 and 2 of Listing 4 tell vdirsyncer where to store the synchronization status of all the vdirs it manages.

Lines 3 to 15 define a pair of address books to sync. The address book on your local computer, my_contacts_local (aliased as a), is stored in normal folders located inside the ~/.contacts directory and contains .vcf files. The Nextcloud remote address book (lines 11 to 15) is accessible with the CardDAV protocol at the URL shown on line 13, with the credentials shown in lines 14 and 15. Line 6 means that vdirsyncer must sync all the collections in each pair of vdirs.

The only difference in lines 17 to 29, visible in the variable names and file extensions, is that this section tells vdirsyncer to sync two sets of calendars instead of address books.

After the initial configuration, but before the actual synchronization, you need to make vdirsyncer "discover" all the data on the remote server (unless you configured to only sync one calendar, as explained in the documentation). Listing 5 shows how to do this plus includes an excerpt of the output.

01 #> vdirsyncer -c vdirsyncer-nextcloud.conf discover
02 Discovering collections for pair contacts_nextcloud_to_local
03 my_contacts_local:
04   - "contacts"
05 my_contacts_nextcloud:
06   - "contacts" ("Contacts")
07 Saved for contacts_nextcloud_to_local: collections = ["contacts"]
08 Discovering collections for pair cal_nextcloud_to_local
09 my_cal_local:
10 my_cal_nextcloud:
11   - "work" ("Work")
12 .....
13 warning: No collection "work" found for storage my_cal_local.
14 Should vdirsyncer attempt to create it? [y/N]: y
15 Saved for cal_nextcloud_to_local: collections = ["work"]

The output means that everything went well, and vdirsyncer discovered everything it was supposed to find. Passing the configuration file with the -c option is necessary when you want to use vdirsyncer with different servers. The actual synchronization is done with the following single command (which you could make a regular cron job):

#> vdirsyncer -c vdirsyncer-nextcloud.conf sync

Vdirsyncer has many other options and uses. Due to space constraints, I will only discuss how to handle conflicts. If two calendars contain two different records for the same event, you must decide which calendar is the master that should win such conflicts and explicitly tell vdirsyncer. (Listing 5), if the Nextcloud calendar is the master, you should add a line like this:

conflict_resolution = "b wins"

right after the lines 6 and 20 of Listing 5. In addition to the vdirsyncer manual [6], see [7] for more conflict resolution examples and vdirsyncer configuration tips.

khal and khard

In addition to vdirsyncer, two other tools, khard [8] for address books and khal [9] for calendars, offer a simple solution for this tutorial. Khard is the most essential. Khal can be used interactively: Figure 3 displays the same calendar shown in Figure 2 read from the local copy previously created by vdirsyncer. Both khard and khal can be installed from the standard repositories of major Linux distributions.

Figure 2: Part of a Nextcloud calendar in monthly mode shows events entered from its own graphical interface.

Figure 3: The command-line khal calendar, showing a local copy of the same calendar shown in Figure 2.

Khal's configuration can be as simple as Listing 6: The calendars section lists all the desired calendars, while the default section sets the options, most with self-explanatory names, that are common to all calendars: default calendar, number of days to display (timedelta), highlighting, and date formatting.

[calendars]
    [[work]]
    path = ~/.pim/work
    type = calendar
    color = dark green
    [[personal]]
    path = ~/.pim/personal
    type = calendar
    color = yellow
[default]
default_calendar = work
highlight_event_days = true
timedelta = 15d
#Dates formatting
[locale]
local_timezone= Europe/Rome
firstweekday = 0
timeformat = %H:%M
dateformat = %Y/%m/%d
longdateformat = %Y/%m/%d
datetimeformat = %Y/%m/%d %H:%M
longdatetimeformat = %Y/%m/%d %H:%M

Khard's configuration (Listing 7) can be almost as simple, if you only use it for importing bulk contacts or as an address grabber for console email clients like Mutt (more on this later).

[addressbooks]
[[All]]
path = ~/.pim/contacts
[general]
debug = no
default_action = list
editor = gedit
merge_editor = /usr/share/doc/khard/examples/sdiff/sdiff_khard_wrapper.sh
[contact table]
display = first_name
group_by_addressbook = no
sort = last_name

The editor and merge_editor options need some additional explanation. If these options do not point to existing programs, khard will not start. In Listing 7, the merge_editor used to handle conflicts is the one included in the khard package.

If you launch khard without options, it will just list all the contacts it found. In my case, the contacts shown in Listing 8 are the same ones that I had created in Nextcloud.

#> khard
Address book: All
Index Name        Phone                E-Mail                   UID
1     Bruce Wayne HOME,VOICE: 555-5555 HOME: batman@example.com b
2     Superman    HOME,VOICE:          HOME:                    3

You can add new contacts on the command line. For the default address book, just type

#> khard new

and answer khard's questions. If you need to work on a non-default address book, specify it with the -a switch. It is also possible to configure Mutt to add or fetch addresses from khard, with the following two instructions that are explained in detail in the khard documentation [10]:

macro pager,index a "<pipe-message>khard add-email<return>" "add the sender address to khard"
set query_command= "khard email --parsable %s"

Khard and khal are very simple, very quick ways to read or write personal data from a Linux terminal. Their real power, however, is their support for automatic import and export of data to or from Nextcloud or any other CalDAV/CardDAV server. To load two new events into my Nextcloud work calendar (Figure 4) I used the following commands, and then ran vdirsyncer again:

khal -c khal-config-local.conf  new -a work '2020/03/24 09:00' '2020/03/24 11:30' 'Engineers meeting'
khal -c khal-config-local.conf  new -a work '2020/03/25 09:00' '2020/03/25 18:00' 'Project meeting'

Figure 4: After synchronization, events created with khal on the desktop also display in the Nextcloud interface.

With khal, you can add hundreds of events to multiple calendars with a shell script that reads the events to be added from a plain text file. The plain text file may be a spreadsheet in CSV format, a database dump, or anything else, as long as it stores one event per line in a format similar to:

Engineers meeting|work|2020/03/24 09:00|2020/03/24 11:30

A script would need to read that file one line at a time, assigning each pipe-separated field to a variable (e.g., in the example above, $EVENT, $CALENDAR, $START, and $END) and then launch khal as follows, once per line:

khal -c $CONFIG_FILE new -a $CALENDAR $START $END $EVENT

In a real script, the above command should be modified to handle spaces inside each variable. But once the input file has been read, one call to vdirsyncer will copy all those events to the Nextcloud calendar.

Importing addresses with khard works in the same way, with one important difference: With khard, you must write all the contact data in a template generated by khard with the following command:

khard -c khard.conf show --format=yaml > template.yaml

Once you have the template, you can replace the value of all parameters with the right ones for your contact inside the file. Listing 9 shows the content of the template.yaml file, when filled with Wonder Woman's contact information; Figure 5 shows the corresponding contact in the Nextcloud address book, after running the following khard command:

khard -c khard.conf new -i template.yaml

Prefix     : Princess
First name : Diana
Additional :
Last name  : Themishira
Suffix     :
Nickname :
Anniversary :
Birthday :
Organisation : League of Justice
Title :
Role  :
Phone :
    HOME,VOICE : 555-5556
Email :
    HOME : wonderwoman@example.com
Address :
    WORK:
        Box      :
        Extended :
        Street   :
        Code     :
        City     :
        Region   : Mount Olympus
        Country  : Planet Earth
Categories :
Webpage : https://en.wikipedia.org/wiki/Wonder_Woman
Private :
    Jabber  :
    Skype   :
    Twitter : wonderwoman_official
Note :

Figure 5: Wonder Woman's contact information displayed in the Nextcloud Contacts app.

Just like with khal, you can use the khard command above inside a loop, which fills the template with new values read from a plain text file at every iteration and calls vdirsyncer when it has finished.

Backups and Post-Processing

As a side benefit of vdirsyncer, you can maintain a full backup of all your CalDAV/CardDAV data, regardless of what you do with it. However, that is not the only way to export or back up that data, and sometimes it is not even the best one. If you want to back up all of the personal data of all your Nextcloud installation's users, for example, you can run the Bash script, calcardbackup [11], on the server as follows:

sudo -u www-data /PATH/TO/calcardbackup $NEXTCLOUD_ROOT_FOLDER

The result will be a tar archive containing several .ics or .vcf files with names like USERNAME-CALENDARNAME, with each file containing one whole calendar or address book. The calcardbackup website also explains how to compress or encrypt backups or just back up the data of selected users.

Personally, I use calcardbackup any time I need to export all my calendar events, in a simpler CSV format, which is much better suited for further processing. To do that, I wrote a very quick and dirty script called ics2csv.pl (Listing 10). Listing 11 shows how to run the script, and its resulting output.

01 #! /usr/bin/perl
02    use strict;
03 use POSIX 'strftime';
04
05 my $TODAY = strftime '%Y%m%d', localtime;
06 my ($CALENDAR, $SUMMARY, $START, $STARTDAY, $END, $ENDDAY, $STARTHOUR, $ENDHOUR);
07 my @APPOINTMENTS;
08
09 print "## All events starting from $TODAY\n\n";
10
11 while (<>) {
12 next unless ($_ =~ m/^(X-WR-CALNAME|SUMMARY|DTSTART|DTEND|END:VEVENT)/);
13
14 s/;VALUE=DATE//;
15 s/;TZID=Europe\/Rome//;
16
17 if ($_ =~ m/^X-WR-CALNAME:(.*)\r\n/)   { $CALENDAR = $1;   $CALENDAR = lc($CALENDAR)}
18
19 if ($_ =~ m/^SUMMARY:(.*)\r\n/){ $SUMMARY  = $1; }
20
21 if ($_ =~ m/^DTSTART:(\w*)/) {
22   $START = $1;
23   $STARTDAY  = substr($START, 0,8);
24   $STARTHOUR = substr($START, 9);
25   substr($STARTDAY,4,0) = '-';
26   substr($STARTDAY,7,0) = '-';
27   if (length($STARTHOUR) > 0) {
28   substr($STARTHOUR,2,0) = ':';
29   substr($STARTHOUR,-2) = '';
30   }
31 }
32
33 if ($_ =~ m/^DTEND:(\w*)/)   {
34   $END = $1;
35   $ENDDAY  = substr($END, 0,8);
36   $ENDHOUR = substr($END, 9);
37   substr($ENDDAY,4,0) = '-';
38   substr($ENDDAY,7,0) = '-';
39   if (length($ENDHOUR) > 0) {
40   substr($ENDHOUR,2,0) = ':';
41   substr($ENDHOUR,-2) = '';
42   }
43 }
44
45 if ($_ =~ m/^END:VEVENT/)   {
46
47   push (@APPOINTMENTS,sprintf("%10.10s  %5.5s | %-10.10s | %-35.35s | %10.10s  %5.5s\n", $STARTDAY,$STARTHOUR,  $CALENDAR, $SUMMARY,$ENDDAY, $ENDHOUR));
48 }
49 }
50
51 for my $APP (sort @APPOINTMENTS) {
52 my $CURRENTDAY = substr($APP,0,10);
53 $CURRENTDAY=~ s/-//g;
54 print "$APP\n" if ($CURRENTDAY >= $TODAY);
55 }

#> ics2csv.pl  calcardbackup-2030-03-01/marco-work.ics > calendar.csv
#> cat calendar.csv
## All events starting from 20200301
2020-03-28 | work | FOSSMeet talk            | 2020-03-29
2020-03-30 | work | Linuxmag: caldav servers | 2020-03-31

Listing 10 shows the power of storing your personal data in any CalDAV/CardDAV server, be it Nextcloud or anything else. Once you have done that, not only can you access the data anywhere on any device, the data also becomes very easy to reuse. It took me less than 15 minutes, testing included, to put together the whole thing (Listing 10), including copying and pasting.

Lines 1 to 10 of Listing 10 call the Perl libraries that perform several variable checks or provide time formatting functions. They then define all the variables I need and print the header line. Lines 11 and 12 scan the .ics file passed as an argument, skipping all lines except those that start with any of the strings in line 12.

Line 17 spots the variable containing the calendar name and saves its lowercase version into $CALENDAR. Line 19 does the same job for the event summary.

Lines 21 to 31 recognize and reformat the current event's start date, which is DTSTART in .ics files. Lines 33 to 43 do the same for the end date.

Whenever it finds the string END:VEVENT (line 45), the script knows that the description of an event has finished, and it appends a string with the calendar name, start time, event description, and end time (separated by pipes) to an array called @APPOINTMENTS.

After the whole .ics file has been scanned, all events are written to standard output, in chronological order (lines 51 to 55).

Radicale

If Nextcloud doesn't meet your needs, another option is the lightweight CalDAV/CardDAV server called Radicale [12]. A Python 3 package, Radicale hosts multiple calendars and contacts for multiple users, with several options to restrict and log accesses. It does not support all the CalDav/CardDAV specifications' features, but it is adequate for small scale usage. Above all, Radicale is very simple to install and configure. Besides Python 3, its only dependencies are the htpasswd utility to encrypt user passwords and the Python 3 libraries to decrypt them.

Radicale's website explains step by step how to install Radicale with the pip package manager and run it for one single user. However, I have found that on Ubuntu and other distributions, a Radicale multi-user installation is as simple as typing the following:

#> sudo apt-get install radicale apache2-utils python3-bcrypt python3-passlib
#> systemctl start radicale

This makes it much easier to keep current afterwards.

The apache2-utils package contains the htpasswd program, and the other two packages contain the required Python libraries. Radicale's default configuration is enough to run and access Radicale only on your local machine. To make it accessible from other computers or from the Internet, you need to "bind" Radicale to other addresses. To learn how to do this simple process, see Radicale's documentation [12]. I strongly suggest, however, that you play with a local-only installation before going live. For that, the only critical settings in /etc/radicale/config define the location and encryption method of usernames and passwords as follows:

[auth]
type = htpasswd
htpasswd_filename = /et/radicale/users
htpasswd_encryption = bcrypt

To create a Radicale user and store the user's encrypted passwords in htpasswd_filename, run the following command for each user and follow its instructions:

sudo  htpasswd -B /etc/radicale/users marco

Now, you can restart Radicale, point your browser to http://localhost:5232/, log in, and start creating calendars and address books in the interface shown in Figure 6. Of course, since Radicale is only a server, you will need clients like khal, khard, or Thunderbird to actually read and write those databases. To import personal data from other servers, just use vdirsyncer, and it will create automatically all the desired calendars and address books. When I configured vdirsyncer as shown in Listing 12, it uploaded all the data previously copied on my computer from the Nextcloud server to my Radicale server (see Figure 7).

[general]
status_path = "~/.vdirsyncer/status/"
[pair cal_local_to_radicale]
a = "my_cal_local"
b = "my_cal_radicale"
collections = ["from a", "from b"]
[storage my_cal_local]
type = "filesystem"
path = "~/.pim/"
fileext = ".ics"
[storage my_cal_radicale]
type = "caldav"
url = "http://localhost:5232"
username = "my_radicale_user_name"
password = "my_radicale_password"

Figure 6: Radicale offers a very simple web interface to create calendars or address books for multiple users, which can then be filled with data from other clients.

Figure 7: Calendars automatically imported from Nextcloud show up under Collections in Radicale.

Conclusions

Personal contacts and calendars are essential data for every computer user. With the utilities and methods described here, you have the basic tools to efficiently create, manage, process, store, and share your personal data.