Personal Data Manager
Tutorial – CalDAV/CardDAV
You can manage your calendars and address books with the CalDAV/CardDAV standards, Nextcloud, and a few open source tools.
If you keep a digital calendar or address book, you want your data to be stored in one central location and accessible from any device, wherever and whenever you need it. You also want ownership of your data and metadata. By using free and open source software (FOSS), you can create calendars and address books in a private cloud that allows you to synchronize and share that data, without being locked into some corporate, data-harvesting walled garden.
In this tutorial, I will explain the open standards, CalDAV and CardDAV, that make independent storing and sharing of calendar and address book data possible. Then, I will show you how to automatically import or export calendars and contacts, from any source, to a Nextcloud instance, process that data, and migrate it to another server. Finally, I will outline how to set up your own standalone calendar and address book.
CalDAV/CardDAV
CalDAV and CardDAV, the open standards that allow centralized storage and management of personal data, are both supersets of the Web Distributed Authoring and Versioning (WebDAV) system. WebDAV's specification describes how software programs can edit remote content over the Internet, using the same protocol (HTTP) that browsers use to load pages from websites. CalDav is the WebDAV extension for calendars, and CardDAV is the extension for personal contacts.
Both CalDAV and CardDAV are client-server protocols: They let many users, each with their own interfaces, simultaneously access the same set of events or contacts stored on a common server.
At the lower level, single events or whole calendars are stored inside files with the iCalendar extension .ics
(also .ical
or .icalender
) [1]. Files that contain address books, instead, have the Virtual Contact File extension .vcf
[2].
Both .ics
(Listing 1) and .vcf
(Listing 2) files use plain text formats with a relatively simple syntax that could be written manually with any text editor. The calendar event in Listing 1 shows an annual all staff meeting that must occur from February 12 to 13 (2008 in this example), as described in the RRULE
(recurrence rule) field. The DESCRIPTION
variable describes the meeting's purpose (a project status checkup) and the GEO
variable gives the meeting's location. The whole event is enclosed by BEGIN:VEVENT
and END:VEVENT
tags, inside a VCALENDAR
that may contain other events both before or after this event.
Listing 1
An .ics Event
BEGIN:VCALENDAR BEGIN:VEVENT SUMMARY:All Staff Meeting STATUS:CONFIRMED RRULE:FREQ=YEARLY;INTERVAL=1;BYMONTH=2;BYMONTHDAY=12 DTSTART:20080212 DTEND:20080213 CATEGORIES:teamwork, project management GEO:87.5739497;-45.7399606 DESCRIPTION: project status check-up END:VEVENT END:VCALENDAR
Listing 2
A .vcf Contact
BEGIN:VCARD VERSION:4.0 FN:Bruce Wayne ADR;TYPE=WORK:;;;Gotham City;USA;;Planet Earth EMAIL;TYPE=HOME:batman@example.com TEL;TYPE="HOME,VOICE":555-5555 URL:https://batman.example.com ORG:Billionaire TITLE:Dark Knight END:VCARD
Besides VEVENT
s, an .ics
calendar may also store to-do items (VTODO
), journal entries (VJOURNAL
), and time zone information (VTIMEZONE
). Rich text in HTML format can be specified by the parameter X-ALT-DESC
.
Both .ics
and .vcf
files also contain unique identifiers (UID) that allow the servers to index them. On the host computers, the files are stored in folders with a defined structure called vdir
. Each vdir
corresponds to a different calendar or address book. While it is possible to create, delete, or edit a vdir
's contents, I recommend using a proper CalDAV or CardDAV client to prevent corruption of vdir
's indexing.
Nextcloud
In this tutorial, I use Nextcloud [3], which is currently the most promising solution for open source, self-hosted cloud services. To install Nextcloud and understand its importance, please see my blog [4].
Under the hood, both the calendars and contacts of all of a Nextcloud instance's users are managed by an embedded CalDAV/CardDAV server.
Figures 1 and 2 show details of the address book and calendar interfaces that I created for a test Nextcloud account. In Figure 1, you can see the data that Nextcloud saved in Listing 2.
I describe the Nextcloud GUIs shown in Figures 1 and 2 in more detail in my blog [4]. However, there are at least two instances when there is a better way to process this personal data: The first is mass migration to or from another server or bulk export to some other database; the second is whenever you want to simultaneously modify contacts or calendars. Especially in the second instance, it can be much faster to make local copies of all the records, modify them with a script, and then re-upload everything to the server. The tools described in this tutorial can be used for all these tasks.
vdirsyncer
Theoretically, the simplest way to process automatically a CalDAV or CardDAV server's contents would be a command-line client that can connect directly to that server. In practice, I have found an alternate approach that seems easier to implement with standard tools and even more flexible: Copy the server data on your computer, change as needed those copies, and then sync the vdir
s on the server with those on your local archive. This approach also has the advantage of keeping contacts and events accessible even when the server is unreachable. I use vdirsyncer [5] to perform the synchronization.
In addition to installing manually, you can find vdirsyncer binary packages for several distributions, but not in the usual places. On my Ubuntu 19.10 desktop, for instance, I had to install from a custom repository with the sequence of commands shown in Listing 3, which I found in vdirsyncer's very complete manual [6].
Listing 3
Installing vdirsyncer on Ubuntu
01 curl -s https://packagecloud.io/install/repositories/pimutils/vdirsyncer/script.deb.sh | sudo bash 02 sudo apt-get update 03 sudo apt-get install vdirsyncer-latest 04 export PATH="$PATH:/opt/venvs/vdirsyncer-latest/bin"
The command in line 1 of Listing 3 downloads and executes a shell script that fetches data for the custom repository and adds them to the Ubuntu package database. Line 3 is what actually installs vdirsyncer on your system. Line 4 tells Ubuntu to add the vdirsyncer executable's folder (/opt/...
) to the list of folders ($PATH
) where programs are stored.
Once vdirsyncer is installed, it needs to know which vdir
s it should sync and their location. You do this with configuration files like vdirsyncer-nextcloud.conf
(Listing 4).
Listing 4
vdirsyncer-nextcloud.conf
01 [general] 02 status_path = "~/.vdirsyncer/status/" 03 [pair contacts_nextcloud_to_local] 04 a = "my_contacts_local" 05 b = "my_contacts_nextcloud" 06 collections = ["from a", "from b"] 07 [storage my_contacts_local] 08 type = "filesystem" 09 path = "~/.contacts/" 10 fileext = ".vcf" 11 [storage my_contacts_nextcloud] 12 type = "carddav" 13 url = https://full-url-of-my-private-nextcloud-instance/" 14 username = "nextcloud_test_user" 15 password = "nextcloud_test_password" 16 17 [pair cal_nextcloud_to_local] 18 a = "my_cal_local" 19 b = "my_cal_nextcloud" 20 collections = ["from a", "from b"] 21 [storage my_cal_local] 22 type = "filesystem" 23 path = "~/.calendars/" 24 fileext = ".ics" 25 [storage my_cal_nextcloud] 26 type = "caldav" 27 url = https://full-url-of-my-private-nextcloud-instance/" 28 username = "nextcloud_test_user" 29 password = "nextcloud_test_password"
Lines 1 and 2 of Listing 4 tell vdirsyncer where to store the synchronization status of all the vdir
s it manages.
Lines 3 to 15 define a pair of address books to sync. The address book on your local computer, my_contacts_local
(aliased as a
), is stored in normal folders located inside the ~/.contacts
directory and contains .vcf
files. The Nextcloud remote address book (lines 11 to 15) is accessible with the CardDAV protocol at the URL shown on line 13, with the credentials shown in lines 14 and 15. Line 6 means that vdirsyncer must sync all the collections in each pair of vdir
s.
The only difference in lines 17 to 29, visible in the variable names and file extensions, is that this section tells vdirsyncer to sync two sets of calendars instead of address books.
After the initial configuration, but before the actual synchronization, you need to make vdirsyncer "discover" all the data on the remote server (unless you configured to only sync one calendar, as explained in the documentation). Listing 5 shows how to do this plus includes an excerpt of the output.
Listing 5
Discovering Data on the Remote Server
01 #> vdirsyncer -c vdirsyncer-nextcloud.conf discover 02 Discovering collections for pair contacts_nextcloud_to_local 03 my_contacts_local: 04 - "contacts" 05 my_contacts_nextcloud: 06 - "contacts" ("Contacts") 07 Saved for contacts_nextcloud_to_local: collections = ["contacts"] 08 Discovering collections for pair cal_nextcloud_to_local 09 my_cal_local: 10 my_cal_nextcloud: 11 - "work" ("Work") 12 ..... 13 warning: No collection "work" found for storage my_cal_local. 14 Should vdirsyncer attempt to create it? [y/N]: y 15 Saved for cal_nextcloud_to_local: collections = ["work"]
The output means that everything went well, and vdirsyncer discovered everything it was supposed to find. Passing the configuration file with the -c
option is necessary when you want to use vdirsyncer with different servers. The actual synchronization is done with the following single command (which you could make a regular cron job):
#> vdirsyncer -c vdirsyncer-nextcloud.conf sync
Vdirsyncer has many other options and uses. Due to space constraints, I will only discuss how to handle conflicts. If two calendars contain two different records for the same event, you must decide which calendar is the master that should win such conflicts and explicitly tell vdirsyncer. (Listing 5), if the Nextcloud calendar is the master, you should add a line like this:
conflict_resolution = "b wins"
right after the lines 6 and 20 of Listing 5. In addition to the vdirsyncer manual [6], see [7] for more conflict resolution examples and vdirsyncer configuration tips.
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Support Our Work
Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.
News
-
Gnome OS Transitioning Toward a General-Purpose Distro
If you're looking for the perfectly vanilla take on the Gnome desktop, Gnome OS might be for you.
-
Fedora 41 Released with New Features
If you're a Fedora fan or just looking for a Linux distribution to help you migrate from Windows, Fedora 41 might be just the ticket.
-
AlmaLinux OS Kitten 10 Gives Power Users a Sneak Preview
If you're looking to kick the tires of AlmaLinux's upstream version, the developers have a purrfect solution.
-
Gnome 47.1 Released with a Few Fixes
The latest release of the Gnome desktop is all about fixing a few nagging issues and not about bringing new features into the mix.
-
System76 Unveils an Ampere-Powered Thelio Desktop
If you're looking for a new desktop system for developing autonomous driving and software-defined vehicle solutions. System76 has you covered.
-
VirtualBox 7.1.4 Includes Initial Support for Linux kernel 6.12
The latest version of VirtualBox has arrived and it not only adds initial support for kernel 6.12 but another feature that will make using the virtual machine tool much easier.
-
New Slimbook EVO with Raw AMD Ryzen Power
If you're looking for serious power in a 14" ultrabook that is powered by Linux, Slimbook has just the thing for you.
-
The Gnome Foundation Struggling to Stay Afloat
The foundation behind the Gnome desktop environment is having to go through some serious belt-tightening due to continued financial problems.
-
Thousands of Linux Servers Infected with Stealth Malware Since 2021
Perfctl is capable of remaining undetected, which makes it dangerous and hard to mitigate.
-
Halcyon Creates Anti-Ransomware Protection for Linux
As more Linux systems are targeted by ransomware, Halcyon is stepping up its protection.