Managing open data with CKAN
Fast Track
CKAN, a versatile data management system, lets you build a portal to share your open data.
Open data would be of little use if it wasn't easy to automatically publish and update and subsequently find, download, and integrate with other open data, straight from the primary sources.
The Comprehensive Knowledge Archive Network (CKAN) [1], a key component of the Open Knowledge Foundation's mission to promote open data [2], is a collection of tools for implementing open data hubs and portals. If you have a dataset that you'd like to make available as open data, CKAN gives you a head start on implementing the necessary infrastructure.
The CKAN tools help you manage and federate the data. It also comes with advanced geospacial features, search capabilities, metadata, and visualization options. Using CKAN tools and best practices, you can build a portal for your open data with the least possible effort.
How CKAN Works
CKAN is based on open standards and free/open source software. At its core, most of CKAN is Python code, released under an aGPL license. Other main components are the PostgreSQL database engine and the Apache Solr search server [3].
CKAN websites make it easy to find, browse, and analyze open data manually. CKAN's real value, however, is its consistent support for creating federated catalogs of open data and, importantly, finding and reusing the same data automatically.
CKAN does this through application programming interfaces (APIs) [4] that support REST-style data exchange and announce and export entire datasets according to the Data Catalogue Vocabulary Application Profile (DCAT-AP) [5]. DCAT-AP is an open standard for describing public sector datasets, developed in Europe specifically to make those datasets easier to search. DCAT-AP rules standardize the description of datasets, defining which metadata should be published and the corresponding machine-readable formats.
For this reason, many organizations (mainly, but not exclusively, public administrations) use CKAN to collaboratively build a worldwide, coherent catalog of open data. CKAN's tools and methods completely automate not just the initial online publication of open data with all its metadata, but also its constant updating, straight from the original sources, whether from databases or spreadsheet files.
When loading one or more sets of open data onto a CKAN website, its administrator may enable its end users (both human and other software) to immediately download current dataset snapshots in JSON format or browse and visualize the raw data directly in several ways.
Automatic usage is possible thanks to other CKAN functions that, once a dataset is loaded, announce its existence with Resource Description Framework (RDF) declarations. Reading the RDF declarations, other portals may then expose the corresponding datasets, as well as datasets of many other CKAN-compatible sources, as part of one single catalog of their choosing. Even better, that catalog will always automatically contain complete, current versions of all the datasets it lists.
In Italy, for example, some regional administrations' CKAN catalogs automatically feed into the national data portal [6], which then automatically feeds into the European portal [7]. Worth noting here, the national data portal is not a CKAN portal, but it can still federate successfully by supporting and using the DCAT-AP standards and software tools. That's the beauty of open standards.
For this to work in practice, two conditions must be met. First, a website must announce that it's running CKAN (or at least following the compatible interfaces and standards) and be configured to make it discoverable by other CKAN (or CKAN-compatible) websites (which I am told is not always the case). Second, each dataset's metadata must always exactly correspond to its actual scope, freshness, and content. Otherwise, the whole thing either breaks or becomes practically useless. While this synchronization requires some effort, CKAN remains the fastest path to standardized, collaborative publishing of open data.
CKAN in Action!
To demonstrate CKAN's usefulness, I interviewed two of the main Italian CKAN experts: Maurizio Napolitano [8] and Vincenzo Patruno [9]. Napolitano works for Fondazione Bruno Kessler (FBK, Bruno Kessler Foundation) in Trento, Italy, as coordinator of the FBK Digital Commons Lab. Patruno is the IT/data manager for the Italian National Institute of Statistics and the vice president of onData APS [10] (a nonprofit association that promotes open data). Both Napolitano and Patruno got involved with open data because they believe "the most powerful form of revolution we have today [is] to give more opportunities to everybody with good ideas to improve our society."
Trento Tourist Spots
Napolitano provides the first of example of CKAN's power with a dataset that lists the tourist points of interest in the Valle dei Laghi (Valley of Lakes) near Trento in Northern Italy. To access this dataset (shown in Figure 1), you can use the federated OPENdata Trentino portal to search for "Valle dei Laghi" [11].
On the right, each Esplora (explore) drop-down menu has two options: Scaricare (download the raw dataset) and Anteprima (preview). Choosing Anteprima opens the interface shown in Figure 2.
The default Grid view lists the several entries in columns, allowing users to filter or sort them as needed. Clicking on the Map tab shows each entry on the map. However, in Figure 3, the map is empty. The reason for this is because the latitude and longitude columns have not been labeled correctly in the Grid view in Figure 2. CKAN expects these columns to be named Latitude and Longitude to recognize and use the geographic coordinates, but Figure 2 uses the Italian column names Latitudine and Longitudine. To get around this problem, you can tell CKAN explicitly where to find those values as shown in Figure 4. Once you have assigned the latitude and longitude fields, all the entries appear on the map (Figure 5).
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Support Our Work
Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.
News
-
AlmaLinux 10.0 Beta Released
The AlmaLinux OS Foundation has announced the availability of AlmaLinux 10.0 Beta ("Purple Lion") for all supported devices with significant changes.
-
Gnome 47.2 Now Available
Gnome 47.2 is now available for general use but don't expect much in the way of newness, as this is all about improvements and bug fixes.
-
Latest Cinnamon Desktop Releases with a Bold New Look
Just in time for the holidays, the developer of the Cinnamon desktop has shipped a new release to help spice up your eggnog with new features and a new look.
-
Armbian 24.11 Released with Expanded Hardware Support
If you've been waiting for Armbian to support OrangePi 5 Max and Radxa ROCK 5B+, the wait is over.
-
SUSE Renames Several Products for Better Name Recognition
SUSE has been a very powerful player in the European market, but it knows it must branch out to gain serious traction. Will a name change do the trick?
-
ESET Discovers New Linux Malware
WolfsBane is an all-in-one malware that has hit the Linux operating system and includes a dropper, a launcher, and a backdoor.
-
New Linux Kernel Patch Allows Forcing a CPU Mitigation
Even when CPU mitigations can consume precious CPU cycles, it might not be a bad idea to allow users to enable them, even if your machine isn't vulnerable.
-
Red Hat Enterprise Linux 9.5 Released
Notify your friends, loved ones, and colleagues that the latest version of RHEL is available with plenty of enhancements.
-
Linux Sees Massive Performance Increase from a Single Line of Code
With one line of code, Intel was able to increase the performance of the Linux kernel by 4,000 percent.
-
Fedora KDE Approved as an Official Spin
If you prefer the Plasma desktop environment and the Fedora distribution, you're in luck because there's now an official spin that is listed on the same level as the Fedora Workstation edition.