HDF5 for efficient I/O
Fast Containers
© Lead Image © Kirsty Pargeter, 123RF.com
HDF5 is a flexible, self-describing, and portable hierarchical filesystem supported by a number of languages and tools, with the ability to run processes in parallel.
Input/output operations are a very important part of many applications, sometimes involving a huge amount of data and a large number of reads and writes. Therefore, applications can use a very significant portion of their total run time to perform I/O, which becomes critical in Big Data, machine learning, and high-performance computing (HPC).
In a previous article [1], I discussed options for improving I/O performance, focusing on parallel I/O. One of the options mentioned was to use a high-level library to perform the I/O. A great example of such a library is the Hierarchical Data Format (HDF) [2], a standard library used primarily for scientific computing.
In this article, I introduce HDF5 and focus on the concepts and its strengths in performing I/O; then, I look at some simple Python and Fortran code examples, before ending with an example of parallel I/O with HDF5 and Fortran.
[...]
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Support Our Work
Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.
News
-
Framework Laptop 13 Pro Competes with the Best
Framework has released what might be considered the MacBook of Linux devices.
-
The Latest CachyOS Features Supercharged Kernel
The latest release of CachyOS brings with it an enhanced version of the latest Linux kernel.
-
Kernel 7.0 Is a Bit More Rusty
Linux kernel 7.0 has been released for general availability, with Rust finally getting its due.
-
France Says "Au Revoir" to Microsoft
In a move that should surprise no one, France announced plans to reduce its reliance on US technology, and Microsoft Windows is the first to get the boot.
-
CIQ Releases Compatibility Catalog for Rocky Linux
The company behind Rocky Linux is making an open catalog available to developers, hobbyists, and other contributors, so they can verify and publish compatibility with the CIQ lineup.
-
KDE Gets Some Resuscitation
KDE is bringing back two themes that vanished a few years ago, putting a bit more air under its wings.
-
Ubuntu 26.04 Beta Arrives with Some Surprises
Ubuntu 26.04 is almost here, but the beta version has been released, and it might surprise some people.
-
Ubuntu MATE Dev Leaving After 12 years
Martin Wimpress, the maintainer of Ubuntu MATE, is now searching for his successor. Are you the next in line?
-
Kali Linux Waxes Nostalgic with BackTrack Mode
For those who've used Kali Linux since its inception, the changes with the new release are sure to put a smile on your face.
-
Gnome 50 Smooths Out NVIDIA GPU Issues
Gamers rejoice, your favorite pastime just got better with Gnome 50 and NVIDIA GPUs.