HDF5 for efficient I/O
Fast Containers
© Lead Image © Kirsty Pargeter, 123RF.com
HDF5 is a flexible, self-describing, and portable hierarchical filesystem supported by a number of languages and tools, with the ability to run processes in parallel.
Input/output operations are a very important part of many applications, sometimes involving a huge amount of data and a large number of reads and writes. Therefore, applications can use a very significant portion of their total run time to perform I/O, which becomes critical in Big Data, machine learning, and high-performance computing (HPC).
In a previous article [1], I discussed options for improving I/O performance, focusing on parallel I/O. One of the options mentioned was to use a high-level library to perform the I/O. A great example of such a library is the Hierarchical Data Format (HDF) [2], a standard library used primarily for scientific computing.
In this article, I introduce HDF5 and focus on the concepts and its strengths in performing I/O; then, I look at some simple Python and Fortran code examples, before ending with an example of parallel I/O with HDF5 and Fortran.
[...]
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
US / Canada
UK / Australia
Direct Download
Read full article as PDF:
Price $2.95
News
-
Honey I Shrunk Ubuntu
Canonical has released a new version of Ubuntu that’s 50% smaller than the standard Ubuntu Server.
-
Linux Mint 19 Released
New version comes with Timeshifting capabilities.
-
Red Hat Adds GPLv3 Cure Clause to its Code-Base
New provision will provide violators with a grace period to achieve license compliance.
-
SUSE Linux Enterprise 15 Released
It’s a modular OS.
-
GitLabs Drops Pricing After Microsoft GitHub Acquisition
GitLabs is creating more incentives for users to switch from GitHub.
-
KDE Plasma 5.13 Is Here
The desktop environment is designed to be more resource efficient.
-
SUSE Forks Red Hat’s Spacewalk Project
SUSE relies heavily on the project for its own SUSE Manager.
-
openSUSE Leap 15 Announced
The release was announced at the openSUSE Conference in Prague, Czech Republic.
-
SoftMaker FreeOffice 2018 for Linux
SoftMaker Software GmbH has announced FreeOffice 2018, a free of cost office suite for Linux.
-
A New CentOS
Latest version of the community Linux based on Red Hat Enterprise Linux 7.5