Improving RAID performance with filesystem and RAID parameters
Bringing LVM into the Mix
When you create an XFS filesystem directly on a software RAID device, mkfs.xfs is smart enough to work out the chunk and stripe sizes for you and configure the XFS filesystem optimally. Unfortunately, if you are using a hardware RAID card or using LVM between XFS and the software RAID, it is up to you to make sure the filesystem is created optimally.
I created both a hand-aligned XFS and a XFS filesystem with default parameters to test what difference, if any, this made to performance. Many folks use LVM on top of a RAID device for improved system management and flexibility. Being able to store many filesystems on a single RAID device is very convenient. I retested the ext3 configurations to see whether using the extra LVM layer sapped much performance.
Figure 5 shows the performance of ext3 and XFS on the same RAID device. For the "raw" configuration, the filesystem is created directly on the RAID device. In the "lvm" configuration, an LVM physical volume is created on the RAID device, and an LVM logical volume is created on the LVM physical volume. The filesystem is then created on the LVM logical volume.
Notice that the gains in read performance resulting from alignment of the ext3 filesystem are gone and that read performance across all filesystems is identical. Aligning the stripe on the XFS filesystem actually made write performance worse than with no alignment attempt. LVM itself appears to be limiting block read performance, so attempts to align the filesystem are not resulting in any performance differences.
Interestingly, though, block output for XFS was greatly improved by using LVM. XFS attempts to delay writes to files for as long as possible. The delayed writes might be the reason that XFS is faster at writing with the use of LVM – if LVM maintains a write cache of its own, then perhaps data can be streamed out to disk from that cache more effectively when it is presented in large chunks by the filesystem. Notice that the read performance is significantly reduced when LVM is used.
Figure 6 shows the block rewrite bonnie++ benchmark. Here you can see that the gains in correctly aligning the ext3 filesystem have been preserved to some degree, with the aligned ext3 filesystem being almost 1MB/sec faster than the unaligned ext3 on LVM. The differences for correct alignment of ext3 are less when using LVM, however: 1MB/sec for LVM compared with about 2.5MB/sec on raw RAID.

Although differences of a megabyte per second might not seem so important, in the worst case, this means the aligned ext3 on raw RAID would be more than 15% faster than unaligned ext3 on LVM. When you consider that this is running on the same machine and same disks, 15% free performance is something you might like to have.
Because it can be somewhat difficult to compare two three-dimensional graphs by eye, I decided to graph not the two original datasets, but the difference between the aligned and non-aligned datasets. In Figure 7, I'm showing you the (ext3aligned – ext3) IOzone random read performance for LVM. Although it is not completely in favor of the ext3aligned filesystem, you can see significant performance peaks where the ext3aligned filesystem is much faster in the 256 to 4,096KB file size range (to the left of the figure).
Figure 8 shows the difference of random write performance for ext3 with and without alignment on LVM. Notice that the peaks and troughs are in the same places, although the extremes are not as pronounced. The dips in performance for the ext3align filesystem are almost eradicated.
Figure 9 shows XFS with alignment and without alignment when created on LVM for random read. Notice that the graph is much less busy for XFS compared with ext3 (shown in Figure 7). Random reads of smaller files are much faster on a properly aligned XFS filesystem, as the peaks on the left side of Figure 9 show. For smaller writes to 4MB files, an aligned XFS filesystem is slightly slower than an XFS filesystem created with the default mkfs.xfs arguments (difference graph not shown). The aligned XFS is faster for the larger writes to files of 16 to 256MB.
Digging a Deeper Hole
The Linux kernel has supported the ext4 filesystem since v2.6.28. Performance improvements over its successor, ext3, include 48-bit block addressing.
One downside of specifying the RAID stripe size at mkfs time is what happens when you add an extra disk to an existing RAID. All of a sudden, you have four data disks on your RAID 5 instead of three, and the stripe alignment that the filesystem is using is incorrect. Luckily, ext4 has a mount option to allow you to set the stripe size at mount time, so if you add a disk to a RAID or extend an LVM volume, you can tell ext4 what you've done so the filesystem can optimize future activity to fit the modified disk configuration.
Unfortunately, specifying the sunit and swidth options when mounting an XFS filesystem is only effective when the filesystem is created directly on a RAID device. So if you use LVM on your RAID and add a disk, you can't tell XFS about the change at mount time. This is once again a case in which XFS does not play well with LVM.
When using LVM, metadata is stored at various abstraction layers. For example, pvcreate has various options for setting the size of metadata and data alignment. So far, I have not found tweaking pvcreate to have the same effect as setting filesystem stripe parameters, but that question is open to further investigation.
Wrap Up
As this study shows, you can get 10% to 20% extra performance if you align your ext3 filesystem to the RAID array chunk and stripe size. Although the differences were not as significant for XFS, proper alignment of XFS can make a large difference to the performance of file creation and deletion.
In summary:
- If you are using the ext3 or ext4 filesystem, you should always use -E stripe-width=x,stride=y to tell the filesystem about your RAID configuration.
- If you are using XFS directly on the RAID device, the system should detect your RAID configuration for you automatically.
- If you are using XFS on an LVM partition created on a RAID, use -d sunit= x,swidth=y at mkfs.xfs time to tell it about your RAID configuration.
- If you are using another filesystem, check that it has detected your RAID configuration – in particular, check the stripe and chunk size – at mkfs time.
The actual benefits that you reap by taking these steps will vary depending on your configuration, but regardless of the numbers, aligning your filesystem parameters with the RAID system is a good strategy for achieving better performance, especially on ext3 and ext4 filesystems.
Infos
- OSDL's Linux RAID: http://linux-raid.osdl.org/index.php/Linux_Raid
- "A Case For Redundant Arrays of Inexpensive Disks (RAID)," by David A. Patterson, Garth Gibson, and Randy H. Katz. http://www-2.cs.cmu.edu/~garth/RAIDpaper/Patterson88.pdf
« Previous 1 2
Buy Linux Magazine
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Support Our Work
Linux Magazine content is made possible with support from readers like you. Please consider contributing when you've found an article to be beneficial.
News
-
The GNU Project Celebrates Its 40th Birthday
September 27 marks the 40th anniversary of the GNU Project, and it was celebrated with a hacker meeting in Biel/Bienne, Switzerland.
-
Linux Kernel Reducing Long-Term Support
LTS support for the Linux kernel is about to undergo some serious changes that will have a considerable impact on the future.
-
Fedora 39 Beta Now Available for Testing
For fans and users of Fedora Linux, the first beta of release 39 is now available, which is a minor upgrade but does include GNOME 45.
-
Fedora Linux 40 to Drop X11 for KDE Plasma
When Fedora 40 arrives in 2024, there will be a few big changes coming, especially for the KDE Plasma option.
-
Real-Time Ubuntu Available in AWS Marketplace
Anyone looking for a Linux distribution for real-time processing could do a whole lot worse than Real-Time Ubuntu.
-
KSMBD Finally Reaches a Stable State
For those who've been looking forward to the first release of KSMBD, after two years it's no longer considered experimental.
-
Nitrux 3.0.0 Has Been Released
The latest version of Nitrux brings plenty of innovation and fresh apps to the table.
-
Linux From Scratch 12.0 Now Available
If you're looking to roll your own Linux distribution, the latest version of Linux From Scratch is now available with plenty of updates.
-
Linux Kernel 6.5 Has Been Released
The newest Linux kernel, version 6.5, now includes initial support for two very exciting features.
-
UbuntuDDE 23.04 Now Available
A new version of the UbuntuDDE remix has finally arrived with all the updates from the Deepin desktop and everything that comes with the Ubuntu 23.04 base.