Linux filesystems – small file performance on HDDs

A handy chart for quick reference.

The benchmark is quite workload-specific; I measured sequential operations on large sets of small files (rather than random reads/writes on a single large file) – an approach similar to filebench or bonnie++.

Methodology:

1. hardware: 4-core x86-64, 10 15kRPM SAS HDDs in RAID 10, LSI MegaRAID, write cache enabled,

2. software: Linux 2.6.32.x, irqbalance, mkfs.* etc.

3. popular Linux filesystems: ext{2,3,4}, XFS, ReiserFS 3,

4. JFS was intentionally left out: it performs so poorly with small files (3-4x worse than XFS), including it in the benchmark made no sense at all.

5. All filesystems were created with default values and mounted with the noatime flag only.

6. Data set: each process operated on its individual set of 350,000 files with an average size of 50 kB, 200 files per directory.

7. VFS cache was dropped (or warmed up) where needed.

8. Operations labeled on the chart are not discrete (syscalls). Rather, compound operations were performed on the data sets, i.e.

   • unlink and stat are: opendir, readdir, unlink or stat in sequence, closedir
   • write is: mkdir (if necessary), open, write, close, chown, chmod
   • rewrite is: mkdir (if necessary), open, unlink and the rest as with write
   • read is: opendir, readdir, open/read/close in sequence.

9. All operations were in sequential (i.e. readdir) order (rather than random).

10. Operation times are in seconds.

11. All values were averaged from 10 samples.

12. Op times are capped at 450 seconds on the chart, for real times see:

13. raw data (CSV format)

14. ext2 is included in the benchmark, as some systems still use it when journaling is not required.

15. This benchmark is valid only for HDDs. On SSDs, the differences are significantly less pronounced due to no (low) seek time penalty.

16. YMMV

Conclusions: nothing new here, obviously. XFS simply sucks for small files, ext3 is slow, ext2 is faster than ext3, ReiserFS 3 performs nicely with everything but concurrent writes (it uses a global lock for writes: the Big Kernel Lock up to 2.6.33 and now a mutex-based solution). ext4 is the new hotness.

The chart (click for full-size version):