[lustre-discuss] Lustre 2.10.3 on ZFS - slow read performance

Alex Vodeyko alex.vodeyko at gmail.com
Sat Mar 31 12:11:21 PDT 2018

Hi Stephane,

thank you!
With ashift=9 reads are much better (1.7 GB/s vs 0.5 GB/s).
However reads are still 5-10 times slower in the mixed read/write workflows.
I'm continuing tuning/benchmarking and will keep you updated.


2018-03-30 20:00 GMT+03:00 Stephane Thiell <sthiell at stanford.edu>:
> Hi Alex,
> I’m no ZFS expert, but for a new project I recently faced some read performance issues too when doing some zfs 0.7 testing, but not at all as bad as you… so I feel sorry for you as you seem to have done quite a good work so far… so perhaps some ideas:
> - check that arc is fully enabled (primarycache=all)
> - restraint arc memory usage (zfs_arc_min = zfs_arc_max = 50% of the RAM is what we usually use)
> - try different values of ashift (unlikely but try to keep the default 9, and/or try 11, 13 or more but it has a volume cost)
> - turn off readahead but you already did that...
> - bump zfs_vdev_cache_max to 131072
> - bump zfs_read_chunk_size to 1310720
> - bump zfs_vdev_cache_bshift to 17
> - bump zfs_vdev_async_read_min_active to 8
> - bump zfs_vdev_async_read_max_active to 32
> - bump zfs_vdev_sync_read_max_active to 32
> /!\ These tunings are not to take as is and are meant for performance testing, plus some of them might be obsolete with 0.7, but we still use a few of these on our ZFS on Linux systems.
> As a rule of thumb, do not explicitly tune if you don’t really need it.
> To get the best read performance from nearline drives with Lustre (without the use of any SSD), we went with mdraid/ldiskfs and we’re VERY happy with that.  With the same hardware, a ZFS backend will definitively provide better writes though.
> Good luck...
> Stephane

More information about the lustre-discuss mailing list