<div dir="ltr">Hi All,<div><br></div><div> I've been evaluating some of the newer options that should be available with Lustre 2.9 on top of ZFSonLinux 0.7.0 (currently at rc3). Specifically, trying 16MB RPCS/blocks on the OSTs and large dnodes on the MDTs.</div><div><br></div><div> I've gathered bits and pieces from discussions in the zfs and lustre development areas, presentations for earlier versions, etc., but have probably missed some low-hanging fruit. Is there a central resource for good starting sets of parameters?</div><div><br></div><div>Brief summary of test system (all on CentOS 7.3x):</div><div>1 MDS (2x E5-2650 v4, 24 cores, 128GB RAM) with three MDTs.</div><div> Each MDT on its own 2-SSD mirror zpool (1TB each)</div><div>pool options:</div><div> ashift=9 (for space reasons)</div><div>dataset options:</div><div> recordsize=128K</div><div> compression=off</div><div> atime=off</div><div> xattr=sa</div><div> dnodesize=auto</div><div>zfs module options: </div><div> zfs_prefetch_disable=1</div><div> metaslab_debug_unload=1</div><div> zfs_dirty_data_max=2147483648</div><div> zfs_vdev_async_write_min_active=5</div><div> zfs_vdev_async_write_max_active=15</div><div> zfs_vdev_async_write_active_min_dirty_percent=20</div><div> zfs_vdev_scheduler=deadline</div><div> zfs_arc_max=103079215104</div><div> zfs_arc_meta_limit=103079215104<br clear="all"><div><br></div><div>1 OSS (2x E5-2650 v4, 24 cores, 768GB RAM) with three ODTs.</div><div> Each ODT on its own 30-disk pool (3, 10-disk raidz2 vdevs) using 8TB drives</div><div>pool options:</div><div> ashift=12</div><div>dataset options:</div><div> recordsize=16M</div><div> compression=lz4</div><div> atime=off</div><div> xattr=sa</div><div> dnodesize=auto</div><div>zfs module options:</div><div> metaslab_aliquot=2097152</div><div> metaslab_debug_unload=1</div><div> zfs_dirty_data_max=2147483648</div><div> zfs_dirty_data_sync=134217728</div><div> zfs_max_recordsize=16777216</div><div> zfs_prefetch_disable=1</div><div> zfs_txg_history=10</div><div> zfs_vdev_aggregation_limit=16777216</div><div> zfs_vdev_async_write_min_active=5</div><div> zfs_vdev_async_write_max_active=15</div><div> zfs_vdev_async_write_active_min_dirty_percent=20</div><div> zfs_vdev_scheduler=deadline</div><div> zfs_arc_max=751619276800</div><div><br></div><div>All nodes use Mellanox EDR IB and the distribution's RDMA stack (mlx5).</div><div>ko2iblnd module options:</div><div><div> concurrent_sends=63</div><div> credits=2560</div><div> fmr_flush_trigger=1024</div><div> fmr_pool_size=1280</div><div> map_on_demand=256</div><div> peer_buffer_credits=0</div><div> peer_credits=16</div><div> peer_credits_hiw=31</div><div> peer_timeout=100</div></div><div><br></div><div>Lustre properly picks up the OST block size, and clients connect with 16M. The obdfilter-survey tool gives 2-5GB/s write and rewrite over record sizes of 1M-16M. At 16MB, most of the blocks displayed with "zpool iostat -r 5" are the expected 2MB for 10-disk raidz2 vdevs.</div><div><br></div><div>The mds-survey tool also gives not-terrible results, although I don't have a good reference:</div><div><div><font face="monospace, monospace">dir_count=6 thrlo=12 thrhi=12 file_count=300000 stripe_count=1 mds-survey</font></div><div><font face="monospace, monospace">mdt 3 file 300000 dir 6 thr 12 create 52875.87 [ 47998.37, 57997.85] lookup 579347.67 [ 370250.61, 919106.38] md_getattr 204802.03 [ 180678.81, 218144.06] setxattr 132886.29 [ 129995.32, 133994.51] destroy 20594.50 [ 11999.21, 34998.01] </font></div></div><div><br></div><div>I fully expect that some of the zfs module tuning above is either no longer needed (metaslab_debug_unload?), or counterproductive. If anyone has suggestions for changes, especially for making insane-numbers-of-tiny-files less deadly, I'd like to try them. This isn't going to go into production as scratch space for a while.</div><div><br></div><div>Thanks,</div><div>Nate</div><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr"><pre>Dr. Nathan Crawford <a href="mailto:nathan.crawford@uci.edu" target="_blank">nathan.crawford@uci.edu</a>
Modeling Facility Director
Department of Chemistry
1102 Natural Sciences II Office: 2101 Natural Sciences II
University of California, Irvine Phone: 949-824-4508
Irvine, CA 92697-2025, USA</pre></div></div>
</div></div>