<div dir="ltr">Andreas, <div><br></div><div>Thank you again for your detailed reply and time, I will have a look further at the lustre IO kit and hopefully, get to the bottom of things.</div><div><br></div><div>Cheers,</div><div>Finn</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, 21 Apr 2022 at 00:52, Andreas Dilger <<a href="mailto:adilger@whamcloud.com">adilger@whamcloud.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div style="overflow-wrap: break-word;">
Finn,
<div>I can't really say for sure where the performance limitation in your system is coming from.</div>
<div><br>
</div>
<div><br>
<div>You'd have to re-run the tests against the local ldiskfs filesystem to see how the performance compares to with that of Lustre. The important part of benchmark testing is to systematically build a complete picture from the ground up to see what
the capabilities of the various components of the storage stack are, and then determine where any bottlenecks are being hit.</div>
<div><br>
</div>
<div>That is what the "lustre-iokit" is intended to do - benchmark starting on the raw storage (sgpdd-survey), on the local disk filesystem (obdfilter-survey for local OSDs), then the network (lnet-selftest), and finally on the client (obdfilter-survey
for network OSDs).</div>
<div><br>
</div>
<div>For example, running sgpdd-survey (or "fio") with small and large IO sizes against the storage devices, individually *AND IN PARALLEL* to determine their performance characteristics. Also running in parallel is critical, since you may see e.g.
3GB/s reads, 2GB/s writes from a single NVMe device, but *not* see 4x that performance when running on 4x NVMe devices because of CPU and/or PCI and/or memory bandwidth limitations. Similarly, you may see reasonable per-OSS performance from a single OSS,
but network congestion (on the client, switch(es), or server) may prevent the performance from scaling as more servers are added.</div>
<div><br>
</div>
<div>This is described in some detail at <a href="https://github.com/DDNStorage/lustre_manual_markdown/blob/master/04.02-Benchmarking%20Lustre%20File%20System%20Performance%20(Lustre%20IO%20Kit).md" target="_blank">https://github.com/DDNStorage/lustre_manual_markdown/blob/master/04.02-Benchmarking%20Lustre%20File%20System%20Performance%20(Lustre%20IO%20Kit).md</a></div>
<div><br>
</div>
<div>Cheers, Andreas<br>
<div><br>
<blockquote type="cite">
<div>On Apr 20, 2022, at 12:03, Finn Rawles Malliagh <<a href="mailto:up883044@myport.ac.uk" target="_blank">up883044@myport.ac.uk</a>> wrote:</div>
<br>
<div>
<div dir="ltr">Hi Andreas,
<div><br>
</div>
<div>Thank you for taking the time to reply with such a detailed response.</div>
<div>I have taken your advice on board and made some changes. Firstly, I have swapped from ZFS and am now using striped LVM groups (Including the P4800X instead of using it as a cache drive). I have also modified io500.sh to include the optimisation
listed above. Rerunning the IO500 benchmark provides the metadata results below:</div>
<div><br>
</div>
<div>With ZFS<br>
[RESULT] mdtest-easy-write 0.931693 kIOPS : time 31.028 seconds [INVALID]<br>
[RESULT] mdtest-hard-write 0.427000 kIOPS : time 31.070 seconds [INVALID]<br>
[RESULT] find 25.311534 kIOPS : time 1.631 seconds<br>
[RESULT] mdtest-easy-stat 0.570021 kIOPS : time 50.067 seconds<br>
[RESULT] mdtest-hard-stat 1.834985 kIOPS : time 7.998 seconds<br>
[RESULT] mdtest-easy-delete 1.715750 kIOPS : time 17.308 seconds<br>
[RESULT] mdtest-hard-read 1.006240 kIOPS : time 13.759 seconds<br>
[RESULT] mdtest-hard-delete 1.624117 kIOPS : time 8.910 seconds<br>
[SCORE ] Bandwidth 2.271383 GiB/s : IOPS 1.526825 kiops : TOTAL 1.862258 [INVALID]<br>
<br>
With LVM:<br>
[RESULT] mdtest-easy-write 3.057249 kIOPS : time 27.177 seconds [INVALID]<br>
[RESULT] mdtest-hard-write 1.576865 kIOPS : time 51.740 seconds [INVALID]<br>
[RESULT] find 71.979457 kIOPS : time 2.234 seconds<br>
[RESULT] mdtest-easy-stat 1.841655 kIOPS : time 44.443 seconds<br>
[RESULT] mdtest-hard-stat 1.779211 kIOPS : time 45.967 seconds<br>
[RESULT] mdtest-easy-delete 1.559825 kIOPS : time 52.301 seconds<br>
[RESULT] mdtest-hard-read 0.631109 kIOPS : time 127.765 seconds<br>
[RESULT] mdtest-hard-delete 0.856858 kIOPS : time 94.372 seconds<br>
[SCORE ] Bandwidth 0.948100 GiB/s : IOPS 2.359024 kiops : TOTAL 1.495524 [INVALID]<br>
</div>
<div><br>
</div>
<div>I believe these scores are more in line with what I should expect, however, it seems that my throughput performance is still lacking(?). In your expert opinion do you think this would be just a case of tuning IO500/lvm parameters further or something
more fundamental about the configuration of this Lustre cluster?</div>
<div><br>
</div>
<div>With LVM</div>
<div>[RESULT] ior-easy-write 2.127026 GiB/s : time 122.305 seconds [INVALID]<br>
[RESULT] ior-hard-write 1.408638 GiB/s : time 1.246 seconds [INVALID]<br>
[RESULT] ior-easy-read 1.549550 GiB/s : time 167.881 seconds<br>
[RESULT] ior-hard-read 0.174036 GiB/s : time 10.063 seconds<br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>Kind Regards,</div>
<div>Finn</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Wed, 20 Apr 2022 at 09:24, Andreas Dilger <<a href="mailto:adilger@whamcloud.com" target="_blank">adilger@whamcloud.com</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>On Apr 16, 2022, at 22:51, Finn Rawles Malliagh via lustre-discuss <<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a>> wrote:<br>
<div>
<blockquote type="cite"><br>
<div>
<div dir="ltr">Hi all,
<div><br>
</div>
<div>I have just set up a three-node Lustre configuration, and initial testing shows what I think are slow results. The current configuration is 2 OSS, 1 MDS-MGS; each OSS/MGS has 4x Intel P3600, 1x Intel P4800, Intel E810 100Gbe eth, 2x 6252, 380GB
dram</div>
<div>I am using Lustre 2.12.8, ZFS 0.7.13, ice-1.8.3, rdma-core-35.0 (RoCEv2 is enabled)</div>
<div>All zpools are setup identical for OST1, OST2, and MDT1</div>
<div><br>
</div>
<div>[root@stor3 ~]# zpool status<br>
pool: osstank<br>
state: ONLINE<br>
scan: none requested<br>
config:<br>
NAME STATE READ WRITE CKSUM<br>
osstank ONLINE 0 0 0<br>
nvme1n1 ONLINE 0 0 0<br>
nvme2n1 ONLINE 0 0 0<br>
nvme3n1 ONLINE 0 0 0<br>
cache<br>
nvme0n1 ONLINE 0 0 0<br>
</div>
</div>
</div>
</blockquote>
<div><br>
</div>
It's been a while since I've done anything with ZFS, but I see a few potential issues here:</div>
<div>- firstly, it doesn't make sense IMHO to have an NVMe cache device when the main storage</div>
<div> pool is also NVMe. You could better use that capacity/bandwidth for storing more data</div>
<div> instead of duplicating it into the cache device. Also, Lustre cannot use the ZIL.</div>
<div>- in general ZFS is not very good at IOPS workloads because of the high overhead per block.</div>
<div> Lustre can't use the ZIL, so no opportunity to accelerate heavy IOPS workloads.</div>
<div><br>
<blockquote type="cite">
<div>
<div dir="ltr">
<div>When running "./io500 ./config-minimalLUST.ini" on my lustre client, I get these performance numbers:</div>
<div>IO500 version io500-isc22_v1 (standard)<br>
[RESULT] ior-easy-write 1.173435 GiB/s : time 31.703 seconds [INVALID]<br>
[RESULT] ior-hard-write 0.821624 GiB/s : time 1.070 seconds [INVALID]</div>
</div>
</div>
</blockquote>
<blockquote type="cite">
<div dir="ltr">
<div>[RESULT] ior-easy-read 5.177930 GiB/s : time 7.187 seconds<br>
</div>
</div>
</blockquote>
<blockquote type="cite">
<div dir="ltr">
<div>[RESULT] ior-hard-read 5.331791 GiB/s : time 0.167 seconds<br>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>
<blockquote type="cite">
<div dir="ltr">
<div>When running "./io500 ./config-minimalLOCAL.ini" on a singular locally mounted ZFS pool I get the following performance numbers:<br>
</div>
<div>IO500 version io500-isc22_v1 (standard)<br>
[RESULT] ior-easy-write 1.304500 GiB/s : time 33.302 seconds [INVALID]<br>
</div>
</div>
</blockquote>
<div>
<blockquote type="cite">
<div dir="ltr">
<div>[RESULT] ior-hard-write 0.485283 GiB/s : time 1.806 seconds [INVALID]<br>
</div>
</div>
</blockquote>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>[RESULT] ior-easy-read 3.078668 GiB/s : time 14.111 seconds<br>
</div>
</div>
</blockquote>
<blockquote type="cite">
<div dir="ltr">
<div>[RESULT] ior-hard-read 3.183521 GiB/s : time 0.275 seconds</div>
</div>
</blockquote>
<br>
</div>
<div>There are definitely some file layout tunables that can improve IO500 performance for these workloads.</div>
<div>See the default io500.sh file, where they are commented out by default:</div>
<div><br>
</div>
<div> # Example commands to create output directories for Lustre. Creating<br>
# top-level directories is allowed, but not the whole directory tree.<br>
#if (( $(lfs df $workdir | grep -c MDT) > 1 )); then<br>
# lfs setdirstripe -D -c -1 $workdir<br>
#fi<br>
#lfs setstripe -c 1 $workdir<br>
#mkdir $workdir/ior-easy $workdir/ior-hard<br>
#mkdir $workdir/mdtest-easy $workdir/mdtest-hard<br>
#local osts=$(lfs df $workdir | grep -c OST)<br>
# Try overstriping for ior-hard to improve scaling, or use wide striping<br>
#lfs setstripe -C $((osts * 4)) $workdir/ior-hard ||<br>
# lfs setstripe -c -1 $workdir/ior-hard<br>
# Try to use DoM if available, otherwise use default for small files<br>
#lfs setstripe -E 64k -L mdt $workdir/mdtest-easy || true #DoM?<br>
#lfs setstripe -E 64k -L mdt $workdir/mdtest-hard || true #DoM?<br>
#lfs setstripe -E 64k -L mdt $workdir/mdtest-rnd<br>
<br>
</div>
<div><br>
</div>
<div>As you can see above, the IO performance of Lustre isn't really much different than the local storage</div>
<div>performance of ZFS. You are always going to lose some percentage over the network and because</div>
<div>of the added distributed locking. That said, for the hardware that you have, it should be getting about</div>
<div>2-3GB/s per NVMe device, and up to 10GB/s over the network, so the limitation here is really ZFS.</div>
<div>It would be useful to test with ldiskfs on tje same hardware, maybe with LVM aggregating the NVMes.</div>
<div><br>
</div>
<div>
<blockquote type="cite">
<div dir="ltr">
<div>When running "./io500 ./config-minimalLUST.ini" on my lustre client, I get these performance numbers:</div>
<div>IO500 version io500-isc22_v1 (standard)<br>
</div>
</div>
</blockquote>
</div>
<blockquote type="cite">
<div>
<div dir="ltr">
<div>[RESULT] mdtest-easy-write 0.931693 kIOPS : time 31.028 seconds [INVALID]</div>
</div>
</div>
</blockquote>
<blockquote type="cite">
<div>
<div dir="ltr">
<div>[RESULT] mdtest-hard-write 0.427000 kIOPS : time 31.070 seconds [INVALID]<br>
[RESULT] find 25.311534 kIOPS : time 1.631 seconds<br>
[RESULT] mdtest-easy-stat 0.570021 kIOPS : time 50.067 seconds<br>
[RESULT] mdtest-hard-stat 1.834985 kIOPS : time 7.998 seconds<br>
[RESULT] mdtest-easy-delete 1.715750 kIOPS : time 17.308 seconds<br>
[RESULT] mdtest-hard-read 1.006240 kIOPS : time 13.759 seconds<br>
[RESULT] mdtest-hard-delete 1.624117 kIOPS : time 8.910 seconds<br>
[SCORE ] Bandwidth 2.271383 GiB/s : IOPS 1.526825 kiops : TOTAL 1.862258 [INVALID]<br>
</div>
<div><br>
</div>
<div>When running "./io500 ./config-minimalLOCAL.ini" on a singular locally mounted ZFS pool I get the following performance numbers:<br>
</div>
<div>IO500 version io500-isc22_v1 (standard)</div>
</div>
</div>
</blockquote>
<blockquote type="cite">
<div>
<div dir="ltr">
<div>[RESULT] mdtest-easy-write 47.979181 kIOPS : time 1.838 seconds [INVALID]<br>
[RESULT] mdtest-hard-write 27.801814 kIOPS : time 2.443 seconds [INVALID]<br>
[RESULT] find 1384.774433 kIOPS : time 0.074 seconds<br>
[RESULT] mdtest-easy-stat 343.232733 kIOPS : time 1.118 seconds<br>
[RESULT] mdtest-hard-stat 333.241620 kIOPS : time 1.123 seconds<br>
[RESULT] mdtest-easy-delete 45.723381 kIOPS : time 1.884 seconds<br>
[RESULT] mdtest-hard-read 73.637312 kIOPS : time 1.546 seconds<br>
[RESULT] mdtest-hard-delete 42.191867 kIOPS : time 1.956 seconds<br>
[SCORE ] Bandwidth 1.578256 GiB/s : IOPS 114.726763 kiops : TOTAL 13.456159 [INVALID]<br>
</div>
</div>
</div>
</blockquote>
<div><br>
</div>
Definitely the metadata performance is lower here, because each Lustre file has to create (at least)</div>
<div>two objects (one on MDT, one or more on OST(s)) and then write and access them again.</div>
<div>Lustre metadata performance would definitely benefit from enabling PFL and Data-on-MDT (per</div>
<div>above default commands), since it only needs to do the MDT create/access.</div>
<div><br>
<blockquote type="cite">
<div dir="ltr">
<div>I have run an iperf3 test and I was able to reach speeds of around 40GbE so I don't think the network links are the issue (Maybe it's something to do with lnet?)</div>
<div><br>
</div>
<div>
<div>If anyone more knowledgeable than me would please educate me on why the performance of the local three disk ZFS is more performant than the lustre FS.</div>
<div>I'm very new to this kind of benchmarking so it may also be that I am misinterpreting the results/ not applying the test correctly.</div>
</div>
</div>
</blockquote>
</div>
<div><br>
</div>
<br>
<div>
<div dir="auto" style="letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="auto" style="letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="auto" style="letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="auto" style="letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="auto" style="letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="auto" style="letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div>Cheers, Andreas</div>
<div>--</div>
<div>Andreas Dilger</div>
<div>Lustre Principal Architect</div>
<div>Whamcloud</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
</div>
<br>
<br>
</div>
<br>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
<br>
<div>
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div>Cheers, Andreas</div>
<div>--</div>
<div>Andreas Dilger</div>
<div>Lustre Principal Architect</div>
<div>Whamcloud</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
</div>
<br>
<br>
</div>
<br>
</div>
</div>
</div>
</blockquote></div>