[lustre-discuss] Large file read performance degradation from multiple OST's

[Vicker, Darby (JSC-EG311)]

Hello,

We recently noticed that the large file read performance on our 2.9 LFS is dramatically worse than it used to be.  The attached plot is the result of a test script that uses dd to write a large file (50GB) to disk, read that file and then copy it to a 2nd file to test write, read and read/write speeds for large files for various stripe sizes and counts.  The two sets of data on this plot are on the same server and client hardware.  The LFS was originally built and formatted with 2.8.0 but we eventually upgraded to 2.9.0 on the servers and clients.  The behavior we are used to seeing is increasing performance as you increase the stripe count with a peak in performance around 4 or 6 OST's and a degradation after that as more OST's are used.  This is what we saw under 2.8.  With 2.9 we still get very good write performance (almost line rate on our 10 GbE clients).  But for reads we see extremely good performance with a single OST and significantly degraded performance for multiple OST's – red lines in the plots.  We searched a little on JIRA and think this might be related.

https://jira.hpdd.intel.com/browse/LU-9214

I think this is a client side patch – we tried a 2.9.0 client compile with this patch but it did not help.  If this was supposed be a server side patch, please let me know.

We contemplating rolling back to 2.8 but I'm a little weary about downgrading.  It would be a little hard because lustre version 2.8 doesn't compile on the older RHEL kernels so we'd have to roll back the OS too.  Besides that, I'd like to hear any thoughts on the cons of downgrading lustre from 2.9 to 2.8.  I think we'd probably be OK since the LFS was originally formatted with 2.8.

Any other thoughts?

Thanks,
Darby

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20170522/078f1f40/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: lustre_performance.pdf
Type: application/pdf
Size: 11462 bytes
Desc: lustre_performance.pdf
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20170522/078f1f40/attachment.pdf>