[lustre-discuss] Spiking OSS load?

Jason Williams jasonw at jhu.edu
Tue Aug 1 12:07:43 PDT 2017


Hello,

First off, the Lustre that we run here is one that was installed by Intel, so figuring out the exact version seems to be a table lookup on a table internal to Intel, but I'm told it's probably 2.5-ish...

Recently, I finally installed a monitoring system on my OSS/MDS servers.  And over the last week or so, the OSS servers have been spiking to a 100+ load average (sometimes much higher.)  They are not going unresponsive, from what I can tell, and the processes that are causing it seem to be the ll_ost_io07_XXX processes (where XXX is a number) because they are going into "D" state (io wait state)

I recently attended the "The 3rd International Workshop on the Lustre Ecosystem" (GREAT WORKSHOP!!) and via a couple of the talks it got me thinking about tunables.

One tunable in particular was the ost.threads_max.  That guidance on the lustre.org says (1/128MB * num_cpu) which, on my system, works out to well over the max thread count allowable of 512. So my OSS machines are all set to a threads_max of 512 and indeed on all of the machines, the threads_started is 512. (It's a VERY busy file system)

This leads me to the following questions (and possibly more, but let's start with these):


1)      Is 512 threads a reasonable setting or should it be lower?

2)      Is high load "normal" if the file system is under heavy use?  At the time I see a lot of open and attr calls which I thought would load the MDS over the OSS... but my under-the-hood understanding is limited at best.

3)      Should I be looking at other tunables?

I realize the information provided in this initial email is limited as well, so if you are curious about anything else, please let me know what else might be interesting.

Oh and as for the MDS/OSS setup, here's a brief overview too:

2x MDS in failover mode with one MDT
12x OSS in fail over pairs with 12 OST per pair 6 running on each OSS. (72 OST, 6 active on each of 12 OSS for load sharing.)
                Each OSS pair is hooked to the same set of 2x RAID Array (Dell MD3460)


--
Jason Williams
Assistant Director
Systems and Data Center Operations.
Maryland Advanced Research Computing Center (MARCC)
Johns Hopkins University
jasonw at jhu.edu<mailto:jasonw at jhu.edu>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20170801/21c190fa/attachment.htm>


More information about the lustre-discuss mailing list