[lustre-discuss] Meaning of 'slow creates' messages on MDS

[Russell Dekema]

Greetings,

We have been having various kinds of trouble with our Lustre
filesystem lately; right now the main problem we are having is
intermittent severe slowness (such as 30 seconds for an 'ls' of a
directory containing 100 files to return) when 'cd' and 'ls'ing around
our Lustre filesystem.

As far as I can tell [although I don't think we have perfect
visibility into this], our underlying metadata and object storage
arrays are not overloaded, either in general or specifically when we
see the (presumably) metadata-related slowdowns.

That said, during the slow periods, the load average on our metadata
server is usually in the low single digits, and the load averages on
our OSSes tend to be in the hundreds.

I have noticed a number of error messages like the following in the
system log on the metadata server, but I don't know quite how to
interpret them:

May 28 15:00:40 scr-mds0 kernel: : Lustre:
scratch-OST001e-osc-MDT0000: slow creates,
last=[0x1001e0000:0x58858e1:0x0], next=[0x1001e0000:0x58858e1:0x0],
reserved=0, syn_changes=173, syn_rpc_in_progress=100, status=0

(The OST mentioned in these messages varies; the MDT is always 0000
since that is our only MDT.)

Can anyone explain how to interpret these error messages? For example,
is the "slow create" in question being caused by problems/delays on
the MDT or on the OST?

Thanks in advance,
Rusty Dekema
University of Michigan
Advanced Research Computing - Technical Services