[Lustre-discuss] MDT extremely slow after restart

[Thomas Roth]

Hi all,

we are suffering from a sever metadata performance degradation on our 1.8.4 cluster and are pretty clueless.
- We moved the MDT to a new hardware, since the old one was failing
- We increased the size of the MDT with 'resize2fs' (+ mounted it and saw all the files)
- We found the performance of the new mds dreadful
- We restarted the MDT on the old hardware with the failed RAID controller replaced, but without doing anything with OSS or clients
The machine crashed three minutes after recovery was over
- Moved back to the new hardware, but the system was now pretty messed up: persistent  "still busy with N RPCs" and some "going back to sleep"
messages (by the way, there is no way to find out what these RPCs are, and how to kill them? Of course I wouldn't mind switching off some clients or
even rebooting some OSS if I only new which ones...)
- Shut down the entire cluster, writeconf, restart without any client mounts - worked fine
- Mounted Lustre and tried to "ls" a directory with 100 files:   takes several minutes(!)
- Being patient and then trying the same on a second client:     takes msecs.

I have done complete shutdowns before, lastly to upgrade from 1.6 to 1.8, then without writeconf and without performance loss. Before to change the
IPs of all servers (moving into a subnet), with writeconf, but without recollection of the metadata behavior afterwards.
It is clear that after writeconf some information has to be regenerated, but this is really extreme - also normal?

The MDT now behaves more like an xrootd master which makes first contact to its file servers and has to read in the entire database (would be nice to
have in Lustre to regenerate the MDT in case of desaster ;-) ).
Which caches are being filled now when I ls through the cluster? May I expect the MDT to explode once it has learned about a certain percentage of the
system? ;-) I mean, we have 100 mio files now and the current MDT hardware has just 32GB memory...
In any case this is not the Lustre behavior we are used to.

Thanks for any hints,
Thomas