[Lustre-discuss] Lustre write performances 4 times faster after removing SD_IOSTATS from kernel

Peter Grandi pg_lus at lus.for.sabi.co.UK
Wed Feb 15 03:49:42 PST 2012

>>> On Tue, 14 Feb 2012 14:55:03 -0500, "Charland, Denis"
>>> <Denis.Charland at imi.cnrc-nrc.gc.ca> said:

> [ ... ] On a Lustre 1.8.7 client, I untared a tarfile of a
> directory containing about 225,000 files (6.4GB).

Writing lots of small files is not the most appropriate.
Also, it may make a big difference where the 'tar' file was
stored, and in how many directories the files ended up.

> [ ... ] it took about 4 minutes to untar the tarfile when
> running the kernel without SD_IOSTATS compared to about 16
> minutes when running the kernel with SD_IOSTATS. [ ... ]

On the client or the OSS or MDS?

Anyhow stats collection in itself should not make a difference.
There may something else involved. For example logging volume,
which is known to make a significant difference.  Perhaps also
whether barriers (or 'sync') are enabled, which may conceivably
remotely related to 'SD_IOSTATS'. Possibly you may have a stats
collector program that results in a higher load on the MDS if
'SD_IOSTATS' is enabled.

But looking at your numbers, in the faster case you were able to
commit ~950 inodes/s and 26 MB/s, and even with Lustre not being
the best for small files, that's not so good. 

Overall your report is vague and not very useful. Have you
looked for example at 'vmstat' and 'iostat -dx' rates on the
MDS/OSS hosts? Have you looked at the logs on both client and
servers? Checked mount options, 'sync' and barriers?

More information about the lustre-discuss mailing list