[lustre-discuss] Read performance bad, telepathy in Lustre

[Thomas Roth]

Hi all,

Lustre 2.10.6, 45 OSS with 7 OSTs each on ZFS 0.7.9, 3 MDTs (ldiskfs), clients 2.10 and 2.12. 
Infiniband network, Mellanox FDR w half bisectional bandwidth.

A sample of ~250.000 files, stripe count 1, average size 100 MB. is read with dd, output > /dev/null.

The location of the files has been recorded, from this we have drawn up separate file lists for each OSS.


In the first run, one client reads the files on one OSS and gets a read performance X, e.g. 2 GB/s.

In the second run, this setup is simply multiplied by 10 or 40: Client 1 still reads from OSS 1, 
Client 2 works with the files on OSS2, client 3 with OSS 3, ...

With only 12 pairs of this kind we see 2 or 3 pairs whose performance dropsto < 500 MB/s. The other 
pairs keep the read rate as seen before. Once they have finished, the remaining 2 -3 pairs jump back 
to original performance.

When the runs are repeated, the affected OSS are not the same as before.

This should exclude effects of bad hardware: servers, disks, cables, switches.

Since this behaviour is reproducible, the effects of interactions with other jobs/users can also be 
excluded.




By now I am able to reproduce the behavior on a test system, same configuration, with just 2 
client-OSS pairs, nobody else on there.

56 parallel dd processes on client 1, reading files on server 1: 440 MB/s
56 parallel dd processes on client 2, reading files on server 2: 1.6 GB/s

Then kill all processes on client 2. Client 1 continues, rising to 1.1 GB/s


These processes are not even visible on the MDS of this system, and from all I understand the metadata 
server should be the only connecting element between the two pairs?
How do they know about each other, who, what tells client-1-server-1 to keep it low while client-2 is 
working on server-1?

Curioser and curioser,
Thomas




-- 
--------------------------------------------------------------------
Thomas Roth
Department: Informationstechnologie
Location: SB3 2.291
Phone: +49-6159-71 1453  Fax: +49-6159-71 2986


GSI Helmholtzzentrum für Schwerionenforschung GmbH
Planckstraße 1, 64291 Darmstadt, Germany, www.gsi.de

Commercial Register / Handelsregister: Amtsgericht Darmstadt, HRB 1528
Managing Directors / Geschäftsführung:
Professor Dr. Paolo Giubellino, Jörg Blaurock
Chairman of the Supervisory Board / Vorsitzender des GSI-Aufsichtsrats:
State Secretary / Staatssekretär Dr. Volkmar Dietz