<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Thomas,
<div class="">If you are positive that the two sets of clients are not reading files on other on the OSTs, I don't think there is anything at the Lustre level that communicates between OSSes to balance traffic or anything like that.</div>
<div class=""><br class="">
</div>
<div class="">One possibility is congestion control at the network level, possibly at the switch?</div>
<div class=""><br class="">
</div>
<div class="">Cheers, Andreas<br class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Jan 23, 2020, at 08:01, Thomas Roth <<a href="mailto:t.roth@gsi.de" class="">t.roth@gsi.de</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="">Hi all,<br class="">
<br class="">
Lustre 2.10.6, 45 OSS with 7 OSTs each on ZFS 0.7.9, 3 MDTs (ldiskfs), clients 2.10 and 2.12. Infiniband network, Mellanox FDR w half bisectional bandwidth.<br class="">
<br class="">
A sample of ~250.000 files, stripe count 1, average size 100 MB. is read with dd, output > /dev/null.<br class="">
<br class="">
The location of the files has been recorded, from this we have drawn up separate file lists for each OSS.<br class="">
<br class="">
<br class="">
In the first run, one client reads the files on one OSS and gets a read performance X, e.g. 2 GB/s.<br class="">
<br class="">
In the second run, this setup is simply multiplied by 10 or 40: Client 1 still reads from OSS 1, Client 2 works with the files on OSS2, client 3 with OSS 3, ...<br class="">
<br class="">
With only 12 pairs of this kind we see 2 or 3 pairs whose performance dropsto < 500 MB/s. The other pairs keep the read rate as seen before. Once they have finished, the remaining 2 -3 pairs jump back to original performance.<br class="">
<br class="">
When the runs are repeated, the affected OSS are not the same as before.<br class="">
<br class="">
This should exclude effects of bad hardware: servers, disks, cables, switches.<br class="">
<br class="">
Since this behaviour is reproducible, the effects of interactions with other jobs/users can also be excluded.<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
By now I am able to reproduce the behavior on a test system, same configuration, with just 2 client-OSS pairs, nobody else on there.<br class="">
<br class="">
56 parallel dd processes on client 1, reading files on server 1: 440 MB/s<br class="">
56 parallel dd processes on client 2, reading files on server 2: 1.6 GB/s<br class="">
<br class="">
Then kill all processes on client 2. Client 1 continues, rising to 1.1 GB/s<br class="">
<br class="">
<br class="">
These processes are not even visible on the MDS of this system, and from all I understand the metadata server should be the only connecting element between the two pairs?<br class="">
How do they know about each other, who, what tells client-1-server-1 to keep it low while client-2 is working on server-1?<br class="">
<br class="">
Curioser and curioser,<br class="">
Thomas<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
-- <br class="">
--------------------------------------------------------------------<br class="">
Thomas Roth<br class="">
Department: Informationstechnologie<br class="">
Location: SB3 2.291<br class="">
Phone: +49-6159-71 1453 Fax: +49-6159-71 2986<br class="">
<br class="">
<br class="">
GSI Helmholtzzentrum für Schwerionenforschung GmbH<br class="">
Planckstraße 1, 64291 Darmstadt, Germany, <a href="http://www.gsi.de" class="">www.gsi.de</a><br class="">
<br class="">
Commercial Register / Handelsregister: Amtsgericht Darmstadt, HRB 1528<br class="">
Managing Directors / Geschäftsführung:<br class="">
Professor Dr. Paolo Giubellino, Jörg Blaurock<br class="">
Chairman of the Supervisory Board / Vorsitzender des GSI-Aufsichtsrats:<br class="">
State Secretary / Staatssekretär Dr. Volkmar Dietz<br class="">
<br class="">
_______________________________________________<br class="">
lustre-discuss mailing list<br class="">
<a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a><br class="">
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org<br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div>Cheers, Andreas</div>
<div>--</div>
<div>Andreas Dilger</div>
<div>Principal Lustre Architect</div>
<div>Whamcloud</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br class="">
</div>
</body>
</html>