<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<span style="font-size: 14px;" class="">Whether there are problems with a large number of OSS and/or MDS nodes depends on whether you are using TCP or IB networking.</span>
<div class=""><span style="font-size: 14px;" class=""><br class="">
</span></div>
<div class=""><span style="font-size: 14px;" class="">With socklnd there are 3 TCP connections per client-server pair (bulk read, bulk write, and small message) so the maximum you could have would be around (65536 - 1024)/3 = 21500 (or likely fewer) clients
or servers, unless you also configured LNet routers in between (which would allow more clients, but not more servers). That isn't a limitation for most deployments, but at least one known limitation. For IB there is no such connection limit that I'm aware
of.</span></div>
<div class=""><span style="font-size: 14px;" class=""><br class="">
</span></div>
<div class=""><span style="font-size: 14px;" class="">There are likely other factors such as memory consumption per target, but I don't think that would be the first thing to cause problems on modern systems with hundreds of GB of RAM.</span></div>
<div class=""><span style="font-size: 14px;" class=""><br class="">
</span></div>
<div class=""><span style="font-size: 14px;" class="">Cheers, Andreas<br class="">
</span>
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Oct 4, 2019, at 01:45, Degremont, Aurelien <<a href="mailto:degremoa@amazon.com" class="">degremoa@amazon.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span lang="EN-US" class="">Thanks for this info. But actually I was really looking at the number of OSS, not OSTs :)<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span lang="EN-US" class="">This is really more how Lustre client nodes and MDT will cope with very large number of OSSes.<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span lang="EN-US" class=""><o:p class=""> </o:p></span></div>
<div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(181, 196, 223); padding: 3pt 0cm 0cm;" class="">
<div style="margin: 0cm 0cm 0.0001pt 35.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<b class=""><span style="font-size: 12pt;" class="">De :<span class="Apple-converted-space"> </span></span></b><span style="font-size: 12pt;" class="">Andreas Dilger <<a href="mailto:adilger@whamcloud.com" style="color: purple; text-decoration: underline;" class="">adilger@whamcloud.com</a>><br class="">
<b class="">Date :<span class="Apple-converted-space"> </span></b>vendredi 4 octobre 2019 à 04:54<br class="">
<b class="">À :<span class="Apple-converted-space"> </span></b>"Degremont, Aurelien" <<a href="mailto:degremoa@amazon.com" style="color: purple; text-decoration: underline;" class="">degremoa@amazon.com</a>><br class="">
<b class="">Objet :<span class="Apple-converted-space"> </span></b>Re: [lustre-discuss] Limit to number of OSS?<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0cm 0cm 0.0001pt 35.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
</div>
<div style="margin: 0cm 0cm 0.0001pt 35.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
On Oct 3, 2019, at 07:55, Degremont, Aurelien <<a href="mailto:degremoa@amazon.com" style="color: purple; text-decoration: underline;" class="">degremoa@amazon.com</a>> wrote:<o:p class=""></o:p></div>
<div class="">
<blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class="">
<div style="margin: 0cm 0cm 0.0001pt 35.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div class="">
<div class="">
<div style="margin: 0cm 0cm 0.0001pt 35.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Hello all!<br class="">
<br class="">
This doc from the wiki says "Lustre can support up to 2000 OSS per file system" (<a href="http://wiki.lustre.org/Lustre_Server_Requirements_Guidelines" style="color: purple; text-decoration: underline;" class="">http://wiki.lustre.org/Lustre_Server_Requirements_Guidelines</a>).<br class="">
<br class="">
I'm a bit surprised by this statement. Does somebody has information about the upper limit to the number of OSSes?<br class="">
Or what could be the scaling limitator for this number of OSS? Network limit? Memory consumption? Other?<o:p class=""></o:p></div>
</div>
</div>
</blockquote>
</div>
<div class="">
<div style="margin: 0cm 0cm 0.0001pt 35.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
</div>
<div style="margin: 0cm 0cm 0.0001pt 35.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
That's likely a combination of a bit of confusion and a bit of safety on the part of Intel writing that document.<o:p class=""></o:p></div>
<div class="">
<div style="margin: 0cm 0cm 0.0001pt 35.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
</div>
<div class="">
<div style="margin: 0cm 0cm 0.0001pt 35.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
The Lustre Operations Manual writes:<o:p class=""></o:p></div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class="">
<div style="margin: 0cm 0cm 0.0001pt 35.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 10pt; font-family: Times;" class="">Although a single file can only be striped over 2000 objects, Lustre file systems can have thousands of OSTs. The I/O bandwidth to access a single file is the aggregated I/O bandwidth to the objects
in a file, which can be as much as a bandwidth of up to 2000 servers. On systems with more than 2000 OSTs, clients can do I/O using multiple files to utilize the full file system bandwidth. </span><o:p class=""></o:p></div>
</blockquote>
<div class="">
<div style="margin: 0cm 0cm 0.0001pt 35.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
I think PNNL once tested up to 4000 OSTs, and I think the compile-time limit is/was 8000 OSTs (maybe it was made dynamic, I don't recall offhand), but the current code could _probably_ handle up to 65000 OSTs without significant problems. Beyond that, there
is the 16-bit OST index limit in the filesystem device labels and the __u16 lov_user_md_v1->lmm_stripe_offset to specify the starting OST index for "lfs setstripe", but that could be overcome with some changes.<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0cm 0cm 0.0001pt 35.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
</div>
<div class="">
<div style="margin: 0cm 0cm 0.0001pt 35.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Given OSTs are starting to approach 1PB with large drives and declustered-parity RAID, this would get us in the range 8-65EB, which is over 2^64 bytes (16EB), so I don't think it is an immediate concern. Let me know if you have any trouble with a 9000-OST
filesystem... :-)<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0cm 0cm 0.0001pt 35.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div style="margin: 0cm 0cm 0.0001pt 35.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="" class="">Cheers, Andreas<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0cm 0cm 0.0001pt 35.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="" class="">--<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0cm 0cm 0.0001pt 35.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="" class="">Andreas Dilger<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0cm 0cm 0.0001pt 35.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="" class="">Principal Lustre Architect<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0cm 0cm 0.0001pt 35.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="" class="">Whamcloud</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div>Cheers, Andreas</div>
<div>--</div>
<div>Andreas Dilger</div>
<div>Principal Lustre Architect</div>
<div>Whamcloud</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br class="">
</div>
</body>
</html>