<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
On Oct 3, 2019, at 07:55, Degremont, Aurelien <<a href="mailto:degremoa@amazon.com" class="">degremoa@amazon.com</a>> wrote:<br class="">
<div>
<blockquote type="cite" class=""><br class="Apple-interchange-newline">
<div class="">
<div class="">Hello all!<br class="">
<br class="">
This doc from the wiki says "Lustre can support up to 2000 OSS per file system" (<a href="http://wiki.lustre.org/Lustre_Server_Requirements_Guidelines" class="">http://wiki.lustre.org/Lustre_Server_Requirements_Guidelines</a>).<br class="">
<br class="">
I'm a bit surprised by this statement. Does somebody has information about the upper limit to the number of OSSes?<br class="">
Or what could be the scaling limitator for this number of OSS? Network limit? Memory consumption? Other?<br class="">
</div>
</div>
</blockquote>
</div>
<div class=""><br class="">
</div>
That's likely a combination of a bit of confusion and a bit of safety on the part of Intel writing that document.
<div class=""><br class="">
</div>
<div class="">The Lustre Operations Manual writes:</div>
<div class="">
<div class="page" title="Page 37">
<div class="layoutArea">
<div class="column">
<p class=""></p>
<blockquote type="cite" class=""><span style="font-size: 10.000000pt; font-family: 'Times'" class="">Although a single file can only be striped over 2000 objects, Lustre file systems can have thousands of OSTs. The I/O bandwidth to access a single file is the
aggregated I/O bandwidth to the objects in a file, which can be as much as a bandwidth of up to 2000 servers. On systems with more than 2000 OSTs, clients can do I/O using multiple files to utilize the full file system bandwidth. </span></blockquote>
<p></p>
<div class="">I think PNNL once tested up to 4000 OSTs, and I think the compile-time limit is/was 8000 OSTs (maybe it was made dynamic, I don't recall offhand), but the current code could _probably_ handle up to 65000 OSTs without significant problems. Beyond
that, there is the 16-bit OST index limit in the filesystem device labels and the __u16 lov_user_md_v1->lmm_stripe_offset to specify the starting OST index for "lfs setstripe", but that could be overcome with some changes.</div>
<div class=""><br class="">
</div>
<div class="">Given OSTs are starting to approach 1PB with large drives and declustered-parity RAID, this would get us in the range 8-65EB, which is over 2^64 bytes (16EB), so I don't think it is an immediate concern. Let me know if you have any trouble with
a 9000-OST filesystem... :-)</div>
<div class=""><br class="">
</div>
</div>
</div>
</div>
</div>
<div class="">
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div>Cheers, Andreas</div>
<div>--</div>
<div>Andreas Dilger</div>
<div>Principal Lustre Architect</div>
<div>Whamcloud</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br class="">
</div>
</body>
</html>