<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
I would recommend *against* mounting all 175 OSTs at the same time. There are (or at least were*) some issues with the MGS registration RPCs timing out when too many config changes happen at once. Your "mount and wait 2 sec" is more robust and doesn't take
very much time (a few minutes) vs. having to restart if some of the OSTs have problems registering. Also, the config logs will have the OSTs in a nice order, which doesn't affect any functionality, but makes it easier for the admin to see if some device is
connected in "lctl dl" output.
<div class=""><br class="">
</div>
<div class="">Cheers, Andreas<br class="">
<div><br class="">
</div>
<div><br class="">
</div>
<div>[*] some fixes have landed over time to improve registration RPC resend.</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Jan 8, 2024, at 11:57, Thomas Roth via lustre-discuss <<a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="">Yes, sorry, I meant the actual procedure of mounting the OSTs for the first time.<br class="">
<br class="">
Last year I did that with 175 OSTs - replacements for EOL hardware. All OSTs had been formatted with a specific index, so probably creating a suitable /etc/fstab everywhere and sending a 'mount -a -t lustre' to all OSTs simultaneously would have worked.<br class="">
<br class="">
But why the hurry? Instead, I logged in to my new OSS, mounted the OSTs with 2 sec between each mount command, watched the OSS log, watched the MDS log, saw the expected log messages, proceeded to the new OSS - all fine ;-) Such a leisurely approach takes
its time, of course.<br class="">
<br class="">
Once all OSTs were happily incorporated, we put the max_create_count (set to 0 before) to some finite value and started file migration. As long as the migration is more effective, faster, than the users's file creations, the result should be evenly filled OSTs
with a good mixture of files (file sizes, ages, types).<br class="">
<br class="">
<br class="">
Cheers<br class="">
Thomas<br class="">
<br class="">
On 1/8/24 19:07, Andreas Dilger wrote:<br class="">
<blockquote type="cite" class="">The need to rebalance depends on how full the existing OSTs are. My recommendation if you know that the data will continue to grow is to add new OSTs when the existing ones are at 60-70% full, and add them in larger groups
rather than one at a time.<br class="">
Cheers, Andreas<br class="">
<blockquote type="cite" class="">On Jan 8, 2024, at 09:29, Thomas Roth via lustre-discuss <<a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a>> wrote:<br class="">
<br class="">
\ufeffJust mount the OSTs, one by one and perhaps not if your system is heavily loaded. Follow what happens in the MDS log and the OSS log.<br class="">
And try to rebalance the OSTs fill levels afterwards - very empty OSTs will attract all new files, which might be hot and direct your users's fire to your new OSS only.<br class="">
<br class="">
Regards,<br class="">
Thomas<br class="">
<br class="">
<blockquote type="cite" class="">On 1/8/24 15:38, Backer via lustre-discuss wrote:<br class="">
Hi,<br class="">
Good morning and happy new year!<br class="">
I have a quick question on extending a lustre file system. The extension is performed online. I am looking for any best practices or anything to watchout while doing the file system extension. The file system extension is done adding new OSS and many OSTs within
these servers.<br class="">
Really appreciate your help on this.<br class="">
Regards,<br class="">
_______________________________________________<br class="">
lustre-discuss mailing list<br class="">
<a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a><br class="">
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org<br class="">
</blockquote>
_______________________________________________<br class="">
lustre-discuss mailing list<br class="">
<a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a><br class="">
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org<br class="">
</blockquote>
</blockquote>
_______________________________________________<br class="">
lustre-discuss mailing list<br class="">
<a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a><br class="">
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org<br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div>Cheers, Andreas</div>
<div>--</div>
<div>Andreas Dilger</div>
<div>Lustre Principal Architect</div>
<div>Whamcloud</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
<br class="Apple-interchange-newline">
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br class="">
</div>
</body>
</html>