<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:10.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">Hello Andreas,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Thanks for your response.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Yes our hardware is a bit old. It was acquired six years ago, in April 2016. But we use a quite recent versions of Lustre, RHEL kernel, OS and MOFED. Here are some details:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:11.0pt">Presently:<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:72.0pt"><span style="font-size:11.0pt">Lustre: 2.15.4 compiled from git repository<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:72.0pt"><span style="font-size:11.0pt">MOFED: 23.10-2.1.3.1<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:72.0pt"><span style="font-size:11.0pt">File server node: RHEL 8.9 kernel 4.18.0-513.24.1 patched for Lustre, , all other non-kernel RPMs being updated on a weekly basis with latest RHEL 8.10<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:72.0pt"><span style="font-size:11.0pt">Head/Compute nodes: RHEL 9.3 kernel 5.14.0-362.24.1, , all other non-kernel RPMs being updated on a weekly basis with latest RHEL 9.5<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:72.0pt"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:11.0pt">After a planned update next week:<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:72.0pt"><span style="font-size:11.0pt">Lustre: git commit
</span><span style="font-size:9.0pt;font-family:"Courier New";color:black;background:white">a71369eb9cb0aa89ede41cb01b2cd9cdcd8e9680</span><span style="font-size:11.0pt"> (2.15.6 + 3 patches: LU-18085 llite: use RCU to protect the dentry_data) compiled from
git repository<br>
MOFED: 24.10-2.1.8.0<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:72.0pt"><span style="font-size:11.0pt">File server node: RHEL 8.10 kernel 4.18.0-553.27.1 patched for Lustre, all other non-kernel RPMs being updated on a weekly basis with latest RHEL 8.10<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:72.0pt"><span style="font-size:11.0pt">Head/Compute nodes: RHEL 9.5 kernel 5.14.0-503.14.1, all other non-kernel RPMs being updated on a weekly basis with latest RHEL 9.5+<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><br>
When we will add the two new OSTs, in two weeks maybe, we plan to have our compute and head nodes powered OFF of other reasons. So the race condition is absolutely not a potential problem in our case. But thanks for explaining this potential race problem.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><br>
Now I have another question: it seems that the OSS contact the MGS server to announce their OST and the MGS simply accept them. I am a bit surprised to see that nothing needs to be done MGS side to restrict which OSS server can offer OSTs. I guess it is like
that to keep the basic scenario simple. But if we want to improve security, is there some mechanism to restrict which server can provide an OST ? In our case it is very simple since MGS, MDS and OSTs are all running on the same server.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Thanks, <br>
<br>
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Martin<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:11.0pt">From:</span></b><span style="font-size:11.0pt"> Andreas Dilger <adilger@ddn.com>
<br>
<b>Sent:</b> April 5, 2025 3:00<br>
<b>To:</b> Audet, Martin <Martin.Audet@cnrc-nrc.gc.ca><br>
<b>Cc:</b> lustre-discuss@lists.lustre.org; Raymond, Stephane <Stephane.Raymond@cnrc-nrc.gc.ca><br>
<b>Subject:</b> EXT: Re: [lustre-discuss] Is it that simple to add a pair of new OSTs ?<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><b>***Attention*** This email originated from outside of the NRC. ***Attention*** Ce courriel provient de l'extérieur du CNRC.</b><span style="font-size:11.0pt"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">It really is that simple. <o:p>
</o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">You didn't mention what version you are using, but based on the hardware and sizes I would assume it is not the latest. <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">As such, there is a race that if clients are actively creating and writing new files at the instant the OSTs are added, those files may be inaccessible on some clients for a few seconds until the new OSTs
are visible on all clients. <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"> If the clients accessing the filesystem are quiesced during the initial mount then there is no race. Very recent servers and clients have fixed this race. <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">Cheers, Andreas<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><br>
<br>
<o:p></o:p></span></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:11.0pt">On Apr 3, 2025, at 15:12, Audet, Martin via lustre-discuss <<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a>> wrote:<o:p></o:p></span></p>
</blockquote>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<div id="divtagdefaultwrapper">
<div id="divtagdefaultwrapper">
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">Hello Lustre community,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">We are operating a small HPC cluster (576 compute cores) using a small Lustre parallel filesystem (64 TB) connected by Infiniband EDR network. The Lustre filesystem is implemented by a single HPE
DL380 Gen10 server acting as MGS, MDS and OSS. It has two 32 TB OSTs (HPE MSA 2050). As new space is required, we will soon install 160 TB of additional storage implemented two 80 TB OSTs (HPE MSA 2060).<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">We looked in the Lustre documentation (10.2.1. Scaling the Lustre File System:
<a href="https://doc.lustre.org/lustre_manual.xhtml#idm140220261007664">https://doc.lustre.org/lustre_manual.xhtml#idm140220261007664</a>) and made tests with small VMs. It appear that in our case adding this new storage would be very simple. From what we understand
we should do something like this:<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
</div>
</div>
<blockquote style="margin-left:30.0pt;margin-right:0cm">
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas;color:black"># Create mount points for the new OSTs<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas;color:black">mkdir /mnt/ost{2,3}<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas;color:black"># The MGS is running on the same node as the OSTs <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas;color:black">mgs_node="$(sed -n -e 's/^ *- *nid: *//; T; p' < /etc/lnet.conf)"<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas;color:black"># Set the devices corresponding to the new OSTs using invariant names<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas;color:black">ost2_device=/dev/disk/by-path/...<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas;color:black">ost3_device=/dev/disk/by-path/...<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas;color:black"># Create the file systems on the new OSTs<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas;color:black">mkfs.lustre --fsname=lustrevm --mgsnode=$mgs_node --ost --index=2 $ost2_device <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas;color:black">mkfs.lustre --fsname=lustrevm --mgsnode=$mgs_node --ost --index=3 $ost3_device <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas;color:black"># Update fstab<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas;color:black">cat >> /etc/fstab << _EOF_<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas;color:black">$ost2_device /mnt/ost2 lustre defaults,_netdev 0 0<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas;color:black">$ost3_device /mnt/ost3 lustre defaults,_netdev 0 0<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas;color:black">_EOF_<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas;color:black"># Mount the new OSTs<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas;color:black">mount /mnt/ost2<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas;color:black">mount /mnt/ost3<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Consolas;color:black"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
</div>
</div>
</blockquote>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">This appears too simple. Are we missing something ? Will the new files created by the clients use the four OSTs with no additional effort ?<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">Thanks in advance !<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">Martin Audet<o:p></o:p></span></p>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt">_______________________________________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><o:p></o:p></span></p>
</div>
</blockquote>
</div>
</div>
</div>
</body>
</html>