<div dir="ltr"><div>what does tune2fs report for /dev/sdb on the MDS?</div><div><br></div><div>(Also sorry, this somehow got lost in my inbox)<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Nov 22, 2021 at 8:57 AM STEPHENS, DEAN - US <<a href="mailto:dean.stephens@caci.com">dean.stephens@caci.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div style="overflow-wrap: break-word;" lang="EN-US">
<div class="gmail-m_-1734618617407989705WordSection1">
<p class="MsoNormal">Colin and Andreas, so to clarify some points for you, This is what I am seeing:<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal" style="text-indent:0.5in">Rpm -qa | grep lustre<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in">Kmod_lustre-2.12.6-1.el7.x86_64<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in">Lustre-iokit-2.12.6-1.el7.x86_64<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in">Lustre_test-2.12.6-1.el7.x86_64<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in">Kernel-devel-3.10.0-1160.2.el7_lustre.x86_64<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in">Lustre-osd-ldiskfs-2.12.6-1.el7.x86_64<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in">Kmod-lustre-osd-ldiskfs-2.12.6-1.el7.x86_64<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in">Kmod-lustre-tests-2.12.6-1.el7.x86_64<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in">lustre-resource-agents-2.12.6-1.el7.x86_64<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in">kernel-3.10.0-1160.2.el7_lustre.x86_64<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in">lustre-2.12.6-1.el7.x86_64<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal" style="text-indent:0.5in">rpm -qa | grep e2fs<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in">e2fsprogs-libs-1.45.6.wc1-0.el7.x86_64<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in">e2fsprogs-1.45.6.wc1-0.el7.x86_64<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">With all of that installed and the successful running and clean up of the llmount.sh and llmountcleanup.sh I am still getting the errors:<u></u><u></u></p>
<p class="MsoNormal">“Unable to mount /dev/sdb: Invalid argument”<u></u><u></u></p>
<p class="MsoNormal">“tunefs.luster: FATAL: failed to write local files and tunefs.luster: exiting with 22 (Invalid argument)”<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">When I use the command tunefs.lustre /dev/sdb (which is one of the lustre LUNs that is attached as a “disk” to the VM)<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Full output of the tunefs.luster /dev/sdb command (as mush as I can show anyway):<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal" style="margin-left:0.5in">Tunefs.lustre /dev/sdb<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in">Checking for existing lustre data: found<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in">Reading CONFIGS/mountdata<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><u></u> <u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"> Read previous values:<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in">Target: <name>-OST0009<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in">Index: 9<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in">Luster FS: <name><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in">Mount type: ldiskfs<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in">Flags: 0x1002<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"> (OST no_primmode )<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in">Persistent mount opts: errors=remount-ro<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in">Parameters: mgsnode=<IP of the 1<sup>st</sup> MGS node>@tcp mgsnode=<IP of the 2<sup>nd</sup> MGS node>@tcp failover.node=<IP of the 1<sup>st</sup> OSS node>@tcp failover.node=<IP of the 2<sup>nd</sup> OSS node>@tcp<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><u></u> <u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"> Permanent disk data:<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in">Target: <name>-OST0009<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in">Index: 9<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in">Luster FS: <name><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in">Mount type: ldiskfs<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in">Flags: 0x1002<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"> (OST no_primmode )<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in">Persistent mount opts: errors=remount-ro<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in">Parameters: mgsnode=<IP of the 1<sup>st</sup> MGS node>@tcp mgsnode=<IP of the 2<sup>nd</sup> MGS node>@tcp failover.node=<IP of the 1<sup>st</sup> OSS node>@tcp failover.node=<IP of the 2<sup>nd</sup> OSS node>@tcp<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><u></u> <u></u></p>
<p class="MsoNormal" style="margin-left:0.5in">tunefs.luster: Unable to mount /dev/sdb: Invalid argument<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><u></u> <u></u></p>
<p class="MsoNormal" style="margin-left:0.5in">tunefs.luster: FATAL: failed to write local files<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in">tunefs.luster: exiting with 22 (Invalid argument<b>)<u></u><u></u></b></p>
<p class="MsoNormal"><b><u></u> <u></u></b></p>
<p class="MsoNormal">Now to be clear the MDS nodes are not working correctly as I am not able to mount /dev/sdb on them where the existing meta data is served out from. To this point I have been concentrating on the OSS nodes as that is where the lustre data
is coming from. I have installed the lustre kernel and the same software on the MDS nodes in the same way that I have on the OSS nodes. When I try to use tunefs.lustre /dev/sdb on the MDS nodes I get an error saying:<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal" style="margin-left:0.5in">Checking for existing lustre data: not found<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><u></u> <u></u></p>
<p class="MsoNormal" style="margin-left:0.5in">tunefs.luster: FATAL: device /dev/sdb has not been formatted with mkfs.lustre<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in">tunefs.luster: exiting with 19 (no such device<b>)<u></u><u></u></b></p>
<p class="MsoNormal"><b><u></u> <u></u></b></p>
<p class="MsoNormal">I am assuming that this is correct as that attached LUN does not need to have lustre data on it as it is the meta data server. Is there anything that I can/need to check on the MDS nodes to see what is running/working correctly?<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">I know that this is a lot and I appreciate any help that you can give me to troubleshoot this.<u></u><u></u></p>
<p class="MsoNormal"><br>
Dean<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div style="border-color:rgb(225,225,225) currentcolor currentcolor;border-style:solid none none;border-width:1pt medium medium;padding:3pt 0in 0in">
<p class="MsoNormal"><b>From:</b> STEPHENS, DEAN - US <br>
<b>Sent:</b> Monday, November 22, 2021 5:58 AM<br>
<b>To:</b> Andreas Dilger <<a href="mailto:adilger@whamcloud.com" target="_blank">adilger@whamcloud.com</a>><br>
<b>Cc:</b> Colin Faber <<a href="mailto:cfaber@gmail.com" target="_blank">cfaber@gmail.com</a>>; <a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
<b>Subject:</b> RE: [lustre-discuss] Lustre and server upgrade<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Thanks for the clarification. I am using llmount.sh to test the install of the OST and MDT not run in production. I hope to have more done today and will reach out to let you all know what I find.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Dean<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div style="border-color:rgb(225,225,225) currentcolor currentcolor;border-style:solid none none;border-width:1pt medium medium;padding:3pt 0in 0in">
<p class="MsoNormal"><b>From:</b> Andreas Dilger <<a href="mailto:adilger@whamcloud.com" target="_blank">adilger@whamcloud.com</a>>
<br>
<b>Sent:</b> Friday, November 19, 2021 5:25 PM<br>
<b>To:</b> STEPHENS, DEAN - US <<a href="mailto:dean.stephens@caci.com" target="_blank">dean.stephens@caci.com</a>><br>
<b>Cc:</b> Colin Faber <<a href="mailto:cfaber@gmail.com" target="_blank">cfaber@gmail.com</a>>; <a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">
lustre-discuss@lists.lustre.org</a><br>
<b>Subject:</b> Re: [lustre-discuss] Lustre and server upgrade<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Dean, <u></u><u></u></p>
<div>
<p class="MsoNormal">it should be emphasized that "llmount.sh" and "llmountcleanup.sh" are for quickly formatting and mounting *TEST* filesystems. They only create a few small (400MB) loopback files in /tmp and format them as OSTs and MDTs. This should *NOT*
be used on a production system, or you will be very sad when the files in /tmp disappear after the server is rebooted and/or they reformat your real filesystem devices.
<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I mention this here because it isn't clear to me whether you are using them for testing, or trying to get a real filesystem mounted.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Cheers, Andreas<u></u><u></u></p>
<div>
<p class="MsoNormal" style="margin-bottom:12pt"><u></u> <u></u></p>
<blockquote style="margin-top:5pt;margin-bottom:5pt">
<div>
<p class="MsoNormal">On Nov 19, 2021, at 13:25, STEPHENS, DEAN - US via lustre-discuss <<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a>> wrote:<u></u><u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">I also figure out how to clean up after the llmount.sh script is run. There is a llmountcleanup.sh that will do that.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Dean<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<div style="border-color:rgb(225,225,225) currentcolor currentcolor;border-style:solid none none;border-width:1pt medium medium;padding:3pt 0in 0in">
<div>
<p class="MsoNormal"><b>From:</b><span class="gmail-m_-1734618617407989705apple-converted-space"> </span>STEPHENS, DEAN - US<span class="gmail-m_-1734618617407989705apple-converted-space"> </span><br>
<b>Sent:</b><span class="gmail-m_-1734618617407989705apple-converted-space"> </span>Friday, November 19, 2021 1:08 PM<br>
<b>To:</b><span class="gmail-m_-1734618617407989705apple-converted-space"> </span>Colin Faber <<a href="mailto:cfaber@gmail.com" target="_blank">cfaber@gmail.com</a>><br>
<b>Cc:</b><span class="gmail-m_-1734618617407989705apple-converted-space"> </span><a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
<b>Subject:</b><span class="gmail-m_-1734618617407989705apple-converted-space"> </span>RE: [lustre-discuss] Lustre and server upgrade<u></u><u></u></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">One more thing that I have noticed using the llmount.sh script, the directories that were created by the script under /mnt have 000 set for the permissions. The ones that I have configure under /mnt/lustre are set to 750 permissions.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Is this something that needs to be fixed. I have these server being configure via puppet and that is how the /mnt/lustre directories are being created and the permissions set.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Dean<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<div style="border-color:rgb(225,225,225) currentcolor currentcolor;border-style:solid none none;border-width:1pt medium medium;padding:3pt 0in 0in">
<div>
<p class="MsoNormal"><b>From:</b><span class="gmail-m_-1734618617407989705apple-converted-space"> </span>STEPHENS, DEAN - US<span class="gmail-m_-1734618617407989705apple-converted-space"> </span><br>
<b>Sent:</b><span class="gmail-m_-1734618617407989705apple-converted-space"> </span>Friday, November 19, 2021 7:14 AM<br>
<b>To:</b><span class="gmail-m_-1734618617407989705apple-converted-space"> </span>Colin Faber <<a href="mailto:cfaber@gmail.com" target="_blank">cfaber@gmail.com</a>><br>
<b>Cc:</b><span class="gmail-m_-1734618617407989705apple-converted-space"> </span><a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
<b>Subject:</b><span class="gmail-m_-1734618617407989705apple-converted-space"> </span>RE: [lustre-discuss] Lustre and server upgrade<u></u><u></u></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">The other question that I have is how to clean up after the llmount.sh has been run? If I do a df on the server I see that mds1, osd1, and ost2 are still mounted to /mnt. Do I need to manually umount them since the llmount.sh completed
successfully?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Also I have not done anything to my MDS node so some direction on what to do there will be helpful as well.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Dean<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<div style="border-color:rgb(225,225,225) currentcolor currentcolor;border-style:solid none none;border-width:1pt medium medium;padding:3pt 0in 0in">
<div>
<p class="MsoNormal"><b>From:</b><span class="gmail-m_-1734618617407989705apple-converted-space"> </span>STEPHENS, DEAN - US<span class="gmail-m_-1734618617407989705apple-converted-space"> </span><br>
<b>Sent:</b><span class="gmail-m_-1734618617407989705apple-converted-space"> </span>Friday, November 19, 2021 7:00 AM<br>
<b>To:</b><span class="gmail-m_-1734618617407989705apple-converted-space"> </span>Colin Faber <<a href="mailto:cfaber@gmail.com" target="_blank">cfaber@gmail.com</a>><br>
<b>Cc:</b><span class="gmail-m_-1734618617407989705apple-converted-space"> </span><a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
<b>Subject:</b><span class="gmail-m_-1734618617407989705apple-converted-space"> </span>RE: [lustre-discuss] Lustre and server upgrade<u></u><u></u></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Thanks for the help yesterday and I was able to install the Lustre kernel and software on a VM to include the test RPM.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">This is what I did following<span class="gmail-m_-1734618617407989705apple-converted-space"> </span><a href="https://wiki.lustre.org/Installing_the_Lustre_Software#Lustre_Servers_with_LDISKFS_OSD_Support" target="_blank">these directions</a>:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Installed the Lustre kernel and kernel-devel (the other RPMs listed were not in my luster-server repo)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Rebooted the VM<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Installed kmod-lustre kmod-lustre-osd-ldiskfs lustre-osd-ldiskfs-mount lustre lustre-resource-agents lustre-tests<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Ran modprobe -v lustre (did not show that it loaded kernel modules as it has done in the past)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Ran lustre_rmmod (got an error Module Luster in use)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Rebooted again<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Ran llmount.sh and it looked like it completed successfully<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Ran tunefs.lustre /dev/sdb (at the bottom of the output I am seeing tunefs.luster: Unable to mount /dev/sdb: Invalid argument and tunefs.luster: FATAL: failed to write local files and tunefs.luster: exiting with 22 (Invalid argument))<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Any idea what the “invalid argument” is talking about?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Dean<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div style="border-color:rgb(225,225,225) currentcolor currentcolor;border-style:solid none none;border-width:1pt medium medium;padding:3pt 0in 0in">
<div>
<p class="MsoNormal"><b>From:</b><span class="gmail-m_-1734618617407989705apple-converted-space"> </span>Colin Faber <<a href="mailto:cfaber@gmail.com" target="_blank">cfaber@gmail.com</a>><span class="gmail-m_-1734618617407989705apple-converted-space"> </span><br>
<b>Sent:</b><span class="gmail-m_-1734618617407989705apple-converted-space"> </span>Thursday, November 18, 2021 3:34 PM<br>
<b>To:</b><span class="gmail-m_-1734618617407989705apple-converted-space"> </span>STEPHENS, DEAN - US <<a href="mailto:dean.stephens@caci.com" target="_blank">dean.stephens@caci.com</a>><br>
<b>Cc:</b><span class="gmail-m_-1734618617407989705apple-converted-space"> </span><a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
<b>Subject:</b><span class="gmail-m_-1734618617407989705apple-converted-space"> </span>Re: [lustre-discuss] Lustre and server upgrade<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<div>
<p class="MsoNormal">The VM will need a full install of all server packages, as well as the tests package to allow for this test.<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal">On Thu, Nov 18, 2021 at 2:26 PM STEPHENS, DEAN - US <<a href="mailto:dean.stephens@caci.com" target="_blank">dean.stephens@caci.com</a>> wrote:<u></u><u></u></p>
</div>
</div>
<blockquote style="border-color:currentcolor currentcolor currentcolor rgb(204,204,204);border-style:none none none solid;border-width:medium medium medium 1pt;padding:0in 0in 0in 6pt;margin:5pt 0in 5pt 4.8pt">
<div>
<div>
<div>
<p class="MsoNormal">I have not tried that but I can do that on a new VM that I can create. I assume that is all that I need is the lustre-tests RPM and associated dependencies and not the full blown lustre install?<span class="gmail-m_-1734618617407989705apple-converted-space"> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Dean<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div style="border-style:solid none none;border-width:1pt medium medium;padding:3pt 0in 0in;border-color:currentcolor">
<div>
<p class="MsoNormal"><b>From:</b><span class="gmail-m_-1734618617407989705apple-converted-space"> </span>Colin Faber <<a href="mailto:cfaber@gmail.com" target="_blank">cfaber@gmail.com</a>><span class="gmail-m_-1734618617407989705apple-converted-space"> </span><br>
<b>Sent:</b><span class="gmail-m_-1734618617407989705apple-converted-space"> </span>Thursday, November 18, 2021 2:22 PM<br>
<b>To:</b><span class="gmail-m_-1734618617407989705apple-converted-space"> </span>STEPHENS, DEAN - US <<a href="mailto:dean.stephens@caci.com" target="_blank">dean.stephens@caci.com</a>><br>
<b>Cc:</b><span class="gmail-m_-1734618617407989705apple-converted-space"> </span><a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
<b>Subject:</b><span class="gmail-m_-1734618617407989705apple-converted-space"> </span>Re: [lustre-discuss] Lustre and server upgrade<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal">So that indicates that your installation is incomplete or something else is preventing lustre, ldiskfs, and possibly other modules from loading. Have you been able to reproduce this behavior on a fresh rhel install with lustre 2.12.7?
(i.e. llmount.sh failing)?<u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">-cf<u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal">On Thu, Nov 18, 2021 at 2:20 PM STEPHENS, DEAN - US <<a href="mailto:dean.stephens@caci.com" target="_blank">dean.stephens@caci.com</a>> wrote:<u></u><u></u></p>
</div>
</div>
<blockquote style="border-style:none none none solid;border-width:medium medium medium 1pt;padding:0in 0in 0in 6pt;margin:5pt 0in 5pt 4.8pt;border-color:currentcolor currentcolor currentcolor rgb(204,204,204)">
<div>
<div>
<div>
<p class="MsoNormal">Thanks for the direction. I found it and installed lustre-tests.x86_64 and now I have the llmount but it was defaulted to /usr/lib64/lustre/tests/llmount.sh and when I ran it but it failed with:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Stopping clients: <hostname> /mnt/lustre (opts: -f)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Stopping clients: <hostname> /mnt/lustre2 (opts: -f)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Loading modules from /usr/lib64/lustre/tests/..<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Detected 2 online CPUs by sysfs<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Force libcfs to create 2 CPU partitions<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Formatting mgs, mds, osts<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Format mds1: /tmp/lustre-mdt1<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Mkfs.lustre: Unable to mount /dev/loop0: No such device (even though /dev/loop0 is a thing)<br>
Is the ldiskfs module loaded?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Mkfs.lustre FATAL: failed to write local files<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Mkfs.lustre: exiting with 19 (no such device)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div style="border-style:solid none none;border-width:1pt medium medium;padding:3pt 0in 0in;border-color:currentcolor">
<div>
<p class="MsoNormal"><b>From:</b><span class="gmail-m_-1734618617407989705apple-converted-space"> </span>Colin Faber <<a href="mailto:cfaber@gmail.com" target="_blank">cfaber@gmail.com</a>><span class="gmail-m_-1734618617407989705apple-converted-space"> </span><br>
<b>Sent:</b><span class="gmail-m_-1734618617407989705apple-converted-space"> </span>Thursday, November 18, 2021 2:03 PM<br>
<b>To:</b><span class="gmail-m_-1734618617407989705apple-converted-space"> </span>STEPHENS, DEAN - US <<a href="mailto:dean.stephens@caci.com" target="_blank">dean.stephens@caci.com</a>><br>
<b>Cc:</b><span class="gmail-m_-1734618617407989705apple-converted-space"> </span><a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
<b>Subject:</b><span class="gmail-m_-1734618617407989705apple-converted-space"> </span>Re: [lustre-discuss] Lustre and server upgrade<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<div>
<p class="MsoNormal">This would be part of the lustre-tests RPM package and will install llmount.sh to /usr/lib/lustre/tests/llmount.sh I believe.<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal">On Thu, Nov 18, 2021 at 1:45 PM STEPHENS, DEAN - US <<a href="mailto:dean.stephens@caci.com" target="_blank">dean.stephens@caci.com</a>> wrote:<u></u><u></u></p>
</div>
</div>
<blockquote style="border-style:none none none solid;border-width:medium medium medium 1pt;padding:0in 0in 0in 6pt;margin:5pt 0in 5pt 4.8pt;border-color:currentcolor currentcolor currentcolor rgb(204,204,204)">
<div>
<div>
<div>
<p class="MsoNormal">Not sure what you mean by “If you install the test suite”. I am not seeing a llmount.sh file on the server using “locate llmount.sh” at this point. What are the steps to install the test suite?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Dean<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div style="border-style:solid none none;border-width:1pt medium medium;padding:3pt 0in 0in;border-color:currentcolor">
<div>
<p class="MsoNormal"><b>From:</b><span class="gmail-m_-1734618617407989705apple-converted-space"> </span>Colin Faber <<a href="mailto:cfaber@gmail.com" target="_blank">cfaber@gmail.com</a>><span class="gmail-m_-1734618617407989705apple-converted-space"> </span><br>
<b>Sent:</b><span class="gmail-m_-1734618617407989705apple-converted-space"> </span>Thursday, November 18, 2021 1:34 PM<br>
<b>To:</b><span class="gmail-m_-1734618617407989705apple-converted-space"> </span>STEPHENS, DEAN - US <<a href="mailto:dean.stephens@caci.com" target="_blank">dean.stephens@caci.com</a>><br>
<b>Cc:</b><span class="gmail-m_-1734618617407989705apple-converted-space"> </span><a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
<b>Subject:</b><span class="gmail-m_-1734618617407989705apple-converted-space"> </span>Re: [lustre-discuss] Lustre and server upgrade<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal">Hm.. If you install the test suite does llmount.sh succeed? This should setup a single node cluster on whatever node you're running lustre on, I believe it will load modules as needed (IIRC), if this test succeeds, then you know that lustre
is installed correctly (or correctly enough), if not, I'd focus on the installation as the target issue may be a redheirring<u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">-cf<u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal">On Thu, Nov 18, 2021 at 1:01 PM STEPHENS, DEAN - US <<a href="mailto:dean.stephens@caci.com" target="_blank">dean.stephens@caci.com</a>> wrote:<u></u><u></u></p>
</div>
</div>
<blockquote style="border-style:none none none solid;border-width:medium medium medium 1pt;padding:0in 0in 0in 6pt;margin:5pt 0in 5pt 4.8pt;border-color:currentcolor currentcolor currentcolor rgb(204,204,204)">
<div>
<div>
<div>
<p class="MsoNormal">Thanks for the fast reply.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">When I do the tunefs.lustre /dev/sdX command I get:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Target: <name>-OST0009<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Index: 9<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Target: <name>-OST0008<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Index: 8<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">I spot checked some others and they seem to be good with the exception of one. It shows:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Target: <name>-OST000a<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Index: 10<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">But since there are 11 LUNs attached that make sense to me.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">As far as the upgrade it was a fresh install using the legacy targets as the OSS and MDS nodes are virtual machine with the LUN disks attached to them so that Red Hat sees them as /dev/sdX devices.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">When I loaded Lustre on the server I did a yum install lustre and since we were pointed at the lustre-2.12 repo in our environment it picked up the following RPMs to install:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Luster-resource-agents-2.12.6-1.el7.x86_64<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Kmod-lustre-2.12.6-1.el7.x86_64<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Kmod-zfs-3.10.0-1160.2.1.el7_lustre.x86_64-09.7.13-1.el7.x86_64<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Kmod-lustre-osd-zfs-2.12.6-1.el7.x86_64<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Lustre-2.12.6-1.el7.x86_64<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Kmod-spl-3.10.0-1160.2.1.el7_lustre.x86_64-09.7.13-1.el7.x86_64<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Lustre-osd-zfs-mount-2.12.6-1.el7.x86_64<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Lustre-osd-ldiskfs-mount-2.12.6-1.el7.x86_64<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Dean<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<div style="border-style:solid none none;border-width:1pt medium medium;padding:3pt 0in 0in;border-color:currentcolor">
<div>
<p class="MsoNormal"><b>From:</b><span class="gmail-m_-1734618617407989705apple-converted-space"> </span>Colin Faber <<a href="mailto:cfaber@gmail.com" target="_blank">cfaber@gmail.com</a>><span class="gmail-m_-1734618617407989705apple-converted-space"> </span><br>
<b>Sent:</b><span class="gmail-m_-1734618617407989705apple-converted-space"> </span>Thursday, November 18, 2021 12:35 PM<br>
<b>To:</b><span class="gmail-m_-1734618617407989705apple-converted-space"> </span>STEPHENS, DEAN - US <<a href="mailto:dean.stephens@caci.com" target="_blank">dean.stephens@caci.com</a>><br>
<b>Cc:</b><span class="gmail-m_-1734618617407989705apple-converted-space"> </span><a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
<b>Subject:</b><span class="gmail-m_-1734618617407989705apple-converted-space"> </span>Re: [lustre-discuss] Lustre and server upgrade<u></u><u></u></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<p class="gmail-m_-1734618617407989705gmail-m-5415300209567740660gmail-m-1795078219693914619gmail-m-1374272708024244046gmail-m-8166945065793112345msonospacing">
<span style="font-size:12pt;color:red">EXTERNAL EMAIL - This email originated from outside of CACI. Do not click any links or attachments unless you recognize and trust the sender.</span><u></u><u></u></p>
<p class="gmail-m_-1734618617407989705gmail-m-5415300209567740660gmail-m-1795078219693914619gmail-m-1374272708024244046gmail-m-8166945065793112345msonospacing">
<span style="font-size:12pt;color:red"> </span><u></u><u></u></p>
<p class="gmail-m_-1734618617407989705gmail-m-5415300209567740660gmail-m-1795078219693914619gmail-m-1374272708024244046gmail-m-8166945065793112345msonospacing">
<span style="font-size:12pt"> </span><u></u><u></u></p>
<div>
<div>
<div>
<div>
<p class="MsoNormal">Hi,<u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">I believe in 2.10 sometime (someone correct me if I'm wrong) that the index parameter was required and needs to be specified. On an existing system this should already be set, but can you check the parameters line with tunefs.lustre for
correct index=N values across your storage nodes?<u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">Also, with your "upgrade", was this a fresh install utilizing legacy targets?<u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">The last thing I can think of IIRC, there was on-disk format changes between 2.5 and 2.12, these should be transparent to you, but it may be some other issue is preventing successful upgrade, though the missing module error really speaks
to possible issues around how lustre was installed and loaded on the system.<u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">Cheers!<u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">-cf<u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal">On Thu, Nov 18, 2021 at 12:24 PM STEPHENS, DEAN - US via lustre-discuss <<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a>> wrote:<u></u><u></u></p>
</div>
</div>
<blockquote style="border-style:none none none solid;border-width:medium medium medium 1pt;padding:0in 0in 0in 6pt;margin:5pt 0in 5pt 4.8pt;border-color:currentcolor currentcolor currentcolor rgb(204,204,204)">
<div>
<div>
<div>
<p class="MsoNormal">I am by no means a Lustre expert and am seeking some help with our system. I am not able to get log file to post as the servers are in the closed area with no access to the Internet.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Here is a bit of history of our system:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">The OSS and MDS nodes were RHEL6 and running a Luster server the kernel 2.6.32-431.23.3.el6_lustre.x86_64 and the Lustre version of 2.5.3. the client version was 2.10. That was in a working state.<span class="gmail-m_-1734618617407989705apple-converted-space"> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">We upgraded the OSS ad MDS nodes to RHEL7 and installed Lustre server 2.12 software and kernel.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">The attached 11 LUNs are showing up as /dev/sdb - /dev/sdl<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Right now, on the OSS nodes, if I use the command tunefs.luster /dev/sdb I get some data back saying that Lustre data has been found but at the bottom of the out put it shows “tunefs.lustre: Unable to mount /dev/sdb: No such device” and
“Is the ldiskfs module available”<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">When I do a “modprobe -v lustre” I do not see ldiskfs.ko as being loaded even though there is a ldiskfs.ko file in /lib/modules/3.10.0-1160.2.1.el7_lustre.x86_64/extra/lustre/fs directory. I am not sure how to get it to load in the modprobe
command.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">I used “insmod /lib/modules/3.10.0-1160.2.1.el7_lustre.x86_64/extra/lustre/fs/ ldiskfs.ko” and re-ran the “tunefs.luster /dev/sdb” command with the same result.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">If I use the same command on the MDS nodes I get “no Lustre data found and /dev/sdb has not been formatted with mkfs.lustre”. I am not sure that is what is needed here as the MDS nodes do not really have the lustre data as it is the meta
data server.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">I tried to use the command “tunefs.lustre --mgs --erase_params --mgsnode=<IP address>@tcp --writeconf --dryrun /dev/sdb” and get the error “/dev/sdb has not been formatted with mkfs.lustre”.<span class="gmail-m_-1734618617407989705apple-converted-space"> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">I need some help and guidance and I can provide what may be needed though it will need to be typed out as I am not able to get actual log files from the system.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Dean Stephens<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">CACI<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Linux System Admin<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div class="MsoNormal" style="text-align:center" align="center">
<hr width="100%" size="2" align="center">
</div>
<div>
<p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:gray"><br>
This electronic message contains information from CACI International Inc or subsidiary companies, which may be company sensitive, proprietary, privileged or otherwise protected from disclosure. The information is intended to be used solely by the recipient(s)
named above. If you are not an intended recipient, be aware that any review, disclosure, copying, distribution or use of this transmission or its contents is prohibited. If you have received this transmission in error, please notify the sender immediately.</span><u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal">_______________________________________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><u></u><u></u></p>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><span style="font-size:9pt;font-family:"Helvetica",sans-serif">_______________________________________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a></span><u></u><u></u></p>
</div>
</blockquote>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="color:black">Cheers, Andreas<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">--<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">Andreas Dilger<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">Lustre Principal Architect<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">Whamcloud<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"><u></u> <u></u></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><span style="color:black"><u></u> <u></u></span></p>
</div>
<p class="MsoNormal" style="margin-bottom:12pt"><u></u> <u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</div>
</div>
</blockquote></div>