<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:Consolas;
color:windowtext;
font-weight:normal;
font-style:normal;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">Hello everyone!<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">I have a lustre client node that is external to our "bright-stack" and is not seen by the bright cluster manager<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">extern lustre client background info:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> root@aapksmansfi01 ~ # cat /etc/os-release<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> NAME="Ubuntu"<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> VERSION="20.04.6 LTS (Focal Fossa)"<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> ID=ubuntu<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> ID_LIKE=debian<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> PRETTY_NAME="Ubuntu 20.04.6 LTS"<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> VERSION_ID="20.04"<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> HOME_URL=<a href="https://www.ubuntu.com/">https://www.ubuntu.com/</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> SUPPORT_URL=<a href="https://help.ubuntu.com/">https://help.ubuntu.com/</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> BUG_REPORT_URL=<a href="https://bugs.launchpad.net/ubuntu/">https://bugs.launchpad.net/ubuntu/</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> PRIVACY_POLICY_URL=<a href="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy">https://www.ubuntu.com/legal/terms-and-policies/privacy-policy</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> VERSION_CODENAME=focal<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> UBUNTU_CODENAME=focal<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> root@aapksmansfi01 ~ # uname --all<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> Linux aapksmansfi01.usda.net 5.4.0-148-generic #165-Ubuntu SMP Tue Apr 18 08:53:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> root@aapksmansfi01 ~ # ofed_info | head -n 1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> MLNX_OFED_LINUX-5.4-3.6.8.1 (OFED-5.4-3.6.8):<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> root@aapksmansfi01 ~ # lctl lustre_build_version<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> Lustre version: 2.14.0_21_gd4b9557<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> root@aapksmansfi01 ~ # lctl list_nids<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> <a href="mailto:10.149.255.253@o2ib">
10.149.255.253@o2ib</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> <a href="mailto:10.149.255.253@tcp">
10.149.255.253@tcp</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> root@aapksmansfi01 ~ # lnetctl net show<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> net:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> - net type: lo<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> local NI(s):<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> - nid: 0@lo<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> status: up<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> - net type: o2ib<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> local NI(s):<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> - nid:
<a href="mailto:10.149.255.253@o2ib">10.149.255.253@o2ib</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> status: up<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> interfaces:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> 0: ibs1f1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> - net type: tcp<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> local NI(s):<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> - nid:
<a href="mailto:10.149.255.253@tcp">10.149.255.253@tcp</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> status: up<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> interfaces:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> 0: ibs1f1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> root@aapksmansfi01 ~ # lnetctl net show -v<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> net:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> - net type: lo<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> local NI(s):<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> - nid: 0@lo<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> status: up<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> statistics:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> send_count: 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> recv_count: 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> drop_count: 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> tunables:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> peer_timeout: 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> peer_credits: 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> peer_buffer_credits: 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> credits: 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> dev cpt: 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> tcp bonding: 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> CPT: "[0,1]"<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> - net type: o2ib<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> local NI(s):<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> - nid:
<a href="mailto:10.149.255.253@o2ib">10.149.255.253@o2ib</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> status: up<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> interfaces:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> 0: ibs1f1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> statistics:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> send_count: 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> recv_count: 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> drop_count: 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> tunables:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> peer_timeout: 180<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> peer_credits: 8<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> peer_buffer_credits: 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> credits: 256<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> peercredits_hiw: 4<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> map_on_demand: 1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> concurrent_sends: 8<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> fmr_pool_size: 512<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> fmr_flush_trigger: 384<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> fmr_cache: 1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> ntx: 512<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> conns_per_peer: 1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> lnd tunables:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> dev cpt: 1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> tcp bonding: 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> CPT: "[0,1]"<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> - net type: tcp<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> local NI(s):<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> - nid:
<a href="mailto:10.149.255.253@tcp">10.149.255.253@tcp</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> status: up<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> interfaces:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> 0: ibs1f1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> statistics:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> send_count: 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> recv_count: 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> drop_count: 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> tunables:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> peer_timeout: 180<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> peer_credits: 8<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> peer_buffer_credits: 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> credits: 256<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> dev cpt: 1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> tcp bonding: 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> CPT: "[0,1]"<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">I am still unable to mount lustre while pointing to the mds/mgs node<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> root@aapksmansfi01 ~ # mount -t lustre
<a href="mailto:10.149.0.33@o2ib:/lustrefs">10.149.0.33@o2ib:/lustrefs</a> /mnt/lustrefs<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> mount.lustre: mount
<a href="mailto:10.149.0.33@o2ib:/lustrefs">10.149.0.33@o2ib:/lustrefs</a> at /mnt/lustrefs failed: Input/output error<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> Is the MGS running?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> all the lustre modules loaded without errors (from what I could tell)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">If anyone knows or can suggest a solution or any diagnostic tools for this lustre client I would be very grateful!<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">Thank You!<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"><o:p> </o:p></span></p>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="500" style="width:375.0pt;background:white;border-collapse:collapse">
<tbody>
<tr>
<td width="74" valign="top" style="width:55.5pt;padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas;color:#333333"><img border="0" width="72" height="51" style="width:.75in;height:.5312in" id="Picture_x0020_1" src="cid:image001.jpg@01D97C48.59556B00" alt="USDA - APHIS"><o:p></o:p></span></p>
</td>
<td width="428" valign="top" style="width:321.0pt;padding:.75pt .75pt .75pt 5.25pt">
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="421" style="width:315.75pt;border-collapse:collapse">
<tbody>
<tr>
<td width="0" nowrap="" style="width:.3pt;padding:2.25pt 2.25pt 2.25pt 2.25pt">
<p class="MsoNormal"><b><span style="font-size:8.0pt;font-family:Consolas;color:black">Oleg Osipenko<br>
IT SPECIALIST - DATA MANAGEMENT<br>
APHIS Marketing and Regulatory Programs Business Services (MRPBS)<br>
MRP Information Technology (MRP IT) Services<o:p></o:p></span></b></p>
</td>
</tr>
<tr>
<td width="0" style="width:.3pt;padding:2.25pt 2.25pt 2.25pt 2.25pt">
<p class="MsoNormal"><b><span style="font-size:8.0pt;font-family:Consolas;color:#28742E">Laboratory and Scientific IT Support Services, NBAF Branch<o:p></o:p></span></b></p>
</td>
</tr>
<tr>
<td width="0" nowrap="" style="width:.3pt;padding:2.25pt 2.25pt 2.25pt 2.25pt">
<p class="MsoNormal"><b><span style="font-size:8.0pt;font-family:Consolas;color:black">p: 785-844-1946<o:p></o:p></span></b></p>
<p class="MsoNormal"><b><span style="font-size:8.0pt;font-family:Consolas;color:black">o: 785-712-3303<br>
<a href="mailto:oleg.osipenko@usda.gov"><span style="color:#428BCA;border:none windowtext 1.0pt;padding:0in">oleg.osipenko@usda.gov</span></a><br>
1880 Kimball Ave.<br>
Manhattan, KS 66502<o:p></o:p></span></b></p>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Helvetica",sans-serif;color:#333333"><br>
<br>
</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-bottom:7.5pt;background:white"><span style="font-size:8.0pt;font-family:Consolas;color:#333333">CONFIDENTIALITY NOTE: The preceding email message contains information that may be confidential, proprietary, or legally privileged,
and may constitute non-public information. This message is intended to be conveyed only to the intended named recipient(s). If you are not an intended recipient of this message, do not read it; instead, please advise the sender by reply email, and delete this
message and any attachments. Unauthorized individuals or entities are not permitted access to this information. Any disclosure, copying, distribution or taking any action in reliance on the contents of this information, except its delivery to the sender, is
strictly prohibited and may be unlawful.<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<br>
<br>
<br>
<br>
This electronic message contains information generated by the USDA solely for the intended recipients. Any unauthorized interception of this message or the use or disclosure of the information it contains may violate the law and subject the violator to civil
or criminal penalties. If you believe you have received this message in error, please notify the sender and delete the email immediately.
</body>
</html>