<div dir="ltr"><div><div><div><br clear="all"></div>Hello Lustre team,<br><br></div>We have an older production cluster with 1.8.1 , with 1 MDT/MDS server and two OST/OSD servers. The network card of one of the OSD servers (lxsrv3) crashed (eth0), and we put a new one (eth5). Then we changed modprobe.conf.local of the server:<br><br>options lnet networks="tcp0(eth5)"<br><br></div><div>And restarted the server, but the lustre file system was down. so I follow the manual to regenerate configuration logs:<br></div><div><br>1- Shutdown the file system in this order.<br> Unmount the clients.<br> Unmount the MDT.<br> Unmount all OSTs.<br><br><br>.2- On the MDT, run: <br><br>lxsrv4:~ # tunefs.lustre --writeconf /dev/sda1<br><br><br>checking for existing Lustre data: found CONFIGS/mountdata<br>Reading CONFIGS/mountdata<br><br> Read previous values:<br>Target: luster-MDT0000<br>Index: 0<br>Lustre FS: luster<br>Mount type: ldiskfs<br>Flags: 0x5<br> (MDT MGS )<br>Persistent mount opts: errors=remount-ro,iopen_nopriv,user_xattr,acl<br>Parameters: mdt.quota_type=ug2 mdt.group_upcall=/usr/sbin/l_getgroups<br><br><br> Permanent disk data:<br>Target: luster-MDT0000<br>Index: 0<br>Lustre FS: luster<br>Mount type: ldiskfs<br>Flags: 0x105<br> (MDT MGS writeconf )<br>Persistent mount opts: errors=remount-ro,iopen_nopriv,user_xattr,acl<br>Parameters: mdt.quota_type=ug2 mdt.group_upcall=/usr/sbin/l_getgroups<br><br>Writing CONFIGS/mountdata<br><br>3. On the OSTs, run: <br><br>lxsrv1: <br>tunefs.lustre --writeconf /dev/sda<br>tunefs.lustre --writeconf /dev/sdb <br>tunefs.lustre --writeconf /dev/sdc<br><br>lxsrv3: ( this has been the failed server)<br><br>tunefs.lustre --writeconf /dev/disk/by-id/scsi-3600605b000a79eb011b131b81830ddc0<br>tunefs.lustre --writeconf /dev/disk/by-id/scsi-3600605b000a79eb011bfa4ad1861dd29<br>tunefs.lustre --writeconf /dev/disk/by-id/scsi-3600605b000a79eb011b131b81831b1de<br><br>4. Restarted the file system in this order.<br>Mount the MGS <br>Mount the MDT.<br>Mount the OSTs.<br><br></div><div>5- Now the MDT can see all OST. However, when I start a client:<br><br> node43:~ # lfs df<br>UUID 1K-blocks Used Available Use% Mounted on<br>luster-MDT0000_UUID 255466784 4921796 235945520 1% /lustre[MDT:0]<br>luster-OST0003_UUID 1152952228 527535012 566850548 45% /lustre[OST:3]<br>luster-OST0004_UUID 1152952228 607482548 486903012 52% /lustre[OST:4]<br>luster-OST0005_UUID 1152952228 660191440 434194120 57% /lustre[OST:5]<br><br>filesystem summary: 3458856684 1795209000 1487947680 51% /lustre<br><br></div><div>The client can't connect with the OST0000 , OST0001 and OST0002.:<br><br>lctl ping 192.168.1.249@tcp<br>failed to ping 192.168.1.44@tcp: Input/output error<br><br><br></div><div><br>lctl dl<br> 0 UP mgc MGC192.168.1.248@tcp 1d2aa343-6ae6-f6d9-0637-7c10b52a0569 5<br> 1 UP lov luster-clilov-ffff81046e4be400 34862c9c-2cfc-2de0-5e6d-247dd4953a13 4<br> 2 UP mdc luster-MDT0000-mdc-ffff81046e4be400 34862c9c-2cfc-2de0-5e6d-247dd4953a13 5<br> 3 UP osc luster-OST0003-osc-ffff81046e4be400 34862c9c-2cfc-2de0-5e6d-247dd4953a13 5<br> 4 UP osc luster-OST0004-osc-ffff81046e4be400 34862c9c-2cfc-2de0-5e6d-247dd4953a13 5<br> 5 UP osc luster-OST0005-osc-ffff81046e4be400 34862c9c-2cfc-2de0-5e6d-247dd4953a13 5<br> 6 UP osc luster-OST0000-osc-ffff81046e4be400 34862c9c-2cfc-2de0-5e6d-247dd4953a13 5<br> 7 UP osc luster-OST0001-osc-ffff81046e4be400 34862c9c-2cfc-2de0-5e6d-247dd4953a13 5<br> 8 UP osc luster-OST0002-osc-ffff81046e4be400 34862c9c-2cfc-2de0-5e6d-247dd4953a13 5<br> 9 UP lov luster-clilov-ffff81046ed3a400 fc806d37-990e-5e36-0952-1abf92e6b2cc 4<br> 10 UP mdc luster-MDT0000-mdc-ffff81046ed3a400 fc806d37-990e-5e36-0952-1abf92e6b2cc 5<br> 11 UP osc luster-OST0003-osc-ffff81046ed3a400 fc806d37-990e-5e36-0952-1abf92e6b2cc 5<br> 12 UP osc luster-OST0004-osc-ffff81046ed3a400 fc806d37-990e-5e36-0952-1abf92e6b2cc 5<br> 13 UP osc luster-OST0005-osc-ffff81046ed3a400 fc806d37-990e-5e36-0952-1abf92e6b2cc 5<br> 14 UP osc luster-OST0000-osc-ffff81046ed3a400 fc806d37-990e-5e36-0952-1abf92e6b2cc 5<br> 15 UP osc luster-OST0001-osc-ffff81046ed3a400 fc806d37-990e-5e36-0952-1abf92e6b2cc 5<br> 16 UP osc luster-OST0002-osc-ffff81046ed3a400 fc806d37-990e-5e36-0952-1abf92e6b2cc 5<br> <br></div><div>There are two lustre file systems!!!<br><br></div><div>If I umount lustre:<br><br></div><div><br>umount /lustre<br>lctl dl<br> 0 UP mgc MGC192.168.1.248@tcp 31d0a0ed-7df2-7bc6-1b3c-d799685e1e1a 5<br> 1 UP lov luster-clilov-ffff81047039b800 2f0d17b1-906c-59c1-2086-29970887c33f 4<br> 2 UP mdc luster-MDT0000-mdc-ffff81047039b800 2f0d17b1-906c-59c1-2086-29970887c33f 5<br> 3 UP osc luster-OST0003-osc-ffff81047039b800 2f0d17b1-906c-59c1-2086-29970887c33f 5<br> 4 UP osc luster-OST0004-osc-ffff81047039b800 2f0d17b1-906c-59c1-2086-29970887c33f 5<br> 5 UP osc luster-OST0005-osc-ffff81047039b800 2f0d17b1-906c-59c1-2086-29970887c33f 5<br> 6 UP osc luster-OST0000-osc-ffff81047039b800 2f0d17b1-906c-59c1-2086-29970887c33f 5<br> 7 UP osc luster-OST0001-osc-ffff81047039b800 2f0d17b1-906c-59c1-2086-29970887c33f 5<br> 8 UP osc luster-OST0002-osc-ffff81047039b800 2f0d17b1-906c-59c1-2086-29970887c33f 5<br> 14 ST osc luster-OST0000-osc-ffff8104702b2400 d1a75733-a7c4-c7b4-b410-f6daaccf1869 2<br> 15 ST osc luster-OST0001-osc-ffff8104702b2400 d1a75733-a7c4-c7b4-b410-f6daaccf1869 2<br> 16 ST osc luster-OST0002-osc-ffff8104702b2400 d1a75733-a7c4-c7b4-b410-f6daaccf1869 2<br><br> lfs df (anything)<br><br></div><div>I umount lustre again:<br><br>lctl dl<br> 6 ST osc luster-OST0000-osc-ffff81047039b800 2f0d17b1-906c-59c1-2086-29970887c33f 2<br> 7 ST osc luster-OST0001-osc-ffff81047039b800 2f0d17b1-906c-59c1-2086-29970887c33f 2<br> 8 ST osc luster-OST0002-osc-ffff81047039b800 2f0d17b1-906c-59c1-2086-29970887c33f 2<br> 14 ST osc luster-OST0000-osc-ffff81046f414c00 802a6b2d-6954-b924-5556-1f3112b030f3 2<br> 15 ST osc luster-OST0001-osc-ffff81046f414c00 802a6b2d-6954-b924-5556-1f3112b030f3 2<br> 16 ST osc luster-OST0002-osc-ffff81046f414c00 802a6b2d-6954-b924-5556-1f3112b030f3 2<br> <br></div><div>And if I run a ping in the OST server:<br><br> lxsrv3> lctl ping 192.168.1.44@tcp<br><br></div><div>And I I mount again lustre in the client:<br><br></div><div><br></div><div> lctl dl<br> 0 UP mgc MGC192.168.1.248@tcp dd51f58e-c580-b2eb-18f8-6f950aa272bb 5<br> 1 UP lov luster-clilov-ffff810467f2d000 3beb05fe-eaa5-1b21-e416-6caa965f2d1a 4<br> 2 UP mdc luster-MDT0000-mdc-ffff810467f2d000 3beb05fe-eaa5-1b21-e416-6caa965f2d1a 5<br> 3 UP osc luster-OST0003-osc-ffff810467f2d000 3beb05fe-eaa5-1b21-e416-6caa965f2d1a 5<br> 4 UP osc luster-OST0004-osc-ffff810467f2d000 3beb05fe-eaa5-1b21-e416-6caa965f2d1a 5<br> 5 UP osc luster-OST0005-osc-ffff810467f2d000 3beb05fe-eaa5-1b21-e416-6caa965f2d1a 5<br> 6 UP osc luster-OST0000-osc-ffff810467f2d000 3beb05fe-eaa5-1b21-e416-6caa965f2d1a 5<br> 7 UP osc luster-OST0001-osc-ffff810467f2d000 3beb05fe-eaa5-1b21-e416-6caa965f2d1a 5<br> 8 UP osc luster-OST0002-osc-ffff810467f2d000 3beb05fe-eaa5-1b21-e416-6caa965f2d1a 5<br><br><br>lfs df<br>UUID 1K-blocks Used Available Use% Mounted on<br>luster-MDT0000_UUID 255466784 4921812 235945504 1% /lustre[MDT:0]<br>luster-OST0000_UUID 2880829872 1395149424 1339342592 48% /lustre[OST:0]<br>luster-OST0001_UUID 2880829872 1323430808 1411061208 45% /lustre[OST:1]<br>luster-OST0002_UUID 2880829872 1348980996 1385511020 46% /lustre[OST:2]<br>luster-OST0003_UUID 1152952228 527535024 566850536 45% /lustre[OST:3]<br>luster-OST0004_UUID 1152952228 607482556 486903004 52% /lustre[OST:4]<br>luster-OST0005_UUID 1152952228 660191444 434194116 57% /lustre[OST:5]<br><br>filesystem summary: 12101346300 5862770252 5623862476 48% /lustre<br><br></div><div>And voalá , lustre works again. However, If I reboot the client, the problem goes back.<br></div><div>What should do to fix it??<br><br></div><div>Thanks!!<br></div><div><span id="OS428024926"><span style="background-color:rgb(255,255,102)"></span></span></div><div><br><br></div><div><br><br><br></div><div><div><div>-- <br><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><br>--------------------------------------------------------<br><pre>Patricia Santos Marco<br></pre><pre>HPC research group System Administrator</pre><pre>Instituto de Biocomputación y Física de Sistemas Complejos (BIFI)</pre><pre>Universidad de Zaragoza</pre><pre>e-mail: <a href="mailto:artginer@bifi.es" target="_blank">psantos@bifi.es</a></pre><pre>phone: <a href="tel:%28%2B34%29%20976762992" value="+34976762992" target="_blank">(+34) 976762992</a> <br><br><a href="http://bifi.es/~patricia/" target="_blank">http://bifi.es/~patricia/</a><br></pre><br><br><br></div></div></div></div></div></div></div></div>
</div></div></div></div>