<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">OK, so our old MDS had hardware issues so I configured a new MGS / MDS on a VM (this is a backup lustre filesystem and I wanted to separate the MGS / MDS from OSS of the previous), and then did this:<div><br></div><div><!--?xml version="1.0" encoding="UTF-8" standalone="no"?-->
<p style="margin-left: 38px; font-family: Times; "><a name="50438194_pgfId-1307283">For example:</a></p><pre style="font-family: courier; margin-left: 40px; "><a name="50438194_pgfId-1307284">mount -t ldiskfs /dev/old /mnt/ost_old
</a><a name="50438194_pgfId-1307285">mount -t ldiskfs /dev/new /mnt/ost_new
</a><a name="50438194_pgfId-1307286">rsync -aSv /mnt/ost_old/ /mnt/ost_new
</a><a name="50438194_pgfId-1307287"># note trailing slash on ost_old/
</a></pre><ul style="font-family: Times; "><li><a name="50438194_pgfId-1307288">If you are unable to connect both sets of disk to the same computer, use <kbd style="font-family: courier; white-space: pre;">rsync</kbd> to copy over the network using <kbd style="font-family: courier; white-space: pre;">rsh</kbd> (or <kbd style="font-family: courier; white-space: pre;">ssh</kbd> with <kbd style="font-family: courier; white-space: pre;">-e ssh</kbd>):</a></li><li style="list-style: none;"><div style="line-height: 0.5em; "><br class="webkit-block-placeholder"></div></li></ul><pre style="font-family: courier; margin-left: 40px; "><a name="50438194_pgfId-1307289">rsync -aSvz /mnt/ost_old/ new_ost_node:/mnt/ost_new
</a></pre><ul style="font-family: Times; "><li><a name="50438194_pgfId-1307290">Use the same procedure for the MDS, with one additional step:</a></li><li style="list-style: none;"><div style="line-height: 0.5em; "><br class="webkit-block-placeholder"></div></li></ul><pre style="font-family: courier; margin-left: 40px; "><a name="50438194_pgfId-1307291">cd /mnt/mds_old; getfattr -R -e base64 -d . > /tmp/mdsea; \<copy all MDS files as above>; cd /mnt/mds_new; setfattr \--restore=/tmp/mdsea
</a></pre><h2 style="margin-top: 1.5em; font-size: 22pt; font-family: helvetica, sans-serif; "><a name="50438194_pgfId-1307352"></a></h2><div style="font-family: Arial; "><h2 style="margin-top: 1.5em; font-size: 22pt; font-family: helvetica, sans-serif;"></h2></div><div style="font-family: Arial; "><font face="courier">Run on all nodes (clients and OSS's & MDS / MGS) to change MGS (where mgs is the name of the server and /dev/sdb1 is the name of the unmounted oat)</font></div><div style="font-family: Arial; "><span style="font-family: courier;"><br></span></div><div style="font-family: Arial; "><a name="50438194_pgfId-1307352"><font face="courier">tunefs.lustre --mgsnode=mgs@tcp0 /dev/sdb1</font></a></div><div style="font-family: Arial; "><a name="50438194_pgfId-1307352"><font face="courier"><br></font></a></div><div style="font-family: Arial; "><a name="50438194_pgfId-1307352"><font face="courier">tunefs.lustre --erase-param --mgsnode=192.168.16.3@tcp --writeconf /dev/sdc1</font></a></div><div style="font-family: Arial; "><a name="50438194_pgfId-1307352"><font face="courier"><br></font></a></div><div style="font-family: Arial; "><a name="50438194_pgfId-1307352"><pre style="color: rgb(0, 0, 0); ">After copying with rsync, I had to
cd /srv/mdt;
rm CATALOGS OBJECTS/*
on the new MDT partition.</pre><div><br></div></a></div><div style="font-family: Arial; "><a name="50438194_pgfId-1307352"><font face="courier"><br></font></a></div><div style="font-family: Arial; "><a name="50438194_pgfId-1307352"><font face="courier"><br></font></a></div><div style="font-family: Arial; "><a name="50438194_pgfId-1307352"><font face="courier">I also upgraded from 1.8.8 to 2. I managed to mount the Lustre filesystem and if I do lfs df -h, I get:</font></a></div><div style="font-family: Arial; "><a name="50438194_pgfId-1307352"><font face="courier"><br></font></a></div><div style="font-family: Arial; "><a name="50438194_pgfId-1307352"><font face="courier">NB> I deactivated those two OSTs below.</font></a></div><div style="font-family: Arial; "><a name="50438194_pgfId-1307352"><font face="courier"><br></font></a></div><div><a name="50438194_pgfId-1307352"><font face="courier"><div>[root@mgs data]# lfs df -h</div><div>UUID bytes Used Available Use% Mounted on</div><div>AC3-MDT0000_UUID 37.5G 499.5M 34.5G 1% /data[MDT:0]</div><div>AC3-OST0000_UUID 16.4T 2.2T 13.3T 14% /data[OST:0]</div><div>AC3-OST0001_UUID 16.4T 1.8T 13.7T 12% /data[OST:1]</div><div>AC3-OST0002_UUID 6.4T 6.0T 49.2G 99% /data[OST:2]</div><div>AC3-OST0003_UUID 6.4T 6.1T 912.9M 100% /data[OST:3]</div><div>AC3-OST0004_UUID 4.3T 4.1T 17.2G 100% /data[OST:4]</div><div>AC3-OST0005_UUID 1.9T 1.8T 29.0M 100% /data[OST:5]</div><div>AC3-OST0006_UUID 1.9T 1.5T 282.1G 85% /data[OST:6]</div><div>AC3-OST0007_UUID 1.9T 1.8T 434.3M 100% /data[OST:7]</div><div>AC3-OST0008_UUID 1.9T 1.8T 12.9G 99% /data[OST:8]</div><div>OST0009 : Resource temporarily unavailable</div><div>AC3-OST000a_UUID 2.0T 1.9T 971.7M 100% /data[OST:10]</div><div>AC3-OST000b_UUID 2.0T 1.9T 537.4M 100% /data[OST:11]</div><div>AC3-OST000c_UUID 2.0T 1.9T 1.4G 100% /data[OST:12]</div><div>AC3-OST000d_UUID 2.0T 1.6T 234.5G 88% /data[OST:13]</div><div>AC3-OST000e_UUID 2.0T 1.9T 3.1G 100% /data[OST:14]</div><div>OST000f : Resource temporarily unavailable</div><div>AC3-OST0010_UUID 2.0T 1.9T 3.6G 100% /data[OST:16]</div><div>AC3-OST0011_UUID 2.0T 1.6T 248.3G 87% /data[OST:17]</div><div>AC3-OST0012_UUID 2.0T 1.9T 3.0G 100% /data[OST:18]</div><div>AC3-OST0013_UUID 2.0T 1.9T 1.4G 100% /data[OST:19]</div><div>AC3-OST0014_UUID 2.0T 1.9T 2.8G 100% /data[OST:20]</div><div>AC3-OST0015_UUID 2.0T 1.9T 2.7G 100% /data[OST:21]</div><div>AC3-OST0016_UUID 2.0T 1.4T 473.3G 75% /data[OST:22]</div><div>AC3-OST0017_UUID 1.2T 917.9G 230.1G 80% /data[OST:23]</div><div>AC3-OST0018_UUID 2.0T 1.3T 616.7G 68% /data[OST:24]</div><div>AC3-OST0019_UUID 2.0T 1.2T 671.9G 65% /data[OST:25]</div><div>AC3-OST001a_UUID 2.0T 1.1T 769.4G 59% /data[OST:26]</div><div>AC3-OST001b_UUID 2.0T 1.2T 642.2G 66% /data[OST:27]</div><div>AC3-OST001c_UUID 2.0T 1.2T 633.0G 67% /data[OST:28]</div><div><br></div><div>filesystem summary: 92.1T 55.6T 31.9T 64% /data</div><div><br></div><div>However, there are hardly any files or folders there - most of them have disappeared, but they are clearly on the OSTs (according to the above command). Is there any way I can reindex the OST'S / MDS to recognise the files and folders properly?</div><div><br></div><div>When I do lctl dl I get:</div><div><br></div><div><div>[root@mgs data]# lctl dl</div><div> 1 UP mgc MGC192.168.16.3@tcp 285232ea-c153-187d-1033-47c23b432296 5</div><div> 2 UP lov AC3-MDT0000-mdtlov AC3-MDT0000-mdtlov_UUID 4</div><div> 3 UP mdt AC3-MDT0000 AC3-MDT0000_UUID 5</div><div> 4 UP mds mdd_obd-AC3-MDT0000 mdd_obd_uuid-AC3-MDT0000 3</div><div> 5 UP osc AC3-OST0002-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5</div><div> 6 UP osc AC3-OST0003-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5</div><div> 7 UP osc AC3-OST0004-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5</div><div> 8 UP osc AC3-OST000a-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5</div><div> 9 UP osc AC3-OST000b-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5</div><div> 10 UP osc AC3-OST000c-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5</div><div> 11 UP osc AC3-OST000d-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5</div><div> 12 UP osc AC3-OST000e-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5</div><div> 13 UP osc AC3-OST0011-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5</div><div> 14 UP osc AC3-OST0013-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5</div><div> 15 UP osc AC3-OST0012-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5</div><div> 16 UP osc AC3-OST0014-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5</div><div> 17 UP osc AC3-OST0015-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5</div><div> 18 UP osc AC3-OST0016-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5</div><div> 19 UP osc AC3-OST0017-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5</div><div> 20 UP osc AC3-OST0001-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5</div><div> 21 UP osc AC3-OST0000-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5</div><div> 22 UP osc AC3-OST0010-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5</div><div> 23 UP osc AC3-OST0005-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5</div><div> 24 UP osc AC3-OST0006-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5</div><div> 25 UP osc AC3-OST0008-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5</div><div> 26 UP osc AC3-OST0007-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5</div><div> 27 UP osc AC3-OST0018-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5</div><div> 28 UP osc AC3-OST0019-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5</div><div> 29 UP osc AC3-OST001a-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5</div><div> 30 UP osc AC3-OST001b-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5</div><div> 31 UP osc AC3-OST001c-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5</div><div> 32 UP lov AC3-clilov-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 4</div><div> 33 UP lmv AC3-clilmv-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 4</div><div> 34 UP mdc AC3-MDT0000-mdc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5</div><div> 35 UP osc AC3-OST0002-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5</div><div> 36 UP osc AC3-OST0003-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5</div><div> 37 UP osc AC3-OST0004-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5</div><div> 38 UP osc AC3-OST000a-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5</div><div> 39 UP osc AC3-OST000b-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5</div><div> 40 UP osc AC3-OST000c-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5</div><div> 41 UP osc AC3-OST000d-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5</div><div> 42 UP osc AC3-OST000e-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5</div><div> 43 UP osc AC3-OST0011-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5</div><div> 44 UP osc AC3-OST0013-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5</div><div> 45 UP osc AC3-OST0012-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5</div><div> 46 UP osc AC3-OST0014-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5</div><div> 47 UP osc AC3-OST0015-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5</div><div> 48 UP osc AC3-OST0016-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5</div><div> 49 UP osc AC3-OST0017-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5</div><div> 50 UP osc AC3-OST0001-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5</div><div> 51 UP osc AC3-OST0000-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5</div><div> 52 UP osc AC3-OST0010-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5</div><div> 53 UP osc AC3-OST0005-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5</div><div> 54 UP osc AC3-OST0006-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5</div><div> 55 UP osc AC3-OST0008-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5</div><div> 56 UP osc AC3-OST0007-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5</div><div> 57 UP osc AC3-OST0018-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5</div><div> 58 UP osc AC3-OST0019-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5</div><div> 59 UP osc AC3-OST001a-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5</div><div> 60 UP osc AC3-OST001b-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5</div><div> 61 UP osc AC3-OST001c-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5</div></div><div><br></div><div>SO it all looks fine and /var/log/messages and dmesg seem to look OK.</div><div><br></div><div>Thanks in advance!</div></font></a></div><div style="font-family: Arial; "><br></div><div><br></div><div apple-content-edited="true">
<span class="Apple-style-span" style="border-collapse: separate; border-spacing: 0px; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Tahoma; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div><span><br class="Apple-interchange-newline"><font class="Apple-style-span" face="Arial">Alastair Ferguson<br>IT Manager<br>Capital Markets CRC Limited (CMCRC)<br>Telephone: +61 2 8088 4222<br>Mobile: +61 424 235 159<br>Fax: +61 2 8088 4201<br><a href="http://www.cmcrc.com">www.cmcrc.com</a> </font></span></div><div><span><br></span></div></span><br class="Apple-interchange-newline"><span></span><span></span><span></span><span><img height="87" width="253" id="d8695b9a-c7c3-4489-817c-b214447ae569" apple-width="yes" apple-height="yes" src="cid:3340953947_834664"></span><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Tahoma; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Tahoma; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Tahoma; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Tahoma; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><span class="Apple-style-span" style="font-family: Helvetica; "><br></span><font class="Apple-style-span" face="Arial" size="1"><span class="Apple-style-span" style="font-size: 9px; ">Capital Markets CRC Ltd - Confidential Communication<br>The information contained in this e-mail is confidential. It is intended for the addressee only. If you receive this e-mail by mistake please promptly inform us by reply e-mail and then delete the e-mail and destroy any printed copy. You must not disclose or use in any way the information in the e-mail. There is no warranty that this e-mail is error or virus free. It may be a private communication, and if so, does not represent the views of the CMCRC and its associates. If it is a private communication, care should be taken in opening it to ensure that undue offence is not given.</span></font><span class="Apple-style-span" style="font-family: Helvetica; "><br></span><span class="Apple-style-span" style="font-family: Helvetica; "><br></span></span></span></span></span>
</span></div>
<br></div></body></html>