[Lustre-discuss] Lost folders after changing MDS

Alastair Ferguson aferguson at cmcrc.com
Mon Feb 11 19:18:30 PST 2013

OK, so our old MDS had hardware issues so I configured a new MGS / MDS on a VM (this is a backup lustre filesystem and I wanted to separate the MGS / MDS from OSS of the previous), and then did this:

For example:

mount -t ldiskfs /dev/old /mnt/ost_old
mount -t ldiskfs /dev/new /mnt/ost_new
rsync -aSv /mnt/ost_old/ /mnt/ost_new
# note trailing slash on ost_old/

If you are unable to connect both sets of disk to the same computer, use rsync to copy over the network using rsh (or ssh with -e ssh):

rsync -aSvz /mnt/ost_old/ new_ost_node:/mnt/ost_new

Use the same procedure for the MDS, with one additional step:

cd /mnt/mds_old; getfattr -R -e base64 -d . > /tmp/mdsea; \<copy all MDS files as above>; cd /mnt/mds_new; setfattr \--restore=/tmp/mdsea

Run on all nodes (clients and OSS's & MDS / MGS) to change MGS (where mgs is the name of the server and /dev/sdb1 is the name of the unmounted oat)

tunefs.lustre --mgsnode=mgs at tcp0 /dev/sdb1

tunefs.lustre --erase-param --mgsnode= at tcp --writeconf /dev/sdc1

After copying with rsync, I had to
cd   /srv/mdt;
on the new MDT partition.

I also upgraded from 1.8.8 to 2. I managed to mount the Lustre filesystem and if I do lfs df -h, I get:

NB> I deactivated those two OSTs below.

[root at mgs data]# lfs df -h
UUID                       bytes        Used   Available Use% Mounted on
AC3-MDT0000_UUID           37.5G      499.5M       34.5G   1% /data[MDT:0]
AC3-OST0000_UUID           16.4T        2.2T       13.3T  14% /data[OST:0]
AC3-OST0001_UUID           16.4T        1.8T       13.7T  12% /data[OST:1]
AC3-OST0002_UUID            6.4T        6.0T       49.2G  99% /data[OST:2]
AC3-OST0003_UUID            6.4T        6.1T      912.9M 100% /data[OST:3]
AC3-OST0004_UUID            4.3T        4.1T       17.2G 100% /data[OST:4]
AC3-OST0005_UUID            1.9T        1.8T       29.0M 100% /data[OST:5]
AC3-OST0006_UUID            1.9T        1.5T      282.1G  85% /data[OST:6]
AC3-OST0007_UUID            1.9T        1.8T      434.3M 100% /data[OST:7]
AC3-OST0008_UUID            1.9T        1.8T       12.9G  99% /data[OST:8]
OST0009             : Resource temporarily unavailable
AC3-OST000a_UUID            2.0T        1.9T      971.7M 100% /data[OST:10]
AC3-OST000b_UUID            2.0T        1.9T      537.4M 100% /data[OST:11]
AC3-OST000c_UUID            2.0T        1.9T        1.4G 100% /data[OST:12]
AC3-OST000d_UUID            2.0T        1.6T      234.5G  88% /data[OST:13]
AC3-OST000e_UUID            2.0T        1.9T        3.1G 100% /data[OST:14]
OST000f             : Resource temporarily unavailable
AC3-OST0010_UUID            2.0T        1.9T        3.6G 100% /data[OST:16]
AC3-OST0011_UUID            2.0T        1.6T      248.3G  87% /data[OST:17]
AC3-OST0012_UUID            2.0T        1.9T        3.0G 100% /data[OST:18]
AC3-OST0013_UUID            2.0T        1.9T        1.4G 100% /data[OST:19]
AC3-OST0014_UUID            2.0T        1.9T        2.8G 100% /data[OST:20]
AC3-OST0015_UUID            2.0T        1.9T        2.7G 100% /data[OST:21]
AC3-OST0016_UUID            2.0T        1.4T      473.3G  75% /data[OST:22]
AC3-OST0017_UUID            1.2T      917.9G      230.1G  80% /data[OST:23]
AC3-OST0018_UUID            2.0T        1.3T      616.7G  68% /data[OST:24]
AC3-OST0019_UUID            2.0T        1.2T      671.9G  65% /data[OST:25]
AC3-OST001a_UUID            2.0T        1.1T      769.4G  59% /data[OST:26]
AC3-OST001b_UUID            2.0T        1.2T      642.2G  66% /data[OST:27]
AC3-OST001c_UUID            2.0T        1.2T      633.0G  67% /data[OST:28]

filesystem summary:        92.1T       55.6T       31.9T  64% /data

However, there are hardly any files or folders there - most of them have disappeared, but they are clearly on the OSTs (according to the above command). Is there any way I can reindex the OST'S / MDS to recognise the files and folders properly?

When I do lctl dl I get:

[root at mgs data]# lctl dl
  1 UP mgc MGC192.168.16.3 at tcp 285232ea-c153-187d-1033-47c23b432296 5
  2 UP lov AC3-MDT0000-mdtlov AC3-MDT0000-mdtlov_UUID 4
  3 UP mdt AC3-MDT0000 AC3-MDT0000_UUID 5
  4 UP mds mdd_obd-AC3-MDT0000 mdd_obd_uuid-AC3-MDT0000 3
  5 UP osc AC3-OST0002-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5
  6 UP osc AC3-OST0003-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5
  7 UP osc AC3-OST0004-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5
  8 UP osc AC3-OST000a-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5
  9 UP osc AC3-OST000b-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5
 10 UP osc AC3-OST000c-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5
 11 UP osc AC3-OST000d-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5
 12 UP osc AC3-OST000e-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5
 13 UP osc AC3-OST0011-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5
 14 UP osc AC3-OST0013-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5
 15 UP osc AC3-OST0012-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5
 16 UP osc AC3-OST0014-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5
 17 UP osc AC3-OST0015-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5
 18 UP osc AC3-OST0016-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5
 19 UP osc AC3-OST0017-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5
 20 UP osc AC3-OST0001-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5
 21 UP osc AC3-OST0000-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5
 22 UP osc AC3-OST0010-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5
 23 UP osc AC3-OST0005-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5
 24 UP osc AC3-OST0006-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5
 25 UP osc AC3-OST0008-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5
 26 UP osc AC3-OST0007-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5
 27 UP osc AC3-OST0018-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5
 28 UP osc AC3-OST0019-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5
 29 UP osc AC3-OST001a-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5
 30 UP osc AC3-OST001b-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5
 31 UP osc AC3-OST001c-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5
 32 UP lov AC3-clilov-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 4
 33 UP lmv AC3-clilmv-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 4
 34 UP mdc AC3-MDT0000-mdc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5
 35 UP osc AC3-OST0002-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5
 36 UP osc AC3-OST0003-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5
 37 UP osc AC3-OST0004-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5
 38 UP osc AC3-OST000a-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5
 39 UP osc AC3-OST000b-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5
 40 UP osc AC3-OST000c-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5
 41 UP osc AC3-OST000d-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5
 42 UP osc AC3-OST000e-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5
 43 UP osc AC3-OST0011-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5
 44 UP osc AC3-OST0013-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5
 45 UP osc AC3-OST0012-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5
 46 UP osc AC3-OST0014-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5
 47 UP osc AC3-OST0015-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5
 48 UP osc AC3-OST0016-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5
 49 UP osc AC3-OST0017-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5
 50 UP osc AC3-OST0001-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5
 51 UP osc AC3-OST0000-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5
 52 UP osc AC3-OST0010-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5
 53 UP osc AC3-OST0005-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5
 54 UP osc AC3-OST0006-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5
 55 UP osc AC3-OST0008-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5
 56 UP osc AC3-OST0007-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5
 57 UP osc AC3-OST0018-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5
 58 UP osc AC3-OST0019-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5
 59 UP osc AC3-OST001a-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5
 60 UP osc AC3-OST001b-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5
 61 UP osc AC3-OST001c-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5

SO it all looks fine and /var/log/messages and dmesg seem to look OK.

Thanks in advance!

Alastair Ferguson
IT Manager
Capital Markets CRC Limited (CMCRC)
Telephone: +61 2 8088 4222
Mobile: +61 424 235 159
Fax: +61 2 8088 4201

Capital Markets CRC Ltd - Confidential Communication
The information contained in this e-mail is confidential.  It is intended for the addressee only.  If you receive this e-mail by mistake please promptly inform us by reply e-mail and then delete the e-mail and destroy any printed copy. You must not disclose or use in any way the information in the e-mail. There is no warranty that this e-mail is error or virus free. It may be a private communication, and if so, does not represent the views of the CMCRC and its associates.  If it is a private communication, care should be taken in opening it to ensure that undue offence is not given.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20130212/a1d7e466/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 13489 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20130212/a1d7e466/attachment.png>

More information about the lustre-discuss mailing list