[Lustre-discuss] Lost folders after changing MDS
Alastair Ferguson
aferguson at cmcrc.com
Mon Feb 11 19:18:30 PST 2013
OK, so our old MDS had hardware issues so I configured a new MGS / MDS on a VM (this is a backup lustre filesystem and I wanted to separate the MGS / MDS from OSS of the previous), and then did this:
For example:
mount -t ldiskfs /dev/old /mnt/ost_old
mount -t ldiskfs /dev/new /mnt/ost_new
rsync -aSv /mnt/ost_old/ /mnt/ost_new
# note trailing slash on ost_old/
If you are unable to connect both sets of disk to the same computer, use rsync to copy over the network using rsh (or ssh with -e ssh):
rsync -aSvz /mnt/ost_old/ new_ost_node:/mnt/ost_new
Use the same procedure for the MDS, with one additional step:
cd /mnt/mds_old; getfattr -R -e base64 -d . > /tmp/mdsea; \<copy all MDS files as above>; cd /mnt/mds_new; setfattr \--restore=/tmp/mdsea
Run on all nodes (clients and OSS's & MDS / MGS) to change MGS (where mgs is the name of the server and /dev/sdb1 is the name of the unmounted oat)
tunefs.lustre --mgsnode=mgs at tcp0 /dev/sdb1
tunefs.lustre --erase-param --mgsnode=192.168.16.3 at tcp --writeconf /dev/sdc1
After copying with rsync, I had to
cd /srv/mdt;
rm CATALOGS OBJECTS/*
on the new MDT partition.
I also upgraded from 1.8.8 to 2. I managed to mount the Lustre filesystem and if I do lfs df -h, I get:
NB> I deactivated those two OSTs below.
[root at mgs data]# lfs df -h
UUID bytes Used Available Use% Mounted on
AC3-MDT0000_UUID 37.5G 499.5M 34.5G 1% /data[MDT:0]
AC3-OST0000_UUID 16.4T 2.2T 13.3T 14% /data[OST:0]
AC3-OST0001_UUID 16.4T 1.8T 13.7T 12% /data[OST:1]
AC3-OST0002_UUID 6.4T 6.0T 49.2G 99% /data[OST:2]
AC3-OST0003_UUID 6.4T 6.1T 912.9M 100% /data[OST:3]
AC3-OST0004_UUID 4.3T 4.1T 17.2G 100% /data[OST:4]
AC3-OST0005_UUID 1.9T 1.8T 29.0M 100% /data[OST:5]
AC3-OST0006_UUID 1.9T 1.5T 282.1G 85% /data[OST:6]
AC3-OST0007_UUID 1.9T 1.8T 434.3M 100% /data[OST:7]
AC3-OST0008_UUID 1.9T 1.8T 12.9G 99% /data[OST:8]
OST0009 : Resource temporarily unavailable
AC3-OST000a_UUID 2.0T 1.9T 971.7M 100% /data[OST:10]
AC3-OST000b_UUID 2.0T 1.9T 537.4M 100% /data[OST:11]
AC3-OST000c_UUID 2.0T 1.9T 1.4G 100% /data[OST:12]
AC3-OST000d_UUID 2.0T 1.6T 234.5G 88% /data[OST:13]
AC3-OST000e_UUID 2.0T 1.9T 3.1G 100% /data[OST:14]
OST000f : Resource temporarily unavailable
AC3-OST0010_UUID 2.0T 1.9T 3.6G 100% /data[OST:16]
AC3-OST0011_UUID 2.0T 1.6T 248.3G 87% /data[OST:17]
AC3-OST0012_UUID 2.0T 1.9T 3.0G 100% /data[OST:18]
AC3-OST0013_UUID 2.0T 1.9T 1.4G 100% /data[OST:19]
AC3-OST0014_UUID 2.0T 1.9T 2.8G 100% /data[OST:20]
AC3-OST0015_UUID 2.0T 1.9T 2.7G 100% /data[OST:21]
AC3-OST0016_UUID 2.0T 1.4T 473.3G 75% /data[OST:22]
AC3-OST0017_UUID 1.2T 917.9G 230.1G 80% /data[OST:23]
AC3-OST0018_UUID 2.0T 1.3T 616.7G 68% /data[OST:24]
AC3-OST0019_UUID 2.0T 1.2T 671.9G 65% /data[OST:25]
AC3-OST001a_UUID 2.0T 1.1T 769.4G 59% /data[OST:26]
AC3-OST001b_UUID 2.0T 1.2T 642.2G 66% /data[OST:27]
AC3-OST001c_UUID 2.0T 1.2T 633.0G 67% /data[OST:28]
filesystem summary: 92.1T 55.6T 31.9T 64% /data
However, there are hardly any files or folders there - most of them have disappeared, but they are clearly on the OSTs (according to the above command). Is there any way I can reindex the OST'S / MDS to recognise the files and folders properly?
When I do lctl dl I get:
[root at mgs data]# lctl dl
1 UP mgc MGC192.168.16.3 at tcp 285232ea-c153-187d-1033-47c23b432296 5
2 UP lov AC3-MDT0000-mdtlov AC3-MDT0000-mdtlov_UUID 4
3 UP mdt AC3-MDT0000 AC3-MDT0000_UUID 5
4 UP mds mdd_obd-AC3-MDT0000 mdd_obd_uuid-AC3-MDT0000 3
5 UP osc AC3-OST0002-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5
6 UP osc AC3-OST0003-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5
7 UP osc AC3-OST0004-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5
8 UP osc AC3-OST000a-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5
9 UP osc AC3-OST000b-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5
10 UP osc AC3-OST000c-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5
11 UP osc AC3-OST000d-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5
12 UP osc AC3-OST000e-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5
13 UP osc AC3-OST0011-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5
14 UP osc AC3-OST0013-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5
15 UP osc AC3-OST0012-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5
16 UP osc AC3-OST0014-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5
17 UP osc AC3-OST0015-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5
18 UP osc AC3-OST0016-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5
19 UP osc AC3-OST0017-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5
20 UP osc AC3-OST0001-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5
21 UP osc AC3-OST0000-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5
22 UP osc AC3-OST0010-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5
23 UP osc AC3-OST0005-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5
24 UP osc AC3-OST0006-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5
25 UP osc AC3-OST0008-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5
26 UP osc AC3-OST0007-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5
27 UP osc AC3-OST0018-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5
28 UP osc AC3-OST0019-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5
29 UP osc AC3-OST001a-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5
30 UP osc AC3-OST001b-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5
31 UP osc AC3-OST001c-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5
32 UP lov AC3-clilov-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 4
33 UP lmv AC3-clilmv-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 4
34 UP mdc AC3-MDT0000-mdc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5
35 UP osc AC3-OST0002-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5
36 UP osc AC3-OST0003-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5
37 UP osc AC3-OST0004-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5
38 UP osc AC3-OST000a-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5
39 UP osc AC3-OST000b-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5
40 UP osc AC3-OST000c-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5
41 UP osc AC3-OST000d-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5
42 UP osc AC3-OST000e-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5
43 UP osc AC3-OST0011-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5
44 UP osc AC3-OST0013-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5
45 UP osc AC3-OST0012-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5
46 UP osc AC3-OST0014-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5
47 UP osc AC3-OST0015-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5
48 UP osc AC3-OST0016-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5
49 UP osc AC3-OST0017-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5
50 UP osc AC3-OST0001-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5
51 UP osc AC3-OST0000-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5
52 UP osc AC3-OST0010-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5
53 UP osc AC3-OST0005-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5
54 UP osc AC3-OST0006-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5
55 UP osc AC3-OST0008-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5
56 UP osc AC3-OST0007-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5
57 UP osc AC3-OST0018-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5
58 UP osc AC3-OST0019-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5
59 UP osc AC3-OST001a-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5
60 UP osc AC3-OST001b-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5
61 UP osc AC3-OST001c-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5
SO it all looks fine and /var/log/messages and dmesg seem to look OK.
Thanks in advance!
Alastair Ferguson
IT Manager
Capital Markets CRC Limited (CMCRC)
Telephone: +61 2 8088 4222
Mobile: +61 424 235 159
Fax: +61 2 8088 4201
www.cmcrc.com
Capital Markets CRC Ltd - Confidential Communication
The information contained in this e-mail is confidential. It is intended for the addressee only. If you receive this e-mail by mistake please promptly inform us by reply e-mail and then delete the e-mail and destroy any printed copy. You must not disclose or use in any way the information in the e-mail. There is no warranty that this e-mail is error or virus free. It may be a private communication, and if so, does not represent the views of the CMCRC and its associates. If it is a private communication, care should be taken in opening it to ensure that undue offence is not given.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20130212/a1d7e466/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 13489 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20130212/a1d7e466/attachment.png>
More information about the lustre-discuss
mailing list