<div dir="ltr">Here are the notes from my attempt to move MDT from pool4 to lustre3. Any ideas on why the transfer of MDT didn't succeed?<br>And the reason why all the OSTs were marked inactive?<br><br>Thanks,<br>Ron<br>
<br>The following contains the commands we executed and the associated log messages.<br><br>Note:<br>/dev/sda4 on pool4 is a Hardware RAID 1 device<br><br>/dev/mapper/lustrevol-lustrelv on lustre3 is a Volume Group on a LVM2 Physical Volume which is on a software RAID device /dev/md3<br>
<br>[root@lustre3 ~]# pvdisplay<br> --- Physical volume ---<br> PV Name /dev/md3<br> VG Name lustrevol<br> PV Size 592.86 GB / not usable 1.00 MB<br> Allocatable yes<br>
PE Size (KByte) 4096<br> Total PE 151772<br> Free PE 113372<br> Allocated PE 38400<br> PV UUID KRY0HY-BhjD-l8qR-14Qw-cQ1T-NNHT-LXT3Bs<br><br>[root@lustre3 ~]# vgdisplay<br>
--- Volume group ---<br> VG Name lustrevol<br> System ID<br> Format lvm2<br> Metadata Areas 1<br> Metadata Sequence No 2<br> VG Access read/write<br> VG Status resizable<br>
MAX LV 0<br> Cur LV 1<br> Open LV 0<br> Max PV 0<br> Cur PV 1<br> Act PV 1<br> VG Size 592.86 GB<br> PE Size 4.00 MB<br>
Total PE 151772<br> Alloc PE / Size 38400 / 150.00 GB<br> Free PE / Size 113372 / 442.86 GB<br> VG UUID XFvbDk-Ukfg-fTcQ-XcNp-rX0f-hYTl-XbkGKs <br><br>[root@lqcd-pool4 ~]# mount -t ldiskfs /dev/sda4 /mnt/mdt<br>
[root@lqcd-pool4 ~]# cd /mnt/mdt/<br>[root@lqcd-pool4 mdt]# getfattr -R -d -m '.*' -P . > /root/ea.bak<br>[root@lqcd-pool4 mdt]# /usr/bin/rcp /root/ea.bak lustre3:/root/ea.bak<br><br>[root@lustre3 ~]# mkfs.lustre --fsname=lustre --mdt --mgs --param lov.stripecount=1 --mkfsoptions="-m 0" --reformat /dev/mapper/lustrevol-lustrelv<br>
[root@lustre3 ~]# mount -t ldiskfs /dev/mapper/lustrevol-lustrelv /mnt/mdt<br><br>[root@lqcd-pool4 mdt]# export RSYNC_RSH=/usr/bin/rsh<br>[root@lqcd-pool4 mdt]# rsync -aSvz --ignore-existing --ignore-times /mnt/mdt/ lustre3:/mnt/mdt > /tmp/rsync.log 2>&1<br>
<br>[root@lustre3 ~]# cd /mnt/mdt<br>[root@lustre3 mdt]# setfattr --restore=/root/ea.bak<br><br>The following command was executed on all the 24 OSTs<br> tunefs.lustre --erase-param --mgsnode=lustre3 --writeconf /dev/sde1<br>
<br>[root@lustre3 ~]# mount -t lustre /dev/mapper/lustrevol-lustrelv /mnt/mdt<br>mount.lustre: mount /dev/mapper/lustrevol-lustrelv at /mnt/mdt failed: Address already in use<br>The target service's index is already in use. (/dev/mapper/lustrevol-lustrelv)<br>
<br>Oct 6 16:39:28 lustre3 kernel: kjournald starting. Commit interval 5 seconds<br>Oct 6 16:39:28 lustre3 kernel: LDISKFS FS on dm-0, internal journal<br>Oct 6 16:39:28 lustre3 kernel: LDISKFS-fs: mounted filesystem with ordered data mode.<br>
Oct 6 16:39:28 lustre3 kernel: kjournald starting. Commit interval 5 seconds<br>Oct 6 16:39:28 lustre3 kernel: LDISKFS FS on dm-0, internal journal<br>Oct 6 16:39:28 lustre3 kernel: LDISKFS-fs: mounted filesystem with ordered data mode.<br>
Oct 6 16:39:28 lustre3 kernel: Lustre: MGS MGS started<br>Oct 6 16:39:28 lustre3 kernel: LustreError: 13e-c: MDT index must = 0 (until Clustered MetaData feature is ready.)<br>Oct 6 16:39:28 lustre3 kernel: LustreError: 140-5: Server lustre-MDTffff requested index 0, but that index is already in use<br>
Oct 6 16:39:28 lustre3 kernel: LustreError: 5026:0:(mgs_llog.c:1672:mgs_write_log_target()) Can't get index (-98)<br>Oct 6 16:39:28 lustre3 kernel: LustreError: 5026:0:(mgs_handler.c:431:mgs_handle_target_reg()) Failed to write lustre-MDTffff log (-98)<br>
Oct 6 16:39:29 lustre3 kernel: LustreError: 5026:0:(mgs_handler.c:625:mgs_handle()) MGS handle cmd=253 rc=-98<br>Oct 6 16:39:29 lustre3 kernel: LustreError: 5026:0:(ldlm_lib.c:1536:target_send_reply_msg()) @@@ processing error (-98) req@ffff8102152b3450 x117/t0 o253->fb445385-7ef9-62f3-1e58-db8f8dc29917@NET_0x9000000000000_UUID:0/0 lens 4672/4672 e 0 to 0 dl 1223329268 ref 1 fl Interpret:/0/0 rc 0/0<br>
Oct 6 16:39:29 lustre3 kernel: LustreError: 11-0: an error occurred while communicating with 0@lo. The mgs_target_reg operation failed with -98<br>Oct 6 16:39:29 lustre3 kernel: LustreError: 4964:0:(obd_mount.c:1062:server_start_targets()) Required registration failed for lustre-MDTffff: -98<br>
Oct 6 16:39:29 lustre3 kernel: LustreError: 4964:0:(obd_mount.c:1597:server_fill_super()) Unable to start targets: -98<br>Oct 6 16:39:29 lustre3 kernel: LustreError: 4964:0:(obd_mount.c:1382:server_put_super()) no obd lustre-MDTffff<br>
Oct 6 16:39:29 lustre3 kernel: LustreError: 4964:0:(obd_mount.c:119:server_deregister_mount()) lustre-MDTffff not registered<br>Oct 6 16:39:29 lustre3 kernel: Lustre: MGS has stopped.<br>Oct 6 16:39:29 lustre3 kernel: Lustre: server umount lustre-MDTffff complete<br>
Oct 6 16:39:29 lustre3 kernel: LustreError: 4964:0:(obd_mount.c:1951:lustre_fill_super()) Unable to mount (-98)<br><br>[root@lustre3 ~]# tunefs.lustre --erase-params --mgs --mdt --writeconf /dev/lustrevol/lustrelv <br>checking for existing Lustre data: found CONFIGS/mountdata<br>
Reading CONFIGS/mountdata<br><br> Read previous values:<br>Target: lustre-MDTffff<br>Index: unassigned<br>Lustre FS: lustre<br>Mount type: ldiskfs<br>Flags: 0x75<br> (MDT MGS needs_index first_time update )<br>
Persistent mount opts: errors=remount-ro,iopen_nopriv,user_xattr<br>Parameters: lov.stripecount=1 mdt.group_upcall=/usr/sbin/l_getgroups<br><br><br> Permanent disk data:<br>Target: lustre-MDTffff<br>Index: unassigned<br>
Lustre FS: lustre<br>Mount type: ldiskfs<br>Flags: 0x175<br> (MDT MGS needs_index first_time update writeconf )<br>Persistent mount opts: errors=remount-ro,iopen_nopriv,user_xattr<br>Parameters:<br><br>
Writing CONFIGS/mountdata<br><br>[root@lustre3 ~]# mount -t lustre /dev/mapper/lustrevol-lustrelv /mnt/mdt<br><br>[root@lustre3 ~]# mount -v -t lustre /dev/sde1 /mnt/sata1-1-3<br>arg[0] = /sbin/mount.lustre<br>arg[1] = -v<br>
arg[2] = -o<br>arg[3] = rw,noauto,_netdev<br>arg[4] = /dev/sde1<br>arg[5] = /mnt/sata1-1-3<br>source = /dev/sde1 (/dev/sde1), target = /mnt/sata1-1-3<br>options = rw,noauto,_netdev<br>mounting device /dev/sde1 at /mnt/sata1-1-3, flags=0 options=device=/dev/sde1<br>
<br>Oct 6 16:40:57 lustre3 kernel: kjournald starting. Commit interval 5 seconds<br>Oct 6 16:40:57 lustre3 kernel: LDISKFS FS on dm-0, internal journal<br>Oct 6 16:40:57 lustre3 kernel: LDISKFS-fs: mounted filesystem with ordered data mode.<br>
Oct 6 16:41:05 lustre3 kernel: kjournald starting. Commit interval 5 seconds<br>Oct 6 16:41:05 lustre3 kernel: LDISKFS FS on dm-0, internal journal<br>Oct 6 16:41:05 lustre3 kernel: LDISKFS-fs: mounted filesystem with ordered data mode.<br>
Oct 6 16:41:05 lustre3 kernel: kjournald starting. Commit interval 5 seconds<br>Oct 6 16:41:05 lustre3 kernel: LDISKFS FS on dm-0, internal journal<br>Oct 6 16:41:05 lustre3 kernel: LDISKFS-fs: mounted filesystem with ordered data mode.<br>
Oct 6 16:41:05 lustre3 kernel: Lustre: MGS MGS started<br>Oct 6 16:41:05 lustre3 kernel: Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs.<br>Oct 6 16:41:05 lustre3 kernel: Lustre: Enabling user_xattr<br>
Oct 6 16:41:06 lustre3 kernel: LustreError: 5145:0:(fsfilt-ldiskfs.c:1283:fsfilt_ldiskfs_read_record()) can't read block: 0<br>Oct 6 16:41:06 lustre3 kernel: Lustre: MDT lustre-MDT0000 now serving dev (lustre-MDT0000/9b2d9c21-aeec-b2d2-4d55-b8e6d8a37b4a) with recovery enabled<br>
Oct 6 16:41:06 lustre3 kernel: Lustre: Server lustre-MDT0000 on device /dev/mapper/lustrevol-lustrelv has started<br>Oct 6 16:41:55 lustre3 kernel: kjournald starting. Commit interval 5 seconds<br>Oct 6 16:41:55 lustre3 kernel: LDISKFS FS on sde1, internal journal<br>
Oct 6 16:41:55 lustre3 kernel: LDISKFS-fs: mounted filesystem with ordered data mode.<br>Oct 6 16:41:55 lustre3 kernel: kjournald starting. Commit interval 5 seconds<br>Oct 6 16:41:55 lustre3 kernel: LDISKFS FS on sde1, internal journal<br>
Oct 6 16:41:55 lustre3 kernel: LDISKFS-fs: mounted filesystem with ordered data mode.<br>Oct 6 16:41:55 lustre3 kernel: LDISKFS-fs: file extents enabled<br>Oct 6 16:41:55 lustre3 kernel: LDISKFS-fs: mballoc enabled<br>
Oct 6 16:41:55 lustre3 kernel: Lustre: MGS: Regenerating lustre-OST0012 log by user request.<br>Oct 6 16:41:55 lustre3 kernel: Lustre: OST lustre-OST0012 now serving dev (lustre-OST0012/32f8d0ff-18d9-b05e-491b-477b8558b745) with recovery enabled<br>
Oct 6 16:41:55 lustre3 kernel: Lustre: Server lustre-OST0012 on device /dev/sde1 has started<br>Oct 6 16:42:00 lustre3 kernel: Lustre: 5536:0:(quota_master.c:1576:mds_quota_recovery()) Not all osts are active, abort quota recovery<br>
Oct 6 16:42:00 lustre3 kernel: LustreError: 5539:0:(llog_lvfs.c:597:llog_lvfs_create()) error looking up logfile 0x28c8020:0x3c23cd5e: rc -2<br>Oct 6 16:42:00 lustre3 kernel: LustreError: 5539:0:(osc_request.c:3586:osc_llog_init()) failed LLOG_MDS_OST_ORIG_CTXT<br>
Oct 6 16:42:00 lustre3 kernel: LustreError: 5539:0:(osc_request.c:3597:osc_llog_init()) osc 'lustre-OST0012-osc' tgt 'lustre-MDT0000' cnt 1 catid ffffc20000a3b240 rc=-2<br>Oct 6 16:42:00 lustre3 kernel: LustreError: 5539:0:(osc_request.c:3599:osc_llog_init()) logid 0x28c8020:0x3c23cd5e<br>
Oct 6 16:42:00 lustre3 kernel: LustreError: 5539:0:(lov_log.c:214:lov_llog_init()) error osc_llog_init idx 18 osc 'lustre-OST0012-osc' tgt 'lustre-MDT0000' (rc=-2)<br>Oct 6 16:42:00 lustre3 kernel: LustreError: 5539:0:(mds_log.c:207:mds_llog_init()) lov_llog_init err -2<br>
Oct 6 16:42:00 lustre3 kernel: LustreError: 5539:0:(llog_obd.c:394:llog_cat_initialize()) rc: -2<br>Oct 6 16:42:00 lustre3 kernel: LustreError: 5539:0:(mds_lov.c:855:__mds_lov_synchronize()) lustre-OST0012_UUID failed at update_mds: -2<br>
Oct 6 16:42:00 lustre3 kernel: LustreError: 5539:0:(mds_lov.c:898:__mds_lov_synchronize()) lustre-OST0012_UUID sync failed -2, deactivating<br><br><br>[root@lustre3 ~]# lctl dl<br> 0 UP mgs MGS MGS 11<br> 1 UP mgc MGC192.168.241.243@tcp b2bcceae-de69-e1b3-d96f-2971bba2fdfc 5<br>
2 UP mdt MDS MDS_uuid 3<br> 3 UP lov lustre-mdtlov lustre-mdtlov_UUID 4<br> 4 UP mds lustre-MDT0000 lustre-MDT0000_UUID 3<br> 5 UP ost OSS OSS_uuid 3<br> 6 UP obdfilter lustre-OST0012 lustre-OST0012_UUID 5<br> 7 IN osc lustre-OST0012-osc lustre-mdtlov_UUID 5<br>
8 UP obdfilter lustre-OST0013 lustre-OST0013_UUID 5<br> 9 IN osc lustre-OST0013-osc lustre-mdtlov_UUID 5<br> 10 UP obdfilter lustre-OST0014 lustre-OST0014_UUID 5<br> 11 UP obdfilter lustre-OST0015 lustre-OST0015_UUID 5<br>
12 IN osc lustre-OST0014-osc lustre-mdtlov_UUID 5<br> 13 IN osc lustre-OST0015-osc lustre-mdtlov_UUID 5<br> 14 UP obdfilter lustre-OST0016 lustre-OST0016_UUID 5<br> 15 IN osc lustre-OST0016-osc lustre-mdtlov_UUID 5<br> 16 UP obdfilter lustre-OST0017 lustre-OST0017_UUID 5<br>
17 IN osc lustre-OST0017-osc lustre-mdtlov_UUID 5<br> 18 IN osc lustre-OST000c-osc lustre-mdtlov_UUID 5<br> 19 IN osc lustre-OST000d-osc lustre-mdtlov_UUID 5<br> 20 IN osc lustre-OST000e-osc lustre-mdtlov_UUID 5<br> 21 IN osc lustre-OST000f-osc lustre-mdtlov_UUID 5<br>
22 IN osc lustre-OST0010-osc lustre-mdtlov_UUID 5<br> 23 IN osc lustre-OST0011-osc lustre-mdtlov_UUID 5<br> 24 IN osc lustre-OST0000-osc lustre-mdtlov_UUID 5<br> 25 IN osc lustre-OST0001-osc lustre-mdtlov_UUID 5<br> 26 IN osc lustre-OST0002-osc lustre-mdtlov_UUID 5<br>
27 IN osc lustre-OST0003-osc lustre-mdtlov_UUID 5<br> 28 IN osc lustre-OST0004-osc lustre-mdtlov_UUID 5<br> 29 IN osc lustre-OST0005-osc lustre-mdtlov_UUID 5<br> 30 IN osc lustre-OST0006-osc lustre-mdtlov_UUID 5<br> 31 IN osc lustre-OST0007-osc lustre-mdtlov_UUID 5<br>
32 IN osc lustre-OST0008-osc lustre-mdtlov_UUID 5<br> 33 IN osc lustre-OST0009-osc lustre-mdtlov_UUID 5<br> 34 IN osc lustre-OST000a-osc lustre-mdtlov_UUID 5<br> 35 IN osc lustre-OST000b-osc lustre-mdtlov_UUID 5<br><br><br>
<br><br><br></div>