<div dir="ltr"><div class="gmail_default" style="font-family:monospace,monospace">Afternoon</div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">I have an MDS running on spinning media and wish to migrate it to SSD's.</div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace"> Lustre 2.9.52</div><div class="gmail_default" style="font-family:monospace,monospace"> ZFS 0.7.0-rc3</div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">How do I do it?</div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">This is a loaded question :)</div><div class="gmail_default"><font face="monospace, monospace"><br></font></div><div class="gmail_default"><font face="monospace, monospace">The MDT is using ~2TB of space. I used the zfs send | zfs receive method to no avail. It was just too slow (I killed it after a week), gave no indication of how far though it was and the destination FS was using ~4TB of space (more than the 2TB of the source FS).</font></div><div class="gmail_default"><font face="monospace, monospace"><br></font></div><div class="gmail_default"><font face="monospace, monospace">I have tried creating a snapshot, mounting it and rsyncing the contents to the new FS. That fails with</font></div><div class="gmail_default"><font face="monospace, monospace"><br></font></div><div class="gmail_default"><font face="monospace, monospace"><span style="color:rgb(0,0,0)"><br></span></font></div><div class="gmail_default"><font face="monospace, monospace"><span style="color:rgb(0,0,0)">mount.lustre: mount localData/t1-MDT0001 at /localData/t1-MDT0001 failed: Invalid argument</span></font></div>
<p style="margin:0px;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="monospace, monospace">This may have multiple causes.</font></span></p>
<p style="margin:0px;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="monospace, monospace">Are the mount options correct?</font></span></p>
<p style="margin:0px;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="monospace, monospace">Check the syslog for more info.</font></span></p><div><span style="font-variant-ligatures:no-common-ligatures"><font face="monospace, monospace"><br></font></span></div><div><div class="gmail_default"><font face="monospace, monospace"></font></div><div class="gmail_default"><font face="monospace, monospace">and from dmesg</font></div><div class="gmail_default"><font face="monospace, monospace"><br></font></div><div class="gmail_default"><font face="monospace, monospace"><br></font></div><div class="gmail_default"><font face="monospace, monospace"><span style="color:rgb(0,0,0)">[6982824.344872] Lustre: 401:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1500606249/real 1500606249]</span><span style="color:rgb(0,0,0)"> </span><span style="color:rgb(0,0,0)">req@ffff880d70b56300 x1566221596712816/t0(0) o38->t1-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 1 dl 1500606260 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1</span></font></div>
<p style="margin:0px;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="monospace, monospace">[6982829.865323] LustreError: 18881:0:(obd_config.c:574:class_setup()) setup t1-MDT0000 failed (-22)</font></span></p>
<p style="margin:0px;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="monospace, monospace">[6982829.868476] LustreError: 18881:0:(obd_config.c:1710:class_config_llog_handler()) MGC172.16.250.40@tcp: cfg command failed: rc = -22</font></span></p>
<p style="margin:0px;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="monospace, monospace">[6982829.874691] Lustre: cmd=cf003 0:t1-MDT0000 1:t1-MDT0000_UUID 2:0 3:t1-MDT0000-mdtlov 4:f</font></span></p>
<p style="margin:0px;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="monospace, monospace">[6982829.881030] LustreError: 15b-f: MGC172.16.250.40@tcp: The configuration from log 't1-MDT0000'failed from the MGS (-22). Make sure this client and the MGS are running compatible versions of Lustre.</font></span></p>
<p style="margin:0px;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="monospace, monospace">[6982829.887555] LustreError: 18811:0:(obd_mount_server.c:1352:server_start_targets()) failed to start server t1-MDT0000: -22</font></span></p>
<p style="margin:0px;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="monospace, monospace">[6982829.890670] LustreError: 18811:0:(obd_mount_server.c:1844:server_fill_super()) Unable to start targets: -22</font></span></p>
<p style="margin:0px;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="monospace, monospace">[6982829.893651] LustreError: 18811:0:(obd_config.c:621:class_cleanup()) Device 443 not setup</font></span></p>
<p style="margin:0px;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="monospace, monospace">[6982830.300528] Lustre: server umount t1-MDT0000 complete</font></span></p>
<p style="margin:0px;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="monospace, monospace">[6982830.303158] LustreError: 18811:0:(obd_mount.c:1496:lustre_fill_super()) Unable to mount (-22)</font></span></p><div><span style="font-variant-ligatures:no-common-ligatures"><font face="monospace, monospace"><br></font></span></div><div><span style="font-variant-ligatures:no-common-ligatures"><font face="monospace, monospace"><br></font></span></div><div class="gmail_default"><font face="monospace, monospace">While doing the rsync -aXv I get weird error messages</font></div><div class="gmail_default"><font face="monospace, monospace"><br></font></div><div class="gmail_default"><font face="monospace, monospace"><br></font></div><div class="gmail_default"><font face="monospace, monospace"><span style="color:rgb(0,0,0)">file has vanished: "/mnt/mdt.s1/oi.7/0x200000007:0x1:0x0/mpartition"</span></font></div><div><span style="color:rgb(0,0,0)"><font face="monospace, monospace"><br></font></span></div><div><span style="color:rgb(0,0,0)"><font face="monospace, monospace"><br></font></span></div><div class="gmail_default"><font face="monospace, monospace">which is on the ro snapshot of the original FS. It gets even weirder...</font></div><div class="gmail_default"><font face="monospace, monospace"><br></font></div><div class="gmail_default"><font face="monospace, monospace"><br></font></div><p style="margin:0px;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="monospace, monospace">170721112705 pud40:mnt# ls -l /mnt/mdt.s1/oi.7/0x200000007:0x1:0x0/mpartition</font></span></p>
<p style="margin:0px;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="monospace, monospace">ls: cannot read symbolic link /mnt/mdt.s1/oi.7/0x200000007:0x1:0x0/mpartition: No such file or directory</font></span></p>
<p style="margin:0px;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="monospace, monospace">lrwxrwxrwx 1 root root 6 Mar 9 07:50 /mnt/mdt.s1/oi.7/0x200000007:0x1:0x0/mpartition</font></span></p><div class="gmail_default"><font face="monospace, monospace"><br></font></div><p style="margin:0px;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="monospace, monospace"># strace -f -e readlink -- ls -l /mnt/mdt.s1/oi.7/0x200000007:0x1:0x0/mpartition</font></span></p>
<p style="margin:0px;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="monospace, monospace">readlink("/mnt/mdt.s1/oi.7/0x200000007:0x1:0x0/mpartition", 0xc8c320, 7) = -1 ENOENT (No such file or directory)</font></span></p>
<p style="margin:0px;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="monospace, monospace">ls: cannot read symbolic link /mnt/mdt.s1/oi.7/0x200000007:0x1:0x0/mpartition: No such file or directory</font></span></p>
<p style="margin:0px;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="monospace, monospace">lrwxrwxrwx 1 root root 6 Mar 9 07:50 /mnt/mdt.s1/oi.7/0x200000007:0x1:0x0/mpartition</font></span></p>
<p style="margin:0px;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="monospace, monospace">+++ exited with 2 +++</font></span></p>
<p style="margin:0px;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="monospace, monospace"><br></font></span></p><p style="margin:0px;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="monospace, monospace"><br></font></span></p><p style="margin:0px;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="monospace, monospace"># stat /mnt/mdt.s1/oi.7/0x200000007:0x1:0x0/mpartition</font></span></p>
<p style="margin:0px;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="monospace, monospace"> File: ‘/mnt/mdt.s1/oi.7/0x200000007:0x1:0x0/mpartition’stat: cannot read symbolic link ‘/mnt/mdt.s1/oi.7/0x200000007:0x1:0x0/mpartition’: No such file or directory</font></span></p>
<p style="margin:0px;line-height:normal;color:rgb(0,0,0);min-height:11px"><font face="monospace, monospace"><span style="font-variant-ligatures:no-common-ligatures"></span><br></font></p>
<p style="margin:0px;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="monospace, monospace"> Size: 6 Blocks: 2 IO Block: 512 symbolic link</font></span></p>
<p style="margin:0px;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="monospace, monospace">Device: 41h/65d Inode: 738 Links: 1</font></span></p>
<p style="margin:0px;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="monospace, monospace">Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root)</font></span></p>
<p style="margin:0px;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="monospace, monospace">Access: 2017-07-21 10:44:27.000000000 +0800</font></span></p>
<p style="margin:0px;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="monospace, monospace">Modify: 2017-03-09 07:50:43.000000000 +0800</font></span></p>
<p style="margin:0px;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="monospace, monospace">Change: 2017-07-21 10:44:27.000000000 +0800</font></span></p>
<p style="margin:0px;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="monospace, monospace"> Birth: -</font></span></p><div class="gmail_default"><font face="monospace, monospace"><br></font></div><div class="gmail_default"><font face="monospace, monospace"><br></font></div><div class="gmail_default"><font face="monospace, monospace">so the file is in the directory listing as a link but not on the file system??</font></div><div class="gmail_default"><font face="monospace, monospace"><br></font></div><div class="gmail_default"><font face="monospace, monospace">So, this leads me to think that lustre is using features of ZFS that linux can't access? or something??</font></div><div class="gmail_default"><font face="monospace, monospace"><br></font></div><div class="gmail_default"><font face="monospace, monospace">It also raised the issue... how would I migrate off ZFS if I wanted to go back to ldiskfs for the MDT?</font></div><div class="gmail_default"><font face="monospace, monospace"><br></font></div><div class="gmail_default"><font face="monospace, monospace">Any ideas appreciated</font></div><div class="gmail_default"><font face="monospace, monospace"><br></font></div><div class="gmail_default"><font face="monospace, monospace">Thanks.</font></div><div class="gmail_default"><font face="monospace, monospace"><br></font></div><font face="monospace, monospace"><br></font></div><font face="monospace, monospace">-- <br></font><div class="gmail_signature"><font face="monospace, monospace">Dr Stuart Midgley<br><a href="mailto:sdm900@sdm900.com" target="_blank">sdm900@sdm900.com</a></font></div>
</div>