<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Times New Roman \(Body CS\)";
panose-1:2 11 6 4 2 2 2 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word;-webkit-nbsp-mode: space;line-break:after-white-space">
<div class="WordSection1">
<p class="MsoNormal">Thanks Andreas \u2013 I appreciate the info. <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I am dd\u2019ing the MDT block device (both of them \u2013 more details below) to separate storage now.
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I\u2019ve written this up on the ZFS mailing list.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">https://zfsonlinux.topicbox.com/groups/zfs-discuss/Tcb8a3ef663db0031/need-help-with-data-recovery-if-possible<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Actually, in the process of doing that, I think I see what is going on. More details in the ZFS post but it looks like the block device names for the ZFS volumes got swapped on the crash and reboot. So /dev/zd0 is a clone of the February
snapshot and /dev/zd16 is actually our primary (current) MDT. If I mount zd16 and poke around, I see lots of files newer than February.
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">[root@hpfs-fsl-mds1 ~]# mount -t ldiskfs -o ro /dev/zd16 /mnt/mdt_backup/
</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">[root@hpfs-fsl-mds1 ~]# cd /mnt/mdt_backup/<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">[root@hpfs-fsl-mds1 mdt_backup]# ls -l PENDING/<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">total 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">-rw------- 1 ecdavis2 damocles 0 Jun 17 11:03 0x200021094:0x3b26:0x0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">-rw------- 1 rharpold rharpold 0 Jun 14 12:27 0x200021096:0x1337:0x0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">[root@hpfs-fsl-mds1 mdt_backup]#
<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">So it looks like we have a shot at recovery. I hope to get more guidance on the ZFS list on how to properly swap zd0 and zd16 back. I\u2019m also tarring up the contents of the read only mount of zd16. In all:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">dd if=/dev/zd0 of=/internal/zd0.dd.2022.06.22 bs=1M<o:p></o:p></p>
<p class="MsoNormal">dd if=/dev/zd16 of=/internal/zd16.dd.2022.06.22 bs=1M <o:p></o:p></p>
<p class="MsoNormal">cd /mnt/mdt_backup ; tar cf /internal/zd16.tar --xattrs --xattrs-include="trusted.*" --sparse .
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Please let me know if there is something else we should consider doing before attempting recovery.
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Actually, I\u2019m 100% certain this is our current MDT. I see files and directories in
<span style="font-size:10.0pt;font-family:"Courier New"">/mnt/mdt_backup/ROOT that were just created in the last couple weeks. Happy day.
</span><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">One other question. We are seeing a ton of these in the MDS logs since the crash.
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Jun 22 21:53:16 hpfs-fsl-mds1 kernel: LustreError: 14346:0:(qmt_handler.c:699:qmt_dqacq0()) $$$ Release too much! uuid:scratch-MDT0000-lwp-OST000f_UUID release: 67108864 granted:0, total:0 qmt:scratch-QMT0000 pool:dt-0x0 id:5697 enforced:0
hard:0 soft:0 granted:0 time:0 qunit: 0 edquot:0 may_rel:0 revoke:0 default:yes<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I assume this is not unexpected with an MDT that got reverted?
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">Andreas Dilger <adilger@whamcloud.com><br>
<b>Date: </b>Wednesday, June 22, 2022 at 4:48 PM<br>
<b>To: </b>"Vicker, Darby J. (JSC-EG111)[Jacobs Technology, Inc.]" <darby.vicker-1@nasa.gov><br>
<b>Cc: </b>"lustre-discuss@lists.lustre.org" <lustre-discuss@lists.lustre.org><br>
<b>Subject: </b>[EXTERNAL] Re: [lustre-discuss] Help with recovery of data<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal">First thing, if you haven't already done so, would be to make a separate "dd" backup of the ldiskfs MDT(s) to some external storage before you do anything else. That will give you a fallback in case whatever changes you make don't work
out well. <o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I would also suggest to contact the ZFS mailing list to ask if they can help restore the "new version" of the MDT at the ZFS level. You may also want to consider a separate ZFS-level backup because the core of the problem appears to be
ZFS related. Unfortunately, the opportunity to recover a newer version of the ldiskfs MDT at the ZFS level declines the more changes are made to the ZFS pool.
<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I don't think LFSCK will repair the missing files on the MDT, since the OSTs don't have enough information to regenerate the namespace. At most LFSCK will create stub files on the MDT under .lustre/lost+found that connect the objects for
the new files created after your MDT snapshot, but they won't have proper filenames. At most they will have UID/GID/timestamps to identify the owners/age, and the users would need to identify the files by content.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</body>
</html>