<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:697779306;
mso-list-type:hybrid;
mso-list-template-ids:816854746 134807567 134807555 134807557 134807553 134807555 134807557 134807553 134807555 134807557;}
@list l0:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l1
{mso-list-id:1364163699;
mso-list-type:hybrid;
mso-list-template-ids:950682136 134807567 134807577 134807579 134807567 134807577 134807579 134807567 134807577 134807579;}
@list l1:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l1:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l1:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l1:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l1:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l1:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l1:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l1:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l1:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal">Hi All,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">We are seeing an anomaly across all of our RaidInc Lustre filesystems<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><b>Problem description:<o:p></o:p></b></p>
<p class="MsoNormal">File Size < on disk size - currently unexplained, size on disk is 2-3 x file size.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><b>Observations:<o:p></o:p></b></p>
<ol style="margin-top:0cm" start="1" type="1">
<li class="MsoNormal" style="mso-list:l1 level1 lfo1">A potential ZFS filesystem corruption across RaidInc Storage in London?<o:p></o:p></li><li class="MsoNormal" style="mso-list:l1 level1 lfo1">zdb check for leaks, it walks the entire block tree constructing the space maps in memory and then compares them to the ones stored on disk. If they differ it reports the leak.
<o:p></o:p>
<ol style="margin-top:0cm" start="1" type="a">
<li class="MsoNormal" style="mso-list:l1 level2 lfo1">Presuming from the below investigation that the “space leaks” mean the pool is corrupted somehow. zdb (ZFS debug) has detected tons of corruptions.
<o:p></o:p></li></ol>
</li><li class="MsoNormal" style="mso-list:l1 level1 lfo1">zdb did not report space leaks on ZFS Houston SI’s.
<o:p></o:p></li><li class="MsoNormal" style="mso-list:l1 level1 lfo1">Does zdb leaked space means trouble with the pool and could explain the file size < disk size discrepancy?
<o:p></o:p></li><li class="MsoNormal" style="mso-list:l1 level1 lfo1">Is it possible that errors got injected due to failover or hardware errors?<o:p></o:p></li><li class="MsoNormal" style="mso-list:l1 level1 lfo1">It seems to be at least inconsistent which is supposed to never happen with ZFS. Is this indicative of a larger problem? Numerous lockups, etc.?<o:p></o:p></li></ol>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><b>Investigation:<o:p></o:p></b></p>
<p class="MsoNormal">For the troubleshooting, the following file located in WEY, was selected. There are no snapshots/reservations/quotas involved here.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:Consolas">lconnect03]</users/jerome.cousin>$ du -h --apparent-size /lus/lsi022/4388cog/p005j02_2010_SRME_1238A018_2copy/*<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:Consolas">33K /lus/lsi022/4388cog/p005j02_2010_SRME_1238A018_2copy/aux_data<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:Consolas">19K /lus/lsi022/4388cog/p005j02_2010_SRME_1238A018_2copy/descriptor.yaml<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:Consolas">104G /lus/lsi022/4388cog/p005j02_2010_SRME_1238A018_2copy/trace_data.bin<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:Consolas">14G /lus/lsi022/4388cog/p005j02_2010_SRME_1238A018_2copy/trace_header.bin<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:Consolas">[lconnect03]</users/jerome.cousin>$ du -h /lus/lsi022/4388cog/p005j02_2010_SRME_1238A018_2copy/*<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:Consolas">33K /lus/lsi022/4388cog/p005j02_2010_SRME_1238A018_2copy/aux_data<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:Consolas">56K /lus/lsi022/4388cog/p005j02_2010_SRME_1238A018_2copy/descriptor.yaml<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:Consolas">237G /lus/lsi022/4388cog/p005j02_2010_SRME_1238A018_2copy/trace_data.bin<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:Consolas">31G /lus/lsi022/4388cog/p005j02_2010_SRME_1238A018_2copy/trace_header.bin<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<ol style="margin-top:0cm" start="1" type="1">
<li class="MsoNormal" style="mso-list:l0 level1 lfo2">Copy of the dataset onto the same storage.
<o:p></o:p>
<ul style="margin-top:0cm" type="circle">
<li class="MsoNormal" style="mso-list:l0 level2 lfo2">Disk size is different. <o:p>
</o:p></li><li class="MsoNormal" style="mso-list:l0 level2 lfo2">Checksum matches.<o:p></o:p></li></ul>
</li></ol>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:Consolas">[lconnect03]</users/jerome.cousin>$ cp -rp /lus/lsi022/4388cog/p005j02_2010_SRME_1238A018_2copy /lus/lsi022/4388cog/p005j02_2010_SRME_1238A018_JC<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:Consolas">[lconnect03]</users/jerome.cousin>$ md5sum /lus/lsi022/4388cog/p005j02_2010_SRME_1238A018_2copy/*<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:Consolas">md5sum: /lus/lsi022/4388cog/p005j02_2010_SRME_1238A018_2copy/aux_data: Is a directory<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:Consolas">f861b60d2b1b844e5ae252345aa20497 /lus/lsi022/4388cog/p005j02_2010_SRME_1238A018_2copy/descriptor.yaml<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:Consolas">e8ac57c241e52b38b60907e4e767b451 /lus/lsi022/4388cog/p005j02_2010_SRME_1238A018_2copy/trace_data.bin<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:Consolas">0826bc74e525697d769248aabcb195cd /lus/lsi022/4388cog/p005j02_2010_SRME_1238A018_2copy/trace_header.bin<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:Consolas">[lconnect03]</users/jerome.cousin>$ md5sum /lus/lsi022/4388cog/p005j02_2010_SRME_1238A018_JC/*<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:Consolas">md5sum: /lus/lsi022/4388cog/p005j02_2010_SRME_1238A018_JC/aux_data: Is a directory<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:Consolas">f861b60d2b1b844e5ae252345aa20497 /lus/lsi022/4388cog/p005j02_2010_SRME_1238A018_JC/descriptor.yaml<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:Consolas">e8ac57c241e52b38b60907e4e767b451 /lus/lsi022/4388cog/p005j02_2010_SRME_1238A018_JC/trace_data.bin<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:Consolas">0826bc74e525697d769248aabcb195cd /lus/lsi022/4388cog/p005j02_2010_SRME_1238A018_JC/trace_header.bin<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:Consolas">[lconnect03]</users/jerome.cousin>$ du -h /lus/lsi022/4388cog/p005j02_2010_SRME_1238A018_JC/*<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:Consolas">33K /lus/lsi022/4388cog/p005j02_2010_SRME_1238A018_JC/aux_data<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:Consolas">56K /lus/lsi022/4388cog/p005j02_2010_SRME_1238A018_JC/descriptor.yaml<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:Consolas">99G /lus/lsi022/4388cog/p005j02_2010_SRME_1238A018_JC/trace_data.bin<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:Consolas">13G /lus/lsi022/4388cog/p005j02_2010_SRME_1238A018_JC/trace_header.bin<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:Consolas">[lconnect03]</users/jerome.cousin>$ du -h --apparent-size /lus/lsi022/4388cog/p005j02_2010_SRME_1238A018_JC/*<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:Consolas">33K /lus/lsi022/4388cog/p005j02_2010_SRME_1238A018_JC/aux_data<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:Consolas">19K /lus/lsi022/4388cog/p005j02_2010_SRME_1238A018_JC/descriptor.yaml<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:Consolas">104G /lus/lsi022/4388cog/p005j02_2010_SRME_1238A018_JC/trace_data.bin<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:Consolas">14G /lus/lsi022/4388cog/p005j02_2010_SRME_1238A018_JC/trace_header.bin<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<ol style="margin-top:0cm" start="2" type="1">
<li class="MsoNormal" style="mso-list:l0 level1 lfo2">Printing the OST name hosting the given file.<o:p></o:p></li></ol>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:Consolas">[lconnect01]</users/jerome.cousin>$ ./lustre-find-ost-for-file /lus/lsi022/4388cog/p005j02_2010_SRME_1238A018_2copy/trace_data.bin<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:Consolas">15<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:Consolas">/lus/lsi022/4388cog/p005j02_2010_SRME_1238A018_2copy/trace_data.bin: ['lsi022-OST000f'] (lsi022-oss6.lon.compute.pgs.com)<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<ol style="margin-top:0cm" start="3" type="1">
<li class="MsoNormal" style="mso-list:l0 level1 lfo2">Run zdb to check for leaks<o:p></o:p></li></ol>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:Consolas">[root@lsi022-oss6 ~]# zfs list<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:Consolas">NAME USED AVAIL REFER MOUNTPOINT<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:Consolas">lsi022-OST17 48.3T 18.5T 219K none<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:Consolas">lsi022-OST17/lsi022-OST0005 48.3T 18.5T 48.3T none<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:Consolas">lsi022-OST19 49.5T 17.3T 219K none<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:Consolas">lsi022-OST19/lsi022-OST0009 49.5T 17.3T 49.5T none<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:Consolas">lsi022-OST21 47.3T 19.5T 219K none<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:Consolas">lsi022-OST21/lsi022-OST000f 47.3T 19.5T 47.3T none<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:Consolas">lsi022-OST23 51.1T 15.7T 219K none<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:Consolas">lsi022-OST23/lsi022-OST0013 51.1T 15.7T 51.1T none<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-left:72.0pt"><span style="font-family:Consolas">[root@lsi022-oss6 ~]# zdb -b lsi022-OST21<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:72.0pt"><span style="font-family:Consolas">Traversing all blocks to verify nothing leaked ...<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:72.0pt"><span style="font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:72.0pt"><span style="font-family:Consolas">loading space map for vdev 0 of 1, metaslab 180 of 181 ...<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:72.0pt"><span style="font-family:Consolas">62.0T completed (12801MB/s) estimated time remaining: 0hr 00min 07sec
<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:72.0pt"><span style="font-family:Consolas">leaked space: vdev 0, offset 0x1d80003de000, size 1081344<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:72.0pt"><span style="font-family:Consolas">[…]<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:72.0pt">See attachment.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Please would someone be able to advise.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks<o:p></o:p></p>
<p class="MsoNormal">Nick<o:p></o:p></p>
</div>
</body>
</html>