<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div><span style="font-family: Consolas, Courier, monospace; font-size: 10pt;">Hello,</span><br>
</div>
<div><br>
</div>
<div><span style="font-family: Consolas, Courier, monospace; font-size: 10pt;">we are running Lustre 2.12.5 on ZFS 0.8.3, where the MDT is backed by a ZFS pool consisting of 8 mirror vdevs. Recently we had two drives fail in the same mirror vdev, from which
we were able to mostly recover (details on the zfs-discuss list [1]). We have brought the Lustre filesystem back up and mounted it on a single client read-only. Directly after mount, checking the filesystem space with "df" shows the correct/expected size
of the fs. But running it again shows the total/used/available space inflated by a factor of 32 (almost exactly):</span><br>
</div>
<div class="elementToProof" style="font-family: Consolas, Courier, monospace; font-size: 10pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Consolas, Courier, monospace; font-size: 10pt; color: rgb(0, 0, 0);">
<br>
</div>
<div><span style="font-family: Consolas, Courier, monospace; font-size: 10pt;"># mount /lustreTEST; sleep 1; df /lustreTEST</span><br>
</div>
<div><span style="font-family: Consolas, Courier, monospace; font-size: 10pt;">Filesystem 1K-blocks Used Available Use% Mounted on</span><br>
</div>
<div><span style="font-family: Consolas, Courier, monospace; font-size: 10pt;">10.128.104.102@o2ib:10.128.104.101@o2ib:/lustrefs 1131965562880 527126408192 604839130112 47% /lustreTEST</span><br>
</div>
<div><br>
</div>
<div><span style="font-family: Consolas, Courier, monospace; font-size: 10pt;"># df /lustreTEST</span><br>
</div>
<div><span style="font-family: Consolas, Courier, monospace; font-size: 10pt;">Filesystem 1K-blocks Used Available Use% Mounted on</span><br>
</div>
<div><span style="font-family: Consolas, Courier, monospace; font-size: 10pt;">10.128.104.102@o2ib:10.128.104.101@o2ib:/lustrefs 36222898339840 16868045127680 19354852425728 47% /lustreTEST</span><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><span style="font-family: Consolas, Courier, monospace; font-size: 10pt;">For comparison, "lfs df" shows the expected size (1.1PB):</span><br>
</div>
<div><br>
</div>
<div><span style="font-family: Consolas, Courier, monospace; font-size: 10pt;"># lfs df</span><br>
</div>
<div><span style="font-family: Consolas, Courier, monospace; font-size: 10pt;">UUID 1K-blocks Used Available Use% Mounted on</span><br>
</div>
<div><span style="font-family: Consolas, Courier, monospace; font-size: 10pt;">lustrefs-MDT0000_UUID 2236427008 1183398912 1053026048 53% /lustreTEST[MDT:0]</span><br>
</div>
<div><span style="font-family: Consolas, Courier, monospace; font-size: 10pt;">lustrefs-OST0000_UUID 94330528768 42854200320 51476326400 46% /lustreTEST[OST:0]</span><br>
</div>
<div><span style="font-family: Consolas, Courier, monospace; font-size: 10pt;">lustrefs-OST0001_UUID 94329859072 46121248768 48208608256 49% /lustreTEST[OST:1]</span><br>
</div>
<div><span style="font-family: Consolas, Courier, monospace; font-size: 10pt;">lustrefs-OST0002_UUID 94330471424 44617906176 49712563200 48% /lustreTEST[OST:2]</span><br>
</div>
<div><span style="font-family: Consolas, Courier, monospace; font-size: 10pt;">lustrefs-OST0003_UUID 94330831872 42602903552 51727926272 46% /lustreTEST[OST:3]</span><br>
</div>
<div><span style="font-family: Consolas, Courier, monospace; font-size: 10pt;">lustrefs-OST0004_UUID 94330233856 42467627008 51862604800 46% /lustreTEST[OST:4]</span><br>
</div>
<div><span style="font-family: Consolas, Courier, monospace; font-size: 10pt;">lustrefs-OST0005_UUID 94330768384 47377309696 46953456640 51% /lustreTEST[OST:5]</span><br>
</div>
<div><span style="font-family: Consolas, Courier, monospace; font-size: 10pt;">lustrefs-OST0006_UUID 94330462208 42833333248 51497126912 46% /lustreTEST[OST:6]</span><br>
</div>
<div><span style="font-family: Consolas, Courier, monospace; font-size: 10pt;">lustrefs-OST0007_UUID 94330291200 42838326272 51491962880 46% /lustreTEST[OST:7]</span><br>
</div>
<div><span style="font-family: Consolas, Courier, monospace; font-size: 10pt;">lustrefs-OST0008_UUID 94330214400 45438008320 48892204032 49% /lustreTEST[OST:8]</span><br>
</div>
<div><span style="font-family: Consolas, Courier, monospace; font-size: 10pt;">lustrefs-OST0009_UUID 94330670080 44649268224 49681399808 48% /lustreTEST[OST:9]</span><br>
</div>
<div><span style="font-family: Consolas, Courier, monospace; font-size: 10pt;">lustrefs-OST000a_UUID 94330542080 43913293824 50417246208 47% /lustreTEST[OST:10]</span><br>
</div>
<div><span style="font-family: Consolas, Courier, monospace; font-size: 10pt;">lustrefs-OST000b_UUID 94330704896 41412985856 52917716992 44% /lustreTEST[OST:11]</span><br>
</div>
<div><br>
</div>
<div><span style="font-family: Consolas, Courier, monospace; font-size: 10pt;">filesystem_summary: 1131965578240 527126411264 604839142400 47% /lustreTEST</span><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><span style="font-family: Consolas, Courier, monospace; font-size: 10pt;">We are wondering why the df output is so inflated, and how worried we should be about it. For now we've refrained from putting the fs back into production.</span><br>
</div>
<div><br>
</div>
<div class="elementToProof"><span style="font-family: Consolas, Courier, monospace; font-size: 10pt;">Anyway, we are running LFSCK now. Is there a way to estimate how long it will take? I can see progress in the "checked" and "current_position" counters reported
by "lctl get_param osd-zfs.lustrefs-MDT*.oi_scrub / mdd.lustrefs-MDT*.lfsck_layout / mdd.lustrefs-MDT*.lfsck_namespace" but how could we estimate what the 100% values of those counters are? Is it related to how full the MDT is? We seem to have 264M inodes
in use.</span><br>
</div>
<div><br>
</div>
<div><span style="font-family: Consolas, Courier, monospace; font-size: 10pt;">Many thanks,</span><br>
</div>
<div><span style="font-family: Consolas, Courier, monospace; font-size: 10pt;">Frank</span><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><span style="font-family: Consolas, Courier, monospace; font-size: 10pt;">[1] https://zfsonlinux.topicbox.com/groups/zfs-discuss/T28e59deb8ff2c26d/two-failed-drives-in-mirror-vdev</span><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
</body>
</html>