<div dir="ltr"><div>Thanks Andreas,</div>I'm aware that index 51 actually translates to hex 33 (local-OST0033_UUID).<br>I don't believe that's the reason for the failed mount as it is only an index that I increase for every new OST and there are no duplicates.<div><br></div>lctl dk show tens of thousands of lines repeating the same error after attempting to mount the OST:<br><div><br></div><div>00100000:10000000:26.0:1625546374.322973:0:248211:0:(osd_scrub.c:2039:osd_ios_scan_one()) local-OST0033: fail to set LMA for init OI scrub: rc = -30</div>00100000:10000000:26.0:1625546374.322974:0:248211:0:(osd_scrub.c:2039:osd_ios_scan_one()) local-OST0033: fail to set LMA for init OI scrub: rc = -30<br><div>00100000:10000000:26.0:1625546374.322975:0:248211:0:(osd_scrub.c:2039:osd_ios_scan_one()) local-OST0033: fail to set LMA for init OI scrub: rc = -30</div><div><br></div><div>in /var/log/messages I see the following corresponding to dm21 which is the new OST:<br><div><br></div><div>Jul 6 07:38:37 oss03 kernel: LDISKFS-fs warning (device dm-21): ldiskfs_multi_mount_protect:322: MMP interval 42 higher than expected, please wait.</div>Jul 6 07:39:19 oss03 kernel: LDISKFS-fs (dm-21): file extents enabled, maximum tree depth=5<br>Jul 6 07:39:19 oss03 kernel: LDISKFS-fs warning (device dm-21): ldiskfs_clear_journal_err:4862: Filesystem error recorded from previous mount: IO failure<br>Jul 6 07:39:19 oss03 kernel: LDISKFS-fs warning (device dm-21): ldiskfs_clear_journal_err:4863: Marking fs in need of filesystem check.<br>Jul 6 07:39:19 oss03 kernel: LDISKFS-fs (dm-21): warning: mounting fs with errors, running e2fsck is recommended<br>Jul 6 07:39:22 oss03 kernel: LDISKFS-fs (dm-21): recovery complete<br>Jul
6 07:39:22 oss03 kernel: LDISKFS-fs (dm-21): mounted filesystem
with ordered data mode. Opts: user_xattr,errors=remount-ro,acl,no_mbcache,nodelalloc<br>Jul
6 07:39:22 oss03 kernel: LDISKFS-fs error (device dm-21):
htree_dirblock_to_tree:1278: inode #2: block 21233: comm mount.lustre:
bad entry in directory: rec_len is too small for name_len -
offset=4084(4084), inode=0, rec_len=12<br>, name_len=0<br>Jul 6 07:39:22 oss03 kernel: Aborting journal on device dm-21-8.<br>Jul 6 07:39:22 oss03 kernel: LDISKFS-fs (dm-21): Remounting filesystem read-only<br>Jul
6 07:39:24 oss03 kernel: LDISKFS-fs warning (device dm-21):
kmmpd:187: kmmpd being stopped since filesystem has been remounted as
readonly.<br>Jul 6 07:44:22 oss03 kernel: LDISKFS-fs (dm-21): error count since last fsck: 6<br>Jul
6 07:44:22 oss03 kernel: LDISKFS-fs (dm-21): initial error at
time 1625367384: htree_dirblock_to_tree:1278: inode 2: block 21233<br>Jul
6 07:44:22 oss03 kernel: LDISKFS-fs (dm-21): last error at time
1625546362: htree_dirblock_to_tree:1278: inode 2: block 21233<br><br>As I mentioned before mount never completes so the only way out of that is force reboot.<br><br>Thanks,<br>David</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Jul 6, 2021 at 8:07 AM Andreas Dilger <<a href="mailto:adilger@whamcloud.com" target="_blank">adilger@whamcloud.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<br>
<div><br>
<blockquote type="cite">
<div>On Jul 5, 2021, at 09:05, David Cohen <<a href="mailto:cdavid@physics.technion.ac.il" target="_blank">cdavid@physics.technion.ac.il</a>> wrote:</div>
<br>
<div>
<div dir="ltr">
<div>Hi,</div>
<div>I'm using Lustre 2.10.5 and lately tried to add a new OST. <br>
</div>
<div>The OST was formatted with the command below, which other than the index is the exact same one used for all the other OSTs in the system.<br>
</div>
<div><br>
</div>
<div>mkfs.lustre --reformat --mkfsoptions="-t ext4 -T huge" --ost --fsname=local --index=0051 --param ost.quota_type=ug --mountfsoptions='errors=remount-ro,extents,mballoc' --mgsnode=10.0.0.3@tcp --mgsnode=10.0.0.1@tc</div>
p --mgsnode=10.0.0.2@tcp --servicenode=10.0.0.3@tcp --servicenode=10.0.0.1@tcp --servicenode=10.0.0.2@tcp /dev/mapper/OST0051<span style="font-family:monospace"><br>
</span></div>
</div>
</blockquote>
<div><br>
</div>
Note that your "--index=0051" value is probably interpreted as an octal number "41", it should be "--index=0x0051" or "--index=0x51" (hex, to match the OST device name) or "--index=81" (decimal).</div>
<div><br>
<blockquote type="cite">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div dir="ltr"><br>
</div>
When trying to mount the with:<br>
mount.lustre /dev/mapper/OST0051 /Lustre/OST0051</div>
<div dir="ltr"><br>
</div>
<div>The system stays on 100% CPU (one core) forever and the mount never completes, not even after a week.</div>
<br>
I tried tunefs.lustre --writeconf --erase-params on the MDS and all the other targets, but the behaviour remains the same.</div>
</div>
</div>
</blockquote>
</div>
<br>
<div>
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div>Cheers, Andreas</div>
<div>--</div>
<div>Andreas Dilger</div>
<div>Lustre Principal Architect</div>
<div>Whamcloud</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
</div>
<br>
<br>
</div>
<br>
</div>
</blockquote></div>