<div dir="auto">Can you say more about these networking issues?<div dir="auto">Good to make a note of them in case anyone sees similar in the future. </div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, 12 May 2023, 20:40 Jane Liu via lustre-discuss, <<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Jeff,<br>
<br>
Thanks for your response. We discovered later that the network issues <br>
originating from the iDRAC IP were causing the SAS driver to hang or <br>
experience timeouts when trying to access the drives. This resulted in <br>
the drives being kicked out.<br>
<br>
Once we resolved this issue, both the mkfs and mount operations started <br>
working fine.<br>
<br>
Thanks,<br>
Jane<br>
<br>
<br>
<br>
On 2023-05-10 12:43, Jeff Johnson wrote:<br>
> Jane,<br>
> <br>
> You're having hardware errors, the codes in those mpt3sas errors<br>
> define as "PL_LOGINFO_SUB_CODE_OPEN_FAILURE_ORR_TIMEOUT", or in other<br>
> words your SAS HBA cannot open a command dialogue with your disk. I'd<br>
> suspect backplane or cabling issues as an internal disk failure will<br>
> be reported by the target disk with its own error code. In this case<br>
> your HBA can't even talk to it properly.<br>
> <br>
> Is sdah the partner mpath device to sdef? Or is sdah a second failing<br>
> disk interface?<br>
> <br>
> Looking at this, I don't think your hardware is deploy-ready.<br>
> <br>
> --Jeff<br>
> <br>
> On Wed, May 10, 2023 at 9:29\u202fAM Jane Liu via lustre-discuss<br>
> <<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank" rel="noreferrer">lustre-discuss@lists.lustre.org</a>> wrote:<br>
> <br>
>> Hi,<br>
>> <br>
>> We recently attempted to add several new OSS servers ( RHEL 8.7 and<br>
>> Lustre 2.15.2). While creating new OSTs, I noticed that mdstat<br>
>> reported<br>
>> some disk failures after the mkfs, even though the disks were<br>
>> functional<br>
>> before the mkfs command. Our hardware admins managed to resolve the<br>
>> mdstat issue and restore the disks to normal operation. However,<br>
>> when I<br>
>> ran the mount OST command (when network had a problem and mount<br>
>> command<br>
>> timed out), similar problems occurred, and several disks were kicked<br>
>> <br>
>> out. The relevant /var/log/messages are provided below.<br>
>> <br>
>> This problem was consistent across all our OSS servers. Any insights<br>
>> <br>
>> into the possible cause would be appreciated.<br>
>> <br>
>> Jane<br>
>> <br>
>> -----------------------------<br>
>> <br>
>> May 9 13:33:15 sphnxoss47 kernel: LDISKFS-fs (md0): mounted<br>
>> filesystem<br>
>> with ordered data mode. Opts: errors=remount-ro<br>
>> May 9 13:33:15 sphnxoss47 systemd[1]: tmp-mntmirJ5z.mount:<br>
>> Succeeded.<br>
>> May 9 13:33:16 sphnxoss47 kernel: LNet: HW NUMA nodes: 2, HW CPU<br>
>> cores:<br>
>> 72, npartitions: 2<br>
>> May 9 13:33:16 sphnxoss47 kernel: alg: No test for adler32<br>
>> (adler32-zlib)<br>
>> May 9 13:33:16 sphnxoss47 kernel: Key type ._llcrypt registered<br>
>> May 9 13:33:16 sphnxoss47 kernel: Key type .llcrypt registered<br>
>> May 9 13:33:16 sphnxoss47 kernel: Lustre: Lustre: Build Version:<br>
>> 2.15.2<br>
>> May 9 13:33:16 sphnxoss47 kernel: LNet: Added LNI 169.254.1.2@tcp<br>
>> [8/256/0/180]<br>
>> May 9 13:33:16 sphnxoss47 kernel: LNet: Accept secure, port 988<br>
>> May 9 13:33:17 sphnxoss47 kernel: LDISKFS-fs (md0): mounted<br>
>> filesystem<br>
>> with ordered data mode. Opts:<br>
>> errors=remount-ro,no_mbcache,nodelalloc<br>
>> May 9 13:33:17 sphnxoss47 kernel: Lustre: sphnx01-OST0244-osd:<br>
>> enabled<br>
>> 'large_dir' feature on device /dev/md0<br>
>> May 9 13:33:25 sphnxoss47 systemd-logind[8609]: New session 7 of<br>
>> user<br>
>> root.<br>
>> May 9 13:33:25 sphnxoss47 systemd[1]: Started Session 7 of user<br>
>> root.<br>
>> May 9 13:34:36 sphnxoss47 kernel: LustreError: 15f-b:<br>
>> sphnx01-OST0244:<br>
>> cannot register this server with the MGS: rc = -110. Is the MGS<br>
>> running?<br>
>> May 9 13:34:36 sphnxoss47 kernel: LustreError:<br>
>> 45314:0:(obd_mount_server.c:2027:server_fill_super()) Unable to<br>
>> start<br>
>> targets: -110<br>
>> May 9 13:34:36 sphnxoss47 kernel: LustreError:<br>
>> 45314:0:(obd_mount_server.c:1644:server_put_super()) no obd<br>
>> sphnx01-OST0244<br>
>> May 9 13:34:36 sphnxoss47 kernel: LustreError:<br>
>> 45314:0:(obd_mount_server.c:131:server_deregister_mount())<br>
>> sphnx01-OST0244 not registered<br>
>> May 9 13:34:39 sphnxoss47 kernel: Lustre: server umount<br>
>> sphnx01-OST0244<br>
>> complete<br>
>> May 9 13:34:39 sphnxoss47 kernel: LustreError:<br>
>> 45314:0:(super25.c:176:lustre_fill_super()) llite: Unable to mount<br>
>> <unknown>: rc = -110<br>
>> May 9 13:34:40 sphnxoss47 kernel: LDISKFS-fs (md1): mounted<br>
>> filesystem<br>
>> with ordered data mode. Opts: errors=remount-ro<br>
>> May 9 13:34:40 sphnxoss47 systemd[1]: tmp-mntXT85fz.mount:<br>
>> Succeeded.<br>
>> May 9 13:34:41 sphnxoss47 kernel: LDISKFS-fs (md1): mounted<br>
>> filesystem<br>
>> with ordered data mode. Opts:<br>
>> errors=remount-ro,no_mbcache,nodelalloc<br>
>> May 9 13:34:41 sphnxoss47 kernel: Lustre: sphnx01-OST0245-osd:<br>
>> enabled<br>
>> 'large_dir' feature on device /dev/md1<br>
>> May 9 13:36:00 sphnxoss47 kernel: LustreError: 15f-b:<br>
>> sphnx01-OST0245:<br>
>> cannot register this server with the MGS: rc = -110. Is the MGS<br>
>> running?<br>
>> May 9 13:36:00 sphnxoss47 kernel: LustreError:<br>
>> 46127:0:(obd_mount_server.c:2027:server_fill_super()) Unable to<br>
>> start<br>
>> targets: -110<br>
>> May 9 13:36:00 sphnxoss47 kernel: LustreError:<br>
>> 46127:0:(obd_mount_server.c:1644:server_put_super()) no obd<br>
>> sphnx01-OST0245<br>
>> May 9 13:36:00 sphnxoss47 kernel: LustreError:<br>
>> 46127:0:(obd_mount_server.c:131:server_deregister_mount())<br>
>> sphnx01-OST0245 not registered<br>
>> May 9 13:36:08 sphnxoss47 kernel: Lustre: server umount<br>
>> sphnx01-OST0245<br>
>> complete<br>
>> May 9 13:36:08 sphnxoss47 kernel: LustreError:<br>
>> 46127:0:(super25.c:176:lustre_fill_super()) llite: Unable to mount<br>
>> <unknown>: rc = -110<br>
>> May 9 13:36:08 sphnxoss47 kernel: LDISKFS-fs (md2): mounted<br>
>> filesystem<br>
>> with ordered data mode. Opts: errors=remount-ro<br>
>> May 9 13:36:08 sphnxoss47 systemd[1]: tmp-mnt17IOaq.mount:<br>
>> Succeeded.<br>
>> May 9 13:36:09 sphnxoss47 kernel: LDISKFS-fs (md2): mounted<br>
>> filesystem<br>
>> with ordered data mode. Opts:<br>
>> errors=remount-ro,no_mbcache,nodelalloc<br>
>> Show less<br>
>> 11:03 AM<br>
>> <br>
>> -----------------------------<br>
>> <br>
>> it just repeats for all of the md raids, then the errors start and<br>
>> the<br>
>> drive fails and is disabled:<br>
>> <br>
>> May 9 13:44:31 sphnxoss47 kernel: LustreError:<br>
>> 48069:0:(super25.c:176:lustre_fill_super()) llite: Unable to mount<br>
>> <unknown>: rc = -110<br>
>> May 9 13:44:33 sphnxoss47 kernel: mpt3sas_cm1:<br>
>> log_info(0x3112011a):<br>
>> originator(PL), code(0x12), sub_code(0x011a)<br>
>> May 9 13:44:33 sphnxoss47 kernel: mpt3sas_cm1:<br>
>> log_info(0x3112011a):<br>
>> originator(PL), code(0x12), sub_code(0x011a)<br>
>> May 9 13:44:33 sphnxoss47 kernel: mpt3sas_cm1:<br>
>> log_info(0x3112011a):<br>
>> originator(PL), code(0x12), sub_code(0x011a)<br>
>> May 9 13:44:33 sphnxoss47 kernel: mpt3sas_cm1:<br>
>> log_info(0x3112011a):<br>
>> originator(PL), code(0x12), sub_code(0x011a)<br>
>> May 9 13:44:33 sphnxoss47 kernel: mpt3sas_cm1:<br>
>> log_info(0x3112011a):<br>
>> originator(PL), code(0x12), sub_code(0x011a)<br>
>> May 9 13:44:33 sphnxoss47 kernel: mpt3sas_cm1:<br>
>> log_info(0x3112011a):<br>
>> originator(PL), code(0x12), sub_code(0x011a)<br>
>> May 9 13:44:33 sphnxoss47 kernel: mpt3sas_cm1:<br>
>> log_info(0x3112011a):<br>
>> originator(PL), code(0x12), sub_code(0x011a)<br>
>> ....<br>
>> ....<br>
>> May 9 13:44:33 sphnxoss47 kernel: sd 16:0:31:0: [sdef] tag#1102<br>
>> FAILED<br>
>> Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK cmd_age=1s<br>
>> May 9 13:44:33 sphnxoss47 kernel: sd 16:0:31:0: [sdef] tag#1102<br>
>> CDB:<br>
>> Read(10) 28 00 00 00 87 79 00 00 01 00<br>
>> May 9 13:44:33 sphnxoss47 kernel: blk_update_request: I/O error,<br>
>> dev<br>
>> sdef, sector 277448 op 0x0:(READ) flags 0x84700 phys_seg 1 prio<br>
>> class 0<br>
>> May 9 13:44:33 sphnxoss47 kernel: sd 16:0:31:0: [sdef] tag#6800<br>
>> FAILED<br>
>> Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK cmd_age=1s<br>
>> May 9 13:44:33 sphnxoss47 kernel: sd 16:0:31:0: [sdef] tag#6800<br>
>> CDB:<br>
>> Read(10) 28 00 00 00 87 dd 00 00 01 00<br>
>> May 9 13:44:33 sphnxoss47 kernel: blk_update_request: I/O error,<br>
>> dev<br>
>> sdef, sector 278248 op 0x0:(READ) flags 0x84700 phys_seg 1 prio<br>
>> class 0<br>
>> May 9 13:44:33 sphnxoss47 kernel: device-mapper: multipath: 253:52:<br>
>> <br>
>> Failing path 128:112.<br>
>> May 9 13:44:33 sphnxoss47 multipathd[6051]: sdef: mark as failed<br>
>> May 9 13:44:33 sphnxoss47 multipathd[6051]: mpathae: remaining<br>
>> active<br>
>> paths: 1<br>
>> ...<br>
>> ...<br>
>> May 9 13:44:34 sphnxoss47 kernel: mpt3sas_cm0:<br>
>> log_info(0x3112011a):<br>
>> originator(PL), code(0x12), sub_code(0x011a)<br>
>> May 9 13:44:34 sphnxoss47 kernel: mpt3sas_cm0:<br>
>> log_info(0x3112011a):<br>
>> originator(PL), code(0x12), sub_code(0x011a)<br>
>> May 9 13:44:34 sphnxoss47 kernel: mpt3sas_cm0:<br>
>> log_info(0x3112011a):<br>
>> originator(PL), code(0x12), sub_code(0x011a)<br>
>> May 9 13:44:34 sphnxoss47 kernel: md: super_written gets error=-5<br>
>> May 9 13:44:34 sphnxoss47 kernel: md/raid:md8: Disk failure on<br>
>> dm-55,<br>
>> disabling device.<br>
>> May 9 13:44:34 sphnxoss47 kernel: md: super_written gets error=-5<br>
>> May 9 13:44:34 sphnxoss47 kernel: md/raid:md8: Operation continuing<br>
>> on<br>
>> 9 devices.<br>
>> May 9 13:44:34 sphnxoss47 multipathd[6051]: sdah: mark as failed<br>
>> _______________________________________________<br>
>> lustre-discuss mailing list<br>
>> <a href="mailto:lustre-discuss@lists.lustre.org" target="_blank" rel="noreferrer">lustre-discuss@lists.lustre.org</a><br>
>> <a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer noreferrer" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a> [1]<br>
> <br>
> --<br>
> <br>
> ------------------------------<br>
> Jeff Johnson<br>
> Co-Founder<br>
> Aeon Computing<br>
> <br>
> <a href="mailto:jeff.johnson@aeoncomputing.com" target="_blank" rel="noreferrer">jeff.johnson@aeoncomputing.com</a><br>
> <a href="http://www.aeoncomputing.com" rel="noreferrer noreferrer" target="_blank">www.aeoncomputing.com</a> [2]<br>
> t: 858-412-3810 x1001 f: 858-412-3845<br>
> m: 619-204-9061<br>
> <br>
> 4170 Morena Boulevard, Suite C - San Diego, CA 92117<br>
> <br>
> High-Performance Computing / Lustre Filesystems / Scale-out Storage<br>
> <br>
> Links:<br>
> ------<br>
> [1] <br>
> <a href="https://urldefense.com/v3/__http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org__;!!P4SdNyxKAPE!B65twCaGe4aP1xnGrjpUnd-1OYuemL3X9zWyxfWEA54zk2tnvbhhrBFW5x9rXl7nFEkSsZpiRGIbodWHehLDQyvnK6u95iVHjg$" rel="noreferrer noreferrer" target="_blank">https://urldefense.com/v3/__http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org__;!!P4SdNyxKAPE!B65twCaGe4aP1xnGrjpUnd-1OYuemL3X9zWyxfWEA54zk2tnvbhhrBFW5x9rXl7nFEkSsZpiRGIbodWHehLDQyvnK6u95iVHjg$</a><br>
> [2] <br>
> <a href="https://urldefense.com/v3/__http://www.aeoncomputing.com__;!!P4SdNyxKAPE!B65twCaGe4aP1xnGrjpUnd-1OYuemL3X9zWyxfWEA54zk2tnvbhhrBFW5x9rXl7nFEkSsZpiRGIbodWHehLDQyvnK6vvMMT5RQ$" rel="noreferrer noreferrer" target="_blank">https://urldefense.com/v3/__http://www.aeoncomputing.com__;!!P4SdNyxKAPE!B65twCaGe4aP1xnGrjpUnd-1OYuemL3X9zWyxfWEA54zk2tnvbhhrBFW5x9rXl7nFEkSsZpiRGIbodWHehLDQyvnK6vvMMT5RQ$</a><br>
_______________________________________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank" rel="noreferrer">lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer noreferrer" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
</blockquote></div>