[lustre-discuss] Lustre stuck in ldlm_lockd (lock on destroyed export, lock timed out)

[Thomas Roth]

In addition, I noticed that those clients that do reconnect are logged as

Mar 10 13:12:24 lxmds19.gsi.de kernel: Lustre: hebe-MDT0000: Connection restored to  (at 10.20.0.41 at o2ib5)

MDS and MDT have this client listed (/proc/fs/lustre/.../exports/) and there is a uuid there for the client.


Regards
Thomas

On 10.03.21 12:33, Thomas Roth via lustre-discuss wrote:
> Hi all,
> 
> we are in a critical situation where our Lustre is rendered completely inaccessible.
> 
> We are running Lustre 2.12.5 on CentOS 7.8, Whamcloud sources, MDTs on ldiskfs, OSTs on ZFS, 3 MDS.
> 
> The first MDS, running MGS + MDT0, is showing
> ### lock callback timer expired
> evicting clients, and
> ### lock on destroyed export
> for the same client, as in
> 
> 
> Mar 10 09:51:54 lxmds19.gsi.de kernel: LustreError: 4779:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 450s: evicting 
> client at 10.20.4.68 at o2ib5  ns: mdt-hebe-MDT0000_UUID lock: ffff8f1ef6681b00/0xdba5480d76a73ab6 lrc: 3/0,0 mode: PR/PR res: [0x20002db4c:0x14:0x0].0x0 
> bits 0x13/0x0 rrc: 3 type: IBT flags: 0x60200400000020 nid: 10.20.4.68 at o2ib5 remote: 0x5360294b0558b867 expref: 31 pid: 6649 timeout: 4849 lvb_type: 0
> 
> Mar 10 09:51:54 lxmds19.gsi.de kernel: LustreError: 6570:0:(ldlm_lockd.c:1348:ldlm_handle_enqueue0()) ### lock on destroyed export ffff8f1eede90000 
> ns: mdt-hebe-MDT0000_UUID lock: ffff8f1efbded8c0/0xdba5480d76a9e456 lrc: 3/0,0 mode: PR/PR res: [0x20002c52b:0xd92b:0x0].0x0 bits 0x13/0x0 rrc: 175 
> type: IBT flags: 0x50200400000020 nid: 10.20.4.68 at o2ib5 remote: 0x5360294b0558b875 expref: 4 pid: 6570 timeout: 0 lvb_type: 0
> 
> 
> 
> Eventually, there is
> ### lock timed out ; not entering recovery in server code, just going back to sleep
> 
> 
> Restart of the server does not help.
> Recovery runs through, clients show the MDS in 'lfs check mds', but any kind of access (aka 'ls') will hang.
> 
> 
> Any help is much appreciated.
> 
> Regards
> Thomas
> 
> 

-- 
--------------------------------------------------------------------
Thomas Roth
Department: IT
Location: SB3 2.291
Phone: +49-6159-71 1453  Fax: +49-6159-71 2986

GSI Helmholtzzentrum für Schwerionenforschung GmbH
Planckstraße 1, 64291 Darmstadt, Germany, www.gsi.de

Commercial Register / Handelsregister: Amtsgericht Darmstadt, HRB 1528
Managing Directors / Geschäftsführung:
Professor Dr. Paolo Giubellino, Dr. Ulrich Breuer, Jörg Blaurock
Chairman of the Supervisory Board / Vorsitzender des GSI-Aufsichtsrats:
State Secretary / Staatssekretär Dr. Volkmar Dietz