[lustre-discuss] MDT refuses to mount: "no more free slots in catalog" "can't initialize llog"

[Jesse Stroik]

Hi Lustre users,

Recently we had a breaker fail overnight which affected part of one of our data centers, including an older lustre setup. The lustre setup was in use prior to the power failure.

This is lustre 2.15.1 / zfs 2.1.2 running on Rocky 8 and using zfs for the all backend file systems.

When i attempt to start lustre on the MDS, it mounts the MGS and then starts mounting the MDT and goes into recovery before failing with the following messages:

=======
Lustre: 5089:0:(llog_cat.c:101:llog_cat_new_log()) arc15-OST0001-osc-MDT0000: there are no more free slots in catalog [0x5:0x1:0x0]:0
LustreError: 5089:0:(osp_sync.c:1553:osp_sync_init()) arc15-OST0001-osc-MDT0000: can't initialize llog: rc = -28
LustreError: 5089:0:(obd_config.c:774:class_setup()) setup arc15-OST0001-osc-MDT0000 failed (-28)
LustreError: 6519:0:(obd_config.c:2001:class_config_llog_handler()) MGC172.16.23.25 at o2ib: cfg command failed: rc = -28
Lustre:    cmd=cf003 0:arc15-OST0001-osc-MDT0000  1:arc15-OST0001_UUID  2:172.16.23.18 at o2ib  
=======

If i attempt to start lustre again or mount the MDT directly after this first attempt, i also see this message in the logs:

=======
 LustreError: 5350:0:(genops.c:522:class_register_device()) arc15-OST0001-osc-MDT0000: already exists, won't add
=======

LNET communication looks good, and this error happens whether or not the OSS units are powered up and have their OSTs mounted. The system didn't have a changelog user registered and wasn't consuming any changelog space.

If I mount just the MGS i can still see the configuration logs with: "lctl --device MGS llog_print <device>" for all devices.

My planned next steps are to regenerate the lustre configuration logs following the instructions here:

https://doc.lustre.org/lustre_manual.xhtml#lustremaint.regenerateConfigLogs

I do have snapshots of the MGS and MDT stored on a zpool on another server.

Before i move on with that step, is there anything i should check or am missing?

Thanks,
Jesse