[lustre-discuss] New client mounts fail after deactivating OSTs

[Brad Merchant]

We deactivated half of 32 OSTs after draining them. We followed the steps
in section 14.9.3 of the lustre manual

https://doc.lustre.org/lustre_manual.xhtml#lustremaint.remove_ost

After running the steps in subhead "3. Deactivate the OST." on
OST0010-OST001f, new client mounts fail with the below log messages.
Existing client mounts seem to function correctly but are on a bit of a
ticking timebomb because they are configured with autofs.

The llog_cancel steps are new to me and the issues seemed to appear after
those commands were issued (can't say that 100% definitively however).
Servers are running 2.12.5 and clients are on 2.14.x


Jul 10 15:22:40 adm-sup1 kernel: LustreError:
26814:0:(obd_config.c:1514:class_process_config()) no device for:
hydra-OST0010-osc-ffff8be5340c2000
Jul 10 15:22:40 adm-sup1 kernel: LustreError:
26814:0:(obd_config.c:2038:class_config_llog_handler())
MGC172.16.100.101 at o2ib: cfg command failed: rc = -22
Jul 10 15:22:40 adm-sup1 kernel: Lustre:    cmd=cf00f 0:hydra-OST0010-osc
 1:osc.active=0
Jul 10 15:22:40 adm-sup1 kernel: LustreError: 15b-f: MGC172.16.100.101 at o2ib:
Configuration from log hydra-client failed from MGS -22. Check client and
MGS are on compatible version.
Jul 10 15:22:40 adm-sup1 kernel: Lustre: hydra: root_squash is set to 99:99
Jul 10 15:22:40 adm-sup1 systemd-udevd[26823]: Process '/usr/sbin/lctl
set_param 'llite.hydra-ffff8be5340c2000.nosquash_nids=192.168.80.84 at tcp
192.168.80.122 at tcp 192.168.80.21 at tcp 172.16.90.11 at o2ib 172.16.100.211 at o2ib
172.16.100.212 at o2ib 172.16.100.213 at o2ib 172.16.100.214 at o2ib
172.16.100.215 at o2ib 172.16.90.51 at o2ib'' failed with exit code 2.
Jul 10 15:22:40 adm-sup1 kernel: Lustre: Unmounted hydra-client
Jul 10 15:22:40 adm-sup1 kernel: LustreError:
26803:0:(obd_mount.c:1680:lustre_fill_super()) Unable to mount  (-22)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20230710/872985b7/attachment.htm>