[lustre-discuss] Mistake while adding OST - OST0003 : Invalid argument
BALVERS Martin
Martin.BALVERS at danone.com
Thu Aug 14 00:06:32 PDT 2025
Hi Thomas,
Thanks for the reply.
Here is the output:
[root at mds ~]# lctl get_param osp.lustre-OST0003-osc-MDT0000.active
osp.lustre-OST0003-osc-MDT0000.active=1
The problem is caused by the fact that I replaced a previously added OST using the --replace option in the mkfs.lustre command.
OST0004 was added correctly in one go, and that one is visible to the client, so I don't think it is a client problem. The client can reach the OSS that has OST0003.
It seems that the according to the MDS everything is fine. It is only the 'lfs df' command that shows a problem. Regular 'df' shows all the space of the lustre fs, but 'lfs df' does not.
# df -h
Filesystem Size Used Avail Use% Mounted on
192.168.6.1 at tcp:/lustre 487T 223T 264T 46% /lustre
# lfs df -h
UUID bytes Used Available Use% Mounted on
lustre-MDT0000_UUID 11.4T 2.8T 8.6T 25% /lustre[MDT:0]
lustre-OST0000_UUID 97.2T 74.4T 22.8T 77% /lustre[OST:0]
lustre-OST0001_UUID 97.2T 74.0T 23.2T 77% /lustre[OST:1]
lustre-OST0002_UUID 97.2T 74.1T 23.1T 77% /lustre[OST:2]
OST0003 : Invalid argument
lustre-OST0004_UUID 97.2T 7.0M 97.2T 1% /lustre[OST:4]
filesystem_summary: 388.9T 222.5T 166.4T 58% /lustre
The fs is not in use at the moment, no data has been written to it since adding OST0003 an 4. If there is a way to fix this removing and adding OST0003 again then I can try that. As long as the data that is on the other OST's is not lost.
Regards,
Martin Balvers
In case this somehow helps:
[root at mds ~]# lctl get_param osp.lustre-OST0003-osc-MDT0000.*
osp.lustre-OST0003-osc-MDT0000.active=1
osp.lustre-OST0003-osc-MDT0000.blocksize=4096
osp.lustre-OST0003-osc-MDT0000.create_count=32
osp.lustre-OST0003-osc-MDT0000.destroys_in_flight=0
osp.lustre-OST0003-osc-MDT0000.filesfree=3262643968
osp.lustre-OST0003-osc-MDT0000.filestotal=3262644326
error: read_param: '/sys/fs/lustre/osp/lustre-OST0003-osc-MDT0000/force_sync': Permission denied
osp.lustre-OST0003-osc-MDT0000.kbytesavail=104404566016
osp.lustre-OST0003-osc-MDT0000.kbytesfree=104404606976
osp.lustre-OST0003-osc-MDT0000.kbytestotal=104404615168
osp.lustre-OST0003-osc-MDT0000.lfsck_max_rpcs_in_flight=512
osp.lustre-OST0003-osc-MDT0000.max_create_count=20000
osp.lustre-OST0003-osc-MDT0000.max_rpcs_in_flight=8
osp.lustre-OST0003-osc-MDT0000.max_rpcs_in_progress=4096
osp.lustre-OST0003-osc-MDT0000.max_sync_changes=2000000000
osp.lustre-OST0003-osc-MDT0000.maxage=5
osp.lustre-OST0003-osc-MDT0000.old_sync_processed=1
osp.lustre-OST0003-osc-MDT0000.ost_conn_uuid=192.168.6.5 at tcp
osp.lustre-OST0003-osc-MDT0000.prealloc_last_id=33
osp.lustre-OST0003-osc-MDT0000.prealloc_last_seq=0x100030000
osp.lustre-OST0003-osc-MDT0000.prealloc_next_id=2
osp.lustre-OST0003-osc-MDT0000.prealloc_next_seq=0x100030000
osp.lustre-OST0003-osc-MDT0000.prealloc_reserved=0
osp.lustre-OST0003-osc-MDT0000.prealloc_status=0
osp.lustre-OST0003-osc-MDT0000.sync_changes=0
osp.lustre-OST0003-osc-MDT0000.sync_in_flight=0
osp.lustre-OST0003-osc-MDT0000.sync_in_progress=0
osp.lustre-OST0003-osc-MDT0000.uuid=lustre-MDT0000-mdtlov_UUID
osp.lustre-OST0003-osc-MDT0000.connect_flags=
flags=0x2040401443000066
flags2=0x0
lov_index
connect_from_mds
version
request_portal
adaptive_timeouts
lru_resize
fid_is_enabled
skip_orphan
full20
lvb_type
lfsck
bulk_mbits
osp.lustre-OST0003-osc-MDT0000.import=
import:
name: lustre-OST0003-osc-MDT0000
target: lustre-OST0003_UUID
state: FULL
connect_flags: [ lov_index, connect_from_mds, version, request_portal, adaptive_timeouts, lru_resize, fid_is_enabled, skip_orphan, full20, lvb_type, lfsck, bulk_mbits ]
connect_data:
flags: 0x2040401443000066
instance: 1
target_version: 2.15.1.0
mdt_index: 0
target_index: 3
import_flags: [ replayable, pingable, connect_tried ]
connection:
failover_nids: [ 192.168.6.5 at tcp ]
current_connection: 192.168.6.5 at tcp
connection_attempts: 24
generation: 3
in-progress_invalidations: 0
idle: 5 sec
rpcs:
inflight: 0
unregistering: 0
timeouts: 17
avg_waittime: 464 usec
service_estimates:
services: 1 sec
network: 1 sec
transactions:
last_replay: 0
peer_committed: 0
last_checked: 0
osp.lustre-OST0003-osc-MDT0000.ost_server_uuid=lustre-OST0003_UUID FULL
osp.lustre-OST0003-osc-MDT0000.reserved_mb_high=38883
osp.lustre-OST0003-osc-MDT0000.reserved_mb_low=19441
osp.lustre-OST0003-osc-MDT0000.srpc_info=
rpc flavor: null
bulk flavor: null
flags: -,
id: -1
refcount: 23
nctx: 1
gc internal 0
gc next 0
error: read_param: '/sys/kernel/debug/lustre/osp/lustre-OST0003-osc-MDT0000/srpc_sepol': Invalid argument
osp.lustre-OST0003-osc-MDT0000.state=
current_state: FULL
state_history:
- [ 1755076684, CONNECTING ]
- [ 1755076739, DISCONN ]
- [ 1755076739, CONNECTING ]
- [ 1755076794, DISCONN ]
- [ 1755076794, CONNECTING ]
- [ 1755076849, DISCONN ]
- [ 1755076849, CONNECTING ]
- [ 1755076904, DISCONN ]
- [ 1755076904, CONNECTING ]
- [ 1755076959, DISCONN ]
- [ 1755076959, CONNECTING ]
- [ 1755076975, DISCONN ]
- [ 1755078008, CONNECTING ]
- [ 1755078008, EVICTED ]
- [ 1755078008, RECOVER ]
- [ 1755078008, FULL ]
osp.lustre-OST0003-osc-MDT0000.stats=
snapshot_time 1112343.516378917 secs.nsecs
start_time 0.000000000 secs.nsecs
elapsed_time 1112343.516378917 secs.nsecs
req_waittime 14767 samples [usec] 221 1689 6858368 3261577492
req_active 14785 samples [reqs] 1 2 14809 14857
ost_create 4 samples [usec] 221 649 1775 950619
ost_get_info 2 samples [usec] 1403 1689 3092 4821130
ost_connect 6 samples [usec] 269 1095 4323 3583477
ost_statfs 14753 samples [usec] 247 718 6848319 3251852025
obd_ping 2 samples [usec] 404 455 859 370241
osp.lustre-OST0003-osc-MDT0000.timeouts=
last reply : 1755153467, 5s ago
network : cur 1 worst 1 (at 1755078008, 75464s ago) 1 1 1 1
portal 28 : cur 1 worst 1 (at 1755076033, 77439s ago) 1 1 0 0
portal 7 : cur 1 worst 1 (at 1755076033, 77439s ago) 1 1 1 1
________________________________
From: lustre-discuss <lustre-discuss-bounces at lists.lustre.org> on behalf of Thomas Roth via lustre-discuss <lustre-discuss at lists.lustre.org>
Sent: Wednesday, August 13, 2025 15:44
To: lustre-discuss at lists.lustre.org <lustre-discuss at lists.lustre.org>
Subject: [lustre-discuss] Mistake while adding OST - OST0003 : Invalid argument
WARNING - EXTERNAL SENDER - BE CYBERSAFE
Hi,
did you check the status of the new OST on the MDS? Some 'lctl get_param a.b.c.active' command?
Because I'm not sure lctl --device 13 activate
does the trick.
And it isn't a client's problem either?
Regards
Thomas
Ce message électronique et tous les fichiers attachés qu'il contient sont confidentiels et destinés exclusivement à l'usage de la personne à laquelle ils sont adressés. Si vous avez reçu ce message par erreur, merci de le retourner à son émetteur. Les idées et opinions présentées dans ce message sont celles de son auteur, et ne représentent pas nécessairement celles de DANONE ou d'une quelconque de ses filiales. La publication, l'usage, la distribution, l'impression ou la copie non autorisée de ce message et des attachements qu'il contient sont strictement interdits.
This e-mail and any files transmitted with it are confidential and intended solely for the use of the individual to whom it is addressed. If you have received this email in error please send it back to the person that sent it to you. Any views or opinions presented are solely those of its author and do not necessarily represent those of DANONE or any of its subsidiary companies. Unauthorized publication, use, dissemination, forwarding, printing or copying of this email and its associated attachments is strictly prohibited.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20250814/59fff185/attachment-0001.htm>
More information about the lustre-discuss
mailing list