[lustre-devel] ASSERTION( sdio->csd_write_copied ) failed: osc_request.c:2796:osc_build_rpc()
Goswin von Brederlow
brederlo at q-leap.de
Wed Jul 29 07:14:46 UTC 2026
Hi,
we have a login node for a HPC cluster where the users now run a lot of
AI agents and the node crashes regulary. Recently we managed to capture
the console output of such a crash showing an assetion failure in Lustre:
[120969.603402] LustreError:
2000985:0:(osc_cache.c:2226:osc_check_rpcs()) Write request failed with -14
[120969.612646] LustreError:
2000985:0:(osc_cache.c:2226:osc_check_rpcs()) Skipped 633 previous
similar messages
[120973.828779] LustreError:
2114014:0:(osc_cache.c:2226:osc_check_rpcs()) Write request failed with -14
[120973.838019] LustreError:
2114014:0:(osc_cache.c:2226:osc_check_rpcs()) Skipped 602 previous
similar messages
[120985.932014] LustreError:
2052733:0:(osc_cache.c:2226:osc_check_rpcs()) Write request failed with -14
[120985.941261] LustreError:
2052733:0:(osc_cache.c:2226:osc_check_rpcs()) Skipped 521 previous
similar messages
[121123.684584] LustreError:
2113054:0:(osc_cache.c:2226:osc_check_rpcs()) Write request failed with -14
[121123.694010] LustreError:
2113054:0:(osc_cache.c:2226:osc_check_rpcs()) Skipped 1041 previous
similar messages[121239.558182] LustreError:
2159945:0:(cl_io.c:1692:__ll_dio_user_copy()) Unaligned DIO copy
repeatedly short, count 41905787, offset 2218, bytes 1878, copied 0, pos
537381034
[121239.573691] LustreError:
2159945:0:(osc_request.c:2796:osc_build_rpc()) ASSERTION(
sdio->csd_write_copied ) failed:
[121239.584318] LustreError:
2159945:0:(osc_request.c:2796:osc_build_rpc()) LBUG
[121239.591467] CPU: 80 UID: 0 PID: 2159945 Comm: kworker/80:1 Tainted:
GO 6.12.95-ql-generic-14-12 #1
[121239.591471] Tainted: [O]=OOT_MODULE
[121239.591472] Hardware name: Supermicro AS -1124US-TNRP/H12DSU-iN,
BIOS 2.1 05/07/2021
[121239.591475] Workqueue: events brw_queue_work [osc]
[121239.591487] Call Trace:
[121239.591491] <TASK>
[121239.591493] dump_stack_lvl+0x64/0x80
[121239.591503] lbug_with_loc+0x38/0x70 [libcfs]
[121239.591509] osc_build_rpc+0x187/0x1620 [osc]
[121239.591516] osc_check_rpcs+0x9dd/0x1380 [osc]
[121239.591522] ? __update_idle_core+0x22/0xc0
[121239.591529] osc_io_unplug0+0x7b/0xa0 [osc]
[121239.591534] brw_queue_work+0x55/0x100 [osc]
[121239.591539] process_one_work+0x142/0x2a0
[121239.591543] worker_thread+0x180/0x2e0
[121239.591546] ? __pfx_worker_thread+0x10/0x10
[121239.591547] kthread+0xca/0x100
[121239.591552] ? __pfx_kthread+0x10/0x10
[121239.591553] ret_from_fork+0x2c/0x50
[121239.591558] ? __pfx_kthread+0x10/0x10
[121239.591559] ret_from_fork_asm+0x1a/0x30
[121239.591564] </TASK>
Just before the LASSERT the code calls ll_dio_user_copy()
from lustre/obdclass/cl_io.c:
/* copy IO data to/from internal buffer and userspace iovec */
static ssize_t __ll_dio_user_copy(struct cl_sub_dio *sdio)
...
if (unlikely(copied < bytes)) {
short_copies++;
CDEBUG(D_VFSTRACE,
"short copy - copied only %zd of %lu, short %d times\n",
copied, bytes, short_copies);
/* copies will very rarely be interrupted, but we
* should retry in those cases, since the other option
* is giving an IO error and this can occur in normal
* operation such as with racing unaligned AIOs
*
* but of course we should not retry indefinitely
*/
if (short_copies > 2) {
CERROR("Unaligned DIO copy repeatedly short, count %zd,
offset %lu, bytes %lu, copied %zd, pos %lld\n",
count, offset, bytes, copied, pos);
status = -EFAULT;
break;
}
continue;
}
...
if (rw == WRITE && status == 0)
sdio->csd_write_copied = true;
...
/* the total bytes copied, or status */
RETURN(original_count - count ? original_count - count : status);
}
From the comments this seems to be specific to unaligned writes and has
a known race condition where it may randomly fail. The
ll_dio_user_copy() function retries 3 times in that case and then gives
up setting status = -EFAULT. Given the failure
the sdio->csd_write_copied is never set to true.
But then comes the problematic part: The function returns the number of
bytes copied and only reports an error if nothing could be copied at
all. We seem to run into a case where the function will copy some pages
but fail 3 times before it completes. The function therefore retruns the
number of bytes copied with sdio->csd_write_copied = false.
And then the error handling in lustre/osc/osc_request.c seems to be
insufficient causing the assertion failure:
/* for unaligned writes, we do the data copying here */
if (sdio && sdio->csd_unaligned && sdio->csd_write) {
rc = ll_dio_user_copy(sdio);
if (rc < 0)
GOTO(out, rc);
/* dio_user_copy has some concurrency handling in it,
* so we add this assert to ensure it did its job...
*/
LASSERT(sdio->csd_write_copied);
}
As you can see the code check for an error return value. Otherwise it
assert sdio->csd_write_copied. But on a partial copy
the ll_dio_user_copy() function returns the number of bytes copied
without sdio->csd_write_copied set. The case of a partial copy seem to
have been overlooked here.
I have 2 tiny patches for this that I hope will be useful for this. The
first a mitigation to reduce the risk of failures and the second
hopefully adding error handling for partial copies:
diff --git a/lustre/obdclass/cl_io.c b/lustre/obdclass/cl_io.c
index 5175bbd148..f2a9306be9 100644
--- a/lustre/obdclass/cl_io.c
+++ b/lustre/obdclass/cl_io.c
@@ -1676,7 +1676,11 @@ static ssize_t __ll_dio_user_copy(struct
cl_sub_dio *sdio)
count -= copied;
if (unlikely(copied < bytes)) {
- short_copies++;
+ if (coppied == 0) {
+ short_copies++;
+ } else {
+ short_copies = 0;
+ }
CDEBUG(D_VFSTRACE,
"short copy - copied only %zd of %lu,
short %d times\n",
The intent of the short_copies counter seems to be to prevent endless
loops in case copying becomes impossible. The idea of the patch is that
the short_copies counter is reset every time the copying makes progress.
The number of errors is irrelevant as long as progress is made if all
lustre cares about is avoiding an endless loop. I would also suggest
increasing the number of allowed short copies from 2 to something larger.
diff --git a/lustre/osc/osc_request.c b/lustre/osc/osc_request.c
index 772605f678..a1d552fa86 100644
--- a/lustre/osc/osc_request.c
+++ b/lustre/osc/osc_request.c
@@ -2790,6 +2790,8 @@ int osc_build_rpc(const struct lu_env *env, struct
client_obd *cli,
rc = ll_dio_user_copy(sdio);
if (rc < 0)
GOTO(out, rc);
+ if (rc != sdio->csd_bytes) // can't handle short
copies
+ GOTO(out, -EFAULT);
/* dio_user_copy has some concurrency handling
in it,
* so we add this assert to ensure it did its
job...
*/
This hopefully is the right fix for the assertion failure. A partial
copy of the write buffer is considered the same as a total failure. I
assume this will cause data loss on the file but report the error
correctly on e.g. fsync().
What do you think? Am I on the right track?
Regards,
Goswin von Brederlow
More information about the lustre-devel
mailing list