[lustre-discuss] Poor read performance when using o2ib nets over RoCE

[Christian Kuntz]

Hello,

I've been running into a strange issue where my writes are blazingly fast
(5.5 GB/s) over RoCE with Mellanox MCX516A-CCAT cards all running together
over o2ib, but read performance tanks to roughly 100 MB/s. During mixed
read/write situations write performance also plummets to sub 100MB/s.

Curiously, when using tcp these problems disappear and everyone is happy,
hovering around 1.5 GB/s read, 3 GB/s write.

I'm wondering if anyone else has run into this and what the solution may
be?
My setup is:
Debian 10.3, lustre 2.13.0, zfs 0.8.2 with two OSS/OST pairs, a single
mgs/mdt node and a single client node connected over o2ib. Everyone is
cabled together via 100g fiber through a mellanox switch that's configured
for roce and bonding, and they all hit about 98 Gb/s to each other via
ib_send_bw, and simple testing of network file transfers via NFSoRDMA
didn't experience the slowdown that lustre seems to be seeing.

I'd be happy to provide more diagnostic information if that helps, as well
as trace information if needed.

Best,
Christian

-- 
 <https://opendrives.com/nab/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20200211/18f965ab/attachment.html>