<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<meta name="Generator" content="Microsoft Exchange Server">
<!-- converted from text --><style><!-- .EmailQuote { margin-left: 1pt; padding-left: 4pt; border-left: #800000 2px solid; } --></style>
</head>
<body>
<meta content="text/html; charset=UTF-8">
<style type="text/css" style="">
<!--
p
{margin-top:0;
margin-bottom:0}
-->
</style>
<div dir="ltr">
<div id="x_divtagdefaultwrapper" style="font-size:12pt; color:#000000; background-color:#FFFFFF; font-family:Calibri,Arial,Helvetica,sans-serif">
<p>Thomas,</p>
<p><br>
</p>
<p>It is somewhat sideways from your questions, but when Cray has seen this problem historically, it has almost always been due to lots of small direct I/O from a user code.</p>
<p><br>
</p>
<p>- Patrick</p>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> lustre-discuss <lustre-discuss-bounces@lists.lustre.org> on behalf of Thomas Roth <t.roth@gsi.de><br>
<b>Sent:</b> Saturday, September 10, 2016 2:38:37 AM<br>
<b>To:</b> lustre-discuss@lists.lustre.org<br>
<b>Subject:</b> [lustre-discuss] RDMA too fragmented, OSTs unavailable (permanently)</font>
<div> </div>
</div>
</div>
<font size="2"><span style="font-size:10pt;">
<div class="PlainText">Hi all,<br>
<br>
we are running Lustre 2.5.3 on Infiniband. We have massive problems with clients being unable to communicate with any number of OSTs, rendering the
<br>
entire cluster quite unusable.<br>
<br>
Clients show<br>
> LNetError: 1399:0:(o2iblnd_cb.c:1140:kiblnd_init_rdma()) RDMA too fragmented for 10.20.0.242@o2ib1 (256): 231/256 src 231/256 dst frags<br>
> LNetError: 1399:0:(o2iblnd_cb.c:1690:kiblnd_reply()) Can't setup rdma for GET from 10.20.0.242@o2ib1: -90<br>
<br>
which eventually results in OSTs at that nid becoming "temporarily unavailable".<br>
However, the OSTs are never recovered, until they are manually evicted or the host rebooted.<br>
<br>
On the OSS side, this reads<br>
> LNetError: 13660:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.20.0.220@o2ib1 (56): c: 7, oc: 0, rc: 7<br>
<br>
<br>
We have checked the IB fabric, which shows no errors. Since we are not able to reproduce this effect in a simple way, we have also scrutinized the
<br>
user code, so far without results.<br>
<br>
Whenever this happens, the connection between client and OSS is fine under all IB test commands.<br>
Communication between client and OSS is still going on, but obviously when Lustre tries to replay the missed transaction, this fragmentation limit is
<br>
hit again, so the OST never becomes available again.<br>
<br>
If we understand correctly, the map_on_demand parameter should be increased as a workaround.<br>
The ko2iblnd module seems to provide this parameter,<br>
> modinfo ko2iblnd<br>
> parm: map_on_demand:map on demand (int)<br>
<br>
but no matter what we load the module with, map_on_demand always remains at the default value,<br>
> cat /sys/module/ko2iblnd/parameters/map_on_demand<br>
> 0<br>
<br>
Is there any way to understand<br>
- why this memory fragmentation occurs/becomes so large?<br>
- how to measure the real fragmentation degree (o2iblnd simply stops at 256, perhaps we are at 1000?)<br>
- why map_on_demand cannot be changed?<br>
<br>
<br>
Of course this all looks very much like LU-5718, but our clients are not behind LNET routers.<br>
<br>
There is one router which connects to the campus network but is not in use. And there are some routers which connect to an older cluster, but of
<br>
course the old (1.8) clients never show any of these errors.<br>
<br>
<br>
Cheers,<br>
Thomas<br>
<br>
--------------------------------------------------------------------<br>
Thomas Roth<br>
Department: HPC<br>
Location: SB3 1.262<br>
Phone: +49-6159-71 1453 Fax: +49-6159-71 2986<br>
<br>
GSI Helmholtzzentrum für Schwerionenforschung GmbH<br>
Planckstraße 1<br>
64291 Darmstadt<br>
www.gsi.de<br>
<br>
Gesellschaft mit beschränkter Haftung<br>
Sitz der Gesellschaft: Darmstadt<br>
Handelsregister: Amtsgericht Darmstadt, HRB 1528<br>
<br>
Geschäftsführung: Professor Dr. Karlheinz Langanke<br>
Ursula Weyrich<br>
Jörg Blaurock<br>
<br>
Vorsitzender des Aufsichtsrates: St Dr. Georg Schütte<br>
Stellvertreter: Ministerialdirigent Dr. Rolf Bernhardt<br>
_______________________________________________<br>
lustre-discuss mailing list<br>
lustre-discuss@lists.lustre.org<br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
</div>
</span></font>
</body>
</html>