<div style="font-family: -apple-system, system-ui; font-size: 14px; color: rgb(0, 0, 0);"><span style="line-height: 1.6;">hi,</span></div><div style="font-family: -apple-system, system-ui; font-size: 14px; color: rgb(0, 0, 0);"><span style="line-height: 1.6;"><br /></span></div><div style="font-family: -apple-system, system-ui; font-size: 14px; color: rgb(0, 0, 0);"><span style="line-height: 1.6;"> i guess that's kernel version compatible. do you try with kernel 6.8.0-38 ?</span></div><div style="display:inline-block" contenteditable="false"><div style="display:block;width:150px;height:1px;border:none;margin:32px 0px 10px;background:rgb(230, 232, 235)"></div><div><a target="_blank" href="https://wx.mail.qq.com/home/index?t=readmail_businesscard_midpage&nocheck=true&name=%E5%A4%8F%E5%A4%A9&icon=http%3A%2F%2Fthirdqq.qlogo.cn%2Fek_qqapp%2FAQCGejBxtuXtZElGqVKcd5ic7qAFt0UyvAiaa0Ac2aONSNNrrXRqQ5Vemicalponw%2F0&mail=313680712%40qq.com&code=Mdx0r_1DZNJ7VkmyIPQ8g1CDhpslLIO6OiI0k6-zDN2G_jnBqTwjDwMGIhZJATrN2wps2Mq3c4FE6Uj0ipvlLQ" style="text-decoration: underline;display:inline-block;text-decoration:none !important;font-family:-apple-system,BlinkMacSystemFont,PingFang SC,Microsoft YaHei" class="xm_write_card" data-readonly="true"><table cellspacing="0" cellpadding="0" style="table-layout:fixed;padding-right:20px"><tbody><tr valign="top"><td style="width:40px;min-width:40px;padding-top:10px"><div style="width:38px;height:38px;border:1px #FFF solid;border-radius:50%;margin:0;vertical-align:top;box-shadow:0 0 10px 0 rgba(127,152,178,0.14)"><img style="vertical-align: bottom;width:100%;height:100%;border-radius:50%;pointer-events:none" src="http://thirdqq.qlogo.cn/ek_qqapp/AQCGejBxtuXtZElGqVKcd5ic7qAFt0UyvAiaa0Ac2aONSNNrrXRqQ5Vemicalponw/0" /></div></td><td style="padding:10px 0 8px 10px"><div style="font-size:14px;color:#33312E;line-height:20px;padding-bottom:2px;margin:0;font-weight:500" class="businessCard_name">夏天</div><div style="font-size:12px;color:#999896;line-height:18px;margin:0" class="businessCard_mail">313680712@qq.com</div></td></tr></tbody></table></a></div></div>
<div><br /></div><div><br /></div><article><div style="display:flex;align-items:center;padding-top:8px" contenteditable="false">
<div style="color:#959DA6;font-size:12px;line-height:30px;background:#FFF">Original</div>
<hr style="border: none;flex-grow:1;border-top:1px solid rgba(21, 46, 74, 0.07);margin-left:8px" />
</div><table data-uneditable="true" style="line-height: 20px; border-radius: 6px; background-color: rgba(20, 46, 77, 0.05); margin: 0px; width: 100%;"><tbody><tr><td style="width: 866.2px; height: 96px; line-height: 20px; padding: 8px;"><div style="line-height: 20px; font-size: 12px;"><span style="color: rgb(92, 97, 102);">From:</span><span style="color: rgb(0, 0, 0);">lustre-discuss-request</span> <span style="color: rgb(149, 157, 166);"><lustre-discuss-request@lists.lustre.org></span></div><div style="line-height: 20px; font-size: 12px;"><span style="color: rgb(92, 97, 102);">Sent Time:</span><span style="color: rgb(0, 0, 0);">2025-02-8- 02:27</span></div><div style="line-height: 20px; font-size: 12px;"><span style="color: rgb(92, 97, 102);">To:</span><span style="color: rgb(0, 0, 0);">lustre-discuss</span> <span style="color: rgb(149, 157, 166);"><lustre-discuss@lists.lustre.org></span></div><div style="line-height: 20px; font-size: 12px;"><span style="color: rgb(92, 97, 102);">Subject:</span><span style="color: rgb(0, 0, 0);">lustre-discuss Digest, Vol 227, Issue 4</span></div></td></tr></tbody></table><div><br /></div>
Send lustre-discuss mailing list submissions to<br /> lustre-discuss@lists.lustre.org<br /><br />To subscribe or unsubscribe via the World Wide Web, visit<br /> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org<br />or, via email, send a message with subject or body 'help' to<br /> lustre-discuss-request@lists.lustre.org<br /><br />You can reach the person managing the list at<br /> lustre-discuss-owner@lists.lustre.org<br /><br />When replying, please edit your Subject line so it is more specific<br />than "Re: Contents of lustre-discuss digest..."<br /><br /><br />Today's Topics:<br /><br />1. Problem with 2.16.1 server on Ubuntu Noble 24.04 (?ke Sandgren)<br />2. Re: Lnet not going up with InfiniHost III Lx HCA card<br />(Jesse Stroik)<br />3. Re: Lustre and ZFS draid (Cameron Harr)<br /><br /><br />----------------------------------------------------------------------<br /><br />Message: 1<br />Date: Fri, 7 Feb 2025 15:43:02 +0000<br />From: ?ke Sandgren<br />To: "lustre-discuss@lists.lustre.org"<br /> <br />Subject: [lustre-discuss] Problem with 2.16.1 server on Ubuntu Noble<br /> 24.04<br />Message-ID:<br /> <GVZP280MB0652BE277CCAFED07F805377E8F12@GVZP280MB0652.SWEP280.PROD.OUTLOOK.COM><br /> <br />Content-Type: text/plain; charset="iso-8859-1"<br /><br />Hi!<br /><br />I'm trying to get 2.16.1 working on Ubuntu 24.04 (kernel 6.8.0-52-generic Ubuntu)<br />I have everything built according to the procedure I used when doing this with 2.14 on Ubuntu 20.04 and 22.04.<br />I.e. after a bit of updating ldiskfs patches I have everything built cleanly.<br /><br />However, when I try to load lnet it fails and I get this back:<br />===<br />[Fri Feb 7 16:10:28 2025] BPF: type_id=13 bits_offset=256<br />[Fri Feb 7 16:10:28 2025] BPF:<br />[Fri Feb 7 16:10:28 2025] BPF: Invalid name<br />[Fri Feb 7 16:10:28 2025] BPF:<br />[Fri Feb 7 16:10:28 2025] failed to validate module [libcfs] BTF: -22<br />===<br /><br />Any ideas for what I managed to mess up?<br />I can't find anything relevant in master branch that might fix this, so did I mess up my kernel build or is this a new problem?<br /><br />---<br />Ake Sandgren, HPC2N, Umea University, S-90187 Umea, Sweden<br />Internet: ake@hpc2n.umu.se ?Mobile: +46 70 7716134 ?Fax: +46 90-580 14<br />WWW: http://www.hpc2n.umu.se<br /><br />------------------------------<br /><br />Message: 2<br />Date: Fri, 7 Feb 2025 18:24:16 +0000<br />From: Jesse Stroik<br />To: Ramiro Alba Queipo ,<br /> "lustre-discuss@lists.lustre.org"<br />Subject: Re: [lustre-discuss] Lnet not going up with InfiniHost III Lx<br /> HCA card<br />Message-ID:<br /> <PH0PR06MB77218ABE432182A84CB31CF2DAF12@PH0PR06MB7721.namprd06.prod.outlook.com><br /> <br />Content-Type: text/plain; charset="iso-8859-1"<br /><br />Hi Ramiro,<br /><br />The invalid MR size looks like you're running into a limit with your cards setting up the RDMA (o2ib) LND when bringing up the network. There may be adjustments or workarounds for it possibly including setting map_on_demand=0 as an argument to the lnet module there.<br /><br />And since you are using older IB hardware on a newer OS, just a heads up: we recently ran into an issue with connectx-3 IB cards after upgrading our operating systems where we found RMDA communication to be unreliable possibly because they often would exceed the amount of connection queue pairs they could create. For us, the workaround was to use the ksocklnd instead of o2iblnd. If you have trouble getting the o2ib lustre network driver to work with this older hardware due to RDMA problems, that could be a workaround although it may not be feasible to implement depending on your networking setup.<br /><br />Best,<br />Jesse<br /><br /><br />________________________________________<br />From: lustre-discuss <lustre-discuss-bounces@lists.lustre.org> on behalf of Ramiro Alba Queipo<br />Sent: Thursday, February 6, 2025 3:34 AM<br />To: lustre-discuss@lists.lustre.org<br />Subject: [lustre-discuss] Lnet not going up with InfiniHost III Lx HCA card<br /><br /><br />Hi all,<br /><br />I am testing Ubuntu 24.04 (6.8.0-52-generic) client with Lustre 2.16.1 over Infiniband and using an old Mellanox DDR card (InfiniHost III Lx HCA).<br /><br />- # ip -br a<br /><br />options lnet networks=o2ib0(ib0)<br /><br />- # modprobe lnet<br />- # lctl network up<br /><br />LNET configure error 100: Network is down<br /><br />- # tail -10 /var/log/kernel.log<br /><br />LNetError: 5071:0:(o2iblnd.c:2866:kiblnd_hdev_get_attr()) Invalid mr size: 0xffffffffffffffff<br />LNetError: 5071:0:(o2iblnd.c:3103:kiblnd_dev_failover()) Can't get device attributes: -22<br />LNetError: 5071:0:(o2iblnd.c:3831:kiblnd_startup()) ko2iblnd: Can't initialize device: rc = -22<br />LNetError: Error -100 starting up LNI o2ib<br /><br />Lustre 2.15.0 and Ubuntu 20.04 (kernel 5.4.0-198-generic) is working fine with the same hardware<br /><br />Can anyone give me some advice or idea to make it work?<br /><br />Thans in advance<br />Best regards<br /><br />--<br />Ramiro Alba<br /><br />Centre Tecnol?gic de Tranfer?ncia de Calor<br />http://www.cttc.upc.edu<br /><br />Escola T?cnica Superior d'Enginyeries<br />Industrial i Aeron?utica de Terrassa<br />Colom 11, E-08222, Terrassa, Barcelona, Spain<br />Tel: (+34) 93 739 8928<br /><br /><br />------------------------------<br /><br />Message: 3<br />Date: Fri, 7 Feb 2025 10:26:49 -0800<br />From: Cameron Harr<br />To: lustre-discuss@lists.lustre.org<br />Subject: Re: [lustre-discuss] Lustre and ZFS draid<br />Message-ID: <80ce762b-3921-42a4-a3ce-1d89209528d7@llnl.gov><br />Content-Type: text/plain; charset="utf-8"; Format="flowed"<br /><br />We've been using draid in production since 2020 and I think were<br />generally happy with it. We have quite a few Lustre clusters and on the<br />majority of them, we run 90-drive JBODs with 1 OST/OSS node, 1 OST/pool<br />and 1 pool/JBOD. We use a draid2:8d:90c:2s config and let the<br />distributed spares rebuild (~2-4 hours) before replacing the 16TB<br />physical disk, which then rebuilds within a day or so.<br /><br />An important note with this configuration is that we also include NVMe<br />in the pool as special allocation devices configured to store small<br />blocks up to 16K. We probably have much more NVMe space than we need due<br />to large NVMe drives (zpool list -v shows each mirror capacity is still<br />low), but we're happy with performance.<br /><br /> NAME STATE READ WRITE CKSUM<br /> asp8 ONLINE 0 0 0<br /> draid2:8d:90c:2s-0 ONLINE 0 0 0<br /> L0 ONLINE 0 0 0<br /> L1 ONLINE 0 0 0<br />...<br /> L88 ONLINE 0 0 0<br /> L89 ONLINE 0 0 0<br /> special <br /> mirror-1 ONLINE 0 0 0<br /> N6 ONLINE 0 0 0<br /> N7 ONLINE 0 0 0<br /> mirror-2 ONLINE 0 0 0<br /> N8 ONLINE 0 0 0<br /> N9 ONLINE 0 0 0<br /> mirror-3 ONLINE 0 0 0<br /> N10 ONLINE 0 0 0<br /> N11 ONLINE 0 0 0<br /> spares<br /> draid2-0-0 AVAIL<br /> draid2-0-1 AVAIL<br /><br />On our newest systems, we have some 106-drive JBODs with 20TB drives and<br />in order to reduce the chance of multiple disk failures in a single<br />draid device, we reconfigured the pools to have 2 draid devices per<br />pool, though still one OST per pool and one OST per OSS. In this config<br />we only have one distributed spare per draid. Due to significant write<br />performance reasons we also (reluctantly) started spanning pools across<br />2 JBODs. An additional difference is we had much less NVMe capacity on<br />these systems with just one small pair of NVMe drives per enclosure, so<br />we configure them as special devices for pool metadata rather than for<br />small block storage. The config for one of those pools looks like the<br />following:<br /><br /> NAME STATE READ WRITE CKSUM<br /> merced239 ONLINE 0 0 0<br /> draid2:11d:53c:1s-0 ONLINE 0 0 0<br /> L0 ONLINE 0 0 0<br /> L2 ONLINE 0 0 0<br /> L4 ONLINE 0 0 0<br />...<br /> L100 ONLINE 0 0 0<br /> L102 ONLINE 0 0 0<br /> L104 ONLINE 0 0 0<br /> draid2:11d:53c:1s-1 ONLINE 0 0 0<br /> U1 ONLINE 0 0 0<br /> U3 ONLINE 0 0 0<br /> U5 ONLINE 0 0 0<br />...<br /> U101 ONLINE 0 0 0<br /> U103 ONLINE 0 0 0<br /> U105 ONLINE 0 0 0<br /> special <br /> mirror-2 ONLINE 0 0 0<br /> N2 ONLINE 0 0 0<br /> N3 ONLINE 0 0 0<br /> spares<br /> draid2-0-0 AVAIL<br /> draid2-1-0 AVAIL<br /><br />Hope this helps,<br />Cameron<br /><br />On 2/6/25 11:29 AM, Nehring, Shane R [ITS] wrote:<br />> Hello All,<br />><br />> I didn't want to hijack the other thread today about draid, but I have been<br />> meaning to ask questions about it and folks' experience with it in the context<br />> of Lustre. Most of my questions come from not having a chance to really play<br />> around with draid much.<br />><br />> Have you been generally satisfied with performance of a single draid vdev vs<br />> either multiple pools/osts per node or single osts on a pool spanning multiple<br />> raidz(2) vdev members? Is random io comparable to a span of raidz2 vdevs? I know<br />> one of the pain points (more from a space usage perspective as I understand it)<br />> is the fixed stripe width and how that impacts small files, but does small file<br />> io perform particularly badly on draid vs a span raidz2?<br />><br />> I've got hardware on order (a couple 60 bay jbods and heads) that's going to<br />> replace some of the older OSTs in our current volume and I'm leaning toward a<br />> single draid pool OST per OSS. I plan to do some benchmarking of the pools in<br />> various configurations, but it's hard to generate a benchmark that's actually<br />> representative of real world usage.<br />><br />> If you've got any insights or anecdotes regarding your experience with draid and<br />> Lustre I'd love to hear them!<br />><br />> Thanks,<br />> Shane<br />><br />> _______________________________________________<br />> lustre-discuss mailing list<br />> lustre-discuss@lists.lustre.org<br />> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org<br />-------------- next part --------------<br />An HTML attachment was scrubbed...<br />URL:<br /><br />------------------------------<br /><br />Subject: Digest Footer<br /><br />_______________________________________________<br />lustre-discuss mailing list<br />lustre-discuss@lists.lustre.org<br />http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org<br /><br /><br />------------------------------<br /><br />End of lustre-discuss Digest, Vol 227, Issue 4<br />**********************************************
</article><div><br /></div>