<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body>
<br>
Neil,<br>
<br>
We do indeed have such functionality (it’s called DVS and it’s basically a high speed file system projection framework, ala NFS but faster), so the ability to build lnet separately is valuable to us. While it is being open sourced under the GPL, I don’t think
there’s any intention to try to upstream it. The current code isn’t even usable off of Cray systems as it depends on info from user space (that is provided, in the end, from Cray proprietary hardware) to keep its connection/routing tables up to date. That’s
supposedly in the pipeline to get fixed, but it’s still pretty far from generally usable.<br>
<br>
But we’d still really appreciate it if lnet stayed separate. Don’t know if that’s enough for you - I know sometimes *small* stuff is done for out of tree users. Hopefully this meets that standard.<br>
<br>
- Patrick<br>
<br>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Andreas Dilger <adilger@whamcloud.com><br>
<b>Sent:</b> Wednesday, June 27, 2018 6:01:27 AM<br>
<b>To:</b> NeilBrown; James Simmons<br>
<b>Cc:</b> Patrick Farrell; Oleg Drokin; Lustre Development List<br>
<b>Subject:</b> Re: [lustre-devel] [PATCH 00/24] lustre - more cleanups including module reduction.</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt;">
<div class="PlainText">On Jun 27, 2018, at 05:08, NeilBrown <neilb@suse.com> wrote:<br>
> <br>
> On Tue, Jun 26 2018, Patrick Farrell wrote:<br>
> <br>
>> Ah, sorry, lost your mail in the shuffle, Neil.<br>
>> <br>
>> The name, mostly. ptlrpc is one subdirectory and one subsystem, so I don’t want to use it for a module that explicitly includes all of several. I’m not sure of a better name immediately - is this module intended to be shared between client and server?<br>
>> <br>
> <br>
> Thanks for clarifying.<br>
> Note that I'm still feeling my way around here so "intended" can at most<br>
> be a soft intention, open to change. But yes, I expect this module<br>
> would ultimately be shared with the server.<br>
> <br>
> I think things should only be in different modules if it might sometimes<br>
> make sense to use one without the other.<br>
> <br>
> So everything that is always used for a lustre client mount, and is only<br>
> used for a lustre client mount, should be in a module called "lustre".<br>
> And everything always and only used for a lustre server should be in a<br>
> module called (something like) "lustre-server".<br>
> <br>
> It appears to me that ptlrpc, ldlm, and obdclass are always and only<br>
> used together, but can be used for client or server. So I think they<br>
> should be together in a module.<br>
> <br>
> I'm tempted to add lnet to that list. There has been mention that Cray<br>
> has some other functionality that uses lnet - if that doesn't use<br>
> ptlrpc, then maybe that is a case for keeping lnet separate. However I<br>
> would rather see it as a potential case to separate lnet from the others<br>
> at a later date if that functionality from cray ever goes into mainline.<br>
> <br>
> The klnds modules can presumably be used independently(?) so they can<br>
> sensibly remain as separate modules, though I'm beginning to wonder if<br>
> lnet can actually function without socklnd, so I wonder if that should<br>
> be permanently part of the lnet module (with NFS, the xprt_sock code is<br>
> a permanent part of "sunrpc", while the xprt_rdma (aka rpcrdma) code<br>
> can be a separate module).<br>
<br>
Definitely LNet can run without ksocklnd, and it does so on clients that<br>
only have IB connections. Lots of HPC systems have client nodes that do<br>
not have Ethernet interfaces, since it is a cabling/networking nightmare<br>
to add a few thousand additional Ethernet cables/ports in addition to the<br>
IB cables/ports, and it is always possible to run IPoIB for TCP traffic.<br>
<br>
> In the interests of a concrete strawman, what objections would I get if<br>
> I suggested that ptlrpc, ldlm, obdclass, lnet, and socklnd were all included in<br>
> the one module named "lnet" ??<br>
<br>
This would break all of the module parameters for ptlrpc. There are only<br>
a few obdclass module parameters that are rarely used, so I'm less worried<br>
about those ones.<br>
<br>
Given that lnet is a separate thing, I'd prefer to keep "ptlrpc" and "lnet"<br>
as separate modules, then "lustre" for the remainder of the client code.<br>
<br>
The "ptlrpc" module should also include the "fld", "fid", "quota", "mgc"<br>
modules, since these are shared with the server as well.<br>
<br>
Server modules:<br>
[adilger@mookie ~]$ lsmod | grep ^obdclass<br>
obdclass 1731964 83 mdd,lod,mdt,osp,ofd,lfsck,ost,mgs,mgc,osd_ldiskfs,fid,fld,lquota,ptlrpc<br>
<br>
Client modules:<br>
[adilger@twoshoes ~]$ lsmod | grep ^obdclass<br>
obdclass 1568040 86 osc,mgc,lustre,lov,mdc,fid,lmv,fld,ptlrpc<br>
<br>
<br>
James, if we are going to move in the direction of having a separate<br>
"lustre-server" module, it would make sense to land a patch to master<br>
now that adds both a module alias, as well as a filesystem type alias<br>
for lustre-server, so that we can start moving systems over to using<br>
"lustre-server" as the filesystem type in /etc/fstab or HA mounting<br>
scripts, and it will auto-load the module at the same time. This might<br>
also allow the Lustre mount code to be simplified, since I think it has<br>
some messy code to have the same code, but distinguish between client<br>
and server mounts.<br>
<br>
Cheers, Andreas<br>
---<br>
Andreas Dilger<br>
Principal Lustre Architect<br>
Whamcloud<br>
<br>
<br>
<br>
<br>
<br>
<br>
</div>
</span></font></div>
</body>
</html>