[Lustre-discuss] InfiniBand QoS with Lustre ko2iblnd.

[Sébastien Buisson]

Sébastien Buisson a écrit :
> Hi all,
> 
> We have been thinking about this IB QoS thing in Lustre for a while, and 
> we would like to express a need that may not be satisfied by the current 
> solution exposed by Isaac (which consists in using the ko2iblnd 
> 'service' option).
> 
> Let's consider we have two sets of OSSes, each set serving a different 
> Lustre file system (i.e. all the OSTs of an OSS are part of the same 
> Lustre file system). The same Lustre clients have access to both 
> filesystems.
> In these conditions, how can we enforce different IB QoS in Lustre for 
> the 2 file systems?
> - by using the ko2iblnd 'service' option, the o2iblnd SL would be the 
> same for all connections initiated by a given Lustre client, regardless 
> the destination file system. So we would not achieve our goal.
> Unless what really matters is the SL of the connections created by the 
> servers (I think I have seen in the Lustre debug logs that the 'real' 
> data transfers are always done via the servers connections).
> What do you think?

I have tried to make communicate a client for which I set the ko2iblnd 
'service' option to 986, with a server for which I set the ko2iblnd 
'service' option to 987: it does not work.
This is not surprising because the ko2iblnd 'service' parameter is used 
  on the client side in the kiblnd_connect_peer function to designate to 
port of the remote peer (the server in this case).
So, the ko2iblnd 'service' option must be the same for all the nodes 
participating in the same file system.

In our case where the same clients access both file systems, it means 
that we will not be able to set different o2iblnd SLs for the two file 
systems.


> - if the 'service id' information was stored on the MGS on a file system 
> basis, one could imagine to retrieve it at mount time on the clients. 
> The 'service id' information stored on the MGS could consist in a port 
> space and a port id. Thus it would be possible to affect different 
> service ports to the various connections initiated by the client, 
> depending on the target file system.
> What do you think? Would you say this is feasible, or can you see major 
> issues with this proposal?
> 

The peer's port information could be stored in the kib_peer_t structure. 
That way, it would be possible to make clients connect to servers which 
listen on different ports.
What do you think?


> 
> Thanks in advance.
> Sebastien.
> 
> 
> Isaac Huang a écrit :
>> On Mon, May 18, 2009 at 12:04:37PM +0200, Daniel Kobras wrote:
>>> Hi!
>>>
>>> Does anyone know how to use QoS with Lustre's o2ib LND? The Voltaire IB
>>> LND allowed to #define a service level, but I couldn't find a similar
>>> facility in o2ib. Is there a different way to apply QoS rules?
>> The o2iblnd SL is set by the OFED RDMA CM, indirectly based on the
>> o2iblnd service port (set via ko2iblnd option 'service', 987 by
>> default) and its port space (RDMA_PS_TCP). For a complete, and more
>> complicated story, please see:
>> https://bugzilla.lustre.org/show_bug.cgi?id=18360#c2
>>
>> Isaac
>> _______________________________________________
>> Lustre-discuss mailing list
>> Lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>>
>>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
> 
>