[Lustre-discuss] InfiniBand QoS with Lustre ko2iblnd.

Sébastien Buisson sebastien.buisson at bull.net
Mon Jun 22 07:49:03 PDT 2009


Hi all,

We have been thinking about this IB QoS thing in Lustre for a while, and 
we would like to express a need that may not be satisfied by the current 
solution exposed by Isaac (which consists in using the ko2iblnd 
'service' option).

Let's consider we have two sets of OSSes, each set serving a different 
Lustre file system (i.e. all the OSTs of an OSS are part of the same 
Lustre file system). The same Lustre clients have access to both 
filesystems.
In these conditions, how can we enforce different IB QoS in Lustre for 
the 2 file systems?
- by using the ko2iblnd 'service' option, the o2iblnd SL would be the 
same for all connections initiated by a given Lustre client, regardless 
the destination file system. So we would not achieve our goal.
Unless what really matters is the SL of the connections created by the 
servers (I think I have seen in the Lustre debug logs that the 'real' 
data transfers are always done via the servers connections).
What do you think?
- if the 'service id' information was stored on the MGS on a file system 
basis, one could imagine to retrieve it at mount time on the clients. 
The 'service id' information stored on the MGS could consist in a port 
space and a port id. Thus it would be possible to affect different 
service ports to the various connections initiated by the client, 
depending on the target file system.
What do you think? Would you say this is feasible, or can you see major 
issues with this proposal?


Thanks in advance.
Sebastien.


Isaac Huang a écrit :
> On Mon, May 18, 2009 at 12:04:37PM +0200, Daniel Kobras wrote:
>> Hi!
>>
>> Does anyone know how to use QoS with Lustre's o2ib LND? The Voltaire IB
>> LND allowed to #define a service level, but I couldn't find a similar
>> facility in o2ib. Is there a different way to apply QoS rules?
> 
> The o2iblnd SL is set by the OFED RDMA CM, indirectly based on the
> o2iblnd service port (set via ko2iblnd option 'service', 987 by
> default) and its port space (RDMA_PS_TCP). For a complete, and more
> complicated story, please see:
> https://bugzilla.lustre.org/show_bug.cgi?id=18360#c2
> 
> Isaac
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
> 
> 



More information about the lustre-discuss mailing list