[Lustre-discuss] Infiniband hot spot avoidance with LMC>0

Nathan Dauchy Nathan.Dauchy at noaa.gov
Tue Apr 28 11:51:43 PDT 2009


Isaac Huang wrote:
> On Mon, Apr 27, 2009 at 12:21:41PM -0600, Nathan Dauchy wrote:
>> Greetings,
>>
>> Does Lustre's o2ib LND take advantage of Infiniband's LID Mask Count
>> (LMC) capability?  Might it be included in the future?  I'm looking for
>> something similar to the "MV2_USE_HSAM=1" option for Hot-Spot Avoidance
>> with MVAPICH2.
> 
> Nothing like this exists so far in the o2iblnd. Currently between any
> two ports there's only one QP which uses the LIDs as returned by the
> SM.

OK, thanks for confirming, Isaac.

> The MVAPICH2 seems to be striping outgoing data over multiple paths
> and adjusting path weights dynamically based on perceived speed of the
> paths. We'd be interested to take a look if there's a high-level
> description of the mechanism.
> 

The best high-level document I can find is a presentation: "Hot-Spot
Avoidance with Multi-Pathing over InfiniBand: An MPI Perspective", by
           Abhinav Vishnu, Matthew Koop, et. al.
http://nowlab.cse.ohio-state.edu/publications/conf-presentations/2007/vishnu-ccgrid07.pdf

I imagine the Ohio State guys would be happy to work with you.

And, if Sun's code licensing plans allow for it, you can always look at
the MVAPICH code. :)

-Nathan



More information about the lustre-discuss mailing list