[Lustre-discuss] OST selection considering load and qos_threshold_rr

Yujun Wu yujun at phys.ufl.edu
Thu May 28 11:58:38 PDT 2009


Hello Andreas,

Thanks a lot for your detailed info.

On Thu, 28 May 2009, Andreas Dilger wrote:

> On May 28, 2009  09:40 -0400, Yujun Wu wrote:
> > I am trying to watch over how the OSTs are selected for 
> > putting files on. One thing I find is that, the files
> > can be put on an OST although it is already busy handling
> > other files while some other OSTs are idling. Does Lustre 
> > select an OST considering its current load or not?
> 
> No, currently there are only two modes for OST selection:
> - round robin, used most of the time because it distributes the
>   space usage/load best in most situations
> - space-weighted random selection that selects OSTs with more free
>   space more often than OSTs with less free space
> 
> Per bug 18547 we would like to change the space-weighted random OST
> selection with a weighted round-robin algorithm to avoid the random
> collisions (due to birthday paradox) that result in the same OST being
> selected more often than it should.
Yes, it will be good to avoid this.
> 
> Once we have a better weighted round-robin OST selector we can add in
> other parameters such as throughput, iops rate, etc.  There are reserved
> fields in the obd_statfs structure that can be used to pass a number of
> performance metrics from the OST to the MDS.

Looking forward to seeing features like this. 
 
> The danger of using the _current_ IO load for the OST object selection
> is that this information may be somewhat stale by the time the MDS does
> OST selection and it might result in poor load balancing.  The MDS only
> gets updated statfs information every 5s at most.  At a minimum the
> MDS should estimate the load it is placing on each OST as it allocates
> objects there so that its calculations don't become completely irrelevant
> after the first few allocations.
I think some study is needed on how to use some dynamic info like
"_current_ IO load". It is not necessarily update all the time. I think it
could be useful to use this info to avoid some OST becomes too heavily
used while others are idling.
> 
> > I looked at the Lustre document and thought I can do
> > some adjustment with qos_threshold_rr at least with Round-Robin(RR):
> > 
> > http://manual.lustre.org/manual/LustreManual16_HTML/LustreProc.html#50401407_pgfId-1290499
> > 
> > But I couldn't find qos_threshold_rr proc tunable on the Lustre MDS I have
> > installed. The version of my installation is 1.6.7. Am I missing
> > something or it has changed?
> 
> Per bug 18334 this patch was not landed in the 1.6.7 release, only for
> 1.6.8 and 1.8.0.  This should probably be made clear in the 1.6 manual.
Thanks for your info. Maybe I should try a later version of Lustre.


Regards,
Yujun
> 
> Cheers, Andreas
> --
> Andreas Dilger
> Sr. Staff Engineer, Lustre Group
> Sun Microsystems of Canada, Inc.
> 
> 




More information about the lustre-discuss mailing list