[Lustre-discuss] Balancing I/O Load

[Tom.Wang]

Charles Taylor wrote:
> We are seeing some disturbing (probably due to our ignorance)  
> behavior from lustre 1.6.3 right now.     We have 8 OSSs with 3 OSTs  
> per OSS (24 physical LUNs).   We just created a brand new lustre file  
> system across this configuration using the default mkfs.lustre  
> formatting options.    We have this file system mounted across 400  
> clients.
>
> At the moment, we have 63 IOzone threads running on roughly 60  
> different clients.    The balance among the OSSs is terrible and  
> within each OSS, the balance across the OSTs (luns) is even worse.     
> We have one OSS with a load of 100 and another that is not being  
> touched.    On several of the OSSs, only one OST (luns) is being used  
> while the other two are ignored entirely.
>
> This is really just a bnuch of random I/O (both large and small  
> block) from a bunch of random clients (as will occur in real-life)  
> and our lustre implementation is not making very good use of the  
> available resources.   Can this be tuned?    What are we doing  
> wrong?    The 1.6 operations manual (version 1.9) does not say a lot  
> about options for balancing the work load among OSSs/OSTs.     
> Shouldn't lustre be doing a better job (by default) of distributing  
> the workload?
>   
Actually, lustre 1.6 of course consider balancing ost load. When it create
the objects, it will choose the OST by its usages and also  try to  
choose the ost
from different OSS.  If the ost usages is about the same, lustre choose 
the OST
by rr policy.  I think  you  should  check  your  ost usage. And also
use lfs find to get the files located in that heave load OSS and copy 
them to redistribute that.

Thanks
WangDi

 

> Charlie Taylor
> UF HPC Center
>
> FWIW, the servers are dual-processor, dual-core Opterons (275s) with  
> 4GB RAM each.   They are running CentOS 5 w/ a  
> 2.6.18-8.1.14.el5Lustre (patched lustre, smp kernel) and the deadline  
> I/O scheduler.   If it matters, our OSTs are atop LVM2 volumes (for  
> management).    The back-end storage is all Fibre-channel RAID  
> (Xyratex).    We have tuned the servers and know that we can get  
> roughly 500MB/s per server across a striped *local* file system. 
>    
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at clusterfs.com
> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
>