[Lustre-discuss] Line rate performance for clients

[Andreas Dilger]

On 2011-07-29, at 11:33 AM, Brock Palen wrote:
> I think this is a networking question.
> 
> We have lustre 1.8 clients with 1gig-e interfaces that according to ethtool are running full duplex.
> 
> If I do the following:
> 
> cp /lustre/largeilfe.h5 /tmp/
> 
> I get 117MB/s
> 
> If I then use globus-url-copy to move that file from /tmp/ to -> remove tape archive I get 117MB/s
> 
> If I go directly from  /lustre -> archive  I get 50MB/s,  

Strace your globus-url-copy and see what IO size it is using.  "cp" has long ago been modified to use the blocksize reported by stat(2) for copying, and Lustre reports a 2MB IO size for striped files (1MB for unstriped).  If your globus tool is using e.g. 4kB reads then it will be very inefficient for Lustre, but much less so than from /tmp.

> this is consistently reproducible.  It doesn't mater if I just copy a large file on lustre to lustre,  or scp, or globus.  If I try to ingest and outgest data I get what looks like half duplex performance. 
> 
> Anyone have ideas why I cannot do 1Gig-e full duplex?

I don't think this has anything to do with "full duplex".  117MB/s is pretty much  the maximum line rate for GigE (and pretty good for Lustre, if I do say so myself) in one direction.  There is presumably no data moving in the other direction at that time.

Cheers, Andreas
--
Andreas Dilger 
Principal Engineer
Whamcloud, Inc.