[Lustre-discuss] Lustre NFs expport

[Oleg Drokin]

Hello!

On Dec 13, 2007, at 7:50 PM, Mertol Ozyoney wrote:

> Is there any experiance about exporting Lustre via NFS to outside  
> world? Any whitepapers, how to guides, schemas etc..

There are some people who do this, also HP support this with their  
Lustre spin-off and I think they are one of the biggest
users at the moment.
Their latest tests indicate that lustre reexported with NFS achieves  
around 90% of the lustre link speed (this was tested with
gigabit connections, but I see nothing that would prevent us from this  
on higher connection speeds provided there is enough CPU
and bus bandwidth). These results were achieved with 2.6 kernel on  
(2.4 kernels perform significantly worse, like 2x worse
due to very small i/o submitted to lustre). To obtain this sort of  
speed special patch (bug 13371 in our bugzilla) was used
on lustre 1.4.11, that is not part of released lustre yet, though it  
will likely be included into 1.4.12.
Port of the patch is also underway for 1.6 lustre releases.
Metadata speed is mostly around 1/2 or 2/3 compared to NFS exported  
off ext3 if lustre transport is tcp. For lower-latency
network (like infiniband) we do better, but I do not have any numbers  
at hand.
Exporting lustre via NFS is not unlike any other filesystem, you just  
write /etc/exports entry for fs you want to export and
you are mostly done (for some older releases you also need to specify  
fsid parameter).
Some people (HP) also export one lustre filesystem via several NFS  
servers (to many clients) that allows to achieve better
speed when lustre link is significantly faster than NFS side link  
(also for load-balancing?), for this case normally you need to have
identical fsid= setting in your /etc/exports, though current 1.6  
release eliminate this requirement. Also in this case you cannot
have consistent posix locking unless you use 2.6.22 kernel or above.  
(also sles9 kernels have special patch that allows this.
The patch was dropped in sles10, though).

Bye,
     Oleg