[Lustre-discuss] gridftp and Lustre

[Andreas Dilger]

On May 26, 2009  12:44 -0700, Jeffrey Bennett wrote:
> Is it technically possible to implement this without using the
> Lustre client on the OSS and accessing directly to the objects on the
> OSS? I guess this would imply the development of some sort of Lustre
> Client/Gridftp server all together in the same code. Not that I am going
> to implement it, but I am curious...

There is no such thing as "direct" access to the object.  What you are
describing is essentially just putting a GridFTP server accessing Lustre
via the Lustre client.  If you want to put a "DSI module" that just writes
to a mounted client filesystem on the OSS that is fine too, but I don't
see how this is significantly different than just running multiple GridFTP
servers, one on each OSS.

Yujun Wu <yujun at phys.ufl.edu> wrote:
> As the frontend and backend are separated, the security risk is
> not big. On the other hand, I have another question: if I install a
> gridftp server on an OSS, it will write to that OSS ONLY or data can go
> to other OSSs as well?

The client would be a normal client - it can access all of the OSS
storage in the Lustre filesystem.  While it might be possible to
create a DSI that knows the internal structure of the Lustre striping
and drive the traffic directly to a specific OST, this would require
a non-trivial amount of ongoing effort.  It would be much simpler to
just run multiple GridFTP servers on Lustre clients (whether local to
the OSS nodes or on standalone clients).

It would seem that even a small number of Lustre clients could accept
all of the GridFTP traffic that your WAN link could handle (10Gbit/s?)
The biggest obstacle to performance is the TCP data copy overhead,
which is probably independent of Lustre itself.

> > -----Original Message-----
> > From: Andreas.Dilger at sun.com [mailto:Andreas.Dilger at sun.com] 
> > On Behalf Of Andreas Dilger
> > Sent: Monday, May 25, 2009 4:14 PM
> > To: Jeffrey Bennett
> > Cc: Yujun Wu; lustre-discuss at lists.lustre.org
> > Subject: Re: [Lustre-discuss] gridftp and Lustre
> > 
> > On May 22, 2009  10:33 -0700, Jeffrey Bennett wrote:
> > > I guess you could implement some sort of gridftp plugin, a 
> > DSI module 
> > > is what they call it, to write data directly to the OSS and 
> > then you 
> > > could install gridftp servers on the OSS. This is basically 
> > what the 
> > > HPSS DSI module for Gridftp does, it writes data directly 
> > to the HPSS 
> > > system using the HPSS API. However, I am not sure it's technically 
> > > possible with Lustre.
> > 
> > It is possible to mount Lustre clients on the OSS nodes, if 
> > you are willing to make at least the GridFTP port accessible 
> > to the outside network on
> > those nodes.   That said, this exposes the back-end storage 
> > to attack by
> > a remote client, if the GridFTP server is not secure.
> > 
> > Cheers, Andreas
> > --
> > Andreas Dilger
> > Sr. Staff Engineer, Lustre Group
> > Sun Microsystems of Canada, Inc.
> > 
> > 

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.