[Lustre-discuss] Gluster then DRBD now Lustre?

[Troy Benjegerdes]

On Mon, Jun 16, 2008 at 11:55:12AM -0700, Kilian CAVALOTTI wrote:
> On Monday 16 June 2008 11:40:40 am Andreas Dilger wrote:
> > > NYC == New York City?  What
> > > is SJC?
> >
> > SJC == San Jose, California
> 
> That's why I thought, but if so, the following part loses me: 
> 
> > This is working in a test setup, however there are some down sides.
> > The first is that DRBD only supports IP, so we have to run IPoIB over
> > our our infiniband adapters, not an ideal solution.
> 
> Nathan, you won't be able to use Infiniband between Ney Work City and 
> San Jose, CA, anyway, right? Even without considering IB cables' length 
> limitation, and unless you can use some kind of dedicated, 
> special-purpose link between your sites, the public Internet is not 
> really able to provide bandwidth nor latencies compatible with 
> Infiniband standards.
> 
> IP is probably your best bet, here, and DRBD would probably be an 
> appropriate candidate for this kind of job. Although, you probably 
> don't want your synchronization data unencrypted over the public pipes, 
> and you may need an extra VPN-ish layer to ensure data confidentiality.

If you have a dedicated gigabit link (no congestion), InfiniBand might
work pretty well. I've used the Obsidian Longbow IB WAN extenders, and
got better performance using IB than over TCP. I believe there is also a
version that does AES encryption as well.

Has anyone tried lustre over a 100ms latency IB wan link?

That being said, IB WAN stuff is pretty new, so if you want to try this,
it has some promise, but expect to do a lot of experimenting.