[Lustre-discuss] IB storage as an OST target

David Dillow dillowda at ornl.gov
Thu Mar 31 08:02:07 PDT 2011


On Thu, 2011-03-31 at 10:19 -0400, chas williams - CONTRACTOR wrote:
> On Wed, 30 Mar 2011 12:00:04 -0400
> David Dillow <dave at thedillows.org> wrote:
> 
> > While it is true both are full duplex, there are also setup messages
> > flowing in both directions to set up the large transfers. In the past,
> > we've certainly seen problems at scale with small messages getting
> > blocked behind large bulk traffic on LNET. It would be interesting to
> > see how much self-interference is generated when running storage over
> > the same HCA as LNET, versus having them on separate NICs -- especially
> 
> i kind of gather that if your clients are doing a mix of i/o (reads and
> writes) that this is going to happen regardless.  two hcas/ports get a
> sort of parallelism to help alleviate some of this congestion.  but a
> faster port (say 40G) would just as helpful?

That makes sense, but then if we we're doing QDR ports, we'd probably
matching that with QDR on the storage as well. If you have DDR on the
storage, then using a single QDR HCA would probably work; I'd still want
to test a bit before running with it to be sure.

-- 
Dave Dillow
National Center for Computational Science
Oak Ridge National Laboratory
(865) 241-6602 office





More information about the lustre-discuss mailing list