[Lustre-discuss] bad 1.6.3 striped write performance

[Andrei Maslennikov]

On Nov 26, 2007 3:32 PM, Robin Humble <rjh+lustre at cita.utoronto.ca> wrote:

> >> I'm seeing what can only be described as dismal striped write
> >> performance from lustre 1.6.3 clients :-/
> >> 1.6.2 and 1.6.1 clients are fine. 1.6.4rc3 clients (from cvs a couple
> >> of days ago) are also terrible.

I have 3 OSTs capable to deliver 300+ MB/sec each for large streaming writes
with 1M blocksize. On one client, with one OST I can see almost all
this bandwidth
over Infiniband. If I run three processes in parallel on this very client, each
writing into a separate OST, I arrive to 520 MB/sec aggregate (3 streams at
approx 170+ MB/sec each).

If I try to stripe over these three OSTs on this client, performance of one
stream drops to 60+ MB/sec. Changing stripesize to a smaller one (1/3 MB)
makes things worse. Writing with larger block sizes (9M, 30M) does not improve
things. Increasing the stripesize to 25 MB allows to approach the speed
of a single OST, as one would expect (blocks are round robined over all three
OSTs). But never more. Zeroing checksums on the client does not help.

Will now be downgrading the client to 1.6.2 to see if this helps.

Andrei.