[Lustre-discuss] lnet router tuning

[Nic Henke]

On 09/10/2010 10:48 AM, Michael Kluge wrote:
> OK, IB back to back is at 1,2 GB/s, 10GE back to back at 950 MB/s, with
> additional lnet router I see 550 MB/s. Time for lnet tuning?
>

This smells like latency - you doubled the number of hops and lost ~50% 
of your performance. Is your later email correct for this test - a 
single client->router->server ?

I'd try adding higher concurrency or more clients with the same 
concurrenct to see if you can ameliorate some of that.

I'd also benchmark each link separately with higher concurrency to see 
what the limits are. You should be able to graph concurrency/client 
count on the X axis and see a nice smooth curve that flattens out to 
some maximum performance.

Cheers,
Nic

P.S to get more accurate performance numbers, you probably want to run 
'lst stat writers 5 & sleep 30'. You might want to take the middle 3 
data points and average them, discarding the first and last that could 
be subject to LST overhead and ramp up/down. FWIW, this is what I do 
when I'm benchmarking throughput rates.

Nic