[Lustre-discuss] Slow read performance across OSSes

[James Robnett]

   After reading through my first post I felt some clarification was
probably warranted.

   In this test setup there are two OSS, call them OSS-1 and OSS-2,
each has an OST, call them OSS-1-A, OSS-1-B and OSS-2-A, OSS-2-B.

   The MDS, OSSes and client all have 1Gbit ethernet connections.

   The following table illustrates the data rates I see in MB/s.

OST(s)                                 Read    Write
OSS-1-A                                 113      95
OSS-1-B                                 112      93
OSS-1-A OSS-1-B                         112      98
OSS-2-A                                 105      93
OSS-2-B                                 115      94
OSS-2-A OSS-2-B                         115      98
OSS-1-B OSS-2-A                     ---> 42     113
OSS-1-A OSS-2-B                     ---> 42     114
OSS-1-A OSS-1-B OSS-2-A OSS-2-B     ---> 46     114

   The write numbers are almost exactly what I'd expect across
1Gbit.  96MB/s or so between the client and a single OSS and
nearly full rate (112MB/s) with two OSSes.

   The 113MB/s read numbers for a single OSS (one or more OST's) are
also pretty much exactly what I'd expect.  It's the 40MB/s reads
when utilizing 2 OSSes that are throwing me.

   I can envision that there would be more re-assembly overhead on
the client in the case of 2 OSSes(1) but I'm surprised it's that high.

   Is this an expected result ?

   If it's unexpected is there a common misconfiguration or client
short coming that causes it to be slower when reading from multiple
OSSes ?

   Is there some command I could run or data I could provide that would
help identify the issue ?  I'm fairly new to Lustre so I'm just as
likely to flood noise as signal if I just randomly appended data
beyond raw rates.

   I just upgraded to 1.8.1.1 which had no effect.

James Robnett
NRAO/NM

1) I'm assuming in the case of a single OSS with 2 OSTs the OSS
presents the client with a single stream.  If assembly of two data
streams is required on the client in both the single and dual OSS
(both with 2 OSTs) cases then I'm even more confused about those
results.

James Robnett wrote:
>    The nodes are a bit cobbled together from what I had handy.
>
> One MDS: Dual quad-core 2.5GHz nehalem 8GB RAM  E1000 gigabit NIC
>           MDT is just a partition on a 1TB SAS Seagate
> Two OSS: Single dual core 2.8GHz Xeon, 4GB RAM single gigabit NIC
>           Dual 3ware 9550SX cards with 7+1 RAID 5 across 400GB WD SATA
>           drives.
> Two OST/OSS: 2TB. Configured as LVM.  1 and 4MB stripe size tried.
> Client:  Dual quad-core 2.5 GHz Xeon, 8GB RAM single gigabit NIC
> Network:  Dedicated Cisco 2960g Gigabit switch