[Lustre-discuss] Slow read performance across OSSes

[James Robnett]

   A bit more (likely not useful) info on this odd read performance
problem across multiple OSSes.

   We have a different prototype Lustre installation in another
group.  The hardware is a bit different but is also a RHEL 5.3
based system installed from RPM's (v1.8). That system also sends
traffic through a Cisco Catalyst 2960 48port gigabit switch.

   We see the same odd performance issue on that Lustre system.
Reads limited to a single OSS are client network limited, reads that
involve more than one OSS are capped around 40MB/s.

   Considering nobody else has reported this it seems like it
must be either some oddity in our base RHEL 5.3 install or
some oddity in that Cisco switch.

    The switch has a 32gbit/s backplane and otherwise seems
perfectly capable of handling traffic from multiple clients at
much faster rates than we're seeing.  Nor do I see any evidence
on the OSSes or switch that the switch becomes congested trying
to direct multiple read streams to the client's 1gbit interface.

   It's quite possible I'm looking for the wrong thing there,
suggestions welcome.

   I'm more inclined to believe it's our base OS install.  It
continues to appear that the experienced read rate is driven by
the request rate from the client and not some reply bottleneck.
The client simply isn't requesting reads at full speed if multiple
OSSes are involved.  There's some evidence that this other Lustre
system used to get more typical read/write rates.  That suggests
some subsequent RHEL 5.3 patch that's affecting the performance.

   BTW,  I've now tried bonnie++, iozone and IOR, all give similar
results so that rules out some bonnie++ pathology.

James Robnett
NRAO/NM

   8 OST's across all 4 OSSes

[root at casa-dev-13 C]# lfs getstripe /lustre/casa-store/IOR/
OBDS:
0: lustre-OST0000_UUID ACTIVE
1: lustre-OST0001_UUID ACTIVE
2: lustre-OST0002_UUID ACTIVE
3: lustre-OST0003_UUID ACTIVE
4: lustre-OST0004_UUID ACTIVE
5: lustre-OST0005_UUID ACTIVE
6: lustre-OST0006_UUID ACTIVE
7: lustre-OST0007_UUID ACTIVE
/lustre/casa-store/IOR/
stripe_count: -1 stripe_size: 4194304 stripe_offset: 0

Run began: Tue Oct 20 10:39:22 2009
Command line used: ./IOR -r -w -b 64K -t 64K -s 270000 -N 1 -o 
/lustre/casa-store/IOR/junkMachine: Linux casa-dev-13

Summary:
         api                = POSIX
         test filename      = /lustre/casa-store/IOR/junk
         access             = single-shared-file
         ordering in a file = sequential offsets
         ordering inter file= no tasks offsets
         clients            = 1 (1 per node)
         repetitions        = 1
         xfersize           = 65536 bytes
         blocksize          = 65536 bytes
         aggregate filesize = 16.48 GiB

Operation  Max (MiB)  Min (MiB)  Mean (MiB)   Std Dev  Max (OPs)  Min 
(OPs)  Mean (OPs)   Std Dev  Mean (s)Op grep #Tasks tPN reps  fPP reord 
reordoff reordrand seed segcnt blksiz xsize aggsize
---------  ---------  ---------  ----------   -------  --------- 
---------  ----------   -------   -------
write         116.01     116.01      116.01      0.00        0.01 
0.01        0.01      0.00 145.46395   1 1 1 0 0 1 0 0 270000 65536 
65536 17694720000 -1 POSIX EXCEL
read           44.29      44.29       44.29      0.00        0.00 
0.00        0.00      0.00 381.01059   1 1 1 0 0 1 0 0 270000 65536 
65536 17694720000 -1 POSIX EXCEL

Max Write: 116.01 MiB/sec (121.64 MB/sec)
Max Read:  44.29 MiB/sec (46.44 MB/sec)


   2 OSTs (0000 and 0001 on OSS-1, confirmed via iostat on OSS-1).

lfs getstripe /lustre/casa-store/IOR-OST-0-1/
OBDS:
0: lustre-OST0000_UUID ACTIVE
1: lustre-OST0001_UUID ACTIVE
2: lustre-OST0002_UUID ACTIVE
3: lustre-OST0003_UUID ACTIVE
4: lustre-OST0004_UUID ACTIVE
5: lustre-OST0005_UUID ACTIVE
6: lustre-OST0006_UUID ACTIVE
7: lustre-OST0007_UUID ACTIVE
/lustre/casa-store/IOR-OST-0-1/
stripe_count: 2 stripe_size: 4194304 stripe_offset: 0

Run began: Tue Oct 20 10:31:32 2009
Command line used: ./IOR -r -w -b 64K -t 64K -s 270000 -N 1 -o 
/lustre/casa-store/IOR-OST-0-1/junkMachine: Linux casa-dev-13

Summary:
         api                = POSIX
         test filename      = /lustre/casa-store/IOR-OST-0-1/junk
         access             = single-shared-file
         ordering in a file = sequential offsets
         ordering inter file= no tasks offsets
         clients            = 1 (1 per node)
         repetitions        = 1
         xfersize           = 65536 bytes
         blocksize          = 65536 bytes
         aggregate filesize = 16.48 GiB

Operation  Max (MiB)  Min (MiB)  Mean (MiB)   Std Dev  Max (OPs)  Min 
(OPs)  Mean (OPs)   Std Dev  Mean (s)Op grep #Tasks tPN reps  fPP reord 
reordoff reordrand seed segcnt blksiz xsize aggsize
---------  ---------  ---------  ----------   -------  --------- 
---------  ----------   -------   -------
write         118.45     118.45      118.45      0.00        0.01 
0.01        0.01      0.00 142.46609   1 1 1 0 0 1 0 0 270000 65536 
65536 17694720000 -1 POSIX EXCEL
read          117.64     117.64      117.64      0.00        0.01 
0.01        0.01      0.00 143.44616   1 1 1 0 0 1 0 0 270000 65536 
65536 17694720000 -1 POSIX EXCEL

Max Write: 118.45 MiB/sec (124.20 MB/sec)
Max Read:  117.64 MiB/sec (123.35 MB/sec)