[Lustre-discuss] Poor multithreaded I/O performance

Thu May 26 13:57:19 PDT 2011

Hi David, 
I am writing to an existing directory that has supposedly been configured on
8 OSTs. 

I have shown below the output of lfs getstripe on the directory and on the
output file generated by the program. 
It seems that the file is correctly striped across 8 OSTs. In one of the
previous emails, Wojciech suggested I make sure that the OSTs belong to
different OSSes using the OST pools feature. Can someone suggest how I can
verify that an existing directory is configured on OSTs belonging to
different OSSes (though I have a hint that that's not the problem, since
writing to separate files on the same directory does give me ~700Mbytes/sec)

lfs getstripe ../../ss_8/ --verbose
----------------------------------------------------------------------------
-------------------------------------
OBDS:
0: fastfs-OST0000_UUID ACTIVE
1: fastfs-OST0001_UUID ACTIVE
2: fastfs-OST0002_UUID ACTIVE
3: fastfs-OST0003_UUID ACTIVE
4: fastfs-OST0004_UUID ACTIVE
5: fastfs-OST0005_UUID ACTIVE
6: fastfs-OST0006_UUID ACTIVE
7: fastfs-OST0007_UUID ACTIVE
8: fastfs-OST0008_UUID ACTIVE
9: fastfs-OST0009_UUID ACTIVE
10: fastfs-OST000a_UUID ACTIVE
11: fastfs-OST000b_UUID ACTIVE
12: fastfs-OST000c_UUID ACTIVE
13: fastfs-OST000d_UUID ACTIVE
14: fastfs-OST000e_UUID ACTIVE
15: fastfs-OST000f_UUID ACTIVE
16: fastfs-OST0010_UUID ACTIVE
17: fastfs-OST0011_UUID ACTIVE
18: fastfs-OST0012_UUID ACTIVE
19: fastfs-OST0013_UUID ACTIVE
20: fastfs-OST0014_UUID ACTIVE
21: fastfs-OST0015_UUID ACTIVE
22: fastfs-OST0016_UUID ACTIVE
23: fastfs-OST0017_UUID ACTIVE
24: fastfs-OST0018_UUID ACTIVE
25: fastfs-OST0019_UUID ACTIVE
26: fastfs-OST001a_UUID ACTIVE
27: fastfs-OST001b_UUID ACTIVE
28: fastfs-OST001c_UUID ACTIVE
29: fastfs-OST001d_UUID ACTIVE
30: fastfs-OST001e_UUID ACTIVE
31: fastfs-OST001f_UUID ACTIVE
32: fastfs-OST0020_UUID ACTIVE
33: fastfs-OST0021_UUID ACTIVE
34: fastfs-OST0022_UUID ACTIVE
35: fastfs-OST0023_UUID ACTIVE
36: fastfs-OST0024_UUID ACTIVE
37: fastfs-OST0025_UUID ACTIVE
38: fastfs-OST0026_UUID ACTIVE
39: fastfs-OST0027_UUID ACTIVE
40: fastfs-OST0028_UUID ACTIVE
41: fastfs-OST0029_UUID ACTIVE
42: fastfs-OST002a_UUID ACTIVE
43: fastfs-OST002b_UUID ACTIVE
44: fastfs-OST002c_UUID ACTIVE
45: fastfs-OST002d_UUID ACTIVE
46: fastfs-OST002e_UUID ACTIVE
47: fastfs-OST002f_UUID ACTIVE
48: fastfs-OST0030_UUID ACTIVE
49: fastfs-OST0031_UUID ACTIVE
50: fastfs-OST0032_UUID ACTIVE
51: fastfs-OST0033_UUID ACTIVE
52: fastfs-OST0034_UUID ACTIVE
53: fastfs-OST0035_UUID ACTIVE
54: fastfs-OST0036_UUID ACTIVE
55: fastfs-OST0037_UUID ACTIVE
56: fastfs-OST0038_UUID ACTIVE
57: fastfs-OST0039_UUID ACTIVE
58: fastfs-OST003a_UUID ACTIVE
59: fastfs-OST003b_UUID ACTIVE
60: fastfs-OST003c_UUID ACTIVE
61: fastfs-OST003d_UUID ACTIVE
62: fastfs-OST003e_UUID ACTIVE
63: fastfs-OST003f_UUID ACTIVE
../../ss_8
stripe_count: 8 stripe_size: 0 stripe_offset: -1
----------------------------------------------------------------------------
------------------------------------

Running lfs getstripe on the 16Gbyte file generated by the program shows
this:
----------------------------------------------------------------------------
------------------------------------
lfs getstripe ../../ss_8/kmtest.txt --verbose
OBDS:
0: fastfs-OST0000_UUID ACTIVE
1: fastfs-OST0001_UUID ACTIVE
2: fastfs-OST0002_UUID ACTIVE
3: fastfs-OST0003_UUID ACTIVE
4: fastfs-OST0004_UUID ACTIVE
5: fastfs-OST0005_UUID ACTIVE
6: fastfs-OST0006_UUID ACTIVE
7: fastfs-OST0007_UUID ACTIVE
8: fastfs-OST0008_UUID ACTIVE
9: fastfs-OST0009_UUID ACTIVE
10: fastfs-OST000a_UUID ACTIVE
11: fastfs-OST000b_UUID ACTIVE
12: fastfs-OST000c_UUID ACTIVE
13: fastfs-OST000d_UUID ACTIVE
14: fastfs-OST000e_UUID ACTIVE
15: fastfs-OST000f_UUID ACTIVE
16: fastfs-OST0010_UUID ACTIVE
17: fastfs-OST0011_UUID ACTIVE
18: fastfs-OST0012_UUID ACTIVE
19: fastfs-OST0013_UUID ACTIVE
20: fastfs-OST0014_UUID ACTIVE
21: fastfs-OST0015_UUID ACTIVE
22: fastfs-OST0016_UUID ACTIVE
23: fastfs-OST0017_UUID ACTIVE
24: fastfs-OST0018_UUID ACTIVE
25: fastfs-OST0019_UUID ACTIVE
26: fastfs-OST001a_UUID ACTIVE
27: fastfs-OST001b_UUID ACTIVE
28: fastfs-OST001c_UUID ACTIVE
29: fastfs-OST001d_UUID ACTIVE
30: fastfs-OST001e_UUID ACTIVE
31: fastfs-OST001f_UUID ACTIVE
32: fastfs-OST0020_UUID ACTIVE
33: fastfs-OST0021_UUID ACTIVE
34: fastfs-OST0022_UUID ACTIVE
35: fastfs-OST0023_UUID ACTIVE
36: fastfs-OST0024_UUID ACTIVE
37: fastfs-OST0025_UUID ACTIVE
38: fastfs-OST0026_UUID ACTIVE
39: fastfs-OST0027_UUID ACTIVE
40: fastfs-OST0028_UUID ACTIVE
41: fastfs-OST0029_UUID ACTIVE
42: fastfs-OST002a_UUID ACTIVE
43: fastfs-OST002b_UUID ACTIVE
44: fastfs-OST002c_UUID ACTIVE
45: fastfs-OST002d_UUID ACTIVE
46: fastfs-OST002e_UUID ACTIVE
47: fastfs-OST002f_UUID ACTIVE
48: fastfs-OST0030_UUID ACTIVE
49: fastfs-OST0031_UUID ACTIVE
50: fastfs-OST0032_UUID ACTIVE
51: fastfs-OST0033_UUID ACTIVE
52: fastfs-OST0034_UUID ACTIVE
53: fastfs-OST0035_UUID ACTIVE
54: fastfs-OST0036_UUID ACTIVE
55: fastfs-OST0037_UUID ACTIVE
56: fastfs-OST0038_UUID ACTIVE
57: fastfs-OST0039_UUID ACTIVE
58: fastfs-OST003a_UUID ACTIVE
59: fastfs-OST003b_UUID ACTIVE
60: fastfs-OST003c_UUID ACTIVE
61: fastfs-OST003d_UUID ACTIVE
62: fastfs-OST003e_UUID ACTIVE
63: fastfs-OST003f_UUID ACTIVE
../../ss_8/kmtest.txt
lmm_magic:          0x0BD10BD0
lmm_object_gr:      0
lmm_object_id:      0x3c52894
lmm_stripe_count:   8
lmm_stripe_size:    1048576
lmm_stripe_pattern: 1
        obdidx           objid          objid            group
            10         6352973       0x60f04d                0
            20         6260051       0x5f8553                0
             4         5733251       0x577b83                0
            22         6381603       0x616023                0
            17         6265103       0x5f990f                0
            45         6133999       0x5d98ef                0
            31         6383869       0x6168fd                0
            58         6234719       0x5f225f                0
----------------------------------------------------------------------------
------------------------------------

Thanks, 
Kshitij

-----Original Message-----
From: David Vasil [mailto:dvasil at ddn.com] 
Sent: Thursday, May 26, 2011 3:01 PM
To: kmehta at cs.uh.edu
Subject: Re: [Lustre-discuss] Poor multithreaded I/O performance

Hi Kshitij,
   Did you create your files with 'lfs setstripe -c <stripe count> <file>'
before writing to it or did you create a directory with a default stripe
size greater than 1?  It sounds like you are only striping across 1 file.

After writing your file out, perform an:

lfs getstripe <file>

Try pre-creating a more widely striped file with:

lfs setstripe -c N <file>

where N is > 1.

You can create a directory where all files under the hierarchy will be
striped using more OSTs in the same manner with lfs setstripe.

_____
David Vasil
DataDirect Networks
615.307.0865
dvasil at ddn.com

On 05/26/2011 02:02 PM, kmehta at cs.uh.edu wrote:
> Ok I ran the following tests:
>
> [1]
> Application spawns 8 threads. I write to Lustre having 8 OSTs.
> Each thread writes data in blocks of 1 Mbyte in a round robin fashion,
i.e.
>
> T0 writes to offsets 0, 8MB, 16MB, etc.
> T1 writes to offsets 1MB, 9MB, 17MB, etc.
> The stripe size being 1MByte, every thread ends up writing to only 1 OST.
>
> I see a bandwidth of 280 Mbytes/sec, similar to the single thread 
> performance.
>
> [2]
> I also ran the same test such that every thread writes data in blocks 
> of 8 Mbytes for the same stripe size. (Thus, every thread will write 
> to every OST). I still get similar performance, ~280Mbytes/sec, so 
> essentially I see no difference between each thread writing to a 
> single OST vs each thread writing to all OSTs.
>
> And as I said before, if all threads write to their own separate file, 
> the resulting bandwidth is ~700Mbytes/sec.
>
> I have attached my C file (simple_io_test.c) herewith. Maybe you could 
> run it and see where the bottleneck is. Comments and instructions for 
> compilation have been included in the file. Do let me know if you need 
> any clarification on that.
>
> Your help is appreciated,
> Kshitij