[Lustre-discuss] IOR writing to a shared file, performance does not scale

Kshitij Mehta kmehta at cs.uh.edu
Fri Feb 10 14:34:20 PST 2012


An `lfs getstripe` on the output file shows that the file is being striped.

> lfs getstripe km_ior_shared.out
> OBDS:
> 0: fastfs-OST0000_UUID ACTIVE
> 1: fastfs-OST0001_UUID ACTIVE
> 2: fastfs-OST0002_UUID ACTIVE
> 3: fastfs-OST0003_UUID ACTIVE
> 4: fastfs-OST0004_UUID ACTIVE
> 5: fastfs-OST0005_UUID ACTIVE
> 6: fastfs-OST0006_UUID ACTIVE
> 7: fastfs-OST0007_UUID ACTIVE
> 8: fastfs-OST0008_UUID ACTIVE
> 9: fastfs-OST0009_UUID ACTIVE
> 10: fastfs-OST000a_UUID ACTIVE
> 11: fastfs-OST000b_UUID ACTIVE
> 12: fastfs-OST000c_UUID ACTIVE
> 13: fastfs-OST000d_UUID ACTIVE
> 14: fastfs-OST000e_UUID ACTIVE
> 15: fastfs-OST000f_UUID ACTIVE
> 16: fastfs-OST0010_UUID ACTIVE
> 17: fastfs-OST0011_UUID ACTIVE
> 18: fastfs-OST0012_UUID ACTIVE
> 19: fastfs-OST0013_UUID ACTIVE
> 20: fastfs-OST0014_UUID ACTIVE
> 21: fastfs-OST0015_UUID ACTIVE
> 22: fastfs-OST0016_UUID ACTIVE
> 23: fastfs-OST0017_UUID ACTIVE
> 24: fastfs-OST0018_UUID ACTIVE
> 25: fastfs-OST0019_UUID ACTIVE
> 26: fastfs-OST001a_UUID ACTIVE
> 27: fastfs-OST001b_UUID ACTIVE
> 28: fastfs-OST001c_UUID ACTIVE
> 29: fastfs-OST001d_UUID ACTIVE
> 30: fastfs-OST001e_UUID ACTIVE
> 31: fastfs-OST001f_UUID ACTIVE
> 32: fastfs-OST0020_UUID ACTIVE
> 33: fastfs-OST0021_UUID ACTIVE
> 34: fastfs-OST0022_UUID ACTIVE
> 35: fastfs-OST0023_UUID ACTIVE
> 36: fastfs-OST0024_UUID ACTIVE
> 37: fastfs-OST0025_UUID ACTIVE
> 38: fastfs-OST0026_UUID ACTIVE
> 39: fastfs-OST0027_UUID ACTIVE
> 40: fastfs-OST0028_UUID ACTIVE
> 41: fastfs-OST0029_UUID ACTIVE
> 42: fastfs-OST002a_UUID ACTIVE
> 43: fastfs-OST002b_UUID ACTIVE
> 44: fastfs-OST002c_UUID ACTIVE
> 45: fastfs-OST002d_UUID ACTIVE
> 46: fastfs-OST002e_UUID ACTIVE
> 47: fastfs-OST002f_UUID ACTIVE
> 48: fastfs-OST0030_UUID ACTIVE
> 49: fastfs-OST0031_UUID ACTIVE
> 50: fastfs-OST0032_UUID ACTIVE
> 51: fastfs-OST0033_UUID ACTIVE
> 52: fastfs-OST0034_UUID ACTIVE
> 53: fastfs-OST0035_UUID ACTIVE
> 54: fastfs-OST0036_UUID ACTIVE
> 55: fastfs-OST0037_UUID ACTIVE
> 56: fastfs-OST0038_UUID ACTIVE
> 57: fastfs-OST0039_UUID ACTIVE
> 58: fastfs-OST003a_UUID ACTIVE
> 59: fastfs-OST003b_UUID ACTIVE
> 60: fastfs-OST003c_UUID ACTIVE
> 61: fastfs-OST003d_UUID ACTIVE
> 62: fastfs-OST003e_UUID ACTIVE
> 63: fastfs-OST003f_UUID ACTIVE
> km_ior_shared.out
>         obdidx           objid          objid            group
>              0        16799778      0x1005822                0
>              1        16940072      0x1027c28                0
>              2        17167283      0x105f3b3                0
>              3        15988448       0xf3f6e0                0
>              4        16859503      0x101416f                0
>              5        17364548      0x108f644                0
>              6        16849540      0x1011a84                0
>              7        12546146       0xbf7062                0
>              8        17143798      0x10597f6                0
>              9        17196912      0x1066770                0
>             10        16916803      0x1022143                0
>             11         7991491       0x79f0c3                0
>             12        16118156       0xf5f18c                0
>             13        16984031      0x10327df                0
>             14        17412454      0x109b166                0
>             15        16599931       0xfd4b7b                0
>             16        16323314       0xf912f2                0
>             17        16664028       0xfe45dc                0
>             18        17292865      0x107de41                0
>             19        17480546      0x10abb62                0
>             20        16527238       0xfc2f86                0
>             21        16436520       0xfacd28                0
>             22        17084101      0x104aec5                0
>             23        15701612       0xef966c                0
>             24        17139117      0x10585ad                0
>             25        16889152      0x101b540                0
>             26        17344287      0x108a71f                0
>             27        17290338      0x107d462                0
>             28        16794234      0x100427a                0
>             29        16153907       0xf67d33                0
>             30        16540825       0xfc6499                0
>             31        17413630      0x109b5fe                0
>             32        13592810       0xcf68ea                0
>             33        16801687      0x1005f97                0
>             34        17519073      0x10b51e1                0
>             35        16898364      0x101d93c                0
>             36        17455291      0x10a58bb                0
>             37        16185055       0xf6f6df                0
>             38        13208545       0xc98be1                0
>             39        16651852       0xfe164c                0
>             40        16845326      0x1010a0e                0
>             41        16305051       0xf8cb9b                0
>             42         9322279       0x8e3f27                0
>             43        17295009      0x107e6a1                0
>             44        16681757       0xfe8b1d                0
>             45        16674804       0xfe6ff4                0
>             46        17476089      0x10aa9f9                0
>             47        16979487      0x103161f                0
>             48        15884086       0xf25f36                0
>             49        16325833       0xf91cc9                0
>             50        15216532       0xe82f94                0
>             51        14917514       0xe39f8a                0
>             52        17016478      0x103a69e                0
>             53        17102674      0x104f752                0
>             54        16167305       0xf6b189                0
>             55        16794224      0x1004270                0
>             56        17047378      0x1041f52                0
>             57        16873050      0x101765a                0
>             58        16293381       0xf89e05                0
>             59        17106865      0x10507b1                0
>             60        16704274       0xfee312                0
>             61        16704006       0xfee206                0
>             62        16554665       0xfc9aa9                0
>             63        16949690      0x102a1ba                0


On 02/10/2012 04:31 PM, Hedges, Richard M. wrote:
> Did you set up the shared file for striping?
>
>> hype356{rhedges}395: lfs help setstripe
>> setstripe: Create a new file with a specific striping pattern or
>> set the default striping pattern on an existing directory or
>> delete the default striping pattern from an existing directory
>> usage: setstripe [--size|-s stripe_size] [--count|-c stripe_count]
>>                   [--index|-i|--offset|-o start_ost_index]
>>                   [--pool|-p<pool>]<directory|filename>
>>         or
>>         setstripe -d<directory>    (to delete default striping)
>>      stripe_size:  Number of bytes on each OST (0 filesystem default)
>>                    Can be specified with k, m or g (in KB, MB and GB
>>                    respectively)
>>      start_ost_index: OST index of first stripe (-1 default)
>>      stripe_count: Number of OSTs to stripe over (0 default, -1 all)
>>      pool:         Name of OST pool to use (default none)
>
>
> On 2/10/12 2:27 PM, "Kshitij Mehta"<kmehta at cs.uh.edu>  wrote:
>
>> We have lustre 1.6.7 configured using 64 OSTs.
>> I am testing the performance using IOR, which is a file system benchmark.
>>
>> When I run IOR using mpi such that processes write to a shared file,
>> performance does not scale. I tested with 1,2 and 4 processes, and the
>> performance remains constant at 230 MBps.
>>
>> When processes write to separate files, performance improves greatly,
>> reaching 475 MBps.
>>
>> Note that all processes are spawned on a single node.
>>
>> Here is the output:
>> Writing to a shared file:
>>
>>> Command line used: ./IOR -a POSIX -b 2g -e -t 32m -w -o
>>> /fastfs/gabriel/ss_64/km_ior.out
>>> Machine: Linux deimos102
>>>
>>> Summary:
>>>          api                = POSIX
>>>          test filename      = /fastfs/gabriel/ss_64/km_ior.out
>>>          access             = single-shared-file
>>>          ordering in a file = sequential offsets
>>>          ordering inter file= no tasks offsets
>>>          clients            = 4 (4 per node)
>>>          repetitions        = 1
>>>          xfersize           = 32 MiB
>>>          blocksize          = 2 GiB
>>>          aggregate filesize = 8 GiB
>>>
>>> Operation  Max (MiB)  Min (MiB)  Mean (MiB)   Std Dev  Max (OPs)  Min
>>> (OPs)  Mean (OPs)   Std Dev  Mean (s)
>>> ---------  ---------  ---------  ----------   -------  ---------
>>> ---------  ----------   -------  --------
>>> write         233.61     233.61      233.61      0.00       7.30
>>> 7.30        7.30      0.00  35.06771   EXCEL
>>>
>>> Max Write: 233.61 MiB/sec (244.95 MB/sec)
>> Writing to separate files:
>>
>>> Command line used: ./IOR -a POSIX -b 2g -e -t 32m -w -o
>>> /fastfs/gabriel/ss_64/km_ior.out -F
>>> Machine: Linux deimos102
>>>
>>> Summary:
>>>          api                = POSIX
>>>          test filename      = /fastfs/gabriel/ss_64/km_ior.out
>>>          access             = file-per-process
>>>          ordering in a file = sequential offsets
>>>          ordering inter file= no tasks offsets
>>>          clients            = 4 (4 per node)
>>>          repetitions        = 1
>>>          xfersize           = 32 MiB
>>>          blocksize          = 2 GiB
>>>          aggregate filesize = 8 GiB
>>>
>>> Operation  Max (MiB)  Min (MiB)  Mean (MiB)   Std Dev  Max (OPs)  Min
>>> (OPs)  Mean (OPs)   Std Dev  Mean (s)
>>> ---------  ---------  ---------  ----------   -------  ---------
>>> ---------  ----------   -------  --------
>>> write         475.95     475.95      475.95      0.00      14.87
>>> 14.87       14.87      0.00  17.21191   EXCEL
>>>
>>> Max Write: 475.95 MiB/sec (499.07 MB/sec)
>> I am trying to understand where the bottleneck is, when processes write
>> to a shared file.
>> Your help is appreciated.
>
> ====================================================
>
> Richard Hedges
> Customer Support and Test - File Systems Project
> Development Environment Group - Livermore Computing
> Lawrence Livermore National Laboratory
> 7000 East Avenue, MS L-557
> Livermore, CA    94551
>
> v:    (925) 423-2699
> f:    (925) 423-6961
> E:    richard-hedges at llnl.gov

-- 
Kshitij Mehta
PhD candidate
Parallel Software Technologies Lab (pstl.cs.uh.edu)
Dept. of Computer Science
University of Houston
Houston, Texas, USA




More information about the lustre-discuss mailing list