[Lustre-discuss] IOR writing to a shared file, performance does not scale
Kshitij Mehta
kmehta at cs.uh.edu
Fri Feb 10 14:34:20 PST 2012
An `lfs getstripe` on the output file shows that the file is being striped.
> lfs getstripe km_ior_shared.out
> OBDS:
> 0: fastfs-OST0000_UUID ACTIVE
> 1: fastfs-OST0001_UUID ACTIVE
> 2: fastfs-OST0002_UUID ACTIVE
> 3: fastfs-OST0003_UUID ACTIVE
> 4: fastfs-OST0004_UUID ACTIVE
> 5: fastfs-OST0005_UUID ACTIVE
> 6: fastfs-OST0006_UUID ACTIVE
> 7: fastfs-OST0007_UUID ACTIVE
> 8: fastfs-OST0008_UUID ACTIVE
> 9: fastfs-OST0009_UUID ACTIVE
> 10: fastfs-OST000a_UUID ACTIVE
> 11: fastfs-OST000b_UUID ACTIVE
> 12: fastfs-OST000c_UUID ACTIVE
> 13: fastfs-OST000d_UUID ACTIVE
> 14: fastfs-OST000e_UUID ACTIVE
> 15: fastfs-OST000f_UUID ACTIVE
> 16: fastfs-OST0010_UUID ACTIVE
> 17: fastfs-OST0011_UUID ACTIVE
> 18: fastfs-OST0012_UUID ACTIVE
> 19: fastfs-OST0013_UUID ACTIVE
> 20: fastfs-OST0014_UUID ACTIVE
> 21: fastfs-OST0015_UUID ACTIVE
> 22: fastfs-OST0016_UUID ACTIVE
> 23: fastfs-OST0017_UUID ACTIVE
> 24: fastfs-OST0018_UUID ACTIVE
> 25: fastfs-OST0019_UUID ACTIVE
> 26: fastfs-OST001a_UUID ACTIVE
> 27: fastfs-OST001b_UUID ACTIVE
> 28: fastfs-OST001c_UUID ACTIVE
> 29: fastfs-OST001d_UUID ACTIVE
> 30: fastfs-OST001e_UUID ACTIVE
> 31: fastfs-OST001f_UUID ACTIVE
> 32: fastfs-OST0020_UUID ACTIVE
> 33: fastfs-OST0021_UUID ACTIVE
> 34: fastfs-OST0022_UUID ACTIVE
> 35: fastfs-OST0023_UUID ACTIVE
> 36: fastfs-OST0024_UUID ACTIVE
> 37: fastfs-OST0025_UUID ACTIVE
> 38: fastfs-OST0026_UUID ACTIVE
> 39: fastfs-OST0027_UUID ACTIVE
> 40: fastfs-OST0028_UUID ACTIVE
> 41: fastfs-OST0029_UUID ACTIVE
> 42: fastfs-OST002a_UUID ACTIVE
> 43: fastfs-OST002b_UUID ACTIVE
> 44: fastfs-OST002c_UUID ACTIVE
> 45: fastfs-OST002d_UUID ACTIVE
> 46: fastfs-OST002e_UUID ACTIVE
> 47: fastfs-OST002f_UUID ACTIVE
> 48: fastfs-OST0030_UUID ACTIVE
> 49: fastfs-OST0031_UUID ACTIVE
> 50: fastfs-OST0032_UUID ACTIVE
> 51: fastfs-OST0033_UUID ACTIVE
> 52: fastfs-OST0034_UUID ACTIVE
> 53: fastfs-OST0035_UUID ACTIVE
> 54: fastfs-OST0036_UUID ACTIVE
> 55: fastfs-OST0037_UUID ACTIVE
> 56: fastfs-OST0038_UUID ACTIVE
> 57: fastfs-OST0039_UUID ACTIVE
> 58: fastfs-OST003a_UUID ACTIVE
> 59: fastfs-OST003b_UUID ACTIVE
> 60: fastfs-OST003c_UUID ACTIVE
> 61: fastfs-OST003d_UUID ACTIVE
> 62: fastfs-OST003e_UUID ACTIVE
> 63: fastfs-OST003f_UUID ACTIVE
> km_ior_shared.out
> obdidx objid objid group
> 0 16799778 0x1005822 0
> 1 16940072 0x1027c28 0
> 2 17167283 0x105f3b3 0
> 3 15988448 0xf3f6e0 0
> 4 16859503 0x101416f 0
> 5 17364548 0x108f644 0
> 6 16849540 0x1011a84 0
> 7 12546146 0xbf7062 0
> 8 17143798 0x10597f6 0
> 9 17196912 0x1066770 0
> 10 16916803 0x1022143 0
> 11 7991491 0x79f0c3 0
> 12 16118156 0xf5f18c 0
> 13 16984031 0x10327df 0
> 14 17412454 0x109b166 0
> 15 16599931 0xfd4b7b 0
> 16 16323314 0xf912f2 0
> 17 16664028 0xfe45dc 0
> 18 17292865 0x107de41 0
> 19 17480546 0x10abb62 0
> 20 16527238 0xfc2f86 0
> 21 16436520 0xfacd28 0
> 22 17084101 0x104aec5 0
> 23 15701612 0xef966c 0
> 24 17139117 0x10585ad 0
> 25 16889152 0x101b540 0
> 26 17344287 0x108a71f 0
> 27 17290338 0x107d462 0
> 28 16794234 0x100427a 0
> 29 16153907 0xf67d33 0
> 30 16540825 0xfc6499 0
> 31 17413630 0x109b5fe 0
> 32 13592810 0xcf68ea 0
> 33 16801687 0x1005f97 0
> 34 17519073 0x10b51e1 0
> 35 16898364 0x101d93c 0
> 36 17455291 0x10a58bb 0
> 37 16185055 0xf6f6df 0
> 38 13208545 0xc98be1 0
> 39 16651852 0xfe164c 0
> 40 16845326 0x1010a0e 0
> 41 16305051 0xf8cb9b 0
> 42 9322279 0x8e3f27 0
> 43 17295009 0x107e6a1 0
> 44 16681757 0xfe8b1d 0
> 45 16674804 0xfe6ff4 0
> 46 17476089 0x10aa9f9 0
> 47 16979487 0x103161f 0
> 48 15884086 0xf25f36 0
> 49 16325833 0xf91cc9 0
> 50 15216532 0xe82f94 0
> 51 14917514 0xe39f8a 0
> 52 17016478 0x103a69e 0
> 53 17102674 0x104f752 0
> 54 16167305 0xf6b189 0
> 55 16794224 0x1004270 0
> 56 17047378 0x1041f52 0
> 57 16873050 0x101765a 0
> 58 16293381 0xf89e05 0
> 59 17106865 0x10507b1 0
> 60 16704274 0xfee312 0
> 61 16704006 0xfee206 0
> 62 16554665 0xfc9aa9 0
> 63 16949690 0x102a1ba 0
On 02/10/2012 04:31 PM, Hedges, Richard M. wrote:
> Did you set up the shared file for striping?
>
>> hype356{rhedges}395: lfs help setstripe
>> setstripe: Create a new file with a specific striping pattern or
>> set the default striping pattern on an existing directory or
>> delete the default striping pattern from an existing directory
>> usage: setstripe [--size|-s stripe_size] [--count|-c stripe_count]
>> [--index|-i|--offset|-o start_ost_index]
>> [--pool|-p<pool>]<directory|filename>
>> or
>> setstripe -d<directory> (to delete default striping)
>> stripe_size: Number of bytes on each OST (0 filesystem default)
>> Can be specified with k, m or g (in KB, MB and GB
>> respectively)
>> start_ost_index: OST index of first stripe (-1 default)
>> stripe_count: Number of OSTs to stripe over (0 default, -1 all)
>> pool: Name of OST pool to use (default none)
>
>
> On 2/10/12 2:27 PM, "Kshitij Mehta"<kmehta at cs.uh.edu> wrote:
>
>> We have lustre 1.6.7 configured using 64 OSTs.
>> I am testing the performance using IOR, which is a file system benchmark.
>>
>> When I run IOR using mpi such that processes write to a shared file,
>> performance does not scale. I tested with 1,2 and 4 processes, and the
>> performance remains constant at 230 MBps.
>>
>> When processes write to separate files, performance improves greatly,
>> reaching 475 MBps.
>>
>> Note that all processes are spawned on a single node.
>>
>> Here is the output:
>> Writing to a shared file:
>>
>>> Command line used: ./IOR -a POSIX -b 2g -e -t 32m -w -o
>>> /fastfs/gabriel/ss_64/km_ior.out
>>> Machine: Linux deimos102
>>>
>>> Summary:
>>> api = POSIX
>>> test filename = /fastfs/gabriel/ss_64/km_ior.out
>>> access = single-shared-file
>>> ordering in a file = sequential offsets
>>> ordering inter file= no tasks offsets
>>> clients = 4 (4 per node)
>>> repetitions = 1
>>> xfersize = 32 MiB
>>> blocksize = 2 GiB
>>> aggregate filesize = 8 GiB
>>>
>>> Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min
>>> (OPs) Mean (OPs) Std Dev Mean (s)
>>> --------- --------- --------- ---------- ------- ---------
>>> --------- ---------- ------- --------
>>> write 233.61 233.61 233.61 0.00 7.30
>>> 7.30 7.30 0.00 35.06771 EXCEL
>>>
>>> Max Write: 233.61 MiB/sec (244.95 MB/sec)
>> Writing to separate files:
>>
>>> Command line used: ./IOR -a POSIX -b 2g -e -t 32m -w -o
>>> /fastfs/gabriel/ss_64/km_ior.out -F
>>> Machine: Linux deimos102
>>>
>>> Summary:
>>> api = POSIX
>>> test filename = /fastfs/gabriel/ss_64/km_ior.out
>>> access = file-per-process
>>> ordering in a file = sequential offsets
>>> ordering inter file= no tasks offsets
>>> clients = 4 (4 per node)
>>> repetitions = 1
>>> xfersize = 32 MiB
>>> blocksize = 2 GiB
>>> aggregate filesize = 8 GiB
>>>
>>> Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min
>>> (OPs) Mean (OPs) Std Dev Mean (s)
>>> --------- --------- --------- ---------- ------- ---------
>>> --------- ---------- ------- --------
>>> write 475.95 475.95 475.95 0.00 14.87
>>> 14.87 14.87 0.00 17.21191 EXCEL
>>>
>>> Max Write: 475.95 MiB/sec (499.07 MB/sec)
>> I am trying to understand where the bottleneck is, when processes write
>> to a shared file.
>> Your help is appreciated.
>
> ====================================================
>
> Richard Hedges
> Customer Support and Test - File Systems Project
> Development Environment Group - Livermore Computing
> Lawrence Livermore National Laboratory
> 7000 East Avenue, MS L-557
> Livermore, CA 94551
>
> v: (925) 423-2699
> f: (925) 423-6961
> E: richard-hedges at llnl.gov
--
Kshitij Mehta
PhD candidate
Parallel Software Technologies Lab (pstl.cs.uh.edu)
Dept. of Computer Science
University of Houston
Houston, Texas, USA
More information about the lustre-discuss
mailing list