[Lustre-discuss] Lustre MPI-IO performance on CNL

[Marty Barnaby]

My, perhaps, misunderstanding was that a Lustre FS had a maximum lfs 
stripe-count of 160. Is this not a constant set  in the LFS, but just 
some local configuration? Could you be more specific about the actual 
lfs stripe-count of the file or files you wrote?

MLB


Weikuan Yu wrote:
> Hi,
>
> The I/O performance of CNL (as measured with IOR) seems quite different
> for a shared file, compared to the same with separated files.
>
> Here are some numbers on a smaller file system on XT system at ORNL. All
> files are striped to 72OSTs. I deliberately use a block size 8512m.
>
> 1. sample tests with separate files
> # aprun -n 32 -N 1 ~/benchmarks/IOR-2.9.1/src/C/IOR -a MPIIO -b 8512m -t
> 64m -d 1 -i 2 -w -r -g -F -o iortes
> Max Write: 9978.18 MiB/sec (10462.88 MB/sec)
> Max Read:  5612.78 MiB/sec (5885.43 MB/sec)
>
> 2. sample share file performance
> # aprun -n 32 -N 1 ~/benchmarks/IOR-2.9.1/src/C/IOR -a MPIIO -b 8512m -t
> 64m -d 1 -i 2 -w -r -g -o iortes
> Max Write: 6817.31 MiB/sec (7148.47 MB/sec)
> Max Read:  5591.98 MiB/sec (5863.62 MB/sec)
>
> In addition, using my experimental MPI-IO library, I noticed that
> enabling direct I/O can have various effects for I/O on CNL.
>
> 3. sample seprate files with direct I/O
> export MPIO_DIRECT_WRITE=true; export MPIO_DIRECT_READ=true; aprun -n 32
> -N 1 ~/benchmarks/IOR-2.10.1/src/C/IOR -a MPIIO -b 8512m -t 64m -d 1 -i
> 2 -w -r -g -F -k -o lustre:iortest
> Max Write: 9353.66 MiB/sec (9808.03 MB/sec)
> Max Read:  8269.28 MiB/sec (8670.97 MB/sec)
>
> 4. sample share file performance with direct IO
> # export MPIO_DIRECT_WRITE=true; export MPIO_DIRECT_READ=true; aprun -n
> 32 -N 1 ~/benchmarks/IOR-2.10.1/src/C/IOR -a MPIIO -b 8512m -t 64m -d 1
> -i 2 -w -r -g -k -o lustre:iortes
> Max Write: 9484.11 MiB/sec (9944.81 MB/sec)
> Max Read:  7929.63 MiB/sec (8314.81 MB/sec)
>
> It seems direct I/O helps quite a bit on the performance of parallel
> reads, but not on writes. The shared file mode appears to benefit more
> from direct write.
>
> While it is understandable that the client cache can play a big role
> here,  I am not sure how it could help the share-file mode much better.
> Anybody can help with some explanations on the comparison between reads
> and writes and the same for shared-file and separated-files?
>
> Also let me know if I am not clear in my descriptions.
>
> --
> Weikuan Yu <+> 1-865-574-7990
> http://ft.ornl.gov/~wyu/
>
> P.S.:
> What shown are the good numbers from several runs. So you may consider
> them as consistent results.
>
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>
>