[lustre-discuss] How Lustre stores hyperslabs and chunks of HDF5?

[Dilger, Andreas]

Currently there is no direct connection between Lustre layout and HDF5 file layout.  The only option is RAID-0 striping across OST objects with a fixed stripe size. If HDF5 is aware of this stripe size and can take advantage of it, that is great.

There is a project that has started to implement Progressive File Layout (PFL) that allows different extents of a file to have different stripe counts and stripe sizes, which could be leveraged by libraries like HDF5 in the future.

See http://cdn.opensfs.org/wp-content/uploads/2015/04/Progressive-File-Layouts_Hammond.pdf and/or https://www.youtube.com/watch?v=5rm6Nlmqdp0 for more details on PFL prototype development.

Cheers, Andreas

On Jul 19, 2015, at 20:37, "Prakrati.Agrawal at shell.com<mailto:Prakrati.Agrawal at shell.com>" <Prakrati.Agrawal at shell.com<mailto:Prakrati.Agrawal at shell.com>> wrote:

Hi,

I wanted to understand how Lustre stores the chunks and hyperslabs in HDF5 framework on the OSTs?
If I set the chunk size and each rank is writing a hyperslab, then OST0 has chunk0, OST1 has chunk1 and so on or is it that OST0 has hyperslab0, OST1 has hyperslab1.
Is there any way of finding that out?

Thanks and Regards,
Prakrati
_______________________________________________
lustre-discuss mailing list
lustre-discuss at lists.lustre.org<mailto:lustre-discuss at lists.lustre.org>
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org