[Lustre-discuss] Checksums of files on disk
Kevin Van Maren
kevin.van.maren at oracle.com
Wed May 25 07:03:30 PDT 2011
Christopher J.Walker wrote:
> The application I use, StoRM[1] can store checksums on disk in an
> extended user attribute - and use that to ensure the integrity of files
> on disk. The algorithm currently used is adler32. The intention is to
> perform end to end checksumming from file creation through storage,
> transfer over the WAN and storage at a site.
>
> Looking at
> http://wiki.lustre.org/manual/LustreManual20_HTML/ManagingFileSystemIO.html#50438211_pgfId-1291975
>
> I see that Lustre has some checksum support (though not for checksumming
> the file on the OST - so we'd still need to use the user attribute for
> that).
>
>
> http://wiki.lustre.org/manual/LustreManual18_HTML/LustreTuning.html#50651264_pgfId-1291287
>
>
> Is the value of the checksum user accessible? Or to be more specific,
> I'd potentially get a big speedup if I were able to ask the diskserver
> to tell me the checksum of a file without actually transferring it over
> the network. Is it easy to do this?
>
No, the checksum is not currently available, and is not being stored on
disk. That being said, feel free to send patches! There were some
plans to merge the client-side checksum with the ZFS checksum when the
backing store is ZFS, but I have not been following the ZFS status
closely enough to know the status of that enhancement. Do note that the
Lustre checksums only cover the RPC, so at best each 1MB file chunk
would have a separate checksum, generated on the client before doing the
RPC (so not quite as end-to-end as an application checksum).
Also note that the checksums are not used when using mmap(). See Bug
11742 for the details (it is sent, but failures are ignored).
Kevin
> Chris
>
> [1] http://storm.forge.cnaf.infn.it/home This is an SRM implementation
> we use to give an grid authentication to our storage (we store data for
> the LHC).
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>
More information about the lustre-discuss
mailing list