[Lustre-discuss] Checksums of files on disk

Kevin Van Maren kevin.van.maren at oracle.com
Wed May 25 07:03:30 PDT 2011


Christopher J.Walker wrote:
> The application I use, StoRM[1] can store checksums on disk in an
> extended user attribute - and use that to ensure the integrity of files
> on disk. The algorithm currently used is adler32. The intention is to
> perform end to end checksumming from file creation through storage,
> transfer over the WAN and storage at a site.
>
> Looking at
> http://wiki.lustre.org/manual/LustreManual20_HTML/ManagingFileSystemIO.html#50438211_pgfId-1291975
>
> I see that Lustre has some checksum support (though not for checksumming
> the file on the OST - so we'd still need to use the user attribute for
> that).
>
>
> http://wiki.lustre.org/manual/LustreManual18_HTML/LustreTuning.html#50651264_pgfId-1291287
>
>
> Is the value of the checksum user accessible? Or to be more specific,
> I'd potentially get a big speedup if I were able to ask the diskserver
> to tell me the checksum of a file without actually transferring it over
> the network. Is it easy to do this?
>   

No, the checksum is not currently available, and is not being stored on 
disk.  That being said, feel free to send patches!  There were some 
plans to merge the client-side checksum with the ZFS checksum when the 
backing store is ZFS, but I have not been following the ZFS status 
closely enough to know the status of that enhancement.  Do note that the 
Lustre checksums only cover the RPC, so at best each 1MB file chunk 
would have a separate checksum, generated on the client before doing the 
RPC (so not quite as end-to-end as an application checksum).

Also note that the checksums are not used when using mmap().  See Bug 
11742 for the details (it is sent, but failures are ignored).

Kevin

> Chris
>
> [1] http://storm.forge.cnaf.infn.it/home This is an SRM implementation
> we use to give an grid authentication to our storage (we store data for
> the LHC).
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>   




More information about the lustre-discuss mailing list