[Lustre-devel] changelog for whole filesystem?
andreas.dilger at oracle.com
Fri Nov 12 15:58:17 PST 2010
On 2010-11-12, at 16:41, Robert Read wrote:
> It seems simpler to do the iteration on a snapshot, instead of a live filesystem, and allow post-snapshot changes to accumulate on the regular changelog for processing once the "from empty" iteration was complete.
Just need to implement snapshot support for Lustre. :-)
> On Nov 11, 2010, at 10:10 AM, Eric Barton wrote:
>> Nathan wrote...
>>> This same "initial population of a database from objects" problem
>>> occurs when trying to replicate a Lustre filesystem using the
>> That must be a great test case. If we can reliably reconstruct an
>> exact replica using a "from empty" changelog, we must have got it
>> right :)
>>> The problem is actually more complicated even for a single changelog
>>> consumer: since iterating through the virtual changelog takes non-0
>>> time, you're not sure if a virtual record was created before or
>>> after an actual changelog entry. E.g. you might try to resolve the
>>> name of a file for a virtual record either before or after a rename,
>>> and then later you would see the rename in the changelog, leading to
>>> an inconsistent view of the namespace.
>>> If you don't care about having the exact right name, then it's easy
>>> enough to ignore inapplicable changelog records.
>> I was trying to hint at the need to filter changes for individual
>> consumers depending on how far through the initial "from empty"
>> iteration they are in my original post. Andreas stated it more
>> explicitly. I think you just need to enumerate the cases - e.g. for
>> Iterated over yet? Record
>> Source Inode Target Inode emitted
>> no no none
>> no yes delete target
>> yes yes rename source + delete target
>> yes no rename source
>> The final case just looks like a rename where the target name didn't
>> exist already. In any case, the only real requirement on the stream
>> of changelog records constructed in the initial iteration is that
>> consistent filesystem state be reconstructed after the last record
>> is consumed.
>> Eric Barton
>> CTO Whamcloud, Inc.
>> Tel: +44 117 330 1575
>> Mob: +44 7920 797 273
>> Lustre-devel mailing list
>> Lustre-devel at lists.lustre.org
Lustre Technical Lead
Oracle Corporation Canada Inc.
More information about the lustre-devel