[Lustre-devel] Commit on share

[Mikhail Pershin]

Hi,

It seems this mail wasn't received by subscribers though it is in  
lustre-devel archive already. I paste the original answer below.

----

Hello Peter,

Thanks for review. Alexander is on vacation so I will answer as co-author.

On Tue, 27 May 2008 14:44:18 +0400, Peter Braam <Peter.Braam at Sun.COM>
wrote:

> This HLD is definitely not ready at all.  It is very short, lacks
> interaction diagrams and the arguments made are not sufficiently  
> detailed.
>
> * the second sentence is not right.  Commit should happen before
> un-committed data coming from a client is shared with a 2nd client.

Can you provide any issues with that? Committing data after but not before
current operation has following benefits:
1) no need in starting/commiting separate transaction, simple code due to
that
2) less syncs. E.g. if we are committing current operation too then we
resolve possible share case related to current operation and later commit
will not needed. Therefore we will have less synced operations.
E.g. the worst case for COS is:
client1 operation
client2 operation -> sync
client1 op -> sync
client2 op -> sync
...
COS will do commit for each operation if they are on the same object and
sync is happened before current operation.
But including the current operation in commit will reduce number of
commits in twice:
client1 operation
client2 op -> sync (including this op)
client1 op - no sync because no uncommitted shared data
client2 op -> sync
...

> * Is COS dependent on VBR ­ no it is not, and can equally apply to normal
> recovery

Agree, COS is like just lever to turn sync commit on/off depending on some
conditions. These conditions maybe quite simple like now - just
comparision of clients - or maybe more complex and include checking of
operation types, etc.

But COS was requested initially as optional feature of VBR, so we didn't
review COS-only configuration. Without VBR any missed client will invoke
the eviction of all clients with replays after the gap. Therefore COS will
not helps until we will change current recovery to don't evict clients if
COS is enabled. But we should know actually was COS enabled before the
server failure to be sure that the excluding gap transactions is safe. Do
we need COS-only use-case actually?

> * Section 3.2 is wrong: the recovery process will not fail with gaps in  
> the
> sequence when there is VBR.  It only fails if there are gaps in the
> versions, and this is rare.

the 3.2 section is talking only about gap in versions. Maybe it is not
correct grammatically though.
"... Highly probably we have non-trivial gaps version in the sequence and
the recovery process fails"
Could you mark what is wrong with 3.2? just rewrite the sentence to make
it more clear about what gaps we mean?

> * 3.3 parallel creations in one directory are protected with different,
> independent lock resources.  Isn¹t that sufficient to allow parallel
> operations with COS?

it is HEAD feature, but this design is for 1.8 (1.6-based) Lustre with one
dir lock. If this is not mentioned in HLD then it should be fixed.
But the issue is not about the lock only.
The 'simple' COS checks only clients nids to determine the dependency.
Therefore if two clients are creating objects in the same directory then
we will have frequent syncs due to COS (operations from different nids)
although there is no need for sync at all becase the operations are not
dependent.
The same will be with parallel locking if we will not check type of
operation to determine the dependency.

> * 3.6 provide a detailed explanation please
"When enabled, COS makes REP-ACK not needed."
Basically the COS is about two things:
1) dependency tracking. This functionality which try to determine is
current operation depending on some other uncommitted one. It may be
simple and check only nids of clients, maybe more sophisticated and
include type of operation checking or any other additional data.
2) COS itself, the doing sync commit of current operation if there is
dependency.

So if we have 1) and 2) we have only the following cases:
- there is dependency determined and commit is needed to remove it. No ACK
is needed.
- there is no dependency and we don't need no ACK and lock nor commit
because client's replays are not dependent
Therefore the ACK is not needed in both cases. The COS don't need to wait
on repack lock, it determine the share case and do commit.

how ACK is related to 'simple' COS (the only client NIDs are matter):
1) client1 did operation and lock object until ACK from it will come to
server
2) client2 is waiting for ACK or commit to access the object
3) if there was no commit yet, then client2 determine the sharing exists
and force commit

The only positive effect of ACK is delay before doing sync, that give us
the chance to wait for commit without doing force sync. But that can be
done with timer to get the same results.

In HLD we propose the following:
1) client1 got lock, did operation, unlock object after operation is done
2) client2 got lock on object and check was there the dependency
3) if dependency then force commit (or wait for it as alternative way)
4) otherwise update dependency info for next check, unlock object when
operation is done

This is generic way and will work with any dependency tracking (on NIDs,
on types of operations, etc.)

> * GC thread is wrong mechanism this is what we have commit callbacks for

Well, with callbacks we have to scan through all hash to find data to
delete on each callback. As alex said there can be about 10K uncommitted
transactions in high load easily, therefore using callback may become the
bottlneck. There was discussion recently in devel@ about that originated
by zam. Although I agree the HLD should be clear about why we choose that
way and what is wrong with another.

> * Why not use the DLM, then we can simply keep the client waiting ­ the
> mechanism already exists for repack; I am not convinced at all by the
> reasoning that rep-ack is so different ­ no real facts are quoted

Let's estimate how RepACK lock is suitable as dependency tracking
functionality. In fact it is more like 'possible dependency prevention'
mechanism, and block object always because we can't predict the next
operation, so should keep lock taken for ALL modifying operations. It is
not 'tracking' but 'prediction' mechanism, it blocks access to the object
until client will got reply just because the conflicting operation is
possible but not because it really happen.
Moreover it conflicts in general with dependency tracking we needed,
because it will serialize operations even when they may not depend.

With RepACK lock we are entering in operation AFTER the checks and we
don't know the result of this check - was there operation from different
client? are changes committed? Should we do sync or not? RepACK lock
doesn't answer this question and we can't decide about sync is needed or
not.

For example, the client2 will wait for commit or ACK before entering in
locked area.
1) ACK is got but no commit yet. So client2 enter in locked area and now
should determine was commit done or not. How to do that? This is vital
because if there was no commit yet then we should do it. We may use
version of object possible and check it against last_committed, but this
will work only with VBR.
So we need extra data per-object like transno.
2) Commit was done. We should still do the same as for 1) to be sure about
was commit done or not because it is not known why lock was unlocked - due
to ACK or commit.
3) But we don't know still is there conflict or not because we should
check client uuids, but we don't store such info anywhere and waiting on
lock is not reflected somehow. So we need extra data (or extra information
  from ldlm?) again to store uuid of client who did latest operation on that
object.

The only way how dlm can work without any additional data is to unlock
only when commit. But in that case we don't need COS at all. Each
conflicting client will wait on lock until previous changes will be
committed. But this may lead to huge latency for requests, comparing with
commit interval and it is not what we need.

> * It is left completely without explanation how the hash table (which I
> think we don¹t need/want) is used

hash table store the following data per object:
struct lu_dep_info {
          struct ll_fid     di_object;
          struct obd_uuid   di_client;
          __u64             di_transno;
};

it contains uuid of client and transno of last change from this client.
The uuid is compared to determine is there is conflict of not, the transno
shows was that data committed already or not. I described above why it is
needed. It is 1.6-related issue because we have only inode of object and
no any extra structure. The HEAD has lu_object enveloping inodes, and hash
will not needed, the dependency info may be stored per lu_object.

>
> Regards,
>
> Peter



-- 
Mikhail Pershin
Staff Engineer
Lustre Group
Sun Microsystems, Inc.