[Lustre-devel] Simpifying Interoperation

Eric Barton eeb at sun.com
Thu Sep 25 08:54:51 PDT 2008


Here are some first thoughts on Huang Hua's idea to simplify version
interoperation, and an invitation for further comments...

1. This scheme should not interfere with upgrade via failover pairs,
   and it must also allow the MDS to be upgraded separately from the
   OSSs.  I think this means in general that we have to allow
   piecemeal server upgrades.

2. This scheme need a mechanism that...

   a) notifies clients when a particular server is about to upgrade so
      that update operations are blocked until the upgrade completes
      and the client reconnects to the upgraded (and/or failed over)
      server.

   b) notifies the server when all clients have completed preparation
      for the upgrade so that no further requests require resend.

   c) notifies clients when all outstanding updates have been
      committed.  If the server crashes before this point, client
      replay is still required.  Clients must not poll for this since
      the server is shutting down.

   The DLM seems the right basic mechanism to notify clients, however
   current assumptions about acquisition timeouts might be an issue.

   We must also ensure that the race between this server upgrade
   process and connection establishment (including any new
   notification locks) by new clients is handled consistently.
   
3. It's not clear to me that we need to evict, or even clean the
   client cache provided the client doesn't attempt any more writes
   until it has connected to the failover server.  The client can
   re-acquire all the locks covering its cache during recovery after
   the upgrade - and there is no need for request refomatting here
   since locks are replayed explicitly (i.e. new requests are
   formatted from scratch using the correct protocol version).

   It does seem advisable however to clean the cache before such a
   significant system incident.

4. We can avoid reformatting requests during open replay if this is
   also done explicity.

5. This scheme prevents recovery on clients that were disconnected
   when the upgrade began.  Such clients will simply be evicted when
   they reconnect even though the server should actually have
   committed all their replayable requests.

   If this can be prevented, we can probably also dispense with much
   of the notification described in (2) above.  However it would
   require (a) a change in the connection protocol to get clients to
   purge their own replay queue and (b) changes to ensure resent
   requests can be reconstructed from scratch (but maybe (b) is just
   another way of saying "request reformatting").

   If this is doable - it further begs the question of whether simply
   making all server requests synchronous during upgrades is enough to
   simply most interoperation issues.

6. This is all about client/server communications. Are there any
   issues for inter-server interoperation?

7. Clients and servers may have to run with different versions for
   extended periods (one customer ran like this for months).  Does
   this raise any issues with this scheme?

    Cheers,
              Eric




More information about the lustre-devel mailing list