From eeb at sun.com  Mon Oct  6 02:53:26 2008
From: eeb at sun.com (Eric Barton)
Date: Mon, 06 Oct 2008 03:53:26 +0100
Subject: [Lustre-devel] MDWBC and how much to trust clients
Message-ID: <1FEDA9D966B34039AF0EC15DBD0E4BAA@eblap>

Nikita,

Do you agree that a buggy or malicious MDWBC could disrupt the
namespace (e.g. links to missing files, orphaned files) if
it splits up operations across multiple MDTs into sub-operations
for the individual targets?  I think it will be an issue for
security if we just trust the MDWBC to do such operations
correctly, and so I'm wondering how we can fix this.  

Using a master MDT to coordinate the operation across itself and
the remaining MDTs seems part of, but not all of the solution.
We have to process batches in bulk to retain a significant
performance advantage, so I wonder if that requires us to trust
that these batches have been created correctly.  

If so, we're stuck with the MDWBC being something we can only
do in a single trust domain - i.e. not across a WAN. That seems
unfortunate since WAN performance should be a major beneficiary
of the MDWBC.  Maybe in this case, we can still send batches over
the WAN, but to a single target which proxies for the remote client
and can be trusted to split multi-target ops over batches correctly.

Thoughts?

    Cheers,
              Eric


From Peter.Braam at Sun.COM  Mon Oct  6 03:19:50 2008
From: Peter.Braam at Sun.COM (Peter Braam)
Date: Sun, 05 Oct 2008 21:19:50 -0600
Subject: [Lustre-devel] MDWBC and how much to trust clients
In-Reply-To: <1FEDA9D966B34039AF0EC15DBD0E4BAA@eblap>
Message-ID: <C50EDFF6.5F4C%peter.braam@sun.com>

We discussed this in Moscow recently.  It seems possible to avoid much
mis-behavior by building relationships that have to be confirmed before a
commit can happen.

For example a directory entry creation must be accompanied by an object
creation or link-count change.

I think it is possible for an MDS or MDS cluster to know in which cases such
relationships need to be present  for operations to transition the name
space to a new namespace (and clients can indicate what operations are
correlated).

Peter



On 10/5/08 8:53 PM, "Eric Barton" <eeb at sun.com> wrote:

> Nikita,
> 
> Do you agree that a buggy or malicious MDWBC could disrupt the
> namespace (e.g. links to missing files, orphaned files) if
> it splits up operations across multiple MDTs into sub-operations
> for the individual targets?  I think it will be an issue for
> security if we just trust the MDWBC to do such operations
> correctly, and so I'm wondering how we can fix this.
> 
> Using a master MDT to coordinate the operation across itself and
> the remaining MDTs seems part of, but not all of the solution.
> We have to process batches in bulk to retain a significant
> performance advantage, so I wonder if that requires us to trust
> that these batches have been created correctly.
> 
> If so, we're stuck with the MDWBC being something we can only
> do in a single trust domain - i.e. not across a WAN. That seems
> unfortunate since WAN performance should be a major beneficiary
> of the MDWBC.  Maybe in this case, we can still send batches over
> the WAN, but to a single target which proxies for the remote client
> and can be trusted to split multi-target ops over batches correctly.
> 
> Thoughts?
> 
>     Cheers,
>               Eric
> 
> _______________________________________________
> Lustre-devel mailing list
> Lustre-devel at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-devel



From Nikita.Danilov at Sun.COM  Mon Oct  6 15:55:48 2008
From: Nikita.Danilov at Sun.COM (Nikita Danilov)
Date: Mon, 06 Oct 2008 19:55:48 +0400
Subject: [Lustre-devel] MDWBC and how much to trust clients
In-Reply-To: <1FEDA9D966B34039AF0EC15DBD0E4BAA@eblap>
References: <1FEDA9D966B34039AF0EC15DBD0E4BAA@eblap>
Message-ID: <18666.13572.686218.818001@gargle.gargle.HOWL>

Eric Barton writes:
 > Nikita,

Hello,

 > 
 > Do you agree that a buggy or malicious MDWBC could disrupt the
 > namespace (e.g. links to missing files, orphaned files) if
 > it splits up operations across multiple MDTs into sub-operations
 > for the individual targets?  I think it will be an issue for
 > security if we just trust the MDWBC to do such operations
 > correctly, and so I'm wondering how we can fix this.  

as Peter mentioned, we discussed this topic during the Moscow
meeting. If I am not mistaken, we converged to the idea that before
committing an epoch, every mdt composes some kind of a `summary',
containing enough information for verification of a global consistency,
and this summary is passed though every server as a ticket, with every
server `approving' some bits in the summary accumulated so far, and
adding new ones. For example, one server adds 

        (SETATTR: FID: fid1, UPDATE: nlink += 2) 

to the summary, then another server having 

        (LINK: PARENT_FID: fid2, NAME: "foo", CHILD_FID: fid1),

in its local epoch replaces UPDATE part of the SETATTR record above with
nlink += 1, and yet another server with

        (LINK: PARENT_FID: fid3, NAME: "bar", CHILD_FID: fid1),

can cancel SETATTR completely. Note, that LINK might cancel UNLINK or
RENAME as well as SETATTR. Global consistency is verified when all
summary records are similarly canceled. All this is still very vague to
me:

    - it is not clear how to start summary exchange (round robin
      perhaps, based on an epoch number)?

    - what state should be kept in a summary?

    - is it always possible to prove consistency in one cycle?

 > 
 > Using a master MDT to coordinate the operation across itself and
 > the remaining MDTs seems part of, but not all of the solution.
 > We have to process batches in bulk to retain a significant
 > performance advantage, so I wonder if that requires us to trust
 > that these batches have been created correctly.  
 > 
 > If so, we're stuck with the MDWBC being something we can only
 > do in a single trust domain - i.e. not across a WAN. That seems
 > unfortunate since WAN performance should be a major beneficiary
 > of the MDWBC.  Maybe in this case, we can still send batches over
 > the WAN, but to a single target which proxies for the remote client
 > and can be trusted to split multi-target ops over batches correctly.
 > 
 > Thoughts?
 > 
 >     Cheers,
 >               Eric

Nikita.

From Nikita.Danilov at Sun.COM  Tue Oct  7 09:13:08 2008
From: Nikita.Danilov at Sun.COM (Nikita Danilov)
Date: Tue, 07 Oct 2008 13:13:08 +0400
Subject: [Lustre-devel] MDWBC and how much to trust clients
In-Reply-To: <18666.13572.686218.818001@gargle.gargle.HOWL>
References: <1FEDA9D966B34039AF0EC15DBD0E4BAA@eblap>
	<18666.13572.686218.818001@gargle.gargle.HOWL>
Message-ID: <18667.10276.857149.242018@gargle.gargle.HOWL>

Nikita Danilov writes:
 > Eric Barton writes:
 >  > Nikita,
 > 
 > Hello,

[...]

 > 
 > as Peter mentioned, we discussed this topic during the Moscow
 > meeting. If I am not mistaken, we converged to the idea that before
 > committing an epoch, every mdt composes some kind of a `summary',
 > containing enough information for verification of a global consistency,
 > and this summary is passed though every server as a ticket, with every

This can be simplified. Suppose total amount of `data', describing all
updates within given epoch is D, and there are N md servers in a cmd
cluster. Then total network traffic incurred by this algorithm is

             D   /* updates from client to all servers */ +
             D*N /* cycle summary through all servers */

that is, (N + 1)*D bytes, transferred in 2*N messages. So we won't
increase network traffic by broadcasting _all_ epoch updates to _every_
server (so that each server gets complete set of all updates within the
epoch). In this latter case, servers can prove that epoch is consistent
by

    - checking global consistency locally,

    - calculating md5 signature of all epoch updates, and

    - exchanging these signatures, to check that client sent the same
      set of updates to everybody.

This results in

             D*N /* broadcast epoch updates to all servers */ +
             e*N /* exchange signatures */

that is N*(D + e), for some small e, bytes transferred in 2*N
messages. Having complete set of updates on every server would probably
help in other places too.


 > server `approving' some bits in the summary accumulated so far, and

[...]

 >  > 
 >  > Thoughts?
 >  > 
 >  >     Cheers,
 >  >               Eric
 > 

Nikita.

From Alexander.Zarochentsev at Sun.COM  Wed Oct  8 06:40:30 2008
From: Alexander.Zarochentsev at Sun.COM (Alexander Zarochentsev)
Date: Wed, 08 Oct 2008 10:40:30 +0400
Subject: [Lustre-devel] COS performance issues
Message-ID: <200810081040.31363.alexander.zarochentsev@sun.com>

Hello,

the cos performance testing made discovered COS performance
problems in tests which were not supposed to be problematic.

The tests are: 
(3a) parallel file creation from several clients, each client creates files in own directory.
(3b) parallel file creation from several clients in one shared directory.

The test results published in Bugzilla bug 17235. The slowdown depends on hardware, for the test clusters
Washie1,3 it is enough big, the difference between COS=0 and COS=1 was about 20%
initially and PTLDEBUG=0 reduces the slowdown to 12-13% (COS patch adds debugging
stuff like ENTRY/EXIT to ptlrpc routines).

COS code finds no inter-client dependencies and adds no extra commits
during those tests, there are about 60 commits in each of 300 seconds tests
with COS=0 and COS=1, one ldiskfs commit in 5 seconds.

However, oprofile results are terrible for COS=1.

COS code profiling results:
== COS=1 ==

376924    4.5556  ptlrpc.ko                ptlrpc                   ptlrpc_server_handle_reply
288668    3.4889  vmlinux                  vmlinux                  .text.task_rq_lock
281528    3.4026  ptlrpc.ko                ptlrpc                   ldlm_resource_find
180107    2.1768  vmlinux                  vmlinux                  .text.__wake_up_common
173166    2.0929  obdclass.ko              obdclass                 htable_lookup
166829    2.0163  e1000e.ko                e1000e                   e1000_irq_enable
150765    1.8222  vmlinux                  vmlinux                  schedule
127059    1.5357  e1000e.ko                e1000e                   e1000_intr_msi
119343    1.4424  ptlrpc.ko                ptlrpc                   target_send_reply
118733    1.4350  lvfs.ko                  lvfs                     lprocfs_counter_add
99568     1.2034  vmlinux                  vmlinux                  __find_get_block
93356     1.1283  vmlinux                  vmlinux                  __switch_to
91259     1.1030  oprofiled                oprofiled                (no symbols)
91147     1.1016  vmlinux                  vmlinux                  try_to_wake_up
89665     1.0837  vmlinux                  vmlinux                  kfree
87774     1.0609  vmlinux                  vmlinux                  module_refcount
85149     1.0291  vmlinux                  vmlinux                  memset
84822     1.0252  vmlinux                  vmlinux                  mwait_idle
78850     0.9530  vmlinux                  vmlinux                  __kmalloc
70324     0.8500  ptlrpc.ko                ptlrpc                   ptlrpc_main
69333     0.8380  vmlinux                  vmlinux                  memmove

== COS=0 ==

237314    2.7642  e1000e.ko                e1000e                   e1000_irq_enable
211304    2.4613  obdclass.ko              obdclass                 htable_lookup
197515    2.3007  vmlinux                  vmlinux                  schedule
180431    2.1017  e1000e.ko                e1000e                   e1000_intr_msi
128113    1.4923  vmlinux                  vmlinux                  __switch_to
127683    1.4873  lvfs.ko                  lvfs                     lprocfs_counter_add
122777    1.4301  vmlinux                  vmlinux                  kfree
122584    1.4279  vmlinux                  vmlinux                  module_refcount
116800    1.3605  vmlinux                  vmlinux                  __find_get_block
110418    1.2862  vmlinux                  vmlinux                  mwait_idle
101909    1.1870  vmlinux                  vmlinux                  memset
101100    1.1776  oprofiled                oprofiled                (no symbols)
96397     1.1228  vmlinux                  vmlinux                  __kmalloc
91745     1.0686  ptlrpc.ko                ptlrpc                   ptlrpc_main
80605     0.9389  ksocklnd.ko              ksocklnd                 ksocknal_scheduler
79782     0.9293  vmlinux                  vmlinux                  kmem_cache_free
76968     0.8965  mds.ko                   mds                      mds_lov_dump_objids
75094     0.8747  vmlinux                  vmlinux                  .text.task_rq_lock
75087     0.8746  vmlinux                  vmlinux                  memmove
72627     0.8460  vmlinux                  vmlinux                  find_get_page
67242     0.7832  vmlinux                  vmlinux                  kmem_cache_alloc

== events distribution in ptlrpc_server_handle_reply, the most cpu consuming function ==

CPU: P4 / Xeon, speed 2800.2 MHz (estimated)
Counted GLOBAL_POWER_EVENTS events (time during which processor is not stopped)
with a unit mask of 0x01 (mandatory) count 100000

vma      samples  %        image name               app name                 symbol name
00000000000498f0 376924    4.5556  ptlrpc.ko                ptlrpc                   ptlrpc_server_handle_reply
  00000000000498f0 1        2.7e-04
  00000000000498f4 10        0.0027
  0000000000049905 3        8.0e-04
  0000000000049912 22        0.0058
  0000000000049972 1        2.7e-04
  0000000000049977 658       0.1746
  000000000004997e 24        0.0064
  0000000000049a23 5         0.0013
  0000000000049a35 21        0.0056
  0000000000049a80 1        2.7e-04
  0000000000049a85 4         0.0011
  0000000000049a92 2        5.3e-04
  0000000000049a99 22        0.0058
  0000000000049aa9 1        2.7e-04
  0000000000049ab5 371741   98.6249
  ...              ^^^^^^   spin_lock (&obd->obd_uncommitted_replies_lock);

I have a patch to avoid using of obd_uncommitted_replies_lock 
in ptlrpc_server_handle_reply but it has minimal effect, 
ptlrpc_server_handle_reply still the most cpu consuming function
because of svc->srv_lock contention.

I think the problem is that COS defers processing of replies to transaction commit time.
When commit happens, MDS has to process thousands of replies (about 14k replies per
commit in the test 3.a) in short period of time. I guess the mdt service threads 
all woken up and spin trying to get the service svr_lock. Processing of new requests may
also suffer of this.

I ran the tests with with CONFIG_DEBUG_SPINLOCK_SLEEP debugging compiled into a kernel, it found no
sleep under spinlock bugs.

Further optimization may include 
1. per-reply spin locks.
2. per-cpu structures and threads to process reply queues.

Any comments?

Thanks.

PS. the test results are much better when MDS server is sata20 machine with 4 cores
(the MDS from Washie1 has 2 cores), COS=0 and COS=1 have only %3 difference:

COS=1
Rate: 3101.77 creates/sec (total: 2 threads 930530 creates 300 secs)
Rate: 3096.94 creates/sec (total: 2 threads 929083 creates 300 secs)

COS=0
Rate: 3184.01 creates/sec (total: 2 threads 958388 creates 301 secs)
Rate: 3152.89 creates/sec (total: 2 threads 945868 creates 300 secs)

-- 
Alexander "Zam" Zarochentsev
Staff Engineer
Lustre Group, Sun Microsystems

From Alexander.Zarochentsev at Sun.COM  Wed Oct  8 11:44:07 2008
From: Alexander.Zarochentsev at Sun.COM (Alexander Zarochentsev)
Date: Wed, 08 Oct 2008 15:44:07 +0400
Subject: [Lustre-devel] COS performance issues
Message-ID: <200810081544.08292.alexander.zarochentsev@sun.com>

sending to lustre-recovery@ as well

Hello,

the cos performance testing made discovered COS performance
problems in tests which were not supposed to be problematic.

The tests are: 
(3a) parallel file creation from several clients, each client creates files in own directory.
(3b) parallel file creation from several clients in one shared directory.

The test results published in Bugzilla bug 17235. The slowdown depends on hardware, for the test clusters
Washie1,3 it is enough big, the difference between COS=0 and COS=1 was about 20%
initially and PTLDEBUG=0 reduces the slowdown to 12-13% (COS patch adds debugging
stuff like ENTRY/EXIT to ptlrpc routines).

COS code finds no inter-client dependencies and adds no extra commits
during those tests, there are about 60 commits in each of 300 seconds tests
with COS=0 and COS=1, one ldiskfs commit in 5 seconds.

However, oprofile results are terrible for COS=1.

COS code profiling results:
== COS=1 ==

376924    4.5556  ptlrpc.ko                ptlrpc                   ptlrpc_server_handle_reply
288668    3.4889  vmlinux                  vmlinux                  .text.task_rq_lock
281528    3.4026  ptlrpc.ko                ptlrpc                   ldlm_resource_find
180107    2.1768  vmlinux                  vmlinux                  .text.__wake_up_common
173166    2.0929  obdclass.ko              obdclass                 htable_lookup
166829    2.0163  e1000e.ko                e1000e                   e1000_irq_enable
150765    1.8222  vmlinux                  vmlinux                  schedule
127059    1.5357  e1000e.ko                e1000e                   e1000_intr_msi
119343    1.4424  ptlrpc.ko                ptlrpc                   target_send_reply
118733    1.4350  lvfs.ko                  lvfs                     lprocfs_counter_add
99568     1.2034  vmlinux                  vmlinux                  __find_get_block
93356     1.1283  vmlinux                  vmlinux                  __switch_to
91259     1.1030  oprofiled                oprofiled                (no symbols)
91147     1.1016  vmlinux                  vmlinux                  try_to_wake_up
89665     1.0837  vmlinux                  vmlinux                  kfree
87774     1.0609  vmlinux                  vmlinux                  module_refcount
85149     1.0291  vmlinux                  vmlinux                  memset
84822     1.0252  vmlinux                  vmlinux                  mwait_idle
78850     0.9530  vmlinux                  vmlinux                  __kmalloc
70324     0.8500  ptlrpc.ko                ptlrpc                   ptlrpc_main
69333     0.8380  vmlinux                  vmlinux                  memmove

== COS=0 ==

237314    2.7642  e1000e.ko                e1000e                   e1000_irq_enable
211304    2.4613  obdclass.ko              obdclass                 htable_lookup
197515    2.3007  vmlinux                  vmlinux                  schedule
180431    2.1017  e1000e.ko                e1000e                   e1000_intr_msi
128113    1.4923  vmlinux                  vmlinux                  __switch_to
127683    1.4873  lvfs.ko                  lvfs                     lprocfs_counter_add
122777    1.4301  vmlinux                  vmlinux                  kfree
122584    1.4279  vmlinux                  vmlinux                  module_refcount
116800    1.3605  vmlinux                  vmlinux                  __find_get_block
110418    1.2862  vmlinux                  vmlinux                  mwait_idle
101909    1.1870  vmlinux                  vmlinux                  memset
101100    1.1776  oprofiled                oprofiled                (no symbols)
96397     1.1228  vmlinux                  vmlinux                  __kmalloc
91745     1.0686  ptlrpc.ko                ptlrpc                   ptlrpc_main
80605     0.9389  ksocklnd.ko              ksocklnd                 ksocknal_scheduler
79782     0.9293  vmlinux                  vmlinux                  kmem_cache_free
76968     0.8965  mds.ko                   mds                      mds_lov_dump_objids
75094     0.8747  vmlinux                  vmlinux                  .text.task_rq_lock
75087     0.8746  vmlinux                  vmlinux                  memmove
72627     0.8460  vmlinux                  vmlinux                  find_get_page
67242     0.7832  vmlinux                  vmlinux                  kmem_cache_alloc

== events distribution in ptlrpc_server_handle_reply, the most cpu consuming function ==

CPU: P4 / Xeon, speed 2800.2 MHz (estimated)
Counted GLOBAL_POWER_EVENTS events (time during which processor is not stopped)
with a unit mask of 0x01 (mandatory) count 100000

vma      samples  %        image name               app name                 symbol name
00000000000498f0 376924    4.5556  ptlrpc.ko                ptlrpc                   ptlrpc_server_handle_reply
  00000000000498f0 1        2.7e-04
  00000000000498f4 10        0.0027
  0000000000049905 3        8.0e-04
  0000000000049912 22        0.0058
  0000000000049972 1        2.7e-04
  0000000000049977 658       0.1746
  000000000004997e 24        0.0064
  0000000000049a23 5         0.0013
  0000000000049a35 21        0.0056
  0000000000049a80 1        2.7e-04
  0000000000049a85 4         0.0011
  0000000000049a92 2        5.3e-04
  0000000000049a99 22        0.0058
  0000000000049aa9 1        2.7e-04
  0000000000049ab5 371741   98.6249
  ...              ^^^^^^   spin_lock (&obd->obd_uncommitted_replies_lock);

I have a patch to avoid using of obd_uncommitted_replies_lock 
in ptlrpc_server_handle_reply but it has minimal effect, 
ptlrpc_server_handle_reply still the most cpu consuming function
because of svc->srv_lock contention.

I think the problem is that COS defers processing of replies to transaction commit time.
When commit happens, MDS has to process thousands of replies (about 14k replies per
commit in the test 3.a) in short period of time. I guess the mdt service threads 
all woken up and spin trying to get the service svr_lock. Processing of new requests may
also suffer of this.

I ran the tests with with CONFIG_DEBUG_SPINLOCK_SLEEP debugging compiled into a kernel, it found no
sleep under spinlock bugs.

Further optimization may include 
1. per-reply spin locks.
2. per-cpu structures and threads to process reply queues.

Any comments?

Thanks.

PS. the test results are much better when MDS server is sata20 machine with 4 cores
(the MDS from Washie1 has 2 cores), COS=0 and COS=1 have only %3 difference:

COS=1
Rate: 3101.77 creates/sec (total: 2 threads 930530 creates 300 secs)
Rate: 3096.94 creates/sec (total: 2 threads 929083 creates 300 secs)

COS=0
Rate: 3184.01 creates/sec (total: 2 threads 958388 creates 301 secs)
Rate: 3152.89 creates/sec (total: 2 threads 945868 creates 300 secs)

-- 
Alexander "Zam" Zarochentsev
Staff Engineer
Lustre Group, Sun Microsystems

From Alex.Zhuravlev at Sun.COM  Wed Oct  8 11:48:50 2008
From: Alex.Zhuravlev at Sun.COM (Alex Zhuravlev)
Date: Wed, 08 Oct 2008 15:48:50 +0400
Subject: [Lustre-devel] COS performance issues
In-Reply-To: <200810081544.08292.alexander.zarochentsev@sun.com>
References: <200810081544.08292.alexander.zarochentsev@sun.com>
Message-ID: <48EC9E22.2090403@sun.com>

try to profile with single CPU? you'll probably get an idea how "per-cpu"
approach can help.

thanks, Alex

Alexander Zarochentsev wrote:
> I have a patch to avoid using of obd_uncommitted_replies_lock 
> in ptlrpc_server_handle_reply but it has minimal effect, 
> ptlrpc_server_handle_reply still the most cpu consuming function
> because of svc->srv_lock contention.
> 
> I think the problem is that COS defers processing of replies to transaction commit time.
> When commit happens, MDS has to process thousands of replies (about 14k replies per
> commit in the test 3.a) in short period of time. I guess the mdt service threads 
> all woken up and spin trying to get the service svr_lock. Processing of new requests may
> also suffer of this.
> 
> I ran the tests with with CONFIG_DEBUG_SPINLOCK_SLEEP debugging compiled into a kernel, it found no
> sleep under spinlock bugs.
> 
> Further optimization may include 
> 1. per-reply spin locks.
> 2. per-cpu structures and threads to process reply queues.
> 
> Any comments?
> 
> Thanks.
> 
> PS. the test results are much better when MDS server is sata20 machine with 4 cores
> (the MDS from Washie1 has 2 cores), COS=0 and COS=1 have only %3 difference:
> 
> COS=1
> Rate: 3101.77 creates/sec (total: 2 threads 930530 creates 300 secs)
> Rate: 3096.94 creates/sec (total: 2 threads 929083 creates 300 secs)
> 
> COS=0
> Rate: 3184.01 creates/sec (total: 2 threads 958388 creates 301 secs)
> Rate: 3152.89 creates/sec (total: 2 threads 945868 creates 300 secs)
> 


From Peter.Braam at Sun.COM  Thu Oct  9 14:04:08 2008
From: Peter.Braam at Sun.COM (Peter Braam)
Date: Thu, 09 Oct 2008 08:04:08 -0600
Subject: [Lustre-devel] MDWBC and how much to trust clients
In-Reply-To: <18667.10276.857149.242018@gargle.gargle.HOWL>
Message-ID: <C5136B78.6085%peter.braam@sun.com>

You'll need to limit this to the requests that have dependencies.  With the
algorithm below every server starts looking at every request - that probably
kills the scaling you want to achieve.

Peter


On 10/7/08 3:13 AM, "Nikita Danilov" <Nikita.Danilov at Sun.COM> wrote:

> Nikita Danilov writes:
>> Eric Barton writes:
>>> Nikita,
>> 
>> Hello,
> 
> [...]
> 
>> 
>> as Peter mentioned, we discussed this topic during the Moscow
>> meeting. If I am not mistaken, we converged to the idea that before
>> committing an epoch, every mdt composes some kind of a `summary',
>> containing enough information for verification of a global consistency,
>> and this summary is passed though every server as a ticket, with every
> 
> This can be simplified. Suppose total amount of `data', describing all
> updates within given epoch is D, and there are N md servers in a cmd
> cluster. Then total network traffic incurred by this algorithm is
> 
>              D   /* updates from client to all servers */ +
>              D*N /* cycle summary through all servers */
> 
> that is, (N + 1)*D bytes, transferred in 2*N messages. So we won't
> increase network traffic by broadcasting _all_ epoch updates to _every_
> server (so that each server gets complete set of all updates within the
> epoch). In this latter case, servers can prove that epoch is consistent
> by
> 
>     - checking global consistency locally,
> 
>     - calculating md5 signature of all epoch updates, and
> 
>     - exchanging these signatures, to check that client sent the same
>       set of updates to everybody.
> 
> This results in
> 
>              D*N /* broadcast epoch updates to all servers */ +
>              e*N /* exchange signatures */
> 
> that is N*(D + e), for some small e, bytes transferred in 2*N
> messages. Having complete set of updates on every server would probably
> help in other places too.
> 
> 
>> server `approving' some bits in the summary accumulated so far, and
> 
> [...]
> 
>>> 
>>> Thoughts?
>>> 
>>>     Cheers,
>>>               Eric
>> 
> 
> Nikita.



From Nikita.Danilov at Sun.COM  Thu Oct  9 16:13:18 2008
From: Nikita.Danilov at Sun.COM (Nikita Danilov)
Date: Thu, 09 Oct 2008 20:13:18 +0400
Subject: [Lustre-devel] MDWBC and how much to trust clients
In-Reply-To: <C5136B78.6085%peter.braam@sun.com>
References: <18667.10276.857149.242018@gargle.gargle.HOWL>
	<C5136B78.6085%peter.braam@sun.com>
Message-ID: <18670.11678.435196.734601@gargle.gargle.HOWL>

Peter Braam writes:
 > You'll need to limit this to the requests that have dependencies.  With the
 > algorithm below every server starts looking at every request - that probably
 > kills the scaling you want to achieve.

I agree that total amount of data can be reduced significantly, but
won't it be sometimes useful to have complete epoch state on all
servers? E.g., we can do server->server replay forward instead of a roll
back.

After all, additional requests are only `looked at' rather than actually
processed. Moreover, global consistency check can be done by one server
only (selected round-robin for each epoch), after which this server
sends md5 signature of total epoch state to other servers to verify.

 > 
 > Peter

Nikita.

 > 
 > 
 > On 10/7/08 3:13 AM, "Nikita Danilov" <Nikita.Danilov at Sun.COM> wrote:
 > 
 > > Nikita Danilov writes:
 > >> Eric Barton writes:
 > >>> Nikita,
 > >> 
 > >> Hello,
 > > 
 > > [...]
 > > 
 > >> 
 > >> as Peter mentioned, we discussed this topic during the Moscow
 > >> meeting. If I am not mistaken, we converged to the idea that before
 > >> committing an epoch, every mdt composes some kind of a `summary',
 > >> containing enough information for verification of a global consistency,
 > >> and this summary is passed though every server as a ticket, with every
 > > 
 > > This can be simplified. Suppose total amount of `data', describing all
 > > updates within given epoch is D, and there are N md servers in a cmd
 > > cluster. Then total network traffic incurred by this algorithm is
 > > 
 > >              D   /* updates from client to all servers */ +
 > >              D*N /* cycle summary through all servers */
 > > 
 > > that is, (N + 1)*D bytes, transferred in 2*N messages. So we won't
 > > increase network traffic by broadcasting _all_ epoch updates to _every_
 > > server (so that each server gets complete set of all updates within the
 > > epoch). In this latter case, servers can prove that epoch is consistent
 > > by
 > > 
 > >     - checking global consistency locally,
 > > 
 > >     - calculating md5 signature of all epoch updates, and
 > > 
 > >     - exchanging these signatures, to check that client sent the same
 > >       set of updates to everybody.
 > > 
 > > This results in
 > > 
 > >              D*N /* broadcast epoch updates to all servers */ +
 > >              e*N /* exchange signatures */
 > > 
 > > that is N*(D + e), for some small e, bytes transferred in 2*N
 > > messages. Having complete set of updates on every server would probably
 > > help in other places too.
 > > 
 > > 
 > >> server `approving' some bits in the summary accumulated so far, and
 > > 
 > > [...]
 > > 
 > >>> 
 > >>> Thoughts?
 > >>> 
 > >>>     Cheers,
 > >>>               Eric
 > >> 
 > > 
 > > Nikita.
 > 
 > 
 > _______________________________________________
 > Lustre-devel mailing list
 > Lustre-devel at lists.lustre.org
 > http://lists.lustre.org/mailman/listinfo/lustre-devel

From Nathan.Rutman at Sun.COM  Thu Oct  9 19:11:10 2008
From: Nathan.Rutman at Sun.COM (Nathaniel Rutman)
Date: Thu, 09 Oct 2008 12:11:10 -0700
Subject: [Lustre-devel] HSM cache-miss locking
In-Reply-To: <20081009080425.GJ2009@webber.adilger.int>
References: <20081009080425.GJ2009@webber.adilger.int>
Message-ID: <48EE574E.2010808@sun.com>


Andreas Dilger wrote:
> Nathan,
> Eric and I had a lengthy discussion today about HSM and the copy-in
> process.  This was largely driven by Braam's assertion that having a
> copy-in process that blocks all access to the file data is not sufficient
> to meet customer demands.  Some customers require processes be able to
> access the file data as soon as it is present in the objects.
>
> Eric and I both agreed that we want to start with as simple an HSM solution
> as possible and incrementally provide improvements, so long as the early
> implementation is not a "throw-away" that consumes significant developer
> resources but doesn't provide long term benefits.  In both the "simple"
> and the "complex" copy-in the client has no knowledge/participation
> of the process being done by the HSM/coordinator.
>
> We both agreed that the simplest copy-in process is a reasonable starting
> point and can be used by many customers.  To review the simple case
> (I hope this also matches your recollection):
>
> 1) client tries to access a file that has been purged
>   a) if client is only doing getattr, attributes can be returned from MDS
>     - MDS holds file size[*]
>     - client may get MDS attribute read locks, but not layout lock
>     -> DONE
>   b) if client is trying to do an open (read or write)
>     - layout lock is required by client to do any read/write of the file
>     - client enqueues layout lock request
>     - MDS notices that file is purged, does upcall to coordinator to
>       start copy-in on FID N
>   
s/does upcall/asks/.  We expect coordinator to be in-kernel for LNET 
comms to agents.
> 2) client is blocked waiting for layout lock
>   - if MDS crashes at this point, client will resend open to MDS, goto 1b
>   - MDS should send early replies indicating lock will be slow to grant
>   
The reply to the layout lock request includes a "wait forever" flag 
(this is the one client code change required for HSM at this point.)  
There are no early replies for lock enqueue requests.  Maybe indefinite 
ongoing early replies for lock enqueus are a requirement for HSM copyin?
>   ? need to have a mechanism to ensure copy-in hasn't failed?
>   
Coordinator needs to decide is copy-in has failed, and redistribute the 
request to a new agent. (Needs detail: timer?  progress messages from 
agent?) There's nothing the client or MDT can do at this point (except 
fail the request), so we may as well just wait.
> 3) coordinator contacts agent(s) to retrieve FID N from HSM
>   - agent(s) create temp file X (new or backed-up layout parameters) [!]
>   
backed up in EA with original copyout request.  We should try to respect 
specific layout settings (pool, stripecount, stripesize), but be 
flexible if e.g. pool doesn't exist anymore.  Maybe we want to ignore 
offset and/or specific ost allocations in order to rebalance.
>   - agent(s) restore data into temp file
>   - agent or coordinator do ioctl on file to move file X objects to
>     file N, old objects are destroyed on file close, or 
>   - agent or coordinator do ioctl on file to notify MDS copy-in is done
>   
I was thinking the latter, and MDT moves the layout from X to N.
> 4) MDS handles ioctl, drops layout lock
> 5) client(s) waiting on layout lock are granted the layout lock by MDS
>   - client(s) get OST extent locks
>   - client(s) read/write file data
>   -> DONE
>
> [*] The MDS will already store the file size today, even without SOM, if
>     the file does not have any objects/striping.  If SOM is not implemented
>     then the "purged" state and object removal (with destroy llog entries)
>     would need to be a synchronous operation BEFORE the objects are actually
>     destroyed.  Otherwise, SOM-like recovery of the object purge state is
>     needed.  Avoiding the sync is desirable, but making HSM dependent upon
>     SOM is undesirable.
>   
All we really have to do is insure that the destroy llog entry is 
committed, right?  Then the OSTs should eventually purge the objects 
during orphan recovery, yes?
> [!] If MDS kept layout then it could pre-create the temp file and pass the
>     restore-to FID to the coordinator/agent, to keep agent more similar to
>     "complex" case where it is restoring directly into real file.  The only
>     reason the agent is restoring into the temp file is to avoid needing
>     to open the file while the MDS is blocking layout lock access, but maybe
>     that isn't a big obstacle (e.g. open flag).
>   
You mean open flag O_IGNORE_LAYOUT_LOCK?  So the one problem I see with 
this is the case of a stuck agent - if we want to start another agent 
doing copyin we have to insure that the first agent doesn't try to write 
anything else.  Or we give them two separate temp files, but this 
remains a problem with the direct restore into real file case.  Although 
I suppose this is already handled by write extent locks and eviction...
> In the "complex" case, the clients should be able to read/write the file
> data as soon as possible and the OSTs need to prevent access to the parts
> of the file which have not yet been restored.
>
> 1) client tries to access a file that has been purged
>   a) if client is only doing getattr, attributes can be returned from MDS
>     - MDS holds file size[*]
>     - client may get MDS attribute read locks, but not layout lock
>     -> DONE
>   b) if client is trying to do an open (read or write)
>     - layout lock is required by client to do any read/write of the file
>     - client enqueues layout lock request
>   
           -  MDT generates new layout based on old lov EA, assigning 
newly created OST objects.
>     - MDS grants layout lock to client
> 2) client enqueues extent lock on OST
>     - object was previously marked fully/partly invalid during purge
>     - object may have persistent invalid map of extent(s) that indicate
>       which parts of object require copy-in
>   
I'll read this as if you're proposing your 2,3 (call it "per-object 
invalid ranges held on OSTs") as a new method to do the copyin 
in-place.  This is not the original in-place idea proposed in Menlo Park 
(see below), and so I'll comment with an eye toward the differences.
 
I think we can't assume we're restoring back to the original OSTs. 
Therefore the MDT must create new empty objects on the OSTs and have the 
OSTs mark them purged before the layout lock can be granted to the clients.
>     - access to invalid parts of object trigger copy-in upcall to coordinator
>   
Now we need to figure out how to map the object back to a particular 
range extent of a particular file (are we storing this in an EA with 
each object now?)  We also need to initiate OST->coordinator 
communication, so either coordinator becomes a distributed function on 
the OSTs or we need new services going the reverse of the normal 
mdt->ost direction.  Maybe the coordinator-as-distributed-function works 
- the coordinators must all choose the same agent for objects belonging 
to the same file, yet distribute load among agents: I think the 
coordinator just got a lot more complicated.
>     ? group locks on invalid part of file block writes to missing data
>   
The issue here is that we can't allow any client to write and then have 
the agent overwrite the new data with old data being restored.  So we 
could have the OST give a group lock to agent via coordinator, 
preventing all other writes.  But it seems that we can check the special 
"clear invalid" flag used by the agent (see (3) below), and silently 
drop writes into areas not in the "invalid extents" list.  Any client 
write to any extent will clear the invalid flag for those extents.  And 
then we only ever need to block on reading.
What about reads to missing data?  OST refuses to grant read locks on 
invalid extents, needs clients to wait forever.
>     - clients block waiting on extent locks for invalid parts of objects
>   
We'll have to set this extent lock enqueue timeout to wait forever.
>     - OST crash at this timek restarts enqueue process
>   
Agent crash will still have to be detected and restarted by coordinator

> 3) coordinator contacts agent(s) to retrieve FID N from HSM
>     - agents write to actual object to be restored with "clear invalid" flag
>     - writes by agent shrink invalid extent, periodically update on-disk
>       invalid extent and release locks on that part of file (on commit?)
>   
The OST should keep track of all invalid extents.  Invalid extents list 
changes should be stored on disk, transactionally with the data write.
>     - client or agent agent crash doesn't want to access parts of multi-
>       part archive it will
>   
??

Invalid extents list will be accurate regardless of client, agent, or 
OST crash.  I hope.  Subsequent requests to missing data will result in 
new OST requests to coordinator.
> 4) client is granted extent lock when that part of file is copied in
>   

So that actually doesn't sound too bad.  I think the original idea of 
keeping the locking (and the coordinator) on the MDT (below) is still 
simpler, but I think it's going to be the recovery issues that decide 
this one way or the other.

Original in-place copyin idea:
When MDT generates new layout, it takes PW write locks on all extents of 
every stripe on behalf of the agent, and then somehow transfers these 
locks to the agent (this transferability was the point of using the 
group lock).  The agent then releases extent locks as it copies in data.
This was the first design we discussed in Menlo Park:

    (older idea, for posterity)
    Open intent enques layout lock.  MDT checks "purged" bit; if purged,
    MDT selects new layout and populates MD.  MDT takes group extent
    locks on all objects, then grants layout read lock to client,
    allowing open to finish successfully, quickly.  (Client reads/writes
    will block forever on extents enqueues until group lock has been
    dropped.)  MDT then sends request to coordinator requesting copyin
    FID XXXX with group lock id YYYY (and extents 0-end).  Coordinator
    distributes that request to an appropriate agent.  Agent retrieves
    file from HSM and writes into /.lustre/fid/XXXX:XXXX using group
    lock YYYY.  Agent takes group lock, MDT still holds group lock. 
    When finished, the agent clears "purged" bit from EA, and drops the
    group lock.  Clearing purged bit causes MDT to drop group lock as
    well, allowing the client to read/write.

It gets fuzzy at the end there, about exactly when the MDT drops the 
group lock in order to do handle the dead agent case.  It seems the safe 
thing to do is for the MDT to keep it until the agent is done, but then 
this blocks access to completed extents.  If the MDT drops the group 
lock as soon as the agent takes it, then somehow the agent converts the 
group lock to regular write lock, then other clients can get read/write 
locks on released extents.  But if the agent dies, the extent locks will 
be freed at eviction, and other clients are free to start reading 
(missing) data.


From eeb at sun.com  Thu Oct  9 20:38:38 2008
From: eeb at sun.com (Eric Barton)
Date: Thu, 09 Oct 2008 21:38:38 +0100
Subject: [Lustre-devel] proposal to increase seek in sgpdd_survey
In-Reply-To: <1222817771.26779.75.camel@pc.ilinx>
References: <1222817771.26779.75.camel@pc.ilinx>
Message-ID: <244CBACA73094649889DFE28A43F7B2E@ebpc>

Brian, Andreas,

I'm cc-ing lustre-devel since this is of general interest.

My comments inline...

> -----Original Message-----
> From: Brian.Murrell at Sun.COM [mailto:Brian.Murrell at Sun.COM] 
> Sent: 01 October 2008 12:36 AM
> To: Andreas.Dilger
> Cc: Eric.Barton at Sun.COM
> Subject: proposal to increase seek in sgpdd_survey
> 
> Andreas, Eric,
> 
> [ I can repost this to lustre-eng or some other place  or to other
> people if you think that a wider/alternate audience is more
> appropriate ]
> 
> We have been asked about considering increasing the seek range on
> multi-region sgpdd_survey tests and I wonder what others' thoughts are.
> 
> Currently, even with multi-region spgdd_survey runs, if the total amount
> of data the survey writes, across all regions is only a small portion of
> the total size of the drive, the results don't factor in what could be
> arguably real world seek penalties on a large drive.
> 
> The proposal is to space out the regions to cover more of the drive.  So
> for example, a 2 region test would write one region at 0% of the drive
> and the second region at 50%.  A 4 region test would write at 0%, 25%,
> 50% and 75%, etc.  Perhaps the inter-region spacing should be such that
> the last byte of the last region is at the end of the disk to maximize
> the seek penalty.  Perhaps not.

Indeed - this is how it should be done.

> In any case, a patch has been submitted in bug 17218 to do just this and
> the reporter's test results and summary are as such:
> 
>         Original test, size=1GB, 128KB reads:
>         0     rd1     rd2     rd4     rd8     rd16    rd32    rd64    
>         1     65.45   -       -       -       -       -       -       
>         2     65.90   18.08   -       -       -       -       -       
>         4     65.98   54.73   20.62   -       -       -       -       
>         8     65.27   64.75   45.56   20.38   -       -       -       
>         16    62.85   59.49   54.38   41.72   20.90   -       -       
>         32    -       58.75   56.54   53.02   38.78   21.94   -       
>         64    -       -       54.47   56.99   52.72   37.94   26.78   
>         128   -       -       -       -       52.22   -       37.84 
>         
>         Original test, size=32GB, 128KB reads (reduced parameters for
>         speed):
>         0     rd1     rd2     rd4     rd8     rd16    rd32    
>         1     -       -       -       -       -       -       
>         2     -       -       -       -       -       -       
>         4     -       -       -       -       -       -       
>         8     -       -       -       -       -       -       
>         16    -       -       -       -       18.96   -       
>         32    -       -       -       -       30.10   20.16   
>         64    -       -       -       -       40.99   -  
>         
>         
>         Modified test, size=1GB, 128KB reads:
>         0     rd1     rd2     rd4     rd8     rd16    rd32    rd64    
>         1     65.00   -       -       -       -       -       -       
>         2     61.65   16.22   -       -       -       -       -       
>         4     57.88   50.18   15.47   -       -       -       -       
>         8     56.95   54.91   47.47   17.09   -       -       -       
>         16    65.54   46.22   51.34   39.85   18.99   -       -       
>         32    -       55.17   51.49   47.62   29.05   19.81   -       
>         64    -       -       50.44   46.41   39.21   26.35   20.23   
>         128   -       -       -       -       -       -       24.74  
>         
>         Modified test, size=32GB, 128KB reads (reduced parameters for
>         speed):
>         0     rd1     rd2     rd4     rd8     rd16    rd32    rd64    
>         1     -       -       -       -       -       -       -       
>         2     -       -       -       -       -       -       -       
>         4     -       -       -       -       -       -       -       
>         8     -       -       -       -       -       -       -       
>         16    -       -       -       -       18.79   -       -       
>         32    -       -       -       -       29.15   19.92   -       
>         64    -       -       -       -       40.90   27.61   20.47
>         
>         
>         Note that these results behave exactly as expected: the 32 GB
>         run is very close to the new 1GB run (and the new 32GB run).
>         The one abnormality, the original 1GB run, has exaggerated
>         performance, and should not be used as it is a very poor
>         predictor of performance.
> 
> So my question for you is, does this proposal match the intent of
> sgpdd_survey or was it intended to be a pure speed measure, minimizing
> seeking cost?

This proposal _totally_ matches the original intent.  This feature of
the current implementation is a bug.  Mea maxima culpa...

    Cheers,
              Eric


From Nathan.Rutman at Sun.COM  Thu Oct  9 21:05:24 2008
From: Nathan.Rutman at Sun.COM (Nathaniel Rutman)
Date: Thu, 09 Oct 2008 14:05:24 -0700
Subject: [Lustre-devel] HSM cache-miss locking
In-Reply-To: <48EE574E.2010808@sun.com>
References: <20081009080425.GJ2009@webber.adilger.int>
	<48EE574E.2010808@sun.com>
Message-ID: <48EE7214.4080200@sun.com>

Note that Andreas' simple vs. complex case seems to fundamentally affect 
the design of the coordinator (whether it is associated with the MDT or 
the OSTs), and so I don't see a clear non-throw-away path from one to 
the other.   I think the "original in-place copyin" idea is more 
compatible with the simple case.  Also note that Braam also posited that 
the copyin at open is a desired simplification for the "Simplified HSM 
for Lustre" (lustre-devel 7/16).

Nathaniel Rutman wrote:
> Andreas Dilger wrote:
>   
>> Nathan,
>> Eric and I had a lengthy discussion today about HSM and the copy-in
>> process.  This was largely driven by Braam's assertion that having a
>> copy-in process that blocks all access to the file data is not sufficient
>> to meet customer demands.  Some customers require processes be able to
>> access the file data as soon as it is present in the objects.
>>
>> Eric and I both agreed that we want to start with as simple an HSM solution
>> as possible and incrementally provide improvements, so long as the early
>> implementation is not a "throw-away" that consumes significant developer
>> resources but doesn't provide long term benefits.  In both the "simple"
>> and the "complex" copy-in the client has no knowledge/participation
>> of the process being done by the HSM/coordinator.
>>
>> We both agreed that the simplest copy-in process is a reasonable starting
>> point and can be used by many customers.  To review the simple case
>> (I hope this also matches your recollection):
>>
>> 1) client tries to access a file that has been purged
>>   a) if client is only doing getattr, attributes can be returned from MDS
>>     - MDS holds file size[*]
>>     - client may get MDS attribute read locks, but not layout lock
>>     -> DONE
>>   b) if client is trying to do an open (read or write)
>>     - layout lock is required by client to do any read/write of the file
>>     - client enqueues layout lock request
>>     - MDS notices that file is purged, does upcall to coordinator to
>>       start copy-in on FID N
>>   
>>     
> s/does upcall/asks/.  We expect coordinator to be in-kernel for LNET 
> comms to agents.
>   
>> 2) client is blocked waiting for layout lock
>>   - if MDS crashes at this point, client will resend open to MDS, goto 1b
>>   - MDS should send early replies indicating lock will be slow to grant
>>   
>>     
> The reply to the layout lock request includes a "wait forever" flag 
> (this is the one client code change required for HSM at this point.)  
> There are no early replies for lock enqueue requests.  Maybe indefinite 
> ongoing early replies for lock enqueus are a requirement for HSM copyin?
>   
>>   ? need to have a mechanism to ensure copy-in hasn't failed?
>>   
>>     
> Coordinator needs to decide is copy-in has failed, and redistribute the 
> request to a new agent. (Needs detail: timer?  progress messages from 
> agent?) There's nothing the client or MDT can do at this point (except 
> fail the request), so we may as well just wait.
>   
>> 3) coordinator contacts agent(s) to retrieve FID N from HSM
>>   - agent(s) create temp file X (new or backed-up layout parameters) [!]
>>   
>>     
> backed up in EA with original copyout request.  We should try to respect 
> specific layout settings (pool, stripecount, stripesize), but be 
> flexible if e.g. pool doesn't exist anymore.  Maybe we want to ignore 
> offset and/or specific ost allocations in order to rebalance.
>   
>>   - agent(s) restore data into temp file
>>   - agent or coordinator do ioctl on file to move file X objects to
>>     file N, old objects are destroyed on file close, or 
>>   - agent or coordinator do ioctl on file to notify MDS copy-in is done
>>   
>>     
> I was thinking the latter, and MDT moves the layout from X to N.
>   
>> 4) MDS handles ioctl, drops layout lock
>> 5) client(s) waiting on layout lock are granted the layout lock by MDS
>>   - client(s) get OST extent locks
>>   - client(s) read/write file data
>>   -> DONE
>>
>> [*] The MDS will already store the file size today, even without SOM, if
>>     the file does not have any objects/striping.  If SOM is not implemented
>>     then the "purged" state and object removal (with destroy llog entries)
>>     would need to be a synchronous operation BEFORE the objects are actually
>>     destroyed.  Otherwise, SOM-like recovery of the object purge state is
>>     needed.  Avoiding the sync is desirable, but making HSM dependent upon
>>     SOM is undesirable.
>>   
>>     
> All we really have to do is insure that the destroy llog entry is 
> committed, right?  Then the OSTs should eventually purge the objects 
> during orphan recovery, yes?
>   
>> [!] If MDS kept layout then it could pre-create the temp file and pass the
>>     restore-to FID to the coordinator/agent, to keep agent more similar to
>>     "complex" case where it is restoring directly into real file.  The only
>>     reason the agent is restoring into the temp file is to avoid needing
>>     to open the file while the MDS is blocking layout lock access, but maybe
>>     that isn't a big obstacle (e.g. open flag).
>>   
>>     
> You mean open flag O_IGNORE_LAYOUT_LOCK?  So the one problem I see with 
> this is the case of a stuck agent - if we want to start another agent 
> doing copyin we have to insure that the first agent doesn't try to write 
> anything else.  Or we give them two separate temp files, but this 
> remains a problem with the direct restore into real file case.  Although 
> I suppose this is already handled by write extent locks and eviction...
>   
>> In the "complex" case, the clients should be able to read/write the file
>> data as soon as possible and the OSTs need to prevent access to the parts
>> of the file which have not yet been restored.
>>
>> 1) client tries to access a file that has been purged
>>   a) if client is only doing getattr, attributes can be returned from MDS
>>     - MDS holds file size[*]
>>     - client may get MDS attribute read locks, but not layout lock
>>     -> DONE
>>   b) if client is trying to do an open (read or write)
>>     - layout lock is required by client to do any read/write of the file
>>     - client enqueues layout lock request
>>   
>>     
>            -  MDT generates new layout based on old lov EA, assigning 
> newly created OST objects.
>   
>>     - MDS grants layout lock to client
>> 2) client enqueues extent lock on OST
>>     - object was previously marked fully/partly invalid during purge
>>     - object may have persistent invalid map of extent(s) that indicate
>>       which parts of object require copy-in
>>   
>>     
> I'll read this as if you're proposing your 2,3 (call it "per-object 
> invalid ranges held on OSTs") as a new method to do the copyin 
> in-place.  This is not the original in-place idea proposed in Menlo Park 
> (see below), and so I'll comment with an eye toward the differences.
>  
> I think we can't assume we're restoring back to the original OSTs. 
> Therefore the MDT must create new empty objects on the OSTs and have the 
> OSTs mark them purged before the layout lock can be granted to the clients.
>   
>>     - access to invalid parts of object trigger copy-in upcall to coordinator
>>   
>>     
> Now we need to figure out how to map the object back to a particular 
> range extent of a particular file (are we storing this in an EA with 
> each object now?)  We also need to initiate OST->coordinator 
> communication, so either coordinator becomes a distributed function on 
> the OSTs or we need new services going the reverse of the normal 
> mdt->ost direction.  Maybe the coordinator-as-distributed-function works 
> - the coordinators must all choose the same agent for objects belonging 
> to the same file, yet distribute load among agents: I think the 
> coordinator just got a lot more complicated.
>   
>>     ? group locks on invalid part of file block writes to missing data
>>   
>>     
> The issue here is that we can't allow any client to write and then have 
> the agent overwrite the new data with old data being restored.  So we 
> could have the OST give a group lock to agent via coordinator, 
> preventing all other writes.  But it seems that we can check the special 
> "clear invalid" flag used by the agent (see (3) below), and silently 
> drop writes into areas not in the "invalid extents" list.  Any client 
> write to any extent will clear the invalid flag for those extents.  And 
> then we only ever need to block on reading.
> What about reads to missing data?  OST refuses to grant read locks on 
> invalid extents, needs clients to wait forever.
>   
>>     - clients block waiting on extent locks for invalid parts of objects
>>   
>>     
> We'll have to set this extent lock enqueue timeout to wait forever.
>   
>>     - OST crash at this timek restarts enqueue process
>>   
>>     
> Agent crash will still have to be detected and restarted by coordinator
>
>   
>> 3) coordinator contacts agent(s) to retrieve FID N from HSM
>>     - agents write to actual object to be restored with "clear invalid" flag
>>     - writes by agent shrink invalid extent, periodically update on-disk
>>       invalid extent and release locks on that part of file (on commit?)
>>   
>>     
> The OST should keep track of all invalid extents.  Invalid extents list 
> changes should be stored on disk, transactionally with the data write.
>   
>>     - client or agent agent crash doesn't want to access parts of multi-
>>       part archive it will
>>   
>>     
> ??
>
> Invalid extents list will be accurate regardless of client, agent, or 
> OST crash.  I hope.  Subsequent requests to missing data will result in 
> new OST requests to coordinator.
>   
>> 4) client is granted extent lock when that part of file is copied in
>>   
>>     
>
> So that actually doesn't sound too bad.  I think the original idea of 
> keeping the locking (and the coordinator) on the MDT (below) is still 
> simpler, but I think it's going to be the recovery issues that decide 
> this one way or the other.
>
> Original in-place copyin idea:
> When MDT generates new layout, it takes PW write locks on all extents of 
> every stripe on behalf of the agent, and then somehow transfers these 
> locks to the agent (this transferability was the point of using the 
> group lock).  The agent then releases extent locks as it copies in data.
> This was the first design we discussed in Menlo Park:
>
>     (older idea, for posterity)
>     Open intent enques layout lock.  MDT checks "purged" bit; if purged,
>     MDT selects new layout and populates MD.  MDT takes group extent
>     locks on all objects, then grants layout read lock to client,
>     allowing open to finish successfully, quickly.  (Client reads/writes
>     will block forever on extents enqueues until group lock has been
>     dropped.)  MDT then sends request to coordinator requesting copyin
>     FID XXXX with group lock id YYYY (and extents 0-end).  Coordinator
>     distributes that request to an appropriate agent.  Agent retrieves
>     file from HSM and writes into /.lustre/fid/XXXX:XXXX using group
>     lock YYYY.  Agent takes group lock, MDT still holds group lock. 
>     When finished, the agent clears "purged" bit from EA, and drops the
>     group lock.  Clearing purged bit causes MDT to drop group lock as
>     well, allowing the client to read/write.
>
> It gets fuzzy at the end there, about exactly when the MDT drops the 
> group lock in order to do handle the dead agent case.  It seems the safe 
> thing to do is for the MDT to keep it until the agent is done, but then 
> this blocks access to completed extents.  If the MDT drops the group 
> lock as soon as the agent takes it, then somehow the agent converts the 
> group lock to regular write lock, then other clients can get read/write 
> locks on released extents.  But if the agent dies, the extent locks will 
> be freed at eviction, and other clients are free to start reading 
> (missing) data.
>
> _______________________________________________
> Lustre-devel mailing list
> Lustre-devel at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-devel
>   


From eeb at sun.com  Thu Oct  9 21:54:01 2008
From: eeb at sun.com (Eric Barton)
Date: Thu, 09 Oct 2008 22:54:01 +0100
Subject: [Lustre-devel] Simplified Interoperation
Message-ID: <0EA267EF312B4BFA837511EFD41F4E6E@ebpc>

Hi,

I've written some notes on simplified interoperation which
you can find at...

http://arch.lustre.org/index.php?title=Simplified_Interoperation


    Cheers,
              Eric


From Nathan.Rutman at Sun.COM  Thu Oct  9 22:37:21 2008
From: Nathan.Rutman at Sun.COM (Nathaniel Rutman)
Date: Thu, 09 Oct 2008 15:37:21 -0700
Subject: [Lustre-devel] "Simple" HSM straw man
Message-ID: <48EE87A1.7000003@sun.com>

This is intended to be a starting point for discussion; the concepts 
here have been hashed through a few times and hopefully represent the 
best current thinking.


Baseline concepts
1. all single-file coherency issues are in kernel space (file locking, 
recovery)
2. all policy decisions are in user space (using changelogs, df, etc)
3. coordinator/mover communication will use LNET
4. "simple" refers to
    a. integration with HPSS only
    b. depends on changelog for policy decisions
    c. restore on file open, not data read/write
5. HSM tracks entire files, not stripe objects
6. HSM namespace is flat, all files are addressed by FID only
7. Desired: coordinator and movers can be reused by (non-HSM) replication


Components
1. Mover
    a. combined kernel (LNET comms) and userspace processes
    b. userspace processes will use Lustre clients for data i/o
    c. will use special fid directory for file access (.lustre/fid/XXXX)
    d. interfaces with hardware-specific copy tool to access HSM files
    e. kernel process encompasses service threads listening for 
coordinator requests, passes these up to userspace process via upcall.  
No interaction with the client is needed; this is a simple message 
passing service.
2. Coordinator
    a. decides and dispatches copyin and copyout requests to movers
    b. consolidates repeat requests
    c. re-queues requests to a new agent if an agent becomes unresponsive
    d. kernel space, associated with the MDT for cache-miss
    e. ioctl interface for copyout, purge requests from policy engine
3. Policy engine (aka Space Manager)
    a. makes policy decisions for copyout, purge
    b. normally uses changelogs and 'df' for input; rarely is allowed to 
scan filesystem
    c. userspace process, requests copyout and purge via ioctl to 
coordinator
4. MDT changes
    a. Per-file layout lock

    A new layout lock is created for every file.  Private writer lock is
    taken by the MDT when allocating/changing file layout (LOV EA). 
    Shared reader locks are taken by anyone reading the layout (client
    opens, lfs getstripe).  Anyone taking a new extent lock anywhere in
    the file must first hold the layout lock.
    Problem: Layout lock can't be held by liblustre during i/o? 

    b. lov EA changes
       i.  flags: file_is_purged "purged", copyout_begin, 
file_in_HSM_is_out_of_date "hsm_dirty", copyout_complete.  The purged 
flag is always manipulated under a write layout lock, the other flags 
are not.
       ii: "window" EA range of non-purged data (rev2)
    c. new file ioctls: HSMCopyOut, HSMPurge, HSMCopyinDone


Algorithms
1. copyout
    a. Policy engine decides to copy a file to HSM, executes HSMCopyOut 
ioctl on file
    b. ioctl handled by MDT, which passes request to Coordinator
    c. coordinator dispatches request to mover.  request should include 
file extents (for future purposes)
    d. normal extents read lock is taken by mover running on client
    e. mover sets "copyout_begin" bit and clears "hsm_dirty" bit in EA.
    f. any writes to the file set the "hsm_dirty" bit (may be 
lazy/delayed with mtime or filesize change updates on MDT).  Note that 
file writes need not cancel copyout; for a fs with a single big file, we 
don't want to keep interrupting copyout or it will never finish. 
    g. when done, mover checks hsm_dirty bit.  If set, clears 
copyout_begin, indicating current file is not in HSM.  If not set,  
mover sets "copyout_complete" bit.  File layout write lock is not taken 
during mover flag manipulation.  (Note: file modifications after copyout 
is complete will have both copyout_complete and hsm_dirty bits set.)

2. purge (aka punch)
    a. Policy engine decides to purge a file, exectues HSMPurge ioctl on 
file
    b. ioctl handled by MDT
    c. MDT takes a write lock on the file layout lock
    d. MDT enques write locks on all extents of the file.  After these 
are granted, then no client has any dirty cache and no child can take 
new extent locks until layout lock is released.  MDT drops all extent locks.
    e. MDT verifies that hsm_dirty bit is clear and copyout_complete bit 
is set
    f. MDT marks the LOV EA as "purged"
    g. MDT sends destroys the OST objects, using destroy llog entries to 
guard against object leakage during OST failover
    h. MDT drops layout lock.

3. restore (aka copyin aka cache miss)
    a. Client open intent enques layout read lock. 
    b. MDT checks "purged" bit; if purged, lock request response 
includes "wait forever" flag, causing client to block the open.
    c. MDT creates a new layout with a similar stripe pattern as the 
original, allocating new objects on new OSTs.  (We should try to respect 
specific layout settings (pool, stripecount, stripesize), but be 
flexible if e.g. pool doesn't exist anymore.  Maybe we want to ignore 
offset and/or specific ost allocations in order to rebalance.)
    d. MDT sends request to coordinator requesting copyin of the file to 
.lustre/fid/XXXX with extents 0-EOF. Extents may be used in the future 
to (a) copy in part of a file, in low-disk-space situations; (b) copy in 
individual stripes simultaneously on multiple OSTs.
    e. Coordinator distributes that request to an appropriate mover.
    f. Writes into .lustre/fid/* are not required to hold layout read 
lock (or special flag is passed to open, or group write lock on layout 
is passed to mover)
    g. Mover copies data from HSM
    h. When finished, mover calls ioctl HSM_COPYIN_DONE on the file
    i. MDT clears "purged" bit from LOV EA
    j. MDT releases the layout lock
    k. This sends a completion AST to the original client, who now 
completes his open. 


State machines
TBD - I think there's enough in here to chew on for awhile

Things requiring a more detailed look
1. configuration of HSM/movers
2. policy engine
3. "complex" HSM roadmap
    a. partial access to files during restore
    b. partial purging for file type identification, image thumbnails, ??
    c. integration with other HSM backends (ADM, ??)
4. layout locks and liblustre


From rread at sun.com  Thu Oct  9 22:57:20 2008
From: rread at sun.com (Robert Read)
Date: Thu, 09 Oct 2008 15:57:20 -0700
Subject: [Lustre-devel] Simplified Interoperation
In-Reply-To: <0EA267EF312B4BFA837511EFD41F4E6E@ebpc>
References: <0EA267EF312B4BFA837511EFD41F4E6E@ebpc>
Message-ID: <68D776E8-A7E1-45D5-8F06-C6A92418454C@sun.com>

Hi,

Thanks for summarizing this, comments inline.

> Description
> At the start of a controlled shutdown, the server notifies all its  
> connected clients that it is shutting down and refuses connection  
> requests from new (but not currently connected) clients.
>

Why refuse connections to new clients? Now that we are adding a  
quiescent mode to the client, we can use that instead of failing new  
mounts. (We could do the same thing when we receive new connection  
during recovery, too, for that matter.)  I'd just hate to add another  
source of mount failures when it seems we can avoid it.

> The clients prepare for shutdown by ensuring at a minimum that no  
> further requests are sent to the server and they have cleaned and  
> evicted all cached server state.
>

Clients need to notify the server when they are finished flushing state.

> The server notifies all clients when all outstanding requests have  
> been committed.
>
There is already a mechanism in place for the server to notify the  
clients of the last committed, so we don't need to add anything for  
this. I'm not convinced the server needs to do anything here except  
failover, but we could withhold the reply to the clients' "i'm done"  
request mentioned above until the server is ready to shutdown. That  
reply would have the current last committed and as a side-effect would  
cause the clients to flush their replay queues. The same thing will  
happen when the clients reconnect, though, so I'm not sure it's worth  
adding another special reply.

> The clients may then disconnect and the server can halt when all  
> clients have disconnected.
>
> When the server restarts, clients reconnect, replay open files and  
> proceed.
>


If the clients disconnect right away, then they will have no way of  
knowing when they need to reconnect. They need to remain connected and  
continue pinging so they will detect when the server has failed and  
recover normally.

One last thing - the clients need to know when it is safe to being  
sending new requests again. Do we do this automatically after  
recovery?  Or is this an explicit operation done by the admin? Also,  
the admin might decide the cancel the upgrade before failing the  
server, so we'll need a way to resume normal operations without going  
through recovery.

robert


On Oct 9, 2008, at 14:54 , Eric Barton wrote:

> Hi,
>
> I've written some notes on simplified interoperation which
> you can find at...
>
> http://arch.lustre.org/index.php?title=Simplified_Interoperation
>
>
>    Cheers,
>              Eric
>
> _______________________________________________
> Lustre-devel mailing list
> Lustre-devel at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-devel


From eeb at sun.com  Thu Oct  9 23:21:52 2008
From: eeb at sun.com (Eric Barton)
Date: Fri, 10 Oct 2008 00:21:52 +0100
Subject: [Lustre-devel] Simplified Interoperation
In-Reply-To: <68D776E8-A7E1-45D5-8F06-C6A92418454C@sun.com>
References: <0EA267EF312B4BFA837511EFD41F4E6E@ebpc>
	<68D776E8-A7E1-45D5-8F06-C6A92418454C@sun.com>
Message-ID: <AF70C16AF3AF481287E4EBB781837B39@ebpc>

> Thanks for summarizing this, comments inline.
> 
> > Description
> > At the start of a controlled shutdown, the server notifies all its  
> > connected clients that it is shutting down and refuses connection  
> > requests from new (but not currently connected) clients.
> >
> 
> Why refuse connections to new clients? Now that we are adding a  
> quiescent mode to the client, we can use that instead of failing new  
> mounts. (We could do the same thing when we receive new connection  
> during recovery, too, for that matter.)  I'd just hate to add another  
> source of mount failures when it seems we can avoid it.

Just to prevent new clients connecting to it as if it wasn't there at
all - it's about to go in any case and a new server is about to start
up in its place, which these clients should shortly succeed in
connecting to.

> > The clients prepare for shutdown by ensuring at a minimum that no  
> > further requests are sent to the server and they have cleaned and  
> > evicted all cached server state.
> >
> 
> Clients need to notify the server when they are finished flushing state.

Yes indeed - e.g. releasing the lock used for the shutdown notification
BAST.

> > The server notifies all clients when all outstanding requests have  
> > been committed.
> >
> There is already a mechanism in place for the server to notify the  
> clients of the last committed, so we don't need to add anything for  
> this. I'm not convinced the server needs to do anything here except  
> failover, but we could withhold the reply to the clients' "i'm done"  
> request mentioned above until the server is ready to shutdown. That  
> reply would have the current last committed and as a side-effect would  
> cause the clients to flush their replay queues. The same thing will  
> happen when the clients reconnect, though, so I'm not sure it's worth  
> adding another special reply.

I really want the replay queue to be empty when the client disconnects.

> > The clients may then disconnect and the server can halt when all  
> > clients have disconnected.
> >
> > When the server restarts, clients reconnect, replay open files and  
> > proceed.
> 
> If the clients disconnect right away, then they will have no way of  
> knowing when they need to reconnect. They need to remain connected and  
> continue pinging so they will detect when the server has failed and  
> recover normally.

What's the difference between remaining connected and pinging, and
disconnecting and attempting reconnection?

> One last thing - the clients need to know when it is safe to being  
> sending new requests again. Do we do this automatically after  
> recovery?  

Yes.

> Or is this an explicit operation done by the admin? 

No.

> Also,  
> the admin might decide the cancel the upgrade before failing the  
> server, so we'll need a way to resume normal operations without going  
> through recovery.

We're not actually failing the server - we're just doing an orderly
shutdown that guarantees to minimize client state and simplify recovery
on reconnection.  Nothing bad happens if the server reboots with the
same version - the client just does the same minimal recovery it would
do with a version-upped server.

    Cheers,
              Eric



From Brian.Murrell at Sun.COM  Fri Oct 10 19:02:37 2008
From: Brian.Murrell at Sun.COM (Brian J. Murrell)
Date: Fri, 10 Oct 2008 15:02:37 -0400
Subject: [Lustre-devel] proposal to increase seek in sgpdd_survey
In-Reply-To: <244CBACA73094649889DFE28A43F7B2E@ebpc>
References: <1222817771.26779.75.camel@pc.ilinx>
	<244CBACA73094649889DFE28A43F7B2E@ebpc>
Message-ID: <1223665357.11840.138.camel@brian-laptop>

On Thu, 2008-10-09 at 21:38 +0100, Eric Barton wrote:
> Brian, Andreas,
> 
> I'm cc-ing lustre-devel since this is of general interest.

Sounds good.

> 
> My comments inline...

Mine too, as always.  :-)

I just want to clarify, this idea is not mine.  I don't want to take
credit for something that is not actually mine.  The idea comes from a
bug filed by one of our support engineers, Kevin Van Maren.

> > -----Original Message-----
> > From: Brian.Murrell at Sun.COM [mailto:Brian.Murrell at Sun.COM] 
> ...
> > The proposal is to space out the regions to cover more of the drive.  So
> > for example, a 2 region test would write one region at 0% of the drive
> > and the second region at 50%.  A 4 region test would write at 0%, 25%,
> > 50% and 75%, etc.  Perhaps the inter-region spacing should be such that
> > the last byte of the last region is at the end of the disk to maximize
> > the seek penalty.  Perhaps not.
> 
> Indeed - this is how it should be done.

Just for clarification Eric...  Are you advocating num_gaps ==
num_regions or num_gaps == num_regions-1?  i.e. in a two region test
should the second region start at 50% of the disk or 100%-$region_size
so that the last byte of the second region is written to the last byte
of the disk?

It seems to me that the first option provides a more "middle of the
road" (which might correlate to real-world averages more closely)
average as it only incorporates a seek of half the disk whereas the
second option provides a more worst case performance result.

> This proposal _totally_ matches the original intent.  This feature of
> the current implementation is a bug.

Fair enough.  :-)

>   Mea maxima culpa...

:-)

b.



From eeb at sun.com  Fri Oct 10 20:11:32 2008
From: eeb at sun.com (Eric Barton)
Date: Fri, 10 Oct 2008 21:11:32 +0100
Subject: [Lustre-devel] proposal to increase seek in sgpdd_survey
In-Reply-To: <1223665357.11840.138.camel@brian-laptop>
References: <1222817771.26779.75.camel@pc.ilinx>
	<244CBACA73094649889DFE28A43F7B2E@ebpc>
	<1223665357.11840.138.camel@brian-laptop>
Message-ID: <BC160A91CD43413FB9B898E767854EC2@ebpc>

Brian,

> Just for clarification Eric...  Are you advocating num_gaps ==
> num_regions or num_gaps == num_regions-1?  i.e. in a two region test
> should the second region start at 50% of the disk or 100%-$region_size
> so that the last byte of the second region is written to the last byte
> of the disk?
> 
> It seems to me that the first option provides a more "middle of the
> road" (which might correlate to real-world averages more closely)
> average as it only incorporates a seek of half the disk whereas the
> second option provides a more worst case performance result.

Not quite so bothered by that and it gets less intersting with more
regions.  Maybe leave it in as a non-default option, in case it's
a real disk killer.

    Cheers,
              Eric


From adilger at sun.com  Sat Oct 11 16:00:10 2008
From: adilger at sun.com (Andreas Dilger)
Date: Sat, 11 Oct 2008 10:00:10 -0600
Subject: [Lustre-devel] COS performance issues
In-Reply-To: <200810081544.08292.alexander.zarochentsev@sun.com>
References: <200810081544.08292.alexander.zarochentsev@sun.com>
Message-ID: <20081011160010.GR2009@webber.adilger.int>

On Oct 08, 2008  15:44 +0400, Alexander Zarochentsev wrote:
> I think the problem is that COS defers processing of replies to
> transaction commit time.  When commit happens, MDS has to process
> thousands of replies (about 14k replies per commit in the test 3.a)
> in short period of time. I guess the mdt service threads all woken
> up and spin trying to get the service svr_lock. Processing of new
> requests may also suffer of this.

Can you please explain what replies are being blocked?  It can't be the
create replies or the clients would be blocked waiting after starting a
single create each.

I think the thread and lock contention is only part of the issue - if all
of these replies are blocked until transaction commit this wastes all of
the bandwidth on the network while the replies are being held.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.


From eeb at sun.com  Sat Oct 11 20:18:35 2008
From: eeb at sun.com (Eric Barton)
Date: Sat, 11 Oct 2008 21:18:35 +0100
Subject: [Lustre-devel] COS performance issues
In-Reply-To: <20081011160010.GR2009@webber.adilger.int>
References: <200810081544.08292.alexander.zarochentsev@sun.com>
	<20081011160010.GR2009@webber.adilger.int>
Message-ID: <097E12B55A98421C880928B9F9A33DE7@ebpc>

Andreas,

Zam doesn't mean the reply messages being blocked - he means
the reply state that is holding lock references.  Previously
they got cleaned up one at a time as the repACKs came in.  Now
they get scheduled by the 1000s on the commit callback.  I'm
betting that we're getting thundering herds of MDS threads
at this time and that we should have dedicated 1-per-CPU threads
doing the cleanup to minimize interference with normal request
processing.  Bzzz thinks it's purely spinlock contention that's
the source of the inefficiency.  So Zam is going to run an
experiment with a single CPU MDS to see if the performance issue
goes away.  I'm betting it won't and Bzzz is betting it will.
We should see soon...

    Cheers,
              Eric

>  
> 
> -----Original Message-----
> From: Andreas.Dilger at Sun.COM [mailto:Andreas.Dilger at Sun.COM] On Behalf Of Andreas Dilger
> Sent: 11 October 2008 5:00 PM
> To: Alexander Zarochentsev
> Cc: lustre-devel at lists.lustre.org; lustre-recovery at Sun.COM
> Subject: Re: [Lustre-devel] COS performance issues
> 
> On Oct 08, 2008  15:44 +0400, Alexander Zarochentsev wrote:
> > I think the problem is that COS defers processing of replies to
> > transaction commit time.  When commit happens, MDS has to process
> > thousands of replies (about 14k replies per commit in the test 3.a)
> > in short period of time. I guess the mdt service threads all woken
> > up and spin trying to get the service svr_lock. Processing of new
> > requests may also suffer of this.
> 
> Can you please explain what replies are being blocked?  It can't be the
> create replies or the clients would be blocked waiting after starting a
> single create each.
> 
> I think the thread and lock contention is only part of the issue - if all
> of these replies are blocked until transaction commit this wastes all of
> the bandwidth on the network while the replies are being held.
> 
> Cheers, Andreas
> --
> Andreas Dilger
> Sr. Staff Engineer, Lustre Group
> Sun Microsystems of Canada, Inc.
> 
> 


From Alexander.Zarochentsev at Sun.COM  Sun Oct 12 15:51:56 2008
From: Alexander.Zarochentsev at Sun.COM (Alexander Zarochentsev)
Date: Sun, 12 Oct 2008 19:51:56 +0400
Subject: [Lustre-devel] COS performance issues
In-Reply-To: <20081011160010.GR2009@webber.adilger.int>
References: <200810081544.08292.alexander.zarochentsev@sun.com>
	<20081011160010.GR2009@webber.adilger.int>
Message-ID: <200810121951.56922.alexander.zarochentsev@sun.com>

On 11 October 2008 20:00:10 Andreas Dilger wrote:
> On Oct 08, 2008  15:44 +0400, Alexander Zarochentsev wrote:
> > I think the problem is that COS defers processing of replies to
> > transaction commit time.  When commit happens, MDS has to process
> > thousands of replies (about 14k replies per commit in the test 3.a)
> > in short period of time. I guess the mdt service threads all woken
> > up and spin trying to get the service svr_lock. Processing of new
> > requests may also suffer of this.
>
> Can you please explain what replies are being blocked?  It can't be
> the create replies or the clients would be blocked waiting after
> starting a single create each.
>
> I think the thread and lock contention is only part of the issue - if
> all of these replies are blocked until transaction commit this wastes
> all of the bandwidth on the network while the replies are being held.

as Eric explained already, replies are sent already but the reply state 
objects remain in memory.  The cleanup process is almost serialized by 
per-service spinlock. It might be only a part of the problem but the 
first task to solve.

> Cheers, Andreas

WBR,
-- 
Alexander "Zam" Zarochentsev
Staff Engineer
Lustre Group, Sun Microsystems

From Alexander.Zarochentsev at Sun.COM  Sun Oct 12 18:41:58 2008
From: Alexander.Zarochentsev at Sun.COM (Alexander Zarochentsev)
Date: Sun, 12 Oct 2008 22:41:58 +0400
Subject: [Lustre-devel] COS performance issues
In-Reply-To: <48EC9E22.2090403@sun.com>
References: <200810081544.08292.alexander.zarochentsev@sun.com>
	<48EC9E22.2090403@sun.com>
Message-ID: <200810122241.58982.alexander.zarochentsev@sun.com>

On 8 October 2008 15:48:50 Alex Zhuravlev wrote:
> try to profile with single CPU? you'll probably get an idea how
> "per-cpu" approach can help.

I booted the MDS server with maxcpus=1 kernel parameter and here are the 
results:

cos=0
2039.31 creates/sec (total: 2 threads 611794 creates 300 secs)
2037.80 creates/sec (total: 2 threads 611341 creates 300 secs)
2076.21 creates/sec (total: 2 threads 622864 creates 300 secs)

cos=1
1874.93 creates/sec (total: 2 threads 564354 creates 301 secs)
1923.97 creates/sec (total: 2 threads 577191 creates 300 secs)
1892.61 creates/sec (total: 2 threads 567783 creates 300 secs)
1874.74 creates/sec (total: 2 threads 562421 creates 300 secs)

unfortunately profiling info isn't available yet, the results are done 
with SLES10 which can boot with maxcpus=1 but has no oprofile 
installed.

> Alexander Zarochentsev wrote:
> > I have a patch to avoid using of obd_uncommitted_replies_lock
> > in ptlrpc_server_handle_reply but it has minimal effect,
> > ptlrpc_server_handle_reply still the most cpu consuming function
> > because of svc->srv_lock contention.
> >
> > I think the problem is that COS defers processing of replies to
> > transaction commit time. When commit happens, MDS has to process
> > thousands of replies (about 14k replies per commit in the test 3.a)
> > in short period of time. I guess the mdt service threads all woken
> > up and spin trying to get the service svr_lock. Processing of new
> > requests may also suffer of this.
> >
> > I ran the tests with with CONFIG_DEBUG_SPINLOCK_SLEEP debugging
> > compiled into a kernel, it found no sleep under spinlock bugs.
> >
> > Further optimization may include
> > 1. per-reply spin locks.
> > 2. per-cpu structures and threads to process reply queues.
> >
> > Any comments?
> >
> > Thanks.
> >
> > PS. the test results are much better when MDS server is sata20
> > machine with 4 cores (the MDS from Washie1 has 2 cores), COS=0 and
> > COS=1 have only %3 difference:
> >
> > COS=1
> > Rate: 3101.77 creates/sec (total: 2 threads 930530 creates 300
> > secs) Rate: 3096.94 creates/sec (total: 2 threads 929083 creates
> > 300 secs)
> >
> > COS=0
> > Rate: 3184.01 creates/sec (total: 2 threads 958388 creates 301
> > secs) Rate: 3152.89 creates/sec (total: 2 threads 945868 creates
> > 300 secs)
>
> _______________________________________________
> Lustre-devel mailing list
> Lustre-devel at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-devel

-- 
Alexander "Zam" Zarochentsev
Staff Engineer
Lustre Group, Sun Microsystems

From Alex.Zhuravlev at Sun.COM  Sun Oct 12 19:12:10 2008
From: Alex.Zhuravlev at Sun.COM (Alex Zhuravlev)
Date: Sun, 12 Oct 2008 23:12:10 +0400
Subject: [Lustre-devel] COS performance issues
In-Reply-To: <200810122241.58982.alexander.zarochentsev@sun.com>
References: <200810081544.08292.alexander.zarochentsev@sun.com>
	<48EC9E22.2090403@sun.com>
	<200810122241.58982.alexander.zarochentsev@sun.com>
Message-ID: <48F24C0A.9070301@sun.com>

would be good to look at profiles as the next one was ldlm_resource_get()

thanks, Alex

Alexander Zarochentsev wrote:
> On 8 October 2008 15:48:50 Alex Zhuravlev wrote:
>> try to profile with single CPU? you'll probably get an idea how
>> "per-cpu" approach can help.
> 
> I booted the MDS server with maxcpus=1 kernel parameter and here are the 
> results:
> 
> cos=0
> 2039.31 creates/sec (total: 2 threads 611794 creates 300 secs)
> 2037.80 creates/sec (total: 2 threads 611341 creates 300 secs)
> 2076.21 creates/sec (total: 2 threads 622864 creates 300 secs)
> 
> cos=1
> 1874.93 creates/sec (total: 2 threads 564354 creates 301 secs)
> 1923.97 creates/sec (total: 2 threads 577191 creates 300 secs)
> 1892.61 creates/sec (total: 2 threads 567783 creates 300 secs)
> 1874.74 creates/sec (total: 2 threads 562421 creates 300 secs)
> 
> unfortunately profiling info isn't available yet, the results are done 
> with SLES10 which can boot with maxcpus=1 but has no oprofile 
> installed.
> 
>> Alexander Zarochentsev wrote:
>>> I have a patch to avoid using of obd_uncommitted_replies_lock
>>> in ptlrpc_server_handle_reply but it has minimal effect,
>>> ptlrpc_server_handle_reply still the most cpu consuming function
>>> because of svc->srv_lock contention.
>>>
>>> I think the problem is that COS defers processing of replies to
>>> transaction commit time. When commit happens, MDS has to process
>>> thousands of replies (about 14k replies per commit in the test 3.a)
>>> in short period of time. I guess the mdt service threads all woken
>>> up and spin trying to get the service svr_lock. Processing of new
>>> requests may also suffer of this.
>>>
>>> I ran the tests with with CONFIG_DEBUG_SPINLOCK_SLEEP debugging
>>> compiled into a kernel, it found no sleep under spinlock bugs.
>>>
>>> Further optimization may include
>>> 1. per-reply spin locks.
>>> 2. per-cpu structures and threads to process reply queues.
>>>
>>> Any comments?
>>>
>>> Thanks.
>>>
>>> PS. the test results are much better when MDS server is sata20
>>> machine with 4 cores (the MDS from Washie1 has 2 cores), COS=0 and
>>> COS=1 have only %3 difference:
>>>
>>> COS=1
>>> Rate: 3101.77 creates/sec (total: 2 threads 930530 creates 300
>>> secs) Rate: 3096.94 creates/sec (total: 2 threads 929083 creates
>>> 300 secs)
>>>
>>> COS=0
>>> Rate: 3184.01 creates/sec (total: 2 threads 958388 creates 301
>>> secs) Rate: 3152.89 creates/sec (total: 2 threads 945868 creates
>>> 300 secs)
>> _______________________________________________
>> Lustre-devel mailing list
>> Lustre-devel at lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-devel
> 


From Alexander.Zarochentsev at Sun.COM  Mon Oct 13 14:36:36 2008
From: Alexander.Zarochentsev at Sun.COM (Alexander Zarochentsev)
Date: Mon, 13 Oct 2008 18:36:36 +0400
Subject: [Lustre-devel] COS performance issues
In-Reply-To: <48F24C0A.9070301@sun.com>
References: <200810081544.08292.alexander.zarochentsev@sun.com>
	<200810122241.58982.alexander.zarochentsev@sun.com>
	<48F24C0A.9070301@sun.com>
Message-ID: <200810131836.37447.alexander.zarochentsev@sun.com>

On 12 October 2008 23:12:10 Alex Zhuravlev wrote:
> would be good to look at profiles as the next one was
> ldlm_resource_get()

Alex, look, ptlrpc_server_handle_reply has gone:

CPU: P4 / Xeon, speed 2800.25 MHz (estimated)
Counted GLOBAL_POWER_EVENTS events (time during which processor is not stopped
with a unit mask of 0x01 (mandatory) count
 100000
samples  %        image name               app name                 symbol name
312318    4.7857  vmlinux                  vmlinux                  try_to_wake_up
259690    3.9792  ptlrpc.ko                ptlrpc                   ldlm_resource_find
175916    2.6956  vmlinux                  vmlinux                  __wake_up_common
160912    2.4657  e1000.ko                 e1000                    e1000_irq_enable
157031    2.4062  obdclass.ko              obdclass                 htable_lookup
134309    2.0580  e1000.ko                 e1000                    e1000_intr_msi
121261    1.8581  lvfs.ko                  lvfs                     lprocfs_counter_add
87613     1.3425  vmlinux                  vmlinux                  __find_get_block
87562     1.3417  oprofiled                oprofiled                (no symbols)
77926     1.1941  vmlinux                  vmlinux                  memset
74561     1.1425  vmlinux                  vmlinux                  __kmalloc
70952     1.0872  vmlinux                  vmlinux                  __switch_to
67288     1.0311  mds.ko                   mds                      mds_lov_dump_objids
67085     1.0279  vmlinux                  vmlinux                  memmove
66521     1.0193  vmlinux                  vmlinux                  kfree
55112     0.8445  ptlrpc.ko                ptlrpc                   ptlrpc_main
54357     0.8329  vmlinux                  vmlinux                  schedule
53501     0.8198  vmlinux                  vmlinux                  find_get_page

> thanks, Alex
>
> Alexander Zarochentsev wrote:
> > On 8 October 2008 15:48:50 Alex Zhuravlev wrote:
> >> try to profile with single CPU? you'll probably get an idea how
> >> "per-cpu" approach can help.
> >
> > I booted the MDS server with maxcpus=1 kernel parameter and here
> > are the results:
> >
> > cos=0
> > 2039.31 creates/sec (total: 2 threads 611794 creates 300 secs)
> > 2037.80 creates/sec (total: 2 threads 611341 creates 300 secs)
> > 2076.21 creates/sec (total: 2 threads 622864 creates 300 secs)
> >
> > cos=1
> > 1874.93 creates/sec (total: 2 threads 564354 creates 301 secs)
> > 1923.97 creates/sec (total: 2 threads 577191 creates 300 secs)
> > 1892.61 creates/sec (total: 2 threads 567783 creates 300 secs)
> > 1874.74 creates/sec (total: 2 threads 562421 creates 300 secs)
> >
> > unfortunately profiling info isn't available yet, the results are
> > done with SLES10 which can boot with maxcpus=1 but has no oprofile
> > installed.
> >
> >> Alexander Zarochentsev wrote:
> >>> I have a patch to avoid using of obd_uncommitted_replies_lock
> >>> in ptlrpc_server_handle_reply but it has minimal effect,
> >>> ptlrpc_server_handle_reply still the most cpu consuming function
> >>> because of svc->srv_lock contention.
> >>>
> >>> I think the problem is that COS defers processing of replies to
> >>> transaction commit time. When commit happens, MDS has to process
> >>> thousands of replies (about 14k replies per commit in the test
> >>> 3.a) in short period of time. I guess the mdt service threads all
> >>> woken up and spin trying to get the service svr_lock. Processing
> >>> of new requests may also suffer of this.
> >>>
> >>> I ran the tests with with CONFIG_DEBUG_SPINLOCK_SLEEP debugging
> >>> compiled into a kernel, it found no sleep under spinlock bugs.
> >>>
> >>> Further optimization may include
> >>> 1. per-reply spin locks.
> >>> 2. per-cpu structures and threads to process reply queues.
> >>>
> >>> Any comments?
> >>>
> >>> Thanks.
> >>>
> >>> PS. the test results are much better when MDS server is sata20
> >>> machine with 4 cores (the MDS from Washie1 has 2 cores), COS=0
> >>> and COS=1 have only %3 difference:
> >>>
> >>> COS=1
> >>> Rate: 3101.77 creates/sec (total: 2 threads 930530 creates 300
> >>> secs) Rate: 3096.94 creates/sec (total: 2 threads 929083 creates
> >>> 300 secs)
> >>>
> >>> COS=0
> >>> Rate: 3184.01 creates/sec (total: 2 threads 958388 creates 301
> >>> secs) Rate: 3152.89 creates/sec (total: 2 threads 945868 creates
> >>> 300 secs)
> >>
> >> _______________________________________________
> >> Lustre-devel mailing list
> >> Lustre-devel at lists.lustre.org
> >> http://lists.lustre.org/mailman/listinfo/lustre-devel

-- 
Alexander "Zam" Zarochentsev
Staff Engineer
Lustre Group, Sun Microsystems

From Alex.Zhuravlev at Sun.COM  Mon Oct 13 15:04:51 2008
From: Alex.Zhuravlev at Sun.COM (Alex Zhuravlev)
Date: Mon, 13 Oct 2008 19:04:51 +0400
Subject: [Lustre-devel] COS performance issues
In-Reply-To: <200810131836.37447.alexander.zarochentsev@sun.com>
References: <200810081544.08292.alexander.zarochentsev@sun.com>
	<200810122241.58982.alexander.zarochentsev@sun.com>
	<48F24C0A.9070301@sun.com>
	<200810131836.37447.alexander.zarochentsev@sun.com>
Message-ID: <48F36393.9020406@sun.com>

cool! do you have profile with COS disabled? how is it different?
I guess ldlm_resorce_find() is result of COS as now we've got 10K times
more locks and resorces, but what about try_to_wake_up() and __wake_up_common() ?


thanks, Alex

Alexander Zarochentsev wrote:
> On 12 October 2008 23:12:10 Alex Zhuravlev wrote:
>> would be good to look at profiles as the next one was
>> ldlm_resource_get()
> 
> Alex, look, ptlrpc_server_handle_reply has gone:
> 
> CPU: P4 / Xeon, speed 2800.25 MHz (estimated)
> Counted GLOBAL_POWER_EVENTS events (time during which processor is not stopped
> with a unit mask of 0x01 (mandatory) count
>  100000
> samples  %        image name               app name                 symbol name
> 312318    4.7857  vmlinux                  vmlinux                  try_to_wake_up
> 259690    3.9792  ptlrpc.ko                ptlrpc                   ldlm_resource_find
> 175916    2.6956  vmlinux                  vmlinux                  __wake_up_common
> 160912    2.4657  e1000.ko                 e1000                    e1000_irq_enable
> 157031    2.4062  obdclass.ko              obdclass                 htable_lookup
> 134309    2.0580  e1000.ko                 e1000                    e1000_intr_msi
> 121261    1.8581  lvfs.ko                  lvfs                     lprocfs_counter_add
> 87613     1.3425  vmlinux                  vmlinux                  __find_get_block
> 87562     1.3417  oprofiled                oprofiled                (no symbols)
> 77926     1.1941  vmlinux                  vmlinux                  memset
> 74561     1.1425  vmlinux                  vmlinux                  __kmalloc
> 70952     1.0872  vmlinux                  vmlinux                  __switch_to
> 67288     1.0311  mds.ko                   mds                      mds_lov_dump_objids
> 67085     1.0279  vmlinux                  vmlinux                  memmove
> 66521     1.0193  vmlinux                  vmlinux                  kfree
> 55112     0.8445  ptlrpc.ko                ptlrpc                   ptlrpc_main
> 54357     0.8329  vmlinux                  vmlinux                  schedule
> 53501     0.8198  vmlinux                  vmlinux                  find_get_page
> 
>> thanks, Alex
>>
>> Alexander Zarochentsev wrote:
>>> On 8 October 2008 15:48:50 Alex Zhuravlev wrote:
>>>> try to profile with single CPU? you'll probably get an idea how
>>>> "per-cpu" approach can help.
>>> I booted the MDS server with maxcpus=1 kernel parameter and here
>>> are the results:
>>>
>>> cos=0
>>> 2039.31 creates/sec (total: 2 threads 611794 creates 300 secs)
>>> 2037.80 creates/sec (total: 2 threads 611341 creates 300 secs)
>>> 2076.21 creates/sec (total: 2 threads 622864 creates 300 secs)
>>>
>>> cos=1
>>> 1874.93 creates/sec (total: 2 threads 564354 creates 301 secs)
>>> 1923.97 creates/sec (total: 2 threads 577191 creates 300 secs)
>>> 1892.61 creates/sec (total: 2 threads 567783 creates 300 secs)
>>> 1874.74 creates/sec (total: 2 threads 562421 creates 300 secs)
>>>
>>> unfortunately profiling info isn't available yet, the results are
>>> done with SLES10 which can boot with maxcpus=1 but has no oprofile
>>> installed.
>>>
>>>> Alexander Zarochentsev wrote:
>>>>> I have a patch to avoid using of obd_uncommitted_replies_lock
>>>>> in ptlrpc_server_handle_reply but it has minimal effect,
>>>>> ptlrpc_server_handle_reply still the most cpu consuming function
>>>>> because of svc->srv_lock contention.
>>>>>
>>>>> I think the problem is that COS defers processing of replies to
>>>>> transaction commit time. When commit happens, MDS has to process
>>>>> thousands of replies (about 14k replies per commit in the test
>>>>> 3.a) in short period of time. I guess the mdt service threads all
>>>>> woken up and spin trying to get the service svr_lock. Processing
>>>>> of new requests may also suffer of this.
>>>>>
>>>>> I ran the tests with with CONFIG_DEBUG_SPINLOCK_SLEEP debugging
>>>>> compiled into a kernel, it found no sleep under spinlock bugs.
>>>>>
>>>>> Further optimization may include
>>>>> 1. per-reply spin locks.
>>>>> 2. per-cpu structures and threads to process reply queues.
>>>>>
>>>>> Any comments?
>>>>>
>>>>> Thanks.
>>>>>
>>>>> PS. the test results are much better when MDS server is sata20
>>>>> machine with 4 cores (the MDS from Washie1 has 2 cores), COS=0
>>>>> and COS=1 have only %3 difference:
>>>>>
>>>>> COS=1
>>>>> Rate: 3101.77 creates/sec (total: 2 threads 930530 creates 300
>>>>> secs) Rate: 3096.94 creates/sec (total: 2 threads 929083 creates
>>>>> 300 secs)
>>>>>
>>>>> COS=0
>>>>> Rate: 3184.01 creates/sec (total: 2 threads 958388 creates 301
>>>>> secs) Rate: 3152.89 creates/sec (total: 2 threads 945868 creates
>>>>> 300 secs)
>>>> _______________________________________________
>>>> Lustre-devel mailing list
>>>> Lustre-devel at lists.lustre.org
>>>> http://lists.lustre.org/mailman/listinfo/lustre-devel
> 


From aurelien.degremont at cea.fr  Mon Oct 13 16:39:16 2008
From: aurelien.degremont at cea.fr (Aurelien Degremont)
Date: Mon, 13 Oct 2008 18:39:16 +0200
Subject: [Lustre-devel] "Simple" HSM straw man
In-Reply-To: <48EE87A1.7000003@sun.com>
References: <48EE87A1.7000003@sun.com>
Message-ID: <48F379B4.6040209@cea.fr>

Nathaniel Rutman a écrit :
>     c. restore on file open, not data read/write

take care of the difficulties to move this behavious to a 
restore-on-first-I/O later

>     d. interfaces with hardware-specific copy tool to access HSM files
rather "HSM-specific"

>     e. kernel process encompasses service threads listening for 
> coordinator requests, passes these up to userspace process via upcall.  
> No interaction with the client is needed; this is a simple message 
> passing service.

Depending on how you can manage a user-space process, but, AFAIK, to be 
able to manage the copy tool process, that means:
  - start this process
  - send a signal
  - get its output (for "complex" hsm, we will need feedback from copy 
tool process)
  - wait for the process end
  - ...
All of this is easily doable from userspace, and very hard in 
kernel-space (we cannot use the fire-and-forget call call_usermodehelper).
So I rather imagine:
  - a kernel space mover, simply getting LNET messages and passing them 
to user-space mover
  - a user-space mover, forking, spawning and managing the copy tool 
process.

Maybe it will need to manage several copy tool processes, so it will 
need queues, process list, etc...

So I think this tool needs a bit more than just a "simple message 
passing service".

>     b. lov EA changes
>        i.  flags: file_is_purged "purged", copyout_begin, 
> file_in_HSM_is_out_of_date "hsm_dirty", copyout_complete.  The purged 
> flag is always manipulated under a write layout lock, the other flags 
> are not.
>        ii: "window" EA range of non-purged data (rev2)

If you add a window EA (will be needed anyway for hsm v2), you do not 
need a purged flag:

window.start ==window.end is comparable to a purged flag unset. (or 
window.end == 0)


>     c. new file ioctls: HSMCopyOut, HSMPurge, HSMCopyinDone
> 
> 
> Algorithms
> 1. copyout
>     a. Policy engine decides to copy a file to HSM, executes HSMCopyOut 
> ioctl on file
>     b. ioctl handled by MDT, which passes request to Coordinator
>     c. coordinator dispatches request to mover.  request should include 
> file extents (for future purposes)
>     d. normal extents read lock is taken by mover running on client
>     e. mover sets "copyout_begin" bit and clears "hsm_dirty" bit in EA.
>     f. any writes to the file set the "hsm_dirty" bit (may be 
> lazy/delayed with mtime or filesize change updates on MDT).  Note that 
> file writes need not cancel copyout; for a fs with a single big file, we 
> don't want to keep interrupting copyout or it will never finish. 

Is it interesting to have a file that is outdated and possibly uncoherent?

>     g. when done, mover checks hsm_dirty bit.  If set, clears 
> copyout_begin, indicating current file is not in HSM.  If not set,  
> mover sets "copyout_complete" bit.  File layout write lock is not taken 
> during mover flag manipulation.  (Note: file modifications after copyout 
> is complete will have both copyout_complete and hsm_dirty bits set.)
> 
> 2. purge (aka punch)
>     a. Policy engine decides to purge a file, exectues HSMPurge ioctl on 
> file
>     b. ioctl handled by MDT
>     c. MDT takes a write lock on the file layout lock
>     d. MDT enques write locks on all extents of the file.  After these 
> are granted, then no client has any dirty cache and no child can take 
> new extent locks until layout lock is released.  MDT drops all extent locks.
>     e. MDT verifies that hsm_dirty bit is clear and copyout_complete bit 
> is set
>     f. MDT marks the LOV EA as "purged"
>     g. MDT sends destroys the OST objects, using destroy llog entries to 
> guard against object leakage during OST failover

Are you sure you want to remove those objects if we will need them 
later, in "complex" HSM?
As this mecanism will need to change a lot when we will implement the 
restore-in-place feature, i'm not sure this is the best idea.


>     h. MDT drops layout lock.
> 
> 3. restore (aka copyin aka cache miss)
>     a. Client open intent enques layout read lock. 
>     b. MDT checks "purged" bit; if purged, lock request response 
> includes "wait forever" flag, causing client to block the open.
>     c. MDT creates a new layout with a similar stripe pattern as the 
> original, allocating new objects on new OSTs.  (We should try to respect 
> specific layout settings (pool, stripecount, stripesize), but be 
> flexible if e.g. pool doesn't exist anymore.  Maybe we want to ignore 
> offset and/or specific ost allocations in order to rebalance.)
>     d. MDT sends request to coordinator requesting copyin of the file to 
> .lustre/fid/XXXX with extents 0-EOF. Extents may be used in the future 
> to (a) copy in part of a file, in low-disk-space situations; (b) copy in 
> individual stripes simultaneously on multiple OSTs.
>     e. Coordinator distributes that request to an appropriate mover.
>     f. Writes into .lustre/fid/* are not required to hold layout read 
> lock (or special flag is passed to open, or group write lock on layout 
> is passed to mover)
>     g. Mover copies data from HSM
>     h. When finished, mover calls ioctl HSM_COPYIN_DONE on the file
>     i. MDT clears "purged" bit from LOV EA
>     j. MDT releases the layout lock
>     k. This sends a completion AST to the original client, who now 
> completes his open. 




Concerning the new flag copyout_begin/copyout_complete, I'm not a 
ldlm/recovery specialist but is it possible to have the mover to take a 
kind of write extent lock on the area it has to copied in/out and 
downgrade it on a smaller range as the copy tool goes along.

Copy-out
- Mover take a specific lock on range (0-EOF for the moment)
- On this range, reads pass, writes raise a callback on the mover.
- Receiving this callback, if the mover release its lock, the copyout is 
cancelled, if not, the write i/o is blocked
- When the mover has copied [0 - cursor], it can downgrade its lock to 
[cursor - EOF] and release the lock on [ 0 - cursor].

Same thing could be done for copy in.

The two key points are:
  - Could we have a layout lock on a specific range?
  - Is it possible to downgrade a range lock with ldlm?



-- 
Aurelien Degremont
CEA

From adilger at sun.com  Tue Oct 14 17:48:45 2008
From: adilger at sun.com (Andreas Dilger)
Date: Tue, 14 Oct 2008 11:48:45 -0600
Subject: [Lustre-devel] HSM cache-miss locking
In-Reply-To: <48EE574E.2010808@sun.com>
References: <20081009080425.GJ2009@webber.adilger.int>
	<48EE574E.2010808@sun.com>
Message-ID: <20081014174845.GN2009@webber.adilger.int>

On Oct 09, 2008  12:11 -0700, Nathaniel Rutman wrote:
> Andreas Dilger wrote:
>>     The only reason the agent is restoring into the temp file is to avoid
>>     needing to open the file while the MDS is blocking layout lock access,
>>     but maybe that isn't a big obstacle (e.g. open flag).
>
> You mean open flag O_IGNORE_LAYOUT_LOCK?  So the one problem I see with  
> this is the case of a stuck agent - if we want to start another agent  
> doing copyin we have to insure that the first agent doesn't try to write  
> anything else.

Having two agents on the same file wouldn't itself be harmful, because they
should both be restoring the same data to the same place.  That said, we
would still want to be able to kill the stuck agent to avoid it continuing
to "restore" the file over new user data after the second agent had reported
"file is available" and the user process started writing to it.

>> 2) client enqueues extent lock on OST
>>     - object was previously marked fully/partly invalid during purge
>>     - object may have persistent invalid map of extent(s) that indicate
>>       which parts of object require copy-in
>
> I'll read this as if you're proposing your 2,3 (call it "per-object  
> invalid ranges held on OSTs") as a new method to do the copyin in-place.  
> This is not the original in-place idea proposed in Menlo Park (see 
> below), and so I'll comment with an eye toward the differences.

Correct, this is something Eric and I recently discussed in the context
of being able to begin using a file before copyin had completed.

> I think we can't assume we're restoring back to the original OSTs.  

Definitely not.

> Therefore the MDT must create new empty objects on the OSTs and have the  
> OSTs mark them purged before the layout lock can be granted to the 
> clients.

Correct.

>>     - access to invalid parts of object trigger copy-in upcall to coordinator
>   
> Now we need to figure out how to map the object back to a particular  
> range extent of a particular file (are we storing this in an EA with  
> each object now?)

We had also discussed the need for this with migration.  The OSTs already
store the MDS FID on each object, and even if the OSTs cannot do the
object->file extent mapping, their upcall to the coordinator can do this
with the LOV EA and the object extent.

> We also need to initiate OST->coordinator  
> communication, so either coordinator becomes a distributed function on  
> the OSTs or we need new services going the reverse of the normal  
> mdt->ost direction.  Maybe the coordinator-as-distributed-function works  
> - the coordinators must all choose the same agent for objects belonging  
> to the same file, yet distribute load among agents: I think the  
> coordinator just got a lot more complicated.

I don't think this implies the need for a distributed coordinator.  The
OSTs would contact the coordinator (as MDS does at file access in "simple"
model) with MDS FID (+OST extent?) and coordinator determines if there is
an existing copyin for that FID or not.

>>     ? group locks on invalid part of file block writes to missing data
>
> The issue here is that we can't allow any client to write and then have  
> the agent overwrite the new data with old data being restored.  So we  
> could have the OST give a group lock to agent via coordinator,  
> preventing all other writes.  But it seems that we can check the special  
> "clear invalid" flag used by the agent (see (3) below), and silently  
> drop writes into areas not in the "invalid extents" list.  Any client  
> write to any extent will clear the invalid flag for those extents.  And  
> then we only ever need to block on reading.

Eric and I discussed this at length.  The solution we came up with is to
have "agent" writes that are restoring the file be flagged as such and
only be allowed for parts of the file which are still marked "in HSM".
This allows normal writes to proceed without danger of being overwritten,
and for operations like "truncate" it would remove the need to restore
some/any of the file data because it would also clear the "in HSM" marker
from the truncated parts of the file.

NB: we haven't discussed truncates/unlinks in the context of HSM, but this
should _definitely_ not start a copyin of the file data.

> What about reads to missing data?  OST refuses to grant read locks on  
> invalid extents, needs clients to wait forever.

This would also trigger HSM copy-in.  If the HSM decides this data is
permanently inaccessible then the object (or parts thereof) should be
marked as such and client reads should get -EIO.

>> 3) coordinator contacts agent(s) to retrieve FID N from HSM
>>     - agents write to actual object to be restored with "clear invalid" flag
>>     - writes by agent shrink invalid extent, periodically update on-disk
>>       invalid extent and release locks on that part of file (on commit?)
>
> The OST should keep track of all invalid extents.  Invalid extents list  
> changes should be stored on disk, transactionally with the data write.

Yes, definitely it needs to be stored on disk, and it should be kept with
the object itself.  For completely purged objects, the MDS needs to mark
the whole file as "in HSM", and it would also truncate the objects to the
right size as soon as they are created (this already happens today when
the MDS file has no objects and is storing the size).

Remember this is all in the "complex" case where we want concurrent file
access with HSM copyin, and in simple case client will just block until
the copyin is finished.  Similarly, if copyin crashes in the middle, it
would have to start at the beginning, but that should be rare enough to
ignore it until the full solution is implemented.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.


From Nathan.Rutman at Sun.COM  Tue Oct 14 19:41:14 2008
From: Nathan.Rutman at Sun.COM (Nathaniel Rutman)
Date: Tue, 14 Oct 2008 12:41:14 -0700
Subject: [Lustre-devel] "Simple" HSM straw man
In-Reply-To: <48F379B4.6040209@cea.fr>
References: <48EE87A1.7000003@sun.com> <48F379B4.6040209@cea.fr>
Message-ID: <48F4F5DA.7090000@sun.com>

Aurelien Degremont wrote:
> Nathaniel Rutman a écrit :
>>     c. restore on file open, not data read/write
>
> take care of the difficulties to move this behavious to a 
> restore-on-first-I/O later.
Indeed.  From a client point of view, it only changes which locks its 
waiting on, but from a server point of view the OSTs would need to 
become involved in HSM knowledge.  It is more work, but I don't think 
there would be much "throwaway" code from the former to the latter.
>
>>     d. interfaces with hardware-specific copy tool to access HSM files
> rather "HSM-specific"
>
>>     e. kernel process encompasses service threads listening for 
>> coordinator requests, passes these up to userspace process via 
>> upcall.  No interaction with the client is needed; this is a simple 
>> message passing service.
>
> Depending on how you can manage a user-space process, but, AFAIK, to 
> be able to manage the copy tool process, that means:
>  - start this process
>  - send a signal
>  - get its output (for "complex" hsm, we will need feedback from copy 
> tool process)
>  - wait for the process end
>  - ...
> All of this is easily doable from userspace, and very hard in 
> kernel-space (we cannot use the fire-and-forget call 
> call_usermodehelper).
> So I rather imagine:
>  - a kernel space mover, simply getting LNET messages and passing them 
> to user-space mover
>  - a user-space mover, forking, spawning and managing the copy tool 
> process.
>
> Maybe it will need to manage several copy tool processes, so it will 
> need queues, process list, etc...
>
> So I think this tool needs a bit more than just a "simple message 
> passing service".
As we discussed in the HSM concall this morning, the return path can 
mostly take place through the file itself via ioctl calls.  The mover 
will open the destination file location in Lustre and then can indicate 
status through an ioctl: starting, waiting for HSM, periodic pinging or 
"% complete" messages, copyin complete.  This is the "fire-and-forget" 
model, and can be started from call_usermodehelper.  The in-kernel code 
will only have to deal with one-way requests from coordinator to mover.

We also specified 4 types of requests from coordinator:
1. copyin FID
2. copyout FID
3. abort copy(in|out) FID
4. purge FID from HSM

To accomplish 3, it might make sense to store the PID of the process 
started from the upcall in the kernel (this is returned by the upcall).  
Closing the file could clear the pid from the kernel list.
>
>>     b. lov EA changes
>>        i.  flags: file_is_purged "purged", copyout_begin, 
>> file_in_HSM_is_out_of_date "hsm_dirty", copyout_complete.  The purged 
>> flag is always manipulated under a write layout lock, the other flags 
>> are not.
>>        ii: "window" EA range of non-purged data (rev2)
>
> If you add a window EA (will be needed anyway for hsm v2), you do not 
> need a purged flag:
>
> window.start ==window.end is comparable to a purged flag unset. (or 
> window.end == 0)
True, but I don't really see a large market for partially purged files, 
so I don't really believe that it is worth the effort.  One of the 
important points here is that we are deleting stripes off the OSTs, 
freeing up space, and we won't necessarily restore to those same OSTs.  
As soon as we have partially purged files that's no longer the case, and 
I think complicates things too much.
>
>
>>     c. new file ioctls: HSMCopyOut, HSMPurge, HSMCopyinDone
>>
>>
>> Algorithms
>> 1. copyout
>>     a. Policy engine decides to copy a file to HSM, executes 
>> HSMCopyOut ioctl on file
>>     b. ioctl handled by MDT, which passes request to Coordinator
>>     c. coordinator dispatches request to mover.  request should 
>> include file extents (for future purposes)
>>     d. normal extents read lock is taken by mover running on client
>>     e. mover sets "copyout_begin" bit and clears "hsm_dirty" bit in EA.
>>     f. any writes to the file set the "hsm_dirty" bit (may be 
>> lazy/delayed with mtime or filesize change updates on MDT).  Note 
>> that file writes need not cancel copyout; for a fs with a single big 
>> file, we don't want to keep interrupting copyout or it will never 
>> finish. 
>
> Is it interesting to have a file that is outdated and possibly 
> uncoherent?
It is probably useful in some cases -- simulation checkpoints maybe.
>
>>     g. when done, mover checks hsm_dirty bit.  If set, clears 
>> copyout_begin, indicating current file is not in HSM.  If not set,  
>> mover sets "copyout_complete" bit.  File layout write lock is not 
>> taken during mover flag manipulation.  (Note: file modifications 
>> after copyout is complete will have both copyout_complete and 
>> hsm_dirty bits set.)
>>
>> 2. purge (aka punch)
>>     a. Policy engine decides to purge a file, exectues HSMPurge ioctl 
>> on file
>>     b. ioctl handled by MDT
>>     c. MDT takes a write lock on the file layout lock
>>     d. MDT enques write locks on all extents of the file.  After 
>> these are granted, then no client has any dirty cache and no child 
>> can take new extent locks until layout lock is released.  MDT drops 
>> all extent locks.
>>     e. MDT verifies that hsm_dirty bit is clear and copyout_complete 
>> bit is set
>>     f. MDT marks the LOV EA as "purged"
>>     g. MDT sends destroys the OST objects, using destroy llog entries 
>> to guard against object leakage during OST failover
>
> Are you sure you want to remove those objects if we will need them 
> later, in "complex" HSM?
> As this mecanism will need to change a lot when we will implement the 
> restore-in-place feature, i'm not sure this is the best idea.
Ah, I think it is important that we do NOT restore in place to the old 
OST objects.  The OSTs may now be full, or indeed not exist anymore.  
The restore in place for complex HSM is at the file level; the objects 
may move around.  "Complex" in this case just means that clients will 
have access to partially restored files.
>
>
>>     h. MDT drops layout lock.
>>
>> 3. restore (aka copyin aka cache miss)
>>     a. Client open intent enques layout read lock.     b. MDT checks 
>> "purged" bit; if purged, lock request response includes "wait 
>> forever" flag, causing client to block the open.
>>     c. MDT creates a new layout with a similar stripe pattern as the 
>> original, allocating new objects on new OSTs.  (We should try to 
>> respect specific layout settings (pool, stripecount, stripesize), but 
>> be flexible if e.g. pool doesn't exist anymore.  Maybe we want to 
>> ignore offset and/or specific ost allocations in order to rebalance.)
>>     d. MDT sends request to coordinator requesting copyin of the file 
>> to .lustre/fid/XXXX with extents 0-EOF. Extents may be used in the 
>> future to (a) copy in part of a file, in low-disk-space situations; 
>> (b) copy in individual stripes simultaneously on multiple OSTs.
>>     e. Coordinator distributes that request to an appropriate mover.
>>     f. Writes into .lustre/fid/* are not required to hold layout read 
>> lock (or special flag is passed to open, or group write lock on 
>> layout is passed to mover)
>>     g. Mover copies data from HSM
>>     h. When finished, mover calls ioctl HSM_COPYIN_DONE on the file
>>     i. MDT clears "purged" bit from LOV EA
>>     j. MDT releases the layout lock
>>     k. This sends a completion AST to the original client, who now 
>> completes his open. 
>
>
>
>
> Concerning the new flag copyout_begin/copyout_complete, I'm not a 
> ldlm/recovery specialist but is it possible to have the mover to take 
> a kind of write extent lock on the area it has to copied in/out and 
> downgrade it on a smaller range as the copy tool goes along.
This is called "lock conversion" and is not yet implemented, but has 
been a general Lustre design goal for some time.  So yes, for "complex" 
HSM this is what we would want to do.
>
> Copy-out
> - Mover take a specific lock on range (0-EOF for the moment)
> - On this range, reads pass, writes raise a callback on the mover.
> - Receiving this callback, if the mover release its lock, the copyout 
> is cancelled, if not, the write i/o is blocked 
I don't think we want to block the write just because the HSM copy isn't 
done yet.  If the data is changing, then the policy engine shouldn't 
have started a copyout process in the first place.  If the customer's 
goal is to do a coherent checkpoint, then it should explicitly wait for 
the copyout to be done.  If it's just the policy engine that got it 
wrong, it doesn't matter if it finishes or not; the file will be marked 
"hsm_dirty", and so the policy engine should re-queue it for copyout 
again later, and it can't be purged in the meantime since the dirty bit 
is set.

> - When the mover has copied [0 - cursor], it can downgrade its lock to 
> [cursor - EOF] and release the lock on [ 0 - cursor].
>
> Same thing could be done for copy in.
>
> The two key points are:
>  - Could we have a layout lock on a specific range?
Not the layout lock - layout means the striping pattern, and must be 
held first before any extent locks can be taken.  So I think what you 
are asking we plan to do with two locks: the layout lock plus another 
extent lock.
>  - Is it possible to downgrade a range lock with ldlm?
>
Not yet, but as I said, lock conversion is a general Lustre goal.

From support at google.com  Sat Oct 11 16:44:42 2008
From: support at google.com (Google-AdWords-Noreply)
Date: Sat, 11 Oct 2008 17:44:42 +0100
Subject: [Lustre-devel] Submit your payment information.
Message-ID: <01c92bc9$0a6b6100$c0445252@fisky>

-------------------------

Hello 


Please update your credit card information in order to trigger our
billing system to try processing your payment again. If you plan to use
the same credit card(s), please use the 'Retry Card' button on the
Billing Preferences page of your account. Otherwise, please follow the
steps below to update the information in your AdWords account.

Our system was unable to process a payment  for your 
outstanding Google AdWords account balance using your primary credit
card. For the time being, your account is still open, and your ads are
still running. However, we require you to update the payment
information in your AdWords account very soon in order to ensure
continued ad serving.

1. Log in to your AdWords account
2. Click the 'My Account' tab.
3. Click 'Billing Preferences' link.
4. Click Edit next to the appropriate 'Payment Details' section.
5. Enter your new or updated payment information.
6. Click 'Save Changes' when you have finished.

Tip: You can review the status of your billing on the Billing Summary
page, under the 'My Account' tab. If a payment has been declined,
click 'Payment Declined' beside the line item to review information for
that particular payment. Once your payment has been processed
successfully, you can view and print an invoice from your Billing
Summary page.

---------------------------------------------------------------------
This message was sent from a notification-only email address that does
not accept incoming email. Please do not reply to this message. If you
have any questions, please visit the Google AdWords Help Centre at
https://adwords.google.com/support/?hl=en_GB to find answers to
frequently asked questions and a 'contact us' link near the bottom of
the page.
------------------------------------------------
-----------------------------------

Thank you for advertising with Google AdWords. We look forward to
providing you with the most effective advertising available.



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel_lists.lustre.org/attachments/20081011/cd527f13/attachment.html>

From Nathan.Rutman at Sun.COM  Wed Oct 15 21:52:53 2008
From: Nathan.Rutman at Sun.COM (Nathaniel Rutman)
Date: Wed, 15 Oct 2008 14:52:53 -0700
Subject: [Lustre-devel] "Simple" HSM straw man
In-Reply-To: <48F4F5DA.7090000@sun.com>
References: <48EE87A1.7000003@sun.com> <48F379B4.6040209@cea.fr>
	<48F4F5DA.7090000@sun.com>
Message-ID: <48F66635.3040006@sun.com>

Nathaniel Rutman wrote:
>
>>>     b. lov EA changes
>>>        i.  flags: file_is_purged "purged", copyout_begin, 
>>> file_in_HSM_is_out_of_date "hsm_dirty", copyout_complete.  The purged 
>>> flag is always manipulated under a write layout lock, the other flags 
>>> are not.
>>>        ii: "window" EA range of non-purged data (rev2)
>>>       
>> If you add a window EA (will be needed anyway for hsm v2), you do not 
>> need a purged flag:
>>
>> window.start ==window.end is comparable to a purged flag unset. (or 
>> window.end == 0)
>>     
> True, but I don't really see a large market for partially purged files, 
> so I don't really believe that it is worth the effort.  One of the 
> important points here is that we are deleting stripes off the OSTs, 
> freeing up space, and we won't necessarily restore to those same OSTs.  
> As soon as we have partially purged files that's no longer the case, and 
> I think complicates things too much.
>   
Ok, I've been told I'm dead wrong here, and this will absolutely be 
required for "complex" HSM (not "simple"), and so therefore we should at 
least think about the arch now.  Supposedly we need to keep X bytes at 
the beginning of the file for the unix "file" command, and supposedly 
icon/preview data, and Y bytes at the end of the file, not sure exactly 
why.
We would still plan on deleting the OST objects in the middle.  And 
clearly, a simple beginning/ending byte count is insufficient for the 
final "complex" requirement of enabling partial file reads while doing a 
copyin (where we would at a minimum need a per-object cursor). 
Anyhow, as I write this none of this sounds like something that can't be 
implemented at a later time, so I think we should stick with the 
simplest of the simple options for now.



From aurelien.degremont at cea.fr  Thu Oct 16 14:09:09 2008
From: aurelien.degremont at cea.fr (Aurelien Degremont)
Date: Thu, 16 Oct 2008 16:09:09 +0200
Subject: [Lustre-devel] "Simple" HSM straw man
In-Reply-To: <48F66635.3040006@sun.com>
References: <48EE87A1.7000003@sun.com> <48F379B4.6040209@cea.fr>
	<48F4F5DA.7090000@sun.com> <48F66635.3040006@sun.com>
Message-ID: <48F74B05.7060401@cea.fr>

Nathaniel Rutman a écrit :
> Ok, I've been told I'm dead wrong here, and this will absolutely be 
> required for "complex" HSM (not "simple"), and so therefore we should at 
> least think about the arch now.  Supposedly we need to keep X bytes at 
> the beginning of the file for the unix "file" command, and supposedly 
> icon/preview data, and Y bytes at the end of the file, not sure exactly 
> why.
> We would still plan on deleting the OST objects in the middle.  And 
> clearly, a simple beginning/ending byte count is insufficient for the 
> final "complex" requirement of enabling partial file reads while doing a 
> copyin (where we would at a minimum need a per-object cursor). Anyhow, 
> as I write this none of this sounds like something that can't be 
> implemented at a later time, so I think we should stick with the 
> simplest of the simple options for now.
>

Ok.
Can you just sum up the inplace copy-in mechanism that have been decided 
(between Menlo Park version and the other ones)?


-- 
Aurelien Degremont
CEA

From eeb at sun.com  Thu Oct 16 21:56:34 2008
From: eeb at sun.com (Eric Barton)
Date: Thu, 16 Oct 2008 22:56:34 +0100
Subject: [Lustre-devel] "Simple" HSM straw man
In-Reply-To: <48F4F5DA.7090000@sun.com>
References: <48EE87A1.7000003@sun.com> <48F379B4.6040209@cea.fr>
	<48F4F5DA.7090000@sun.com>
Message-ID: <CE2BF0610E4C4692A3DDDD842249AAE2@ebpc>

Nathan,

> True, but I don't really see a large market for partially purged files, 
> so I don't really believe that it is worth the effort.  One of the 
> important points here is that we are deleting stripes off the OSTs, 
> freeing up space, and we won't necessarily restore to those same OSTs.  
> As soon as we have partially purged files that's no longer the case, and 
> I think complicates things too much.

Partially purged files is a requirement to allow graphical file browsers
to retrieve icons from within the file.  It's OK to miss this out in the
first version, but it has to be there for the full product.

> >> Algorithms
> >> 1. copyout
> >>     a. Policy engine decides to copy a file to HSM, executes 
> >>        HSMCopyOut ioctl on file
> >>     b. ioctl handled by MDT, which passes request to Coordinator
> >>     c. coordinator dispatches request to mover.  request should 
> >>        include file extents (for future purposes)
> >>     d. normal extents read lock is taken by mover running on client
> >>     e. mover sets "copyout_begin" bit and clears "hsm_dirty" bit in EA.
> >>     f. any writes to the file set the "hsm_dirty" bit (may be 
> >>        lazy/delayed with mtime or filesize change updates on MDT).  Note 
> >>        that file writes need not cancel copyout; for a fs with a single big 
> >>        file, we don't want to keep interrupting copyout or it will never 
> >>        finish. 
> >
> > Is it interesting to have a file that is outdated and possibly 
> > uncoherent?
> It is probably useful in some cases -- simulation checkpoints maybe.

A corrupt simulation checkpoint is useless.  We _must_ provide a way to
ensure the HSM copy of a file is a known good snapshot.  We don't necessarily
have to abort the copyout if there is an update that could mean the
HSM copy would be corrupt since we can always just copy it out again,
but it doesn't seem hugely complicated to notify the backend, if not
the agent and let it decide.

> > Are you sure you want to remove those objects if we will need them 
> > later, in "complex" HSM?
> > As this mecanism will need to change a lot when we will implement the 
> > restore-in-place feature, i'm not sure this is the best idea.
> Ah, I think it is important that we do NOT restore in place to the old 
> OST objects.  The OSTs may now be full, or indeed not exist anymore.  
> The restore in place for complex HSM is at the file level; the objects 
> may move around.  "Complex" in this case just means that clients will 
> have access to partially restored files.

Can't the "complex" HSM restore to new objects?  It just depends on when
the new-being-restored objects become the new contents of the file doesn't it?

> > Copy-out
> > - Mover take a specific lock on range (0-EOF for the moment)
> > - On this range, reads pass, writes raise a callback on the mover.
> > - Receiving this callback, if the mover release its lock, the copyout 
> > is cancelled, if not, the write i/o is blocked 
> I don't think we want to block the write just because the HSM copy isn't 
> done yet.  If the data is changing, then the policy engine shouldn't 
> have started a copyout process in the first place. 

Indeed.

> If the customer's 
> goal is to do a coherent checkpoint, then it should explicitly wait for 
> the copyout to be done.  

Disagree - the customer doesn't have to know a copyout is in progress.
The HSM should abort the copyout or mark the copy corrupt.

> If it's just the policy engine that got it 
> wrong, it doesn't matter if it finishes or not; the file will be marked 
> "hsm_dirty", and so the policy engine should re-queue it for copyout 
> again later, and it can't be purged in the meantime since the dirty bit 
> is set.

Indeed.


    Cheers,
              Eric


From Alexander.Zarochentsev at Sun.COM  Fri Oct 17 08:07:15 2008
From: Alexander.Zarochentsev at Sun.COM (Alexander Zarochentsev)
Date: Fri, 17 Oct 2008 12:07:15 +0400
Subject: [Lustre-devel] COS performance issues
In-Reply-To: <48F36393.9020406@sun.com>
References: <200810081544.08292.alexander.zarochentsev@sun.com>
	<200810131836.37447.alexander.zarochentsev@sun.com>
	<48F36393.9020406@sun.com>
Message-ID: <200810171207.16313.alexander.zarochentsev@sun.com>

On 13 October 2008 19:04:51 Alex Zhuravlev wrote:
> cool! do you have profile with COS disabled? how is it different?
> I guess ldlm_resorce_find() is result of COS as now we've got 10K
> times more locks and resorces, but what about try_to_wake_up() and
> __wake_up_common() ?

What I got after number of performance optimizations:

332019    4.4010  ptlrpc.ko                ptlrpc                   ldlm_resource_find
190465    2.5247  obdclass.ko              obdclass                 htable_lookup
175260    2.3231  e1000.ko                 e1000                    e1000_irq_enable
146324    1.9396  vmlinux                  vmlinux                  schedule
139799    1.8531  e1000.ko                 e1000                    e1000_intr_msi
121729    1.6136  lvfs.ko                  lvfs                     lprocfs_counter_add
105164    1.3940  vmlinux                  vmlinux                  __find_get_block
102296    1.3560  vmlinux                  vmlinux                  memset
96728     1.2822  vmlinux                  vmlinux                  __switch_to
92123     1.2211  vmlinux                  vmlinux                  kfree
90847     1.2042  oprofiled                oprofiled                (no symbols)
84152     1.1155  vmlinux                  vmlinux                  mwait_idle
82378     1.0919  vmlinux                  vmlinux                  __kmalloc
80368     1.0653  vmlinux                  vmlinux                  try_to_wake_up
79110     1.0486  mds.ko                   mds                      mds_lov_dump_objids
71730     0.9508  vmlinux                  vmlinux                  memmove


The ptlrpc_server_handle_reply has gone due to per-reply spin lock implemented,
no try_to_wake_up due to dedicated reply handling threads,
there is only ldlm_resource_find and htable_lookup left, I hope to fix that with the next patch.

The massive create tests shows the following results:

COS=1
Rate: 2085.73 creates/sec (total: 2 threads 625719 creates 300 secs)
Rate: 2072.38 creates/sec (total: 2 threads 621713 creates 300 secs)
Rate: 2053.80 creates/sec (total: 2 threads 616141 creates 300 secs)

COS=0
Rate: 2128.22 creates/sec (total: 2 threads 638465 creates 300 secs)
Rate: 2151.05 creates/sec (total: 2 threads 645316 creates 300 secs)
Rate: 2169.91 creates/sec (total: 2 threads 650974 creates 300 secs)

Clean HEAD

Rate: 2199.67 creates/sec (total: 2 threads 659900 creates 300 secs)
Rate: 2213.95 creates/sec (total: 2 threads 664184 creates 300 secs)
Rate: 2186.74 creates/sec (total: 2 threads 658209 creates 301 secs)
Rate: 2206.10 creates/sec (total: 2 threads 661831 creates 300 secs)
Rate: 2217.49 creates/sec (total: 2 threads 665247 creates 300 secs)
Rate: 2230.86 creates/sec (total: 2 threads 669258 creates 300 secs)

WBR,
-- 
Alexander "Zam" Zarochentsev
Staff Engineer
Lustre Group, Sun Microsystems

From aurelien.degremont at cea.fr  Fri Oct 17 09:47:47 2008
From: aurelien.degremont at cea.fr (Aurelien Degremont)
Date: Fri, 17 Oct 2008 11:47:47 +0200
Subject: [Lustre-devel] "Simple" HSM straw man
In-Reply-To: <CE2BF0610E4C4692A3DDDD842249AAE2@ebpc>
References: <48EE87A1.7000003@sun.com> <48F379B4.6040209@cea.fr>
	<48F4F5DA.7090000@sun.com> <CE2BF0610E4C4692A3DDDD842249AAE2@ebpc>
Message-ID: <48F85F43.2050007@cea.fr>

Eric Barton a écrit :
 > Partially purged files is a requirement to allow graphical file browsers
 > to retrieve icons from within the file.  It's OK to miss this out in the
 > first version, but it has to be there for the full product.

Think also of command like

$ file foo*

 >>> Is it interesting to have a file that is outdated and possibly
 >>> uncoherent?
 >> It is probably useful in some cases -- simulation checkpoints maybe.
 >
 > A corrupt simulation checkpoint is useless.  We _must_ provide a way to
 > ensure the HSM copy of a file is a known good snapshot.  We don't 
necessarily
 > have to abort the copyout if there is an update that could mean the
 > HSM copy would be corrupt since we can always just copy it out again,
 > but it doesn't seem hugely complicated to notify the backend, if not
 > the agent and let it decide.

Indeed, this is important

 >> I don't think we want to block the write just because the HSM copy 
isn't
 >> done yet.  If the data is changing, then the policy engine shouldn't
 >> have started a copyout process in the first place.
 >
 > Indeed.

You were speaking of a FS with only one big file and so we need to have 
a way to be sure it will be copied at least once, even if people are 
writting on it.
In this case, with a classical policy engine, this file will never be 
copied out because data is constantly changing.



-- 
Aurelien Degremont
CEA

From eeb at sun.com  Fri Oct 17 10:10:07 2008
From: eeb at sun.com (Eric Barton)
Date: Fri, 17 Oct 2008 11:10:07 +0100
Subject: [Lustre-devel] "Simple" HSM straw man
In-Reply-To: <48F85F43.2050007@cea.fr>
References: <48EE87A1.7000003@sun.com> <48F379B4.6040209@cea.fr>
	<48F4F5DA.7090000@sun.com> <CE2BF0610E4C4692A3DDDD842249AAE2@ebpc>
	<48F85F43.2050007@cea.fr>
Message-ID: <FE8C312998514690AA1ED80B9125D11E@ebpc>

Aurelien,

>  >> I don't think we want to block the write just because the HSM
>  >> copy isn't done yet.  If the data is changing, then the policy
>  >> engine shouldn't have started a copyout process in the first
>  >> place.
>  >
>  > Indeed.
> 
> You were speaking of a FS with only one big file and so we need to
> have a way to be sure it will be copied at least once, even if
> people are writting on it.  In this case, with a classical policy
> engine, this file will never be copied out because data is
> constantly changing.

I'm not so sure that's a realistic case.  If this file is so active
that it's impossible to take a consistent copy of it without some
sort of a snapshot facility, does that really mean it's a candidate
for archiving?

    Cheers,
              Eric


From Peter.Braam at Sun.COM  Fri Oct 17 13:54:36 2008
From: Peter.Braam at Sun.COM (Peter Braam)
Date: Fri, 17 Oct 2008 07:54:36 -0600
Subject: [Lustre-devel] "Simple" HSM straw man
In-Reply-To: <48F85F43.2050007@cea.fr>
Message-ID: <C51DF53C.6247%peter.braam@sun.com>




On 10/17/08 3:47 AM, "Aurelien Degremont" <aurelien.degremont at cea.fr> wrote:

> Eric Barton a écrit :
>> Partially purged files is a requirement to allow graphical file browsers
>> to retrieve icons from within the file.  It's OK to miss this out in the
>> first version, but it has to be there for the full product.
> 
> Think also of command like
> 
> $ file foo*
> 
>>>> Is it interesting to have a file that is outdated and possibly
>>>> uncoherent?

99.99% of (probably more 9's) backup systems do work this way, with
relatively little harm.

Also remember that many files are append only - for those it might be fine.

Philosophically it is a disaster of course.  I would offer archiving of
files that are active, and in due course use snapshots.

Peter


>>> It is probably useful in some cases -- simulation checkpoints maybe.
>> 
>> A corrupt simulation checkpoint is useless.  We _must_ provide a way to
>> ensure the HSM copy of a file is a known good snapshot.  We don't
> necessarily
>> have to abort the copyout if there is an update that could mean the
>> HSM copy would be corrupt since we can always just copy it out again,
>> but it doesn't seem hugely complicated to notify the backend, if not
>> the agent and let it decide.
> 
> Indeed, this is important
> 
>>> I don't think we want to block the write just because the HSM copy
> isn't
>>> done yet.  If the data is changing, then the policy engine shouldn't
>>> have started a copyout process in the first place.
>> 
>> Indeed.
> 
> You were speaking of a FS with only one big file and so we need to have
> a way to be sure it will be copied at least once, even if people are
> writting on it.
> In this case, with a classical policy engine, this file will never be
> copied out because data is constantly changing.
> 
> 



From rread at sun.com  Fri Oct 17 20:55:03 2008
From: rread at sun.com (Robert Read)
Date: Fri, 17 Oct 2008 13:55:03 -0700
Subject: [Lustre-devel] COS performance issues
In-Reply-To: <200810171207.16313.alexander.zarochentsev@sun.com>
References: <200810081544.08292.alexander.zarochentsev@sun.com>
	<200810131836.37447.alexander.zarochentsev@sun.com>
	<48F36393.9020406@sun.com>
	<200810171207.16313.alexander.zarochentsev@sun.com>
Message-ID: <41338A16-5CFA-4979-BD68-35FE8D80455A@sun.com>


On Oct 17, 2008, at 01:07 , Alexander Zarochentsev wrote:
> The massive create tests shows the following results:
>
> COS=1
> Rate: 2085.73 creates/sec (total: 2 threads 625719 creates 300 secs)
> Rate: 2072.38 creates/sec (total: 2 threads 621713 creates 300 secs)
> Rate: 2053.80 creates/sec (total: 2 threads 616141 creates 300 secs)
>
> COS=0
> Rate: 2128.22 creates/sec (total: 2 threads 638465 creates 300 secs)
> Rate: 2151.05 creates/sec (total: 2 threads 645316 creates 300 secs)
> Rate: 2169.91 creates/sec (total: 2 threads 650974 creates 300 secs)
>
> Clean HEAD
>
> Rate: 2199.67 creates/sec (total: 2 threads 659900 creates 300 secs)
> Rate: 2213.95 creates/sec (total: 2 threads 664184 creates 300 secs)
> Rate: 2186.74 creates/sec (total: 2 threads 658209 creates 301 secs)
> Rate: 2206.10 creates/sec (total: 2 threads 661831 creates 300 secs)
> Rate: 2217.49 creates/sec (total: 2 threads 665247 creates 300 secs)
> Rate: 2230.86 creates/sec (total: 2 threads 669258 creates 300 secs)
>

Any idea why COS=0 was consistently a bit slower than HEAD?

robert



From Alexander.Zarochentsev at Sun.COM  Sat Oct 18 10:34:54 2008
From: Alexander.Zarochentsev at Sun.COM (Alexander Zarochentsev)
Date: Sat, 18 Oct 2008 14:34:54 +0400
Subject: [Lustre-devel] COS performance issues
In-Reply-To: <41338A16-5CFA-4979-BD68-35FE8D80455A@sun.com>
References: <200810081544.08292.alexander.zarochentsev@sun.com>
	<200810171207.16313.alexander.zarochentsev@sun.com>
	<41338A16-5CFA-4979-BD68-35FE8D80455A@sun.com>
Message-ID: <200810181434.54745.alexander.zarochentsev@sun.com>

On 18 October 2008 00:55:03 Robert Read wrote:
> On Oct 17, 2008, at 01:07 , Alexander Zarochentsev wrote:
> > The massive create tests shows the following results:
> >
> > COS=1
> > Rate: 2085.73 creates/sec (total: 2 threads 625719 creates 300
> > secs) Rate: 2072.38 creates/sec (total: 2 threads 621713 creates
> > 300 secs) Rate: 2053.80 creates/sec (total: 2 threads 616141
> > creates 300 secs)
> >
> > COS=0
> > Rate: 2128.22 creates/sec (total: 2 threads 638465 creates 300
> > secs) Rate: 2151.05 creates/sec (total: 2 threads 645316 creates
> > 300 secs) Rate: 2169.91 creates/sec (total: 2 threads 650974
> > creates 300 secs)
> >
> > Clean HEAD
> >
> > Rate: 2199.67 creates/sec (total: 2 threads 659900 creates 300
> > secs) Rate: 2213.95 creates/sec (total: 2 threads 664184 creates
> > 300 secs) Rate: 2186.74 creates/sec (total: 2 threads 658209
> > creates 301 secs) Rate: 2206.10 creates/sec (total: 2 threads
> > 661831 creates 300 secs) Rate: 2217.49 creates/sec (total: 2
> > threads 665247 creates 300 secs) Rate: 2230.86 creates/sec (total:
> > 2 threads 669258 creates 300 secs)
>
> Any idea why COS=0 was consistently a bit slower than HEAD?

I only suspect the ENTRY/EXIT stuff I added to several ptlrpc functions.
next time I will try to build Lustre with CDEBUG_ENTRY_EXIT=0.

> robert

The latest results I got after increasing of ldlm resource
hashtable size are:

zam at lts-head:~$ zgrep Rate: m-ht-improved-WASHIE3.4-20081017.log

COS = 1
Rate: 1887.57 creates/sec (total: 2 threads 566271 creates 300 secs)
Rate: 1980.18 creates/sec (total: 2 threads 594053 creates 300 secs)
Rate: 1923.76 creates/sec (total: 2 threads 577129 creates 300 secs)

COS = 0
Rate: 1804.92 creates/sec (total: 2 threads 541475 creates 300 secs)
Rate: 1915.29 creates/sec (total: 2 threads 574587 creates 300 secs)
Rate: 2025.70 creates/sec (total: 2 threads 607711 creates 300 secs)


CLEAN HEAD:
zam at lts-head:~$ zgrep Rate: m-cleanhead-WASHIE3.4-20081017.log
Rate: 1986.13 creates/sec (total: 2 threads 595839 creates 300 secs)
Rate: 2009.84 creates/sec (total: 2 threads 602952 creates 300 secs)
Rate: 1887.05 creates/sec (total: 2 threads 566115 creates 300 secs)
Rate: 2060.59 creates/sec (total: 2 threads 618178 creates 300 secs)
Rate: 1868.89 creates/sec (total: 2 threads 560666 creates 300 secs)
Rate: 1976.95 creates/sec (total: 2 threads 594084 creates 301 secs)
zam at lts-head:~$

average values:

COS=1 : 1930.5
COS=0 : 1915.3
HEAD  : 1964.9

The results are done on another test cluster 
so they are not directly comparable with the previous ones.

The profiling chart for COS=1, there is no ldlm_resource_find anymore in top20:

198152    2.9052  e1000.ko                 e1000                    e1000_irq_enable
158535    2.3243  e1000.ko                 e1000                    e1000_intr_msi
152375    2.2340  vmlinux                  vmlinux                  schedule
148709    2.1803  obdclass.ko              obdclass                 htable_lookup
109440    1.6045  lvfs.ko                  lvfs                     lprocfs_counter_add
102773    1.5068  vmlinux                  vmlinux                  __switch_to
95607     1.4017  vmlinux                  vmlinux                  __find_get_block
90447     1.3261  vmlinux                  vmlinux                  kfree
88713     1.3006  vmlinux                  vmlinux                  mwait_idle
84511     1.2390  vmlinux                  vmlinux                  try_to_wake_up
83801     1.2286  vmlinux                  vmlinux                  memset
79735     1.1690  vmlinux                  vmlinux                  __kmalloc
78594     1.1523  oprofiled                oprofiled                (no symbols)
71184     1.0437  ksocklnd.ko              ksocklnd                 ksocknal_scheduler
65613     0.9620  vmlinux                  vmlinux                  kmem_cache_free
64525     0.9460  mds.ko                   mds                      mds_lov_dump_objids
64208     0.9414  ptlrpc.ko                ptlrpc                   ptlrpc_main
63260     0.9275  vmlinux                  vmlinux                  memmove
56148     0.8232  vmlinux                  vmlinux                  find_get_page
55011     0.8065  vmlinux                  vmlinux                  kmem_cache_alloc

I attached all performance enhancement patches 
(except the latest patch of ldlm resource hash table size increase) 
to the COS bug (Bug 15393).

Thanks,
-- 
Alexander "Zam" Zarochentsev
Staff Engineer
Lustre Group, Sun Microsystems

From Nathan.Rutman at Sun.COM  Mon Oct 20 23:09:12 2008
From: Nathan.Rutman at Sun.COM (Nathaniel Rutman)
Date: Mon, 20 Oct 2008 16:09:12 -0700
Subject: [Lustre-devel] HSM arch wiki
In-Reply-To: <48F379B4.6040209@cea.fr>
References: <48EE87A1.7000003@sun.com> <48F379B4.6040209@cea.fr>
Message-ID: <48FD0F98.9040300@sun.com>

High-level architecture page for the Lustre HSM project
http://arch.lustre.org/index.php?title=HSM_Migration


HSM core team - this is intended to be sufficient to write a full 
HLD/DLD from.  What is it missing?

From aurelien.degremont at cea.fr  Tue Oct 21 13:21:37 2008
From: aurelien.degremont at cea.fr (Aurelien Degremont)
Date: Tue, 21 Oct 2008 15:21:37 +0200
Subject: [Lustre-devel] HSM arch wiki
In-Reply-To: <48FD0F98.9040300@sun.com>
References: <48EE87A1.7000003@sun.com> <48F379B4.6040209@cea.fr>
	<48FD0F98.9040300@sun.com>
Message-ID: <48FDD761.1070602@cea.fr>

Nathaniel Rutman a écrit :
> High-level architecture page for the Lustre HSM project
> http://arch.lustre.org/index.php?title=HSM_Migration
>
>
> HSM core team - this is intended to be sufficient to write a full 
> HLD/DLD from.  What is it missing?
>
I think after tuestday conf call, all the main element for writting hld 
will be their. May be some small points will be missing but we could 
discuss them by e-mail or at confcalls. I think most of it is already there.

-- 
Aurelien Degremont
CEA


From H.Huang at Sun.COM  Wed Oct 22 05:44:06 2008
From: H.Huang at Sun.COM (Huang Hua)
Date: Wed, 22 Oct 2008 13:44:06 +0800
Subject: [Lustre-devel] Simpifying Interoperation
In-Reply-To: <CF0D4EA95C1945F18F74426BB95008CE@ebpc>
References: <CF0D4EA95C1945F18F74426BB95008CE@ebpc>
Message-ID: <48FEBDA6.4070200@sun.com>

Hello Eric,
I have some updates on the interop development.

1. I have implemented a "Barrier" on client, (by a read write semaphore,
but Andreas suggests to use mutex).
Before upgrade the MDS/OSS, we setup barrier on all clients, stopping
new requests being sent to MDS and OSS.
Currently, this will be done manually, e.g. run a command on all clients
to barrier them.
And then, user can explicitly sync lustre from clients, MDS and OSS, to
insure that no outstanding requests are there.
And then, cancel all mdc and osc locks on clients manually. Maybe this
step is optional. I will do more testing.

2. User can upgrade MDS now. In this step, we need to "tunefs.lustre
--write_conf" to erase all configuration, because these
configuration can not be recognized by 2.0 MDS.

3. User has to upgrade OSS, or restart OSS to re-generate the configuration.
Both are OK.

4. After that, we can cleanup the barrier on client manually. Client
will reconnect to server, recover, and continue to run seamlessly.


The problems here is that we have to setup and cleanup barrier on all
clients by hand.
Ideally, we should do it on a MDS/MGS by DLM lock, or something similar.
If this is strongly required, I will implement this later.
Till now, preliminary upgrade/downgrade tests passed. More testing are
underway.

I will answer some of your questions inlinely. Please see the followings.




Eric Barton wrote:
> Here are some first thoughts on Huang Hua's idea to simplify version
> interoperation, and an invitation for further comments...
>
> 1. This scheme should not interfere with upgrade via failover pairs,
>    and it must also allow the MDS to be upgraded separately from the
>    OSSs.  I think this means in general that we have to allow
>    piecemeal server upgrades.
>
> 2. This scheme need a mechanism that...
>
>    a) notifies clients when a particular server is about to upgrade so
>       that update operations are blocked until the upgrade completes
>       and the client reconnects to the upgraded (and/or failed over)
>       server.
>   
Now, this notification is done on every client, manually, by a command.

>    b) notifies the server when all clients have completed preparation
>       for the upgrade so that no further requests require resend.
>   
This is done when barriers have been setup on all clients, and sync have
been run on clients.

>    c) notifies clients when all outstanding updates have been
>       committed.  If the server crashes before this point, client
>       replay is still required.  Clients must not poll for this since
>       the server is shutting down.
>
>    The DLM seems the right basic mechanism to notify clients, however
>    current assumptions about acquisition timeouts might be an issue.
>
>    We must also ensure that the race between this server upgrade
>    process and connection establishment (including any new
>    notification locks) by new clients is handled consistently.
>   
I think these races should be avoided by user.


>    
> 3. It's not clear to me that we need to evict, or even clean the
>    client cache provided the client doesn't attempt any more writes
>    until it has connected to the failover server. 
While upgrade, we do not need to evict the clients.
but, in downgrade, we have to evict clients, because
the 1.8 mds server does not understand FIDs, it only knows inode number.
But while 1.8 client is talking to 2.0 MDS, they talk in FIDs, and know
nothing about real inode numbers.


>  The client can
>    re-acquire all the locks covering its cache during recovery after
>    the upgrade - and there is no need for request refomatting here
>    since locks are replayed explicitly (i.e. new requests are
>    formatted from scratch using the correct protocol version).
>
>    It does seem advisable however to clean the cache before such a
>    significant system incident.
>
> 4. We can avoid reformatting requests during open replay if this is
>    also done explicity.
>   
while upgrading, the client will do open replay.
The server has already committed this open.
So, 2.0 MDS only need to "open" that file, and return handle back to client.


> 5. This scheme prevents recovery on clients that were disconnected
>    when the upgrade began.  Such clients will simply be evicted when
>    they reconnect even though the server should actually have
>    committed all their replayable requests.
>
>    If this can be prevented, we can probably also dispense with much
>    of the notification described in (2) above.  However it would
>    require (a) a change in the connection protocol to get clients to
>    purge their own replay queue and (b) changes to ensure resent
>    requests can be reconstructed from scratch (but maybe (b) is just
>    another way of saying "request reformatting").
>
>    If this is doable - it further begs the question of whether simply
>    making all server requests synchronous during upgrades is enough to
>    simply most interoperation issues.
>
> 6. This is all about client/server communications. Are there any
>    issues for inter-server interoperation?
>   

The protocol for mds-oss does not change much.
As I tested, there is no inter-server interop issues.


> 7. Clients and servers may have to run with different versions for
>    extended periods (one customer ran like this for months).  Does
>    this raise any issues with this scheme?
>   
I do not see any issues.
More testing is needed.

Thanks,
Huang Hua

>     Cheers,
>               Eric
>
>   


From adilger at sun.com  Wed Oct 22 17:21:40 2008
From: adilger at sun.com (Andreas Dilger)
Date: Wed, 22 Oct 2008 11:21:40 -0600
Subject: [Lustre-devel] Simpifying Interoperation
In-Reply-To: <48FEBDA6.4070200@sun.com>
References: <CF0D4EA95C1945F18F74426BB95008CE@ebpc> <48FEBDA6.4070200@sun.com>
Message-ID: <20081022172139.GI3184@webber.adilger.int>

On Oct 22, 2008  13:44 +0800, Huang Hua wrote:
> 1. I have implemented a "Barrier" on client, (by a read write semaphore,
> but Andreas suggests to use mutex).
> Before upgrade the MDS/OSS, we setup barrier on all clients, stopping
> new requests being sent to MDS and OSS.
> Currently, this will be done manually, e.g. run a command on all clients
> to barrier them.
> And then, user can explicitly sync lustre from clients, MDS and OSS, to
> insure that no outstanding requests are there.

I don't think that "sync" will cause existing RPCs to be sent.  Not a bad
idea, but I don't think it is implemented.

> And then, cancel all mdc and osc locks on clients manually. Maybe this
> step is optional. I will do more testing.
> 
> 2. User can upgrade MDS now. In this step, we need to "tunefs.lustre
> --write_conf" to erase all configuration, because these
> configuration can not be recognized by 2.0 MDS.

Can you explain why this is necessary?  Is this only for the MDT and OST
configurations, or the client configuration also?  The only change I was
aware of with the configuration log was for sptlrpc and that has been
fixed by Eric Mei to use a separate log.  Rewriting the client config
log while the clients are mounted would cause problems later on the client.

> 3. User has to upgrade OSS, or restart OSS to re-generate the configuration.
> Both are OK.

I didn't think the configured devices on the OSS were changed (e.g. OSD
is not yet in use), so what is the reason to rewrite the config there?

> 4. After that, we can cleanup the barrier on client manually. Client
> will reconnect to server, recover, and continue to run seamlessly.
> 
> 
> The problems here is that we have to setup and cleanup barrier on all
> clients by hand.
> Ideally, we should do it on a MDS/MGS by DLM lock, or something similar.
> If this is strongly required, I will implement this later.

In some customer configurations there is no easy mechanism for running
e.g. pdsh on all of the clients in advance of the upgrade.

> Till now, preliminary upgrade/downgrade tests passed. More testing are
> underway.
> 
> I will answer some of your questions inlinely. Please see the followings.
> 
> 
> 
> 
> Eric Barton wrote:
> > Here are some first thoughts on Huang Hua's idea to simplify version
> > interoperation, and an invitation for further comments...
> >
> > 1. This scheme should not interfere with upgrade via failover pairs,
> >    and it must also allow the MDS to be upgraded separately from the
> >    OSSs.  I think this means in general that we have to allow
> >    piecemeal server upgrades.
> >
> > 2. This scheme need a mechanism that...
> >
> >    a) notifies clients when a particular server is about to upgrade so
> >       that update operations are blocked until the upgrade completes
> >       and the client reconnects to the upgraded (and/or failed over)
> >       server.
> >   
> Now, this notification is done on every client, manually, by a command.
> 
> >    b) notifies the server when all clients have completed preparation
> >       for the upgrade so that no further requests require resend.
>
> This is done when barriers have been setup on all clients, and sync have
> been run on clients.

Even with the current manual operation, the MDS itself doesn't know when
the clients have all been updated.  There should be some mechanism by
which the MDS knows for sure that the clients have done their barrier,
and clients which have not responded should be evicted.

This is easily done by having a DLM read lock that all clients hold all of
the time.  It should not be put on the LRU, and the only reason it should
conflict is if the server is doing a shutdown.  The server would revoke
that lock (by enqueueing a conflicting lock) and then setting a local
flag which ensures that no more RPCs will be processed by the server.

> >    c) notifies clients when all outstanding updates have been
> >       committed.  If the server crashes before this point, client
> >       replay is still required.  Clients must not poll for this since
> >       the server is shutting down.
> >
> >    The DLM seems the right basic mechanism to notify clients, however
> >    current assumptions about acquisition timeouts might be an issue.
> >
> >    We must also ensure that the race between this server upgrade
> >    process and connection establishment (including any new
> >    notification locks) by new clients is handled consistently.
> >   
> I think these races should be avoided by user.
> 
> 
> >    
> > 3. It's not clear to me that we need to evict, or even clean the
> >    client cache provided the client doesn't attempt any more writes
> >    until it has connected to the failover server. 
> While upgrade, we do not need to evict the clients.
> but, in downgrade, we have to evict clients, because
> the 1.8 mds server does not understand FIDs, it only knows inode number.
> But while 1.8 client is talking to 2.0 MDS, they talk in FIDs, and know
> nothing about real inode numbers.

Yes, this is to be expected, and in that case the server should be
unmounted with "-f" or recovery aborted when 1.8 is restarted.

> >  The client can
> >    re-acquire all the locks covering its cache during recovery after
> >    the upgrade - and there is no need for request refomatting here
> >    since locks are replayed explicitly (i.e. new requests are
> >    formatted from scratch using the correct protocol version).
> >
> >    It does seem advisable however to clean the cache before such a
> >    significant system incident.
> >
> > 4. We can avoid reformatting requests during open replay if this is
> >    also done explicity.
> >   
> while upgrading, the client will do open replay.
> The server has already committed this open.
> So, 2.0 MDS only need to "open" that file, and return handle back to client.

Does the 2.0 MDS have support for handling the 1.8 MDS RPC open request
format?

> > 5. This scheme prevents recovery on clients that were disconnected
> >    when the upgrade began.  Such clients will simply be evicted when
> >    they reconnect even though the server should actually have
> >    committed all their replayable requests.
> >
> >    If this can be prevented, we can probably also dispense with much
> >    of the notification described in (2) above.  However it would
> >    require (a) a change in the connection protocol to get clients to
> >    purge their own replay queue and (b) changes to ensure resent
> >    requests can be reconstructed from scratch (but maybe (b) is just
> >    another way of saying "request reformatting").
> >
> >    If this is doable - it further begs the question of whether simply
> >    making all server requests synchronous during upgrades is enough to
> >    simply most interoperation issues.
> >
> > 6. This is all about client/server communications. Are there any
> >    issues for inter-server interoperation?
> 
> The protocol for mds-oss does not change much.
> As I tested, there is no inter-server interop issues.
> 
> > 7. Clients and servers may have to run with different versions for
> >    extended periods (one customer ran like this for months).  Does
> >    this raise any issues with this scheme?
>
> I do not see any issues.
> More testing is needed.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.


From rread at sun.com  Wed Oct 22 18:06:11 2008
From: rread at sun.com (Robert Read)
Date: Wed, 22 Oct 2008 11:06:11 -0700
Subject: [Lustre-devel] Simpifying Interoperation
In-Reply-To: <48FEBDA6.4070200@sun.com>
References: <CF0D4EA95C1945F18F74426BB95008CE@ebpc> <48FEBDA6.4070200@sun.com>
Message-ID: <26338A6E-C0A5-4E7F-8957-A8694AD566E1@sun.com>

Huang,

Have you seen the new recovery interop architecture that came out of  
the menlo park meetings?

http://arch.lustre.org/index.php?title=Simplified_Interoperation

robert

On Oct 21, 2008, at 22:44 , Huang Hua wrote:

> Hello Eric,
> I have some updates on the interop development.
>
> 1. I have implemented a "Barrier" on client, (by a read write  
> semaphore,
> but Andreas suggests to use mutex).
> Before upgrade the MDS/OSS, we setup barrier on all clients, stopping
> new requests being sent to MDS and OSS.
> Currently, this will be done manually, e.g. run a command on all  
> clients
> to barrier them.
> And then, user can explicitly sync lustre from clients, MDS and OSS,  
> to
> insure that no outstanding requests are there.
> And then, cancel all mdc and osc locks on clients manually. Maybe this
> step is optional. I will do more testing.
>
> 2. User can upgrade MDS now. In this step, we need to "tunefs.lustre
> --write_conf" to erase all configuration, because these
> configuration can not be recognized by 2.0 MDS.
>
> 3. User has to upgrade OSS, or restart OSS to re-generate the  
> configuration.
> Both are OK.
>
> 4. After that, we can cleanup the barrier on client manually. Client
> will reconnect to server, recover, and continue to run seamlessly.
>
>
> The problems here is that we have to setup and cleanup barrier on all
> clients by hand.
> Ideally, we should do it on a MDS/MGS by DLM lock, or something  
> similar.
> If this is strongly required, I will implement this later.
> Till now, preliminary upgrade/downgrade tests passed. More testing are
> underway.
>
> I will answer some of your questions inlinely. Please see the  
> followings.
>
>
>
>
> Eric Barton wrote:
>> Here are some first thoughts on Huang Hua's idea to simplify version
>> interoperation, and an invitation for further comments...
>>
>> 1. This scheme should not interfere with upgrade via failover pairs,
>>   and it must also allow the MDS to be upgraded separately from the
>>   OSSs.  I think this means in general that we have to allow
>>   piecemeal server upgrades.
>>
>> 2. This scheme need a mechanism that...
>>
>>   a) notifies clients when a particular server is about to upgrade so
>>      that update operations are blocked until the upgrade completes
>>      and the client reconnects to the upgraded (and/or failed over)
>>      server.
>>
> Now, this notification is done on every client, manually, by a  
> command.
>
>>   b) notifies the server when all clients have completed preparation
>>      for the upgrade so that no further requests require resend.
>>
> This is done when barriers have been setup on all clients, and sync  
> have
> been run on clients.
>
>>   c) notifies clients when all outstanding updates have been
>>      committed.  If the server crashes before this point, client
>>      replay is still required.  Clients must not poll for this since
>>      the server is shutting down.
>>
>>   The DLM seems the right basic mechanism to notify clients, however
>>   current assumptions about acquisition timeouts might be an issue.
>>
>>   We must also ensure that the race between this server upgrade
>>   process and connection establishment (including any new
>>   notification locks) by new clients is handled consistently.
>>
> I think these races should be avoided by user.
>
>
>>
>> 3. It's not clear to me that we need to evict, or even clean the
>>   client cache provided the client doesn't attempt any more writes
>>   until it has connected to the failover server.
> While upgrade, we do not need to evict the clients.
> but, in downgrade, we have to evict clients, because
> the 1.8 mds server does not understand FIDs, it only knows inode  
> number.
> But while 1.8 client is talking to 2.0 MDS, they talk in FIDs, and  
> know
> nothing about real inode numbers.
>
>
>> The client can
>>   re-acquire all the locks covering its cache during recovery after
>>   the upgrade - and there is no need for request refomatting here
>>   since locks are replayed explicitly (i.e. new requests are
>>   formatted from scratch using the correct protocol version).
>>
>>   It does seem advisable however to clean the cache before such a
>>   significant system incident.
>>
>> 4. We can avoid reformatting requests during open replay if this is
>>   also done explicity.
>>
> while upgrading, the client will do open replay.
> The server has already committed this open.
> So, 2.0 MDS only need to "open" that file, and return handle back to  
> client.
>
>
>> 5. This scheme prevents recovery on clients that were disconnected
>>   when the upgrade began.  Such clients will simply be evicted when
>>   they reconnect even though the server should actually have
>>   committed all their replayable requests.
>>
>>   If this can be prevented, we can probably also dispense with much
>>   of the notification described in (2) above.  However it would
>>   require (a) a change in the connection protocol to get clients to
>>   purge their own replay queue and (b) changes to ensure resent
>>   requests can be reconstructed from scratch (but maybe (b) is just
>>   another way of saying "request reformatting").
>>
>>   If this is doable - it further begs the question of whether simply
>>   making all server requests synchronous during upgrades is enough to
>>   simply most interoperation issues.
>>
>> 6. This is all about client/server communications. Are there any
>>   issues for inter-server interoperation?
>>
>
> The protocol for mds-oss does not change much.
> As I tested, there is no inter-server interop issues.
>
>
>> 7. Clients and servers may have to run with different versions for
>>   extended periods (one customer ran like this for months).  Does
>>   this raise any issues with this scheme?
>>
> I do not see any issues.
> More testing is needed.
>
> Thanks,
> Huang Hua
>
>>    Cheers,
>>              Eric
>>
>>
>
> _______________________________________________
> Lustre-devel mailing list
> Lustre-devel at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-devel


From Nathan.Rutman at Sun.COM  Mon Oct 27 23:02:34 2008
From: Nathan.Rutman at Sun.COM (Nathaniel Rutman)
Date: Mon, 27 Oct 2008 16:02:34 -0700
Subject: [Lustre-devel] HSM comments
In-Reply-To: <4905F121.6000501@cea.fr>
References: <4905F121.6000501@cea.fr>
Message-ID: <4906488A.8030006@sun.com>

Aurelien Degremont wrote:
> Hello all
>
> I'm sending this e-mail directly because for some unknown reasons, it 
> did not reach the mailing lists (either lustre-hsm-core-ext or 
> lustre-devel) last week.
>
> Find attached, 2 schemas presenting the various messages exchanged by 
> Lustre HSM components for copyin and copyout. Tell me if this is not 
> what you were expecting, I can fix them for tomorrow conf call. I 
> think I will add some other schemas also.
>
>
>
> Some points we did not discuss at last conf call:
>
> * File unlinks:
>
> - HSM object removal should be async.
Agreed, trigger should just be changelog unlink entry.
>
> - We should not link hsm object, even in v1. Could we manage hsm object
> deletion like ost object deletion and manage orphan in the same way?
Since the unlink event trigger is the changelog record, the policy 
engine should simply not cancel the changelog record until the HSM 
confirms the unlink.  
>
> - Presently, we could also leak hsm objects if the file is dirtied when
> being copied out. In this case, the file will be tagged dirtied, with no
> copyout_begin/complete flag. So the MDT will not request for HSM removal
> but their is something to delete there. Maybe the copyout mechanisms
> should be adapted.
How about we never clear the copyout_begin bit?  This is really for the 
coordinator's benefit so it knows a copyout is in progress on that file, 
but since we're having regular status updates to the coordinator from 
the agent, there's no real need for that bit.  So instead we have the 
bit "a_file_exists_in_hsm" aka hsm_exists.
But we don't even need that - the MDT does not "request for HSM 
removal", but instead the policy engine just watches the changelog for 
unlink events.  Ah, now I see the problem with using the changelog - 
this forces the policy engine to remember which files are on HSM, or 
accept an error return code, but in any case may result in much undue 
load on the HSM when deleting non-HSM'ed files.  So what do we do?  
Ignore the changelog and have the MDT directly signal the coordinator to 
do HSM unlinks?  That may be fine.  In that case, I think if we leak 
files after we tell the coordinator to delete them it is not much of a 
problem.

>
> * HSM dirty bit.
>
> - should be updated with laziness.
> - Is it possible to implement it like the lazy file size? That means,
> manage the dirty bit, per OST object, and lazily update it on the MDT?
Since file mtime/size is already updated this way, we can just use any 
attr change as the dirty indicator; we don't need an actual bit per 
object.  Any setattr should update MDT dirty bit, most setxattr should 
(not the hardlink/parent xattr however, maybe no XATTR_TRUSTED_PREFIX 
ones).  
>
> - Also, if, instead of setting hsm_dirty bit to 1 when the file is
> modified, can we do counter += 1 ? That way 'counter' could be use as
> 'light' file revision. You compare two versions of this variable, is 
> their differ, the file has been modified  (this is not
> intended to check 'counter_c1 < counter_c2' but just 'counter_c1 !=
> counter_c2', that way, you can have circular counters.) 
I have no objection, although I don't see the benefit right now.  E.g. 
how is that different than checking the mtime?
> - Could a policy test could be based on file path (not just filename and
> properties) ? This is a rule we presently used in our hsm tools. I do
> not see how have the filepath from the changelogs data ?
The changelog data has file and parent FID, if you want more path than 
this you can do a "lfs fid2path" to reconstruct the entire path name.  
Note however this returns only the "first" path of a hardlinked file.  
(Is this a limitation?  Do I need to fix fid2path?)
>
> - Could this flag be exposed to userspace via liblustreapi? Maybe this
> flag should be set on file creation also? Doing this, Policy Engines
> could use this flag to know easily if the file is udate to date in hsm
> or not.
Sounds good.
>
> * Policy Engine
>
> - It needs to:
> . read changelogs (mdt)
> . df (mdt/client)
> . lfs df (per ost) (mdt/client)
> . scan namespace (client)
> . lfs getstripe by fid (client)
> . stat file by fid (client)
>
> The only thing the engine will lack on a client is the changelogs. May
> be it could be a good idea to export the changelogs on some 'trusted
> clients' ?
I think it's sticky to impose certain priviledged clients, but maybe 
exporting to all clients isn't so bad.  Superuser privs on any client 
gives them access.  If anyone really hates this, we can add a tunable on 
the MDT to allow/disallow all client access.
> If not, we will be force to have MDT, client mount and policy engine on
> the same node or split the policy engine into two components (very bad
> idea to impose that on the engine). Potential overhead?
>
>
ok, client access to changelogs sounds like a reasonable requirement.  
[Note: this actually happens to solve a problem I haven't figured out 
yet, which is to limit access to only disk-committed changelog records.]
>
>
> ------------------------------------------------------------------------
>
>
#10 is "open reply", not "i/o reply", but a very nice diagram!  Can you 
add these to the wiki?
> ------------------------------------------------------------------------
>


From jeffd at sicortex.com  Wed Oct 22 17:12:07 2008
From: jeffd at sicortex.com (Jeff Darcy)
Date: Wed, 22 Oct 2008 13:12:07 -0400
Subject: [Lustre-devel] [Lustre-discuss] Misaligned buffers being handed to
	LND?
Message-ID: <48FF5EE7.4090205@sicortex.com>

Have any other LND authors (either at Sun or elsewhere) noticed recent 
Lustre versions passing iovs with non-page-aligned middle elements down 
to the LND layer?  We've always seen plenty of misaligned start/end 
elements, but prior to 1.6.3 we never saw it with middle elements.  We 
were advised that this would never be the case, and some code was 
written based on that assumption.  Now we do see this behavior from lnet 
- always associated with large/complex "ld" invocations while building 
programs, oddly enough.  This is a little non-trivial to deal with in 
our LND for the SiCortex fabric interconnect, and is likely to remain a 
performance issue even with code in place to handle it.  Have any of the 
folks working on other LNDs hit this?
_______________________________________________
Lustre-discuss mailing list
Lustre-discuss at lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

From peter.braam at gmail.com  Mon Oct 27 15:06:26 2008
From: peter.braam at gmail.com (Peter J. Braam)
Date: Mon, 27 Oct 2008 09:06:26 -0600
Subject: [Lustre-devel] stepping back from Lustre
Message-ID: <7bb097cc0810270806l1fd3b366ra146147377bf91a3@mail.gmail.com>

I am writing today to let you know that after working for almost exactly 10
years on Lustre, I am going to step back from the project, slow down a bit
and return to research.  Today is my last day at Sun.  Peter Bojanic will
continue as the project lead, and the principal developers on the team led
by Eric Barton will do the architecture.

The community and the team have created ideas and technology that went much
further than I ever thought it would do.  I thank both for their support,
and I will be following with curiosity how things develop further.  I expect
to at least occasionally make suggestions on the lists and you can also
reach me at  first.last at gmail.com (with appropriate substitutions).

Best wishes,

Peter Braam

ps: I will not be attending SC 2008 because of a once-in-a-lifetime family
event
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel_lists.lustre.org/attachments/20081027/8ce172bb/attachment.html>

From aurelien.degremont at cea.fr  Tue Oct 28 15:42:09 2008
From: aurelien.degremont at cea.fr (DEGREMONT Aurelien)
Date: Tue, 28 Oct 2008 16:42:09 +0100
Subject: [Lustre-devel] HSM comments
In-Reply-To: <4906488A.8030006@sun.com>
References: <4905F121.6000501@cea.fr> <4906488A.8030006@sun.com>
Message-ID: <490732D1.8060103@cea.fr>

Nathaniel Rutman a écrit :
>> - HSM object removal should be async.
> Agreed, trigger should just be changelog unlink entry.
I'm not sure Lustre need the policy engine for managing the hsm removals.
It could triggers them automatically (like the copy-in mechanisms) when 
the file is deleted in Lustre.
Lustre could still live for a long moment without the PolicyEngine/Space 
Manager, we could imagine this for several hours.

>> - We should not link hsm object, even in v1. Could we manage hsm object
>> deletion like ost object deletion and manage orphan in the same way?
> Since the unlink event trigger is the changelog record, the policy 
> engine should simply not cancel the changelog record until the HSM 
> confirms the unlink.
For the moment, the PolicyEngine has no way to know the copytool has 
successfully deleted the file.
> How about we never clear the copyout_begin bit?  This is really for 
> the coordinator's benefit so it knows a copyout is in progress on that 
> file, but since we're having regular status updates to the coordinator 
> from the agent, there's no real need for that bit.  So instead we have 
> the bit "a_file_exists_in_hsm" aka hsm_exists.
> But we don't even need that - the MDT does not "request for HSM 
> removal", but instead the policy engine just watches the changelog for 
> unlink events.  Ah, now I see the problem with using the changelog - 
> this forces the policy engine to remember which files are on HSM, or 
> accept an error return code, but in any case may result in much undue 
> load on the HSM when deleting non-HSM'ed files.  So what do we do?  
> Ignore the changelog and have the MDT directly signal the coordinator 
> to do HSM unlinks?  That may be fine.  In that case, I think if we 
> leak files after we tell the coordinator to delete them it is not much 
> of a problem.
If we store in a llog the hsm objects that need to be removed and only 
delete them when copytool says it's fine, we will not leak files and if 
coordinator crashes, the copyin and removal requests will be resent 
automatically. The PolicyEngine will also re-send copy-out requests.
>> * HSM dirty bit.
>>
>> - should be updated with laziness.
>> - Is it possible to implement it like the lazy file size? That means,
>> manage the dirty bit, per OST object, and lazily update it on the MDT?
>
> Since file mtime/size is already updated this way, we can just use any 
> attr change as the dirty indicator; we don't need an actual bit per 
> object.  
Dirty means data were changed, not metadata.

>> - Also, if, instead of setting hsm_dirty bit to 1 when the file is
>> modified, can we do counter += 1 ? That way 'counter' could be use as
>> 'light' file revision. You compare two versions of this variable, is 
>> their differ, the file has been modified  (this is not
>> intended to check 'counter_c1 < counter_c2' but just 'counter_c1 !=
>> counter_c2', that way, you can have circular counters.) 
> I have no objection, although I don't see the benefit right now.  E.g. 
> how is that different than checking the mtime?
mtime could not be trust. mtime is a user exposed value that could be 
changed by user as he likes it.

$ touch -t 200101010000 foo
$ ls -l foo
-rw-r--r-- 1 degremont user 0 Jan  1  2001 foo

> The changelog data has file and parent FID, if you want more path than 
> this you can do a "lfs fid2path" to reconstruct the entire path name.  
> Note however this returns only the "first" path of a hardlinked file.  
> (Is this a limitation?  Do I need to fix fid2path?)
Ok this is fine, enough for our needs.

> #10 is "open reply", not "i/o reply", but a very nice diagram!  Can 
> you add these to the wiki?
>
Thanks. Done.


Aurélien


From adilger at sun.com  Tue Oct 28 18:26:13 2008
From: adilger at sun.com (Andreas Dilger)
Date: Tue, 28 Oct 2008 12:26:13 -0600
Subject: [Lustre-devel] HSM comments
In-Reply-To: <4907249E.60904@cea.fr>
References: <4905F121.6000501@cea.fr>
	<20081027231204.GP3184@webber.adilger.int> <4907249E.60904@cea.fr>
Message-ID: <20081028182613.GX3184@webber.adilger.int>

On Oct 28, 2008  15:41 +0100, DEGREMONT Aurelien wrote:
> Andreas Dilger a écrit :
>> I would recommend that we can keep a reference to a "dirty" HSM object
>> even if the copyout did not complete successfully, and HSM policy engine
>> should decide if the dirty object is kept or deleted.  In some cases
>> it may never be possible to do a complete copyout of the file, and having
>> some copy of the file would be better than having none at all.
>   
> So we will have three states for the hsm object:
> - exist
> - completed (exist & copy was completed)
> - uptodate (exist, copy was completed and the lustre file was not dirtied)

We already need to have the distinction between "completed" and "uptodate"
because an "uptodate" HSM copy stop being uptodate as soon as the file is
again modified.

I was a bit unclear when I wrote "...a complete copyout of the file".  What
I meant was "impossible to ever complete an uptodate copyout of the file
if it is continually changing".  I don't think we need to make any distinction
between "exist" and "complete" because both mean "not uptodate".

>>> - Also, if, instead of setting hsm_dirty bit to 1 when the file is
>>> modified, can we do counter += 1 ? That way 'counter' could be use as
>>> 'light' file revision. You compare two versions of this variable, is
>>> their differ, the file has been modified  (this is not intended to check
>>> 'counter_c1 < counter_c2' but just 'counter_c1 != counter_c2', that way,
>>> you can have circular counters.)
>>>     
>> The MDS in 1.8 (and soon 2.0) will already keep a version counter for all
>> changes to the MDS inode.  The OSTs will also keep version numbers for
>> all of the objects there.
>>   
> In our first (and old) hlds, we based several mechanisms on such  
> information.  But we were told that the version will be available
> for MDT but not for OST object

This hasn't changed - the OST object versions are local to the OSTs.
We could get this OST version information at the agent (just like with
file size being fetched from all OSTs) and store it with the HSM archive
copy.  This can be a later optimization, however.  I think mtime is
itself sufficient for an initial indication for file data changes.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.


From adilger at sun.com  Mon Oct 27 23:12:04 2008
From: adilger at sun.com (Andreas Dilger)
Date: Mon, 27 Oct 2008 17:12:04 -0600
Subject: [Lustre-devel] HSM comments
In-Reply-To: <4905F121.6000501@cea.fr>
References: <4905F121.6000501@cea.fr>
Message-ID: <20081027231204.GP3184@webber.adilger.int>

On Oct 27, 2008  17:49 +0100, Aurelien Degremont wrote:
> Some points we did not discuss at last conf call:
>
> * File unlinks:
>
> - HSM object removal should be async.
> - We should not link hsm object, even in v1. Could we manage hsm object
> deletion like ost object deletion and manage orphan in the same way?

I wouldn't object to this - there could be an "HSM unlink" llog, similar
to the OST unlink llog that the HSM coordinator (either in the kernel,
or in userspace) processes at startup.  The difficulty is to know when
the llog record can be cancelled.

> - Presently, we could also leak hsm objects if the file is dirtied when
> being copied out. In this case, the file will be tagged dirtied, with no
> copyout_begin/complete flag. So the MDT will not request for HSM removal
> but their is something to delete there. Maybe the copyout mechanisms
> should be adapted.

I would recommend that we can keep a reference to a "dirty" HSM object
even if the copyout did not complete successfully, and HSM policy engine
should decide if the dirty object is kept or deleted.  In some cases
it may never be possible to do a complete copyout of the file, and having
some copy of the file would be better than having none at all.

> * HSM dirty bit.
>
> - should be updated with laziness.
> - Is it possible to implement it like the lazy file size? That means,
> manage the dirty bit, per OST object, and lazily update it on the MDT?

> - Also, if, instead of setting hsm_dirty bit to 1 when the file is
> modified, can we do counter += 1 ? That way 'counter' could be use as
> 'light' file revision. You compare two versions of this variable, is
> their differ, the file has been modified  (this is not intended to check
> 'counter_c1 < counter_c2' but just 'counter_c1 != counter_c2', that way,
> you can have circular counters.)

The MDS in 1.8 (and soon 2.0) will already keep a version counter for all
changes to the MDS inode.  The OSTs will also keep version numbers for
all of the objects there.

> - Could this flag be exposed to userspace via liblustreapi? Maybe this
> flag should be set on file creation also? Doing this, Policy Engines
> could use this flag to know easily if the file is udate to date in hsm
> or not.
>
> * Policy Engine
>
> - It needs to:
> . read changelogs (mdt)
> . df (mdt/client)
> . lfs df (per ost) (mdt/client)
> . scan namespace (client)
> . lfs getstripe by fid (client)
> . stat file by fid (client)
>
> The only thing the engine will lack on a client is the changelogs. May
> be it could be a good idea to export the changelogs on some 'trusted
> clients' ?

Yes, that is already considered.

> If not, we will be force to have MDT, client mount and policy engine on
> the same node or split the policy engine into two components (very bad
> idea to impose that on the engine). Potential overhead?

If the policy engine is running via an external database (e.g. MySQL),
it wouldn't be impossible to just have the Changelog reader do the
database insertions remotely, after looking up the pathname.

> - Could a policy test could be based on file path (not just filename and
> properties) ? This is a rule we presently used in our hsm tools. I do
> not see how have the filepath from the changelogs data (lfs fid2path?) ?

I believe the Changelog will report a full pathname (relative to the root
of the filesystem).  This will be exported via llapi to userspace.  Nathan?



Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.


From josephin at psc.edu  Fri Oct 31 04:54:06 2008
From: josephin at psc.edu (Josephine Palencia)
Date: Fri, 31 Oct 2008 00:54:06 -0400 (EDT)
Subject: [Lustre-devel] Interoperability between Lustre 1.9.50 and 1.9.80?
Message-ID: <Pine.LNX.4.64.0810310042430.4907@pscuxa.psc.edu>


Hi,

NB: Kerberos cross-realm authentication (testing) enabled

Mounting of lustre-1.9.50 lustre filesystem by a client built 
with lustre-1.9.5 produces the following error below:

> > Lustre: Added LNI 198.202.115.111 at tcp [8/256]
> > Lustre: Accept secure, port 988
> > Lustre: OBD class driver, http://www.lustre.org/
> > Lustre:         Lustre Version: 1.9.80
> > Lustre:         Build Version:
> >
> 1.9.80-19691231160000-PRISTINE-.usr.src.redhat.BUILD.kernel-lustre-2.6
> > .1 8.linux-2.6.18.x86_64-2.6.18-92.1.10.el5_lustre.1.9.80
> > Lustre: Lustre LU module (ffffffff88506740).
> > Lustre: 3586:0:(sec.c:1340:sptlrpc_import_sec_adapt()) import
> > ffff8104264a7000 (MGC128.182.112.70 at tcp->MGC128.182.112.70 at tcp_0)
> > netid
> > 20000: select initial flavor (null, null/null)
> > LustreError: 3620:0:(pack_generic.c:391:lustre_msg_buf_v2()) msg
> > ffff810425c37480 buffer[0] size 88 too small (required 152)
> > LustreError:
> > 3620:0:(pack_generic.c:577:lustre_unpack_ptlrpc_body_v2())
> > error unpacking ptlrpc body
> > LustreError: 3620:0:(pack_generic.c:1083:lustre_msg_get_status())
> > invalid msg ffff810425c37480: no ptlrpc body!
> > LustreError: 3620:0:(pack_generic.c:805:lustre_msg_get_flags())
> > invalid msg ffff810425c37480: no ptlrpc body!
> > LustreError: 3620:0:(client.c:291:unpack_reply()) @@@ unpack ptlrpc
> > body
> > failed: -14  req at ffff8104264b0800 x1284919012622337/t0(0)
> > o250->MGS at MGC128.182.112.70@tcp_0:26/25 lens 368/200 e 0 to 11 dl
> > 1225394272 ref 1 fl Rpc:R/0/0 rc 0/-22
> > LustreError: 3620:0:(layout.c:1410:__req_capsule_get()) @@@ Wrong
> > buffer for field `cdata' (1 of 2) in format `MDS_CONNECT':
> 72 vs. 200
> > (server)  req at ffff8104264b0800 x1284919012622337/t0(0)
> > o250->MGS at MGC128.182.112.70@tcp_0:26/25 lens 368/200 e 0 to 11 dl
> > 1225394272 ref 1 fl Interpret:R/0/0 rc -71/-22
> > LustreError: 3586:0:(client.c:850:ptlrpc_import_delay_req()) @@@
> > IMP_INVALID  req at ffff8104264b0400 x1284919012622339/t0(0)
> > o501->MGS at MGC128.182.112.70@tcp_0:26/25 lens 264/240 e 0 to 11 dl 0
> > o501->ref
> > 1 fl Rpc:/0/ffffffff rc 0/-1
> > LustreError: 15c-8: MGC128.182.112.70 at tcp: The
> configuration from log
> > 'testfs-client' failed (-108). This may be the result of
> communication
> > errors between this node and the MGS, a bad configuration, or other
> > errors. See the syslog for more information.
> > LustreError: 3586:0:(llite_lib.c:1079:ll_fill_super()) Unable to
> > process
> > log: -108
> > Lustre: client ffff81043be0c800 umount complete
> > LustreError: 3586:0:(obd_mount.c:1982:lustre_fill_super())
> Unable to
> > mount  (-108)


Error seems to point to the size of ptlrpc_body which is 152byes on 2.0 
and 88bytes in 1.8. However both lustre versions are minor versions of the 
major release..?

Short of me rebuilding/upgrading to lustre 1.9.80 or the client site 
downgrading to 1.9.50, any suggested work-arounds or something I am 
missing?


THank you,
Josephine


From josephin at psc.edu  Fri Oct 31 04:56:04 2008
From: josephin at psc.edu (Josephine Palencia)
Date: Fri, 31 Oct 2008 00:56:04 -0400 (EDT)
Subject: [Lustre-devel] Interoperability between Lustre 1.9.50 and
	1.9.80?
In-Reply-To: <Pine.LNX.4.64.0810310042430.4907@pscuxa.psc.edu>
References: <Pine.LNX.4.64.0810310042430.4907@pscuxa.psc.edu>
Message-ID: <Pine.LNX.4.64.0810310055380.5445@pscuxa.psc.edu>


Correction.

..client build with lustre-1.9.80..



On Fri, 31 Oct 2008, Josephine Palencia wrote:

>
> Hi,
>
> NB: Kerberos cross-realm authentication (testing) enabled
>
> Mounting of lustre-1.9.50 lustre filesystem by a client built with 
> lustre-1.9.5 produces the following error below:
>
>> > Lustre: Added LNI 198.202.115.111 at tcp [8/256]
>> > Lustre: Accept secure, port 988
>> > Lustre: OBD class driver, http://www.lustre.org/
>> > Lustre:         Lustre Version: 1.9.80
>> > Lustre:         Build Version:
>> >
>> 1.9.80-19691231160000-PRISTINE-.usr.src.redhat.BUILD.kernel-lustre-2.6
>> > .1 8.linux-2.6.18.x86_64-2.6.18-92.1.10.el5_lustre.1.9.80
>> > Lustre: Lustre LU module (ffffffff88506740).
>> > Lustre: 3586:0:(sec.c:1340:sptlrpc_import_sec_adapt()) import
>> > ffff8104264a7000 (MGC128.182.112.70 at tcp->MGC128.182.112.70 at tcp_0)
>> > netid
>> > 20000: select initial flavor (null, null/null)
>> > LustreError: 3620:0:(pack_generic.c:391:lustre_msg_buf_v2()) msg
>> > ffff810425c37480 buffer[0] size 88 too small (required 152)
>> > LustreError:
>> > 3620:0:(pack_generic.c:577:lustre_unpack_ptlrpc_body_v2())
>> > error unpacking ptlrpc body
>> > LustreError: 3620:0:(pack_generic.c:1083:lustre_msg_get_status())
>> > invalid msg ffff810425c37480: no ptlrpc body!
>> > LustreError: 3620:0:(pack_generic.c:805:lustre_msg_get_flags())
>> > invalid msg ffff810425c37480: no ptlrpc body!
>> > LustreError: 3620:0:(client.c:291:unpack_reply()) @@@ unpack ptlrpc
>> > body
>> > failed: -14  req at ffff8104264b0800 x1284919012622337/t0(0)
>> > o250->MGS at MGC128.182.112.70@tcp_0:26/25 lens 368/200 e 0 to 11 dl
>> > 1225394272 ref 1 fl Rpc:R/0/0 rc 0/-22
>> > LustreError: 3620:0:(layout.c:1410:__req_capsule_get()) @@@ Wrong
>> > buffer for field `cdata' (1 of 2) in format `MDS_CONNECT':
>> 72 vs. 200
>> > (server)  req at ffff8104264b0800 x1284919012622337/t0(0)
>> > o250->MGS at MGC128.182.112.70@tcp_0:26/25 lens 368/200 e 0 to 11 dl
>> > 1225394272 ref 1 fl Interpret:R/0/0 rc -71/-22
>> > LustreError: 3586:0:(client.c:850:ptlrpc_import_delay_req()) @@@
>> > IMP_INVALID  req at ffff8104264b0400 x1284919012622339/t0(0)
>> > o501->MGS at MGC128.182.112.70@tcp_0:26/25 lens 264/240 e 0 to 11 dl 0
>> > o501->ref
>> > 1 fl Rpc:/0/ffffffff rc 0/-1
>> > LustreError: 15c-8: MGC128.182.112.70 at tcp: The
>> configuration from log
>> > 'testfs-client' failed (-108). This may be the result of
>> communication
>> > errors between this node and the MGS, a bad configuration, or other
>> > errors. See the syslog for more information.
>> > LustreError: 3586:0:(llite_lib.c:1079:ll_fill_super()) Unable to
>> > process
>> > log: -108
>> > Lustre: client ffff81043be0c800 umount complete
>> > LustreError: 3586:0:(obd_mount.c:1982:lustre_fill_super())
>> Unable to
>> > mount  (-108)
>
>
> Error seems to point to the size of ptlrpc_body which is 152byes on 2.0 and 
> 88bytes in 1.8. However both lustre versions are minor versions of the major 
> release..?
>
> Short of me rebuilding/upgrading to lustre 1.9.80 or the client site 
> downgrading to 1.9.50, any suggested work-arounds or something I am missing?
>
>
> THank you,
> Josephine
>
>

From Nathan.Rutman at Sun.COM  Fri Oct 31 20:32:43 2008
From: Nathan.Rutman at Sun.COM (Nathaniel Rutman)
Date: Fri, 31 Oct 2008 13:32:43 -0700
Subject: [Lustre-devel] HSM comments
In-Reply-To: <490732D1.8060103@cea.fr>
References: <4905F121.6000501@cea.fr> <4906488A.8030006@sun.com>
	<490732D1.8060103@cea.fr>
Message-ID: <490B6B6B.1070401@sun.com>

Our conclusions from the meeting about HSM unlink, from 
http://arch.lustre.org/index.php?title=HSM_Migration


      5.4 unlink

   1. A client issues an unlink for a file to the MDT.
   2. The MDT includes the "hsm_exists" bit in the changelog unlink entry
   3. The policy engine determines if the file should be removed from HSM
   4. Policy engine sends HSMunlink FID to coordinator via MDT ioctl
         1. Yuck - we can't do direct ioctls on the MDT from a client
            node. We can only do ioctls on a file.

                maybe we need to implement a .lustre/device/XXX dir,
                where all MDT/OSTs are listed, and act as stub files for
                handling ioctls. 
                or maybe policy engine talks to agent / tool directly
                for unlinks? 

   5. The coordinator sends a request to one of its agent for the
      corresponding removal.
   6. The agent spawns the HSM tool to do this removal.
   7. When HSM removal is complete, policy engine cancels changelog
      unlink record
         1. How does agent/HSM tool signal to policy engine that HSM
            removal is complete?
   8. In case of agent crash, unlink record will remain uncancelled in
      the changelog; policy engine should restart processing at the
      first uncancelled record.


There's two open issues:
- How for policy engine to tell coordinator to unlink an HSM object, 
when no corresponding object exists on the MDT for us to ioctl() on
    -which coordinator to talk to for CMD?
    -since unlink isn't data movement, maybe all unlinks can be 
originated from policy engine directly?  (direct call to HSMunlinkHelper 
executable)
- How does HSMunlinkHelper return a signal to the policy engine that the 
removal is complete
    -if policy engine directly calls HSMunlinkHelper this is easy...

DEGREMONT Aurelien wrote:
>
>>> * HSM dirty bit.
>>>
>>> - should be updated with laziness.
>>> - Is it possible to implement it like the lazy file size? That means,
>>> manage the dirty bit, per OST object, and lazily update it on the MDT?
>>>       
>> Since file mtime/size is already updated this way, we can just use any 
>> attr change as the dirty indicator; we don't need an actual bit per 
>> object.  
>>     
> Dirty means data were changed, not metadata.
>   
Actually a file is dirty if either changed, depending on what you are 
storing in HSM:
filename / path?
EAs?
My point was that you can use the mtime attr change as an indicator that 
some data
possibly changed.  It is not sufficient to show that it absolutely has 
changed, so
policy engine could do something else to try to verify the change, or 
simply assume
that the file is changed, mark it dirty, and reschedule for copyout -- 
no real harm done.
Yes, I agree it would be ideal to have a true verifyable "this file has 
changed" versioning,
but since that doesn't exist yet, I don't think we need to hold up HSM 
development for it.