<div dir="ltr">Hello,<div style="font-size:14px"><br></div><div style="font-size:14px">><span style="font-size:13px">Can someone offer any insight on why the behavior appears to be different or provide insight on if this is incorrect behavior</span>
</div><div style="font-size:14px"><span style="font-size:13px"><br></span></div><div style="font-size:14px">"set_param -P" command syntax is differ from "conf_param" by design. It is same as "set_param" syntax. This allows use "set_param" for temporary commands and "set_param -P" for permanent. "set_param -P" allows change the same parameters that "set_param". So command set is wider, BUT existing commands should be changed to qualify "set_param" syntax. Below is the example:
</div><div style="font-size:14px"><div>[root@devvm-sl6-1 MRP-2661]# lctl get_param jobid_var
</div><div>jobid_var=disable
</div></div><div style="font-size:14px"><br></div><div style="font-size:14px">[root@devvm-sl6-1 MRP-2661]# lctl set_param -P client.jobid_var=SLURM_JOB_ID<br>
</div><div style="font-size:14px"><br></div><div style="font-size:14px"><div>[root@devvm-sl6-1 MRP-2661]# lctl get_param jobid_var
</div><div>jobid_var=SLURM_JOB_ID
</div><div><br></div><div>><span style="font-size:13px">When using conf_param, any changes to the property are written to /proc/fs/lustre/jobid_var within a short period of time. This is not the case with set_param -P.</span>
</div><span style="font-size:13px"> </span></div><div style="font-size:14px">After "set_param -P" applied, lock for "params" file is canceled that initiate this log is acquired and "executed" by targets and clients. This requires some time.
</div><div style="font-size:14px"><br></div><div style="font-size:14px">><span style="font-size:13px">The change is never reflected nor is it stored in some other file within /proc.</span>
</div><div style="font-size:14px"><br></div><div style="font-size:14px">Changes stored in "params" file in CONFIG directory. Can be viewed using log_reader.
</div><div style="font-size:14px"><br></div><div style="font-size:14px">><span style="font-size:13px">In short, it appears that conf_param attempts to apply the changes to all the targets while set_param does not (it does not recognize it as a global property).</span>
</div><div style="font-size:14px"><br></div><div style="font-size:14px">Really "set_param -P" attempts to apply the new changes to all the targets too. Command is executed on all targets, and all commands have "general" as 0th parameter of command in llog file params.
</div><div style="font-size:14px"><br></div><div style="font-size:14px">>Is it the intent of "set_param -P" that the specified changes only take
</div>effect after components are restarted? And if so, why?<div style="font-size:14px"><span style="font-size:13px"><br></span></div><div style="font-size:14px">No, changes applied just after command executed, but some time is required to cancel lock for params file.
</div><div style="font-size:14px"><br></div><div style="font-size:14px">><span style="font-size:13px">How would a normal system administrator go about finding out what</span>
</div><div style="font-family:'Helvetica Neue';font-size:14px"><span style="font-family:arial,sans-serif;font-size:13px">settings are currently set permanently?</span></div><div style="font-family:'Helvetica Neue';font-size:14px"><span style="font-family:arial,sans-serif;font-size:13px"><br></span></div><div style="font-size:14px">Seme settings as "set_param" sets can be set using "set_param -P". Finally all command that stored using "set_param" executed using "lctl set_param" upcall on targets.
</div><div style="font-size:14px"><br></div><div style="font-size:14px">><span style="font-size:13px">From a user-interface standpoint though, presenting a single namespace</span>
</div><div style="font-family:'Helvetica Neue';font-size:14px"><span style="font-family:arial,sans-serif;font-size:13px">for all nodes in the entire center seems like less than desirable</span></div><div style="font-family:'Helvetica Neue';font-size:14px"><span style="font-family:arial,sans-serif;font-size:13px">choice. </span></div><div style="font-family:'Helvetica Neue';font-size:14px"><span style="font-family:arial,sans-serif;font-size:13px"><br></span></div><div style="font-size:14px">Yes, probably. All commands are saved in one namespace. It called "general". But I believe we could fix this issue easy, changing "general" to filesystem name, so targets checks this namespace before executing upcall.
</div><div style="font-size:14px"><br></div><div style="font-size:14px">Actually, command that have filesystem part (for example started as "testfs.") faintly failed on wrong nodes and succeed only on addressed nodes. So probably, we do not need change this, but we can use reserved field if we would.
</div><div style="font-size:14px"><br></div><div style="font-size:14px">><span style="font-size:13px">Given that not all paths</span>
</div><div style="font-family:'Helvetica Neue';font-size:14px"><span style="font-family:arial,sans-serif;font-size:13px">under /proc have differentiating strings in their path, there are some</span></div><div style="font-family:'Helvetica Neue';font-size:14px"><span style="font-family:arial,sans-serif;font-size:13px">things that can only be set completely globally in this design.</span></div><div style="font-family:'Helvetica Neue';font-size:14px"><span style="font-family:arial,sans-serif;font-size:13px"><br></span></div><div style="font-size:14px">Yes, this is case there we could change "general" to filename. Or fix this issue changing procfs paths.
</div><div style="font-size:14px"><br></div><div style="font-size:14px">><span style="font-size:13px">And what about sites that use an MGS per filesystem, rather than a</span>
</div><div style="font-family:'Helvetica Neue';font-size:14px"><span style="font-family:arial,sans-serif;font-size:13px">single MGS for the entire site? If one MGS says that the debug level</span></div><div style="font-family:'Helvetica Neue';font-size:14px"><span style="font-family:arial,sans-serif;font-size:13px">should be one value, and another says that the debug value should be</span></div><div style="font-family:'Helvetica Neue';font-size:14px"><span style="font-family:arial,sans-serif;font-size:13px">another value, is it entirely random which debug setting will appear on</span></div><div style="font-family:'Helvetica Neue';font-size:14px"><span style="font-family:arial,sans-serif;font-size:13px">any given node?</span></div><div style="font-family:'Helvetica Neue';font-size:14px"><span style="font-family:arial,sans-serif;font-size:13px"><br></span></div><div style="font-size:14px"><span style="font-size:13px">Good </span>example there we could use reserved field. Do you think this can solve the issue? If so, we could extend "set_param -P" now.
</div><div style="font-size:14px"><br></div><div style="font-size:14px">><span style="font-size:13px">We might want to just remove the conf_param deprecation message</span>
</div><div style="font-family:'Helvetica Neue';font-size:14px"><span style="font-family:arial,sans-serif;font-size:13px">altogether. I think there is more than just a simple bug in the</span></div><div style="font-size:14px"><span style="font-size:13px">"set_param -P" implementation. </span></div><div style="font-size:14px"><br></div><div style="font-size:14px">I believe we need move to "set_param -P". Yes, this require some scripts to be changed, but fixing this now we get method with this benefits instead of conf_param:
</div><div style="font-size:14px">1. Identical format for permanent and temporary commands
</div><div style="font-size:14px">2. wildcarding in strings
</div><div style="font-size:14px">3. no unimplemented paths (e.g. ptlrpc services)
</div><div style="font-size:14px">4. simpler implementation
</div><div style="font-size:14px"><br></div><div style="font-size:14px">><span style="font-size:13px">From what I am seeing, it looks to me </span><span style="font-size:13px">like we have some design issues. </span>
</div><div style="font-size:14px"><span style="font-size:13px"><br></span></div><div style="font-size:14px">All mentioned issues can be fixed easily without changing design. The problem of namespace distinguish already is part of design. Special field is reserved.
</div><div style="font-size:14px"><br></div><div style="font-size:14px">><span style="font-size:13px">Without anyone committed to working</span>
</div><div style="font-family:'Helvetica Neue';font-size:14px"><span style="font-family:arial,sans-serif;font-size:13px">out a better design and implementing it, set_param -P looks like</span></div><div style="font-family:'Helvetica Neue';font-size:14px"><span style="font-family:arial,sans-serif;font-size:13px">technical debt with no clear resolution in sight. Maybe we should even</span></div><div style="font-family:'Helvetica Neue';font-size:14px"><span style="font-family:arial,sans-serif;font-size:13px">consider stripping it back back out of the tree.</span></div><div style="font-family:'Helvetica Neue';font-size:14px"><br></div><div style="font-size:14px">I belive we could solve found issues and move forward using "set_param -P"
</div><div style="font-size:14px">I offered "set_param -P topic to last LAD, but unfortunately it was not accepted. Lets' discuss this together. Moving conf_param discarding date forward gives us some time for this.
</div><div style="font-size:14px"><br></div><div style="font-size:14px">Thanks. </div><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Aug 15, 2015 at 11:04 PM, <span dir="ltr"><<a href="mailto:lustre-devel-request@lists.lustre.org" target="_blank">lustre-devel-request@lists.lustre.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Send lustre-devel mailing list submissions to<br>
<a href="mailto:lustre-devel@lists.lustre.org">lustre-devel@lists.lustre.org</a><br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.lustre.org_listinfo.cgi_lustre-2Ddevel-2Dlustre.org&d=AwICAg&c=IGDlg0lD0b-nebmJJ0Kp8A&r=f-9bup-jYTTZb-zqZbdOzQTObV_VGrTP7-8xROGj35I&m=zc0hAFYXhRlbqxXHoLOSK0PoVteDFbf33MzwVDBMADY&s=NedsjGFvLB9S2GAZw1bC2c9WNT-N8YBC0YHvMimBTvw&e=" rel="noreferrer" target="_blank">https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.lustre.org_listinfo.cgi_lustre-2Ddevel-2Dlustre.org&d=AwICAg&c=IGDlg0lD0b-nebmJJ0Kp8A&r=f-9bup-jYTTZb-zqZbdOzQTObV_VGrTP7-8xROGj35I&m=zc0hAFYXhRlbqxXHoLOSK0PoVteDFbf33MzwVDBMADY&s=NedsjGFvLB9S2GAZw1bC2c9WNT-N8YBC0YHvMimBTvw&e=</a><br>
or, via email, send a message with subject or body 'help' to<br>
<a href="mailto:lustre-devel-request@lists.lustre.org">lustre-devel-request@lists.lustre.org</a><br>
<br>
You can reach the person managing the list at<br>
<a href="mailto:lustre-devel-owner@lists.lustre.org">lustre-devel-owner@lists.lustre.org</a><br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of lustre-devel digest..."<br>
<br>
<br>
Today's Topics:<br>
<br>
1. Re: lctl conf_param vs set_param -P (Christopher J. Morrone)<br>
2. Re: lctl conf_param vs set_param -P (Dilger, Andreas)<br>
3. Re: lctl conf_param vs set_param -P (Christopher J. Morrone)<br>
<br>
<br>
----------------------------------------------------------------------<br>
<br>
Message: 1<br>
Date: Fri, 14 Aug 2015 13:41:03 -0700<br>
From: "Christopher J. Morrone" <<a href="mailto:morrone2@llnl.gov">morrone2@llnl.gov</a>><br>
To: <a href="mailto:lustre-devel@lists.lustre.org">lustre-devel@lists.lustre.org</a><br>
Subject: Re: [lustre-devel] lctl conf_param vs set_param -P<br>
Message-ID: <<a href="mailto:55CE525F.3020505@llnl.gov">55CE525F.3020505@llnl.gov</a>><br>
Content-Type: text/plain; charset=windows-1252; format=flowed<br>
<br>
I too am confused. And a bit dismayed that there is so little in the<br>
way of code comments to explain the intent.<br>
<br>
Is it the intent of "set_param -P" that the specified changes only take<br>
effect after components are restarted? And if so, why?<br>
<br>
How would a normal system administrator go about finding out what<br>
settings are currently set permanently?<br>
<br>
I read through LU-3155 and see the discussion about using a single llog<br>
file for all nodes, so I will withhold comment about that for now.<br>
<br>
From a user-interface standpoint though, presenting a single namespace<br>
for all nodes in the entire center seems like less than desirable<br>
choice. Might we not want to set settings differently on different<br>
clusters (be they client or server clusters)? Given that not all paths<br>
under /proc have differentiating strings in their path, there are some<br>
things that can only be set completely globally in this design.<br>
<br>
And what about sites that use an MGS per filesystem, rather than a<br>
single MGS for the entire site? If one MGS says that the debug level<br>
should be one value, and another says that the debug value should be<br>
another value, is it entirely random which debug setting will appear on<br>
any given node?<br>
<br>
Chris<br>
<br>
On 08/13/2015 03:43 PM, Di Natale, Giuseppe wrote:<br>
> Greetings,<br>
><br>
> In an effort to change test-framework.sh to not utilize the deprecated<br>
> conf_param option in lctl, I stumbled upon what appears to be<br>
> inconsistent behavior between lctl's conf_param and set_param -P<br>
> options. The permanent option test-framework.sh is attempting to change<br>
> is jobid_var. When using conf_param, any changes to the property are<br>
> written to /proc/fs/lustre/jobid_var within a short period of time. This<br>
> is not the case with set_param -P. The change is never reflected nor is<br>
> it stored in some other file within /proc. I started digging into the<br>
> MGS logs and found that the behavior for both are different (the<br>
> relevant segments of the logs are attached to this email and are named<br>
> accordingly). In short, it appears that conf_param attempts to apply the<br>
> changes to all the targets while set_param does not (it does not<br>
> recognize it as a global property). Can someone offer any insight on why<br>
> the behavior appears to be different or provide insight on if this is<br>
> incorrect behavior?<br>
><br>
> Thanks,<br>
> Giuseppe Di Natale<br>
><br>
><br>
> _______________________________________________<br>
> lustre-devel mailing list<br>
> <a href="mailto:lustre-devel@lists.lustre.org">lustre-devel@lists.lustre.org</a><br>
> <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.lustre.org_listinfo.cgi_lustre-2Ddevel-2Dlustre.org&d=AwICAg&c=IGDlg0lD0b-nebmJJ0Kp8A&r=f-9bup-jYTTZb-zqZbdOzQTObV_VGrTP7-8xROGj35I&m=zc0hAFYXhRlbqxXHoLOSK0PoVteDFbf33MzwVDBMADY&s=NedsjGFvLB9S2GAZw1bC2c9WNT-N8YBC0YHvMimBTvw&e=" rel="noreferrer" target="_blank">https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.lustre.org_listinfo.cgi_lustre-2Ddevel-2Dlustre.org&d=AwICAg&c=IGDlg0lD0b-nebmJJ0Kp8A&r=f-9bup-jYTTZb-zqZbdOzQTObV_VGrTP7-8xROGj35I&m=zc0hAFYXhRlbqxXHoLOSK0PoVteDFbf33MzwVDBMADY&s=NedsjGFvLB9S2GAZw1bC2c9WNT-N8YBC0YHvMimBTvw&e=</a><br>
><br>
<br>
<br>
<br>
------------------------------<br>
<br>
Message: 2<br>
Date: Sat, 15 Aug 2015 00:07:22 +0000<br>
From: "Dilger, Andreas" <<a href="mailto:andreas.dilger@intel.com">andreas.dilger@intel.com</a>><br>
To: "Di Natale, Giuseppe" <<a href="mailto:dinatale2@llnl.gov">dinatale2@llnl.gov</a>><br>
Cc: "<a href="mailto:lustre-devel@lists.lustre.org">lustre-devel@lists.lustre.org</a>" <<a href="mailto:lustre-devel@lists.lustre.org">lustre-devel@lists.lustre.org</a>><br>
Subject: Re: [lustre-devel] lctl conf_param vs set_param -P<br>
Message-ID: <<a href="mailto:D1F3DE3E.101528%25andreas.dilger@intel.com">D1F3DE3E.101528%andreas.dilger@intel.com</a>><br>
Content-Type: text/plain; charset="us-ascii"<br>
<br>
Coincidentally (or maybe that was what drove your investigations?), I'd just filed <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__jira.hpdd.intel.com_browse_LU-2D7004&d=AwICAg&c=IGDlg0lD0b-nebmJJ0Kp8A&r=f-9bup-jYTTZb-zqZbdOzQTObV_VGrTP7-8xROGj35I&m=zc0hAFYXhRlbqxXHoLOSK0PoVteDFbf33MzwVDBMADY&s=h6zLIi4EW2KHxq37PO71vZIbOskaw7V_qJJGV9mplo4&e=" rel="noreferrer" target="_blank">https://urldefense.proofpoint.com/v2/url?u=https-3A__jira.hpdd.intel.com_browse_LU-2D7004&d=AwICAg&c=IGDlg0lD0b-nebmJJ0Kp8A&r=f-9bup-jYTTZb-zqZbdOzQTObV_VGrTP7-8xROGj35I&m=zc0hAFYXhRlbqxXHoLOSK0PoVteDFbf33MzwVDBMADY&s=h6zLIi4EW2KHxq37PO71vZIbOskaw7V_qJJGV9mplo4&e=</a> about this issue. It looks like the "lctl set_param -P" feature needs more testing, and I've removed the deprecation warning for "lctl conf_param" for 2.8.<br>
<br>
Cheers, Andreas<br>
--<br>
Andreas Dilger<br>
Lustre Software Architect<br>
Intel High Performance Data Division<br>
<br>
On 2015/08/13, 4:43 PM, "lustre-devel on behalf of Di Natale, Giuseppe" <<a href="mailto:lustre-devel-bounces@lists.lustre.org">lustre-devel-bounces@lists.lustre.org</a><mailto:<a href="mailto:lustre-devel-bounces@lists.lustre.org">lustre-devel-bounces@lists.lustre.org</a>> on behalf of <a href="mailto:dinatale2@llnl.gov">dinatale2@llnl.gov</a><mailto:<a href="mailto:dinatale2@llnl.gov">dinatale2@llnl.gov</a>>> wrote:<br>
<br>
Greetings,<br>
<br>
In an effort to change test-framework.sh to not utilize the deprecated conf_param option in lctl, I stumbled upon what appears to be inconsistent behavior between lctl's conf_param and set_param -P options. The permanent option test-framework.sh is attempting to change is jobid_var. When using conf_param, any changes to the property are written to /proc/fs/lustre/jobid_var within a short period of time. This is not the case with set_param -P. The change is never reflected nor is it stored in some other file within /proc. I started digging into the MGS logs and found that the behavior for both are different (the relevant segments of the logs are attached to this email and are named accordingly). In short, it appears that conf_param attempts to apply the changes to all the targets while set_param does not (it does not recognize it as a global property). Can someone offer any insight on why the behavior appears to be different or provide insight on if this is incorrect behavior?<br>
<br>
Thanks,<br>
Giuseppe Di Natale<br>
<br>
<br>
------------------------------<br>
<br>
Message: 3<br>
Date: Fri, 14 Aug 2015 17:36:47 -0700<br>
From: "Christopher J. Morrone" <<a href="mailto:morrone2@llnl.gov">morrone2@llnl.gov</a>><br>
To: <a href="mailto:lustre-devel@lists.lustre.org">lustre-devel@lists.lustre.org</a><br>
Subject: Re: [lustre-devel] lctl conf_param vs set_param -P<br>
Message-ID: <<a href="mailto:55CE899F.5060403@llnl.gov">55CE899F.5060403@llnl.gov</a>><br>
Content-Type: text/plain; charset=windows-1252; format=flowed<br>
<br>
It was coincidental this time. We saw the deprecation message during<br>
llmount.sh and figured that might be an easy first task for Giuseppe to<br>
get familiarity with the patch submission process. But it turned out to<br>
be a little more difficult than I thought. :)<br>
<br>
We might want to just remove the conf_param deprecation message<br>
altogether. I think there is more than just a simple bug in the<br>
"set_param -P" implementation. From what I am seeing, it looks to me<br>
like we have some design issues. Without anyone committed to working<br>
out a better design and implementing it, set_param -P looks like<br>
technical debt with no clear resolution in sight. Maybe we should even<br>
consider stripping it back back out of the tree.<br>
<br>
Chris<br>
<br>
On 08/14/2015 05:07 PM, Dilger, Andreas wrote:<br>
> Coincidentally (or maybe that was what drove your investigations?), I'd just filed <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__jira.hpdd.intel.com_browse_LU-2D7004&d=AwICAg&c=IGDlg0lD0b-nebmJJ0Kp8A&r=f-9bup-jYTTZb-zqZbdOzQTObV_VGrTP7-8xROGj35I&m=zc0hAFYXhRlbqxXHoLOSK0PoVteDFbf33MzwVDBMADY&s=h6zLIi4EW2KHxq37PO71vZIbOskaw7V_qJJGV9mplo4&e=" rel="noreferrer" target="_blank">https://urldefense.proofpoint.com/v2/url?u=https-3A__jira.hpdd.intel.com_browse_LU-2D7004&d=AwICAg&c=IGDlg0lD0b-nebmJJ0Kp8A&r=f-9bup-jYTTZb-zqZbdOzQTObV_VGrTP7-8xROGj35I&m=zc0hAFYXhRlbqxXHoLOSK0PoVteDFbf33MzwVDBMADY&s=h6zLIi4EW2KHxq37PO71vZIbOskaw7V_qJJGV9mplo4&e=</a> about this issue. It looks like the "lctl set_param -P" feature needs more testing, and I've removed the deprecation warning for "lctl conf_param" for 2.8.<br>
><br>
> Cheers, Andreas<br>
> --<br>
> Andreas Dilger<br>
> Lustre Software Architect<br>
> Intel High Performance Data Division<br>
><br>
> On 2015/08/13, 4:43 PM, "lustre-devel on behalf of Di Natale, Giuseppe" <<a href="mailto:lustre-devel-bounces@lists.lustre.org">lustre-devel-bounces@lists.lustre.org</a><mailto:<a href="mailto:lustre-devel-bounces@lists.lustre.org">lustre-devel-bounces@lists.lustre.org</a>> on behalf of <a href="mailto:dinatale2@llnl.gov">dinatale2@llnl.gov</a><mailto:<a href="mailto:dinatale2@llnl.gov">dinatale2@llnl.gov</a>>> wrote:<br>
><br>
> Greetings,<br>
><br>
> In an effort to change test-framework.sh to not utilize the deprecated conf_param option in lctl, I stumbled upon what appears to be inconsistent behavior between lctl's conf_param and set_param -P options. The permanent option test-framework.sh is attempting to change is jobid_var. When using conf_param, any changes to the property are written to /proc/fs/lustre/jobid_var within a short period of time. This is not the case with set_param -P. The change is never reflected nor is it stored in some other file within /proc. I started digging into the MGS logs and found that the behavior for both are different (the relevant segments of the logs are attached to this email and are named accordingly). In short, it appears that conf_param attempts to apply the changes to all the targets while set_param does not (it does not recognize it as a global property). Can someone offer any insight on why the behavior appears to be different or provide insight on if this is incorrect behavio<br>
r?<br>
><br>
><br>
> Thanks,<br>
> Giuseppe Di Natale<br>
> _______________________________________________<br>
> lustre-devel mailing list<br>
> <a href="mailto:lustre-devel@lists.lustre.org">lustre-devel@lists.lustre.org</a><br>
> <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.lustre.org_listinfo.cgi_lustre-2Ddevel-2Dlustre.org&d=AwICAg&c=IGDlg0lD0b-nebmJJ0Kp8A&r=f-9bup-jYTTZb-zqZbdOzQTObV_VGrTP7-8xROGj35I&m=zc0hAFYXhRlbqxXHoLOSK0PoVteDFbf33MzwVDBMADY&s=NedsjGFvLB9S2GAZw1bC2c9WNT-N8YBC0YHvMimBTvw&e=" rel="noreferrer" target="_blank">https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.lustre.org_listinfo.cgi_lustre-2Ddevel-2Dlustre.org&d=AwICAg&c=IGDlg0lD0b-nebmJJ0Kp8A&r=f-9bup-jYTTZb-zqZbdOzQTObV_VGrTP7-8xROGj35I&m=zc0hAFYXhRlbqxXHoLOSK0PoVteDFbf33MzwVDBMADY&s=NedsjGFvLB9S2GAZw1bC2c9WNT-N8YBC0YHvMimBTvw&e=</a><br>
><br>
<br>
<br>
<br>
------------------------------<br>
<br>
Subject: Digest Footer<br>
<br>
_______________________________________________<br>
lustre-devel mailing list<br>
<a href="mailto:lustre-devel@lists.lustre.org">lustre-devel@lists.lustre.org</a><br>
<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.lustre.org_listinfo.cgi_lustre-2Ddevel-2Dlustre.org&d=AwICAg&c=IGDlg0lD0b-nebmJJ0Kp8A&r=f-9bup-jYTTZb-zqZbdOzQTObV_VGrTP7-8xROGj35I&m=zc0hAFYXhRlbqxXHoLOSK0PoVteDFbf33MzwVDBMADY&s=NedsjGFvLB9S2GAZw1bC2c9WNT-N8YBC0YHvMimBTvw&e=" rel="noreferrer" target="_blank">https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.lustre.org_listinfo.cgi_lustre-2Ddevel-2Dlustre.org&d=AwICAg&c=IGDlg0lD0b-nebmJJ0Kp8A&r=f-9bup-jYTTZb-zqZbdOzQTObV_VGrTP7-8xROGj35I&m=zc0hAFYXhRlbqxXHoLOSK0PoVteDFbf33MzwVDBMADY&s=NedsjGFvLB9S2GAZw1bC2c9WNT-N8YBC0YHvMimBTvw&e=</a><br>
<br>
<br>
------------------------------<br>
<br>
End of lustre-devel Digest, Vol 103, Issue 2<br>
********************************************<br>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr">Artem Blagodarenko Ph.D.<strong>·</strong> SW Developer on <a href="http://my.seagate.com" target="_blank">my.seagate.com</a><br>
Seagate Technology, LLC<br>
<a href="http://www.seagate.com" target="_blank">www.seagate.com</a><br></div></div>
</div></div>