<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
Some background on this NUMA change:
<div class=""><br class="">
</div>
<div class="">First off, this is just a first step to a bigger set of changes which include changes to the Lustre utilities. This was done as part of the Multi-Rail feature. One of the systems that feature is meant to support is the SGI UV system (now HPE)
which has a massive number of NUMA nodes connected by a NUMA Link. There are multiple fabric cards spread throughout the system and Multi-Rail needs to know which fabric cards are nearest to the NUMA node we are running on. To do that, the “distance” between
NUMA nodes needs to be configured.</div>
<div class=""><br class="">
</div>
<div class="">This patch is preparing the infrastructure for the Multi-Rail feature to support configuring NUMA node distances. Technically, this patch should be landing with the Multi-Rail feature (still to be pushed) for it to make proper sense.</div>
<div class=""><br class="">
</div>
<div class="">Doug</div>
<div class=""> <br class="">
<div>
<blockquote type="cite" class="">
<div class="">On Jun 24, 2018, at 5:39 PM, NeilBrown <<a href="mailto:neilb@suse.com" class="">neilb@suse.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">On
Sun, Jun 24 2018, James Simmons wrote:</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">
From: Amir Shehata <<a href="mailto:amir.shehata@intel.com" class="">amir.shehata@intel.com</a>><br class="">
<br class="">
This patch adds NUMA node support. NUMA node information is stored<br class="">
in the CPT table. A NUMA node mask is maintained for the entire<br class="">
table as well as for each CPT to track the NUMA nodes related to<br class="">
each of the CPTs. Add new function cfs_cpt_of_node() which returns<br class="">
the CPT of a particular NUMA node.<br class="">
</blockquote>
<br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">I
note that you didn't respond to Greg's questions about this patch.</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">I'll
accept it anyway in the interests of moving forward, but I think</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">his
comments were probably valid, and need to be considered at some</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">stage.</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">There
is a bug though....</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">
<br class="">
Signed-off-by: Amir Shehata <<a href="mailto:amir.shehata@intel.com" class="">amir.shehata@intel.com</a>><br class="">
WC-bug-id: <a href="https://jira.whamcloud.com/browse/LU-7734" class="">https://jira.whamcloud.com/browse/LU-7734</a><br class="">
Reviewed-on: <a href="http://review.whamcloud.com/18916" class="">http://review.whamcloud.com/18916</a><br class="">
Reviewed-by: Olaf Weber <<a href="mailto:olaf@sgi.com" class="">olaf@sgi.com</a>><br class="">
Reviewed-by: Doug Oucharek <<a href="mailto:dougso@me.com" class="">dougso@me.com</a>><br class="">
Signed-off-by: James Simmons <<a href="mailto:jsimmons@infradead.org" class="">jsimmons@infradead.org</a>><br class="">
---<br class="">
.../lustre/include/linux/libcfs/libcfs_cpu.h | 11 +++++++++++<br class="">
drivers/staging/lustre/lnet/libcfs/libcfs_cpu.c | 21 +++++++++++++++++++++<br class="">
2 files changed, 32 insertions(+)<br class="">
<br class="">
diff --git a/drivers/staging/lustre/include/linux/libcfs/libcfs_cpu.h b/drivers/staging/lustre/include/linux/libcfs/libcfs_cpu.h<br class="">
index 1b4333d..ff3ecf5 100644<br class="">
--- a/drivers/staging/lustre/include/linux/libcfs/libcfs_cpu.h<br class="">
+++ b/drivers/staging/lustre/include/linux/libcfs/libcfs_cpu.h<br class="">
@@ -103,6 +103,8 @@ struct cfs_cpt_table {<br class="">
<span class="Apple-tab-span" style="white-space: pre;"></span>int<span class="Apple-tab-span" style="white-space: pre;">
</span><span class="Apple-tab-span" style="white-space: pre;"></span><span class="Apple-tab-span" style="white-space: pre;"></span><span class="Apple-tab-span" style="white-space: pre;"></span>*ctb_cpu2cpt;<br class="">
<span class="Apple-tab-span" style="white-space: pre;"></span>/* all cpus in this partition table */<br class="">
<span class="Apple-tab-span" style="white-space: pre;"></span>cpumask_var_t<span class="Apple-tab-span" style="white-space: pre;">
</span><span class="Apple-tab-span" style="white-space: pre;"></span><span class="Apple-tab-span" style="white-space: pre;"></span>ctb_cpumask;<br class="">
+<span class="Apple-tab-span" style="white-space: pre;"> </span>/* shadow HW node to CPU partition ID */<br class="">
+<span class="Apple-tab-span" style="white-space: pre;"> </span>int<span class="Apple-tab-span" style="white-space: pre;">
</span><span class="Apple-tab-span" style="white-space: pre;"></span><span class="Apple-tab-span" style="white-space: pre;"></span><span class="Apple-tab-span" style="white-space: pre;"></span>*ctb_node2cpt;<br class="">
<span class="Apple-tab-span" style="white-space: pre;"></span>/* all nodes in this partition table */<br class="">
<span class="Apple-tab-span" style="white-space: pre;"></span>nodemask_t<span class="Apple-tab-span" style="white-space: pre;">
</span><span class="Apple-tab-span" style="white-space: pre;"></span><span class="Apple-tab-span" style="white-space: pre;"></span>*ctb_nodemask;<br class="">
};<br class="">
@@ -143,6 +145,10 @@ struct cfs_cpt_table {<br class="">
*/<br class="">
int cfs_cpt_of_cpu(struct cfs_cpt_table *cptab, int cpu);<br class="">
/**<br class="">
+ * shadow HW node ID \a NODE to CPU-partition ID by \a cptab<br class="">
+ */<br class="">
+int cfs_cpt_of_node(struct cfs_cpt_table *cptab, int node);<br class="">
+/**<br class="">
* bind current thread on a CPU-partition \a cpt of \a cptab<br class="">
*/<br class="">
int cfs_cpt_bind(struct cfs_cpt_table *cptab, int cpt);<br class="">
@@ -299,6 +305,11 @@ void cfs_cpt_unset_nodemask(struct cfs_cpt_table *cptab,<br class="">
<span class="Apple-tab-span" style="white-space: pre;"></span>return 0;<br class="">
}<br class="">
<br class="">
+static inline int cfs_cpt_of_node(struct cfs_cpt_table *cptab, int node)<br class="">
+{<br class="">
+<span class="Apple-tab-span" style="white-space: pre;"> </span>return 0;<br class="">
+}<br class="">
+<br class="">
static inline int<br class="">
cfs_cpt_bind(struct cfs_cpt_table *cptab, int cpt)<br class="">
{<br class="">
diff --git a/drivers/staging/lustre/lnet/libcfs/libcfs_cpu.c b/drivers/staging/lustre/lnet/libcfs/libcfs_cpu.c<br class="">
index 33294da..8c5cf7b 100644<br class="">
--- a/drivers/staging/lustre/lnet/libcfs/libcfs_cpu.c<br class="">
+++ b/drivers/staging/lustre/lnet/libcfs/libcfs_cpu.c<br class="">
@@ -102,6 +102,15 @@ struct cfs_cpt_table *<br class="">
<span class="Apple-tab-span" style="white-space: pre;"></span>memset(cptab->ctb_cpu2cpt, -1,<br class="">
<span class="Apple-tab-span" style="white-space: pre;"></span><span class="Apple-converted-space"> </span> nr_cpu_ids * sizeof(cptab->ctb_cpu2cpt[0]));<br class="">
<br class="">
+<span class="Apple-tab-span" style="white-space: pre;"> </span>cptab->ctb_node2cpt = kvmalloc_array(nr_node_ids,<br class="">
+<span class="Apple-tab-span" style="white-space: pre;"> </span><span class="Apple-tab-span" style="white-space: pre;"></span><span class="Apple-tab-span" style="white-space: pre;"></span><span class="Apple-tab-span" style="white-space: pre;"></span><span class="Apple-tab-span" style="white-space: pre;"></span><span class="Apple-converted-space"> </span> sizeof(cptab->ctb_node2cpt[0]),<br class="">
+<span class="Apple-tab-span" style="white-space: pre;"> </span><span class="Apple-tab-span" style="white-space: pre;"></span><span class="Apple-tab-span" style="white-space: pre;"></span><span class="Apple-tab-span" style="white-space: pre;"></span><span class="Apple-tab-span" style="white-space: pre;"></span><span class="Apple-converted-space"> </span> GFP_KERNEL);<br class="">
+<span class="Apple-tab-span" style="white-space: pre;"> </span>if (!cptab->ctb_node2cpt)<br class="">
+<span class="Apple-tab-span" style="white-space: pre;"> </span><span class="Apple-tab-span" style="white-space: pre;"></span>goto failed_alloc_node2cpt;<br class="">
+<br class="">
+<span class="Apple-tab-span" style="white-space: pre;"> </span>memset(cptab->ctb_node2cpt, -1,<br class="">
+<span class="Apple-tab-span" style="white-space: pre;"> </span><span class="Apple-converted-space"> </span> nr_node_ids * sizeof(cptab->ctb_node2cpt[0]));<br class="">
+<br class="">
<span class="Apple-tab-span" style="white-space: pre;"></span>cptab->ctb_parts = kvmalloc_array(ncpt, sizeof(cptab->ctb_parts[0]),<br class="">
<span class="Apple-tab-span" style="white-space: pre;"></span><span class="Apple-tab-span" style="white-space: pre;"></span><span class="Apple-tab-span" style="white-space: pre;"></span><span class="Apple-tab-span" style="white-space: pre;"></span><span class="Apple-tab-span" style="white-space: pre;"></span><span class="Apple-converted-space"> </span> GFP_KERNEL);<br class="">
<span class="Apple-tab-span" style="white-space: pre;"></span>if (!cptab->ctb_parts)<br class="">
@@ -133,6 +142,8 @@ struct cfs_cpt_table *<br class="">
<br class="">
<span class="Apple-tab-span" style="white-space: pre;"></span>kvfree(cptab->ctb_parts);<br class="">
failed_alloc_ctb_parts:<br class="">
+<span class="Apple-tab-span" style="white-space: pre;"> </span>kvfree(cptab->ctb_node2cpt);<br class="">
+failed_alloc_node2cpt:<br class="">
<span class="Apple-tab-span" style="white-space: pre;"></span>kvfree(cptab->ctb_cpu2cpt);<br class="">
failed_alloc_cpu2cpt:<br class="">
<span class="Apple-tab-span" style="white-space: pre;"></span>kfree(cptab->ctb_nodemask);<br class="">
@@ -150,6 +161,7 @@ struct cfs_cpt_table *<br class="">
<span class="Apple-tab-span" style="white-space: pre;"></span>int i;<br class="">
<br class="">
<span class="Apple-tab-span" style="white-space: pre;"></span>kvfree(cptab->ctb_cpu2cpt);<br class="">
+<span class="Apple-tab-span" style="white-space: pre;"> </span>kvfree(cptab->ctb_node2cpt);<br class="">
<br class="">
<span class="Apple-tab-span" style="white-space: pre;"></span>for (i = 0; cptab->ctb_parts && i < cptab->ctb_nparts; i++) {<br class="">
<span class="Apple-tab-span" style="white-space: pre;"></span><span class="Apple-tab-span" style="white-space: pre;"></span>struct cfs_cpu_partition *part = &cptab->ctb_parts[i];<br class="">
@@ -515,6 +527,15 @@ struct cfs_cpt_table *<br class="">
}<br class="">
EXPORT_SYMBOL(cfs_cpt_of_cpu);<br class="">
<br class="">
+int cfs_cpt_of_node(struct cfs_cpt_table *cptab, int node)<br class="">
+{<br class="">
+<span class="Apple-tab-span" style="white-space: pre;"> </span>if (node < 0 || node > nr_node_ids)<br class="">
+<span class="Apple-tab-span" style="white-space: pre;"> </span><span class="Apple-tab-span" style="white-space: pre;"></span>return CFS_CPT_ANY;<br class="">
+<br class="">
+<span class="Apple-tab-span" style="white-space: pre;"> </span>return cptab->ctb_node2cpt[node];<br class="">
+}<br class="">
</blockquote>
<br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">So
if node == nr_node_ids, we access beyond the end of the ctb_node2cpt array.</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Oops.</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">I've
fixed this before applying.</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Thanks,</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">NeilBrown</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">
+EXPORT_SYMBOL(cfs_cpt_of_node);<br class="">
+<br class="">
int<br class="">
cfs_cpt_bind(struct cfs_cpt_table *cptab, int cpt)<br class="">
{<br class="">
--<span class="Apple-converted-space"> </span><br class="">
1.8.3.1<br class="">
</blockquote>
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">_______________________________________________</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">lustre-devel
mailing list</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<a href="mailto:lustre-devel@lists.lustre.org" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">lustre-devel@lists.lustre.org</a><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<a href="http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org</a></div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>