<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="FR" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">This is a known issue, see
<a href="https://jira.whamcloud.com/browse/LU-11840">https://jira.whamcloud.com/browse/LU-11840</a> and
<a href="https://jira.whamcloud.com/browse/LU-13548">https://jira.whamcloud.com/browse/LU-13548</a><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">Aurélien<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="margin-left:35.4pt"><b><span style="font-size:12.0pt;color:black">De :
</span></b><span style="font-size:12.0pt;color:black">lustre-discuss <lustre-discuss-bounces@lists.lustre.org> au nom de Mark Lundie <mark.lundie@manchester.ac.uk><br>
<b>Date : </b>mardi 1 décembre 2020 à 13:16<br>
<b>À : </b>fırat yılmaz <firatyilmazz@gmail.com><br>
<b>Cc : </b>"lustre-discuss@lists.lustre.org" <lustre-discuss@lists.lustre.org><br>
<b>Objet : </b>RE: [EXTERNAL] [lustre-discuss] lnet routing issue - 2.12.5 client with 2.10.3 server<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
</div>
<div>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="625" style="width:468.8pt;margin-left:35.4pt;border-collapse:collapse">
<tbody>
<tr style="height:15.25pt">
<td width="625" valign="top" style="width:468.8pt;border:solid #ED7D31 1.5pt;padding:0cm 5.4pt 0cm 5.4pt;height:15.25pt">
<p><strong><span style="font-family:"Calibri",sans-serif;color:black;background:#FFFF99">CAUTION</span></strong><span style="color:black;background:#FFFF99">: This email originated from outside of the organization. Do not click links or open attachments unless
you can confirm the sender and know the content is safe.</span><o:p></o:p></p>
</td>
</tr>
</tbody>
</table>
</div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black">Hi Firat,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black"><br>
Thanks for your reply. Apologies if I am being silly here, but there is no route configured for that network. We have the networks tcp (10.110.0.0/16) and tcp1 (10.10.0.0/16). The servers have interfaces on both, but the clients only have an interface on tcp1.
I'm not sure why the client is trying to route to 10.110.0.21@tcp:<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black">client # mount /net/lustre/
<o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black">mount.lustre: mount hmeta1@tcp1:hmeta2@tcp1:/lustre at /net/lustre failed: Input/output error<o:p></o:p></span></p>
</div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black">Is the MGS running?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black">hmeta1 resolves to 10.10.0.91, on tcp1.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black">Thanks,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black">Mark<o:p></o:p></span></p>
</div>
<div class="MsoNormal" align="center" style="margin-left:35.4pt;text-align:center">
<hr size="0" width="100%" align="center">
</div>
<div id="divRplyFwdMsg">
<p class="MsoNormal" style="margin-left:35.4pt"><b><span style="color:black">From:</span></b><span style="color:black"> fırat yılmaz <firatyilmazz@gmail.com><br>
<b>Sent:</b> 01 December 2020 11:55<br>
<b>To:</b> Mark Lundie <mark.lundie@manchester.ac.uk><br>
<b>Cc:</b> lustre-discuss@lists.lustre.org <lustre-discuss@lists.lustre.org><br>
<b>Subject:</b> Re: [lustre-discuss] lnet routing issue - 2.12.5 client with 2.10.3 server</span>
<o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Hi Mark, <o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><span style="font-size:12.0pt;color:black">[Tue Dec 1 11:07:55 2020] LNetError: 2127:0:(lib-move.c:1999:lnet_handle_find_routed_path()) no route to 10.110.0.21@tcp from <?></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">I would suggest checking lnetctl routing show and remove the route to 10.110.0.21@tcp and try to mount.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><a href="https://wiki.lustre.org/LNet_Router_Config_Guide">https://wiki.lustre.org/LNet_Router_Config_Guide</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:12.0pt;margin-left:35.4pt">
<o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
</div>
</div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">On Tue, Dec 1, 2020 at 2:41 PM Mark Lundie <<a href="mailto:mark.lundie@manchester.ac.uk">mark.lundie@manchester.ac.uk</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black">Hi all,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black">I've just run in to an issue mounting on a newly upgraded client running 2.12.5 with 2.10.3 servers. Just to give some background, we're about to replace
our existing Lustre storage, but will run it concurrently with the replacement for a couple of months. We'll be running 2.12.5 server on the new MDS and OSSs and I plan to update all clients to the same version. I would like to avoid updating the existing
servers though.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black">The problem is this. The servers have two tcp LNET networks, tcp and tcp1, on separate subnets and VLANs. The clients only see tcp1 (a small number are
also on tcp3, routed via 2 lnet routers), which has been fine until now. With the 2.12.5 client, however, it is trying to mount from tcp. 2.10.3 to 2.12.5 is obviously a bit of a jump, but does anyone have any ideas on what has changed and what I could do
here please? <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black">meta# lnetctl net show
<o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black">net:<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black"> - net type: lo<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black"> local NI(s):<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black"> - nid: 0@lo<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black"> status: up<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black"> - net type: tcp<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black"> local NI(s):<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black"> - nid: 10.110.0.21@tcp<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black"> status: up<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black"> interfaces:<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black"> 0: bond0.22<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black"> - net type: tcp1<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black"> local NI(s):<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black"> - nid: 10.10.0.91@tcp1<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black"> status: up<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black"> interfaces:<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black"> 0: bond0<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black">meta# lnetctl route show
<o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black">route:<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black"> - net: tcp2<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black"> gateway: 10.10.0.254@tcp1<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black"> - net: tcp3<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black"> gateway: 10.10.0.254@tcp1<o:p></o:p></span></p>
</div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black"><br>
client# lnetctl net show <o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black">net:<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black"> - net type: lo<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black"> local NI(s):<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black"> - nid: 0@lo<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black"> status: up<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black"> - net type: o2ib<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black"> local NI(s):<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black"> - nid: 10.12.170.47@o2ib<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black"> status: up<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black"> interfaces:<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black"> 0: ib0<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black"> - net type: tcp1<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black"> local NI(s):<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black"> - nid: 10.10.170.47@tcp1<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black"> status: up<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black"> interfaces:<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black"> 0: em1<o:p></o:p></span></p>
</div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black">[Tue Dec 1 11:07:55 2020] LNetError: 2127:0:(lib-move.c:1999:lnet_handle_find_routed_path()) no route to 10.110.0.21@tcp from <?>
<o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black">[Tue Dec 1 11:08:01 2020] LustreError: 1792:0:(mgc_request.c:249:do_config_log_add()) MGC10.10.0.91@tcp1: failed processing log, type 1: rc = -5<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black">[Tue Dec 1 11:08:08 2020] LustreError: 2169:0:(mgc_request.c:599:do_requeue()) failed processing log: -5<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black">[Tue Dec 1 11:08:19 2020] LNetError: 2127:0:(lib-move.c:1999:lnet_handle_find_routed_path()) no route to 10.110.0.22@tcp from <?><o:p></o:p></span></p>
</div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black">[Tue Dec 1 11:08:30 2020] LustreError: 15c-8: MGC10.10.0.91@tcp1: The configuration from log 'lustre-client' failed (-5). This may be the result of
communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black">client# lctl ping 10.10.0.91@tcp1
<o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black">12345-0@lo<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black">12345-10.110.0.21@tcp<o:p></o:p></span></p>
</div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black">12345-10.10.0.91@tcp1<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black">Any suggestions will be greatly appreciated!<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black">Many thanks,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt;background:white"><span style="font-size:12.0pt;color:black">Mark<o:p></o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div id="x_gmail-m_3535144636297484693Signature">
<div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><b><span style="font-family:"Courier New";color:#3D3D3D">Dr Mark Lundie</span></b><span style="font-family:"Courier New";color:#3D3D3D;background:white"> | Research IT Systems Administrator | Research IT | Directorate
of IT Services | B39, Sackville Street Building | The University of Manchester | Manchester | M1 3WE | 0161 275 8403 | </span><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif">ri.itservices.manchester.ac.uk<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><span style="font-family:"Courier New";color:#3D3D3D;background:white"><br>
<br>
</span><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><span style="font-family:"Courier New";color:#3D3D3D">Working Hours: Tues - Thurs 0730-1730; Fri 0730-1630</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal" style="margin-left:35.4pt">_______________________________________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><o:p></o:p></p>
</blockquote>
</div>
</div>
</div>
</div>
</body>
</html>