<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML xmlns="http://www.w3.org/TR/REC-html40" xmlns:v =
"urn:schemas-microsoft-com:vml" xmlns:o =
"urn:schemas-microsoft-com:office:office" xmlns:w =
"urn:schemas-microsoft-com:office:word" xmlns:m =
"http://schemas.microsoft.com/office/2004/12/omml"><HEAD>
<META http-equiv=Content-Type content="text/html; charset=iso-2022-jp">
<META content="MSHTML 6.00.2900.6182" name=GENERATOR><!--[if !mso]>
<STYLE>
v\:* {
BEHAVIOR: url(#default#VML)
}
o\:* {
BEHAVIOR: url(#default#VML)
}
w\:* {
BEHAVIOR: url(#default#VML)
}
.shape {
BEHAVIOR: url(#default#VML)
}
DIV.FoxDiv20120315205304086892 {
MARGIN: 7.5pt; COLOR: #000000
}
BODY {
FONT-SIZE: 10.5pt; COLOR: #000080; LINE-HEIGHT: 1.5; FONT-FAMILY: 宋体
}
</STYLE>
<![endif]-->
<STYLE>@font-face {
font-family: SimSun;
}
@font-face {
font-family: SimSun;
}
@font-face {
font-family: Calibri;
}
@font-face {
font-family: Tahoma;
}
@font-face {
font-family: @SimSun;
}
@page WordSection1 {size: 8.5in 11.0in; margin: 1.0in 1.0in 1.0in 1.0in; }
P.MsoNormal {
FONT-SIZE: 12pt; MARGIN-LEFT: 0in; MARGIN-RIGHT: 0in; FONT-FAMILY: "SimSun","serif"; mso-margin-top-alt: auto; mso-margin-bottom-alt: auto; mso-believe-normal-left: yes
}
LI.MsoNormal {
FONT-SIZE: 12pt; MARGIN-LEFT: 0in; MARGIN-RIGHT: 0in; FONT-FAMILY: "SimSun","serif"; mso-margin-top-alt: auto; mso-margin-bottom-alt: auto; mso-believe-normal-left: yes
}
DIV.MsoNormal {
FONT-SIZE: 12pt; MARGIN-LEFT: 0in; MARGIN-RIGHT: 0in; FONT-FAMILY: "SimSun","serif"; mso-margin-top-alt: auto; mso-margin-bottom-alt: auto; mso-believe-normal-left: yes
}
A:link {
COLOR: blue; TEXT-DECORATION: underline; mso-style-priority: 99
}
SPAN.MsoHyperlink {
COLOR: blue; TEXT-DECORATION: underline; mso-style-priority: 99
}
A:visited {
COLOR: purple; TEXT-DECORATION: underline; mso-style-priority: 99
}
SPAN.MsoHyperlinkFollowed {
COLOR: purple; TEXT-DECORATION: underline; mso-style-priority: 99
}
P {
FONT-SIZE: 12pt; MARGIN: 0px 0in; FONT-FAMILY: "SimSun","serif"; mso-style-priority: 99
}
SPAN.EmailStyle18 {
COLOR: #1f497d; FONT-FAMILY: "Calibri","sans-serif"; mso-style-type: personal-reply
}
.MsoChpDefault {
FONT-SIZE: 10pt; mso-style-type: export-only
}
DIV.WordSection1 {
page: WordSection1
}
</STYLE>
<![if mso 9]><style>p.MsoNormal
{margin-left:7.5pt;}
</style><![endif]><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<STYLE>BLOCKQUOTE {
MARGIN-TOP: 0px; MARGIN-BOTTOM: 0px; MARGIN-LEFT: 2em
}
OL {
MARGIN-TOP: 0px; MARGIN-BOTTOM: 0px
}
UL {
MARGIN-TOP: 0px; MARGIN-BOTTOM: 0px
}
</STYLE>
</HEAD>
<BODY lang=EN-US style="MARGIN: 10px" vLink=purple link=blue>
<DIV>Thanks a lot , Charles.</DIV>
<DIV> </DIV>
<DIV>I agree with you about this problem.</DIV>
<DIV> </DIV>
<DIV>And I did more tests with the following steps:</DIV>
<DIV style="TEXT-INDENT: 2em">0) Use 3 subnets to assign the 3
nodes.</DIV>
<DIV style="TEXT-INDENT: 2em">1)Run "netperf" in the two OSS
separately, run "netserver" in "client";this step could simulate
the </DIV>
<DIV>networking scenario: "client" reads data from two OSS, but here
is no disk i/o or other r/w;</DIV>
<DIV style="TEXT-INDENT: 2em">2) two OSS netperf's results are about
200 M/s, totally are 400M/s. so low - -!</DIV>
<DIV style="TEXT-INDENT: 2em">3) run only netperf at one OSS, the test
result is 950M/s.. this res is ok. </DIV>
<DIV style="TEXT-INDENT: 2em">4) All the upper steps prove that, the
networking is the bottleneck of the read performance.</DIV>
<DIV> </DIV>
<DIV style="TEXT-INDENT: 2em">When 2 NODEs send TCP stream at the same time, and
only 1 NODE recv TCP stream. The total throughput is half of normal value.
so oddball..</DIV>
<DIV style="TEXT-INDENT: 2em">What induced that? Thanks a lot</DIV>
<DIV style="TEXT-INDENT: 2em"> </DIV>
<DIV>
<HR style="WIDTH: 210px; HEIGHT: 1px" align=left color=#b5c4df SIZE=1>
</DIV>
<DIV>Best regards</DIV>
<DIV><SPAN>feng</SPAN></DIV>
<DIV> </DIV>
<DIV
style="BORDER-RIGHT: medium none; PADDING-RIGHT: 0cm; BORDER-TOP: #b5c4df 1pt solid; PADDING-LEFT: 0cm; PADDING-BOTTOM: 0cm; BORDER-LEFT: medium none; PADDING-TOP: 3pt; BORDER-BOTTOM: medium none">
<DIV
style="PADDING-RIGHT: 8px; PADDING-LEFT: 8px; FONT-SIZE: 12px; BACKGROUND: #efefef; PADDING-BOTTOM: 8px; COLOR: #000000; PADDING-TOP: 8px">
<DIV><B>From:</B> <A href="mailto:chammitt@email.unc.edu">Hammitt, Charles
Allen</A></DIV>
<DIV><B>Date:</B> 2012-03-15 20:50</DIV>
<DIV><B>To:</B> <A href="mailto:zf5984599@gmail.com">zf5984599</A>; <A
href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</A></DIV>
<DIV><B>Subject:</B> RE: [Lustre-discuss] Lustre read performance decay
when OSSes are assigned in two different subnet</DIV></DIV></DIV>
<DIV>
<DIV class=FoxDiv20120315205304086892>
<META content="Microsoft Word 12 (filtered medium)" name=Generator><!--[if !mso]>
<STYLE>v\:* {
BEHAVIOR: url(#default#VML)
}
o\:* {
BEHAVIOR: url(#default#VML)
}
w\:* {
BEHAVIOR: url(#default#VML)
}
.shape {
BEHAVIOR: url(#default#VML)
}
</STYLE>
<![endif]-->
<STYLE>@font-face {
font-family: SimSun;
}
@font-face {
font-family: SimSun;
}
@font-face {
font-family: Calibri;
}
@font-face {
font-family: Tahoma;
}
@font-face {
font-family: @SimSun;
}
@page WordSection1 {size: 8.5in 11.0in; margin: 1.0in 1.0in 1.0in 1.0in; }
P.MsoNormal {
FONT-SIZE: 12pt; MARGIN-LEFT: 0in; MARGIN-RIGHT: 0in; FONT-FAMILY: "SimSun","serif"; mso-margin-top-alt: auto; mso-margin-bottom-alt: auto; mso-believe-normal-left: yes
}
LI.MsoNormal {
FONT-SIZE: 12pt; MARGIN-LEFT: 0in; MARGIN-RIGHT: 0in; FONT-FAMILY: "SimSun","serif"; mso-margin-top-alt: auto; mso-margin-bottom-alt: auto; mso-believe-normal-left: yes
}
DIV.MsoNormal {
FONT-SIZE: 12pt; MARGIN-LEFT: 0in; MARGIN-RIGHT: 0in; FONT-FAMILY: "SimSun","serif"; mso-margin-top-alt: auto; mso-margin-bottom-alt: auto; mso-believe-normal-left: yes
}
A:link {
COLOR: blue; TEXT-DECORATION: underline; mso-style-priority: 99
}
SPAN.MsoHyperlink {
COLOR: blue; TEXT-DECORATION: underline; mso-style-priority: 99
}
A:visited {
COLOR: purple; TEXT-DECORATION: underline; mso-style-priority: 99
}
SPAN.MsoHyperlinkFollowed {
COLOR: purple; TEXT-DECORATION: underline; mso-style-priority: 99
}
P {
FONT-SIZE: 12pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "SimSun","serif"; mso-style-priority: 99
}
SPAN.EmailStyle18 {
COLOR: #1f497d; FONT-FAMILY: "Calibri","sans-serif"; mso-style-type: personal-reply
}
.MsoChpDefault {
FONT-SIZE: 10pt; mso-style-type: export-only
}
DIV.WordSection1 {
page: WordSection1
}
</STYLE>
<![if mso 9]><style>p.MsoNormal
{margin-left:7.5pt;}
</style><![endif]><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<DIV class=WordSection1>
<P class=MsoNormal><SPAN
style="FONT-SIZE: 11pt; COLOR: #1f497d; FONT-FAMILY: 'Calibri','sans-serif'"><o:p> </o:p></SPAN></P>
<P class=MsoNormal><SPAN
style="FONT-SIZE: 11pt; COLOR: #1f497d; FONT-FAMILY: 'Calibri','sans-serif'">Networking
overhead… vlan routing perhaps; 1) with either adding an extra network
device hop and latency from a network device/router or 2) overburdened switch
handling the routing itself still introducing network latency.
Latency is the storage and network i/o bandwidth
killer.<o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN
style="FONT-SIZE: 11pt; COLOR: #1f497d; FONT-FAMILY: 'Calibri','sans-serif'"><o:p> </o:p></SPAN></P>
<P class=MsoNormal><SPAN
style="FONT-SIZE: 11pt; COLOR: #1f497d; FONT-FAMILY: 'Calibri','sans-serif'">I’m
willing to bet two things: <o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN
style="FONT-SIZE: 11pt; COLOR: #1f497d; FONT-FAMILY: 'Calibri','sans-serif'">1)
changing your stripe size from 2 to 1 will make similar bandwidth results to the
diagram 2 [54.3MB/s], even if the layout is as diagram 1 [separate
nets].<o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN
style="FONT-SIZE: 11pt; COLOR: #1f497d; FONT-FAMILY: 'Calibri','sans-serif'">2)
If all your OSS/MDS and Clients nodes were in the same single vlan network…you’d
see better performance than diagram’s 2 54.3MB/sec bandwidth
throughput.<o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN
style="FONT-SIZE: 11pt; COLOR: #1f497d; FONT-FAMILY: 'Calibri','sans-serif'">So,
drop classful subnets…go with cidr / supernetting networks to get the ip spaces
you need and drop the extra routing latency.<o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN
style="FONT-SIZE: 11pt; COLOR: #1f497d; FONT-FAMILY: 'Calibri','sans-serif'"><o:p> </o:p></SPAN></P>
<P class=MsoNormal><SPAN
style="FONT-SIZE: 11pt; COLOR: #1f497d; FONT-FAMILY: 'Calibri','sans-serif'">Regards,<o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN
style="FONT-SIZE: 11pt; COLOR: #1f497d; FONT-FAMILY: 'Calibri','sans-serif'">Charles<o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN
style="FONT-SIZE: 11pt; COLOR: #1f497d; FONT-FAMILY: 'Calibri','sans-serif'"><o:p> </o:p></SPAN></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><SPAN
style="FONT-SIZE: 10pt; COLOR: #1f497d; FONT-FAMILY: 'Courier New'">--
<BR>===========================================<BR>Charles Hammitt<BR>Storage
Systems Specialist<BR>ITS Research Computing @ <o:p></o:p></SPAN></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><SPAN
style="FONT-SIZE: 10pt; COLOR: #1f497d; FONT-FAMILY: 'Courier New'">The
University of North
Carolina-CH<BR><BR>===========================================<o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN
style="FONT-SIZE: 11pt; COLOR: #1f497d; FONT-FAMILY: 'Calibri','sans-serif'"><o:p> </o:p></SPAN></P>
<P class=MsoNormal><SPAN
style="FONT-SIZE: 11pt; COLOR: #1f497d; FONT-FAMILY: 'Calibri','sans-serif'"><o:p> </o:p></SPAN></P>
<DIV>
<DIV
style="BORDER-RIGHT: medium none; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; PADDING-LEFT: 0in; PADDING-BOTTOM: 0in; BORDER-LEFT: medium none; PADDING-TOP: 3pt; BORDER-BOTTOM: medium none">
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><B><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: 'Tahoma','sans-serif'">From:</SPAN></B><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: 'Tahoma','sans-serif'">
lustre-discuss-bounces@lists.lustre.org
[mailto:lustre-discuss-bounces@lists.lustre.org] <B>On Behalf Of
</B>zhengfeng<BR><B>Sent:</B> Thursday, March 15, 2012 12:11 AM<BR><B>To:</B>
lustre-discuss@lists.lustre.org<BR><B>Subject:</B> [Lustre-discuss] Lustre read
performance decay when OSSes are assigned in two different
subnet<o:p></o:p></SPAN></P></DIV></DIV>
<P class=MsoNormal><o:p> </o:p></P>
<DIV>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><SPAN
style="FONT-SIZE: 10.5pt; COLOR: navy; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: navy"><o:p></o:p></SPAN></P></DIV>
<DIV>
<DIV style="MARGIN: 7.5pt">
<DIV>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">Dear all,<o:p></o:p></SPAN></P></DIV>
<DIV>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt; TEXT-INDENT: 24pt"><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">We met</SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">one problem about Lustre read
performance decay when OSSes are assigned in two different
subnet.<o:p></o:p></SPAN></P></DIV>
<DIV>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt; TEXT-INDENT: 24pt"><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">Describing that in the following
diagram:<o:p></o:p></SPAN></P></DIV>
<DIV>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt; TEXT-INDENT: 24pt"><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">diagram 1, OSS in different
subnets:<o:p></o:p></SPAN></P></DIV>
<DIV>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt; TEXT-INDENT: 120pt"><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">Client (subnet
10.0.1.2)<o:p></o:p></SPAN></P></DIV>
<DIV>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt; TEXT-INDENT: 120pt"><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black"> |<o:p></o:p></SPAN></P></DIV>
<DIV>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt; TEXT-INDENT: 120pt"><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black"> |<o:p></o:p></SPAN></P></DIV>
<DIV>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt; TEXT-INDENT: 120pt"><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black"> |<o:p></o:p></SPAN></P></DIV>
<DIV>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt; TEXT-INDENT: 24pt"><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">Switch <o:p></o:p></SPAN></P></DIV>
<DIV>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt; TEXT-INDENT: 96pt"><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">|</SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black"> | <o:p></o:p></SPAN></P>
<DIV>
<DIV>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt; TEXT-INDENT: 96pt"><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">|</SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">
|<o:p></o:p></SPAN></P></DIV></DIV></DIV>
<DIV>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt; TEXT-INDENT: 96pt"><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">|</SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black"> |<o:p></o:p></SPAN></P></DIV>
<DIV>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt; TEXT-INDENT: 1in"><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black"> OSS1</SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black"> OSS2<o:p></o:p></SPAN></P></DIV>
<DIV>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt; TEXT-INDENT: 1in"><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">(10.0.2.2)</SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">(10.0.3.2)<o:p></o:p></SPAN></P></DIV>
<DIV>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt; TEXT-INDENT: 24pt"><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">For diagram 1, we made the CLient OSS1
and OSS2 in 3 different subnets. the switch</SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">used</SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">is able forward</SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">all packages.</SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black"><o:p></o:p></SPAN></P></DIV>
<DIV>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt; TEXT-INDENT: 24pt"><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">Use dd cmd to test r/w performance<SPAN
lang=ZH-CN>,</SPAN> write/rad data to/from</SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">to</SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">OSS1 and OSS2 at the same
time:<o:p></o:p></SPAN></P></DIV>
<DIV>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">test result:<o:p></o:p></SPAN></P></DIV>
<DIV>
<DIV>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt; TEXT-INDENT: 24pt"><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">[root@client</SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">client]#</SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">time</SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">dd</SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">if=test2</SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">of=/dev/null</SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">bs=1M</SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">count=2000<o:p></o:p></SPAN></I></P></DIV>
<DIV>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt; TEXT-INDENT: 24pt"><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">2000+0</SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">records</SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">in<o:p></o:p></SPAN></I></P></DIV>
<DIV>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt; TEXT-INDENT: 24pt"><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">2000+0</SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">records</SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">out<o:p></o:p></SPAN></I></P></DIV>
<DIV>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt; TEXT-INDENT: 24pt"><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">2097152000</SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">bytes</SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">(2.1</SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">GB)</SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">copied,</SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">53.5922</SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">seconds,</SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">39.1</SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">MB/s<o:p></o:p></SPAN></I></P></DIV>
<DIV>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt; TEXT-INDENT: 24pt"><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black"><o:p></o:p></SPAN></I></P></DIV>
<DIV>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt; TEXT-INDENT: 24pt"><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">real
0m53.796s<o:p></o:p></SPAN></I></P></DIV>
<DIV>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt; TEXT-INDENT: 24pt"><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">user
0m0.005s<o:p></o:p></SPAN></I></P></DIV>
<DIV>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt; TEXT-INDENT: 24pt"><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">sys
0m2.914s<o:p></o:p></SPAN></I></P></DIV></DIV>
<DIV>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt; TEXT-INDENT: 24pt"><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black"><o:p></o:p></SPAN></P></DIV>
<DIV>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt; TEXT-INDENT: 24pt"><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black"><o:p></o:p></SPAN></P></DIV>
<DIV>
<DIV>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt; TEXT-INDENT: 24pt"><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">diagram 2, OSS in same
subnet:<o:p></o:p></SPAN></P></DIV>
<DIV>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt; TEXT-INDENT: 120pt"><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">Client (subnet
10.0.1.2)<o:p></o:p></SPAN></P></DIV>
<DIV>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt; TEXT-INDENT: 120pt"><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black"> |<o:p></o:p></SPAN></P></DIV>
<DIV>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt; TEXT-INDENT: 120pt"><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black"> |<o:p></o:p></SPAN></P></DIV>
<DIV>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt; TEXT-INDENT: 120pt"><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black"> |<o:p></o:p></SPAN></P></DIV>
<DIV>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt; TEXT-INDENT: 24pt"><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">Switch <o:p></o:p></SPAN></P></DIV>
<DIV>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt; TEXT-INDENT: 96pt"><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">|</SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black"> | <o:p></o:p></SPAN></P>
<DIV>
<DIV>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt; TEXT-INDENT: 96pt"><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">|</SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">
|<o:p></o:p></SPAN></P></DIV></DIV></DIV>
<DIV>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt; TEXT-INDENT: 96pt"><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">|</SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black"> |<o:p></o:p></SPAN></P></DIV>
<DIV>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt; TEXT-INDENT: 1in"><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black"> OSS1</SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black"> OSS2<o:p></o:p></SPAN></P></DIV>
<DIV>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt; TEXT-INDENT: 1in"><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">(10.0.2.2, 10.0.2.3, at same
subnet)<o:p></o:p></SPAN></P></DIV>
<DIV>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black"><o:p></o:p></SPAN></P></DIV>
<DIV>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">for diagram 2, we assigned OSS1 and OSS2
at the same subnet, then test:<o:p></o:p></SPAN></P></DIV>
<DIV>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">test result:<o:p></o:p></SPAN></P></DIV>
<DIV>
<DIV>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt; TEXT-INDENT: 24pt"><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">[root@client219</SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">client]#</SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">time</SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">dd</SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">of=/dev/null</SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">if=test1</SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">bs=1M</SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black"><o:p></o:p></SPAN></P></DIV></DIV>
<DIV>
<DIV>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">10000+0</SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">records</SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">in<o:p></o:p></SPAN></I></P></DIV>
<DIV>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">10000+0</SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">records</SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">out<o:p></o:p></SPAN></I></P></DIV>
<DIV>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">10485760000</SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">bytes</SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">(10</SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">GB)</SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">copied,</SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">193.07</SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">seconds,</SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">54.3</SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">MB/s<o:p></o:p></SPAN></I></P></DIV>
<DIV>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black"><o:p></o:p></SPAN></I></P></DIV>
<DIV>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN></I><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black"><o:p></o:p></SPAN></I></P></DIV></DIV></DIV>
<DIV>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">conclusion:<o:p></o:p></SPAN></P></DIV>
<DIV>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt; TEXT-INDENT: 24pt"><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">In different subnets, the OSS read
performance is 39.1</SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">MB/s, while OSS
in<o:p></o:p></SPAN></P></DIV>
<DIV>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">same subnet, the read performance is
54.3</SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">MB/s. the performance decays so
much.<o:p></o:p></SPAN></P></DIV>
<DIV>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black"><o:p></o:p></SPAN></P></DIV>
<DIV>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">Question:<o:p></o:p></SPAN></P></DIV>
<DIV>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt; TEXT-INDENT: 24pt"><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">Why using different subnets in lustre,
the performance decayed?<o:p></o:p></SPAN></P></DIV>
<DIV>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt; TEXT-INDENT: 24pt"><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">Anyone had met such problems? Many
thanks for your answers and advice.<o:p></o:p></SPAN></P></DIV>
<DIV>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black"><o:p></o:p></SPAN></P></DIV>
<DIV>
<DIV class=MsoNormal style="MARGIN: 0in 0in 0pt"><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">
<HR style="WIDTH: 157.5pt; COLOR: #b5c4df" align=left width=210 noShade SIZE=1>
</SPAN></DIV></DIV>
<DIV>
<DIV>
<DIV>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><B><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">B.R.<o:p></o:p></SPAN></I></B></P></DIV>
<DIV>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><B><I><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black">Feng<o:p></o:p></SPAN></I></B></P></DIV>
<DIV>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black; FONT-FAMILY: 'Times New Roman','serif'"> </SPAN><SPAN
style="FONT-SIZE: 10.5pt; COLOR: black"><o:p></o:p></SPAN></P></DIV></DIV></DIV></DIV></DIV></DIV></DIV></DIV></BODY></HTML>