<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:0 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:623998187;
mso-list-type:hybrid;
mso-list-template-ids:241086150 -1298742050 67895299 67895301 67895297 67895299 67895301 67895297 67895299 67895301;}
@list l0:level1
{mso-level-start-at:0;
mso-level-number-format:bullet;
mso-level-text:-;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-font-family:Calibri;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:\uf0a7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:\uf0b7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:\uf0a7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:\uf0b7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:\uf0a7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="FR" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-family:Helvetica">></span> read | write<br>
>disk I/Os in flight ios % cum % | ios % cum %<br>
>1: 211177215 61 61 | 29305564 97 97<br>
>2: 41332944 11 72 | 498260 1 99<br>
>[..]<o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:Helvetica">>Does these lines means :<br>
>Since last snapshot there was 211177215x1 and read 41332944x2 I/O in flight ?<br>
<br>
</span><span style="mso-fareast-language:EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">It means (since the last time the statistics were cleared)<o:p></o:p></span></p>
<ul style="margin-top:0cm" type="disc">
<li class="MsoListParagraph" style="margin-left:0cm;mso-list:l0 level1 lfo1"><span style="mso-fareast-language:EN-US">11% of the time, 2 READ I/O requests were "in-flights" to disk, meaning 2 I/O were sent to disks and not yet commit/acknowledged<o:p></o:p></span></li><li class="MsoListParagraph" style="margin-left:0cm;mso-list:l0 level1 lfo1"><span style="mso-fareast-language:EN-US">61 % of the time, only 1 READ I/O request.<o:p></o:p></span></li></ul>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Same principle for write.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">What this means here is that your workload is not feeding the disks with lots of write (97% with 1 I/O in flight), but a bit more reads.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Disks and especially disk arrays are reordering I/O and distributing them across the various drives they are composed of to optimized bandwith. To really take benefits of all the possible bandwith/throughput
your hardward can offer, you often need to be able to have lots of big I/O and possible multiple I/O in flights.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Few I/O in flight could means:<o:p></o:p></span></p>
<ul style="margin-top:0cm" type="disc">
<li class="MsoListParagraph" style="margin-left:0cm;mso-list:l0 level1 lfo1"><span style="mso-fareast-language:EN-US">your workload is not really big<o:p></o:p></span></li><li class="MsoListParagraph" style="margin-left:0cm;mso-list:l0 level1 lfo1"><span style="mso-fareast-language:EN-US">your hardward is fast compared to the throughput coming to this server (ratio disk BW vs network BW by example)<o:p></o:p></span></li></ul>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">This could also help you identify bad performance numbers and find from where the bottleneck comes from.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">De : </span></b><span style="font-size:12.0pt;color:black">lustre-discuss <lustre-discuss-bounces@lists.lustre.org> au nom de Louis Bailleul <Louis.Bailleul@pgs.com><br>
<b>Date : </b>mardi 16 juillet 2019 à 17:49<br>
<b>À : </b>lustre-discuss <lustre-discuss@lists.lustre.org><br>
<b>Objet : </b>Re: [lustre-discuss] [External] Re: obdfilter/mdt stats meaning ?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal"><span style="font-family:Helvetica">Hi Aurélien,<br>
<br>
Thanks for the prompt reply.<br>
For the ost stats, any idea what the preprw and commitrw mean ?<br>
And why there are two entries with different values for statfs ?<br>
<br>
For brw_stats even with the doc I still struggle to read this.<br>
For example how do you make sense of disk I/O in flight ?</span><o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"> read | write<br>
disk I/Os in flight ios % cum % | ios % cum %<br>
1: 211177215 61 61 | 29305564 97 97<br>
2: 41332944 11 72 | 498260 1 99<br>
[..]<o:p></o:p></p>
</blockquote>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-family:Helvetica">Does these lines means :<br>
Since last snapshot there was 211177215x1 and read 41332944x2 I/O in flight ?<br>
<br>
Best regards,<br>
Louis</span><o:p></o:p></p>
<div>
<p class="MsoNormal">On 16/07/2019 15:50, Degremont, Aurelien wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Hi Louis, </span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">About brw_stats, there are a bit of explanation in the Lustre Doc (not that detailed, but still)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><a href="http://doc.lustre.org/lustre_manual.xhtml#dbdoclet.50438271_55057">http://doc.lustre.org/lustre_manual.xhtml#dbdoclet.50438271_55057</a></span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">> </span><span style="font-family:Helvetica">Last thing, is there any way to get the name of the filesystem an OST is part of by using lctl ?</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">I don't know what you want exactly, but the OST names are self explanatory, there always are like: fsname-OSTXXXX</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Where fsname is the lustre filesystem they are part of.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">For obdfilter stats, these are mostly action to OST objects or client connection management RPCs.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal"> setattr: changing an OST object attributes (owner, group, ...)<br>
punch: mostly used for truncate (theorically can do holes in files, like truncate with a start and length)<br>
sync: straighforward, sync OST to disk<br>
destroy: delete an OST object (mostly when a file is deleted)<br>
create: create an OST object<br>
statfs: like 'df' for this specific OST (used by 'lfs df' by example)<br>
(re)connect: when a client connect/reconnect to this OST <br>
ping: when a client ping this OST.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Aurélien<o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">De : </span></b><span style="font-size:12.0pt;color:black">lustre-discuss
<a href="mailto:lustre-discuss-bounces@lists.lustre.org"><lustre-discuss-bounces@lists.lustre.org></a> au nom de Louis Bailleul
<a href="mailto:Louis.Bailleul@pgs.com"><Louis.Bailleul@pgs.com></a><br>
<b>Date : </b>mardi 16 juillet 2019 à 16:38<br>
<b>À : </b>lustre-discuss <a href="mailto:lustre-discuss@lists.lustre.org"><lustre-discuss@lists.lustre.org></a><br>
<b>Objet : </b>[lustre-discuss] obdfilter/mdt stats meaning ?</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="font-family:Helvetica">Hi all,<br>
<br>
I am trying to make sense of some of the OST/MDT stats for 2.12.<br>
Can anybody point me to the doc that explain what the metrics are ?<br>
The wiki only mention read/write/get_info : <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__wiki.lustre.org_Lustre-5FMonitoring-5Fand-5FStatistics-5FGuide&d=DwMGaQ&c=KV_I7O14pmwRcmAVyJ1eg4Jwb8Y2JAxuL5YgMGHpjcQ&r=FTXmt89oLXmbXfP78w86-PxB1XdLYgxG8hEoAnZvCvs&m=UC1t7z9tgmxUE2FWaTFHFT_Y69z_VMH0dEYF1VXadX0&s=cdXTUStD_NPwj3GtNYBqJA2nkJ1Ec53F9aD5UxFo5tw&e=">
http://wiki.lustre.org/Lustre_Monitoring_and_Statistics_Guide</a><br>
But the list I get is quite different :</span><o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="margin-bottom:12.0pt"> obdfilter.OST001.stats=<br>
snapshot_time 1563285450.647120173 secs.nsecs<br>
read_bytes 340177708 samples [bytes] 4096 4194304 396712660910080<br>
write_bytes 30008856 samples [bytes] 24 4194304 78618271501667<br>
setattr 1755 samples [reqs]<br>
punch 73463 samples [reqs]<br>
sync 50606 samples [reqs]<br>
destroy 31990 samples [reqs]<br>
create 956 samples [reqs]<br>
statfs 75378743 samples [reqs]<br>
connect 5798 samples [reqs]<br>
reconnect 3242 samples [reqs]<br>
disconnect 5820 samples [reqs]<br>
statfs 3737980 samples [reqs]<br>
preprw 370186566 samples [reqs]<br>
commitrw 370186557 samples [reqs]<br>
ping 882096292 samples [reqs]<o:p></o:p></p>
</blockquote>
<p class="MsoNormal"><span style="font-family:Helvetica">For the MDT, most are pretty much self explanatory, but I'll still be happy to be pointed to some doc.</span><o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="margin-bottom:12.0pt">mdt.MDT0000.md_stats=<br>
snapshot_time 1563287416.006001068 secs.nsecs<br>
open 3174644054 samples [reqs]<br>
close 3174494603 samples [reqs]<br>
mknod 107564 samples [reqs]<br>
unlink 99625 samples [reqs]<br>
mkdir 199643 samples [reqs]<br>
rmdir 45021 samples [reqs]<br>
rename 12728 samples [reqs]<br>
getattr 50227431 samples [reqs]<br>
setattr 103435 samples [reqs]<br>
getxattr 9051470 samples [reqs]<br>
setxattr 14 samples [reqs]<br>
statfs 7525513 samples [reqs]<br>
sync 20597 samples [reqs]<br>
samedir_rename 207 samples [reqs]<br>
crossdir_rename 12521 samples [reqs]<o:p></o:p></p>
</blockquote>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-family:Helvetica">And anyone knows how to read the OST brw_stats ?</span><o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="margin-bottom:12.0pt">obdfilter.OST0014.brw_stats=<br>
snapshot_time: 1563287631.511085465 (secs.nsecs)<br>
<br>
read | write<br>
pages per bulk r/w rpcs % cum % | rpcs % cum %<br>
1: 231699298 66 66 | 180944 0 0<br>
2: 855611 0 67 | 322359 1 1<br>
4: 541749 0 67 | 5539716 18 20<br>
8: 1281219 0 67 | 67837 0 20<br>
16: 637808 0 67 | 114546 0 20<br>
32: 1342813 0 68 | 3099780 10 31<br>
64: 1559834 0 68 | 173166 0 31<br>
128: 1583127 0 69 | 211512 0 32<br>
256: 10627583 3 72 | 499978 1 34<br>
512: 3909601 1 73 | 1029686 3 37<br>
1K: 92141161 26 100 | 18788597 62 100<br>
<br>
read | write<br>
discontiguous pages rpcs % cum % | rpcs % cum %<br>
0: 346179839 100 100 | 180946 0 0<br>
1: 0 0 100 | 322363 1 1<br>
2: 0 0 100 | 5521062 18 20<br>
3: 0 0 100 | 18650 0 20<br>
4: 0 0 100 | 18159 0 20<br>
5: 0 0 100 | 26664 0 20<br>
6: 0 0 100 | 10830 0 20<br>
7: 0 0 100 | 12189 0 20<br>
8: 0 0 100 | 11365 0 20<br>
9: 0 0 100 | 10253 0 20<br>
10: 0 0 100 | 8810 0 20<br>
11: 0 0 100 | 9825 0 20<br>
12: 0 0 100 | 16740 0 20<br>
13: 0 0 100 | 14421 0 20<br>
14: 0 0 100 | 10513 0 20<br>
15: 0 0 100 | 32655 0 20<br>
16: 0 0 100 | 1418677 4 25<br>
17: 0 0 100 | 1477077 4 30<br>
18: 0 0 100 | 6227 0 30<br>
19: 0 0 100 | 7071 0 30<br>
20: 0 0 100 | 7297 0 30<br>
21: 0 0 100 | 8478 0 30<br>
22: 0 0 100 | 34591 0 30<br>
23: 0 0 100 | 35591 0 30<br>
24: 0 0 100 | 8378 0 30<br>
25: 0 0 100 | 8724 0 30<br>
26: 0 0 100 | 52300 0 30<br>
27: 0 0 100 | 14038 0 30<br>
28: 0 0 100 | 4734 0 30<br>
29: 0 0 100 | 4878 0 31<br>
30: 0 0 100 | 6232 0 31<br>
31: 0 0 100 | 20708383 68 100<br>
read | write<br>
disk I/Os in flight ios % cum % | ios % cum %<br>
1: 211177215 61 61 | 29305564 97 97<br>
2: 41332944 11 72 | 498260 1 99<br>
3: 22250410 6 79 | 86831 0 99<br>
4: 15524737 4 83 | 34513 0 99<br>
5: 12049717 3 87 | 19442 0 99<br>
6: 8904108 2 89 | 13107 0 99<br>
7: 5955503 1 91 | 8748 0 99<br>
8: 3943444 1 92 | 6869 0 99<br>
9: 3115034 0 93 | 5447 0 99<br>
10: 2553941 0 94 | 4593 0 99<br>
11: 2121217 0 95 | 3828 0 99<br>
12: 1709040 0 95 | 3264 0 99<br>
13: 1418541 0 95 | 2800 0 99<br>
14: 1184247 0 96 | 2454 0 99<br>
15: 1047397 0 96 | 2153 0 99<br>
16: 875229 0 96 | 1871 0 99<br>
17: 752555 0 97 | 1643 0 99<br>
18: 656424 0 97 | 1531 0 99<br>
19: 584066 0 97 | 1375 0 99<br>
20: 529630 0 97 | 1267 0 99<br>
21: 477143 0 97 | 1144 0 99<br>
22: 426303 0 97 | 1067 0 99<br>
23: 385707 0 97 | 984 0 99<br>
24: 354584 0 98 | 959 0 99<br>
25: 328332 0 98 | 899 0 99<br>
26: 305886 0 98 | 828 0 99<br>
27: 281444 0 98 | 786 0 99<br>
28: 261958 0 98 | 734 0 99<br>
29: 242335 0 98 | 711 0 99<br>
30: 227010 0 98 | 692 0 99<br>
31: 5203738 1 100 | 13757 0 100<br>
<br>
read | write<br>
I/O time (1/1000s) ios % cum % | ios % cum %<br>
1: 34363647 26 26 | 0 0 0<br>
2: 9013233 7 33 | 0 0 0<br>
4: 3381561 2 36 | 0 0 0<br>
8: 2194196 1 38 | 0 0 0<br>
16: 8767687 6 45 | 0 0 0<br>
32: 25062401 19 64 | 0 0 0<br>
64: 27196704 21 85 | 0 0 0<br>
128: 10760610 8 94 | 0 0 0<br>
256: 4203334 3 97 | 0 0 0<br>
512: 2002196 1 99 | 0 0 0<br>
1K: 785539 0 99 | 0 0 0<br>
2K: 340525 0 99 | 0 0 0<br>
4K: 140336 0 99 | 0 0 0<br>
8K: 6875 0 99 | 0 0 0<br>
16K: 161 0 100 | 0 0 0<br>
<br>
read | write<br>
disk I/O size ios % cum % | ios % cum %<br>
8: 4 0 0 | 0 0 0<br>
16: 0 0 0 | 0 0 0<br>
32: 1 0 0 | 4 0 0<br>
64: 1 0 0 | 5703 0 0<br>
128: 3061 0 0 | 2853 0 0<br>
256: 1 0 0 | 3340 0 0<br>
512: 1 0 0 | 309 0 0<br>
1K: 0 0 0 | 3697 0 0<br>
2K: 2 0 0 | 38311 0 0<br>
4K: 231696225 66 66 | 126727 0 0<br>
8K: 855613 0 67 | 322359 1 1<br>
16K: 541749 0 67 | 5539716 18 20<br>
32K: 1281219 0 67 | 67837 0 20<br>
64K: 637808 0 67 | 114546 0 20<br>
128K: 1342813 0 68 | 3099780 10 31<br>
256K: 1559834 0 68 | 173166 0 31<br>
512K: 1583127 0 69 | 211512 0 32<br>
1M: 10627583 3 72 | 499978 1 34<br>
2M: 3909601 1 73 | 1029686 3 37<br>
4M: 92141161 26 100 | 18788597 62 100<o:p></o:p></p>
</blockquote>
<p class="MsoNormal"><span style="font-family:Helvetica">Last thing, is there any way to get the name of the filesystem an OST is part of by using lctl ?</span><br>
<span style="font-family:Helvetica"><br>
Best regards,<br>
Louis<br>
<br>
<br>
</span><o:p></o:p></p>
</blockquote>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
</div>
</body>
</html>