<html><head><base href="x-msg://164/"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><br><div><div>On May 26, 2011, at 6:01 AM, Eric Barton wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><span class="Apple-style-span" style="border-collapse: separate; font-family: 'Trebuchet MS'; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div bgcolor="white" lang="EN-GB" link="blue" vlink="purple"><div class="WordSection1" style="page: WordSection1; "><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0.0001pt; margin-left: 0cm; font-size: 12pt; font-family: 'Times New Roman', serif; color: black; "><span style="color: rgb(31, 73, 125); ">Nasf,<o:p></o:p></span></div><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0.0001pt; margin-left: 0cm; font-size: 12pt; font-family: 'Times New Roman', serif; color: black; "><span style="color: rgb(31, 73, 125); "><o:p> </o:p></span></div><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0.0001pt; margin-left: 0cm; font-size: 12pt; font-family: 'Times New Roman', serif; color: black; "><span style="color: rgb(31, 73, 125); ">Interesting results. Thank you - especially for graphing the results so thoroughly.<o:p></o:p></span></div><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0.0001pt; margin-left: 0cm; font-size: 12pt; font-family: 'Times New Roman', serif; color: black; "><span style="color: rgb(31, 73, 125); ">I’m attaching them here and cc-ing lustre-devel since these are of general interest.<o:p></o:p></span></div><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0.0001pt; margin-left: 0cm; font-size: 12pt; font-family: 'Times New Roman', serif; color: black; "><span style="color: rgb(31, 73, 125); "><o:p> </o:p></span></div><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0.0001pt; margin-left: 0cm; font-size: 12pt; font-family: 'Times New Roman', serif; color: black; "><span style="color: rgb(31, 73, 125); ">I don’t think your conclusion number (1), to say CLIO locking is slowing us down<o:p></o:p></span></div><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0.0001pt; margin-left: 0cm; font-size: 12pt; font-family: 'Times New Roman', serif; color: black; "><span style="color: rgb(31, 73, 125); ">is as obvious from these results as you imply. If you just compare the 1.8 and<o:p></o:p></span></div><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0.0001pt; margin-left: 0cm; font-size: 12pt; font-family: 'Times New Roman', serif; color: black; "><span style="color: rgb(31, 73, 125); ">patched 2.x per-file times and how they scale with #stripes you get this…<o:p></o:p></span></div><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0.0001pt; margin-left: 0cm; font-size: 12pt; font-family: 'Times New Roman', serif; color: black; "><span style="color: rgb(31, 73, 125); "><o:p> </o:p></span></div><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0.0001pt; margin-left: 0cm; font-size: 12pt; font-family: 'Times New Roman', serif; color: black; "><span style="color: rgb(31, 73, 125); "><span><image001.png></span></span><span style="color: rgb(31, 73, 125); "><o:p></o:p></span></div><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0.0001pt; margin-left: 0cm; font-size: 12pt; font-family: 'Times New Roman', serif; color: black; "><span style="color: rgb(31, 73, 125); "><o:p> </o:p></span></div><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0.0001pt; margin-left: 0cm; font-size: 12pt; font-family: 'Times New Roman', serif; color: black; "><span style="color: rgb(31, 73, 125); ">The gradients of these lines should correspond to the additional time per stripe required<o:p></o:p></span></div><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0.0001pt; margin-left: 0cm; font-size: 12pt; font-family: 'Times New Roman', serif; color: black; "><span style="color: rgb(31, 73, 125); ">to stat each file and I’ve graphed these times below (ignoring the 0-stripe data for this<o:p></o:p></span></div><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0.0001pt; margin-left: 0cm; font-size: 12pt; font-family: 'Times New Roman', serif; color: black; "><span style="color: rgb(31, 73, 125); ">calculation because I’m just interested in the incremental per-stripe overhead).<o:p></o:p></span></div><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0.0001pt; margin-left: 0cm; font-size: 12pt; font-family: 'Times New Roman', serif; color: black; "><span style="color: rgb(31, 73, 125); "><o:p> </o:p></span></div><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0.0001pt; margin-left: 0cm; font-size: 12pt; font-family: 'Times New Roman', serif; color: black; "><span style="color: rgb(31, 73, 125); "><span><image004.png></span></span><span style="color: rgb(31, 73, 125); "><o:p></o:p></span></div><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0.0001pt; margin-left: 0cm; font-size: 12pt; font-family: 'Times New Roman', serif; color: black; "><span style="color: rgb(31, 73, 125); ">They show per-stripe overhead for 1.8 well above patched 2.x for the lower stripe<o:p></o:p></span></div><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0.0001pt; margin-left: 0cm; font-size: 12pt; font-family: 'Times New Roman', serif; color: black; "><span style="color: rgb(31, 73, 125); ">counts, but whereas 1.8 gets better with more stripes, patched 2.x gets worse. I’m<o:p></o:p></span></div><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0.0001pt; margin-left: 0cm; font-size: 12pt; font-family: 'Times New Roman', serif; color: black; "><span style="color: rgb(31, 73, 125); ">guessing that at high stripe counts, 1.8 puts many concurrent glimpses on the wire<o:p></o:p></span></div><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0.0001pt; margin-left: 0cm; font-size: 12pt; font-family: 'Times New Roman', serif; color: black; "><span style="color: rgb(31, 73, 125); ">and does it quite efficiently. I’d like to understand better how you control the #<o:p></o:p></span></div><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0.0001pt; margin-left: 0cm; font-size: 12pt; font-family: 'Times New Roman', serif; color: black; "><span style="color: rgb(31, 73, 125); ">of glimpse-aheads you keep on the wire – is it a single fixed number, or a fixed<o:p></o:p></span></div><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0.0001pt; margin-left: 0cm; font-size: 12pt; font-family: 'Times New Roman', serif; color: black; "><span style="color: rgb(31, 73, 125); ">number per OST or some other scheme? In any case, it will be interesting to see<o:p></o:p></span></div><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0.0001pt; margin-left: 0cm; font-size: 12pt; font-family: 'Times New Roman', serif; color: black; "><span style="color: rgb(31, 73, 125); ">measurements at higher stripe counts.<o:p></o:p></span></div><blockquote style="margin-top: 5pt; margin-bottom: 5pt; "><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0.0001pt; margin-left: 0cm; font-size: 12pt; font-family: 'Times New Roman', serif; color: black; "><span lang="EN-US" style="color: rgb(31, 73, 125); ">Cheers,<span class="Apple-converted-space"> </span><br> Eric<o:p></o:p></span></div></blockquote><div style="border-top-style: none; border-right-style: none; border-bottom-style: none; border-width: initial; border-color: initial; border-left-style: solid; border-left-color: blue; border-left-width: 1.5pt; padding-top: 0cm; padding-right: 0cm; padding-bottom: 0cm; padding-left: 4pt; position: static; z-index: auto; "><div><div style="border-right-style: none; border-bottom-style: none; border-left-style: none; border-width: initial; border-color: initial; border-top-style: solid; border-top-color: rgb(181, 196, 223); border-top-width: 1pt; padding-top: 3pt; padding-right: 0cm; padding-bottom: 0cm; padding-left: 0cm; "><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0.0001pt; margin-left: 0cm; font-size: 12pt; font-family: 'Times New Roman', serif; color: black; "><b><span lang="EN-US" style="font-size: 10pt; font-family: Tahoma, sans-serif; color: windowtext; ">From:</span></b><span lang="EN-US" style="font-size: 10pt; font-family: Tahoma, sans-serif; color: windowtext; "><span class="Apple-converted-space"> </span>Fan Yong [mailto:yong.fan@whamcloud.com]<span class="Apple-converted-space"> </span><br><b>Sent:</b><span class="Apple-converted-space"> </span>12 May 2011 10:18 AM<br><b>To:</b><span class="Apple-converted-space"> </span>Eric Barton<br><b>Cc:</b><span class="Apple-converted-space"> </span>Bryon Neitzel; Ian Colle; Liang Zhen<br><b>Subject:</b><span class="Apple-converted-space"> </span>New test results for "ls -Ul"<o:p></o:p></span></div></div></div><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0.0001pt; margin-left: 0cm; font-size: 12pt; font-family: 'Times New Roman', serif; color: black; "><o:p> </o:p></div><p class="MsoNormal" style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 12pt; margin-left: 0cm; font-size: 12pt; font-family: 'Times New Roman', serif; color: black; ">I have improved statahead load balance mechanism to distribute statahead load to more CPU units on client. And adjusted AGL according to CLIO lock state machine. After those improvement, 'ls -Ul' can run more fast than old patches, especially on large SMP node.<br><br>On the other hand, as the increasing the degree of parallelism, the lower network scheduler is becoming performance bottleneck. So I combine my patches together with Liang's SMP patches in the test.<o:p></o:p></p><table class="MsoNormalTable" border="1" cellpadding="0" width="100%" style="width: 1019px; "><tbody><tr><td valign="top" style="padding-top: 1.5pt; padding-right: 1.5pt; padding-bottom: 1.5pt; padding-left: 1.5pt; "></td><td valign="top" style="padding-top: 1.5pt; padding-right: 1.5pt; padding-bottom: 1.5pt; padding-left: 1.5pt; "><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0.0001pt; margin-left: 0cm; font-size: 12pt; font-family: 'Times New Roman', serif; color: black; ">client (fat-intel-4, 24 cores)<o:p></o:p></div></td><td valign="top" style="padding-top: 1.5pt; padding-right: 1.5pt; padding-bottom: 1.5pt; padding-left: 1.5pt; "><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0.0001pt; margin-left: 0cm; font-size: 12pt; font-family: 'Times New Roman', serif; color: black; ">server (client-xxx, 4 OSSes, 8 OSTs on each OSS)<o:p></o:p></div></td></tr><tr><td valign="top" style="padding-top: 1.5pt; padding-right: 1.5pt; padding-bottom: 1.5pt; padding-left: 1.5pt; "><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0.0001pt; margin-left: 0cm; font-size: 12pt; font-family: 'Times New Roman', serif; color: black; ">b2x_patched<o:p></o:p></div></td><td valign="top" style="padding-top: 1.5pt; padding-right: 1.5pt; padding-bottom: 1.5pt; padding-left: 1.5pt; "><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0.0001pt; margin-left: 0cm; font-size: 12pt; font-family: 'Times New Roman', serif; color: black; ">my patches + SMP patches<o:p></o:p></div></td><td valign="top" style="padding-top: 1.5pt; padding-right: 1.5pt; padding-bottom: 1.5pt; padding-left: 1.5pt; "><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0.0001pt; margin-left: 0cm; font-size: 12pt; font-family: 'Times New Roman', serif; color: black; ">my patches<o:p></o:p></div></td></tr><tr><td valign="top" style="padding-top: 1.5pt; padding-right: 1.5pt; padding-bottom: 1.5pt; padding-left: 1.5pt; "><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0.0001pt; margin-left: 0cm; font-size: 12pt; font-family: 'Times New Roman', serif; color: black; ">b18<o:p></o:p></div></td><td valign="top" style="padding-top: 1.5pt; padding-right: 1.5pt; padding-bottom: 1.5pt; padding-left: 1.5pt; "><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0.0001pt; margin-left: 0cm; font-size: 12pt; font-family: 'Times New Roman', serif; color: black; ">original b1_8<o:p></o:p></div></td><td valign="top" style="padding-top: 1.5pt; padding-right: 1.5pt; padding-bottom: 1.5pt; padding-left: 1.5pt; "><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0.0001pt; margin-left: 0cm; font-size: 12pt; font-family: 'Times New Roman', serif; color: black; ">share the same server with "b2x_patched"<o:p></o:p></div></td></tr><tr><td valign="top" style="padding-top: 1.5pt; padding-right: 1.5pt; padding-bottom: 1.5pt; padding-left: 1.5pt; "><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0.0001pt; margin-left: 0cm; font-size: 12pt; font-family: 'Times New Roman', serif; color: black; ">b2x_original<o:p></o:p></div></td><td valign="top" style="padding-top: 1.5pt; padding-right: 1.5pt; padding-bottom: 1.5pt; padding-left: 1.5pt; "><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0.0001pt; margin-left: 0cm; font-size: 12pt; font-family: 'Times New Roman', serif; color: black; ">original b2_x<o:p></o:p></div></td><td valign="top" style="padding-top: 1.5pt; padding-right: 1.5pt; padding-bottom: 1.5pt; padding-left: 1.5pt; "><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0.0001pt; margin-left: 0cm; font-size: 12pt; font-family: 'Times New Roman', serif; color: black; ">original b2_x<o:p></o:p></div></td></tr></tbody></table><div style="margin-top: 0cm; margin-right: 0cm; margin-bottom: 0.0001pt; margin-left: 0cm; font-size: 12pt; font-family: 'Times New Roman', serif; color: black; "><br>Some notes:<br><br>1) Stripe count affects traversing performance much, and the impact is more than linear. Even if with all the patches applied on b2_x, the degree of stripe count impact is still larger than b1_8. It is related with the complex CLIO lock state machine and tedious iteration/repeat operations. It is not easy to make it run as efficiently as b1_8.<br></div></div></div></div></span></blockquote><div><br></div><div><br></div><div>Hi there,</div><div><br></div><div>I did some tests to investigate the overhead of clio lock state machine and glimpse lock, and I found something new.</div><div><br></div><div>Basically I did the same thing as what Nasf had done, but I only cared about the overhead of glimpse locks. For this purpose, I ran 'ls -lU' twice for each test, and the 1st run is only used to create IBITS UPDATE lock cache for files; then, I dropped cl_locks and ldlm_locks from client side cache by setting zero to lru_size of ldlm namespaces, then do 'ls -lU' once again. In the second run of 'ls -lU', the statahead thread will always find cached IBITS lock(we can check mdc lock_count for sure), so the elapsed time of ls will be glimpse related.</div><div><br></div><div>This is what I got from the test:</div><div><br></div><div></div></div></body></html>