<div dir="ltr"><div class="gmail_default" style="font-family:verdana,sans-serif">All my OSS's are mounted on SSD. So, I think that wouldn't be the reason behind such improvements.</div><div class="gmail_default" style="font-family:verdana,sans-serif"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif">Thanks,</div><div class="gmail_default" style="font-family:verdana,sans-serif">Hasan</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Oct 17, 2021 at 3:54 AM Andreas Dilger <<a href="mailto:adilger@whamcloud.com">adilger@whamcloud.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">



<div dir="auto">
It seems likely if you are using spinning disks (HDD)  that using max_rpcs_in_flight=1 will reduce the seeking on the OSTs, especially with large RPC sizes (16MB). <br>
<br>
<div dir="ltr">Cheers, Andreas</div>
<div dir="ltr"><br>
<blockquote type="cite">On Oct 16, 2021, at 16:11, Md Hasanur Rashid <<a href="mailto:mrashid2@uncc.edu" target="_blank">mrashid2@uncc.edu</a>> wrote:<br>
<br>
</blockquote>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div class="gmail_default" style="font-family:verdana,sans-serif">Hi Everyone,</div>
<div class="gmail_default" style="font-family:verdana,sans-serif"><br>
</div>
<div class="gmail_default" style="font-family:verdana,sans-serif">Thanks for getting back to me, Andreas and Patrick. The reason for me to experiment with max_rpcs_in_flight being 1 is that I have observed significant improvement over the default value when
 the workload is the following one:</div>
<div class="gmail_default" style="font-family:verdana,sans-serif">
<pre style="white-space:pre-wrap;color:rgb(36,41,47);box-sizing:border-box;font-family:ui-monospace,SFMono-Regular,"SF Mono",Menlo,Consolas,"Liberation Mono",monospace;font-size:11.9px;margin-top:0px;margin-bottom:16px;padding:16px;overflow:auto;line-height:1.45;border-radius:6px"><code style="box-sizing:border-box;font-family:ui-monospace,SFMono-Regular,"SF Mono",Menlo,Consolas,"Liberation Mono",monospace;font-size:11.9px;padding:0px;margin:0px;background:transparent;border-radius:6px;word-break:normal;border:0px;display:inline;overflow:visible;line-height:inherit">define fileset name="testF",entries=100,filesize=16m,prealloc,path="/mnt/hasanfs/tmp1"

define process name="readerP",instances=2 {
  thread name="readerT",instances=4 {
  flowop openfile name="openOP",filesetname="testF"
  flowop writewholefile name="writeOP",iters=4,filesetname="testF"
  flowop readwholefile name="readOP",iters=1,filesetname="testF"
  flowop closefile name="closeOP"
  }
}

create files
system "sync"
system "echo 3 > /proc/sys/vm/drop_caches"

run 60</code></pre>
</div>
<div class="gmail_default" style="font-family:verdana,sans-serif">The performance when max_rpcs_in_flight has default value looks like the following:</div>
<div class="gmail_default" style="font-family:verdana,sans-serif"><br>
</div>
<div class="gmail_default" style="font-family:verdana,sans-serif">
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<b><span style="font-variant-ligatures:no-common-ligatures">max_rpcs_in_flight=8, </span><span style="font-variant-ligatures:no-common-ligatures">max_pages_per_rpc=256, </span><span style="font-variant-ligatures:no-common-ligatures">max_dirty_mb=2000</span></b></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p style="margin:0px;font:11px Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">64.164: Per-Operation Breakdown</span></p>
<p style="margin:0px;font:11px Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">closeOP<span>             
</span>5612ops <span>      </span>94ops/s <span>
  </span>0.0mb/s<span>    </span>0.581ms/op [0.278ms - 1.966ms]</span></p>
<p style="margin:0px;font:11px Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">readOP
<span>              </span>5612ops <span>
      </span>94ops/s 5985.6mb/s <span>  </span>
19.335ms/op [14.957ms - 83.853ms]</span></p>
<p style="margin:0px;font:11px Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">writeOP<span>             
</span>22462ops<span>      </span>374ops/s 5989.3mb/s
<span>  </span>16.248ms/op [5.683ms - 626.451ms]</span></p>
<p style="margin:0px;font:11px Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">openOP
<span>              </span>5620ops <span>
      </span>94ops/s <span>  </span>0.0mb/s<span>   
</span>0.539ms/op [0.313ms - 27.266ms]</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">64.164: IO Summary: 39306 ops 655.039 ops/s 94/374 rd/wr 11974.9mb/s 12.206ms/op</span></p>
</div>
<div class="gmail_default" style="font-family:verdana,sans-serif"><br>
</div>
<div class="gmail_default" style="font-family:verdana,sans-serif">If we change the max_rpcs_in_flight value to 1, the performance is following:</div>
<div class="gmail_default" style="font-family:verdana,sans-serif"><br>
</div>
<div class="gmail_default" style="font-family:verdana,sans-serif"><b><span style="color:rgb(0,0,0);font-family:Menlo;font-size:11px;font-variant-ligatures:no-common-ligatures">max_rpcs_in_flight=1, </span><span style="color:rgb(0,0,0);font-family:Menlo;font-size:11px;font-variant-ligatures:no-common-ligatures">max_pages_per_rpc=256, </span><span style="color:rgb(0,0,0);font-family:Menlo;font-size:11px;font-variant-ligatures:no-common-ligatures">max_dirty_mb=2000</span></b><br>
</div>
<div class="gmail_default" style="font-family:verdana,sans-serif"><br>
</div>
<div class="gmail_default" style="font-family:verdana,sans-serif">
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">64.048: Per-Operation Breakdown</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">closeOP<span>             
</span>8631ops<span>      </span>144ops/s <span>
  </span>0.0mb/s<span>    </span>0.593ms/op [0.273ms - 4.810ms]</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">readOP <span>
              </span>8631ops<span>      </span>
144ops/s 9205.6mb/s <span>  </span>20.187ms/op [16.292ms - 37.513ms]</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">writeOP<span>             
</span>34545ops<span>      </span>576ops/s 9211.2mb/s<span>   
</span>8.561ms/op [5.860ms - 55.334ms]</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">openOP <span>
              </span>8639ops<span>      </span>
144ops/s <span>  </span>0.0mb/s<span>   
</span>0.552ms/op [0.293ms - 10.132ms]</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">64.048: IO Summary: 60446 ops 1007.344 ops/s 144/576 rd/wr 18416.8mb/s 7.939ms/op</span></p>
</div>
<div class="gmail_default" style="font-family:verdana,sans-serif"><br>
</div>
<div class="gmail_default" style="font-family:verdana,sans-serif">As you can see from above the performance got improved when the value got changed to 1. Unfortunately, I am still not sure why such improvement is being observed. I ran the workload from a single
 client. The cluster setup looks like the following (5 server machines, 1 for MGS+MDS, and the other 4 for OSSs where each one has 2 OSTs):</div>
<div class="gmail_default" style="font-family:verdana,sans-serif"><br>
</div>
<div class="gmail_default" style="font-family:verdana,sans-serif">
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">UUID <span>
                      </span>bytes<span>       
</span>Used <span>  </span>Available Use% Mounted on</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">hasanfs-MDT0000_UUID<span>     
</span>144.8G <span>      </span>78.1M<span>     
</span>134.8G <span>  </span>0% /mnt/hasanfs[MDT:0]</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">hasanfs-OST0000_UUID<span>     
</span>197.3G<span>        </span>1.1G<span>     
</span>185.4G <span>  </span>1% /mnt/hasanfs[OST:0]</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">hasanfs-OST0001_UUID<span>     
</span>197.3G<span>        </span>1.2G<span>     
</span>185.3G <span>  </span>1% /mnt/hasanfs[OST:1]</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">hasanfs-OST0002_UUID<span>     
</span>197.3G<span>        </span>1.2G<span>     
</span>185.3G <span>  </span>1% /mnt/hasanfs[OST:2]</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">hasanfs-OST0003_UUID<span>     
</span>197.3G<span>        </span>1.2G<span>     
</span>185.2G <span>  </span>1% /mnt/hasanfs[OST:3]</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">hasanfs-OST0004_UUID<span>     
</span>197.3G<span>        </span>1.2G<span>     
</span>185.3G <span>  </span>1% /mnt/hasanfs[OST:4]</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">hasanfs-OST0005_UUID<span>     
</span>197.3G<span>        </span>1.2G<span>     
</span>185.4G <span>  </span>1% /mnt/hasanfs[OST:5]</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">hasanfs-OST0006_UUID<span>     
</span>197.3G<span>        </span>1.2G<span>     
</span>185.3G <span>  </span>1% /mnt/hasanfs[OST:6]</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">hasanfs-OST0007_UUID<span>     
</span>197.3G<span>        </span>1.2G<span>     
</span>185.4G <span>  </span>1% /mnt/hasanfs[OST:7]</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);min-height:13px">
<span style="font-variant-ligatures:no-common-ligatures"></span><br>
</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">filesystem summary:
<span>        </span>1.5T<span>       
</span>9.6G<span>        </span>1.4T <span>
  </span>1% /mnt/hasanfs</span></p>
</div>
<div class="gmail_default" style="font-family:verdana,sans-serif"><br>
</div>
<div class="gmail_default" style="font-family:verdana,sans-serif">It would be really helpful for me and highly appreciated if you can share your intuitions or reasons regarding why such improvement may have been observed.</div>
<div class="gmail_default" style="font-family:verdana,sans-serif"><br>
</div>
<div class="gmail_default" style="font-family:verdana,sans-serif">Thanks,</div>
<div class="gmail_default" style="font-family:verdana,sans-serif">Md. Hasanur Rashid</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Thu, Oct 7, 2021 at 5:45 PM Andreas Dilger <<a href="mailto:adilger@whamcloud.com" target="_blank">adilger@whamcloud.com</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>[<strong><font color="red">Caution</font></strong>: Email from External Sender. Do not click or open links or attachments unless you know this sender.]<br>
<span> </span> On Oct 7, 2021, at 13:19, Md Hasanur Rashid via lustre-discuss <<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a>> wrote:<br>
<div>
<blockquote type="cite"><br>
<div>
<div dir="ltr">
<div class="gmail_default" style="font-family:verdana,sans-serif">Hello Everyone,<br>
</div>
<div class="gmail_quote">
<div dir="ltr">
<div style="font-family:verdana,sans-serif"><br>
I am running the Filebench benchmark in my Lustre cluster. I set the max_rpcs_in_flight value to be 1. Before executing and after executing, I verified that the value of max_rpcs_in_flight is indeed 1. However, when I check the rpc_stats, the stat shows a much
 higher value for RPCs in flight. </div>
</div>
</div>
</div>
</div>
</blockquote>
<div><br>
</div>
If by "much higher than 1" you mean "2", then yes it appears there are mostly (95%) 2 RPCs being processed concurrently on this OST.</div>
<div><br>
</div>
<div>That might happen if you have 2 clients/mountpoints writing to the same OST, it might be an off-by-one logic error allowing an extra RPC in flight, it might be intentional for some reason (e.g. to avoid deadlock, memory pressure, etc), or it might be accounting
 error in the statistics (e.g. counting the next RPC to be sent before the first one is marked finished).</div>
<div><br>
<blockquote type="cite">
<div>
<div dir="ltr">
<div class="gmail_quote">
<div dir="ltr">
<div style="font-family:verdana,sans-serif">Following is the value shown for one OSC just for reference:</div>
<div style="font-family:verdana,sans-serif"><br>
</div>
<div style="font-family:verdana,sans-serif"><font size="1">osc.hasanfs-OST0000-osc-ffff882fcf777000.rpc_stats=<br>
snapshot_time:         1632483604.967291 (secs.usecs)<br>
read RPCs in flight:  0<br>
write RPCs in flight: 0<br>
pending write pages:  0<br>
pending read pages:   0<br>
<br>
read write<br>
pages per rpc         rpcs   % cum % |       rpcs   % cum %<br>
1:         1 100 100   |          0   0   0<br>
2:         0   0 100   |          0   0   0<br>
4:         0   0 100   |          0   0   0<br>
8:         0   0 100   |          0   0   0<br>
16:         0   0 100   |          0   0   0<br>
32:         0   0 100   |          0   0   0<br>
64:         0   0 100   |          0   0   0<br>
128:         0   0 100   |          0   0   0<br>
256:         0   0 100   |       9508 100 100<br>
<br>
read write<br>
rpcs in flight        rpcs   % cum % |       rpcs   % cum %<br>
0:         0   0   0   |          0   0   0<br>
1:         1 100 100   |         10   0   0<br>
2:         0   0 100   |       9033  95  95<br>
3:         0   0 100   |        465   4 100<br>
<br>
read write<br>
offset                rpcs   % cum % |       rpcs   % cum %<br>
0:         1 100 100   |        725   7   7<br>
1:         0   0 100   |          0   0   7<br>
2:         0   0 100   |          0   0   7<br>
4:         0   0 100   |          0   0   7<br>
8:         0   0 100   |          0   0   7<br>
16:         0   0 100   |          0   0   7<br>
32:         0   0 100   |          0   0   7<br>
64:         0   0 100   |          0   0   7<br>
128:         0   0 100   |          0   0   7<br>
256:         0   0 100   |        718   7  15<br>
512:         0   0 100   |       1386  14  29<br>
1024:         0   0 100   |       2205  23  52<br>
2048:         0   0 100   |       1429  15  67<br>
4096:         0   0 100   |       1103  11  79<br>
8192:         0   0 100   |       1942  20 100</font><br>
</div>
<div style="font-family:verdana,sans-serif"><br>
</div>
<div style="font-family:verdana,sans-serif">Can anyone please explain to me why the RPCs in flight shown in the rpc_stats could be higher than the max_rpcs_in_flight?</div>
</div>
</div>
</div>
</div>
</blockquote>
<br>
</div>
<div>I do see a similar behavior with the statistics of my home system, which has the default osc.*.max_rpcs_in_flight=8, but shows many cases of 9 RPCs in flight in the statistics for both read and write, and in a few cases 10 or 11:</div>
<div><br>
</div>
<div><span style="white-space:pre-wrap"></span>read<span style="white-space:pre-wrap">
</span>write<br>
rpcs in flight        rpcs   % cum % |       rpcs   % cum %<br>
1:<span style="white-space:pre-wrap"> </span>       121   2   2   |      27831  93  93<br>
2:<span style="white-space:pre-wrap"> </span>        23   0   3   |        108   0  93<br>
3:<span style="white-space:pre-wrap"> </span>        22   0   3   |         19   0  93<br>
4:<span style="white-space:pre-wrap"> </span>        24   0   4   |         15   0  93<br>
5:<span style="white-space:pre-wrap"> </span>        19   0   5   |         10   0  93<br>
6:<span style="white-space:pre-wrap"> </span>        26   0   5   |         13   0  93<br>
7:<span style="white-space:pre-wrap"> </span>       176   4   9   |         39   0  93<br>
8:<span style="white-space:pre-wrap"> </span>       933  22  32   |         75   0  94<br>
9:<span style="white-space:pre-wrap"> </span>      2802  67  99   |       1207   4  98</div>
10:<span style="white-space:pre-wrap"> </span>        10   0 100   |        543   1  99<br>
11:<span style="white-space:pre-wrap"> </span>         0   0 100   |          1   0 100
<div><br>
</div>
<div><br>
</div>
<div>The good news is that Lustre is open source, so you can look into the lustre/osc code to see why this is happening.  The limit is set by cli->cl_max_rpcs_in_flight, and the stats are accounted by cli->cl_write_rpc_hist.</div>
<div><br>
</div>
<div>Out of curiosity, you don't say _why_ this off-by-one error is of interest?  Definitely it seems like a bug that could be fixed, but it doesn't seem too critical to correct functionality.<br>
<div><br>
<div>
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div>Cheers, Andreas</div>
<div>--</div>
<div>Andreas Dilger</div>
<div>Lustre Principal Architect</div>
<div>Whamcloud</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
</div>
<br>
<br>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>

</blockquote></div>