<div dir="ltr">Anything I can try on this issue Andreas ?</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jun 30, 2022 at 10:46 AM Karan Singh <<a href="mailto:karanveersingh5623@gmail.com">karanveersingh5623@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">I changed the below param , just to check if any differences seen , unfortunately nothing ...all same<div><br><div># lctl set_param ost.OSS.ost.threads_max=1024<br></div></div><div># lctl set_param ost.OSS.ost.threads_min=512</div><div><br></div><div><img src="cid:ii_l50d80610" alt="image.png" width="562" height="205"><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jun 29, 2022 at 2:24 PM Karan Singh <<a href="mailto:karanveersingh5623@gmail.com" target="_blank">karanveersingh5623@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">hahahha...its just a typo...just testing both in order to understand the IOs performance . <div><br></div><div><span style="color:rgb(23,43,77);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen,Ubuntu,"Fira Sans","Droid Sans","Helvetica Neue",sans-serif;font-size:14px;background-color:rgb(183,183,183)">mpirun -hostfile nodefile --map-by node -np 16 /usr/local/bin/ior -a POSIX --posix.odirect -v -i 50 -g -F -e -k -o /mnt/lustre/test.ior -r -t ${xfersize} -b ${blksize} -O summaryFile=./${TESTDIR}/</span><span style="color:rgb(23,43,77);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen,Ubuntu,"Fira Sans","Droid Sans","Helvetica Neue",sans-serif;font-size:14px;background-color:rgb(183,183,183)">iorresult_SeqWrite_p${</span><span style="color:rgb(23,43,77);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen,Ubuntu,"Fira Sans","Droid Sans","Helvetica Neue",sans-serif;font-size:14px;background-color:rgb(183,183,183)">numthreads}_bs${blksize}_tf${</span><span style="color:rgb(23,43,77);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen,Ubuntu,"Fira Sans","Droid Sans","Helvetica Neue",sans-serif;font-size:14px;background-color:rgb(183,183,183)">xfersize}.json,summaryFormat=</span><span style="color:rgb(23,43,77);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen,Ubuntu,"Fira Sans","Droid Sans","Helvetica Neue",sans-serif;font-size:14px;background-color:rgb(183,183,183)">JSON</span><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jun 29, 2022 at 1:22 PM Andreas Dilger <<a href="mailto:adilger@whamcloud.com" target="_blank">adilger@whamcloud.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">



<div>
On Jun 28, 2022, at 21:51, Karan Singh via lustre-discuss <<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a>> wrote:<br>
<div>
<blockquote type="cite"><br>
<div>
<div dir="ltr">Hi team
<div><br>
</div>
<div>Below are the details :</div>
<div>Using 40 lustre docker clients running on 4 x Dell R750 with each lustre docker client running the below mentioned command </div>
<div><br>
</div>
<div><span style="color:rgb(23,43,77);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen,Ubuntu,"Fira Sans","Droid Sans","Helvetica Neue",sans-serif;font-size:14px;background-color:rgb(183,183,183)">${xfersize} = </span><span style="background-color:rgb(183,183,183);color:rgb(23,43,77);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen,Ubuntu,"Fira Sans","Droid Sans","Helvetica Neue",sans-serif;font-size:14px">${blksize}</span><span style="background-color:rgb(183,183,183);color:rgb(23,43,77);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen,Ubuntu,"Fira Sans","Droid Sans","Helvetica Neue",sans-serif;font-size:14px"> </span></div>
<div><span style="background-color:rgb(183,183,183);color:rgb(23,43,77);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen,Ubuntu,"Fira Sans","Droid Sans","Helvetica Neue",sans-serif;font-size:14px">${blksize} = </span><span style="color:rgb(23,43,77);font-family:Roboto,RobotoDraft,Helvetica,Arial,sans-serif;font-size:14px">128M,
 64M, 32M, 16M, 8M, 4M, 2M, 1M</span><br>
</div>
<div><span style="color:rgb(23,43,77);font-family:Roboto,RobotoDraft,Helvetica,Arial,sans-serif;font-size:14px"><br>
</span></div>
<div><span style="color:rgb(23,43,77);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen,Ubuntu,"Fira Sans","Droid Sans","Helvetica Neue",sans-serif;font-size:14px;background-color:rgb(183,183,183)">mpirun -hostfile nodefile
 --map-by node -np 16 /usr/local/bin/ior -a POSIX --posix.odirect -v -i 50 -g -F -e -k -o /mnt/beegfs/test.ior</span></div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>I think the problem is pretty clear right here - you are using BeeGFS for your testing, not Lustre...  :-)</div>
<br>
<blockquote type="cite">
<div>
<div dir="ltr">
<div><span style="color:rgb(23,43,77);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen,Ubuntu,"Fira Sans","Droid Sans","Helvetica Neue",sans-serif;font-size:14px;background-color:rgb(183,183,183)">-r -t ${xfersize} -b
 ${blksize} -O summaryFile=./${TESTDIR}/</span><span style="color:rgb(23,43,77);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen,Ubuntu,"Fira Sans","Droid Sans","Helvetica Neue",sans-serif;font-size:14px;background-color:rgb(183,183,183)">iorresult_SeqWrite_p${</span><span style="color:rgb(23,43,77);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen,Ubuntu,"Fira Sans","Droid Sans","Helvetica Neue",sans-serif;font-size:14px;background-color:rgb(183,183,183)">numthreads}_bs${blksize}_tf${</span><span style="color:rgb(23,43,77);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen,Ubuntu,"Fira Sans","Droid Sans","Helvetica Neue",sans-serif;font-size:14px;background-color:rgb(183,183,183)">xfersize}.json,summaryFormat=</span><span style="color:rgb(23,43,77);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen,Ubuntu,"Fira Sans","Droid Sans","Helvetica Neue",sans-serif;font-size:14px;background-color:rgb(183,183,183)">JSON</span><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><span id="gmail-m_-3424454879616850837gmail-m_2667672281349364185gmail-m_-5961383112123020492cid:ii_l4z1z8dt0"><image.png></span><br>
</div>
<div><br>
</div>
<div>please let me know what is the bottleneck in the above setup ?</div>
<div>Also if you need more info , will be provide it right away .</div>
</div>
</div>
</blockquote>
</div>
<br>
<div>
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div>Cheers, Andreas</div>
<div>--</div>
<div>Andreas Dilger</div>
<div>Lustre Principal Architect</div>
<div>Whamcloud</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
</div>
<br>
<br>
</div>
<br>
</div>

</blockquote></div>
</blockquote></div>
</blockquote></div>