<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<div class="">Hi all,</div>
<div class=""><br class="">
</div>
<div class="">We’re using jobstats under SLURM and have pulled together a tool to integrate SLURM job info and lustre OST/MDT jobstats. The idea is to correlate filesystem use cases with particular applications as targets for refactoring.</div>
<div class=""><br class="">
</div>
<div class="">In doing so, I’m seeing some applications really trigger getxattr on the MDT, and others do not. A particular egregious example is below: 360 cores, ~10s of GB output, ~6500 files, but 16,608,476 calls to getxattr during a 4 hour runtime. And
this is a nominally compute-bound problem, so that I/O pattern is likely compressed into small windows of time.</div>
<div class=""><br class="">
</div>
<div class="">The system is CentOS 7.5 / lustre 2.10.5 / zfs-0.7.9, single mdt, 12 OSS, 2 OST each. Default stripe count of 4.</div>
<div class=""><br class="">
</div>
<div class="">A couple questions:</div>
<div class=""><br class="">
</div>
<div class="">1) should I care about this? We do see sporadic mdt slowness under zfs, but that doesn’t seem rare. I’m looking for a good way to trace that to jobs / use cases.</div>
<div class="">2) what types of operations might be triggering the getxattr usage on a moderate amount of files (e.g. what to watch for in the refactoring process…)</div>
<div class=""><br class="">
</div>
<div class="">Thanks,</div>
<div class=""><br class="">
</div>
-Ben
<div class=""><br class="">
</div>
<div class="">--------------------------<br class="">
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>….</div>
</div>
<div class=""><font face="Courier" class=""><span class="Apple-tab-span" style="white-space:pre"></span>TRES : cpu=360,node=30,billing=360<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>RunTime : 04:59:14<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>GroupId : eg3(3000)<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>ExitCode : 0:0<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>MDT:rename : 373<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>MDT:snapshot_time : 2018-09-21 08:36:29<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>MDT:setattr : 444<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>MDT:mkdir : 361<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>MDT:getattr : 1570<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>MDT:getxattr : 16608476<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>MDT:mknod : 265<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>MDT:rmdir : 1<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>MDT:samedir_rename : 373<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>MDT:close : 6331<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>MDT:unlink : 113<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>MDT:open : 6345<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>OST0009:write_bytes : 3.46 GB<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>OST0008:write_bytes : 3.11 GB<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>OST0001:write_bytes : 1.01 GB<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>OST0000:write_bytes : 396.19 MB<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>OST0005:read_bytes : 8.19 KB<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>OST0005:write_bytes : 2.38 GB<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>OST0005:setattr : 1<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>OST0004:write_bytes : 790.65 MB<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>OST0007:write_bytes : 3.02 GB<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>OST0006:write_bytes : 817.14 MB<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>OST0016:write_bytes : 4.57 GB<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>OST0017:write_bytes : 5.15 GB<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>OST0017:setattr : 1<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>OST0014:write_bytes : 8.8 GB<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>OST0015:write_bytes : 1.37 GB<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>OST0012:write_bytes : 7 GB<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>OST0012:setattr : 1<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>OST0013:read_bytes : 8.39 MB<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>OST0013:write_bytes : 8.4 GB<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>OST0013:setattr : 1<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>OST0010:write_bytes : 1.98 GB<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>OST0011:read_bytes : 27.28 MB<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>OST0011:write_bytes : 9.42 GB<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>OST000c:read_bytes : 131.07 KB<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>OST000c:write_bytes : 5.83 GB<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>OST000c:setattr : 2<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>OST000b:read_bytes : 28.12 MB<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>OST000b:write_bytes : 4.23 GB<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>OST000e:read_bytes : 8.02 MB<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>OST000e:write_bytes : 7.48 GB<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>OST000e:setattr : 1<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>OST000d:write_bytes : 1.21 GB<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>OST000f:write_bytes : 2.88 GB<br class="">
<br class="">
</font></div>
<div class=""></div>
<div class=""><br class="">
</div>
</body>
</html>