<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
On Feb 4, 2020, at 07:23, Åke Sandgren <<a href="mailto:ake.sandgren@hpc2n.umu.se" class="">ake.sandgren@hpc2n.umu.se</a>> wrote:<br class="">
<div>
<blockquote type="cite" class=""><br class="">
<div class="">
<div class="">When I create a large number of files on an OST and then remove them,<br class="">
the used inode count on the OST decreases very slowly, it takes several<br class="">
hours for it to go from 3M to the correct ~10k.<br class="">
<br class="">
(I'm running the io500 test suite)<br class="">
<br class="">
Is there something I can do to make it release them faster?<br class="">
Right now it has gone from 3M to 1.5M in 6 hours, (lfs df -i).<br class="">
</div>
</div>
</blockquote>
<div><br class="">
</div>
It this the object count or the file count? Are you possibly using a lot of</div>
<div>stripes on the files being deleted that is multiplying the work needed?</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">
<div class="">These are SSD based OST's in case it matters.<br class="">
</div>
</div>
</blockquote>
<br class="">
</div>
<div>The MDS controls the destroy of the OST objects, so there is a rate</div>
<div>limit, but ~700/s seems low to me, especially for SSD OSTs.</div>
<div><br class="">
</div>
<div>You could check "lctl get_param osp.*.sync*" on the MDS to see how</div>
<div>many destroys are pending. Also, increasing osp.*.max_rpcs_in_flight</div>
<div>on the MDS might speed this up? It should default to 32 per OST on</div>
<div>the MDS vs. default 8 for clients</div>
<br class="">
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div>Cheers, Andreas</div>
<div>--</div>
<div>Andreas Dilger</div>
<div>Principal Lustre Architect</div>
<div>Whamcloud</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br class="">
</body>
</html>