<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
On Aug 5, 2021, at 17:44, Nathan Dauchy - NOAA Affiliate via lustre-discuss <<a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a>> wrote:<br class="">
<div>
<blockquote type="cite" class=""><br class="Apple-interchange-newline">
<div class="">
<div dir="ltr" class="">
<div dir="ltr" class="">On Thu, Aug 5, 2021 at 3:23 PM Andreas Dilger <<a href="mailto:adilger@whamcloud.com" class="">adilger@whamcloud.com</a>> wrote:<br class="">
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div style="overflow-wrap: break-word;" class="">On Aug 5, 2021, at 13:29, Nathan Dauchy wrote:<br class="">
<div class="">
<blockquote type="cite" class="">
<div class="">
<div dir="ltr" class="">
<div class="">Andreas, thanks as always for your insight. Comments inline...</div>
<br class="">
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Thu, Aug 5, 2021 at 10:48 AM Andreas Dilger <<a href="mailto:adilger@whamcloud.com" target="_blank" class="">adilger@whamcloud.com</a>> wrote:<br class="">
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div class="">On Aug 5, 2021, at 09:28, Nathan Dauchy via lustre-discuss <<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank" class="">lustre-discuss@lists.lustre.org</a>> wrote:<br class="">
<div class="">
<blockquote type="cite" class="">
<div class="">
<div dir="ltr" class="">
<div class="">Question: Is it possible that a flash journal device on an ext4 filesystem can reach a point where there are not enough clean blocks to write to, and they can suffer from very degraded write performance?<br class="">
</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
<div class="">... </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div style="overflow-wrap: break-word;" class="">
<div class="">
<blockquote type="cite" class="">
<div class="">
<div dir="ltr" class="">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div class="">
<div class="">
<blockquote type="cite" class="">
<div class="">
<div dir="ltr" class="">
<div class=""></div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div class="">
<div class="">
<blockquote type="cite" class="">
<div class="">
<div dir="ltr" class="">
<div class="">Another related question would be how to benchmark the journal device on it's own, particularly write performance, without losing data on an existing file system; similar to the very useful obdfilter-survey tool, but at a lower level. But I am
primarily looking to understand the nuances of flash devices and ldiskfs external journals a bit better.</div>
</div>
</div>
</blockquote>
</div>
<div class="">While the external journal device has an ext4 superblock header for identification (UUID/label), and a feature flag that prevents it from being mounted/used directly, it is not really an ext4 filesystem, just a flat "file". You'd need to remove
it from the main ext4/ldiskfs filesystem, reformat it as ext4 and mount locally, and then run benchmarks (e.g. "dd" would best match the JBD2 workload, or fio if you want random IOPS) against it. You could do this before/after trim (could use fstrim at this
point) to see if it affects the performance or not.</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
<div class="">OK, thanks for confirming that there is no magic ext4 journal benchmarking tool. I'll stop searching. ;-)</div>
</div>
</div>
</div>
</blockquote>
</div>
<div class="">Note that there *are* some journal commit statistics - /proc/fs/jbd2/<dev>/info that you might be able to compare between devices. Probably the most interesting is "average transaction commit time", which is how long it takes to write the blocks
to the journal device after the transaction starts to commit.</div>
<div class="">
<div dir="auto" style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class="">
<div dir="auto" style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class="">
<div dir="auto" style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class="">
<div dir="auto" style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class="">
<div dir="auto" style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class="">
<div dir="auto" style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class="">
<div class=""><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
<div class="">Oh, that is interesting!</div>
<div class=""><br class="">
</div>
<div class="">The "average transaction commit time" seems to fluctuate, possibly with load, and doesn't have an obvious correlation to the slower OSTs. But perhaps I'll look at it again when running a clean benchmark during a future dedicated time.</div>
<div class=""><br class="">
</div>
<div class="">I _did_ find a stark difference in other metrics though:</div>
<div class=""><br class="">
</div>
<div class=""># pdsh -g oss "grep 'handles per' /proc/fs/jbd2/md*/info" | sort<br class="">
lfs4n04: 17 handles per transaction<br class="">
lfs4n05: 18 handles per transaction<br class="">
lfs4n06: 18 handles per transaction<br class="">
lfs4n07: 18 handles per transaction<br class="">
lfs4n08: 17 handles per transaction<br class="">
lfs4n09: 17 handles per transaction<br class="">
lfs4n10: 18 handles per transaction<br class="">
lfs4n11: 18 handles per transaction<br class="">
lfs4n12: 18 handles per transaction<br class="">
lfs4n13: 17 handles per transaction<br class="">
lfs4n16: 192 handles per transaction<br class="">
lfs4n17: 178 handles per transaction<br class="">
lfs4n18: 198 handles per transaction<br class="">
lfs4n19: 192 handles per transaction<br class="">
# pdsh -g oss "grep 'logged blocks per' /proc/fs/jbd2/md*/info" | sort<br class="">
lfs4n04: 24 logged blocks per transaction<br class="">
lfs4n05: 24 logged blocks per transaction<br class="">
lfs4n06: 25 logged blocks per transaction<br class="">
lfs4n07: 25 logged blocks per transaction<br class="">
lfs4n08: 24 logged blocks per transaction<br class="">
lfs4n09: 24 logged blocks per transaction<br class="">
lfs4n10: 25 logged blocks per transaction<br class="">
lfs4n11: 24 logged blocks per transaction<br class="">
lfs4n12: 24 logged blocks per transaction<br class="">
lfs4n13: 24 logged blocks per transaction<br class="">
lfs4n16: 103 logged blocks per transaction<br class="">
lfs4n17: 98 logged blocks per transaction<br class="">
lfs4n18: 106 logged blocks per transaction<br class="">
lfs4n19: 103 logged blocks per transaction<br class="">
</div>
<div class=""><br class="">
</div>
<div class="">The last 4 nodes are the expansion OSTs which are performing better. What does that difference indicate?</div>
</div>
</div>
</div>
</blockquote>
<br class="">
</div>
<div>I don't think you can read too much into those numbers. "handles per transaction" usually depend on the IO load, due to more threads joining a single transaction before it commits due to sync. The "blocks per transaction" is strongly correlated, in this
case it looks like only 1 or 2 blocks per handle (maybe random 4KB writes, with occasional metadata updates?)</div>
<div><br class="">
</div>
<div>On the OSTs the transactions are usually forced to commit quickly to allow clients to stop pinning their dirty pages in cache. The pages are kept in case an RPC resend is needed, but can be dropped as soon as they commit to storage on the server. On
the MDTs there is less memory pressure, so transaction commits are often timed out (5s) to aggregate as many changes as possible.</div>
<div><br class="">
</div>
<div>If the new OSTs are less full than the old ones (maybe 4x more free space?) then they would naturally get more objects allocated due to OST space balance (about the 4x seen here). That would lead to 4x more blocks per commit, which would make it appear
that the new OSTs are doing better, but really this is just more aggregation of the very small write sizes.</div>
<br class="">
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div>Cheers, Andreas</div>
<div>--</div>
<div>Andreas Dilger</div>
<div>Lustre Principal Architect</div>
<div>Whamcloud</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
<br class="Apple-interchange-newline">
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br class="">
</body>
</html>