<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div class="">Ake,</div>
<div class="">in this particular case I can answer your question in detail.</div>
<div class=""><br class="">
</div>
Before SFAOS 12.1 (IIRC) the /sys/block/*/queue/rotational setting is set from userspace at mount time via a udev script, and the Lustre detection of "rotational=0" could be racy. Newer versions of SFAOS (12.1+) set the rotational state in the SCSI VPD page
and this is detected directly by the kernel.
<div class=""><br class="">
</div>
<div class="">For EXAScaler systems that may be running older SFAOS releases, there was a patch made (included in <span class="com-google-gerrit-client-change-IncludedInBox_BinderImpl_GenCss_style-includedInElement">2.12.6-ddn72/EXA5.2.5) that revalidates the
rotational device state occasionally in case it has been modified after mount time, and uses that to update the read_cache_enable and writethrough_cache_enable tunables *if they have not been explicitly set*.</span></div>
<div class=""><br class="">
</div>
<div class="">Until you update to a newer EXA and/or SFAOS, you can explicitly tune osd-ldiskfs.*.read_cache_enable=0 and ...writethrough_cache_enable=0, using a wildcard "*" if all of the OSTs/MDTs are flash based. If you have a hybrid NVMe/HDD system, you
can explicitly select a subset of OST/MDT devices to disable the caches.</div>
<div class=""><br class="">
</div>
<div class="">Cheers, Andreas<br class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On May 20, 2022, at 02:49, Åke Sandgren <<a href="mailto:ake.sandgren@hpc2n.umu.se" class="">ake.sandgren@hpc2n.umu.se</a>> wrote:</div>
<div class="">
<div class="">On 5/20/22 09:53, Andreas Dilger via lustre-discuss wrote:<br class="">
<blockquote type="cite" class="">To elaborate a bit on Patrick's answer, there is no mechanism to do this on the *client*, because the performance difference between client RAM and server storage is still fairly significant, especially if the application is
doing sub-page read or write operations.<br class="">
However, on the *server* the OSS and MDS will *not* put flash storage into the page cache, because using the kernel page cache has a measurable overhead, and (at least in our testing) the performance of NVMe IOPS is actually better *without* the page cache
because more CPU is available to handle RPCs. This is controlled on the server with osd-ldiskfs.*.{read_cache_enable,writethrough_cache_enable}, default to 0 if the block device is non-rotational, default to 1 if block device is rotational.<br class="">
</blockquote>
<br class="">
Then my question is, what is it checking to determine non-rotational?<br class="">
<br class="">
On our systems the NVMe disks have read/writethrough_cache_enable = 1 (DDN SFA400NVXE) with<br class="">
===<br class="">
/dev/sde on /lustre/stor10/ost0000 (NVMe)<br class="">
cat /sys/block/sde/queue/rotational<br class="">
0<br class="">
lctl get_param osd-ldiskfs.*.*cache*enable<br class="">
osd-ldiskfs.stor10-OST0000.read_cache_enable=1<br class="">
osd-ldiskfs.stor10-OST0000.writethrough_cache_enable=1<br class="">
<br class="">
EXAScaler SFA CentOS 5.2.3-r5<br class="">
kmod-lustre-2.12.6_ddn58-1.el7.x86_64<br class="">
===<br class="">
<br class="">
-- <br class="">
Ake Sandgren, HPC2N, Umea University, S-90187 Umea, Sweden<br class="">
Internet: <a href="mailto:ake@hpc2n.umu.se" class="">ake@hpc2n.umu.se</a> Mobile: +46 70 7716134 Fax: +46 90-580 14<br class="">
WWW: <a href="http://www.hpc2n.umu.se" class="">http://www.hpc2n.umu.se</a><br class="">
_______________________________________________<br class="">
lustre-discuss mailing list<br class="">
<a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a><br class="">
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org<br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div>Cheers, Andreas</div>
<div>--</div>
<div>Andreas Dilger</div>
<div>Lustre Principal Architect</div>
<div>Whamcloud</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
<br class="Apple-interchange-newline">
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br class="">
</div>
</body>
</html>