<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
</head>
<body dir="ltr">
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif;" dir="ltr">
echo 1 > drop_caches does not generate memory pressure - it requests that the page cache be cleared. It would not be expected to affect slab caches much.
<div><br>
</div>
<div>You could try 3 (1+2 in this case, where 2 is inode and dentry). That might do a bit more because some (maybe many?) of those objects you're looking at would go away if the associated inodes or dentries were removed. But fundamentally, drop caches does
not generate memory pressure, and does not force reclaim. It drops specific, identified caches.</div>
<div><br>
</div>
<div>The only way to force *reclaim* is memory pressure.</div>
<div><br>
</div>
<div>Your note that a lot more memory than expected was freed under pressure does tell us something, though.</div>
<div><br>
</div>
<div><span style="font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 16px;">It's conceivable Lustre needs to set SLAB_RECLAIM_ACCOUNT on
more of its slab caches, so this piqued my curiosity. My conclusion is no, here's why:</span><br>
</div>
<div><br>
The one quality reference I was quickly able to find suggests setting SLAB_RECLAIM_ACCOUNT wouldn't be so simple:<br>
<a href="https://lwn.net/Articles/713076/" class="OWAAutoLink" id="LPlnk131494" previewremoved="true">https://lwn.net/Articles/713076/</a><br>
<br>
GFP_TEMPORARY is - in practice - just another name for __GFP_RECLAIMABLE, and setting SLAB_RECLAIM_ACCOUNT is equivalent to setting __GFP_RECLAIMABLE. That article suggests caution is needed, as this should only be used for memory that is certain to be easily
available, because using this flag changes the allocation behavior on the assumption that the memory can be quickly freed at need. That is often not true of these Lustre objects.<br>
<br>
An easy way to learn more about this sort of question is to compare to other actively developed file systems in the kernel...</div>
<div><br>
Comparing to other file systems, we see that in general, only the inode cache is allocated with SLAB_RECLAIM_ACCOUNT (it varies a bit).</div>
<div><br>
</div>
<div>XFS, for example, has only one use of KM_ZONE_RECLAIM, its name for this flag - the inode cache:<br>
"
<div> xfs_inode_zone =</div>
<div> kmem_zone_init_flags(sizeof(xfs_inode_t), "xfs_inode",</div>
<div> KM_ZONE_HWALIGN | KM_ZONE_RECLAIM | KM_ZONE_SPREAD,</div>
<div> xfs_fs_inode_init_once);</div>
<div>"</div>
<div><br>
</div>
<div>btrfs is the same, just the inode cache. EXT4 has a *few* more caches marked this way, but not everything.</div>
<div><br>
</div>
<div>So, no - I don't think so. It would be atypical for Lustre to set SLAB_RECLAIM_ACCOUNT on its slab caches for internal objects. Presumably this sort of thing is not considered reclaimable enough for this purpose.</div>
<div><br>
</div>
<div>I believe if you tried similar tests with other complex file systems (XFS might be a good start), you'd see broadly similar behavior. (Lustre is probably a bit worse because it has a more complex internal object model, so more slab caches.)</div>
<div><br>
</div>
<div>VM accounting is distinctly imperfect. The design is such that it's often impossible to know how much memory could be made available without actually going and trying to free it. There are good, intrinsic reasons for some of that, and some of that is
design artifacts...</div>
<div><br>
</div>
<div>I've copied in Neil Brown, who I think only reads lustre-devel, just in case he has some particular input on this.</div>
</div>
<div>
<div><br>
</div>
<div>Regards,</div>
<div>- Patrick</div>
</div>
</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> lustre-discuss <lustre-discuss-bounces@lists.lustre.org> on behalf of Jacek Tomaka <jacekt@dug.com><br>
<b>Sent:</b> Sunday, April 14, 2019 3:12:51 AM<br>
<b>To:</b> lustre-discuss@lists.lustre.org<br>
<b>Subject:</b> Re: [lustre-discuss] Lustre client memory and MemoryAvailable</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div dir="ltr">Actually i think it is just a bug with the way slab caches are created. Some of them should be passed a flag that they are reclaimable. </div>
<div>i.e. something like: </div>
<div dir="ltr"><a href="https://patchwork.kernel.org/patch/9360819/">https://patchwork.kernel.org/patch/9360819/</a></div>
<div dir="ltr"><br>
</div>
<div>Regards.</div>
<div>Jacek Tomaka</div>
</div>
<br>
<div class="x_gmail_quote">
<div dir="ltr" class="x_gmail_attr">On Sun, Apr 14, 2019 at 3:27 PM Jacek Tomaka <<a href="mailto:jacekt@dug.com">jacekt@dug.com</a>> wrote:<br>
</div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">Hello, </div>
<div dir="ltr"><br>
<div>TL;DR;</div>
<div>Is there a way to figure out how much memory Lustre will make available under memory pressure?</div>
<div><br>
</div>
<div>Details: </div>
<div>We are running lustre client on a machine with 128GB of memory (Centos 7) Intel Phi KNL machines and at certain situations we see that there can be ~10GB+ of memory allocated on the kernel side i.e. : </div>
<div><br>
</div>
<div>vvp_object_kmem 3535336 3536986 176 46 2 : tunables 0 0 0 : slabdata 76891 76891 0</div>
<div>ll_thread_kmem 33511 33511 344 47 4 : tunables 0 0 0 : slabdata 713 713 0</div>
<div>lov_session_kmem 34760 34760 592 55 8 : tunables 0 0 0 : slabdata 632 632 0</div>
<div>osc_extent_kmem 3549831 3551232 168 48 2 : tunables 0 0 0 : slabdata 73984 73984 0</div>
<div>osc_thread_kmem 14012 14116 2832 11 8 : tunables 0 0 0 : slabdata 1286 1286 0</div>
<div>osc_object_kmem 3546640 3548350 304 53 4 : tunables 0 0 0 : slabdata 66950 66950 0</div>
<div>signal_cache 3702537 3707144 1152 28 8 : tunables 0 0 0 : slabdata 132398 132398 0</div>
<div><br>
</div>
<div>/proc/meminfo: </div>
<div>
<div>MemAvailable: 114196044 kB</div>
<div>Slab: 11641808 kB</div>
<div>SReclaimable: 1410732 kB</div>
<div>SUnreclaim: 10231076 kB</div>
</div>
<div><br>
</div>
<div>After executing </div>
<div>
<pre class="x_gmail-m_-3651746023953936964gmail-code-java" style="margin-top:0px; margin-bottom:0px; padding:0px; max-height:30em; overflow:auto; white-space:pre-wrap; word-wrap:normal; color:rgb(51,51,51); font-size:12px; background-color:rgb(245,245,245)">echo 1 >/proc/sys/vm/drop_caches</pre>
<div>the slabinfo values don't change but when i actually generate memory pressure by: </div>
<div>
<pre class="x_gmail-m_-3651746023953936964gmail-code-java" style="margin-top:0px; margin-bottom:0px; padding:0px; max-height:30em; overflow:auto; white-space:pre-wrap; word-wrap:normal; color:rgb(51,51,51); font-size:12px; background-color:rgb(245,245,245)">java -Xmx117G -Xms117G -XX:+AlwaysPreTouch -version</pre>
</div>
<div><br>
</div>
<div>lots of memory gets freed: </div>
<div>
<div>vvp_object_kmem 127650 127880 176 46 2 : tunables 0 0 0 : slabdata 2780 2780 0</div>
<div>ll_thread_kmem 33558 33558 344 47 4 : tunables 0 0 0 : slabdata 714 714 0</div>
<div>lov_session_kmem 34815 34815 592 55 8 : tunables 0 0 0 : slabdata 633 633 0</div>
<div>osc_extent_kmem 128640 128880 168 48 2 : tunables 0 0 0 : slabdata 2685 2685 0</div>
<div>osc_thread_kmem 14038 14116 2832 11 8 : tunables 0 0 0 : slabdata 1286 1286 0</div>
<div>osc_object_kmem 82998 83263 304 53 4 : tunables 0 0 0 : slabdata 1571 1571 0</div>
<div>signal_cache 38734 44268 1152 28 8 : tunables 0 0 0 : slabdata 1581 1581 0</div>
</div>
<div><br>
</div>
<div>/proc/meminfo: </div>
<div>
<div>MemAvailable: 123146076 kB</div>
<div>Slab: 1959160 kB</div>
<div>SReclaimable: 334276 kB</div>
<div>SUnreclaim: 1624884 kB</div>
</div>
<div><br>
</div>
<div>The similar effect to generating memory pressure we see when executing: </div>
<div>
<pre class="x_gmail-m_-3651746023953936964gmail-code-java" style="margin-top:0px; margin-bottom:0px; padding:0px; max-height:30em; overflow:auto; white-space:pre-wrap; word-wrap:normal; color:rgb(51,51,51); font-size:12px; background-color:rgb(245,245,245)">echo 3 >/proc/sys/vm/drop_caches</pre>
</div>
<div>But this can take very long time (10 minutes). </div>
<div><br>
</div>
<div>So essentially on a machine using Lustre client, MemAvailable is no longer a good predictor of the amount of memory that can be allocated.</div>
<div>Is there a way to query Lustre and compensate for lustre cache memory that will be made available on memory pressure?</div>
<div><br>
</div>
<div>Regards.</div>
-- <br>
<div dir="ltr" class="x_gmail-m_-3651746023953936964gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr"><span>
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div><font color="#000000"><font face="arial,helvetica,sans-serif"><b>Jacek Tomaka</b></font></font><br>
<font color="#000000"><font face="arial,helvetica,sans-serif"><font size="2">Geophysical Software Developer</font></font></font><br>
</div>
<font color="#000000" face="arial,helvetica,sans-serif"></font>
<div><span lang="EN-US"></span><span lang="EN-US"><b><br>
<br>
</b></span></div>
<font color="#000000" face="arial,helvetica,sans-serif"></font><img src="http://drive.google.com/uc?export=view&id=0B4X9ixpc-ZU_NHV0WnluaXp5ZkE"><br>
<br>
<span style="color:rgb(102,102,102)"><font size="2"><b>DownUnder GeoSolutions<br>
<br>
</b></font></span>
<div><span style="color:rgb(102,102,102)"></span><span style="color:rgb(102,102,102)">76 Kings Park Road<br>
</span></div>
<span style="color:rgb(102,102,102)">West Perth 6005 WA, Australia<br>
<i><b>tel </b></i><a href="tel:+61%208%209287%204143" value="+61892874143" target="_blank">+61 8 9287 4143</a><br>
<a href="mailto:jacekt@dug.com" target="_blank">jacekt@dug.com</a><br>
<b><a href="http://www.dug.com" target="_blank">www.dug.com</a></b></span></div>
</div>
</div>
</div>
</div>
</div>
</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr" class="x_gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr"><span>
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div><font color="#000000"><font face="arial,helvetica,sans-serif"><b>Jacek Tomaka</b></font></font><br>
<font color="#000000"><font face="arial,helvetica,sans-serif"><font size="2">Geophysical Software Developer</font></font></font><br>
</div>
<font color="#000000" face="arial,helvetica,sans-serif"></font>
<div><span lang="EN-US"></span><span lang="EN-US"><b><br>
<br>
</b></span></div>
<font color="#000000" face="arial,helvetica,sans-serif"></font><img src="http://drive.google.com/uc?export=view&id=0B4X9ixpc-ZU_NHV0WnluaXp5ZkE"><br>
<br>
<span style="color:rgb(102,102,102)"><font size="2"><b>DownUnder GeoSolutions<br>
<br>
</b></font></span>
<div><span style="color:rgb(102,102,102)"></span><span style="color:rgb(102,102,102)">76 Kings Park Road<br>
</span></div>
<span style="color:rgb(102,102,102)">West Perth 6005 WA, Australia<br>
<i><b>tel </b></i><a href="tel:+61%208%209287%204143" value="+61892874143" target="_blank">+61 8 9287 4143</a><br>
<a href="mailto:jacekt@dug.com" target="_blank">jacekt@dug.com</a><br>
<b><a href="http://www.dug.com" target="_blank">www.dug.com</a></b></span></div>
</div>
</div>
</div>
</div>
</div>
</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>