<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body>
1. Good to know, thank you. I hadnt looked at the code, I was unaware it runs through all the sprinklers.<br>
<br>
2. Right, I know - the article was about it when it was an alias for reclaimable, and hence describes some of the behavior of reclaimable.<br>
<br>
3. Interesting, thats good to know. I would note that it doesnt seem to be standard practice in other file systems, though I didnt look at how many shrinkers theyre registering. Perhaps having special shrinkers is whats unusual.<br>
<br>
BTW, re: mailing list, this is the first devel appropriate thing Ive seen on discuss in a long while. I should instead have encouraged Jacek to use lustre-devel :)<br>
<br>
- Patrick
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> NeilBrown <neilb@suse.com><br>
<b>Sent:</b> Sunday, April 14, 2019 6:38:47 PM<br>
<b>To:</b> Patrick Farrell; Jacek Tomaka; lustre-discuss@lists.lustre.org<br>
<b>Subject:</b> Re: [lustre-discuss] Lustre client memory and MemoryAvailable</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt;">
<div class="PlainText"><br>
(that for the Cc Patrick - maybe I should subscribe to<br>
lustre-discuss...)<br>
<br>
1/ w.r.t drop_caches, "2" is *not* "inode and dentry". The '2' bit<br>
causes all registered shrinkers to be run, until they report there is<br>
nothing left that can be discarded. If this is taking 10 minutes,<br>
then it seems likely that some shrinker is either very inefficient, or<br>
is reporting that there is more work to be done, when really there<br>
isn't.<br>
<br>
lustre registers 5 shrinkers. Any memcache which is not affected by<br>
those shrinkers should *not* be marked SLAB_RECLAIM_ACCOUNT (unless<br>
they are indirectly shrunk by a system shrinker - e.g. if they are<br>
slaves to the icache or dcache). Any which are probably can be.<br>
<br>
1a/ "echo 3 > drop_caches" does the easy part of memory reclaim: it<br>
reclaims anything that can be reclaimed immediately. It doesn't<br>
trigger write-back, and it doesn't start the oom-killer, but all<br>
caches are flushed of everything that is not currently in use, and<br>
does not need to be written-out first. If you run "sync" first,<br>
there should be nothing to write out, so it should drop a lot more.<br>
<br>
2/ GFP_TEMPORARY is gone, it was never really well defined. Best to<br>
ignore it.<br>
<br>
3/ I don't *think* __GFP_RECLAIMABLE has a very big effect. It<br>
primarily tries to keep non-reclaimable allocations together so they<br>
don't cause too much fragmentation. To do this, it groups them<br>
separately from reclaimable allocations. So if a shrinker is<br>
expected to do anything useful, then it makes sense to tag the<br>
related slabs as RECLAIMABLE.<br>
<br>
4/ Patrick is right that accounting is best-effort. But we do want it<br>
to improve. Just last week there was a report<br>
<a href="https://lwn.net/SubscriberLink/784964/9ddad7d7050729e1/">https://lwn.net/SubscriberLink/784964/9ddad7d7050729e1/</a><br>
about making slab-allocated objects movable. If/when that gets off<br>
the ground, it should help the fragmentation problem, so more of the<br>
pages listed as reclaimable should actually be so.<br>
<br>
NeilBrown<br>
<br>
On Sun, Apr 14 2019, Patrick Farrell wrote:<br>
<br>
> echo 1 > drop_caches does not generate memory pressure - it requests that the page cache be cleared. It would not be expected to affect slab caches much.<br>
><br>
> You could try 3 (1+2 in this case, where 2 is inode and dentry). That might do a bit more because some (maybe many?) of those objects you're looking at would go away if the associated inodes or dentries were removed. But fundamentally, drop caches does
not generate memory pressure, and does not force reclaim. It drops specific, identified caches.<br>
><br>
> The only way to force *reclaim* is memory pressure.<br>
><br>
> Your note that a lot more memory than expected was freed under pressure does tell us something, though.<br>
><br>
> It's conceivable Lustre needs to set SLAB_RECLAIM_ACCOUNT on more of its slab caches, so this piqued my curiosity. My conclusion is no, here's why:<br>
><br>
> The one quality reference I was quickly able to find suggests setting SLAB_RECLAIM_ACCOUNT wouldn't be so simple:<br>
> <a href="https://lwn.net/Articles/713076/">https://lwn.net/Articles/713076/</a><br>
><br>
> GFP_TEMPORARY is - in practice - just another name for __GFP_RECLAIMABLE, and setting SLAB_RECLAIM_ACCOUNT is equivalent to setting __GFP_RECLAIMABLE. That article suggests caution is needed, as this should only be used for memory that is certain to be easily
available, because using this flag changes the allocation behavior on the assumption that the memory can be quickly freed at need. That is often not true of these Lustre objects.<br>
><br>
> An easy way to learn more about this sort of question is to compare to other actively developed file systems in the kernel...<br>
><br>
> Comparing to other file systems, we see that in general, only the inode cache is allocated with SLAB_RECLAIM_ACCOUNT (it varies a bit).<br>
><br>
> XFS, for example, has only one use of KM_ZONE_RECLAIM, its name for this flag - the inode cache:<br>
> "<br>
> xfs_inode_zone =<br>
> kmem_zone_init_flags(sizeof(xfs_inode_t), "xfs_inode",<br>
> KM_ZONE_HWALIGN | KM_ZONE_RECLAIM | KM_ZONE_SPREAD,<br>
> xfs_fs_inode_init_once);<br>
> "<br>
><br>
> btrfs is the same, just the inode cache. EXT4 has a *few* more caches marked this way, but not everything.<br>
><br>
> So, no - I don't think so. It would be atypical for Lustre to set SLAB_RECLAIM_ACCOUNT on its slab caches for internal objects. Presumably this sort of thing is not considered reclaimable enough for this purpose.<br>
><br>
> I believe if you tried similar tests with other complex file systems (XFS might be a good start), you'd see broadly similar behavior. (Lustre is probably a bit worse because it has a more complex internal object model, so more slab caches.)<br>
><br>
> VM accounting is distinctly imperfect. The design is such that it's often impossible to know how much memory could be made available without actually going and trying to free it. There are good, intrinsic reasons for some of that, and some of that is design
artifacts...<br>
><br>
> I've copied in Neil Brown, who I think only reads lustre-devel, just in case he has some particular input on this.<br>
><br>
> Regards,<br>
> - Patrick<br>
> ________________________________<br>
> From: lustre-discuss <lustre-discuss-bounces@lists.lustre.org> on behalf of Jacek Tomaka <jacekt@dug.com><br>
> Sent: Sunday, April 14, 2019 3:12:51 AM<br>
> To: lustre-discuss@lists.lustre.org<br>
> Subject: Re: [lustre-discuss] Lustre client memory and MemoryAvailable<br>
><br>
> Actually i think it is just a bug with the way slab caches are created. Some of them should be passed a flag that they are reclaimable.<br>
> i.e. something like:<br>
> <a href="https://patchwork.kernel.org/patch/9360819/">https://patchwork.kernel.org/patch/9360819/</a><br>
><br>
> Regards.<br>
> Jacek Tomaka<br>
><br>
> On Sun, Apr 14, 2019 at 3:27 PM Jacek Tomaka <jacekt@dug.com<mailto:jacekt@dug.com>> wrote:<br>
> Hello,<br>
><br>
> TL;DR;<br>
> Is there a way to figure out how much memory Lustre will make available under memory pressure?<br>
><br>
> Details:<br>
> We are running lustre client on a machine with 128GB of memory (Centos 7) Intel Phi KNL machines and at certain situations we see that there can be ~10GB+ of memory allocated on the kernel side i.e. :<br>
><br>
> vvp_object_kmem 3535336 3536986 176 46 2 : tunables 0 0 0 : slabdata 76891 76891 0<br>
> ll_thread_kmem 33511 33511 344 47 4 : tunables 0 0 0 : slabdata 713 713 0<br>
> lov_session_kmem 34760 34760 592 55 8 : tunables 0 0 0 : slabdata 632 632 0<br>
> osc_extent_kmem 3549831 3551232 168 48 2 : tunables 0 0 0 : slabdata 73984 73984 0<br>
> osc_thread_kmem 14012 14116 2832 11 8 : tunables 0 0 0 : slabdata 1286 1286 0<br>
> osc_object_kmem 3546640 3548350 304 53 4 : tunables 0 0 0 : slabdata 66950 66950 0<br>
> signal_cache 3702537 3707144 1152 28 8 : tunables 0 0 0 : slabdata 132398 132398 0<br>
><br>
> /proc/meminfo:<br>
> MemAvailable: 114196044 kB<br>
> Slab: 11641808 kB<br>
> SReclaimable: 1410732 kB<br>
> SUnreclaim: 10231076 kB<br>
><br>
> After executing<br>
><br>
> echo 1 >/proc/sys/vm/drop_caches<br>
><br>
> the slabinfo values don't change but when i actually generate memory pressure by:<br>
><br>
> java -Xmx117G -Xms117G -XX:+AlwaysPreTouch -version<br>
><br>
> lots of memory gets freed:<br>
> vvp_object_kmem 127650 127880 176 46 2 : tunables 0 0 0 : slabdata 2780 2780 0<br>
> ll_thread_kmem 33558 33558 344 47 4 : tunables 0 0 0 : slabdata 714 714 0<br>
> lov_session_kmem 34815 34815 592 55 8 : tunables 0 0 0 : slabdata 633 633 0<br>
> osc_extent_kmem 128640 128880 168 48 2 : tunables 0 0 0 : slabdata 2685 2685 0<br>
> osc_thread_kmem 14038 14116 2832 11 8 : tunables 0 0 0 : slabdata 1286 1286 0<br>
> osc_object_kmem 82998 83263 304 53 4 : tunables 0 0 0 : slabdata 1571 1571 0<br>
> signal_cache 38734 44268 1152 28 8 : tunables 0 0 0 : slabdata 1581 1581 0<br>
><br>
> /proc/meminfo:<br>
> MemAvailable: 123146076 kB<br>
> Slab: 1959160 kB<br>
> SReclaimable: 334276 kB<br>
> SUnreclaim: 1624884 kB<br>
><br>
> The similar effect to generating memory pressure we see when executing:<br>
><br>
> echo 3 >/proc/sys/vm/drop_caches<br>
><br>
> But this can take very long time (10 minutes).<br>
><br>
> So essentially on a machine using Lustre client, MemAvailable is no longer a good predictor of the amount of memory that can be allocated.<br>
> Is there a way to query Lustre and compensate for lustre cache memory that will be made available on memory pressure?<br>
><br>
> Regards.<br>
> --<br>
> Jacek Tomaka<br>
> Geophysical Software Developer<br>
><br>
><br>
> [<a href="http://drive.google.com/uc?export=view&id=0B4X9ixpc-ZU_NHV0WnluaXp5ZkE">http://drive.google.com/uc?export=view&id=0B4X9ixpc-ZU_NHV0WnluaXp5ZkE</a>]<br>
><br>
> DownUnder GeoSolutions<br>
><br>
> 76 Kings Park Road<br>
> West Perth 6005 WA, Australia<br>
> tel +61 8 9287 4143<tel:+61%208%209287%204143><br>
> jacekt@dug.com<<a href="mailto:jacekt@dug.com">mailto:jacekt@dug.com</a>><br>
> www.dug.com<<a href="http://www.dug.com">http://www.dug.com</a>><br>
><br>
><br>
> --<br>
> Jacek Tomaka<br>
> Geophysical Software Developer<br>
><br>
><br>
> [<a href="http://drive.google.com/uc?export=view&id=0B4X9ixpc-ZU_NHV0WnluaXp5ZkE">http://drive.google.com/uc?export=view&id=0B4X9ixpc-ZU_NHV0WnluaXp5ZkE</a>]<br>
><br>
> DownUnder GeoSolutions<br>
><br>
> 76 Kings Park Road<br>
> West Perth 6005 WA, Australia<br>
> tel +61 8 9287 4143<tel:+61%208%209287%204143><br>
> jacekt@dug.com<<a href="mailto:jacekt@dug.com">mailto:jacekt@dug.com</a>><br>
> www.dug.com<<a href="http://www.dug.com">http://www.dug.com</a>><br>
</div>
</span></font></div>
</body>
</html>