[Lustre-devel] Hangs with cgroup memory controller
robin.humble+lustre at anu.edu.au
Fri Jul 29 00:15:16 PDT 2011
On Wed, Jul 27, 2011 at 07:57:57PM +0100, Mark Hills wrote:
>On Wed, 27 Jul 2011, Andreas Dilger wrote:
>> Possibly you can correlate reproducer cases with Lustre errors on the
>I've managed to catch the bad state, on a clean client too -- there's no
>errors reported from Lustre in dmesg.
>Here's the information reported by the cgroup. It seems that there's a
>discrepancy of 2x pages (the 'cache' field, pgpgin, pgpgout).
>The process which was in the group terminated a long time ago.
>I can leave the machine in this state until tomorrow, so any suggestions
>for data to capture that could help trace this bug would be welcomed.
with zone_reclaim_mode=1 (even without memcg) we saw ~infinite scanning
for pages when doing Lustre i/o + memory pressure, which also hung up a
core in 100% system time. the scanning can be seen with
grep scan /proc/zoneinfo
that zone_reclaim_mode=0 helps our problem could be related to your
memcg semi-missing pages, or perhaps it's a workaround for a core
kernel problem with zones - we only have Lustre so can't distinguish.
secondly, and even more of a long shot - I presume slab isn't accounted
as part of memcg, but you could also try clearing the ldlm locks. Linux
is reluctant to drop inodes caches until the locks are cleared first
lctl set_param ldlm.namespaces.*.lru_size=clear
More information about the lustre-devel