[Lustre-discuss] Fwd: Lustre and Large Pages

Andreas Dilger andreas.dilger at oracle.com
Thu Aug 19 16:07:24 PDT 2010


On 2010-08-19, at 16:44, Kevin Van Maren wrote:
> Easy way to reduce the client memory used by "Lustre" is to have an 
> Epilogue script run by SGE (or whatever scheduler/resource manager) that 
> does something like this on every node:
> # sync ; sleep 1 ; sync
> # echo 3 > /proc/sys/vm/drop_caches

Actually, my understanding is that /proc/sys/vm/drop_caches is NOT safe for production usage in all cases (i.e. there are bugs in some kernels, and it isn't actually meant for regular use from what I've read).

Others use huge pages in their configuration, but they reserve them at node boot time.  See https://bugzilla.lustre.org/show_bug.cgi?id=14323 for details.

If you want to flush all the memory used by a Lustre client between jobs, you can do "lctl set_param ldlm.namespaces.*.lru_size=clear".  Unlike Kevin's suggestion it is Lustre-specific, while drop_caches will try to flush memory from everything.

> Nathan Rutman wrote:
>> Jim, I'm forwarding this to lustre-discuss to get a broader community 
>> input.  I'm sure somebody has some experience with this.
>> 
>> Begin forwarded message:
>>> 
>>> I am looking for information on how Lustre assigns and holds pages on 
>>> client nodes across jobs.  The motivation is that we want to make 
>>> "huge" pages available to users.  We have found that it is almost 
>>> impossible to allocate very many "huge" pages since Lustre holds 
>>> scattered small pages across jobs.  In fact, typically about 1/3 of 
>>> compute node memory can be allocated as huge pages.
>>> 
>>> We have done quite a lot of performance studies which show that a 
>>> substantial percentage of jobs on Ranger have TLB misses as a major 
>>> performance bottleneck.  We estimate we might recover as much as an 
>>> additional 5%-10% throughput if users could use huge pages.
>>> 
>>> Therefore we would like to find a way to minimize the client memory 
>>> which Lustre holds across jobs.
>>> 
>>> Have you had anyone else mention this situation to you?
>>> 
>>> Regards,
>>> 
>>> Jim Browne
>>> 
>>> 
>> 
>> ------------------------------------------------------------------------
>> 
>> _______________________________________________
>> Lustre-discuss mailing list
>> Lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>> 
> 
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss


Cheers, Andreas
--
Andreas Dilger
Lustre Technical Lead
Oracle Corporation Canada Inc.




More information about the lustre-discuss mailing list