[Lustre-discuss] question about size on MDS (MDT) for lustre-1.8

Andreas Dilger adilger at whamcloud.com
Thu Jan 13 13:37:34 PST 2011

On 2011-01-13, at 11:38, Nathan Dauchy wrote:
> On a somewhat related note... we have recently discovered that the
> object caching added in 1.8 consumes all the memory on the OSS nodes,
> leaving insufficient block device cache for the inodes.  This was making
> 'ls -l' and 'du' run 10-20x longer than when we were running
> lustre-1.6.7.  If you are running 1.8 and want to try turning off the
> object caching, these are the settings you should look at:
> lctl conf_param fsname-OST00XX.ost.read_cache_enable=0
> lctl conf_param fsname-OST00XX.ost.writethrough_cache_enable=0

It would probably be better to set:

lctl conf_param fsname-OST00XX.ost.readcache_max_filesize=32M

or similar, to limit the read cache to files 32MB in size or less (or whatever you consider "small" files at your site.  That allows the read cache for config files and such, while not thrashing the cache while accessing large files.

We should probably change this to be the default, but at the time the read cache was introduced, we didn't know what should be considered a small vs. large file, and the amount of RAM and number of OSTs on an OSS, and the uses varies so much that it is difficult to pick a single correct value for this.

Cheers, Andreas
Andreas Dilger 
Principal Engineer
Whamcloud, Inc.

More information about the lustre-discuss mailing list