[Lustre-discuss] Number of inodes on OST

[Stearman, Marc]

All of our file systems show reserved=0 for all OSTs.

Create counts vary from 512 up to 2048.

-Marc

----
D. Marc Stearman
Lustre Operations Lead
stearman2 at llnl.gov
925.423.9670




On Mar 19, 2014, at 4:04 PM, "Dilger, Andreas" <andreas.dilger at intel.com> wrote:

> On 2014/03/11, 1:04 AM, "Grégoire Pichon" <gregoire.pichon at bull.net> wrote:
> 
>> Hi Jinshan,
>> 
>> An additional question on that topic...
>> Is the precreate object cache tunable ? for instance, to increase the
>> number of precreated objects in case of metadata intensive applications ?
> 
> This should be handled dynamically by the MDT instead of having a static
> precreate number.  It starts at 32 objects, and increases up to 20000 if
> the MDS is creating a lot of files.
> 
> If someone had interest to do so, it would be useful to actually do some
> measurements on the MDS to see what the precreate algorithm is doing,
> since it has been a long time since anyone looked at that code in detail.
> In particular, monitoring the precreated objects via:
> 
>        lctl get_param osc.*.prealloc_{{last,next}_id,reserved}
> 
> Ideally, (prealloc_last_id - prealloc_next_id - prealloc_reserved) should
> never hit zero, or it means that the MDS may block waiting for OST
> objects, or may allocate files unevenly across OSTs.  It would be
> interesting to graph these over time during a metadata create workload.
> 
> Cheers, Andreas
> 
>> -----Message d'origine-----
>> De : lustre-discuss-bounces at lists.lustre.org De la part de Xiong, Jinshan
>> Envoyé : mardi 11 mars 2014 04:16
>> À : Chan Ching Yu Patrick
>> Cc : lustre-discuss at lists.lustre.org
>> Objet : Re: [Lustre-discuss] Number of inodes on OST
>> 
>> For performance reason, the MDT creates objects on OSTs in batch with a
>> technology called `precreate'. The MDT caches the `precreated' objects in
>> cache and whenever a client is creating a file on the MDT, it will
>> allocate objects from cache. This is why you can't see the inode count on
>> OSTs increase immediately.
>> 
>> You can keep creating files from client side and monitor the inode count
>> on the OSTs you will see the effect.
>> 
>> Jinshan
>> 
>> On Mar 10, 2014, at 7:05 PM, Chan Ching Yu Patrick
>> <cychan at clustertech.com> wrote:
>> 
>>> Hi all,
>>> 
>>> I found that when I create a file, the number of inodes on OST does not
>>> increase.
>>> 
>>> Let's see the following experiment:
>>> 
>>> [root at client lustre]# lfs df -i
>>> UUID                      Inodes       IUsed       IFree IUse% Mounted
>>> on
>>> data-MDT0000_UUID         524288        3428      520860   1%
>>> /lustre[MDT:0]
>>> data-OST0000_UUID         393216        2398      390818   1%
>>> /lustre[OST:0]
>>> data-OST0001_UUID         393216        2398      390818   1%
>>> /lustre[OST:1]
>>> 
>>> filesystem summary:       524288        3428      520860   1% /lustre
>>> 
>>> 
>>> [root at client lustre]# dd if=/dev/zero of=/lustre/file12 bs=1M count=1000
>>> 1000+0 records in
>>> 1000+0 records out
>>> 1048576000 bytes (1.0 GB) copied, 46.4868 s, 22.6 MB/s
>>> 
>>> 
>>> [root at client lustre]# lfs df -i
>>> UUID                      Inodes       IUsed       IFree IUse% Mounted
>>> on
>>> data-MDT0000_UUID         524288        3429      520859   1%
>>> /lustre[MDT:0]
>>> data-OST0000_UUID         393216        2398      390818   1%
>>> /lustre[OST:0]
>>> data-OST0001_UUID         393216        2398      390818   1%
>>> /lustre[OST:1]
>>> 
>>> filesystem summary:       524288        3429      520859   1% /lustre
>>> 
>>> 
>>> You can see the number of inodes on MDT is increased by 1. But the
>>> number of inodes on OST keep unchanged.
>>> 
>>> 
>>> According to Lustre Manual, the inodes on OST should increase.
>>> 
>>> "Each time a file is created on a Lustre file system, it consumes one
>>> inode on the MDT and one inode for each OST object over which the file
>>> is striped. "
>>> 
>>> 
>>> Anyone knows why?   Thanks very much.
>>> 
>>> Regards,
>>> Patrick
>>> 
>>> _______________________________________________
>>> Lustre-discuss mailing list
>>> Lustre-discuss at lists.lustre.org
>>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>> 
>> _______________________________________________
>> Lustre-discuss mailing list
>> Lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>> _______________________________________________
>> Lustre-discuss mailing list
>> Lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>> 
> 
> 
> Cheers, Andreas
> -- 
> Andreas Dilger
> 
> Lustre Software Architect
> Intel High Performance Data Division
> 
> 
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss