[Lustre-discuss] Number of inodes on OST

[Dilger, Andreas]

Thanks Marc. 

It at least appears to be working correctly at first glance. The reserved value is just a short term non-specific reservation between the time the MDS is trying to pick OSTs for the file and when it actually allocates specific objects for the layout. 

Cheers, Andreas

> On Mar 20, 2014, at 10:46, "Stearman, Marc" <stearman2 at llnl.gov> wrote:
> 
> All of our file systems show reserved=0 for all OSTs.
> 
> Create counts vary from 512 up to 2048.
> 
> -Marc
> 
> ----
> D. Marc Stearman
> Lustre Operations Lead
> stearman2 at llnl.gov
> 925.423.9670
> 
> 
> 
> 
>> On Mar 19, 2014, at 4:04 PM, "Dilger, Andreas" <andreas.dilger at intel.com> wrote:
>> 
>>> On 2014/03/11, 1:04 AM, "Grégoire Pichon" <gregoire.pichon at bull.net> wrote:
>>> 
>>> Hi Jinshan,
>>> 
>>> An additional question on that topic...
>>> Is the precreate object cache tunable ? for instance, to increase the
>>> number of precreated objects in case of metadata intensive applications ?
>> 
>> This should be handled dynamically by the MDT instead of having a static
>> precreate number.  It starts at 32 objects, and increases up to 20000 if
>> the MDS is creating a lot of files.
>> 
>> If someone had interest to do so, it would be useful to actually do some
>> measurements on the MDS to see what the precreate algorithm is doing,
>> since it has been a long time since anyone looked at that code in detail.
>> In particular, monitoring the precreated objects via:
>> 
>>       lctl get_param osc.*.prealloc_{{last,next}_id,reserved}
>> 
>> Ideally, (prealloc_last_id - prealloc_next_id - prealloc_reserved) should
>> never hit zero, or it means that the MDS may block waiting for OST
>> objects, or may allocate files unevenly across OSTs.  It would be
>> interesting to graph these over time during a metadata create workload.
>> 
>> Cheers, Andreas
>> 
>>> -----Message d'origine-----
>>> De : lustre-discuss-bounces at lists.lustre.org De la part de Xiong, Jinshan
>>> Envoyé : mardi 11 mars 2014 04:16
>>> À : Chan Ching Yu Patrick
>>> Cc : lustre-discuss at lists.lustre.org
>>> Objet : Re: [Lustre-discuss] Number of inodes on OST
>>> 
>>> For performance reason, the MDT creates objects on OSTs in batch with a
>>> technology called `precreate'. The MDT caches the `precreated' objects in
>>> cache and whenever a client is creating a file on the MDT, it will
>>> allocate objects from cache. This is why you can't see the inode count on
>>> OSTs increase immediately.
>>> 
>>> You can keep creating files from client side and monitor the inode count
>>> on the OSTs you will see the effect.
>>> 
>>> Jinshan
>>> 
>>> On Mar 10, 2014, at 7:05 PM, Chan Ching Yu Patrick
>>> <cychan at clustertech.com> wrote:
>>> 
>>>> Hi all,
>>>> 
>>>> I found that when I create a file, the number of inodes on OST does not
>>>> increase.
>>>> 
>>>> Let's see the following experiment:
>>>> 
>>>> [root at client lustre]# lfs df -i
>>>> UUID                      Inodes       IUsed       IFree IUse% Mounted
>>>> on
>>>> data-MDT0000_UUID         524288        3428      520860   1%
>>>> /lustre[MDT:0]
>>>> data-OST0000_UUID         393216        2398      390818   1%
>>>> /lustre[OST:0]
>>>> data-OST0001_UUID         393216        2398      390818   1%
>>>> /lustre[OST:1]
>>>> 
>>>> filesystem summary:       524288        3428      520860   1% /lustre
>>>> 
>>>> 
>>>> [root at client lustre]# dd if=/dev/zero of=/lustre/file12 bs=1M count=1000
>>>> 1000+0 records in
>>>> 1000+0 records out
>>>> 1048576000 bytes (1.0 GB) copied, 46.4868 s, 22.6 MB/s
>>>> 
>>>> 
>>>> [root at client lustre]# lfs df -i
>>>> UUID                      Inodes       IUsed       IFree IUse% Mounted
>>>> on
>>>> data-MDT0000_UUID         524288        3429      520859   1%
>>>> /lustre[MDT:0]
>>>> data-OST0000_UUID         393216        2398      390818   1%
>>>> /lustre[OST:0]
>>>> data-OST0001_UUID         393216        2398      390818   1%
>>>> /lustre[OST:1]
>>>> 
>>>> filesystem summary:       524288        3429      520859   1% /lustre
>>>> 
>>>> 
>>>> You can see the number of inodes on MDT is increased by 1. But the
>>>> number of inodes on OST keep unchanged.
>>>> 
>>>> 
>>>> According to Lustre Manual, the inodes on OST should increase.
>>>> 
>>>> "Each time a file is created on a Lustre file system, it consumes one
>>>> inode on the MDT and one inode for each OST object over which the file
>>>> is striped. "
>>>> 
>>>> 
>>>> Anyone knows why?   Thanks very much.
>>>> 
>>>> Regards,
>>>> Patrick
>>>> 
>>>> _______________________________________________
>>>> Lustre-discuss mailing list
>>>> Lustre-discuss at lists.lustre.org
>>>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>>> 
>>> _______________________________________________
>>> Lustre-discuss mailing list
>>> Lustre-discuss at lists.lustre.org
>>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>>> _______________________________________________
>>> Lustre-discuss mailing list
>>> Lustre-discuss at lists.lustre.org
>>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>>> 
>> 
>> 
>> Cheers, Andreas
>> -- 
>> Andreas Dilger
>> 
>> Lustre Software Architect
>> Intel High Performance Data Division
>> 
>> 
>> _______________________________________________
>> Lustre-discuss mailing list
>> Lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>