[Lustre-discuss] Number of inodes on OST

[Dilger, Andreas]

On 2014/03/11, 1:04 AM, "Grégoire Pichon" <gregoire.pichon at bull.net> wrote:

>Hi Jinshan,
>
>An additional question on that topic...
>Is the precreate object cache tunable ? for instance, to increase the
>number of precreated objects in case of metadata intensive applications ?

This should be handled dynamically by the MDT instead of having a static
precreate number.  It starts at 32 objects, and increases up to 20000 if
the MDS is creating a lot of files.

If someone had interest to do so, it would be useful to actually do some
measurements on the MDS to see what the precreate algorithm is doing,
since it has been a long time since anyone looked at that code in detail.
In particular, monitoring the precreated objects via:

        lctl get_param osc.*.prealloc_{{last,next}_id,reserved}

Ideally, (prealloc_last_id - prealloc_next_id - prealloc_reserved) should
never hit zero, or it means that the MDS may block waiting for OST
objects, or may allocate files unevenly across OSTs.  It would be
interesting to graph these over time during a metadata create workload.

Cheers, Andreas

>-----Message d'origine-----
>De : lustre-discuss-bounces at lists.lustre.org De la part de Xiong, Jinshan
>Envoyé : mardi 11 mars 2014 04:16
>À : Chan Ching Yu Patrick
>Cc : lustre-discuss at lists.lustre.org
>Objet : Re: [Lustre-discuss] Number of inodes on OST
>
>For performance reason, the MDT creates objects on OSTs in batch with a
>technology called `precreate'. The MDT caches the `precreated' objects in
>cache and whenever a client is creating a file on the MDT, it will
>allocate objects from cache. This is why you can't see the inode count on
>OSTs increase immediately.
>
>You can keep creating files from client side and monitor the inode count
>on the OSTs you will see the effect.
>
>Jinshan
>
>On Mar 10, 2014, at 7:05 PM, Chan Ching Yu Patrick
><cychan at clustertech.com> wrote:
>
>> Hi all,
>> 
>> I found that when I create a file, the number of inodes on OST does not
>>increase.
>> 
>> Let's see the following experiment:
>> 
>> [root at client lustre]# lfs df -i
>> UUID                      Inodes       IUsed       IFree IUse% Mounted
>>on
>> data-MDT0000_UUID         524288        3428      520860   1%
>>/lustre[MDT:0]
>> data-OST0000_UUID         393216        2398      390818   1%
>>/lustre[OST:0]
>> data-OST0001_UUID         393216        2398      390818   1%
>>/lustre[OST:1]
>> 
>> filesystem summary:       524288        3428      520860   1% /lustre
>> 
>> 
>> [root at client lustre]# dd if=/dev/zero of=/lustre/file12 bs=1M count=1000
>> 1000+0 records in
>> 1000+0 records out
>> 1048576000 bytes (1.0 GB) copied, 46.4868 s, 22.6 MB/s
>> 
>> 
>> [root at client lustre]# lfs df -i
>> UUID                      Inodes       IUsed       IFree IUse% Mounted
>>on
>> data-MDT0000_UUID         524288        3429      520859   1%
>>/lustre[MDT:0]
>> data-OST0000_UUID         393216        2398      390818   1%
>>/lustre[OST:0]
>> data-OST0001_UUID         393216        2398      390818   1%
>>/lustre[OST:1]
>> 
>> filesystem summary:       524288        3429      520859   1% /lustre
>> 
>> 
>> You can see the number of inodes on MDT is increased by 1. But the
>>number of inodes on OST keep unchanged.
>> 
>> 
>> According to Lustre Manual, the inodes on OST should increase.
>> 
>> "Each time a file is created on a Lustre file system, it consumes one
>>inode on the MDT and one inode for each OST object over which the file
>>is striped. "
>> 
>> 
>> Anyone knows why?   Thanks very much.
>> 
>> Regards,
>> Patrick
>> 
>> _______________________________________________
>> Lustre-discuss mailing list
>> Lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>
>_______________________________________________
>Lustre-discuss mailing list
>Lustre-discuss at lists.lustre.org
>http://lists.lustre.org/mailman/listinfo/lustre-discuss
>_______________________________________________
>Lustre-discuss mailing list
>Lustre-discuss at lists.lustre.org
>http://lists.lustre.org/mailman/listinfo/lustre-discuss
>


Cheers, Andreas
-- 
Andreas Dilger

Lustre Software Architect
Intel High Performance Data Division