[Lustre-discuss] explanation about inodes in MDT/OST

[E.S. Rosenberg]

Cool, thanks for all the in-depth answers.
Eli

On Wed, Jun 19, 2013 at 6:33 PM, Dilger, Andreas
<andreas.dilger at intel.com> wrote:
> On 2013-06-19, at 8:52, "Lee, Brett" <brett.lee at intel.com<mailto:brett.lee at intel.com>> wrote:
> We probably should clarify the context of inodes:
>
> 1) Each OST has "inodes" – these inodes refer to objects on that OST that contain block data for a Lustre file.
>
> I prefer to call these as OST objects, to keep them distinct from the MDT inodes.
>
> 2) Each (as of 2.4) MDT has "inodes" – these inodes contain metadata for a single Lustre file.
>
> When adding an OST you gain the inodes on that OST for objects (block data) on that OST, but the total amount of Lustre files (MDT inodes) hasn't changed.
>
> Right, though Mr. Rosenberg is correct that since the MDT uses an ext4-based backing storage it is possible to use offline resize2fs to increase the size of the MDT afterward, if it is using LVM or some other form of storage that can be resized.  Online resizing is blocked behind LU-137 being finished.
>
> Note that resize2fs is not something that we test on any regular basis, because most of our customers have large fixed-size devices for the MDT (over-provisioning on spindles). If someone needs to do this, is encourage them to make a "dd" copy of the MDT to a new device first (could just be a USB SATA disk) and test out the resize2fs first, and verify with e2fsck before and after that the filesystem is still valid.
>
> I'll see whether i can get resize2fs testing added into our normal test suite.
>
> Cheers, Andreas
>
>
> From: lustre-discuss-bounces at lists.lustre.org<mailto:lustre-discuss-bounces at lists.lustre.org> [mailto:lustre-discuss-bounces at lists.lustre.org] On Behalf Of Parinay Kondekar
> Sent: Wednesday, June 19, 2013 8:23 AM
> To: E.S. Rosenberg
> Cc: lustre-discuss at lists.lustre.org<mailto:lustre-discuss at lists.lustre.org>
> Subject: Re: [Lustre-discuss] explanation about inodes in MDT/OST
>
> IMO there are two limiting factors
>
> 1. total number of inodes supported by MDT and
> 2.  OSTs  i.e. num_ost_inodes = 4 * number_of_mds_inodes * default_stripe_count / number_of_osts
>
> so your Q
> I am not sure I understand this 100%, when adding an OST I do or don't
> gain inodes?or you basically only get more inodes in the data part of
> the filesystem but if you don't have enough inodes on the MDT you
> still have a problem?
> A
> Yes, if there are no enough inodes on MDT adding OSTs wont help much.
>
> I don't think you can change no. of inodes w/o formatting. You might want to take a look at
> lfs_migrate, on how to increase the no_of_inodes on OST's.
>
> HTH
> p
>
> On 19 June 2013 19:19, E.S. Rosenberg <esr+lustre at mail.hebrew.edu<mailto:esr+lustre at mail.hebrew.edu>> wrote:
> "
> In a Lustre ldiskfs file system, all the inodes are allocated on the
> MDT and OSTs when the file system is first formatted. The total number
> of inodes on a formatted MDT or OST cannot be easily changed, although
> it is possible to add OSTs with additional space and corresponding
> inodes. Thus, the number of inodes created at format time should be
> generous enough to anticipate future expansion.
> "
> http://build.whamcloud.com/job/lustre-manual/lastSuccessfulBuild/artifact/lustre_manual.xhtml#understandinglustre.fig.cluster
>
>
>
> Also if lustrefs is really based on ext4 doesn't ext4 support growing
> the amount of inodes with resize2fs?
>
> Thanks,
> Eli
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org<mailto:Lustre-discuss at lists.lustre.org>
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org<mailto:Lustre-discuss at lists.lustre.org>
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss