[lustre-discuss] MGTMDT device getting full

[Christopher J. Morrone]

Oh.  Doh.  I missed the low Inode usage that you listed in your first 
email.   Ben was right, that kind of does point to some runaway log 
files like changelogs.  Perhaps you enabled them by accident?  Do you 
have any kind of HSM?  It would be good to check the changelog_users 
regardless.

If that doesn't show anything, you probably need to mount your MDT's 
backend filesystem as a local filesytem (readonly) and look for where 
the space is going.

Chris

On 10/16/2015 10:37 AM, Christopher J. Morrone wrote:
> Hi Torsten,
>
> There is no reason to suspect that space usage on the MDT will be the
> same as the average space usage on the OSTs.
>
> Your MDT is storing the metadata about _all_ of the files in your Lustre
> filesystem.  You can think of this metadata as a whole bunch of
> zero-length files with some extended attributes, because under the
> covers that is basically what the MDT is storing.
>
> So space usage on the MDT will be directly proportional to the total
> number of files in your Lustre filesystem.  The size of those files
> doesn't really matter much, because the contents of the files are stored
> on the OSTs.
>
> So your issue is that, for your filesystem, on average your files are
> too small to allow filling your OSTs.  Some possible solutions are:
>
> * Increase the size of your MDT
> * Encourage/require your users to start using larger average file sizes
>
> Granted, neither approach is terribly easy.
>
> I think that you can pretty safely ignore the changelogs suggestion.
>
> Chris
>
> On 10/16/2015 07:31 AM, Torsten Harenberg wrote:
>> Am 16.10.2015 um 16:01 schrieb Ben Evans:
>>> Looks like you¹ve got some really large changelogs built up.  Did you
>>> have
>>> robin hood, or some other consumer running at some point that has since
>>> stalled?
>>
>> Don't think so, as I never heard about "Robin Hood" in the context of
>> Lustre.
>>
>> The setup is pretty simple, devices were created with
>>
>> mkfs.lustre --fsname=lustre --mgs --mdt --backfstype=ext4
>> --failnode=132.195.124.201 at tcp --verbose /dev/mapper/MGTMDT
>>
>> and
>>
>> mkfs.lustre --fsname=lustre --ost --backfstype=ext4
>> --failnode=132.195.124.204 at tcp --mgsnode=132.195.124.202 at tcp
>> --mgsnode=132.195.124.201 at tcp --verbose /dev/mapper/OST0000
>>
>> respectively
>>
>> The only "add-on" we have is quota support. Back in 2.1.5 that was
>> enabled with:
>>
>> lfs quotacheck –ug /lustre
>> lfs quotaon /lustre
>>
>>
>> The file system is mounted on about 200 nodes and accessed by cluster
>> users.
>>
>> Best regards,
>>
>>    Torsten
>>
>
> .
>