[lustre-discuss] MDT filling up with 4 MB files

Thu Sep 29 09:25:09 PDT 2016

Yes, if you're not consuming the records, you're going to see them eat up
space on the MDT.

On Thu, Sep 29, 2016 at 10:04 AM, Jessica Otey <jotey at nrao.edu> wrote:

>
>
> On 9/29/16 12:36 PM, Colin Faber wrote:
>
> Is the changelogs feature enabled?
>
> Yes, and.. the output of lfs changelogs gives us 360,000 lines... Do you
> think that is the source of all the 'extra' data?
>
> On Thu, Sep 29, 2016 at 8:58 AM, Jessica Otey <jotey at nrao.edu> wrote:
>
>> Hello all,
>> I write on behalf of my colleagues in Chile, who are experiencing a
>> bizarre problem with their MDT, namely, it is filling up with 4 MB files.
>> There is no issue with the number of inodes, of which there are hundreds of
>> millions unused.
>>
>> [root at jaopost-mds ~]# tune2fs -l /dev/sdb2 | grep -i inode
>> device /dev/sdb2 mounted by lustre
>> Filesystem features:      has_journal ext_attr resize_inode dir_index
>> filetype needs_recovery flex_bg dirdata sparse_super large_file huge_file
>> uninit_bg dir_nlink quota
>> Inode count:              239730688
>> Free inodes:              223553405
>> Inodes per group:         32768
>> Inode blocks per group:   4096
>> First inode:              11
>> Inode size:          512
>> Journal inode:            8
>> Journal backup:           inode blocks
>> User quota inode:         3
>> Group quota inode:        4
>>
>> Has anyone ever encountered such a problem? The only thing unusual about
>> this cluster is that it is using 2.5.3 MDS/OSSes while still using 1.8.9
>> clients—something I didn't actually believe was possible, as I thought the
>> last version to work effectively with 1.8.9 clients was 2.4.3. However, for
>> all I know, the version gap may have nothing to do with this phenomena.
>>
>> Any and all advice is appreciated. Any general information on the
>> structure of the MDT also welcome, as such info is in short supply on the
>> internet.
>>
>> Thanks,
>> Jessica
>>
>> Below is a look inside the O folder at the root of the MDT, where there
>> are about 48,000 4MB files:
>>
>> [root at jaopost-mds O]# pwd
>> /lustrebackup/O
>> [root at jaopost-mds O]# tree -L 1
>> .
>> ├── 1
>> ├── 10
>> └── 200000003
>>
>> 3 directories, 0 files
>>
>> [root at jaopost-mds O]# ls -l 1
>> total 2240
>> drwx------ 2 root root 69632 sep 16 16:25 d0
>> drwx------ 2 root root 69632 sep 16 16:25 d1
>> drwx------ 2 root root 61440 sep 16 17:46 d10
>> drwx------ 2 root root 69632 sep 16 17:46 d11
>> drwx------ 2 root root 69632 sep 16 18:04 d12
>> drwx------ 2 root root 65536 sep 16 18:04 d13
>> drwx------ 2 root root 65536 sep 16 18:04 d14
>> drwx------ 2 root root 69632 sep 16 18:04 d15
>> drwx------ 2 root root 61440 sep 16 18:04 d16
>> drwx------ 2 root root 61440 sep 16 18:04 d17
>> drwx------ 2 root root 69632 sep 16 18:04 d18
>> drwx------ 2 root root 69632 sep 16 18:04 d19
>> drwx------ 2 root root 65536 sep 16 16:25 d2
>> drwx------ 2 root root 69632 sep 16 18:04 d20
>> drwx------ 2 root root 69632 sep 16 18:04 d21
>> drwx------ 2 root root 61440 sep 16 18:04 d22
>> drwx------ 2 root root 69632 sep 16 18:04 d23
>> drwx------ 2 root root 61440 sep 16 16:11 d24
>> drwx------ 2 root root 69632 sep 16 16:11 d25
>> drwx------ 2 root root 69632 sep 16 16:11 d26
>> drwx------ 2 root root 69632 sep 16 16:11 d27
>> drwx------ 2 root root 69632 sep 16 16:25 d28
>> drwx------ 2 root root 69632 sep 16 16:25 d29
>> drwx------ 2 root root 69632 sep 16 16:25 d3
>> drwx------ 2 root root 65536 sep 16 16:25 d30
>> drwx------ 2 root root 65536 sep 16 16:25 d31
>> drwx------ 2 root root 69632 sep 16 16:25 d4
>> drwx------ 2 root root 61440 sep 16 16:25 d5
>> drwx------ 2 root root 69632 sep 16 16:25 d6
>> drwx------ 2 root root 73728 sep 16 16:25 d7
>> drwx------ 2 root root 65536 sep 16 17:46 d8
>> drwx------ 2 root root 69632 sep 16 17:46 d9
>> -rw-r--r-- 1 root root     8 ene  4  2016 LAST_ID
>>
>> [root at jaopost-mds d0]# ls -ltr | more
>> total 5865240
>> -rw-r--r-- 1 root root  252544 ene  4  2016 32
>> -rw-r--r-- 1 root root 2396224 ene  9  2016 2720
>> -rw-r--r-- 1 root root 4153280 ene  9  2016 2752
>> -rw-r--r-- 1 root root 4153280 ene 10  2016 2784
>> -rw-r--r-- 1 root root 4153280 ene 10  2016 2816
>> -rw-r--r-- 1 root root 4153280 ene 10  2016 2848
>> -rw-r--r-- 1 root root 4153280 ene 10  2016 2880
>> -rw-r--r-- 1 root root 4153280 ene 10  2016 2944
>> -rw-r--r-- 1 root root 4153280 ene 10  2016 2976
>> -rw-r--r-- 1 root root 4153280 ene 10  2016 3008
>> -rw-r--r-- 1 root root 4153280 ene 10  2016 3040
>> -rw-r--r-- 1 root root 4153280 ene 10  2016 3072
>> -rw-r--r-- 1 root root 4153280 ene 10  2016 3104
>> -rw-r--r-- 1 root root 4153280 ene 10  2016 3136
>> -rw-r--r-- 1 root root 4153280 ene 10  2016 3168
>> -rw-r--r-- 1 root root 4153280 ene 10  2016 3200
>> -rw-r--r-- 1 root root 4153280 ene 10  2016 3232
>> -rw-r--r-- 1 root root 4153280 ene 10  2016 3264
>> -rw-r--r-- 1 root root 4153280 ene 10  2016 3296
>> -rw-r--r-- 1 root root 4153280 ene 10  2016 3328
>>
>>
>>
>> _______________________________________________
>> lustre-discuss mailing list
>> lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20160929/d6ea3f2e/attachment-0001.htm>