[lustre-discuss] FOLLOW UP: MDT filling up with 4 MB files
Jessica Otey
jotey at nrao.edu
Fri Oct 14 15:46:59 PDT 2016
All,
My colleagues in Chile now believe that both of their 2.5.3 file systems
are experiencing this same problem with the MDTs filling up with files.
We have also come across a report from another user from early 2015
denoting the same issue, also with a 2.5.3 system.
See:
https://www.mail-archive.com/search?l=lustre-discuss@lists.lustre.org&q=subject:%22Re%5C%3A+%5C%5Blustre%5C-discuss%5C%5D+MDT+partition+getting+full%22&o=newest
We are confident that these files are not related to the changelog feature.
Does anyone have any other suggestions as to what the cause of this
problem could be?
I'm intrigued that the Lustre version involved in all 3 reports is
2.5.3. Could this be a bug?
Thanks,
Jessica
> On Thu, Sep 29, 2016 at 8:58 AM, Jessica Otey <jotey at nrao.edu
> <mailto:jotey at nrao.edu>> wrote:
>
> Hello all,
> I write on behalf of my colleagues in Chile, who are experiencing
> a bizarre problem with their MDT, namely, it is filling up with 4
> MB files. There is no issue with the number of inodes, of which
> there are hundreds of millions unused. Â
>
> [root at jaopost-mds ~]# tune2fs -l /dev/sdb2 | grep -i inode
> device /dev/sdb2 mounted by lustre
> Filesystem features: Â Â Â has_journal ext_attr resize_inode
> dir_index filetype needs_recovery flex_bg dirdata sparse_super
> large_file huge_file uninit_bg dir_nlink quota
> Inode count: Â Â Â Â Â Â Â 239730688
> Free inodes: Â Â Â Â Â Â Â 223553405
> Inodes per group: Â Â Â Â 32768
> Inode blocks per group: Â 4096
> First inode: Â Â Â Â Â Â Â 11
> Inode size:Â Â Â Â Â 512
> Journal inode: Â Â Â Â Â Â 8
> Journal backup: Â Â Â Â Â inode blocks
> User quota inode: Â Â Â Â 3
> Group quota inode: Â Â Â Â 4
>
> Has anyone ever encountered such a problem? The only thing unusual
> about this cluster is that it is using 2.5.3 MDS/OSSes while still
> using 1.8.9 clients—something I didn't actually believe was
> possible, as I thought the last version to work effectively with
> 1.8.9 clients was 2.4.3. However, for all I know, the version gap
> may have nothing to do with this phenomena.
>
> Any and all advice is appreciated. Any general information on the
> structure of the MDT also welcome, as such info is in short supply
> on the internet.
>
> Thanks,
> Jessica
>
> Below is a look inside the O folder at the root of the MDT, where
> there are about 48,000 4MB files:
>
> [root at jaopost-mds O]# pwd
> /lustrebackup/O
> [root at jaopost-mds O]# tree -L 1
> .
> ├── 1
> ├── 10
> └── 200000003
>
> 3 directories, 0 files
>
> [root at jaopost-mds O]# ls -l 1
> total 2240
> drwx------ 2 root root 69632 sep 16 16:25 d0
> drwx------ 2 root root 69632 sep 16 16:25 d1
> drwx------ 2 root root 61440 sep 16 17:46 d10
> drwx------ 2 root root 69632 sep 16 17:46 d11
> drwx------ 2 root root 69632 sep 16 18:04 d12
> drwx------ 2 root root 65536 sep 16 18:04 d13
> drwx------ 2 root root 65536 sep 16 18:04 d14
> drwx------ 2 root root 69632 sep 16 18:04 d15
> drwx------ 2 root root 61440 sep 16 18:04 d16
> drwx------ 2 root root 61440 sep 16 18:04 d17
> drwx------ 2 root root 69632 sep 16 18:04 d18
> drwx------ 2 root root 69632 sep 16 18:04 d19
> drwx------ 2 root root 65536 sep 16 16:25 d2
> drwx------ 2 root root 69632 sep 16 18:04 d20
> drwx------ 2 root root 69632 sep 16 18:04 d21
> drwx------ 2 root root 61440 sep 16 18:04 d22
> drwx------ 2 root root 69632 sep 16 18:04 d23
> drwx------ 2 root root 61440 sep 16 16:11 d24
> drwx------ 2 root root 69632 sep 16 16:11 d25
> drwx------ 2 root root 69632 sep 16 16:11 d26
> drwx------ 2 root root 69632 sep 16 16:11 d27
> drwx------ 2 root root 69632 sep 16 16:25 d28
> drwx------ 2 root root 69632 sep 16 16:25 d29
> drwx------ 2 root root 69632 sep 16 16:25 d3
> drwx------ 2 root root 65536 sep 16 16:25 d30
> drwx------ 2 root root 65536 sep 16 16:25 d31
> drwx------ 2 root root 69632 sep 16 16:25 d4
> drwx------ 2 root root 61440 sep 16 16:25 d5
> drwx------ 2 root root 69632 sep 16 16:25 d6
> drwx------ 2 root root 73728 sep 16 16:25 d7
> drwx------ 2 root root 65536 sep 16 17:46 d8
> drwx------ 2 root root 69632 sep 16 17:46 d9
> -rw-r--r-- 1 root root 8 ene 4 2016 LAST_ID
>
> [root at jaopost-mds d0]# ls -ltr | more
> total 5865240
> -rw-r--r-- 1 root root 252544 ene 4 2016 32
> -rw-r--r-- 1 root root 2396224 ene 9 2016 2720
> -rw-r--r-- 1 root root 4153280 ene 9 2016 2752
> -rw-r--r-- 1 root root 4153280 ene 10 2016 2784
> -rw-r--r-- 1 root root 4153280 ene 10 2016 2816
> -rw-r--r-- 1 root root 4153280 ene 10 2016 2848
> -rw-r--r-- 1 root root 4153280 ene 10 2016 2880
> -rw-r--r-- 1 root root 4153280 ene 10 2016 2944
> -rw-r--r-- 1 root root 4153280 ene 10 2016 2976
> -rw-r--r-- 1 root root 4153280 ene 10 2016 3008
> -rw-r--r-- 1 root root 4153280 ene 10 2016 3040
> -rw-r--r-- 1 root root 4153280 ene 10 2016 3072
> -rw-r--r-- 1 root root 4153280 ene 10 2016 3104
> -rw-r--r-- 1 root root 4153280 ene 10 2016 3136
> -rw-r--r-- 1 root root 4153280 ene 10 2016 3168
> -rw-r--r-- 1 root root 4153280 ene 10 2016 3200
> -rw-r--r-- 1 root root 4153280 ene 10 2016 3232
> -rw-r--r-- 1 root root 4153280 ene 10 2016 3264
> -rw-r--r-- 1 root root 4153280 ene 10 2016 3296
> -rw-r--r-- 1 root root 4153280 ene 10 2016 3328
>
>
>
> _______________________________________________
> lustre-discuss mailing list
> lustre-discuss at lists.lustre.org
> <mailto:lustre-discuss at lists.lustre.org>
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
> <http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20161014/28196834/attachment.htm>
-------------- next part --------------
_______________________________________________
lustre-discuss mailing list
lustre-discuss at lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
More information about the lustre-discuss
mailing list