[lustre-discuss] MDT filling up with 4 MB files

Thu Sep 29 11:31:10 PDT 2016

You shouldn't enable ChangeLogs unless you have some application consuming the logs (i.e. RobinHood).  Otherwise, as you observe, the ChangeLogs grow to fill all available space.

Cheers, Andreas
--
Andreas Dilger
Lustre Principal Architect
Intel High Performance Data Division

On 2016/09/29, 10:04, "lustre-discuss on behalf of Jessica Otey" <lustre-discuss-bounces at lists.lustre.org<mailto:lustre-discuss-bounces at lists.lustre.org> on behalf of jotey at nrao.edu<mailto:jotey at nrao.edu>> wrote:

On 9/29/16 12:36 PM, Colin Faber wrote:
Is the changelogs feature enabled?

Yes, and.. the output of lfs changelogs gives us 360,000 lines... Do you think that is the source of all the 'extra' data?

On Thu, Sep 29, 2016 at 8:58 AM, Jessica Otey <jotey at nrao.edu<mailto:jotey at nrao.edu>> wrote:
Hello all,
I write on behalf of my colleagues in Chile, who are experiencing a bizarre problem with their MDT, namely, it is filling up with 4 MB files. There is no issue with the number of inodes, of which there are hundreds of millions unused.
[root at jaopost-mds ~]# tune2fs -l /dev/sdb2 | grep -i inode
device /dev/sdb2 mounted by lustre
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery flex_bg dirdata sparse_super large_file huge_file uninit_bg dir_nlink quota
Inode count:              239730688
Free inodes:              223553405
Inodes per group:         32768
Inode blocks per group:   4096
First inode:              11
Inode size:          512
Journal inode:            8
Journal backup:           inode blocks
User quota inode:         3
Group quota inode:        4

Has anyone ever encountered such a problem? The only thing unusual about this cluster is that it is using 2.5.3 MDS/OSSes while still using 1.8.9 clients—something I didn't actually believe was possible, as I thought the last version to work effectively with 1.8.9 clients was 2.4.3. However, for all I know, the version gap may have nothing to do with this phenomena.

Any and all advice is appreciated. Any general information on the structure of the MDT also welcome, as such info is in short supply on the internet.

Thanks,
Jessica

Below is a look inside the O folder at the root of the MDT, where there are about 48,000 4MB files:

[root at jaopost-mds O]# pwd

/lustrebackup/O

[root at jaopost-mds O]# tree -L 1

.

├── 1

├── 10

└── 200000003

3 directories, 0 files

[root at jaopost-mds O]# ls -l 1

total 2240

drwx------ 2 root root 69632 sep 16 16:25 d0

drwx------ 2 root root 69632 sep 16 16:25 d1

drwx------ 2 root root 61440 sep 16 17:46 d10

drwx------ 2 root root 69632 sep 16 17:46 d11

drwx------ 2 root root 69632 sep 16 18:04 d12

drwx------ 2 root root 65536 sep 16 18:04 d13

drwx------ 2 root root 65536 sep 16 18:04 d14

drwx------ 2 root root 69632 sep 16 18:04 d15

drwx------ 2 root root 61440 sep 16 18:04 d16

drwx------ 2 root root 61440 sep 16 18:04 d17

drwx------ 2 root root 69632 sep 16 18:04 d18

drwx------ 2 root root 69632 sep 16 18:04 d19

drwx------ 2 root root 65536 sep 16 16:25 d2

drwx------ 2 root root 69632 sep 16 18:04 d20

drwx------ 2 root root 69632 sep 16 18:04 d21

drwx------ 2 root root 61440 sep 16 18:04 d22

drwx------ 2 root root 69632 sep 16 18:04 d23

drwx------ 2 root root 61440 sep 16 16:11 d24

drwx------ 2 root root 69632 sep 16 16:11 d25

drwx------ 2 root root 69632 sep 16 16:11 d26

drwx------ 2 root root 69632 sep 16 16:11 d27

drwx------ 2 root root 69632 sep 16 16:25 d28

drwx------ 2 root root 69632 sep 16 16:25 d29

drwx------ 2 root root 69632 sep 16 16:25 d3

drwx------ 2 root root 65536 sep 16 16:25 d30

drwx------ 2 root root 65536 sep 16 16:25 d31

drwx------ 2 root root 69632 sep 16 16:25 d4

drwx------ 2 root root 61440 sep 16 16:25 d5

drwx------ 2 root root 69632 sep 16 16:25 d6

drwx------ 2 root root 73728 sep 16 16:25 d7

drwx------ 2 root root 65536 sep 16 17:46 d8

drwx------ 2 root root 69632 sep 16 17:46 d9

-rw-r--r-- 1 root root     8 ene  4  2016 LAST_ID

[root at jaopost-mds d0]# ls -ltr | more

total 5865240

-rw-r--r-- 1 root root  252544 ene  4  2016 32

-rw-r--r-- 1 root root 2396224 ene  9  2016 2720

-rw-r--r-- 1 root root 4153280 ene  9  2016 2752

-rw-r--r-- 1 root root 4153280 ene 10  2016 2784

-rw-r--r-- 1 root root 4153280 ene 10  2016 2816

-rw-r--r-- 1 root root 4153280 ene 10  2016 2848

-rw-r--r-- 1 root root 4153280 ene 10  2016 2880

-rw-r--r-- 1 root root 4153280 ene 10  2016 2944

-rw-r--r-- 1 root root 4153280 ene 10  2016 2976

-rw-r--r-- 1 root root 4153280 ene 10  2016 3008

-rw-r--r-- 1 root root 4153280 ene 10  2016 3040

-rw-r--r-- 1 root root 4153280 ene 10  2016 3072

-rw-r--r-- 1 root root 4153280 ene 10  2016 3104

-rw-r--r-- 1 root root 4153280 ene 10  2016 3136

-rw-r--r-- 1 root root 4153280 ene 10  2016 3168

-rw-r--r-- 1 root root 4153280 ene 10  2016 3200

-rw-r--r-- 1 root root 4153280 ene 10  2016 3232

-rw-r--r-- 1 root root 4153280 ene 10  2016 3264

-rw-r--r-- 1 root root 4153280 ene 10  2016 3296

-rw-r--r-- 1 root root 4153280 ene 10  2016 3328

_______________________________________________
lustre-discuss mailing list
lustre-discuss at lists.lustre.org<mailto:lustre-discuss at lists.lustre.org>
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20160929/4694fb09/attachment-0001.htm>