<div dir="ltr">Hi,<div><br></div><div>There was a bug (sorry don't recall which) that would leave the llog files present and full of logs which should have been cleared, remount was the solution. I don't recall the details here but I'm sure you'd be able to find the LU ticket after some searching.</div><div><br></div><div>-cf</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Oct 15, 2016 at 3:49 PM, Pawel Dziekonski <span dir="ltr"><<a href="mailto:dzieko@wcss.pl" target="_blank">dzieko@wcss.pl</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<br>
we had the same problem on 2.5.3. Robinhood was supposed to<br>
consume changelog but it wasn't. Don't know why. Simply<br>
disabling changelog was not enough - we had to remount MDT.<br>
We did it by simply doing failover to other MDS node (HA<br>
pair).<br>
<br>
The other issue we had with MDT was the size of inodes -<br>
they are (were at that time) created with 512 bytes by<br>
default and when you use the stripe count then it will not<br>
accommodate the lfsck and xattr data on that single inode<br>
and it starts utilizing the disk space. So you have to<br>
create the inodes with proper size, then whole data will be<br>
saved in that same inode and will not occupy additional disk<br>
space. AFAIK, this is a known issue since 2.x.<br>
Unfortunately the only solution was to reformat MDT offline.<br>
<br>
P<br>
<br>
<br>
<br>
<br>
(via failover for<br>
<span class="">On pią, 14 paź 2016 at 06:46:59 -0400, Jessica Otey wrote:<br>
> All,<br>
> My colleagues in Chile now believe that both of their 2.5.3 file<br>
> systems are experiencing this same problem with the MDTs filling up<br>
> with files. We have also come across a report from another user from<br>
> early 2015 denoting the same issue, also with a 2.5.3 system.<br>
><br>
</span>> See: <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__www.mail-2Darchive.com_search-3Fl-3Dlustre-2Ddiscuss-40lists.lustre.org-26q-3Dsubject-3A-2522Re-255C-253A-2B-255C-255Blustre-255C-2Ddiscuss-255C-255D-2BMDT-2Bpartition-2Bgetting-2Bfull-2522-26o-3Dnewest&d=DQIGaQ&c=IGDlg0lD0b-nebmJJ0Kp8A&r=x9pM59OqndbWw-lPPdr8w1Vud29EZigcxcNkz0uw5oQ&m=hrnhInO0YKCrI7g-bxkr6YelhXQXywcc8cF4G8x3rR4&s=ulbEYTQ5HJrTLe_tNQ3LdH_Ylc_qwxZOWD3MlN_a5Bc&e=" rel="noreferrer" target="_blank">https://urldefense.proofpoint.<wbr>com/v2/url?u=https-3A__www.<wbr>mail-2Darchive.com_search-3Fl-<wbr>3Dlustre-2Ddiscuss-40lists.<wbr>lustre.org-26q-3Dsubject-3A-<wbr>2522Re-255C-253A-2B-255C-<wbr>255Blustre-255C-2Ddiscuss-<wbr>255C-255D-2BMDT-2Bpartition-<wbr>2Bgetting-2Bfull-2522-26o-<wbr>3Dnewest&d=DQIGaQ&c=<wbr>IGDlg0lD0b-nebmJJ0Kp8A&r=<wbr>x9pM59OqndbWw-<wbr>lPPdr8w1Vud29EZigcxcNkz0uw5oQ&<wbr>m=hrnhInO0YKCrI7g-<wbr>bxkr6YelhXQXywcc8cF4G8x3rR4&s=<wbr>ulbEYTQ5HJrTLe_tNQ3LdH_Ylc_<wbr>qwxZOWD3MlN_a5Bc&e=</a><br>
<span class="">><br>
> We are confident that these files are not related to the changelog feature.<br>
><br>
> Does anyone have any other suggestions as to what the cause of this<br>
> problem could be?<br>
><br>
> I'm intrigued that the Lustre version involved in all 3 reports is<br>
> 2.5.3. Could this be a bug?<br>
><br>
> Thanks,<br>
> Jessica<br>
><br>
><br>
> >On Thu, Sep 29, 2016 at 8:58 AM, Jessica Otey <<a href="mailto:jotey@nrao.edu">jotey@nrao.edu</a><br>
</span><div><div class="h5">> ><mailto:<a href="mailto:jotey@nrao.edu">jotey@nrao.edu</a>>> wrote:<br>
> ><br>
> > Hello all,<br>
> > I write on behalf of my colleagues in Chile, who are experiencing<br>
> > a bizarre problem with their MDT, namely, it is filling up with 4<br>
> > MB files. There is no issue with the number of inodes, of which<br>
> > there are hundreds of millions unused. Â<br>
> ><br>
> > [root@jaopost-mds ~]# tune2fs -l /dev/sdb2 | grep -i inode<br>
> > device /dev/sdb2 mounted by lustre<br>
> > Filesystem features: Â Â Â has_journal ext_attr resize_inode<br>
> > dir_index filetype needs_recovery flex_bg dirdata sparse_super<br>
> > large_file huge_file uninit_bg dir_nlink quota<br>
> > Inode count: Â Â Â Â Â Â Â 239730688<br>
> > Free inodes: Â Â Â Â Â Â Â 223553405<br>
> > Inodes per group: Â Â Â Â 32768<br>
> > Inode blocks per group: Â 4096<br>
> > First inode: Â Â Â Â Â Â Â 11<br>
> > Inode size:Â Â Â Â Â 512<br>
> > Journal inode: Â Â Â Â Â Â 8<br>
> > Journal backup: Â Â Â Â Â inode blocks<br>
> > User quota inode: Â Â Â Â 3<br>
> > Group quota inode: Â Â Â Â 4<br>
> ><br>
> > Has anyone ever encountered such a problem? The only thing unusual<br>
> > about this cluster is that it is using 2.5.3 MDS/OSSes while still<br>
> > using 1.8.9 clients—something I didn't actually believe was<br>
> > possible, as I thought the last version to work effectively with<br>
> > 1.8.9 clients was 2.4.3. However, for all I know, the version gap<br>
> > may have nothing to do with this phenomena.<br>
> ><br>
> > Any and all advice is appreciated. Any general information on the<br>
> > structure of the MDT also welcome, as such info is in short supply<br>
> > on the internet.<br>
> ><br>
> > Thanks,<br>
> > Jessica<br>
> ><br>
<br>
</div></div>--<br>
Pawel Dziekonski <<a href="mailto:pawel.dziekonski@wcss.pl">pawel.dziekonski@wcss.pl</a>><br>
Wroclaw Centre for Networking & Supercomputing, HPC Department<br>
phone: <a href="tel:%2B48%2071%20320%2037%2039" value="+48713203739">+48 71 320 37 39</a>, fax: <a href="tel:%2B48%2071%20322%2057%2097" value="+48713225797">+48 71 322 57 97</a>, <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__www.wcss.pl&d=DQIGaQ&c=IGDlg0lD0b-nebmJJ0Kp8A&r=x9pM59OqndbWw-lPPdr8w1Vud29EZigcxcNkz0uw5oQ&m=hrnhInO0YKCrI7g-bxkr6YelhXQXywcc8cF4G8x3rR4&s=Ob17Q3c2GcDNu-2FIvqyDwKxVQ-u7xeBzVTMo9we3XI&e=" rel="noreferrer" target="_blank">https://urldefense.proofpoint.<wbr>com/v2/url?u=http-3A__www.<wbr>wcss.pl&d=DQIGaQ&c=IGDlg0lD0b-<wbr>nebmJJ0Kp8A&r=x9pM59OqndbWw-<wbr>lPPdr8w1Vud29EZigcxcNkz0uw5oQ&<wbr>m=hrnhInO0YKCrI7g-<wbr>bxkr6YelhXQXywcc8cF4G8x3rR4&s=<wbr>Ob17Q3c2GcDNu-2FIvqyDwKxVQ-<wbr>u7xeBzVTMo9we3XI&e=</a><br>
<span class="">______________________________<wbr>_________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.<wbr>org</a><br>
</span><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.lustre.org_listinfo.cgi_lustre-2Ddiscuss-2Dlustre.org&d=DQIGaQ&c=IGDlg0lD0b-nebmJJ0Kp8A&r=x9pM59OqndbWw-lPPdr8w1Vud29EZigcxcNkz0uw5oQ&m=hrnhInO0YKCrI7g-bxkr6YelhXQXywcc8cF4G8x3rR4&s=eHOc6uHZuJVxo_TiHizVTzK4FmtxLwtp0jQjbKWFGK8&e=" rel="noreferrer" target="_blank">https://urldefense.proofpoint.<wbr>com/v2/url?u=http-3A__lists.<wbr>lustre.org_listinfo.cgi_<wbr>lustre-2Ddiscuss-2Dlustre.org&<wbr>d=DQIGaQ&c=IGDlg0lD0b-<wbr>nebmJJ0Kp8A&r=x9pM59OqndbWw-<wbr>lPPdr8w1Vud29EZigcxcNkz0uw5oQ&<wbr>m=hrnhInO0YKCrI7g-<wbr>bxkr6YelhXQXywcc8cF4G8x3rR4&s=<wbr>eHOc6uHZuJVxo_<wbr>TiHizVTzK4FmtxLwtp0jQjbKWFGK8&<wbr>e=</a><br>
</blockquote></div><br></div>