[lustre-discuss] MDT partition getting full

[Alexander Zarochentsev]

Hello,

On Thu, Apr 23, 2015 at 9:01 PM, Mohr Jr, Richard Frank (Rick Mohr)
<rmohr at utk.edu> wrote:
>
>> On Apr 23, 2015, at 1:07 PM, Colin Faber <cfaber at gmail.com> wrote:
>>
>>
>> Based on the directory structure here, this appears to be an OST. are you sure your targets are correctly named?
>>
>
> That is what I would have guessed until I took a look at my own MDT.  Sure enough, I have the directories /O/1/d[0-31] and each one seems to have 3 files that are about 3.5MB each (along with some other smaller ones).  Here is what one of those directories looks like:
>
> debugfs:  ls -l /O/1/d14
>  16777293   40700 (2)      0      0    4096 20-Apr-2015 14:09 .
>  16777278   40755 (2)      0      0    4096 13-May-2014 16:51 ..
>   58129  100644 (1)      0      0    8256 13-May-2014 16:51 14
>   58162  100644 (1)      0      0    8256 13-May-2014 16:51 46
>   58197  100644 (1)      0      0    8256 13-May-2014 16:51 78
>   58237  100644 (1)      0      0   37632 13-May-2014 19:28 110
>   58271  100644 (1)      0      0   38464 13-May-2014 19:28 142
>   58305  100644 (1)      0      0   37888 13-May-2014 19:28 174
>   58343  100644 (1)      0      0   37184 13-May-2014 19:28 206
>   58396  100644 (1)      0      0   37312 13-May-2014 19:28 238
>   58429  100644 (1)      0      0   36160 13-May-2014 19:28 270
>   12824  100644 (1)      0      0   3623232 20-Apr-2015 14:09 43150
>   12915  100644 (1)      0      0   3800960 20-Apr-2015 14:09 43182
>   12954  100644 (1)      0      0   3769216 20-Apr-2015 14:09 43214
>
> The three large files seem to have been created the last time the MDT was mounted.  The timestamps for the other smaller files coincides with the Lustre upgrade we performed last year.  But I am not sure what is contained in these files.

can you do "debugfs dump" for one of those 4MB files , run llog_reader
(utility from lustre sources) over it and send the output to the list?

BTW, how long the MDS running without remounts or restarts?

Thanks,

>
> Radu: Are the timestamps for all of your files the same?  Or is the system gradually accumulating them over time for some reason?
>
> --
> Rick Mohr
> Senior HPC System Administrator
> National Institute for Computational Sciences
> http://www.nics.tennessee.edu
>
> _______________________________________________
> lustre-discuss mailing list
> lustre-discuss at lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org



-- 
Alexander Zarochentsev
Seagate Technology, LLC
www.seagate.com