[lustre-discuss] MDT partition getting full

[Mohr Jr, Richard Frank (Rick Mohr)]

> On Apr 29, 2015, at 6:23 PM, Mohr Jr, Richard Frank (Rick Mohr) <rmohr at utk.edu> wrote:
> 
> 
>> On Apr 29, 2015, at 6:01 AM, Alexander Zarochentsev <alexander.zarochentsev at seagate.com> wrote:
>> 
>> 
>> can you increase mds parameters osp.*.max_rpcs_in_progress from 4096
>> (default) to, say, 65536, and check whether those llog files are being
>> deleted faster?
>> 
> 
> I increased the value, and I will let you know what happens.


I checked the current state of the files, and this is what I am seeing in the O/1/d14 directory:

 58129  100644 (1)      0      0    8256 13-May-2014 16:51 14
  58162  100644 (1)      0      0    8256 13-May-2014 16:51 46
  58197  100644 (1)      0      0    8256 13-May-2014 16:51 78
  58237  100644 (1)      0      0   38720 13-May-2014 19:28 110
  58271  100644 (1)      0      0   39488 13-May-2014 19:28 142
  58305  100644 (1)      0      0   38912 13-May-2014 19:28 174
  58343  100644 (1)      0      0   38272 13-May-2014 19:28 206
  58396  100644 (1)      0      0   38400 13-May-2014 19:28 238
  58429  100644 (1)      0      0   37184 13-May-2014 19:28 270
    179  100644 (17)      0      0   4153280 24-Apr-2015 04:14 43278
    188  100644 (17)      0      0   4153280 24-Apr-2015 12:03 43310
    206  100644 (17)      0      0   4153280 24-Apr-2015 18:42 43246
   1304  100644 (17)      0      0   4153280 26-Apr-2015 06:47 43630
   1285  100644 (17)      0      0   4153280 25-Apr-2015 10:17 43470
    120  100644 (17)      0      0   4153280 25-Apr-2015 16:49 43502
    202  100644 (17)      0      0   4153280 26-Apr-2015 11:53 43662
    124  100644 (17)      0      0   4153280 26-Apr-2015 20:44 43694
   1327  100644 (17)      0      0   4153280 27-Apr-2015 15:30 43822
  12671  100644 (17)      0      0   4153280 29-Apr-2015 12:53 44558
   9991  100644 (17)      0      0   4153280 27-Apr-2015 15:13 43790
  10060  100644 (17)      0      0   4153280 28-Apr-2015 04:34 43886
  10039  100644 (17)      0      0   4153280 28-Apr-2015 13:03 43918
  10019  100644 (17)      0      0   4153280 29-Apr-2015 03:01 44046
  10131  100644 (17)      0      0   4153280 29-Apr-2015 00:15 44014
  12661  100644 (17)      0      0   2562624 30-Apr-2015 15:55 44622
  12724  100644 (17)      0      0   1222528  1-May-2015 03:11 44654

New files are still being created, and the old ones are not being deleted.  However, after making the change, it looks like the two most recent files are not as large as the previous ones.  I don’t know if this is coincidence or a direct result of the new setting.  

Of course, there is still the question of where these files are coming from and why they aren’t going away.

--
Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences
http://www.nics.tennessee.edu