<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hi folks,</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Just going to piggy back on this thread as we are experiencing exactly the same thing. We're ldiskfs, though, not zfs. We were 2.10.3 on all servers and, when this first occurred, it brought down our MDT with the following ASSERT:</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Apr 10 01:22:09 hpcmds01.adqimr.ad.lan kernel: LustreError: 183448:0:(osp_precreate.c:634:osp_precreate_send()) qimrb-OST008f-osc-MDT0000: precreate fid [0x1008f0000:0xa65bd3f:0x0] < local used fid [0x1008f0000:0xa65bd3f:0x0]:
<div>Apr 10 01:22:09 hpcmds01.adqimr.ad.lan kernel: LustreError: 57275:0:(osp_precreate.c:1311:osp_precreate_ready_condition()) qimrb-OST008f-osc-MDT0000: precreate failed opd_pre_status -116</div>
<div>Apr 10 01:22:09 hpcmds01.adqimr.ad.lan kernel: LustreError: 183448:0:(osp_precreate.c:1259:osp_precreate_thread()) qimrb-OST008f-osc-MDT0000: cannot precreate objects: rc = -116</div>
<div>Apr 10 01:22:09 hpcmds01.adqimr.ad.lan kernel: LustreError: 8475:0:(lod_qos.c:1624:lod_alloc_qos()) ASSERTION( nfound <= inuse->op_count ) failed: nfound:19, op_count:0</div>
Apr 10 01:22:09 hpcmds01.adqimr.ad.lan kernel: LustreError: 8475:0:(lod_qos.c:1624:lod_alloc_qos()) LBUG<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Attempts to remount the MDT resulted in repeated crashes. Thinking it was <a href="https://jira.whamcloud.com/browse/LU-10297" id="LPlnk626389">https://jira.whamcloud.com/browse/LU-10297</a>, we brought the MDS up to 2.10.4 and were immediately bit by <a href="https://jira.whamcloud.com/browse/LU-11227" id="LPlnk">https://jira.whamcloud.com/browse/LU-11227</a>,
as we have deactivated OSTs so we quickly upgraded MDS/MGS to 2.10.6. We're now still seeing the -52/-116 and we have three OSTs that we similarly can't create objects on with explicit "lfs setstripe -i". OSSs are still on 2.10.3.</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Not sure if I should reply to Marco's request for the various node list, "lfs df" and "getparam"s here, or open up a jira ticket. Leaning towards the latter but I'll spend some time in jira today to ensure it's not a duplicate, first.</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
We're currently up and running again but are looking to resolve the remaining unusable OSTs. And, like Amit, we're working towards a 2.12 upgrade in the near future but we just haven't got there yet. </div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Cheers,</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Scott</div>
<br>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> lustre-discuss <lustre-discuss-bounces@lists.lustre.org> on behalf of Marco Grossi <marco.grossi@ichec.ie><br>
<b>Sent:</b> Tuesday, 10 March 2020 9:22 PM<br>
<b>To:</b> Kumar, Amit <ahkumar@mail.smu.edu><br>
<b>Cc:</b> lustre-discuss@lists.lustre.org <lustre-discuss@lists.lustre.org><br>
<b>Subject:</b> Re: [lustre-discuss] unable to precreate -52/-116</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt;">
<div class="PlainText">Hi Amit,<br>
<br>
Sounds definitely different from my case.<br>
<br>
The only JIRA issue logging a "precreate fid < local used fid" is:<br>
<a href="https://jira.whamcloud.com/browse/LU-11536">https://jira.whamcloud.com/browse/LU-11536</a><br>
<br>
What puzzle me is the "rc = -52" on the "ofd_create_hdl"; if I mapped it<br>
correctly, is a -EBADE error, i.e. "invalid exchange".<br>
<br>
Can you provide:<br>
- HA node list and location of MGS, MDT and OST between nodes<br>
<br>
As well as the output of:<br>
- lfs df<br>
- lfs df -i<br>
- lctl get_param osp.*scratch0-OST0029*.prealloc*<br>
- lctl get_param obdfilter.*scratch0-OST0029*.last_id<br>
<br>
Regards,<br>
Marco<br>
<br>
<br>
On 3/9/20 5:23 PM, Kumar, Amit wrote:<br>
> Hi Marco,<br>
> <br>
> Thank you for the response on this issue. <br>
> <br>
> We have an HA setup, I tried to fail over MDT to the secondary pair and then fail it back. This did not help.
<br>
> I also tried restart of the MDS servers, that did not help.<br>
> I have rebooted OSS servers as well, that did not help<br>
> I also tried completely stopping MDS and unmounting MDS for a little while and that did not help either.
<br>
> <br>
> This error ritually comes back right after MDT is mounted. Additionally I am not able to manually create any files on that particular OST. Any other thoughts.
<br>
> <br>
> Thank you,<br>
> Amit<br>
> <br>
> -----Original Message-----<br>
> From: Marco Grossi <marco.grossi@ichec.ie> <br>
> Sent: Monday, March 9, 2020 11:23 AM<br>
> To: Kumar, Amit <ahkumar@mail.smu.edu><br>
> Cc: lustre-discuss@lists.lustre.org<br>
> Subject: Re: [lustre-discuss] unable to precreate -52/-116<br>
> <br>
> Hi Amit,<br>
> <br>
> We had a similar issue after a set_param of "max_create_count=0"<br>
> <br>
> In our case re-mounting the MDT (not the OST) fixed the issue.<br>
> <br>
> Hope it helps.<br>
> <br>
> Regards,<br>
> Marco<br>
> <br>
> <br>
> On 3/3/20 8:25 PM, Kumar, Amit wrote:<br>
>> Dear Lustre,<br>
>><br>
>> <br>
>><br>
>> Recently we had a degraded(Not failed) RAID and had to wait longer to <br>
>> get compatible disk, as we had received incompatible one and it took <br>
>> over a week to get the correct one back in place.<br>
>><br>
>> <br>
>><br>
>> During this wait I ended up disabling the OST first and then noticed <br>
>> continuous IO to the OST and thought of disabling object creation on <br>
>> it as well. Everything looked normal after that and once the disk was <br>
>> replaced I reenabled object creation and enabled OST. Since then I <br>
>> started seeing these messages on OST<br>
>><br>
>> .(ofd_dev.c:1784:ofd_create_hdl()) scratch0-OST0029: unable to<br>
>> precreate: rc = -52<br>
>><br>
>> And following messages on MDS<br>
>><br>
>> .(osp_precreate.c:1282:osp_precreate_thread())<br>
>> scratch0-OST0029-osc-MDT0000: cannot precreate objects: rc = -116<br>
>><br>
>> .(osp_precreate.c:657:osp_precreate_send())<br>
>> scratch0-OST0029-osc-MDT0000: precreate fid <br>
>> [0x100290000:0x101b39a:0x0] < local used fid <br>
>> [0x100290000:0x101b39a:0x0]: rc = -116<br>
>><br>
>> <br>
>><br>
>> These messages don't seem to stop. I am wondering what impact could <br>
>> these errors have in long run? I have noticed I am not able to create <br>
>> files on this particular OST using lfs setstripe, when I do so it gets <br>
>> me an object on another OST by default. Just want to make sure this is <br>
>> not causing any data loss for files the currently on them and new requests?<br>
>><br>
>> We plan to upgrade to 2.12 in the summer downtime and assuming that <br>
>> has a fix based on LU-9442 & LU-11186. Currently running servers on <br>
>> lustre<br>
>> 10.4.1 over ZFS-0.7.9-1<br>
>><br>
>> <br>
>><br>
>> Any help is greatly appreciated.<br>
>><br>
>> <br>
>><br>
>> Thank you,<br>
>> Amit<br>
>><br>
>><br>
>> _______________________________________________<br>
>> lustre-discuss mailing list<br>
>> lustre-discuss@lists.lustre.org<br>
>> <a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
>><br>
> <br>
> --<br>
> Marco Grossi<br>
> ICHEC Systems Team<br>
> <br>
> <br>
> ----IF CLASSIFICATION START----<br>
> <br>
> ----IF CLASSIFICATION END----<br>
> <br>
<br>
-- <br>
Marco Grossi<br>
ICHEC Systems Team<br>
_______________________________________________<br>
lustre-discuss mailing list<br>
lustre-discuss@lists.lustre.org<br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
</div>
</span></font></div>
</body>
</html>