[lustre-discuss] OST doomed after e2fsck

Riccardo Veraldi Riccardo.Veraldi at cnaf.infn.it
Thu May 31 07:56:00 PDT 2018


thanks a lot! it worked.

On 5/31/18 12:14 AM, Fernando Pérez wrote:
> I had the same problem in the past with 2.4 release.
>
> I solved the problem upgrading the e2fsprogs to the its latest release
> and running again e2fsck in the failed OST.
>
> Regards.
>
> ============================================
> Fernando Pérez
> Institut de Ciències del Mar (CMIMA-CSIC)
> Departament Oceanografía Física i Tecnològica
> Passeig Marítim de la Barceloneta,37-49
> 08003 Barcelona
> Phone:  (+34) 93 230 96 35 <tel:%28+34%29%2093%20230%2096%2035>
> ============================================
>
> El 31 may 2018, a las 5:36, Riccardo Veraldi
> <Riccardo.Veraldi at cnaf.infn.it <mailto:Riccardo.Veraldi at cnaf.infn.it>>
> escribió:
>
>> Hello,
>>
>> after a power outage I had one of my OSTs (total of 60) in an unhappy
>> state.
>>
>> Lustre version 2.4.1
>>
>> I ran then a FS check and here follows:
>>
>> e2fsck 1.42.7.wc1 (12-Apr-2013)
>> Pass 1: Checking inodes, blocks, and sizes
>> Pass 2: Checking directory structure
>> Pass 3: Checking directory connectivity
>> Pass 4: Checking reference counts
>> Unattached inode 25793
>> Connect to /lost+found<y>? yes
>> Inode 25793 ref count is 2, should be 1.  Fix<y>? yes
>> Unattached inode 29096
>> Connect to /lost+found<y>? yes
>> Inode 29096 ref count is 2, should be 1.  Fix<y>? yes
>> Unattached inode 29745
>> Connect to /lost+found<y>? yes
>> Inode 29745 ref count is 2, should be 1.  Fix<y>? yes
>> Unattached inode 29821
>> Connect to /lost+found<y>? yes
>> Inode 29821 ref count is 2, should be 1.  Fix<y>? yes
>> yPass 5: Checking group summary information
>> Inode bitmap differences:  +23902 +29082 +29096 +29130 +29459 +29497
>> -29530 +29552 +29566 +29596 +(29643--29644) +29655 +29668 +29675 +29696
>> +29701 +29720 +29736 +29739 +29745 +29751 +29778 +29787 -29795 +29808
>> +29821
>> Fix<y>? yes
>> Free inodes count wrong for group #70 (1, counted=0).
>> Fix<y>? yes
>> Free inodes count wrong for group #76 (1, counted=0).
>> Fix<y>? yes
>> Free inodes count wrong for group #90 (1, counted=0).
>> Fix<y>? yes
>> Free inodes count wrong for group #93 (3, counted=2).
>> Fix<y>? yes
>> Free inodes count wrong for group #100 (2, counted=0).
>> Fix<y>? yes
>> Free inodes count wrong for group #101 (1, counted=0).
>> Fix<y>? yes
>> Free inodes count wrong for group #113 (5, counted=2).
>> Fix<y>? yes
>> Free inodes count wrong for group #114 (1, counted=0).
>> Fix<y>? yes
>> Free inodes count wrong for group #115 (13, counted=4).
>> Fix<y>? yes
>> Free inodes count wrong for group #116 (149, counted=140).
>> Fix<y>? yes
>> Free inodes count wrong (30493545, counted=30493516).
>> Fix<y>? yes
>> [QUOTA WARNING] Usage inconsistent for ID 0:actual (2083721216, 841) !=
>> expected (2082398208, 678)
>> [QUOTA WARNING] Usage inconsistent for ID 9997:actual (1095815659520,
>> 19800) != expected (664375967744, 19791)
>> [QUOTA WARNING] Usage inconsistent for ID -1597706240:actual (0, 0) !=
>> expected (90112, 1)
>> [QUOTA WARNING] Usage inconsistent for ID -1428439040:actual (0, 0) !=
>> expected (126976, 1)
>> [QUOTA WARNING] Usage inconsistent for ID -1936064512:actual (0, 0) !=
>> expected (12288, 1)
>> [QUOTA WARNING] Usage inconsistent for ID -1684783104:actual (0, 0) !=
>> expected (28672, 1)
>> [QUOTA WARNING] Usage inconsistent for ID -2131947520:actual (0, 0) !=
>> expected (4096, 1)
>> [QUOTA WARNING] Usage inconsistent for ID 963263424:actual (957718528,
>> 49) != expected (957628416, 48)
>> [QUOTA WARNING] Usage inconsistent for ID 987173056:actual (1364516864,
>> 158) != expected (1364426752, 157)
>> [QUOTA WARNING] Usage inconsistent for ID -1537871872:actual (0, 0) !=
>> expected (73728, 1)
>> [QUOTA WARNING] Usage inconsistent for ID -2105077760:actual (0, 0) !=
>> expected (49152, 1)
>> [QUOTA WARNING] Usage inconsistent for ID -2145202176:actual (0, 0) !=
>> expected (24576, 1)
>> [QUOTA WARNING] Usage inconsistent for ID -1422704640:actual (0, 0) !=
>> expected (65536, 1)
>> Update quota info for quota type 0<y>? yes
>> [ERROR] quotaio_tree.c:357:free_dqentry:: Quota structure has offset to
>> other block (0) than it should (472).
>> [ERROR] quotaio_tree.c:357:free_dqentry:: Quota structure has offset to
>> other block (0) than it should (507).
>> [ERROR] quotaio_tree.c:357:free_dqentry:: Quota structure has offset to
>> other block (0) than it should (170).
>> [ERROR] quotaio_tree.c:357:free_dqentry:: Quota structure has offset to
>> other block (0) than it should (435).
>> [ERROR] quotaio_tree.c:357:free_dqentry:: Quota structure has offset to
>> other block (0) than it should (89).
>> [ERROR] quotaio_tree.c:357:free_dqentry:: Quota structure has offset to
>> other block (0) than it should (5).
>> [ERROR] quotaio_tree.c:357:free_dqentry:: Quota structure has offset to
>> other block (0) than it should (130).
>> [ERROR] quotaio_tree.c:357:free_dqentry:: Quota structure has offset to
>> other block (0) than it should (435).
>> [ERROR] quotaio_tree.c:357:free_dqentry:: Quota structure has offset to
>> other block (0) than it should (251).
>> [QUOTA WARNING] Usage inconsistent for ID 0:actual (8301957120, 843) !=
>> expected (5880315904, 677)
>> [QUOTA WARNING] Usage inconsistent for ID 2279:actual (14819280969728,
>> 21842) != expected (14298746671104, 21705)
>> Update quota info for quota type 1<y>? yes
>>
>> ana01-OST000e: ***** FILE SYSTEM WAS MODIFIED *****
>> ana01-OST000e: 29876/30523392 files (22.3% non-contiguous),
>> 3670668872/3906963456 blocks
>>
>>
>> After this when trying to mount the OST again, it makes the lustre
>> kernel module hang and Linux kernel goes to panic.
>> it is reproducible, every time I try to mount the OST fixed with e2fsck.
>> So basically I lost all data on the OST.
>> Any hints on how could I recover it ?
>> thank you.
>>
>> Rick
>>
>>
>>
>>
>>
>> _______________________________________________
>> lustre-discuss mailing list
>> lustre-discuss at lists.lustre.org <mailto:lustre-discuss at lists.lustre.org>
>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20180531/1c64562b/attachment-0001.html>


More information about the lustre-discuss mailing list