X-Mozilla-Keys: 
Message-ID: <4CE568E5.8050006@st-andrews.ac.uk>
Date: Thu, 18 Nov 2010 17:56:53 +0000
From: Herbert Fruchtl <herbert.fruchtl@st-andrews.ac.uk>
User-Agent: Thunderbird 2.0.0.22 (X11/20090625)
MIME-Version: 1.0
To: Kevin Van Maren <Kevin.Van.Maren@oracle.com>
Subject: Re: [Lustre-discuss] Broken client
References: <4CE5199E.6020904@st-andrews.ac.uk>
	<1BCB6A84-E338-4195-8AD3-FE582ECCA189@oracle.com>
	<4CE5323D.2060505@st-andrews.ac.uk>
	<66CA90D4-9DEB-49AF-8410-0376AF22879D@oracle.com>
	<4CE5499C.7070503@oracle.com> <4CE54B7A.8000000@st-andrews.ac.uk>
	<2E3489B3-FB97-4B1A-93F9-7F1DD8D933E8@oracle.com>
In-Reply-To: <2E3489B3-FB97-4B1A-93F9-7F1DD8D933E8@oracle.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Hi Kevin,

That didn't change anything. Umounting the of the OSTs hung (yes, with an LBUG), 
and I did a hard reboot. It came up again, and the status is as before: on the 
MDT server, I can see all files (well, I assume it's all); on the client in 
question some files appear broken. The OST is still "not healthy". I am running 
another lfsck, without much hope. Here's the LBUG:

Nov 18 17:05:16 oss1-fs kernel: LustreError: 
8125:0:(lprocfs_status.c:865:lprocfs_free_client_stats()) LBU

   Herbert

Kevin Van Maren wrote:
> Reboot the server with the unhealthy OST.
> 
> If you look at the logs, there is likely an LBUG that is causing the 
> problems.
> 
> Kevin
> 
> 
> On Nov 18, 2010, at 9:51 AM, Herbert Fruchtl 
> <herbert.fruchtl@st-andrews.ac.uk> wrote:
> 
>>>
>>> It looks like you may have corruption on the mdt or an ost, where the
>>> objects on an OST can't be found for the directory entry. Have you
>>> had a crash recently or run Lustre fsck? You might need to do fsck and
>>> delete (unlink) the "broken" files.
>>>
>> The files do exist (I can see them on the mdt server) and I don't want 
>> to delete
>> them. There was a crash lately, and I have run an lfsck afterwards 
>> (repeatedly,
>> actually.
>>
>>> I suppose it's also possible you're seeing fallout from an earlier 
>>> LBUG or
>>> something. Try 'cat /proc/fs/lustre/health_check' on all the servers.
>>>
>> There seems to be a problem:
>> [root@master ~]# cat /proc/fs/lustre/health_check
>> healthy
>> [root@master ~]# ssh oss1 'cat /proc/fs/lustre/health_check'
>> device home-OST0005 reported unhealthy
>> NOT HEALTHY
>> [root@master ~]# ssh oss2 'cat /proc/fs/lustre/health_check'
>> healthy
>> [root@master ~]# ssh oss3 'cat /proc/fs/lustre/health_check'
>> healthy
>>
>> What do I do about the unhealthy OST?
>>
>>  Herbert
>> -- 
>> Herbert Fruchtl
>> Senior Scientific Computing Officer
>> School of Chemistry, School of Mathematics and Statistics
>> University of St Andrews
>> -- 
>> The University of St Andrews is a charity registered in Scotland:
>> No SC013532

-- 
Herbert Fruchtl
Senior Scientific Computing Officer
School of Chemistry, School of Mathematics and Statistics
University of St Andrews
--
The University of St Andrews is a charity registered in Scotland:
No SC013532