Subject: Re: [Lustre-discuss] Broken client
From: Herbert Fruchtl &lt;[email-masked]&gt;
To: Kevin Van Maren &lt;[email-masked]&gt;
Date: Thu, 18 Nov 2010 17:56:53 +0000

Hi Kevin,

That didn&#x27;t change anything. Umounting the of the OSTs hung (yes, with an LBUG), 
and I did a hard reboot. It came up again, and the status is as before: on the 
MDT server, I can see all files (well, I assume it&#x27;s all); on the client in 
question some files appear broken. The OST is still &quot;not healthy&quot;. I am running 
another lfsck, without much hope. Here&#x27;s the LBUG:

Nov 18 17:05:16 oss1-fs kernel: LustreError: 
8125:0:(lprocfs_status.c:865:lprocfs_free_client_stats()) LBU

   Herbert

Kevin Van Maren wrote:
&gt; Reboot the server with the unhealthy OST.
&gt; 
&gt; If you look at the logs, there is likely an LBUG that is causing the 
&gt; problems.
&gt; 
&gt; Kevin
&gt; 
&gt; 
&gt; On Nov 18, 2010, at 9:51 AM, Herbert Fruchtl 
&gt; &lt;[email-masked]&gt; wrote:
&gt; 
&gt;&gt;&gt;
&gt;&gt;&gt; It looks like you may have corruption on the mdt or an ost, where the
&gt;&gt;&gt; objects on an OST can&#x27;t be found for the directory entry. Have you
&gt;&gt;&gt; had a crash recently or run Lustre fsck? You might need to do fsck and
&gt;&gt;&gt; delete (unlink) the &quot;broken&quot; files.
&gt;&gt;&gt;
&gt;&gt; The files do exist (I can see them on the mdt server) and I don&#x27;t want 
&gt;&gt; to delete
&gt;&gt; them. There was a crash lately, and I have run an lfsck afterwards 
&gt;&gt; (repeatedly,
&gt;&gt; actually.
&gt;&gt;
&gt;&gt;&gt; I suppose it&#x27;s also possible you&#x27;re seeing fallout from an earlier 
&gt;&gt;&gt; LBUG or
&gt;&gt;&gt; something. Try &#x27;cat /proc/fs/lustre/health_check&#x27; on all the servers.
&gt;&gt;&gt;
&gt;&gt; There seems to be a problem:
&gt;&gt; [root@master ~]# cat /proc/fs/lustre/health_check
&gt;&gt; healthy
&gt;&gt; [root@master ~]# ssh oss1 &#x27;cat /proc/fs/lustre/health_check&#x27;
&gt;&gt; device home-OST0005 reported unhealthy
&gt;&gt; NOT HEALTHY
&gt;&gt; [root@master ~]# ssh oss2 &#x27;cat /proc/fs/lustre/health_check&#x27;
&gt;&gt; healthy
&gt;&gt; [root@master ~]# ssh oss3 &#x27;cat /proc/fs/lustre/health_check&#x27;
&gt;&gt; healthy
&gt;&gt;
&gt;&gt; What do I do about the unhealthy OST?
&gt;&gt;
&gt;&gt;  Herbert
&gt;&gt; -- 
&gt;&gt; Herbert Fruchtl
&gt;&gt; Senior Scientific Computing Officer
&gt;&gt; School of Chemistry, School of Mathematics and Statistics
&gt;&gt; University of St Andrews
&gt;&gt; -- 
&gt;&gt; The University of St Andrews is a charity registered in Scotland:
&gt;&gt; No SC013532

-- 
Herbert Fruchtl
Senior Scientific Computing Officer
School of Chemistry, School of Mathematics and Statistics
University of St Andrews
--
The University of St Andrews is a charity registered in Scotland:
No SC013532