[Lustre-discuss] MDS overload, why?

[Arne Brutschy]

Hi,

thanks for replying!

I understand that without further information we can't do much about the
oopses. I was more hoping for some information regarding possible
sources of such an overload. Is it normal that a MDS gets overloaded
like this, while the OSTs have nothing to do, and what can I do about
it? How can I find the source of the problem?

More specifically, what are the operations that lead to a lot of MDS
load and none for the OSTs? Although our MDS (8GB ram, 2x4core, SATA) is
not a top-notch server, it's fairly recent and I feel the load we're
experiencing is not handable by a single MDS.

My problem is that I can't make out major problems in the user's jobs
running on the cluster, and I can't quantify nor track down the problem
because I don't know what behavior might have caused it. 

As I said, ooppses appeared only twice, and all other problems where
just apparent by a non-responsive MDS.

Thanks,
Arne


On Fr, 2009-10-09 at 07:44 -0400, Brian J. Murrell wrote:
> On Fri, 2009-10-09 at 10:26 +0200, Arne Brutschy wrote:
> > 
> > The clients showed the following error:
> > > Oct  8 09:58:55 majorana kernel: LustreError: 3787:0:(events.c:66:request_out_callback()) @@@ type 4, status -5  req at f6222800 x8702488/t0 o250->MGS at 10.255.255.206@tcp:26/25 lens 304/456 e 0 to 1 dl 1254988740 ref 2 fl Rpc:N/0/0 rc 0/0
> > > Oct  8 09:58:55 majorana kernel: LustreError: 3787:0:(events.c:66:request_out_callback()) Skipped 33 previous similar messages
> > 
> > So, my question is: what could cause such a load? The cluster was not
> > exessively used... Is this a bug or a user's job that creates the load?
> > How can I protect lustre against this kind of failure?
> 
> Without any more information we could not possibly know.  If you really
> are getting oopses then you will need console logs (i.e. serial console)
> so that we can see the stack trace.
> 
> b.
> 
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
-- 
Arne Brutschy
Ph.D. Student                    Email    arne.brutschy(AT)ulb.ac.be
IRIDIA CP 194/6                  Web      iridia.ulb.ac.be/~abrutschy
Universite' Libre de Bruxelles   Tel      +32 2 650 3168
Avenue Franklin Roosevelt 50     Fax      +32 2 650 2715
1050 Bruxelles, Belgium          (Fax at IRIDIA secretary)