[Lustre-discuss] Problems with MDS Crashing

[Gary Brooks]

Greg,

We are using CENT and 3ware raid adapters.    We had the MDS on Raid 0 (on
accident) and changed it out for a raid 10 machine with 3ware card.  We did
update the firmware on the 3ware cards and changed the stripe size down to
64k.   It was at 256.  After changing out the equipment the problem stopped.
   I heard that some Western Digital drives might cause strange system
locks.  Network cards are Intel Pro 1000's.

I hope this helps you.


On Tue, May 18, 2010 at 4:10 AM, Gregory Matthews <
greg.matthews at diamond.ac.uk> wrote:

> Gary Brooks wrote:
>
>> Then, all of the sudden the MDS stops responding, ssh sessions die and
>> only hard restart helps. After the restart, /var/log/messages contains
>> normal information (some timeout chit-chat).
>>
>
> is your hardware using the bnx2 NIC driver? We've just been seeing very
> similar issues on Lustre clients on brand new Dell Power Edge R610s. The
> workaround is to turn off MSI-X but there has recently been a fix merged
> into the mainline kernel which has also been backported by Red Hat.
>
>
>  While this happens randomly, there is an almost sure way to trigger it:
>> issue sysctl -w lnet.debug=0 on all clients and servers, after which the
>> file system becomes super responsive, load on MDS is still low, our gig-e
>> link is well utilized (unlike when lnet logging is enabled) and after a few
>> minutes MDS dies as described above.
>>
>
> we have not been able to trigger it in any predictable fashion either.
>
> GREG
>
>
>> I know that this is too little information to ask for help, but maybe you
>> could at least tell me where to look for any information?
>>
>> Gary
>>
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> Lustre-discuss mailing list
>> Lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>>
>
>
> --
> Greg Matthews            01235 778658
> Senior Computer Systems Administrator
> Diamond Light Source, Oxfordshire, UK
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20100518/45303d97/attachment.htm>