[Lustre-discuss] OSTs hanging while running IOR

[Atul Vidwansa]

Rafael,

Can you tell me what RAID stripe_cache_size are you using? Have you 
tuned your OSS nodes with something like:

echo 4096 > /sys/block/md0/md/stripe_cache_size

If you haven't, can you try using above command to tune all your RAID 
arrays?

Cheers,
-Atul

Rafael David Tinoco wrote:
> Forget the file.. sorry
>
> -----Original Message-----
> From: lustre-discuss-bounces at lists.lustre.org [mailto:lustre-discuss-bounces at lists.lustre.org] On Behalf Of Rafael David Tinoco
> Sent: Wednesday, September 09, 2009 7:30 PM
> To: 'Brian J. Murrell'; lustre-discuss at lists.lustre.org
> Subject: Re: [Lustre-discuss] OSTs hanging while running IOR
>
> Im attaching the messages (only the error part) file so we don't have these mail formatting problems.
>
> ------
>
> Can you provide a bit more of the log before the above so we can see what the stack trace is in reference to?  Also, try to
> eliminate the white-space between lines.  Are you getting any other errors or messages from Lustre prior to that?
>
> Perhaps you are getting some messages saying that various operations are "slow"?
>
>   
>>> Even beeing slow, the OST should respond right ? It "hangs".
>>>       
>
> Have you tuned these OSSes with respect to the number of OST threads needed to drive (and not over-drive) your disks?  The lustre
> iokit is useful for that tuning.
>
>   
>>> Ok, tuning for performance is okay, but hanging with 20 nodes (IOR MPI).. strange right ?
>>>       
>
> b.
>
> -----
>
> I'm using 3 raid 5 with 8 disks each and 256 OST threads on each OSS.
>
> root at a02n00:~# cat /etc/mdadm.conf
> ARRAY /dev/md10 level=raid5 num-devices=8 devices=/dev/dm-0,/dev/dm-1,/dev/dm-2,/dev/dm-3,/dev/dm-4,/dev/dm-5,/dev/dm-6,/dev/dm-7
> ARRAY /dev/md11 level=raid5 num-devices=8
> devices=/dev/dm-8,/dev/dm-9,/dev/dm-10,/dev/dm-11,/dev/dm-12,/dev/dm-13,/dev/dm-14,/dev/dm-15
> ARRAY /dev/md12 level=raid5 num-devices=8
> devices=/dev/dm-16,/dev/dm-17,/dev/dm-18,/dev/dm-19,/dev/dm-20,/dev/dm-21,/dev/dm-22,/dev/dm-23
>
> All my OSTs were created with internal journal (for test pourposes).
>
> mkfs.lustre --r --ost --fsname=work --mkfsoptions="-b 4096 -E stride=32,stripe-width=224 -m 0" --mgsnid=a03n00 at o2ib
> --mgsnid=b03n00 at o2ib /dev/md[10|11|12]
>
> Im using separete mdt and mgs:
>
> # MGS
> mkfs.lustre --fsname=work --r --mgs --mkfsoptions="-b 4096 -E stride=4,stripe-width=4 -m 0" --mountfsoptions=acl
> --failnode=b03n00 at o2ib /dev/sdb1
>
> # MDT
> mkfs.lustre --fsname=work --r --mgsnid=a03n00 at o2ib --mgsnid=b03n00 at o2ib --mdt --mkfsoptions="-b 4096 -E stride=4,stripe-width=40 -m
> 0" --mountfsoptions=acl --failnode=b03n00 at o2ib /dev/sdc1
>
> I'm using these packages on server:
> ----------
> root at a03n00:~# rpm -aq | grep -i lustre
> lustre-modules-1.8.1-2.6.18_128.1.14.el5_lustre.1.8.1
> lustre-client-modules-1.8.1-2.6.18_128.1.14.el5_lustre.1.8.1
> lustre-ldiskfs-3.0.9-2.6.18_128.1.14.el5_lustre.1.8.1
> kernel-lustre-headers-2.6.18-128.1.14.el5_lustre.1.8.1
> kernel-lustre-2.6.18-128.1.14.el5_lustre.1.8.1
> lustre-client-1.8.1-2.6.18_128.1.14.el5_lustre.1.8.1
> kernel-lustre-devel-2.6.18-128.1.14.el5_lustre.1.8.1
> lustre-1.8.1-2.6.18_128.1.14.el5_lustre.1.8.1
> kernel-ib-1.4.1-2.6.18_128.1.14.el5_lustre.1.8.1
> ----------
> On client Ive compiled kernel 2.6.18-128.el5 without INFINIBAND support.
> Then compiled OFED 1.4.1 and after that compile patchless client.
> For the patchless client, compiled with:
> --ofa-kernel=/usr/src/ofa_kernel
> ----------
>
> * THE ERROR
>
> Using: 
>
> root at b00n00:~# mpirun -hostfile ./lustre.hosts -np 20 /hpc/IOR -w -r -C -i 2 -b 1G -t 512k -F -o /work/stripe12/teste
>
> for example starts "hanging" the OSTs and the filesystem "hangs". 
> Any atempt to rm or read a file (or df -kh) hangs and keeps forever (not even kill -9 solves).
>
> With that.. I cannot umount my OSTs on the OSSs.
> And I have to "reboot" the server, and my raids starts resyncing.
>
> Tinoco
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
> ------------------------------------------------------------------------
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss