Hey Robin,<br><br>We are still looking into this issue and try to figure out what is causing this problem. We are using a application called gsnap that does use openmpi and RMPI. The next time this happens I will definitely look at lsof and see if there are any /dev/shm related entries. Will report back with more information.<br>
<br>What os are you guys using on your clients? What did you guys end up doing for the long term fix of this issue? I am thinking of downgrading to 2.6.27.29-0.1 kernel and 1.8.1.1 lustre client.<br><br>Regards,<br>-J<br>
<br><div class="gmail_quote">On Tue, Oct 19, 2010 at 9:48 PM, Robin Humble <span dir="ltr"><<a href="mailto:robin.humble%2Blustre@anu.edu.au">robin.humble+lustre@anu.edu.au</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
Hi Jagga,<br>
<div class="im"><br>
On Wed, Oct 13, 2010 at 02:33:35PM -0700, Jagga Soorma wrote:<br>
</div>..<br>
<div class="im">>start seeing this issue. All my clients are setup with SLES11 and the same<br>
>packages with the exception of a newer kernel in the 1.8.4 environment due<br>
>to the lustre dependency:<br>
><br>
>reshpc208:~ # uname -a<br>
>Linux reshpc208 2.6.27.39-0.3-default #1 SMP 2009-11-23 12:57:38 +0100 x86_64 x86_64 x86_64 GNU/Linux<br>
</div>...<br>
<div class="im">>open("/proc/9598/stat", O_RDONLY) = 6<br>
>read(6, "9598 (gsnap) S 9596 9589 9589 0 "..., 1023) = 254<br>
>close(6) = 0<br>
>open("/proc/9598/status", O_RDONLY) = 6<br>
>read(6, "Name:\tgsnap\nState:\tS (sleeping)\n"..., 1023) = 1023<br>
>close(6) = 0<br>
>open("/proc/9598/cmdline", O_RDONLY) = 6<br>
>read(6,<br>
<br>
</div>did you get any further with this?<br>
<br>
we've just seen something similar in that we had D state hung processes<br>
and a strace of ps hung at the same place.<br>
<br>
in the end our hang appeared to be /dev/shm related, and an 'ipcs -ma'<br>
magically caused all the D state processes to continue... we don't have<br>
a good idea why this might be. looks kinda like a generic kernel shm<br>
deadlock, possibly unrelated to Lustre.<br>
<br>
sys_shmdt features in the hung process tracebacks that the kernel<br>
prints out.<br>
<br>
if you do 'lsof' do you see lots of /dev/shm entries for your app?<br>
the app we saw run into trouble was using HPMPI which is common in<br>
commercial packages. does gsnap use HPMPI?<br>
<br>
we are running vanilla 2.6.32.* kernels with Lustre 1.8.4 clients on<br>
this cluster.<br>
<br>
cheers,<br>
robin<br>
--<br>
<div class="im">Dr Robin Humble, HPC Systems Analyst, NCI National Facility<br>
</div>_______________________________________________<br>
Lustre-discuss mailing list<br>
<a href="mailto:Lustre-discuss@lists.lustre.org">Lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss" target="_blank">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a><br>
<font color="#888888"><br>
--<br>
You received this message because you are subscribed to the Google Groups "lustre-discuss-list" group.<br>
To post to this group, send email to <a href="mailto:lustre-discuss-list@googlegroups.com">lustre-discuss-list@googlegroups.com</a>.<br>
To unsubscribe from this group, send email to <a href="mailto:lustre-discuss-list%2Bunsubscribe@googlegroups.com">lustre-discuss-list+unsubscribe@googlegroups.com</a>.<br>
For more options, visit this group at <a href="http://groups.google.com/group/lustre-discuss-list?hl=en" target="_blank">http://groups.google.com/group/lustre-discuss-list?hl=en</a>.<br>
<br>
</font></blockquote></div><br>