Hi Brendon,<br><div class="gmail_quote"><br>So it looks like you Lustre was just stuck in recovery processes after all.<br>It is a bit concerning that you had kernel panics on MDS during recovery. Which Lustre version are you using? Do you have stack traces from the kernel panics?<br>
<br>Wojciech<div><div></div><div class="h5"><br><br><div class="gmail_quote">On 13 January 2011 17:41, Brendon <span dir="ltr"><<a href="mailto:b@brendon.com" target="_blank">b@brendon.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<div>On Tue, Jan 11, 2011 at 3:35 PM, Wojciech Turek <<a href="mailto:wjt27@cam.ac.uk" target="_blank">wjt27@cam.ac.uk</a>> wrote:<br>
</div><div>> Hi Brendon,<br>
><br>
> Can you please provide following:<br>
> 1) output of ifconfig run on each OSS MDS and at least one client<br>
> 2) output of lctl list_nids run on each OSS MDS and at least one client<br>
> 3) output of tunefs.lustre --print --dryrun /dev/<OST_block_device> from<br>
> each OSS<br>
><br>
> Wojciech<br>
<br>
</div><div><div></div><div>After someone looked at the emails I sent out, they grabbed me on IRC.<br>
We had a discussion and basically they interpreted the email as<br>
everything should be working, I just needed to wait for a repair to<br>
run and complete. What I then learned is that first, a client has to<br>
connect for a repair to initiate. Secondly, the code isn't perfect.<br>
The MDS kernel oops'ed twice before it finally completed a repair<br>
successfully. I was in the process of disabling panic on oops, but it<br>
finally completed successfully. Once that was done, I got a clean bill<br>
of health.<br>
<br>
Just to complete this discussion, I have listed the requested output.<br>
I might still learn something :)<br>
<br>
...Looks like I did learn something. OSS0 has an issue with the root<br>
FS and was remounted RO which I discovered when running tunefs.lustre<br>
--print --dryrun /dev/sda5.<br>
<br>
The fun never ends :)<br>
-Brendon<br>
<br>
1) ifconfig info<br>
MDS: # ifconfig<br>
eth0 Link encap:Ethernet HWaddr 00:15:17:5E:46:64<br>
inet addr:10.1.1.1 Bcast:10.1.1.255 Mask:255.255.255.0<br>
inet6 addr: fe80::215:17ff:fe5e:4664/64 Scope:Link<br>
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1<br>
RX packets:49140546 errors:0 dropped:0 overruns:0 frame:0<br>
TX packets:63644404 errors:0 dropped:0 overruns:0 carrier:0<br>
collisions:0 txqueuelen:1000<br>
RX bytes:18963170801 (17.6 GiB) TX bytes:65261762295 (60.7 GiB)<br>
Base address:0xcc00 Memory:f58e0000-f5900000<br>
<br>
eth1 Link encap:Ethernet HWaddr 00:15:17:5E:46:65<br>
inet addr:192.168.0.181 Bcast:192.168.0.255 Mask:255.255.255.0<br>
inet6 addr: fe80::215:17ff:fe5e:4665/64 Scope:Link<br>
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1<br>
RX packets:236738842 errors:0 dropped:0 overruns:0 frame:0<br>
TX packets:458503163 errors:0 dropped:0 overruns:0 carrier:0<br>
collisions:0 txqueuelen:100<br>
RX bytes:15562858193 (14.4 GiB) TX bytes:686167422947 (639.0 GiB)<br>
Base address:0xc880 Memory:f5880000-f58a0000<br>
<br>
OSS : # ifconfig<br>
eth0 Link encap:Ethernet HWaddr 00:1D:60:E0:5B:B2<br>
inet addr:10.1.1.2 Bcast:10.1.1.255 Mask:255.255.255.0<br>
inet6 addr: fe80::21d:60ff:fee0:5bb2/64 Scope:Link<br>
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1<br>
RX packets:3092588 errors:0 dropped:0 overruns:0 frame:0<br>
TX packets:3547204 errors:0 dropped:0 overruns:0 carrier:0<br>
collisions:0 txqueuelen:1000<br>
RX bytes:1320521551 (1.2 GiB) TX bytes:2670089148 (2.4 GiB)<br>
Interrupt:233<br>
<br>
client: # ifconfig<br>
eth0 Link encap:Ethernet HWaddr 00:1E:8C:39:E4:69<br>
inet addr:10.1.1.5 Bcast:10.1.1.255 Mask:255.255.255.0<br>
inet6 addr: fe80::21e:8cff:fe39:e469/64 Scope:Link<br>
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1<br>
RX packets:727922 errors:0 dropped:0 overruns:0 frame:0<br>
TX packets:884188 errors:0 dropped:0 overruns:0 carrier:0<br>
collisions:0 txqueuelen:1000<br>
RX bytes:433349006 (413.2 MiB) TX bytes:231985578 (221.2 MiB)<br>
Interrupt:50<br>
<br>
<br>
<br>
2) lctl list_nids<br>
<br>
client: lctl list_nids<br>
10.1.1.5@tcp<br>
<br>
MDS: lctl list_nids<br>
</div></div><div>10.1.1.1@tcp<br>
<br>
</div>OSS: lctl list_nids<br>
<div>10.1.1.2@tcp<br>
<br>
</div><div><div></div><div>3) tunefs.lustre --print --dryrun /dev/sda5<br>
OSS0: ]# tunefs.lustre --print --dryrun /dev/sda5<br>
checking for existing Lustre data: found CONFIGS/mountdata<br>
tunefs.lustre: Can't create temporary directory /tmp/dirCZXt3k:<br>
Read-only file system<br>
<br>
tunefs.lustre FATAL: Failed to read previous Lustre data from /dev/sda5 (30)<br>
tunefs.lustre: exiting with 30 (Read-only file system)<br>
<br>
OSS1: # tunefs.lustre --print --dryrun /dev/sda5<br>
checking for existing Lustre data: found CONFIGS/mountdata<br>
Reading CONFIGS/mountdata<br>
<br>
Read previous values:<br>
Target: mylustre-OST0001<br>
Index: 1<br>
Lustre FS: mylustre<br>
Mount type: ldiskfs<br>
Flags: 0x2<br>
(OST )<br>
Persistent mount opts: errors=remount-ro,extents,mballoc<br>
Parameters: mgsnode=10.1.1.1@tcp<br>
<br>
<br>
Permanent disk data:<br>
Target: mylustre-OST0001<br>
Index: 1<br>
Lustre FS: mylustre<br>
Mount type: ldiskfs<br>
Flags: 0x2<br>
(OST )<br>
Persistent mount opts: errors=remount-ro,extents,mballoc<br>
Parameters: mgsnode=10.1.1.1@tcp<br>
<br>
exiting before disk write.<br>
<br>
<br>
OSS2: # tunefs.lustre --print --dryrun /dev/sda5<br>
checking for existing Lustre data: found CONFIGS/mountdata<br>
Reading CONFIGS/mountdata<br>
<br>
Read previous values:<br>
Target: mylustre-OST0002<br>
Index: 2<br>
Lustre FS: mylustre<br>
Mount type: ldiskfs<br>
Flags: 0x2<br>
(OST )<br>
Persistent mount opts: errors=remount-ro,extents,mballoc<br>
Parameters: mgsnode=10.1.1.1@tcp<br>
<br>
<br>
Permanent disk data:<br>
Target: mylustre-OST0002<br>
Index: 2<br>
Lustre FS: mylustre<br>
Mount type: ldiskfs<br>
Flags: 0x2<br>
(OST )<br>
Persistent mount opts: errors=remount-ro,extents,mballoc<br>
</div></div>Parameters: mgsnode=10.1.1.1@tcp<br>
<br>
exiting before disk write.<br>
<div><div></div><div>_______________________________________________<br>
Lustre-discuss mailing list<br>
<a href="mailto:Lustre-discuss@lists.lustre.org" target="_blank">Lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss" target="_blank">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a><br>
</div></div></blockquote></div><br><br><br>
</div></div></div>