can you also please post output of 'rpm -qa | grep lustre' run on puppy5-7 ?<br><br><div class="gmail_quote">On 15 July 2010 15:55, Roger Sersted <span dir="ltr"><<a href="mailto:rs1@aps.anl.gov">rs1@aps.anl.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><br>
OK. This looks bad. It appears that I should have upgraded ext3 to ext4, I found instructions for that,<br>
<br>
tune2fs -O extents,uninit_bg,dir_index /dev/XXX<br>
fsck -pf /dev/XXX<br>
<br>
Is the above correct? I'd like to move our systems to ext4. I didn't know those steps were necessary.<br>
<br>
Other answers listed below.<div class="im"><br>
<br>
Wojciech Turek wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Hi Roger,<br>
<br>
Sorry for the delay. From the ldiskfs messages I seem to me that you are using ext4 ldiskfs<br>
(Jun 26 17:54:30 puppy7 kernel: ldiskfs created from ext4-2.6-rhel5).<br>
If you upgrading from 1.6.6 you ldiskfs is ext3 based so I think taht in lustre-1.8.3 you should use ext3 based ldiskfs rpm.<br>
<br>
Can you also tell us a bit more about your setup? From what you wrote so far I understand you have 2 OSS servers and each server has one OST device. In addition to that you have a third server which acts as a MGS/MDS, is that right?<br>
<br>
The logs you provided seem to be only from one server called puppy7 so it does not give a whole picture of the situation. The timeout messages may indicate a problem with communication between the servers but it is really difficult to say without seeing the whole picture or at least more elements of it.<br>
<br>
To check if you have correct rpms installed can you please run 'rpm -qa | grep lustre' on both OSS servers and the MDS?<br>
<br>
Also please provide output from command 'lctl list_nids' run on both OSS servers, MDS and a client?<br>
</blockquote>
<br></div>
puppy5 (MDS/MGS)<div class="im"><br>
172.17.2.5@o2ib<br></div>
172.16.2.5@tcp<br>
<br>
puppy6 (OSS)<br>
172.17.2.6@o2ib<br>
172.16.2.6@tcp<br>
<br>
puppy7 (OSS)<br>
172.17.2.7@o2ib<br>
172.16.2.7@tcp<div class="im"><br>
<br>
<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<br>
In addition to above please run following command on all lustre targets (OSTs and MDT) to display your current lustre configuration<br>
<br>
tunefs.lustre --dryrun --print /dev/<ost_device><br>
</blockquote>
<br></div>
puppy5 (MDS/MGS)<br>
Read previous values:<br>
Target: lustre1-MDT0000<br>
Index: 0<br>
Lustre FS: lustre1<br>
Mount type: ldiskfs<br>
Flags: 0x405<br>
(MDT MGS )<br>
Persistent mount opts: errors=remount-ro,iopen_nopriv,user_xattr<br>
Parameters: lov.stripesize=125K lov.stripecount=2 mdt.group_upcall=/usr/sbin/l_getgroups mdt.group_upcall=NONE mdt.group_upcall=NONE<br>
<br>
<br>
Permanent disk data:<br>
Target: lustre1-MDT0000<br>
Index: 0<br>
Lustre FS: lustre1<br>
Mount type: ldiskfs<br>
Flags: 0x405<br>
(MDT MGS )<br>
Persistent mount opts: errors=remount-ro,iopen_nopriv,user_xattr<br>
Parameters: lov.stripesize=125K lov.stripecount=2 mdt.group_upcall=/usr/sbin/l_getgroups mdt.group_upcall=NONE mdt.group_upcall=NONE<br>
<br>
exiting before disk write.<br>
----------------------------------------------------<br>
puppy6<br>
checking for existing Lustre data: found CONFIGS/mountdata<br>
Reading CONFIGS/mountdata<br>
<br>
Read previous values:<br>
Target: lustre1-OST0000<br>
Index: 0<br>
Lustre FS: lustre1<br>
Mount type: ldiskfs<br>
Flags: 0x2<br>
(OST )<br>
Persistent mount opts: errors=remount-ro,extents,mballoc<br>
Parameters: mgsnode=172.17.2.5@o2ib<br>
<br>
<br>
Permanent disk data:<br>
Target: lustre1-OST0000<br>
Index: 0<br>
Lustre FS: lustre1<br>
Mount type: ldiskfs<br>
Flags: 0x2<br>
(OST )<br>
Persistent mount opts: errors=remount-ro,extents,mballoc<br>
Parameters: mgsnode=172.17.2.5@o2ib<br>
--------------------------------------------------<br>
puppy7 (this is the broken OSS. The "Target" should be "lustre1-OST0001")<br>
checking for existing Lustre data: found CONFIGS/mountdata<br>
Reading CONFIGS/mountdata<br>
<br>
Read previous values:<br>
Target: lustre1-OST0000<br>
Index: 0<br>
Lustre FS: lustre1<br>
Mount type: ldiskfs<br>
Flags: 0x2<br>
(OST )<br>
Persistent mount opts: errors=remount-ro,extents,mballoc<br>
Parameters: mgsnode=172.17.2.5@o2ib<br>
<br>
<br>
Permanent disk data:<br>
Target: lustre1-OST0000<br>
Index: 0<br>
Lustre FS: lustre1<br>
Mount type: ldiskfs<br>
Flags: 0x2<br>
(OST )<br>
Persistent mount opts: errors=remount-ro,extents,mballoc<br>
Parameters: mgsnode=172.17.2.5@o2ib<br>
<br>
exiting before disk write.<br>
<br>
<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div class="im">
<br>
If possible please attach syslog from each machine from the time you mounted lustre targets (OST and MDT).<br>
<br>
Best regards,<br>
<br>
Wojciech<br>
<br></div><div class="im">
On 14 July 2010 20:46, Roger Sersted <<a href="mailto:rs1@aps.anl.gov" target="_blank">rs1@aps.anl.gov</a> <mailto:<a href="mailto:rs1@aps.anl.gov" target="_blank">rs1@aps.anl.gov</a>>> wrote:<br>
<br>
<br>
Any additional info?<br>
<br>
Thanks,<br>
<br>
Roger S.<br>
<br>
<br>
<br>
<br>
-- <br>
--<br>
Wojciech Turek<br>
<br>
<br>
</div></blockquote>
</blockquote></div><br><br clear="all"><br>-- <br>--<br>Wojciech Turek<br><br>Assistant System Manager<br><br>High Performance Computing Service<br>University of Cambridge<br>Email: <a href="mailto:wjt27@cam.ac.uk">wjt27@cam.ac.uk</a><br>
Tel: (+)44 1223 763517 <br>