Hi Roger <br><br>Where did you find this CONFIG hack?<br>Did you make a copy of the CONFIG dir before followed this steps?<br><br><br><br><div class="gmail_quote">On 15 July 2010 20:02, Roger Sersted <span dir="ltr"><<a href="mailto:rs1@aps.anl.gov">rs1@aps.anl.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;"><br>
I am using the ext4 RPMs. I ran the following commands on the MDS and OSS nodes (lustre was not running at the time):<div class="im"><br>
<br>
tune2fs -O extents,uninit_bg,dir_index /dev/XXX<br>
fsck -pf /dev/XXX<br>
<br></div>
I then started Lustre "mount -t lustre /dev/XXX /lustre" on the OSSes and then the MDS. The problem still persisted. I then shutdown Lustre by unmounting the Lustre filesystems on the MDS/OSS nodes.<br>
<br>
My last and most desperate step was to "hack" the CONFIG files. On puppy7, I did the following:<br>
<br>
1. mount -t ldiskfs /dev/sdc /mnt<br>
2. cd /mnt/CONFIG<br>
3. mv lustre1-OST0000 lustre1-OST0001<br>
4. vim -nb lustre1-OST0001 mountdata<br>
5. I changed OST0000 to OST0001.<br>
6. I verified my changes by comparing an "od -c" of before and after.<br>
7. umount /mnt<br>
8. tunefs.lustre -writeconf /dev/sdc<br>
<br>
The output of step 8 is:<br>
<br>
tunefs.lustre -writeconf /dev/sdc<div class="im"><br>
checking for existing Lustre data: found CONFIGS/mountdata<br>
Reading CONFIGS/mountdata<br>
<br>
Read previous values:<br></div>
Target: lustre1-OST0001<div class="im"><br>
Index: 0<br>
Lustre FS: lustre1<br>
Mount type: ldiskfs<br></div>
Flags: 0x102<br>
(OST writeconf )<div class="im"><br>
Persistent mount opts: errors=remount-ro,extents,mballoc<br>
Parameters: mgsnode=172.17.2.5@o2ib<br>
<br>
<br>
Permanent disk data:<br>
Target: lustre1-OST0000<br>
Index: 0<br>
Lustre FS: lustre1<br>
Mount type: ldiskfs<br></div>
Flags: 0x102<br>
(OST writeconf )<div class="im"><br>
Persistent mount opts: errors=remount-ro,extents,mballoc<br>
Parameters: mgsnode=172.17.2.5@o2ib<br>
<br></div>
Writing CONFIGS/mountdata<br>
<br>
Now part of the system seems to have the correct Target value.<br>
<br>
Thanks for your time on this.<br>
<br>
Roger S.<br>
<br>
Wojciech Turek wrote:<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
Hi Roger,<br>
<br><div class="im">
the Lustre 1.8.3 for RHEL5 has to set of RPMS one set for old style ext3 based ldiskfs and one set for the ext4 based ldiskfs. When upgrading from 1.6.6 to 1.8.3 I think you should not try to use the ext4 based packages, can you let us know which RPMs have you used?<br>
<br>
<br>
<br></div><div><div></div><div class="h5">
On 15 July 2010 16:14, Roger Sersted <<a href="mailto:rs1@aps.anl.gov" target="_blank">rs1@aps.anl.gov</a> <mailto:<a href="mailto:rs1@aps.anl.gov" target="_blank">rs1@aps.anl.gov</a>>> wrote:<br>
<br>
<br>
<br>
Wojciech Turek wrote:<br>
<br>
can you also please post output of 'rpm -qa | grep lustre' run<br>
on puppy5-7 ?<br>
<br>
<br>
<br>
[root@puppy5 log]# rpm -qa |grep -i lustre<br>
kernel-2.6.18-164.11.1.el5_lustre.1.8.3<br>
lustre-1.8.3-2.6.18_164.11.1.el5_lustre.1.8.3<br>
lustre-ldiskfs-3.0.9-2.6.18_164.11.1.el5_lustre.1.8.3<br>
mft-2.6.0-2.6.18_164.11.1.el5_lustre.1.8.3<br>
lustre-modules-1.8.3-2.6.18_164.11.1.el5_lustre.1.8.3<br>
<br>
[root@puppy6 log]# rpm -qa | grep -i lustre<br>
kernel-2.6.18-164.11.1.el5_lustre.1.8.3<br>
lustre-1.8.3-2.6.18_164.11.1.el5_lustre.1.8.3<br>
lustre-ldiskfs-3.0.9-2.6.18_164.11.1.el5_lustre.1.8.3<br>
mft-2.6.0-2.6.18_164.11.1.el5_lustre.1.8.3<br>
lustre-modules-1.8.3-2.6.18_164.11.1.el5_lustre.1.8.3<br>
<br>
[root@puppy7 CONFIGS]# rpm -qa | grep -i lustre<br>
kernel-2.6.18-164.11.1.el5_lustre.1.8.3<br>
lustre-1.8.3-2.6.18_164.11.1.el5_lustre.1.8.3<br>
lustre-ldiskfs-3.0.9-2.6.18_164.11.1.el5_lustre.1.8.3<br>
mft-2.6.0-2.6.18_164.11.1.el5_lustre.1.8.3<br>
lustre-modules-1.8.3-2.6.18_164.11.1.el5_lustre.1.8.3<br>
<br>
Thanks,<br>
<br>
Roger S.<br>
<br>
<br>
On 15 July 2010 15:55, Roger Sersted <<a href="mailto:rs1@aps.anl.gov" target="_blank">rs1@aps.anl.gov</a><br></div></div><div><div></div><div class="h5">
<mailto:<a href="mailto:rs1@aps.anl.gov" target="_blank">rs1@aps.anl.gov</a>> <mailto:<a href="mailto:rs1@aps.anl.gov" target="_blank">rs1@aps.anl.gov</a><br>
<mailto:<a href="mailto:rs1@aps.anl.gov" target="_blank">rs1@aps.anl.gov</a>>>> wrote:<br>
<br>
<br>
OK. This looks bad. It appears that I should have upgraded<br>
ext3 to<br>
ext4, I found instructions for that,<br>
<br>
tune2fs -O extents,uninit_bg,dir_index /dev/XXX<br>
fsck -pf /dev/XXX<br>
Is the above correct? I'd like to move our<br>
systems to ext4. I<br>
didn't know those steps were necessary.<br>
<br>
Other answers listed below.<br>
<br>
<br>
Wojciech Turek wrote:<br>
<br>
Hi Roger,<br>
<br>
Sorry for the delay. From the ldiskfs messages I seem to<br>
me that<br>
you are using ext4 ldiskfs<br>
(Jun 26 17:54:30 puppy7 kernel: ldiskfs created from<br>
ext4-2.6-rhel5).<br>
If you upgrading from 1.6.6 you ldiskfs is ext3 based so<br>
I think<br>
taht in lustre-1.8.3 you should use ext3 based ldiskfs rpm.<br>
<br>
Can you also tell us a bit more about your setup? From<br>
what you<br>
wrote so far I understand you have 2 OSS servers and each<br>
server<br>
has one OST device. In addition to that you have a third<br>
server<br>
which acts as a MGS/MDS, is that right?<br>
<br>
The logs you provided seem to be only from one server called<br>
puppy7 so it does not give a whole picture of the<br>
situation. The<br>
timeout messages may indicate a problem with communication<br>
between the servers but it is really difficult to say without<br>
seeing the whole picture or at least more elements of it.<br>
<br>
To check if you have correct rpms installed can you<br>
please run<br>
'rpm -qa | grep lustre' on both OSS servers and the MDS?<br>
<br>
Also please provide output from command 'lctl list_nids'<br>
run on<br>
both OSS servers, MDS and a client?<br>
<br>
<br>
puppy5 (MDS/MGS)<br>
<br>
172.17.2.5@o2ib<br>
172.16.2.5@tcp<br>
<br>
puppy6 (OSS)<br>
172.17.2.6@o2ib<br>
172.16.2.6@tcp<br>
<br>
puppy7 (OSS)<br>
172.17.2.7@o2ib<br>
172.16.2.7@tcp<br>
<br>
<br>
<br>
<br>
In addition to above please run following command on all<br>
lustre<br>
targets (OSTs and MDT) to display your current lustre<br>
configuration<br>
<br>
tunefs.lustre --dryrun --print /dev/<ost_device><br>
<br>
<br>
puppy5 (MDS/MGS)<br>
Read previous values:<br>
Target: lustre1-MDT0000<br>
Index: 0<br>
Lustre FS: lustre1<br>
Mount type: ldiskfs<br>
Flags: 0x405<br>
(MDT MGS )<br>
Persistent mount opts: errors=remount-ro,iopen_nopriv,user_xattr<br>
Parameters: lov.stripesize=125K lov.stripecount=2<br>
mdt.group_upcall=/usr/sbin/l_getgroups mdt.group_upcall=NONE<br>
mdt.group_upcall=NONE<br>
<br>
<br>
Permanent disk data:<br>
Target: lustre1-MDT0000<br>
Index: 0<br>
Lustre FS: lustre1<br>
Mount type: ldiskfs<br>
Flags: 0x405<br>
(MDT MGS )<br>
Persistent mount opts: errors=remount-ro,iopen_nopriv,user_xattr<br>
Parameters: lov.stripesize=125K lov.stripecount=2<br>
mdt.group_upcall=/usr/sbin/l_getgroups mdt.group_upcall=NONE<br>
mdt.group_upcall=NONE<br>
<br>
exiting before disk write.<br>
----------------------------------------------------<br>
puppy6<br>
checking for existing Lustre data: found CONFIGS/mountdata<br>
Reading CONFIGS/mountdata<br>
<br>
Read previous values:<br>
Target: lustre1-OST0000<br>
Index: 0<br>
Lustre FS: lustre1<br>
Mount type: ldiskfs<br>
Flags: 0x2<br>
(OST )<br>
Persistent mount opts: errors=remount-ro,extents,mballoc<br>
Parameters: mgsnode=172.17.2.5@o2ib<br>
<br>
<br>
Permanent disk data:<br>
Target: lustre1-OST0000<br>
Index: 0<br>
Lustre FS: lustre1<br>
Mount type: ldiskfs<br>
Flags: 0x2<br>
(OST )<br>
Persistent mount opts: errors=remount-ro,extents,mballoc<br>
Parameters: mgsnode=172.17.2.5@o2ib<br>
--------------------------------------------------<br>
puppy7 (this is the broken OSS. The "Target" should be<br>
"lustre1-OST0001")<br>
checking for existing Lustre data: found CONFIGS/mountdata<br>
Reading CONFIGS/mountdata<br>
<br>
Read previous values:<br>
Target: lustre1-OST0000<br>
Index: 0<br>
Lustre FS: lustre1<br>
Mount type: ldiskfs<br>
Flags: 0x2<br>
(OST )<br>
Persistent mount opts: errors=remount-ro,extents,mballoc<br>
Parameters: mgsnode=172.17.2.5@o2ib<br>
<br>
<br>
Permanent disk data:<br>
Target: lustre1-OST0000<br>
Index: 0<br>
Lustre FS: lustre1<br>
Mount type: ldiskfs<br>
Flags: 0x2<br>
(OST )<br>
Persistent mount opts: errors=remount-ro,extents,mballoc<br>
Parameters: mgsnode=172.17.2.5@o2ib<br>
<br>
exiting before disk write.<br>
<br>
<br>
<br>
If possible please attach syslog from each machine from<br>
the time<br>
you mounted lustre targets (OST and MDT).<br>
<br>
Best regards,<br>
<br>
Wojciech<br>
<br>
On 14 July 2010 20:46, Roger Sersted <<a href="mailto:rs1@aps.anl.gov" target="_blank">rs1@aps.anl.gov</a><br>
<mailto:<a href="mailto:rs1@aps.anl.gov" target="_blank">rs1@aps.anl.gov</a>><br>
<mailto:<a href="mailto:rs1@aps.anl.gov" target="_blank">rs1@aps.anl.gov</a> <mailto:<a href="mailto:rs1@aps.anl.gov" target="_blank">rs1@aps.anl.gov</a>>><br>
<mailto:<a href="mailto:rs1@aps.anl.gov" target="_blank">rs1@aps.anl.gov</a> <mailto:<a href="mailto:rs1@aps.anl.gov" target="_blank">rs1@aps.anl.gov</a>><br>
<br></div></div><div class="im">
<mailto:<a href="mailto:rs1@aps.anl.gov" target="_blank">rs1@aps.anl.gov</a> <mailto:<a href="mailto:rs1@aps.anl.gov" target="_blank">rs1@aps.anl.gov</a>>>>> wrote:<br>
<br>
<br>
Any additional info?<br>
<br>
Thanks,<br>
<br>
Roger S.<br>
<br>
<br>
<br>
<br>
-- --<br>
Wojciech Turek<br>
<br>
<br>
<br>
<br>
<br>
-- --<br>
Wojciech Turek<br>
<br>
Assistant System Manager<br>
517<br>
<br>
</div></blockquote>
</blockquote></div><br><br clear="all"><br>-- <br>
<div style="visibility: hidden; display: inline;" id="avg_ls_inline_popup"></div><style type="text/css">#avg_ls_inline_popup { position:absolute; z-index:9999; padding: 0px 0px; margin-left: 0px; margin-top: 0px; width: 240px; overflow: hidden; word-wrap: break-word; color: black; font-size: 10px; text-align: left; line-height: 13px;}</style>