<br>Following up on the below, the problem was a broken ethernet configuration. One eth interface had an MTU of 1500 and the other was set to 9000 or similar silliness, I think. Pings made it through the bond0 balance-alb pair, but TCP/IP didn't? It was something along those lines.<br>
<br><br><div class="gmail_quote">On Thu, Feb 21, 2008 at 4:47 PM, <<a href="mailto:aberoham@gmail.com">aberoham@gmail.com</a>> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<br>I have a lustre MGS/MDT hosting one lustre filesystem with three OSTs attached. When trying to attach a forth OST I see the following on the OST's console and the mount command times out with "mount.lustre: mount /dev/lustre2/ost at /mnt/data/ost failed: Input/output error Is the MGS running?". <br>
<br>I am able to mount the lustre filesystem on this un-attachable OST node as a client and am able to ping the MGS/MDT and vice versa.<br><br># mkfs.lustre --reformat --fsname tmonster --ost --mgsnode=tm01@tcp0 --mkfsoptions='-N 1200000' /dev/lustre2/ost<br>
...<br># mount -t lustre /dev/lustre2/ost /mnt/data/ost<br>mount.lustre: mount /dev/lustre2/ost at /mnt/data/ost failed: Input/output error<br>Is the MGS running?<br><br># dmesg<br>Lustre: OBD class driver, <a href="mailto:info@clusterfs.com" target="_blank">info@clusterfs.com</a><br>
Lustre Version: <a href="http://1.6.4.2" target="_blank">1.6.4.2</a><br> Build Version: 1.6.4.2-19691231190000-PRISTINE-.cache.build.BUILD.lustre-kernel-2.6.18.lustre.linux-2.6.18-8.1.14.el5_lustre.1.6.4.2smp<br>
Lustre: Added LNI 192.168.33.5@tcp [8/256]<br>
Lustre: Accept secure, port 988<br>Lustre: Lustre Client File System; <a href="mailto:info@clusterfs.com" target="_blank">info@clusterfs.com</a><br>kjournald starting. Commit interval 5 seconds<br>LDISKFS FS on dm-2, internal journal<br>
LDISKFS-fs: mounted filesystem with ordered data mode.<br>LustreError: 2535:0:(obd_mount.c:247:ldd_parse()) cannot open CONFIGS/mountdata: rc = -2<br>LustreError: 2535:0:(obd_mount.c:1252:server_kernel_mount()) premount parse options failed: rc = -2<br>
LustreError: 2535:0:(obd_mount.c:1533:server_fill_super()) Unable to mount device /dev/lustre2/ost: -2<br>LustreError: 2535:0:(obd_mount.c:1924:lustre_fill_super()) Unable to mount (-2)<br>NET: Registered protocol family 10<br>
lo: Disabled Privacy Extensions<br>IPv6 over IPv4 tunneling driver<br>bond0: no IPv6 routers present<br>eth0: no IPv6 routers present<br>eth1: no IPv6 routers present<br>kjournald starting. Commit interval 5 seconds<br>
LDISKFS FS on dm-2, internal journal<br>
LDISKFS-fs: mounted filesystem with ordered data mode.<br>kjournald starting. Commit interval 5 seconds<br>LDISKFS FS on dm-2, internal journal<br>LDISKFS-fs: mounted filesystem with ordered data mode.<br>kjournald starting. Commit interval 5 seconds<br>
LDISKFS FS on dm-2, internal journal<br>LDISKFS-fs: mounted filesystem with ordered data mode.<br>LDISKFS-fs: file extents enabled<br>LDISKFS-fs: mballoc enabled<br>LustreError: 3157:0:(client.c:975:ptlrpc_expire_one_request()) @@@ timeout (sent at 1203640178, 100s ago) req@ffff81007f5db600 x2/t0 o253->MGS@MGC192.168.33.1@tcp_0:26 lens 4672/4672 ref 1 fl Rpc:/0/0 rc 0/-22<br>
LustreError: 166-1: MGC192.168.33.1@tcp: Connection to service MGS via nid 192.168.33.1@tcp was lost; in progress operations using this service will fail.<br>LustreError: 3157:0:(obd_mount.c:954:server_register_target()) registration with the MGS failed (-5)<br>
LustreError: 3157:0:(obd_mount.c:1054:server_start_targets()) Required registration failed for tmonster-OSTffff: -5<br>LustreError: 15f-b: Communication error with the MGS. Is the MGS running?<br>Lustre: MGC192.168.33.1@tcp: Reactivating import<br>
LustreError: 3157:0:(obd_mount.c:1570:server_fill_super()) Unable to start targets: -5<br>Lustre: MGC192.168.33.1@tcp: Connection restored to service MGS using nid 192.168.33.1@tcp.<br>LustreError: 3157:0:(obd_mount.c:1368:server_put_super()) no obd tmonster-OSTffff<br>
LustreError: 3157:0:(obd_mount.c:119:server_deregister_mount()) tmonster-OSTffff not registered<br>LustreError: 11-0: an error occurred while communicating with 192.168.33.1@tcp. The mgs_disconnect operation failed with -107<br>
LDISKFS-fs: mballoc: 0 blocks 0 reqs (0 success)<br>LDISKFS-fs: mballoc: 0 extents scanned, 0 goal hits, 0 2^N hits, 0 breaks, 0 lost<br>LDISKFS-fs: mballoc: 0 generated and it took 0<br>LDISKFS-fs: mballoc: 0 preallocated, 0 discarded<br>
Lustre: server umount tmonster-OSTffff complete<br>LustreError: 3157:0:(obd_mount.c:1924:lustre_fill_super()) Unable to mount (-5)<br><br><br>MGS/MDT dmesg: (some of these are certainly unrelated to the OST's mount cmd)<br>
<br>Lustre: 2807:0:(ldlm_lib.c:519:target_handle_reconnect()) tmonster-MDT0000: 3d7d98f5-470c-4188-8023-6c0023150148 reconnecting<br>Lustre: 2807:0:(ldlm_lib.c:519:target_handle_reconnect()) Skipped 2 previous similar messages<br>
Lustre: 2806:0:(mds_reint.c:362:mds_steal_ack_locks()) Stealing 1 locks from rs ffff81006ce30000 x895702.t458401 o101 NID 192.168.19.14@tcp<br>Lustre: 2813:0:(service.c:751:ptlrpc_server_handle_reply()) All locks stolen from rs ffff81006ce30000 x895702.t458401 o101 NID 192.168.19.14@tcp<br>
Lustre: 2797:0:(mds_reint.c:362:mds_steal_ack_locks()) Stealing 1 locks from rs ffff81007f93d000 x666557.t458402 o101 NID 192.168.19.15@tcp<br>Lustre: 2817:0:(service.c:751:ptlrpc_server_handle_reply()) All locks stolen from rs ffff81007f93d000 x666557.t458402 o101 NID 192.168.19.15@tcp<br>
Lustre: 2688:0:(router.c:167:lnet_notify()) Ignoring prediction from 192.168.33.1@tcp of 192.168.33.5@tcp down 7854805405 seconds in the future<br>Lustre: 2780:0:(ldlm_lib.c:519:target_handle_reconnect()) MGS: 8eba281a-43bd-3fa2-2491-fbab892dc02c reconnecting<br>
Lustre: 2780:0:(ldlm_lib.c:519:target_handle_reconnect()) Skipped 2 previous similar messages<br>Lustre: MGS: haven't heard from client 8eba281a-43bd-3fa2-2491-fbab892dc02c (at 192.168.33.5@tcp) in 72 seconds. I think it's dead, and I am evicting it.<br>
LustreError: 2780:0:(mgs_handler.c:515:mgs_handle()) lustre_mgs: operation 251 on unconnected MGS<br>LustreError: 2780:0:(ldlm_lib.c:1442:target_send_reply_msg()) @@@ processing error (-107) req@ffff810073150050 x7/t0 o251-><?>@<?>:-1 lens 128/0 ref 0 fl Interpret:/0/0 rc -107/0<br>
LustreError: 2780:0:(ldlm_lib.c:1442:target_send_reply_msg()) Skipped 2 previous similar messages<br><br><br>Mounting the desired lustre filesystem as client on the OST that is having problems --<br><br># mount -t lustre tm01@tcp0:/tmonster /mnt/tmonster<br>
# df -h /mnt/tmonster<br>Filesystem Size Used Avail Use% Mounted on<br>tm01@tcp0:/tmonster 2.8T 181G 2.6T 7% /mnt/tmonster<br><br>I have replaced this OSTs hardware (utilizing same boot/OST disks in different blade) to no avail. Any help is highly appreciated.<br>
<br>Thanks,<br>Abe<br><br>
</blockquote></div><br>