<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>Dear Lustre users and developers</div><div><br></div><div>I couldn't find a solution to work around this problem. So I was hoping that restarting the MGS/MDT would be a good try. But I was definitely wrong. When trying to remote the MGS/MDT device I got the same error:</div><div><br></div><div><div>Aug 31 03:27:59 lustre01 LDISKFS FS on sde, internal journal</div><div>Aug 31 03:27:59 lustre01 LDISKFS-fs: recovery complete.</div><div>Aug 31 03:27:59 lustre01 LDISKFS-fs: mounted filesystem with ordered data mode.</div><div>Aug 31 03:27:59 lustre01 kjournald starting. Commit interval 5 seconds</div><div>Aug 31 03:27:59 lustre01 LDISKFS FS on sde, internal journal</div><div>Aug 31 03:27:59 lustre01 LDISKFS-fs: mounted filesystem with ordered data mode.</div><div>Aug 31 03:27:59 lustre01 Lustre: MGS MGS started</div><div>Aug 31 03:27:59 lustre01 Lustre: Enabling user_xattr</div><div>Aug 31 03:27:59 lustre01 Lustre: 6934:0:(mds_fs.c:446:mds_init_server_data()) RECOVERY: service homefs-MDT0000, 26 recoverable clients, last_transno 5217310552</div><div>Aug 31 03:27:59 lustre01 Lustre: MDT homefs-MDT0000 now serving dev (homefs-MDT0000/983b4a03-68de-a879-44c3-b91decd23fba), but will be in recovery until 26 clients reconnect, or if no clients reconnect for 4:10; during that time new clients will not be allowed to connect. Recovery progress can be monitored by watching /proc/fs/lustre/mds/homefs-MDT0000/recovery_status.</div><div>Aug 31 03:27:59 lustre01 Lustre: 6934:0:(lproc_mds.c:260:lprocfs_wr_group_upcall()) homefs-MDT0000: group upcall set to /usr/sbin/l_getgroups</div><div>Aug 31 03:27:59 lustre01 Lustre: homefs-MDT0000.mdt: set parameter group_upcall=/usr/sbin/l_getgroups</div><div>Aug 31 03:27:59 lustre01 Lustre: 6934:0:(mds_lov.c:858:mds_notify()) MDS homefs-MDT0000: in recovery, not resetting orphans on homefs-OST0001_UUID</div><div>Aug 31 03:27:59 lustre01 Lustre: 6934:0:(mds_lov.c:858:mds_notify()) MDS homefs-MDT0000: in recovery, not resetting orphans on homefs-OST0004_UUID</div><div>Aug 31 03:27:59 lustre01 LustreError: 6842:0:(events.c:55:request_out_callback()) @@@ type 4, status -5 req@ffff81011b56a400 x11/t0 o8-><a href="mailto:homefs-OST0003_UUID@10.1.140.2">homefs-OST0003_UUID@10.1.140.2</a>@tcp:6 lens 240/272 ref 2 fl Rpc:/0/0 rc 0/-22</div><div>Aug 31 03:27:59 lustre01 LustreError: 6842:0:(client.c:975:ptlrpc_expire_one_request()) @@@ network error (sent at 1220146079, 0s ago) req@ffff81011b56a400 x11/t0 o8-><a href="mailto:homefs-OST0003_UUID@10.1.140.2">homefs-OST0003_UUID@10.1.140.2</a>@tcp:6 lens 240/272 ref 1 fl Rpc:/0/0 rc 0/-22</div><div>Aug 31 03:27:59 lustre01 LustreError: 6842:0:(events.c:55:request_out_callback()) @@@ type 4, status -5 req@ffff81011b5bfa00 x13/t0 o8-><a href="mailto:homefs-OST0006_UUID@10.1.140.2">homefs-OST0006_UUID@10.1.140.2</a>@tcp:6 lens 240/272 ref 2 fl Rpc:/0/0 rc 0/-22</div><div>Aug 31 03:27:59 lustre01 LustreError: 6842:0:(client.c:975:ptlrpc_expire_one_request()) @@@ network error (sent at 1220146079, 0s ago) req@ffff81011b5bfa00 x13/t0 o8-><a href="mailto:homefs-OST0006_UUID@10.1.140.2">homefs-OST0006_UUID@10.1.140.2</a>@tcp:6 lens 240/272 ref 1 fl Rpc:/0/0 rc 0/-22</div><div>Aug 31 03:27:59 lustre01 LustreError: 6934:0:(obd_config.c:897:class_process_proc_param()) homefs-OST0002-osc: unknown param activate=0</div><div>Aug 31 03:27:59 lustre01 LustreError: 6934:0:(obd_config.c:1062:class_config_llog_handler()) Err -22 on cfg command:</div><div>Aug 31 03:27:59 lustre01 Lustre: cmd=cf00f 0:homefs-OST0002-osc 1:osc.activate=0 </div><div>Aug 31 03:27:59 lustre01 LustreError: 15b-f: MGC10.1.140.2@tcp: The configuration from log 'homefs-MDT0000' failed (-22). Make sure this client and the MGS are running compatible versions of Lustre.</div><div>Aug 31 03:27:59 lustre01 LustreError: 15c-8: MGC10.1.140.2@tcp: The configuration from log 'homefs-MDT0000' failed (-22). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information.</div><div>Aug 31 03:27:59 lustre01 LustreError: 6934:0:(obd_mount.c:1080:server_start_targets()) failed to start server homefs-MDT0000: -22</div><div>Aug 31 03:27:59 lustre01 LustreError: 6934:0:(obd_mount.c:1570:server_fill_super()) Unable to start targets: -22</div><div>Aug 31 03:27:59 lustre01 Lustre: Failing over homefs-MDT0000</div><div>Aug 31 03:27:59 lustre01 Lustre: *** setting obd homefs-MDT0000 device 'unknown-block(8,64)' read-only ***</div><div>Aug 31 03:27:59 lustre01 Turning device sde (0x800040) read-only</div><div>Aug 31 03:27:59 lustre01 Lustre: MGS has stopped.</div><br></div><div>Still here the -22 (unknown parameter -> homefs-OST0002-osc: unknown param activate=0) error is haunting me. WTF?! Where does this comes from? It doesn't make any sense to me. When trying to mount the MGS/MDT device a second time I get a kernel soft-lockup:</div><div><br></div><div><div>Aug 31 03:34:32 lustre01 LustreError: 7456:0:(mgs_handler.c:150:mgs_setup()) ASSERTION(!lvfs_check_rdonly(lvfs_sbdev(mnt->mnt_sb))) failed</div><div>Aug 31 03:34:32 lustre01 LustreError: 7456:0:(tracefile.c:431:libcfs_assertion_failed()) LBUG</div><div>Aug 31 03:34:32 lustre01 Lustre: 7456:0:(linux-debug.c:168:libcfs_debug_dumpstack()) showing stack for process 7456</div><div>Aug 31 03:34:32 lustre01 mount.lustre R running task 0 7456 7455 (NOTLB)</div><div>Aug 31 03:34:32 lustre01 ffff810077c9d598 000000000000000c 0000000000009c72 0000000000000004</div><div>Aug 31 03:34:32 lustre01 0000000000000004 0000000000000000 0000000000009c55 0000000000000004</div><div>Aug 31 03:34:32 lustre01 0000000000000018 ffff81011c54c180 0000000000000000 00000000ffffffff</div><div>Aug 31 03:34:32 lustre01 Call Trace:</div><div>Aug 31 03:34:32 lustre01 [<ffffffff80249faa>] module_text_address+0x3a/0x50</div><div>Aug 31 03:34:32 lustre01 [<ffffffff80240ada>] kernel_text_address+0x1a/0x30</div><div>Aug 31 03:34:32 lustre01 [<ffffffff80240ada>] kernel_text_address+0x1a/0x30</div><div>Aug 31 03:34:32 lustre01 [<ffffffff8020b3ba>] show_trace+0x20a/0x240</div><div>Aug 31 03:34:32 lustre01 [<ffffffff8020b4fb>] _show_stack+0xeb/0x100</div><div>Aug 31 03:34:32 lustre01 [<ffffffff880869fa>] :libcfs:lbug_with_loc+0x7a/0xc0</div><div>Aug 31 03:34:32 lustre01 [<ffffffff8808e724>] :libcfs:libcfs_assertion_failed+0x54/0x60</div><div>Aug 31 03:34:32 lustre01 [<ffffffff88307a71>] :mgs:cleanup_module+0xa71/0x2470</div><div>Aug 31 03:34:32 lustre01 [<ffffffff880f05cd>] :obdclass:class_new_export+0x52d/0x5b0</div><div>Aug 31 03:34:32 lustre01 [<ffffffff88105cdb>] :obdclass:class_setup+0x8bb/0xbe0</div><div>Aug 31 03:34:32 lustre01 [<ffffffff8810836a>] :obdclass:class_process_config+0x14ca/0x19f0</div><div>Aug 31 03:34:32 lustre01 [<ffffffff88112d94>] :obdclass:do_lcfg+0x9d4/0x15f0</div><div>Aug 31 03:34:32 lustre01 [<ffffffff8042b475>] scsi_disk_put+0x35/0x50</div><div>Aug 31 03:34:32 lustre01 [<ffffffff88114bd0>] :obdclass:lustre_common_put_super+0x1220/0x6890</div><div>Aug 31 03:34:32 lustre01 [<ffffffff88119a3f>] :obdclass:lustre_common_put_super+0x608f/0x6890</div><div>Aug 31 03:34:32 lustre01 [<ffffffff80293405>] __d_lookup+0x85/0x120</div><div>Aug 31 03:34:32 lustre01 [<ffffffff88086f48>] :libcfs:cfs_alloc+0x28/0x60</div><div>Aug 31 03:34:32 lustre01 [<ffffffff8810d8bf>] :obdclass:lustre_init_lsi+0x29f/0x660</div><div>Aug 31 03:34:32 lustre01 [<ffffffff8811a240>] :obdclass:lustre_fill_super+0x0/0x1ae0</div><div>Aug 31 03:34:32 lustre01 [<ffffffff8811bba3>] :obdclass:lustre_fill_super+0x1963/0x1ae0</div><div>Aug 31 03:34:32 lustre01 [<ffffffff802822d0>] set_anon_super+0x0/0xc0</div><div>Aug 31 03:34:32 lustre01 [<ffffffff8811a240>] :obdclass:lustre_fill_super+0x0/0x1ae0</div><div>Aug 31 03:34:32 lustre01 [<ffffffff80282583>] get_sb_nodev+0x63/0xe0</div><div>Aug 31 03:34:32 lustre01 [<ffffffff80281d62>] vfs_kern_mount+0x62/0xb0</div><div>Aug 31 03:34:32 lustre01 [<ffffffff80281e0a>] do_kern_mount+0x4a/0x80</div><div>Aug 31 03:34:32 lustre01 [<ffffffff8029955d>] do_mount+0x6cd/0x770</div><div>Aug 31 03:34:32 lustre01 [<ffffffff80260cb2>] __handle_mm_fault+0x5e2/0xa30</div><div>Aug 31 03:34:32 lustre01 [<ffffffff80384c21>] __up_read+0x21/0xb0</div><div>Aug 31 03:34:32 lustre01 [<ffffffff8021bae7>] do_page_fault+0x447/0x820</div><div>Aug 31 03:34:32 lustre01 [<ffffffff8025a006>] release_pages+0x186/0x1a0</div><div>Aug 31 03:34:32 lustre01 [<ffffffff8025da33>] zone_statistics+0x33/0x90</div><div>Aug 31 03:34:32 lustre01 [<ffffffff8025774b>] __get_free_pages+0x1b/0x40</div><div>Aug 31 03:34:32 lustre01 [<ffffffff8029969b>] sys_mount+0x9b/0x100</div><div>Aug 31 03:34:32 lustre01 [<ffffffff80209cf2>] system_call+0x7e/0x83</div><br></div><div>Is there some kind of log that is replayed when mounting the MGS/MDT? Can I clear it to be able to mount the device again? Or is this just an annoying bug? At the moment the entire file system is down. Is there a way to bring it back online or do I have to reformat it?</div><div><br></div><div>Any help/hints/advice would be appreciated. I really cannot see where I made a mistake.</div><div><br></div><div>Kind regards,</div><div>Reto Gantenbein</div><div><br></div><div><br></div><br><div><div>On Aug 29, 2008, at 3:12 PM, Reto Gantenbein wrote:</div><br><blockquote type="cite"><div>Dear Lustre users<br><br>Some days ago we had a problem that four OSTs were disconnecting <br>themselves. To recover, I deactivated them with 'lctl conf_param <br>homefs-OST0002.osc.active=0', remounted them and waited until they <br>were recovered and activated them again. Some hosts which kept the <br>Lustre file system mounted at this time, resumed to work correctly on <br>the paused devices.<br><br>But when I want to mount Lustre with on a new client:<br><br>node01 ~ # mount -t lustre lustre01@tcp:lustre02@tcp:/homefs /home<br><br> it refuses with the following message:<br><br>LustreError: 3794:0:(obd_config.c:897:class_process_proc_param()) <br>homefs-OST0002-osc-ffff81022f630000: unknown param activate=0<br>LustreError: 3794:0:(obd_config.c:1062:class_config_llog_handler()) <br>Err -22 on cfg command:<br>Lustre: cmd=cf00f 0:homefs-OST0002-osc 1:osc.activate=0<br>LustreError: 15b-f: MGC10.1.140.1@tcp: The configuration from log <br>'homefs-client' failed (-22). Make sure this client and the MGS are <br>running compatible versions of Lustre.<br>LustreError: 15c-8: MGC10.1.140.1@tcp: The configuration from log <br>'homefs-client' failed (-22). This may be the result of communication <br>errors between this node and the MGS, a bad configuration, or other <br>errors. See the syslog for more information.<br>LustreError: 3794:0:(llite_lib.c:1021:ll_fill_super()) Unable to <br>process log: -22<br>LustreError: 3794:0:(mdc_request.c:1273:mdc_precleanup()) client <br>import never connected<br>LustreError: 3794:0:(connection.c:142:ptlrpc_put_connection()) NULL <br>connection<br>Lustre: client ffff81022f630000 umount complete<br>LustreError: 3794:0:(obd_mount.c:1924:lustre_fill_super()) Unable to <br>mount (-22)<br><br>There are no wrong parameters because the same command did work on all <br>the previous attempts. Also there is no connection problem between the <br>hosts:<br><br>lctl > peer_list<br>12345-10.1.140.1@tcp [1]node01->lustre01:988 #6<br>12345-10.1.140.2@tcp [1]node01->lustre02:988 #6<br><br>Why does this cfg command error arise? homefs-OST0002 is properly <br>mounted on the lustre-server and is fully working with the other <br>clients, as far as I can say. Any hints about this or anything I can <br>do to troubleshoot this problem?<br><br>Kind regards,<br>Reto Gantenbein<br><br><br><br><br>_______________________________________________<br>Lustre-discuss mailing list<br><a href="mailto:Lustre-discuss@lists.lustre.org">Lustre-discuss@lists.lustre.org</a><br><a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a><br></div></blockquote></div><br></body></html>