<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 12 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.emailquote, li.emailquote, div.emailquote
{mso-style-name:emailquote;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:1.0pt;
border:none;
padding:0in;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:798844639;
mso-list-template-ids:-1747013880;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Hi,<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>It appears that I’ve resolved the issue and therefore wanted to provide an update to this list. As I noted in the description of my configuration, the client only has a single IB interface. After changing the options for lnet in “/etc/modprobe.conf” (on the client) from “options lnet networks=o2ib0(ib0)” to “options lnet networks=o2ib0(ib0),o2ib1(ib0)”, things started working.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Now, I said “appears” above because I am seeing an issue that I’ve not seen in the past. Occasionally, while testing workloads with 8 concurrent clients, I see a client being evicted. The stack trace is not always the same. Here’s an excerpt from “/var/log/messages”:<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>Feb 26 11:26:05 bmr2-s14 kernel: Lustre: 7648:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1361895936/real 0] req@ffff81013fe3d800 x1428048654366757/t0(0) o4->lustre2-OST0015-osc-ffff810229235c00@192.168.1.31@o2ib1:6/4 lens 456/416 e 0 to 1 dl 1361895943 ref 3 fl Rpc:X/0/ffffffff rc 0/-1<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>Feb 26 11:26:05 bmr2-s14 kernel: Lustre: 7648:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 10 previous similar messages<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>Feb 26 11:26:05 bmr2-s14 kernel: Lustre: lustre2-OST0010-osc-ffff810229235c00: Connection to lustre2-OST0010 (at 192.168.1.20@o2ib) was lost;<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'> in progress operations using this service will wait for recovery to complete<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>Feb 26 11:26:05 bmr2-s14 kernel: Lustre: Skipped 2 previous similar messages<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>Feb 26 11:26:21 bmr2-s14 kernel: Lustre: 7647:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1361895964/real 0] req@ffff8102438a2800 x1428048654378315/t0(0) o400->lustre2-OST0010-osc-ffff810229235c00@192.168.1.20@o2ib:28/4 lens 192/192 e 0 to 1 dl 1361895981 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>Feb 26 11:26:21 bmr2-s14 kernel: Lustre: 7647:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 14 previous similar messages<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>Feb 26 11:26:30 bmr2-s14 kernel: Lustre: lustre2-OST0013-osc-ffff810229235c00: Connection restored to lustre2-OST0013 (at 192.168.1.31@o2ib1)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>Feb 26 11:26:30 bmr2-s14 kernel: Lustre: Skipped 8 previous similar messages<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>Feb 26 11:26:32 bmr2-s14 kernel: LNetError: 7580:0:(o2iblnd_cb.c:2989:kiblnd_check_txs_locked()) Timed out tx: active_txs, 3 seconds<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>Feb 26 11:26:32 bmr2-s14 kernel: LNetError: 7580:0:(o2iblnd_cb.c:3052:kiblnd_check_conns()) Timed out RDMA with 192.168.1.20@o2ib (55): c: 8, oc: 0, rc: 16<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>Feb 26 11:27:21 bmr2-s14 kernel: Lustre: 7644:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1361896015/real 0] req@ffff810082199800 x1428048654380582/t0(0) o8->lustre2-OST0010-osc-ffff810229235c00@192.168.1.20@o2ib:28/4 lens 368/512 e 0 to 1 dl 1361896041 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>Feb 26 11:27:21 bmr2-s14 kernel: Lustre: 7644:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 7 previous similar messages<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>Feb 26 11:29:11 bmr2-s14 kernel: Lustre: 7644:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1361896115/real 0] req@ffff81009d25ec00 x1428048654380680/t0(0) o8->lustre2-OST0010-osc-ffff810229235c00@192.168.1.20@o2ib:28/4 lens 368/512 e 0 to 1 dl 1361896151 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>Feb 26 11:26:30 bmr2-s14 kernel: Lustre: Skipped 8 previous similar messages<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>Feb 26 11:26:32 bmr2-s14 kernel: LNetError: 7580:0:(o2iblnd_cb.c:2989:kiblnd_check_txs_locked()) Timed out tx: active_txs, 3 seconds<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>Feb 26 11:26:32 bmr2-s14 kernel: LNetError: 7580:0:(o2iblnd_cb.c:3052:kiblnd_check_conns()) Timed out RDMA with 192.168.1.20@o2ib (55): c: 8, oc: 0, rc: 16<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>Feb 26 11:27:21 bmr2-s14 kernel: Lustre: 7644:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1361896015/real 0] req@ffff810082199800 x1428048654380582/t0(0) o8->lustre2-OST0010-osc-ffff810229235c00@192.168.1.20@o2ib:28/4 lens 368/512 e 0 to 1 dl 1361896041 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>Feb 26 11:27:21 bmr2-s14 kernel: Lustre: 7644:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 7 previous similar messages<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>Feb 26 11:29:11 bmr2-s14 kernel: Lustre: 7644:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1361896115/real 0] req@ffff81009d25ec00 x1428048654380680/t0(0) o8->lustre2-OST0010-osc-ffff810229235c00@192.168.1.20@o2ib:28/4 lens 368/512 e 0 to 1 dl 1361896151 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>Feb 26 11:29:11 bmr2-s14 kernel: Lustre: 7644:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 5 previous similar messages<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>Feb 26 11:29:15 bmr2-s14 kernel: INFO: task iozone:9201 blocked for more than 120 seconds.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>Feb 26 11:29:15 bmr2-s14 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>Feb 26 11:29:15 bmr2-s14 kernel: iozone D ffffffff801546d1 0 9201 1 9202 9269 7846 (NOTLB)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>Feb 26 11:29:15 bmr2-s14 kernel: ffff8101278f5aa8 0000000000000082 ffff8101278f5ab8 ffffffff80062ff2<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>Feb 26 11:29:15 bmr2-s14 kernel: ffff81021dbaddf0 0000000000000007 ffff81014f521820 ffff810108617100<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>Feb 26 11:29:15 bmr2-s14 kernel: 00003976bb915dac 0000000000001fbe ffff81014f521a08 000000018006ec8f<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>Feb 26 11:29:15 bmr2-s14 kernel: Call Trace:<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>Feb 26 11:29:15 bmr2-s14 kernel: [<ffffffff80062ff2>] thread_return+0x62/0xfe<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>Feb 26 11:29:15 bmr2-s14 kernel: [<ffffffff8006ec8f>] do_gettimeofday+0x40/0x90<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>Feb 26 11:29:15 bmr2-s14 kernel: [<ffffffff80028d0e>] sync_page+0x0/0x43<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>Feb 26 11:29:15 bmr2-s14 kernel: [<ffffffff800637ce>] io_schedule+0x3f/0x67<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>Feb 26 11:29:15 bmr2-s14 kernel: [<ffffffff80028d4c>] sync_page+0x3e/0x43<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>Feb 26 11:29:15 bmr2-s14 kernel: [<ffffffff800639fa>] __wait_on_bit+0x40/0x6e<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>Feb 26 11:29:15 bmr2-s14 kernel: [<ffffffff800350d9>] wait_on_page_bit+0x6c/0x72<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>Feb 26 11:29:15 bmr2-s14 kernel: [<ffffffff800a2e8b>] wake_bit_function+0x0/0x23<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>Feb 26 11:29:15 bmr2-s14 kernel: [<ffffffff80047cae>] pagevec_lookup_tag+0x1a/0x21<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>Feb 26 11:29:15 bmr2-s14 kernel: [<ffffffff8001d19f>] mpage_writepages+0x18d/0x37d<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>Feb 26 11:29:15 bmr2-s14 kernel: [<ffffffff88e7f850>] :lustre:ll_writepage+0x0/0x430<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>Feb 26 11:29:15 bmr2-s14 kernel: [<ffffffff8005a8a6>] do_writepages+0x20/0x2f<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>Feb 26 11:29:15 bmr2-s14 kernel: [<ffffffff8004f767>] __filemap_fdatawrite_range+0x50/0x5b<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>Feb 26 11:29:15 bmr2-s14 kernel: [<ffffffff800c8cf4>] sync_page_range+0x3d/0xa0<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>Feb 26 11:29:15 bmr2-s14 kernel: [<ffffffff800c8ff2>] generic_file_writev+0x8a/0xa3<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>Feb 26 11:29:15 bmr2-s14 kernel: [<ffffffff88ea430d>] :lustre:vvp_io_write_start+0xfd/0x1b0<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>Feb 26 11:29:15 bmr2-s14 kernel: [<ffffffff88aaea50>] :obdclass:cl_io_start+0x90/0xf0<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>Feb 26 11:29:15 bmr2-s14 kernel: [<ffffffff88ab1718>] :obdclass:cl_io_loop+0x88/0x130<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>Feb 26 11:29:15 bmr2-s14 kernel: [<ffffffff88e5d16e>] :lustre:ll_file_io_generic+0x43e/0x480<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>Feb 26 11:29:15 bmr2-s14 kernel: [<ffffffff88e5d335>] :lustre:ll_file_writev+0x185/0x1f0<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>Feb 26 11:29:15 bmr2-s14 kernel: [<ffffffff88e66a71>] :lustre:ll_file_write+0x121/0x190<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>Feb 26 11:29:15 bmr2-s14 kernel: [<ffffffff80016b92>] vfs_write+0xce/0x174<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>Feb 26 11:29:15 bmr2-s14 kernel: [<ffffffff8001745b>] sys_write+0x45/0x6e<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>Feb 26 11:29:15 bmr2-s14 kernel: [<ffffffff8005d28d>] tracesys+0xd5/0xe0<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>Feb 26 11:29:15 bmr2-s14 kernel:<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Here’s some additional info showing loss of connection to 3 of the 6 OSTs located on this OSS (on the .20@o2ib interface):<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>[root@bmr2-s14 ~]# cat /proc/fs/lustre/osc/lustre2-OST*-osc-ffff810229235c00/ost_conn_uuid <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>192.168.1.25@o2ib<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>192.168.1.35@o2ib1<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>192.168.1.25@o2ib<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>192.168.1.35@o2ib1<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>192.168.1.25@o2ib<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>192.168.1.35@o2ib1<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>192.168.1.26@o2ib<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>192.168.1.36@o2ib1<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>192.168.1.26@o2ib<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>192.168.1.36@o2ib1<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>192.168.1.26@o2ib<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>192.168.1.36@o2ib1<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>192.168.1.20@o2ib<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>192.168.1.30@o2ib1<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>192.168.1.20@o2ib<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>192.168.1.30@o2ib1<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>192.168.1.20@o2ib<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>192.168.1.30@o2ib1<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>192.168.1.21@o2ib<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>192.168.1.31@o2ib1<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>192.168.1.21@o2ib<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>192.168.1.31@o2ib1<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>192.168.1.21@o2ib<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>192.168.1.31@o2ib1<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>[root@bmr2-s14 ~]# cat /proc/fs/lustre/osc/lustre2-OST*-osc-ffff810229235c00/ost_server_uuid<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>lustre2-OST0000_UUID FULL<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>lustre2-OST0001_UUID FULL<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>lustre2-OST0002_UUID FULL<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>lustre2-OST0003_UUID FULL<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>lustre2-OST0004_UUID FULL<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>lustre2-OST0005_UUID FULL<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>lustre2-OST0006_UUID FULL<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>lustre2-OST0007_UUID FULL<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>lustre2-OST0008_UUID FULL<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>lustre2-OST0009_UUID FULL<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>lustre2-OST000a_UUID FULL<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>lustre2-OST000b_UUID FULL<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>lustre2-OST000c_UUID CONNECTING<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>lustre2-OST000d_UUID FULL<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>lustre2-OST000e_UUID CONNECTING<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>lustre2-OST000f_UUID FULL<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>lustre2-OST0010_UUID CONNECTING<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>lustre2-OST0011_UUID FULL<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>lustre2-OST0012_UUID FULL<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>lustre2-OST0013_UUID FULL<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>lustre2-OST0014_UUID FULL<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>lustre2-OST0015_UUID FULL<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>lustre2-OST0016_UUID FULL<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>lustre2-OST0017_UUID FULL<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>[root@bmr2-s14 ~]#<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Based on some research, I’ve experimented with setting “options ko2iblnd peer_credits=16 concurrent_sends=16” in /etc/modprobe.conf and this has made the issue occur less frequently. However, it is still occurring. I’m not sure if this has something to do with both server interfaces being located on the same network or something else.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Any input would be appreciated.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Thanks,<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>…Brian<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> lustre-discuss-bounces@lists.lustre.org [mailto:lustre-discuss-bounces@lists.lustre.org] <b>On Behalf Of </b>mages, brian<br><b>Sent:</b> Tuesday, February 12, 2013 3:34 PM<br><b>To:</b> lustre-discuss@lists.lustre.org<br><b>Subject:</b> [Lustre-discuss] Multirail IB Configuration Issue<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'>Hi,<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'> <o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'>I’m having difficulty getting one of my clients to work with a multirail IB configuration. Here’s what I’ve got:<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'> <o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'> <b>Host OS Version Lustre Version Function Storage Interface ib0 Interface ib1</b><o:p></o:p></span></p></div><ol start=1 type=1><li class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo1'><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'>bmr1-s7 CentOS 5.7 2.1.1 MGS,MDS,OSS1 mdt,mdt2,ost1->6,ost13->18 192.168.1.25/24 192.168.1.35/24<o:p></o:p></span></li><li class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo1'><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'>bmr1-s8 CentOS 5.7 2.1.1 OSS2 ost7->12,ost19->24 192.168.1.26/24 192.168.1.36/24<o:p></o:p></span></li><li class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo1'><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'>bmr1-s5 CentOS 5.7 2.1.1 OSS3 ost25->30 192.168.1.20/24 192.168.1.30/24<o:p></o:p></span></li><li class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo1'><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'>bmr1-s6 CentOS 5.7 2.1.1 OSS4 ost31->36 192.168.1.21/24 192.168.1.31/24<o:p></o:p></span></li><li class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo1'><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'>bmr2-s9 CentOS 5.7 2.1.1 Client n/a 192.168.1.209/24<o:p></o:p></span></li></ol><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'> <o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'>The “/lustre” filesystem consists of mdt and ost1->12 (using bmr1-s7 and bmr1-s8).<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'>The “/lustre2” filesystem consists of mdt2 and ost13->36 (using bmr1-s7, bmr1-s8, bmr1-s5, and bmr1-s6).<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'>On each OSS, half the OSTs are available only on ib0 and the other half only on ib1.<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'> <o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'>From bmr1-s5 and bmr1-s6 (using as clients), I can successfully mount and access “/lustre”. I can also successfully mount “/lustre2”.<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'> <o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'>From bmr2-s9, I can neither mount “/lustre” nor “/lustre2”. Originally, the issue with bmr2-s9 was that it was running 1.8.6-wc1 (server on CentOS 5.6). Since this config (i.e., multirail) wasn’t supported on that version, I upgraded to 2.1.1. Originally, I tried installing and testing the 2.1.1 client without success. Then, since it had worked with the 2.1.1 server on both bmr1-s5 and bmr1-s6, I thought I’d try that next. Unfortunately, it still didn’t work.<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'> <o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'>1a) Here’s what I see on the client when I try to mount “/lustre”:<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'> <o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>[root@bmr2-s9 ~]# mount -t lustre <a href="mailto:192.168.1.25@o2ib:/lustre">192.168.1.25@o2ib:/lustre</a> /mnt/lustre</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>mount.lustre: mount <a href="mailto:192.168.1.25@o2ib:/lustre">192.168.1.25@o2ib:/lustre</a> at /mnt/lustre failed: No such file or directory</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Is the MGS specification correct?</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Is the filesystem name correct?</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>If upgrading, is the copied client log valid? (see upgrade docs)</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>[root@bmr2-s9 ~]#</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'> <o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'>1b) Here’s an excerpt from “/var/log/messages” on the client (after executing the above command):<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'> <o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:00:54 bmr2-s9 kernel: Lustre: 5512:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import <a href="mailto:MGC192.168.1.25@o2ib-%3eMGC192.168.1.25@o2ib_0">MGC192.168.1.25@o2ib->MGC192.168.1.25@o2ib_0</a> netid 50000: select flavor null</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:00:54 bmr2-s9 kernel: Lustre: <a href="mailto:MGC192.168.1.25@o2ib">MGC192.168.1.25@o2ib</a>: Reactivating import</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:00:54 bmr2-s9 kernel: LustreError: 5523:0:(ldlm_lib.c:357:client_obd_setup()) can't add initial connection</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:00:54 bmr2-s9 kernel: LustreError: 5523:0:(obd_config.c:522:class_setup()) setup lustre-OST0001-osc-ffff81045d783c00 failed (-2)</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:00:54 bmr2-s9 kernel: LustreError: 5523:0:(obd_config.c:1361:class_config_llog_handler()) Err -2 on cfg command:</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:00:54 bmr2-s9 kernel: Lustre: cmd=cf003 0:lustre-OST0001-osc 1:lustre-OST0001_UUID 2:192.168.1.35@o2ib1 </span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:00:54 bmr2-s9 kernel: LustreError: 15c-8: <a href="mailto:MGC192.168.1.25@o2ib">MGC192.168.1.25@o2ib</a>: The configuration from log 'lustre-client' failed (-2). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information.</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:00:54 bmr2-s9 kernel: LustreError: 5512:0:(llite_lib.c:950:ll_fill_super()) Unable to process log: -2</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:00:54 bmr2-s9 kernel: LustreError: 4923:0:(lov_obd.c:927:lov_cleanup()) lov tgt 0 not cleaned! deathrow=0, lovrc=1</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:00:54 bmr2-s9 kernel: LustreError: 5512:0:(obd_config.c:567:class_cleanup()) Device 5 not setup</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:00:54 bmr2-s9 kernel: LustreError: 5512:0:(ldlm_request.c:1172:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:00:54 bmr2-s9 kernel: LustreError: 5512:0:(ldlm_request.c:1799:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:00:54 bmr2-s9 kernel: Lustre: client ffff81045d783c00 umount complete</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:00:54 bmr2-s9 kernel: LustreError: 5512:0:(obd_mount.c:2164:lustre_fill_super()) Unable to mount (-2)</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'> <o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'>1c) Here’s an excerpt from “/var/log/messages” on the server (after executing the above command):<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'> <o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:00:54 bmr1-s7 kernel: Lustre: 25911:0:(ldlm_lib.c:877:target_handle_connect()) MGS: connection from <a href="mailto:2e13dea0-ec9c-0fbd-0f95-7b16246f2626@192.168.1.209@o2ib">2e13dea0-ec9c-0fbd-0f95-7b16246f2626@192.168.1.209@o2ib</a> t0 exp 0000000000000000 cur 1360699254 last 0</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:00:54 bmr1-s7 kernel: Lustre: 25911:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGS->NET_0x50000c0a801d1_UUID netid 50000: select flavor null</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'> <o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'>2a) Here’s what I see on the client when I try to mount “/lustre” (using the other interface):<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'> <o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>[root@bmr2-s9 ~]# mount -t lustre <a href="mailto:192.168.1.25@o2ib:/lustre">192.168.1.25@o2ib:/lustre</a> /mnt/lustre</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>mount.lustre: mount <a href="mailto:192.168.1.25@o2ib:/lustre">192.168.1.25@o2ib:/lustre</a> at /mnt/lustre failed: No such file or directory</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Is the MGS specification correct?</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Is the filesystem name correct?</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>If upgrading, is the copied client log valid? (see upgrade docs)</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>[root@bmr2-s9 ~]# mount -t lustre <a href="mailto:192.168.1.35@o2ib:/lustre">192.168.1.35@o2ib:/lustre</a> /mnt/lustre</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>mount.lustre: mount <a href="mailto:192.168.1.35@o2ib:/lustre">192.168.1.35@o2ib:/lustre</a> at /mnt/lustre failed: Invalid argument</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>This may have multiple causes.</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Is 'lustre' the correct filesystem name?</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Are the mount options correct?</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Check the syslog for more info.</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>[root@bmr2-s9 ~]#</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'> <o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'>2b) Here’s an excerpt from “/var/log/messages” on the client (after executing the above command):<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'> <o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:06:57 bmr2-s9 kernel: Lustre: 5580:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import <a href="mailto:MGC192.168.1.35@o2ib-%3eMGC192.168.1.35@o2ib_0">MGC192.168.1.35@o2ib->MGC192.168.1.35@o2ib_0</a> netid 50000: select flavor null</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:06:57 bmr2-s9 kernel: Lustre: 4948:0:(client.c:1778:ptlrpc_expire_one_request()) @@@ Request x1426793186721863 sent from <a href="mailto:MGC192.168.1.35@o2ib">MGC192.168.1.35@o2ib</a> to NID <a href="mailto:192.168.1.35@o2ib">192.168.1.35@o2ib</a> has failed due to network error: [sent 1360699617] [real_sent 1360699617] [current 1360699617] [deadline 5s] [delay -5s] req@ffff81043b76e400 x1426793186721863/t0(0) o-1-><a href="mailto:MGS@MGC192.168.1.35@o2ib_0:26/25">MGS@MGC192.168.1.35@o2ib_0:26/25</a> lens 368/512 e 0 to 1 dl 1360699622 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:06:57 bmr2-s9 kernel: Lustre: 4948:0:(client.c:1778:ptlrpc_expire_one_request()) Skipped 1 previous similar message</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:06:57 bmr2-s9 kernel: LustreError: 3074:0:(o2iblnd_cb.c:2615:kiblnd_rejected()) <a href="mailto:192.168.1.35@o2ib">192.168.1.35@o2ib</a> rejected: o2iblnd fatal error</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:06:57 bmr2-s9 kernel: LustreError: 3074:0:(o2iblnd_cb.c:2615:kiblnd_rejected()) Skipped 1 previous similar message</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:07:03 bmr2-s9 kernel: LustreError: 5580:0:(client.c:1049:ptlrpc_import_delay_req()) @@@ send limit expired req@ffff81043b76e000 x1426793186721864/t0(0) o-1-><a href="mailto:MGS@MGC192.168.1.35@o2ib_0:26/25">MGS@MGC192.168.1.35@o2ib_0:26/25</a> lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:W/ffffffff/ffffffff rc 0/-1</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:07:03 bmr2-s9 kernel: LustreError: 5580:0:(client.c:1049:ptlrpc_import_delay_req()) Skipped 6 previous similar messages</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:07:22 bmr2-s9 kernel: Lustre: 4949:0:(import.c:526:import_select_connection()) <a href="mailto:MGC192.168.1.35@o2ib">MGC192.168.1.35@o2ib</a>: tried all connections, increasing latency to 5s</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:07:22 bmr2-s9 kernel: Lustre: 4948:0:(client.c:1778:ptlrpc_expire_one_request()) @@@ Request x1426793186721868 sent from <a href="mailto:MGC192.168.1.35@o2ib">MGC192.168.1.35@o2ib</a> to NID <a href="mailto:192.168.1.35@o2ib">192.168.1.35@o2ib</a> has failed due to network error: [sent 1360699642] [real_sent 1360699642] [current 1360699642] [deadline 10s] [delay -10s] req@ffff810430e30800 x1426793186721868/t0(0) o-1-><a href="mailto:MGS@MGC192.168.1.35@o2ib_0:26/25">MGS@MGC192.168.1.35@o2ib_0:26/25</a> lens 368/512 e 0 to 1 dl 1360699652 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:07:22 bmr2-s9 kernel: LustreError: 3074:0:(o2iblnd_cb.c:2615:kiblnd_rejected()) <a href="mailto:192.168.1.35@o2ib">192.168.1.35@o2ib</a> rejected: o2iblnd fatal error</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:07:24 bmr2-s9 kernel: LustreError: 5591:0:(client.c:1049:ptlrpc_import_delay_req()) @@@ send limit expired req@ffff81045d7ce800 x1426793186721867/t0(0) o-1-><a href="mailto:MGS@MGC192.168.1.35@o2ib_0:26/25">MGS@MGC192.168.1.35@o2ib_0:26/25</a> lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:W/ffffffff/ffffffff rc 0/-1</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:07:24 bmr2-s9 kernel: LustreError: 5591:0:(client.c:1049:ptlrpc_import_delay_req()) Skipped 1 previous similar message</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:07:47 bmr2-s9 kernel: Lustre: 4949:0:(import.c:526:import_select_connection()) <a href="mailto:MGC192.168.1.35@o2ib">MGC192.168.1.35@o2ib</a>: tried all connections, increasing latency to 10s</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:07:47 bmr2-s9 kernel: Lustre: 4948:0:(client.c:1778:ptlrpc_expire_one_request()) @@@ Request x1426793186721872 sent from <a href="mailto:MGC192.168.1.35@o2ib">MGC192.168.1.35@o2ib</a> to NID <a href="mailto:192.168.1.35@o2ib">192.168.1.35@o2ib</a> has failed due to network error: [sent 1360699667] [real_sent 1360699667] [current 1360699667] [deadline 15s] [delay -15s] req@ffff810444576c00 x1426793186721872/t0(0) o-1-><a href="mailto:MGS@MGC192.168.1.35@o2ib_0:26/25">MGS@MGC192.168.1.35@o2ib_0:26/25</a> lens 368/512 e 0 to 1 dl 1360699682 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:07:47 bmr2-s9 kernel: LustreError: 3074:0:(o2iblnd_cb.c:2615:kiblnd_rejected()) <a href="mailto:192.168.1.35@o2ib">192.168.1.35@o2ib</a> rejected: o2iblnd fatal error</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:07:54 bmr2-s9 kernel: LustreError: 156-2: The client profile 'lustre-client' could not be read from the MGS. Does that filesystem exist?</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:07:54 bmr2-s9 kernel: Lustre: client ffff81045f465800 umount complete</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:07:54 bmr2-s9 kernel: LustreError: 5580:0:(obd_mount.c:2164:lustre_fill_super()) Unable to mount (-22)</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'> <o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'>2c) Here’s an excerpt from “/var/log/messages” on the server (after executing the above command):<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'> <o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:06:57 bmr1-s7 kernel: LustreError: 9274:0:(o2iblnd_cb.c:2247:kiblnd_passive_connect()) Can't accept <a href="mailto:192.168.1.209@o2ib">192.168.1.209@o2ib</a> on <a href="mailto:192.168.1.25@o2ib">192.168.1.25@o2ib</a> (ib1:1:192.168.1.35): bad dst nid <a href="mailto:192.168.1.35@o2ib">192.168.1.35@o2ib</a></span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:06:57 bmr1-s7 kernel: LustreError: 9274:0:(o2iblnd_cb.c:2247:kiblnd_passive_connect()) Skipped 2 previous similar messages</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:07:22 bmr1-s7 kernel: LustreError: 9274:0:(o2iblnd_cb.c:2247:kiblnd_passive_connect()) Can't accept <a href="mailto:192.168.1.209@o2ib">192.168.1.209@o2ib</a> on <a href="mailto:192.168.1.25@o2ib">192.168.1.25@o2ib</a> (ib1:1:192.168.1.35): bad dst nid <a href="mailto:192.168.1.35@o2ib">192.168.1.35@o2ib</a></span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'> <o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'>3) Here’s what one of the MDTs looks like (the other is similarly configured):<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'> <o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>[root@bmr1-s7 ~]# tunefs.lustre --dryrun --writeconf /dev/sdp</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>checking for existing Lustre data: found CONFIGS/mountdata</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Reading CONFIGS/mountdata</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'> </span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'> Read previous values:</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Target: lustre-MDT0000</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Index: 0</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Lustre FS: lustre</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Mount type: ldiskfs</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Flags: 0x5</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'> (MDT MGS )</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Persistent mount opts: iopen_nopriv,user_xattr,errors=remount-ro</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Parameters: <a href="mailto:mgsnode=192.168.1.25@o2ib,192.168.1.35@o2ib1,10.244.78.88@tcp,192.168.1.25@tcp1">mgsnode=192.168.1.25@o2ib,192.168.1.35@o2ib1,10.244.78.88@tcp,192.168.1.25@tcp1</a></span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'> <o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'> <o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'> Permanent disk data:</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Target: lustre-MDT0000</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Index: 0</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Lustre FS: lustre</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Mount type: ldiskfs</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Flags: 0x105</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'> (MDT MGS writeconf )</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Persistent mount opts: iopen_nopriv,user_xattr,errors=remount-ro</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Parameters: <a href="mailto:mgsnode=192.168.1.25@o2ib,192.168.1.35@o2ib1,10.244.78.88@tcp,192.168.1.25@tcp1">mgsnode=192.168.1.25@o2ib,192.168.1.35@o2ib1,10.244.78.88@tcp,192.168.1.25@tcp1</a></span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'> <o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>exiting before disk write.</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>[root@bmr1-s7 ~]#</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'> <o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'>4) Here’s what one of the OSTs looks like (the others are similarly configured):<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'> <o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>[root@bmr1-s7 ~]# tunefs.lustre --dryrun --writeconf /dev/sdf</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>checking for existing Lustre data: found CONFIGS/mountdata</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Reading CONFIGS/mountdata</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'> </span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'> Read previous values:</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Target: lustre-OST0000</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Index: 0</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Lustre FS: lustre</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Mount type: ldiskfs</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Flags: 0x2</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'> (OST )</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Persistent mount opts: errors=remount-ro,extents,mballoc</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Parameters: <a href="mailto:mgsnode=192.168.1.25@o2ib,192.168.1.35@o2ib1,10.244.78.88@tcp,192.168.1.25@tcp1">mgsnode=192.168.1.25@o2ib,192.168.1.35@o2ib1,10.244.78.88@tcp,192.168.1.25@tcp1</a> network=o2ib0</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'> </span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'> </span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'> Permanent disk data:</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Target: lustre-OST0000</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Index: 0</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Lustre FS: lustre</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Mount type: ldiskfs</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Flags: 0x102</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'> (OST writeconf )</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Persistent mount opts: errors=remount-ro,extents,mballoc</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Parameters: <a href="mailto:mgsnode=192.168.1.25@o2ib,192.168.1.35@o2ib1,10.244.78.88@tcp,192.168.1.25@tcp1">mgsnode=192.168.1.25@o2ib,192.168.1.35@o2ib1,10.244.78.88@tcp,192.168.1.25@tcp1</a> network=o2ib0</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'> </span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>exiting before disk write.</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>[root@bmr1-s7 ~]#</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'> <o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'>I’d appreciate any help or direction on a potential resolution. Let me know what additional information is needed, if any. Hopefully, I’m just missing something simple.<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'> <o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'>Thanks in advance,<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'>…Brian<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'> <o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Calibri","sans-serif"'> <o:p></o:p></span></p></div></div></body></html>