<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
I\u2019m currently investigating a problem with MOFED 4.x which seems very similar to what you seeing. I have no solution yet.
<div class=""><br class="">
</div>
<div class="">Doug</div>
<div class=""><br class="">
<div>
<blockquote type="cite" class="">
<div class="">On May 7, 2017, at 7:09 AM, HM Li <<a href="mailto:lihm0@163.com" class="">lihm0@163.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div text="#000000" bgcolor="#FFFFFF" class="">
<p class="">Thank you very much.</p>
<p class="">The MLNX used on Multi-Host nodes is MLNX_OFED_LINUX-4.0-1.0.1.0-rhel7.3-x86_64.</p>
<p class="">This driver and lustre(git, 2.9.55_45) can work well on other normal FDR nodes.<br class="">
</p>
<br class="">
<div class="moz-cite-prefix">On 2017\u5e7405\u670806\u65e5 01:14, Oucharek, Doug S wrote:<br class="">
</div>
<blockquote cite="mid:F2457005-6B4B-458F-98C4-DD0F250F5EC5@intel.com" type="cite" class="">
The tag you checked out is missing this fix: <a moz-do-not-send="true" href="https://review.whamcloud.com/#/c/24306/" class="">https://review.whamcloud.com/#/c/24306/</a>. Try applying that.
<div class=""><br class="">
</div>
<div class="">Doug</div>
<div class=""><br class="">
<div class="">
<blockquote type="cite" class="">
<div class="">On May 5, 2017, at 9:51 AM, HM Li <<a moz-do-not-send="true" href="mailto:lihm0@163.com" class="">lihm0@163.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<p style="font-family: Helvetica; font-size: 12px;
font-style: normal; font-variant-caps: normal;
font-weight: normal; letter-spacing: normal; text-align:
start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color:
rgb(255, 255, 255);" class="">
Conformed.</p>
<p style="font-family: Helvetica; font-size: 12px;
font-style: normal; font-variant-caps: normal;
font-weight: normal; letter-spacing: normal; text-align:
start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color:
rgb(255, 255, 255);" class="">
This is a bug of git(2.9.55_45), it works well when using MLNX_OFED_LINUX-3.4-2.1.8.0-rhel7.3-x86_64 and<a moz-do-not-send="true" class="moz-txt-link-freetext" href="https://downloads.hpdd.intel.com/public/lustre/latest-feature-release/el7.3.1611/server/SRPMS/lustre-2.9.0-1.src.rpm">https://downloads.hpdd.intel.com/public/lustre/latest-feature-release/el7.3.1611/server/SRPMS/lustre-2.9.0-1.src.rpm</a>.</p>
<p style="font-family: Helvetica; font-size: 12px;
font-style: normal; font-variant-caps: normal;
font-weight: normal; letter-spacing: normal; text-align:
start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color:
rgb(255, 255, 255);" class="">
node453@tc4600: ~# export LC_ALL=C<br class="">
node453@tc4600: ~# lctl lustre_build_version<br class="">
Lustre version: 2.9.0<br class="">
node453@tc4600: ~# df <span class="Apple-converted-space"> </span><br class="">
Filesystem 1K-blocks Used Available Use% Mounted on<br class="">
10.10.100.6@o2ib:/lxfs 7341068688 10152136 6960149976 1% /home<br class="">
10.10.100.1@o2ib:/sgfs 108704716104 24800951320 78393702908 25% /mnt<br class="">
node453@tc4600: ~# ibv_devinfo<span class="Apple-converted-space"> </span><br class="">
hca_id: mlx5_0<br class="">
transport: InfiniBand (0)<br class="">
fw_ver: 12.17.1010<br class="">
node_guid: 46e3:e861:1f19:4438<br class="">
sys_image_guid: 46e3:e861:1f19:4438<br class="">
vendor_id: 0x02c9<br class="">
vendor_part_id: 4115<br class="">
hw_ver: 0x0<br class="">
board_id: SGN1130110032<br class="">
phys_port_cnt: 1<br class="">
Device ports:<br class="">
port: 1<br class="">
state: PORT_ACTIVE (4)<br class="">
max_mtu: 4096 (5)<br class="">
active_mtu: 4096 (5)<br class="">
sm_lid: 360<br class="">
port_lid: 362<br class="">
port_lmc: 0x00<br class="">
link_layer: InfiniBand<br class="">
<br class="">
</p>
<br style="font-family: Helvetica; font-size: 12px;
font-style: normal; font-variant-caps: normal;
font-weight: normal; letter-spacing: normal; text-align:
start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color:
rgb(255, 255, 255);" class="">
<div class="moz-cite-prefix" style="font-family:
Helvetica; font-size: 12px; font-style: normal;
font-variant-caps: normal; font-weight: normal;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
background-color: rgb(255, 255, 255);">
On 2017\u5e7405\u670803\u65e5 17:02, HM Li wrote:<br class="">
</div>
<blockquote cite="mid:5490037c-1644-d839-868a-198bf20c8957@163.com" type="cite" style="font-family: Helvetica; font-size:
12px; font-style: normal; font-variant-caps: normal;
font-weight: normal; letter-spacing: normal; orphans:
auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; background-color:
rgb(255, 255, 255);" class="">
Dear,<br class="">
I have setup the Lustre-git(2.9.55_45) on CentOS 7.3, but the client(on the same multi-host IB) can't mount lustre. Can you help me? Thank you very much.<br class="">
<ul class="">
<li class="">The<span class="Apple-converted-space"> </span><font class="" color="#ff0000">server</font>:
<ul class="">
<li class="">mkfs.lustre --fsname=lxfs --mgs --mdt --index=0 --reformat /dev/sda5
</li><li class="">mkfs.lustre --fsname=lxfs --mgsnode=10.10.146.1@o2ib1 --servicenode=10.10.146.1@o2ib1 --ost --reformat --index=1 /dev/sda6
</li><li class="">mount -t lustre /dev/sda5 /mnt/mdt<br class="">
</li><li class="">mount -t lustre /dev/sda6 /mnt/ost </li><li class="">mount -t lustre -v 10.10.146.1@o2ib1:/lxfs /home is OK. </li><li class="">lctl list_nids<br class="">
10.10.146.1@o2ib1<br class="">
</li><li class="">lctl ping 10.10.146.2@o2ib1<br class="">
12345-0@lo<br class="">
12345-10.10.146.2@o2ib1<br class="">
<br class="">
</li></ul>
</li><li class="">The<span class="Apple-converted-space"> </span><font class="" color="#3333ff">client</font>:
<ul class="">
<li class="">lctl list_nids<br class="">
10.10.146.2@o2ib1 </li></ul>
<ul class="">
<li class="">lctl ping 10.10.146.1@o2ib1<br class="">
12345-0@lo<br class="">
12345-10.10.146.1@o2ib1 </li></ul>
<ul class="">
<li class="">mount -t lustre -v 10.10.146.1@o2ib1:/lxfs /home/<br class="">
arg[0] = /sbin/mount.lustre<br class="">
arg[1] = -v<br class="">
arg[2] = -o<br class="">
arg[3] = rw<br class="">
arg[4] = 10.10.146.1@o2ib1:/lxfs<br class="">
arg[5] = /home<br class="">
source = 10.10.146.1@o2ib1:/lxfs (10.10.146.1@o2ib1:/lxfs), target = /home<br class="">
options = rw<br class="">
mounting device 10.10.146.1@o2ib1:/lxfs at /home, flags=0x1000000 options=device=10.10.146.1@o2ib1:/lxfs<br class="">
mount.lustre: mount 10.10.146.1@o2ib1:/lxfs at /home failed: Input/output error retries left: 0<br class="">
mount.lustre: mount 10.10.146.1@o2ib1:/lxfs at /home failed: Input/output error<br class="">
Is the MGS running?<br class="">
<br class="">
</li></ul>
</li><li class="">and now on<span class="Apple-converted-space"> </span><font class="" color="#ff0000">server</font><span class="Apple-converted-space"> </span>dmesg show:<br class="">
[82709.336007] Lustre: MGS: Connection restored to 792b2b21-2e57-de7d-3d8f-5e80eb6d7bf2 (at 10.10.146.2@o2ib1)<br class="">
[82709.339324] mlx5_0:dump_cqe:275:(pid 22740): dump error cqe<br class="">
[82709.339508] 00000000 00000000 00000000 00000000<br class="">
[82709.339677] 00000000 00000000 00000000 00000000<br class="">
[82709.339841] 00000000 00000000 00000000 00000000<br class="">
[82709.340006] 00000000 9d005304 08000074 01f1c5d2<br class="">
[82716.343333] Lustre: MGS: Received new LWP connection from 10.10.146.2@o2ib1, removing former export from same NID<br class="">
[82716.343712] Lustre: MGS: Connection restored to 792b2b21-2e57-de7d-3d8f-5e80eb6d7bf2 (at 10.10.146.2@o2ib1)<br class="">
<br class="">
</li><li class=""><font class="" color="#ff0000">IB</font><span class="Apple-converted-space"> </span>information:
<ul class="">
<li class="">ibstat<br class="">
CA 'mlx5_0'<br class="">
CA type: MT4115<br class="">
Number of ports: 1<br class="">
Firmware version: 12.17.1010<br class="">
Hardware version: 0<br class="">
Node GUID: 0x46e3e8611f19443a<br class="">
System image GUID: 0x46e3e8611f194438<br class="">
Port 1:<br class="">
State: Active<br class="">
Physical state: LinkUp<br class="">
Rate: 100<br class="">
Base lid: 360<br class="">
LMC: 2<br class="">
SM lid: 360<br class="">
Capability mask: 0x2651e84a<br class="">
Port GUID: 0x46e3e8611f19443a<br class="">
Link layer: InfiniBand<br class="">
</li><li class="">ibv_devinfo<span class="Apple-converted-space"> </span><br class="">
hca_id: mlx5_0<br class="">
transport: InfiniBand (0)<br class="">
fw_ver: 12.17.1010<br class="">
node_guid: 46e3:e861:1f19:443a<br class="">
sys_image_guid: 46e3:e861:1f19:4438<br class="">
vendor_id: 0x02c9<br class="">
vendor_part_id: 4115<br class="">
hw_ver: 0x0<br class="">
board_id: SGN1130110032<br class="">
phys_port_cnt: 1<br class="">
Device ports:<br class="">
port: 1<br class="">
state: PORT_ACTIVE (4)<br class="">
max_mtu: 4096 (5)<br class="">
active_mtu: 4096 (5)<br class="">
sm_lid: 360<br class="">
port_lid: 360<br class="">
port_lmc: 0x02<br class="">
link_layer: InfiniBand </li><li class="">ibstatus<span class="Apple-converted-space"> </span><br class="">
Infiniband device 'mlx5_0' port 1 status:<br class="">
default gid: fe80:0000:0000:0000:46e3:e861:1f19:443a<br class="">
base lid: 0x168<br class="">
sm lid: 0x168<br class="">
state: 4: ACTIVE<br class="">
phys state: 5: LinkUp<br class="">
rate: 100 Gb/sec (4X EDR)<br class="">
link_layer: InfiniBand </li><li class=""><span class="" lang="EN-US">opensm --create-config /etc/opensm/opensm.conf</span>
</li><li class="">/etc/opensm/opensm.conf has been modified:<br class="">
virt enabled 2<br class="">
qos TRUE<br class="">
lmc 2<br class="">
</li></ul>
</li></ul>
<ul class="">
<li class="">Other information:
<ul class="">
<li class="">selinux disabled </li><li class="">iptables cleaned </li><li class="">uname -r: 3.10.0-514.16.1.el7_lustre.x86_64 </li><li class="">OS: CentOS 7.3<br class="">
</li></ul>
</li></ul>
<br class="">
<fieldset class="mimeAttachmentHeader"></fieldset> <br class="">
<pre class="" wrap="">_______________________________________________
lustre-discuss mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a>
</pre>
</blockquote>
<br style="font-family: Helvetica; font-size: 12px;
font-style: normal; font-variant-caps: normal;
font-weight: normal; letter-spacing: normal; text-align:
start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color:
rgb(255, 255, 255);" class="">
<span style="font-family: Helvetica; font-size: 12px;
font-style: normal; font-variant-caps: normal;
font-weight: normal; letter-spacing: normal; text-align:
start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color:
rgb(255, 255, 255); float: none; display: inline
!important;" class="">_______________________________________________</span><br style="font-family: Helvetica; font-size: 12px;
font-style: normal; font-variant-caps: normal;
font-weight: normal; letter-spacing: normal; text-align:
start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color:
rgb(255, 255, 255);" class="">
<span style="font-family: Helvetica; font-size: 12px;
font-style: normal; font-variant-caps: normal;
font-weight: normal; letter-spacing: normal; text-align:
start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color:
rgb(255, 255, 255); float: none; display: inline
!important;" class="">lustre-discuss
mailing list</span><br style="font-family: Helvetica; font-size: 12px;
font-style: normal; font-variant-caps: normal;
font-weight: normal; letter-spacing: normal; text-align:
start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color:
rgb(255, 255, 255);" class="">
<a moz-do-not-send="true" href="mailto:lustre-discuss@lists.lustre.org" style="font-family: Helvetica; font-size: 12px;
font-style: normal; font-variant-caps: normal;
font-weight: normal; letter-spacing: normal; orphans:
auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; background-color:
rgb(255, 255, 255);" class="">lustre-discuss@lists.lustre.org</a><br style="font-family: Helvetica; font-size: 12px;
font-style: normal; font-variant-caps: normal;
font-weight: normal; letter-spacing: normal; text-align:
start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color:
rgb(255, 255, 255);" class="">
<a moz-do-not-send="true" href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" style="font-family: Helvetica; font-size: 12px;
font-style: normal; font-variant-caps: normal;
font-weight: normal; letter-spacing: normal; orphans:
auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; background-color:
rgb(255, 255, 255);" class="">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br style="font-family: Helvetica; font-size: 12px;
font-style: normal; font-variant-caps: normal;
font-weight: normal; letter-spacing: normal; text-align:
start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color:
rgb(255, 255, 255);" class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</blockquote>
<br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>