<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Thank you very much.</p>
<p>The MLNX used on Multi-Host nodes is
MLNX_OFED_LINUX-4.0-1.0.1.0-rhel7.3-x86_64.</p>
<p>This driver and lustre(git, 2.9.55_45) can work well on other
normal FDR nodes.<br>
</p>
<br>
<div class="moz-cite-prefix">On 2017年05月06日 01:14, Oucharek, Doug S
wrote:<br>
</div>
<blockquote
cite="mid:F2457005-6B4B-458F-98C4-DD0F250F5EC5@intel.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
The tag you checked out is missing this fix: <a
moz-do-not-send="true"
href="https://review.whamcloud.com/#/c/24306/" class="">https://review.whamcloud.com/#/c/24306/</a>.
Try applying that.
<div class=""><br class="">
</div>
<div class="">Doug</div>
<div class=""><br class="">
<div>
<blockquote type="cite" class="">
<div class="">On May 5, 2017, at 9:51 AM, HM Li <<a
moz-do-not-send="true" href="mailto:lihm0@163.com"
class="">lihm0@163.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<p style="font-family: Helvetica; font-size: 12px;
font-style: normal; font-variant-caps: normal;
font-weight: normal; letter-spacing: normal; text-align:
start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color:
rgb(255, 255, 255);" class="">
Conformed.</p>
<p style="font-family: Helvetica; font-size: 12px;
font-style: normal; font-variant-caps: normal;
font-weight: normal; letter-spacing: normal; text-align:
start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color:
rgb(255, 255, 255);" class="">
This is a bug of git(2.9.55_45), it works well when
using MLNX_OFED_LINUX-3.4-2.1.8.0-rhel7.3-x86_64 and<a
moz-do-not-send="true" class="moz-txt-link-freetext"
href="https://downloads.hpdd.intel.com/public/lustre/latest-feature-release/el7.3.1611/server/SRPMS/lustre-2.9.0-1.src.rpm">https://downloads.hpdd.intel.com/public/lustre/latest-feature-release/el7.3.1611/server/SRPMS/lustre-2.9.0-1.src.rpm</a>.</p>
<p style="font-family: Helvetica; font-size: 12px;
font-style: normal; font-variant-caps: normal;
font-weight: normal; letter-spacing: normal; text-align:
start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color:
rgb(255, 255, 255);" class="">
node453@tc4600: ~# export LC_ALL=C<br class="">
node453@tc4600: ~# lctl lustre_build_version<br class="">
Lustre version: 2.9.0<br class="">
node453@tc4600: ~# df <span
class="Apple-converted-space"> </span><br class="">
Filesystem 1K-blocks Used
Available Use% Mounted on<br class="">
10.10.100.6@o2ib:/lxfs 7341068688 10152136
6960149976 1% /home<br class="">
10.10.100.1@o2ib:/sgfs 108704716104 24800951320
78393702908 25% /mnt<br class="">
node453@tc4600: ~# ibv_devinfo<span
class="Apple-converted-space"> </span><br class="">
hca_id: mlx5_0<br class="">
transport: InfiniBand (0)<br class="">
fw_ver: 12.17.1010<br class="">
node_guid: 46e3:e861:1f19:4438<br
class="">
sys_image_guid: 46e3:e861:1f19:4438<br
class="">
vendor_id: 0x02c9<br class="">
vendor_part_id: 4115<br class="">
hw_ver: 0x0<br class="">
board_id: SGN1130110032<br class="">
phys_port_cnt: 1<br class="">
Device ports:<br class="">
port: 1<br class="">
state: PORT_ACTIVE (4)<br
class="">
max_mtu: 4096 (5)<br class="">
active_mtu: 4096 (5)<br class="">
sm_lid: 360<br class="">
port_lid: 362<br class="">
port_lmc: 0x00<br class="">
link_layer: InfiniBand<br class="">
<br class="">
</p>
<br style="font-family: Helvetica; font-size: 12px;
font-style: normal; font-variant-caps: normal;
font-weight: normal; letter-spacing: normal; text-align:
start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color:
rgb(255, 255, 255);" class="">
<div class="moz-cite-prefix" style="font-family:
Helvetica; font-size: 12px; font-style: normal;
font-variant-caps: normal; font-weight: normal;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
background-color: rgb(255, 255, 255);">
On 2017年05月03日 17:02, HM Li wrote:<br class="">
</div>
<blockquote
cite="mid:5490037c-1644-d839-868a-198bf20c8957@163.com"
type="cite" style="font-family: Helvetica; font-size:
12px; font-style: normal; font-variant-caps: normal;
font-weight: normal; letter-spacing: normal; orphans:
auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; background-color:
rgb(255, 255, 255);" class="">
Dear,<br class="">
I have setup the Lustre-git(2.9.55_45) on CentOS 7.3,
but the client(on the same multi-host IB) can't mount
lustre. Can you help me? Thank you very much.<br
class="">
<ul class="">
<li class="">The<span class="Apple-converted-space"> </span><font
class="" color="#ff0000">server</font>:
<ul class="">
<li class="">mkfs.lustre --fsname=lxfs --mgs --mdt
--index=0 --reformat /dev/sda5</li>
<li class="">mkfs.lustre --fsname=lxfs
--mgsnode=10.10.146.1@o2ib1
--servicenode=10.10.146.1@o2ib1 --ost --reformat
--index=1 /dev/sda6</li>
<li class="">mount -t lustre /dev/sda5 /mnt/mdt<br
class="">
</li>
<li class="">mount -t lustre /dev/sda6 /mnt/ost</li>
<li class="">mount -t lustre -v
10.10.146.1@o2ib1:/lxfs /home is OK.</li>
<li class="">lctl list_nids<br class="">
10.10.146.1@o2ib1<br class="">
</li>
<li class="">lctl ping 10.10.146.2@o2ib1<br
class="">
12345-0@lo<br class="">
12345-10.10.146.2@o2ib1<br class="">
<br class="">
</li>
</ul>
</li>
<li class="">The<span class="Apple-converted-space"> </span><font
class="" color="#3333ff">client</font>:
<ul class="">
<li class="">lctl list_nids<br class="">
10.10.146.2@o2ib1</li>
</ul>
<ul class="">
<li class="">lctl ping 10.10.146.1@o2ib1<br
class="">
12345-0@lo<br class="">
12345-10.10.146.1@o2ib1</li>
</ul>
<ul class="">
<li class="">mount -t lustre -v
10.10.146.1@o2ib1:/lxfs /home/<br class="">
arg[0] = /sbin/mount.lustre<br class="">
arg[1] = -v<br class="">
arg[2] = -o<br class="">
arg[3] = rw<br class="">
arg[4] = 10.10.146.1@o2ib1:/lxfs<br class="">
arg[5] = /home<br class="">
source = 10.10.146.1@o2ib1:/lxfs
(10.10.146.1@o2ib1:/lxfs), target = /home<br
class="">
options = rw<br class="">
mounting device 10.10.146.1@o2ib1:/lxfs at
/home, flags=0x1000000
options=device=10.10.146.1@o2ib1:/lxfs<br
class="">
mount.lustre: mount 10.10.146.1@o2ib1:/lxfs at
/home failed: Input/output error retries left: 0<br
class="">
mount.lustre: mount 10.10.146.1@o2ib1:/lxfs at
/home failed: Input/output error<br class="">
Is the MGS running?<br class="">
<br class="">
</li>
</ul>
</li>
<li class="">and now on<span
class="Apple-converted-space"> </span><font
class="" color="#ff0000">server</font><span
class="Apple-converted-space"> </span>dmesg show:<br
class="">
[82709.336007] Lustre: MGS: Connection restored to
792b2b21-2e57-de7d-3d8f-5e80eb6d7bf2 (at
10.10.146.2@o2ib1)<br class="">
[82709.339324] mlx5_0:dump_cqe:275:(pid 22740): dump
error cqe<br class="">
[82709.339508] 00000000 00000000 00000000 00000000<br
class="">
[82709.339677] 00000000 00000000 00000000 00000000<br
class="">
[82709.339841] 00000000 00000000 00000000 00000000<br
class="">
[82709.340006] 00000000 9d005304 08000074 01f1c5d2<br
class="">
[82716.343333] Lustre: MGS: Received new LWP
connection from 10.10.146.2@o2ib1, removing former
export from same NID<br class="">
[82716.343712] Lustre: MGS: Connection restored to
792b2b21-2e57-de7d-3d8f-5e80eb6d7bf2 (at
10.10.146.2@o2ib1)<br class="">
<br class="">
</li>
<li class=""><font class="" color="#ff0000">IB</font><span
class="Apple-converted-space"> </span>information:
<ul class="">
<li class="">ibstat<br class="">
CA 'mlx5_0'<br class="">
CA type: MT4115<br class="">
Number of ports: 1<br class="">
Firmware version: 12.17.1010<br class="">
Hardware version: 0<br class="">
Node GUID: 0x46e3e8611f19443a<br class="">
System image GUID: 0x46e3e8611f194438<br
class="">
Port 1:<br class="">
State: Active<br class="">
Physical state: LinkUp<br class="">
Rate: 100<br class="">
Base lid: 360<br class="">
LMC: 2<br class="">
SM lid: 360<br class="">
Capability mask: 0x2651e84a<br class="">
Port GUID: 0x46e3e8611f19443a<br
class="">
Link layer: InfiniBand<br class="">
</li>
<li class="">ibv_devinfo<span
class="Apple-converted-space"> </span><br
class="">
hca_id: mlx5_0<br class="">
transport: InfiniBand (0)<br
class="">
fw_ver: 12.17.1010<br
class="">
node_guid: 46e3:e861:1f19:443a<br
class="">
sys_image_guid:
46e3:e861:1f19:4438<br class="">
vendor_id: 0x02c9<br class="">
vendor_part_id: 4115<br class="">
hw_ver: 0x0<br class="">
board_id: SGN1130110032<br
class="">
phys_port_cnt: 1<br class="">
Device ports:<br class="">
port: 1<br class="">
state: PORT_ACTIVE (4)<br
class="">
max_mtu: 4096 (5)<br class="">
active_mtu: 4096 (5)<br
class="">
sm_lid: 360<br class="">
port_lid: 360<br class="">
port_lmc: 0x02<br class="">
link_layer: InfiniBand</li>
<li class="">ibstatus<span
class="Apple-converted-space"> </span><br
class="">
Infiniband device 'mlx5_0' port 1 status:<br
class="">
default gid:
fe80:0000:0000:0000:46e3:e861:1f19:443a<br
class="">
base lid: 0x168<br class="">
sm lid: 0x168<br class="">
state: 4: ACTIVE<br class="">
phys state: 5: LinkUp<br class="">
rate: 100 Gb/sec (4X EDR)<br
class="">
link_layer: InfiniBand</li>
<li class=""><span class="" lang="EN-US">opensm
--create-config /etc/opensm/opensm.conf</span></li>
<li class="">/etc/opensm/opensm.conf has been
modified:<br class="">
virt enabled 2<br class="">
qos TRUE<br class="">
lmc 2<br class="">
</li>
</ul>
</li>
</ul>
<ul class="">
<li class="">Other information:
<ul class="">
<li class="">selinux disabled</li>
<li class="">iptables cleaned</li>
<li class="">uname -r:
3.10.0-514.16.1.el7_lustre.x86_64</li>
<li class="">OS: CentOS 7.3<br class="">
</li>
</ul>
</li>
</ul>
<br class="">
<fieldset class="mimeAttachmentHeader"></fieldset>
<br class="">
<pre class="" wrap="">_______________________________________________
lustre-discuss mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a>
</pre>
</blockquote>
<br style="font-family: Helvetica; font-size: 12px;
font-style: normal; font-variant-caps: normal;
font-weight: normal; letter-spacing: normal; text-align:
start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color:
rgb(255, 255, 255);" class="">
<span style="font-family: Helvetica; font-size: 12px;
font-style: normal; font-variant-caps: normal;
font-weight: normal; letter-spacing: normal; text-align:
start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color:
rgb(255, 255, 255); float: none; display: inline
!important;" class="">_______________________________________________</span><br
style="font-family: Helvetica; font-size: 12px;
font-style: normal; font-variant-caps: normal;
font-weight: normal; letter-spacing: normal; text-align:
start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color:
rgb(255, 255, 255);" class="">
<span style="font-family: Helvetica; font-size: 12px;
font-style: normal; font-variant-caps: normal;
font-weight: normal; letter-spacing: normal; text-align:
start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color:
rgb(255, 255, 255); float: none; display: inline
!important;" class="">lustre-discuss mailing list</span><br
style="font-family: Helvetica; font-size: 12px;
font-style: normal; font-variant-caps: normal;
font-weight: normal; letter-spacing: normal; text-align:
start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color:
rgb(255, 255, 255);" class="">
<a moz-do-not-send="true"
href="mailto:lustre-discuss@lists.lustre.org"
style="font-family: Helvetica; font-size: 12px;
font-style: normal; font-variant-caps: normal;
font-weight: normal; letter-spacing: normal; orphans:
auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; background-color:
rgb(255, 255, 255);" class="">lustre-discuss@lists.lustre.org</a><br
style="font-family: Helvetica; font-size: 12px;
font-style: normal; font-variant-caps: normal;
font-weight: normal; letter-spacing: normal; text-align:
start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color:
rgb(255, 255, 255);" class="">
<a moz-do-not-send="true"
href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org"
style="font-family: Helvetica; font-size: 12px;
font-style: normal; font-variant-caps: normal;
font-weight: normal; letter-spacing: normal; orphans:
auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; background-color:
rgb(255, 255, 255);" class="">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br
style="font-family: Helvetica; font-size: 12px;
font-style: normal; font-variant-caps: normal;
font-weight: normal; letter-spacing: normal; text-align:
start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color:
rgb(255, 255, 255);" class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</blockquote>
<br>
</body>
</html>