<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
I’m currently investigating a problem with MOFED 4.x which seems very similar to what you seeing.  I have no solution yet.
<div class=""><br class="">
</div>
<div class="">Doug</div>
<div class=""><br class="">
<div>
<blockquote type="cite" class="">
<div class="">On May 7, 2017, at 7:09 AM, HM Li <<a href="mailto:lihm0@163.com" class="">lihm0@163.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div text="#000000" bgcolor="#FFFFFF" class="">
<p class="">Thank you very much.</p>
<p class="">The MLNX used on Multi-Host nodes is MLNX_OFED_LINUX-4.0-1.0.1.0-rhel7.3-x86_64.</p>
<p class="">This driver and lustre(git, 2.9.55_45) can work well on other normal FDR nodes.<br class="">
</p>
<br class="">
<div class="moz-cite-prefix">On 2017年05月06日 01:14, Oucharek, Doug S wrote:<br class="">
</div>
<blockquote cite="mid:F2457005-6B4B-458F-98C4-DD0F250F5EC5@intel.com" type="cite" class="">
The tag you checked out is missing this fix: <a moz-do-not-send="true" href="https://review.whamcloud.com/#/c/24306/" class="">https://review.whamcloud.com/#/c/24306/</a>.  Try applying that.
<div class=""><br class="">
</div>
<div class="">Doug</div>
<div class=""><br class="">
<div class="">
<blockquote type="cite" class="">
<div class="">On May 5, 2017, at 9:51 AM, HM Li <<a moz-do-not-send="true" href="mailto:lihm0@163.com" class="">lihm0@163.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<p style="font-family: Helvetica; font-size: 12px;
                font-style: normal; font-variant-caps: normal;
                font-weight: normal; letter-spacing: normal; text-align:
                start; text-indent: 0px; text-transform: none;
                white-space: normal; word-spacing: 0px;
                -webkit-text-stroke-width: 0px; background-color:
                rgb(255, 255, 255);" class="">
Conformed.</p>
<p style="font-family: Helvetica; font-size: 12px;
                font-style: normal; font-variant-caps: normal;
                font-weight: normal; letter-spacing: normal; text-align:
                start; text-indent: 0px; text-transform: none;
                white-space: normal; word-spacing: 0px;
                -webkit-text-stroke-width: 0px; background-color:
                rgb(255, 255, 255);" class="">
This is a bug of git(2.9.55_45), it works well when using MLNX_OFED_LINUX-3.4-2.1.8.0-rhel7.3-x86_64 and<a moz-do-not-send="true" class="moz-txt-link-freetext" href="https://downloads.hpdd.intel.com/public/lustre/latest-feature-release/el7.3.1611/server/SRPMS/lustre-2.9.0-1.src.rpm">https://downloads.hpdd.intel.com/public/lustre/latest-feature-release/el7.3.1611/server/SRPMS/lustre-2.9.0-1.src.rpm</a>.</p>
<p style="font-family: Helvetica; font-size: 12px;
                font-style: normal; font-variant-caps: normal;
                font-weight: normal; letter-spacing: normal; text-align:
                start; text-indent: 0px; text-transform: none;
                white-space: normal; word-spacing: 0px;
                -webkit-text-stroke-width: 0px; background-color:
                rgb(255, 255, 255);" class="">
node453@tc4600: ~# export LC_ALL=C<br class="">
node453@tc4600: ~# lctl lustre_build_version<br class="">
Lustre version: 2.9.0<br class="">
node453@tc4600: ~# df      <span class="Apple-converted-space"> </span><br class="">
Filesystem                1K-blocks        Used   Available Use% Mounted on<br class="">
10.10.100.6@o2ib:/lxfs   7341068688    10152136  6960149976   1% /home<br class="">
10.10.100.1@o2ib:/sgfs 108704716104 24800951320 78393702908  25% /mnt<br class="">
node453@tc4600: ~# ibv_devinfo<span class="Apple-converted-space"> </span><br class="">
hca_id:    mlx5_0<br class="">
    transport:            InfiniBand (0)<br class="">
    fw_ver:                12.17.1010<br class="">
    node_guid:            46e3:e861:1f19:4438<br class="">
    sys_image_guid:            46e3:e861:1f19:4438<br class="">
    vendor_id:            0x02c9<br class="">
    vendor_part_id:            4115<br class="">
    hw_ver:                0x0<br class="">
    board_id:            SGN1130110032<br class="">
    phys_port_cnt:            1<br class="">
    Device ports:<br class="">
        port:    1<br class="">
            state:            PORT_ACTIVE (4)<br class="">
            max_mtu:        4096 (5)<br class="">
            active_mtu:        4096 (5)<br class="">
            sm_lid:            360<br class="">
            port_lid:        362<br class="">
            port_lmc:        0x00<br class="">
            link_layer:        InfiniBand<br class="">
<br class="">
</p>
<br style="font-family: Helvetica; font-size: 12px;
                font-style: normal; font-variant-caps: normal;
                font-weight: normal; letter-spacing: normal; text-align:
                start; text-indent: 0px; text-transform: none;
                white-space: normal; word-spacing: 0px;
                -webkit-text-stroke-width: 0px; background-color:
                rgb(255, 255, 255);" class="">
<div class="moz-cite-prefix" style="font-family:
                Helvetica; font-size: 12px; font-style: normal;
                font-variant-caps: normal; font-weight: normal;
                letter-spacing: normal; text-align: start; text-indent:
                0px; text-transform: none; white-space: normal;
                word-spacing: 0px; -webkit-text-stroke-width: 0px;
                background-color: rgb(255, 255, 255);">
On 2017年05月03日 17:02, HM Li wrote:<br class="">
</div>
<blockquote cite="mid:5490037c-1644-d839-868a-198bf20c8957@163.com" type="cite" style="font-family: Helvetica; font-size:
                12px; font-style: normal; font-variant-caps: normal;
                font-weight: normal; letter-spacing: normal; orphans:
                auto; text-align: start; text-indent: 0px;
                text-transform: none; white-space: normal; widows: auto;
                word-spacing: 0px; -webkit-text-size-adjust: auto;
                -webkit-text-stroke-width: 0px; background-color:
                rgb(255, 255, 255);" class="">
Dear,<br class="">
I have setup the Lustre-git(2.9.55_45) on CentOS 7.3, but the client(on the same multi-host IB) can't mount lustre. Can you help me? Thank you very much.<br class="">
<ul class="">
<li class="">The<span class="Apple-converted-space"> </span><font class="" color="#ff0000">server</font>:
<ul class="">
<li class="">mkfs.lustre --fsname=lxfs --mgs --mdt --index=0  --reformat /dev/sda5
</li><li class="">mkfs.lustre --fsname=lxfs --mgsnode=10.10.146.1@o2ib1 --servicenode=10.10.146.1@o2ib1 --ost --reformat --index=1 /dev/sda6
</li><li class="">mount -t lustre /dev/sda5 /mnt/mdt<br class="">
</li><li class="">mount -t lustre /dev/sda6 /mnt/ost </li><li class="">mount -t lustre -v  10.10.146.1@o2ib1:/lxfs /home is OK. </li><li class="">lctl list_nids<br class="">
10.10.146.1@o2ib1<br class="">
</li><li class="">lctl ping 10.10.146.2@o2ib1<br class="">
12345-0@lo<br class="">
12345-10.10.146.2@o2ib1<br class="">
<br class="">
</li></ul>
</li><li class="">The<span class="Apple-converted-space"> </span><font class="" color="#3333ff">client</font>:
<ul class="">
<li class="">lctl list_nids<br class="">
10.10.146.2@o2ib1 </li></ul>
<ul class="">
<li class="">lctl ping 10.10.146.1@o2ib1<br class="">
12345-0@lo<br class="">
12345-10.10.146.1@o2ib1 </li></ul>
<ul class="">
<li class="">mount -t lustre -v 10.10.146.1@o2ib1:/lxfs /home/<br class="">
arg[0] = /sbin/mount.lustre<br class="">
arg[1] = -v<br class="">
arg[2] = -o<br class="">
arg[3] = rw<br class="">
arg[4] = 10.10.146.1@o2ib1:/lxfs<br class="">
arg[5] = /home<br class="">
source = 10.10.146.1@o2ib1:/lxfs (10.10.146.1@o2ib1:/lxfs), target = /home<br class="">
options = rw<br class="">
mounting device 10.10.146.1@o2ib1:/lxfs at /home, flags=0x1000000 options=device=10.10.146.1@o2ib1:/lxfs<br class="">
mount.lustre: mount 10.10.146.1@o2ib1:/lxfs at /home failed: Input/output error retries left: 0<br class="">
mount.lustre: mount 10.10.146.1@o2ib1:/lxfs at /home failed: Input/output error<br class="">
Is the MGS running?<br class="">
<br class="">
</li></ul>
</li><li class="">and now on<span class="Apple-converted-space"> </span><font class="" color="#ff0000">server</font><span class="Apple-converted-space"> </span>dmesg show:<br class="">
[82709.336007] Lustre: MGS: Connection restored to 792b2b21-2e57-de7d-3d8f-5e80eb6d7bf2 (at 10.10.146.2@o2ib1)<br class="">
[82709.339324] mlx5_0:dump_cqe:275:(pid 22740): dump error cqe<br class="">
[82709.339508] 00000000 00000000 00000000 00000000<br class="">
[82709.339677] 00000000 00000000 00000000 00000000<br class="">
[82709.339841] 00000000 00000000 00000000 00000000<br class="">
[82709.340006] 00000000 9d005304 08000074 01f1c5d2<br class="">
[82716.343333] Lustre: MGS: Received new LWP connection from 10.10.146.2@o2ib1, removing former export from same NID<br class="">
[82716.343712] Lustre: MGS: Connection restored to 792b2b21-2e57-de7d-3d8f-5e80eb6d7bf2 (at 10.10.146.2@o2ib1)<br class="">
<br class="">
</li><li class=""><font class="" color="#ff0000">IB</font><span class="Apple-converted-space"> </span>information:
<ul class="">
<li class="">ibstat<br class="">
CA 'mlx5_0'<br class="">
    CA type: MT4115<br class="">
    Number of ports: 1<br class="">
    Firmware version: 12.17.1010<br class="">
    Hardware version: 0<br class="">
    Node GUID: 0x46e3e8611f19443a<br class="">
    System image GUID: 0x46e3e8611f194438<br class="">
    Port 1:<br class="">
        State: Active<br class="">
        Physical state: LinkUp<br class="">
        Rate: 100<br class="">
        Base lid: 360<br class="">
        LMC: 2<br class="">
        SM lid: 360<br class="">
        Capability mask: 0x2651e84a<br class="">
        Port GUID: 0x46e3e8611f19443a<br class="">
        Link layer: InfiniBand<br class="">
</li><li class="">ibv_devinfo<span class="Apple-converted-space"> </span><br class="">
hca_id:    mlx5_0<br class="">
    transport:            InfiniBand (0)<br class="">
    fw_ver:                12.17.1010<br class="">
    node_guid:            46e3:e861:1f19:443a<br class="">
    sys_image_guid:            46e3:e861:1f19:4438<br class="">
    vendor_id:            0x02c9<br class="">
    vendor_part_id:            4115<br class="">
    hw_ver:                0x0<br class="">
    board_id:            SGN1130110032<br class="">
    phys_port_cnt:            1<br class="">
    Device ports:<br class="">
        port:    1<br class="">
            state:            PORT_ACTIVE (4)<br class="">
            max_mtu:        4096 (5)<br class="">
            active_mtu:        4096 (5)<br class="">
            sm_lid:            360<br class="">
            port_lid:        360<br class="">
            port_lmc:        0x02<br class="">
            link_layer:        InfiniBand </li><li class="">ibstatus<span class="Apple-converted-space"> </span><br class="">
Infiniband device 'mlx5_0' port 1 status:<br class="">
    default gid:     fe80:0000:0000:0000:46e3:e861:1f19:443a<br class="">
    base lid:     0x168<br class="">
    sm lid:         0x168<br class="">
    state:         4: ACTIVE<br class="">
    phys state:     5: LinkUp<br class="">
    rate:         100 Gb/sec (4X EDR)<br class="">
    link_layer:     InfiniBand </li><li class=""><span class="" lang="EN-US">opensm --create-config /etc/opensm/opensm.conf</span>
</li><li class="">/etc/opensm/opensm.conf has been modified:<br class="">
virt enabled 2<br class="">
qos TRUE<br class="">
lmc 2<br class="">
</li></ul>
</li></ul>
<ul class="">
<li class="">Other information:
<ul class="">
<li class="">selinux disabled </li><li class="">iptables cleaned </li><li class="">uname -r: 3.10.0-514.16.1.el7_lustre.x86_64 </li><li class="">OS: CentOS 7.3<br class="">
</li></ul>
</li></ul>
<br class="">
<fieldset class="mimeAttachmentHeader"></fieldset> <br class="">
<pre class="" wrap="">_______________________________________________
lustre-discuss mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a>
</pre>
</blockquote>
<br style="font-family: Helvetica; font-size: 12px;
                font-style: normal; font-variant-caps: normal;
                font-weight: normal; letter-spacing: normal; text-align:
                start; text-indent: 0px; text-transform: none;
                white-space: normal; word-spacing: 0px;
                -webkit-text-stroke-width: 0px; background-color:
                rgb(255, 255, 255);" class="">
<span style="font-family: Helvetica; font-size: 12px;
                font-style: normal; font-variant-caps: normal;
                font-weight: normal; letter-spacing: normal; text-align:
                start; text-indent: 0px; text-transform: none;
                white-space: normal; word-spacing: 0px;
                -webkit-text-stroke-width: 0px; background-color:
                rgb(255, 255, 255); float: none; display: inline
                !important;" class="">_______________________________________________</span><br style="font-family: Helvetica; font-size: 12px;
                font-style: normal; font-variant-caps: normal;
                font-weight: normal; letter-spacing: normal; text-align:
                start; text-indent: 0px; text-transform: none;
                white-space: normal; word-spacing: 0px;
                -webkit-text-stroke-width: 0px; background-color:
                rgb(255, 255, 255);" class="">
<span style="font-family: Helvetica; font-size: 12px;
                font-style: normal; font-variant-caps: normal;
                font-weight: normal; letter-spacing: normal; text-align:
                start; text-indent: 0px; text-transform: none;
                white-space: normal; word-spacing: 0px;
                -webkit-text-stroke-width: 0px; background-color:
                rgb(255, 255, 255); float: none; display: inline
                !important;" class="">lustre-discuss
 mailing list</span><br style="font-family: Helvetica; font-size: 12px;
                font-style: normal; font-variant-caps: normal;
                font-weight: normal; letter-spacing: normal; text-align:
                start; text-indent: 0px; text-transform: none;
                white-space: normal; word-spacing: 0px;
                -webkit-text-stroke-width: 0px; background-color:
                rgb(255, 255, 255);" class="">
<a moz-do-not-send="true" href="mailto:lustre-discuss@lists.lustre.org" style="font-family: Helvetica; font-size: 12px;
                font-style: normal; font-variant-caps: normal;
                font-weight: normal; letter-spacing: normal; orphans:
                auto; text-align: start; text-indent: 0px;
                text-transform: none; white-space: normal; widows: auto;
                word-spacing: 0px; -webkit-text-size-adjust: auto;
                -webkit-text-stroke-width: 0px; background-color:
                rgb(255, 255, 255);" class="">lustre-discuss@lists.lustre.org</a><br style="font-family: Helvetica; font-size: 12px;
                font-style: normal; font-variant-caps: normal;
                font-weight: normal; letter-spacing: normal; text-align:
                start; text-indent: 0px; text-transform: none;
                white-space: normal; word-spacing: 0px;
                -webkit-text-stroke-width: 0px; background-color:
                rgb(255, 255, 255);" class="">
<a moz-do-not-send="true" href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" style="font-family: Helvetica; font-size: 12px;
                font-style: normal; font-variant-caps: normal;
                font-weight: normal; letter-spacing: normal; orphans:
                auto; text-align: start; text-indent: 0px;
                text-transform: none; white-space: normal; widows: auto;
                word-spacing: 0px; -webkit-text-size-adjust: auto;
                -webkit-text-stroke-width: 0px; background-color:
                rgb(255, 255, 255);" class="">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br style="font-family: Helvetica; font-size: 12px;
                font-style: normal; font-variant-caps: normal;
                font-weight: normal; letter-spacing: normal; text-align:
                start; text-indent: 0px; text-transform: none;
                white-space: normal; word-spacing: 0px;
                -webkit-text-stroke-width: 0px; background-color:
                rgb(255, 255, 255);" class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</blockquote>
<br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>