<html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">Hi Parinay,<br>
<br>
Yes, I see ib0 in output of ifconfig -a.<br>
I also tried with options lnet networks=<b>o2ib<u>0</u></b>(ib0)
but no luck.<br>
While loading lnet I do see error in var/log/messages:<br>
<br>
kernel: LNet: HW CPU cores: 32, npartitions: 4<br>
alg: No test for crc32 (crc32-table)<br>
kernel: alg: No test for adler32 (adler32-zlib)<br>
kernel: alg: No test for crc32 (crc32-pclmul)<br>
kernel: padlock: VIA PadLock Hash Engine not detected.<br>
modprobe: FATAL: Error inserting padlock_sha
(/lib/modules/2.6.32_358/kernel/drivers/crypto/padlock-sha.ko): No
such device<br>
<br>
But as per below link this should not be a problem?<br>
<a class="moz-txt-link-freetext" href="https://jira.hpdd.intel.com/browse/LU-1599">https://jira.hpdd.intel.com/browse/LU-1599</a><br>
<br>
modprobe lnet completes successfully and I see "Write failed:
Broken pipe" after running "lctl network up" and after this
session gets logout from the server.<br>
<br>
Thanks,<br>
Aayush.<br>
<br>
On 9/30/2014 7:21 PM, Parinay Kondekar wrote:<br>
</div>
<blockquote
cite="mid:CAGt+9fu6uj099rOSHJ1b+SrbNSS=+PM0BbAdszZwzxuXQwc_VQ@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_default"
style="font-family:tahoma,sans-serif;font-size:small">- what
is the output of 'ifconfig -a' , do you see ib0 there ?
mentioning 'options lnet networks=<b>o2ib<u>0</u></b>(ib0)'<b>
</b>should be enough.<br>
</div>
<div class="gmail_default"
style="font-family:tahoma,sans-serif;font-size:small">-
anything in syslog ?<br>
<br>
</div>
<div class="gmail_default"
style="font-family:tahoma,sans-serif;font-size:small">HTH<br>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, Sep 30, 2014 at 6:03 PM, aayush
agrawal <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:aayush.agrawal@calsoftinc.com"
target="_blank">aayush.agrawal@calsoftinc.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> Hi,<br>
<div> <br>
I am trying to build lustre 2.5.0 against
MLNX_OFED_LINUX-2.2-1.0.1-rhel6.4-x86_64 on CentOS6.4
with kernel version 2.6.32-358.<br>
But I am not able to set lnet config settings properly.
I used settings suggested in lustre 2.x manual. But then
not able to get network up using lctl.<br>
<br>
Details:<br>
<br>
I have two server machines, one for mgs+mdt and second
for oss and one client machine. I want to setup
Infiniband on all these machines.<br>
I could run below steps successfully for all the three
machines:<br>
1. Run script mlnxofedinstall <br>
<font face="Courier New"># ./mlnxofedinstall -vvv
--add-kernel-support --without-32bit
--without-fw-update --hpc</font><br>
2. Restart openibd service<br>
<font face="Courier New"># /etc/init.d/openibd
restart</font><br>
3. configure ib0 interface.<br>
4. configure lustre with o2ib<br>
<font face="Courier New">#</font> <font
face="Courier New">./configure
--with-linux=Path_to_linux-2.6.32-358.18.1.el6
--with-o2ib=/usr/src/ofa_kernel/default/</font> <br>
<br>
5. make lustre rpms:<br>
<font face="Courier New"> # make rpms</font><br>
This gave me below compilation error <br>
I looked online for this error and found bug registered
on the same: <a moz-do-not-send="true"
href="https://urldefense.proofpoint.com/v2/url?u=https-3A__jira.hpdd.intel.com_browse_LU-2D4266&d=AAMCAw&c=IGDlg0lD0b-nebmJJ0Kp8A&r=c-1Cg_VH2lcYI_JXS3gypPA6xWmYsO2Md6-EoqjeIzk&m=q_uNuYFdGrDiFyB8x0KjRuPV4TbYGJf20PKQKambrfE&s=Gu0enSN8vm3fdyqEtx0cJjPMhWf9o_TCXmJhHez9HKE&e="
target="_blank">https://jira.hpdd.intel.com/browse/LU-4266</a><br>
Below patch from above link solved the problem and hence
I could build lustre rpms:<br>
<a moz-do-not-send="true"
href="https://urldefense.proofpoint.com/v2/url?u=http-3A__review.whamcloud.com_-23_c_8451_1&d=AAMCAw&c=IGDlg0lD0b-nebmJJ0Kp8A&r=c-1Cg_VH2lcYI_JXS3gypPA6xWmYsO2Md6-EoqjeIzk&m=q_uNuYFdGrDiFyB8x0KjRuPV4TbYGJf20PKQKambrfE&s=BqWJdkdWSRVMHWQkLWAhYaV0yfRwJZDUb61TfAgRss0&e="
target="_blank">http://review.whamcloud.com/#/c/8451/1</a><br>
<br>
Now first I want to do the Infiniband setup for mgs and
mdt on single machine which also has Ethernet IP. Then I
want to format and mount mgs and mdt.<br>
So I installed above created lustre rpms and then added
below line in <span
style="color:rgb(0,0,0);font-family:monospace;font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;display:inline!important;float:none">/etc/modprobe.d/lustre.conf</span><span
style="color:rgb(0,0,0);font-family:monospace;font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;display:inline!important;float:none"><br>
</span>options lnet networks=o2ib(ib0)<br>
<br>
Then I rebooted the machine to remove all lustre related
modules including lnet and then ran<font face="Courier
New"> modprobe lnet</font> command to add above
parameters and the ran <font face="Courier New">lctl
network up</font> which is giving me below error:<br>
LNET configure error 100: Network is down<br>
<br>
I looked online and found below discussion on same
error:<br>
<a moz-do-not-send="true"
href="https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.lustre.org_pipermail_lustre-2Ddiscuss_2010-2DJune_013510.html&d=AAMCAw&c=IGDlg0lD0b-nebmJJ0Kp8A&r=c-1Cg_VH2lcYI_JXS3gypPA6xWmYsO2Md6-EoqjeIzk&m=q_uNuYFdGrDiFyB8x0KjRuPV4TbYGJf20PKQKambrfE&s=aCgXfqCUyJ7IXVRJHjqpk2HCS1_dsKDuaKJrDPmWp4I&e="
target="_blank">http://lists.lustre.org/pipermail/lustre-discuss/2010-June/013510.html</a><br>
<br>
As per suggestion in above mail I tried with below line
in <span
style="color:rgb(0,0,0);font-family:monospace;font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;display:inline!important;float:none">/etc/modprobe.d/lustre.conf</span>.
In below command for IB_IP, I have given infiniband IP.<br>
options lnet <b>networks=o2ib(ib0)</b> routes="tcp0
IB_IP@o2ib"<br>
This command hangs for around 2 to 3 minutes and then
gives error: Write failed: Broken pipe. Same is the case
for "options lnet <b>networks=o2ib(ib0)</b>"<br>
But if I set: options lnet <b>networks=tcp0(eth0),o2ib(ib0)</b>
routes="tcp1 IB_IP@o2ib" then it gives LNET configure
error 100: Network is down.<br>
<br>
It seems that for network=o2ib(ibo) I am getting error
Write failed: Broken pipe.<br>
Am I missing anything while following above steps? Or
how do I resolve above error? <br>
<br>
Thanks,<br>
Aayush. <br>
</div>
</div>
<br>
<html><br>
_______________________________________________<br>
HPDD-discuss mailing list<br>
<a moz-do-not-send="true"
href="mailto:HPDD-discuss@lists.01.org">HPDD-discuss@lists.01.org</a><br>
<a moz-do-not-send="true"
href="https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.01.org_mailman_listinfo_hpdd-2Ddiscuss&d=AAICAg&c=IGDlg0lD0b-nebmJJ0Kp8A&r=c-1Cg_VH2lcYI_JXS3gypPA6xWmYsO2Md6-EoqjeIzk&m=q_uNuYFdGrDiFyB8x0KjRuPV4TbYGJf20PKQKambrfE&s=0hW3r7x0NhgbZ7zgaZKr9K_fk7_E8bs0f-GAlH89rgM&e="
target="_blank">https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.01.org_mailman_listinfo_hpdd-2Ddiscuss&d=AAICAg&c=IGDlg0lD0b-nebmJJ0Kp8A&r=c-1Cg_VH2lcYI_JXS3gypPA6xWmYsO2Md6-EoqjeIzk&m=q_uNuYFdGrDiFyB8x0KjRuPV4TbYGJf20PKQKambrfE&s=0hW3r7x0NhgbZ7zgaZKr9K_fk7_E8bs0f-GAlH89rgM&e=</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
</body>
</html>