<html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">Hi Mike,<br>
<br>
While installing OFED I have used below command:<br>
<font face="Courier New"># ./mlnxofedinstall -vvv
--add-kernel-support --without-32bit --without-fw-update --hpc<br>
<br>
</font>I have used option <font face="Courier New">
--add-kernel-support, </font>Which add kernel support (Run
mlnx_add_kernel_support.sh). This is what you meant to say, right?<br>
<br>
Thanks,<br>
Aayush.<br>
<br>
On 9/30/2014 11:04 PM, Mike Ware wrote:<br>
</div>
<blockquote
cite="mid:CANgwNH9siTWJ1yE38=LLpQQ8+Nc08Yu8spuUdRmX3CHE9D1uGg@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>I knew I had it somewhere</div>
<div><br>
</div>
<a moz-do-not-send="true"
href="http://lists.lustre.org/pipermail/lustre-discuss/2012-November/016988.html">http://lists.lustre.org/pipermail/lustre-discuss/2012-November/016988.html</a> <br>
<div><br>
</div>
<div>Mike</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, Sep 30, 2014 at 10:32 AM, Mike
Ware <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:charnobyl3000@gmail.com" target="_blank">charnobyl3000@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">I had a similar issue using the Mellanox
packages. If i remember correctly I had to recompile the
drivers against the Lustre kernel for the install. I
believe Mellanox had an article on this but I don't have
the link.
<div><br>
</div>
<div>Mike</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">
<div>
<div class="h5">On Tue, Sep 30, 2014 at 8:07 AM,
Parinay Kondekar <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:parinay.kondekar@seagate.com"
target="_blank">parinay.kondekar@seagate.com</a>></span>
wrote:<br>
</div>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div class="h5">
<div dir="ltr">
<div class="gmail_default"
style="font-family:tahoma,sans-serif;font-size:small">IMO
you should try out strace to see if anything
is noticed. <br>
"Write failed: Broken pipe" is quite common
message and difficult to conclude anything
with.<br>
<br>
</div>
<div class="gmail_default"
style="font-family:tahoma,sans-serif;font-size:small">Regards<span><font
color="#888888"><br>
</font></span></div>
<span><font color="#888888">
<div class="gmail_default"
style="font-family:tahoma,sans-serif;font-size:small">parinay<br>
</div>
</font></span></div>
<div>
<div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, Sep 30,
2014 at 8:16 PM, aayush agrawal <span
dir="ltr"><<a moz-do-not-send="true"
href="mailto:aayush.agrawal@calsoftinc.com" target="_blank">aayush.agrawal@calsoftinc.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>Hi Parinay,<br>
<br>
Yes, I see ib0 in output of ifconfig
-a.<br>
I also tried with options lnet
networks=<b>o2ib<u>0</u></b>(ib0)
but no luck.<br>
While loading lnet I do see error in
var/log/messages:<br>
<br>
kernel: LNet: HW CPU cores: 32,
npartitions: 4<br>
alg: No test for crc32 (crc32-table)<br>
kernel: alg: No test for adler32
(adler32-zlib)<br>
kernel: alg: No test for crc32
(crc32-pclmul)<br>
kernel: padlock: VIA PadLock Hash
Engine not detected.<br>
modprobe: FATAL: Error inserting
padlock_sha
(/lib/modules/2.6.32_358/kernel/drivers/crypto/padlock-sha.ko):
No such device<br>
<br>
But as per below link this should
not be a problem?<br>
<a moz-do-not-send="true"
href="https://jira.hpdd.intel.com/browse/LU-1599"
target="_blank">https://jira.hpdd.intel.com/browse/LU-1599</a><br>
<br>
modprobe lnet completes successfully
and I see "Write failed: Broken
pipe" after running "lctl network
up" and after this session gets
logout from the server.<br>
<br>
Thanks,<br>
Aayush.
<div>
<div><br>
<br>
On 9/30/2014 7:21 PM, Parinay
Kondekar wrote:<br>
</div>
</div>
</div>
<div>
<div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_default"
style="font-family:tahoma,sans-serif;font-size:small">-
what is the output of
'ifconfig -a' , do you see
ib0 there ? mentioning
'options lnet networks=<b>o2ib<u>0</u></b>(ib0)'<b>
</b>should be enough.<br>
</div>
<div class="gmail_default"
style="font-family:tahoma,sans-serif;font-size:small">-
anything in syslog ?<br>
<br>
</div>
<div class="gmail_default"
style="font-family:tahoma,sans-serif;font-size:small">HTH<br>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On
Tue, Sep 30, 2014 at 6:03
PM, aayush agrawal <span
dir="ltr"><<a
moz-do-not-send="true"
href="mailto:aayush.agrawal@calsoftinc.com"
target="_blank">aayush.agrawal@calsoftinc.com</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div bgcolor="#FFFFFF"
text="#000000"> Hi,<br>
<div> <br>
I am trying to build
lustre 2.5.0 against
MLNX_OFED_LINUX-2.2-1.0.1-rhel6.4-x86_64
on CentOS6.4 with
kernel version
2.6.32-358.<br>
But I am not able to
set lnet config
settings properly. I
used settings
suggested in lustre
2.x manual. But then
not able to get
network up using lctl.<br>
<br>
Details:<br>
<br>
I have two server
machines, one for
mgs+mdt and second for
oss and one client
machine. I want to
setup Infiniband on
all these machines.<br>
I could run below
steps successfully for
all the three
machines:<br>
1. Run script
mlnxofedinstall <br>
<font
face="Courier New">#
./mlnxofedinstall
-vvv
--add-kernel-support
--without-32bit
--without-fw-update
--hpc</font><br>
2. Restart openibd
service<br>
<font
face="Courier New">#
/etc/init.d/openibd
restart</font><br>
3. configure ib0
interface.<br>
4. configure lustre
with o2ib<br>
<font
face="Courier New">#</font>
<font face="Courier
New">./configure
--with-linux=Path_to_linux-2.6.32-358.18.1.el6
--with-o2ib=/usr/src/ofa_kernel/default/</font> <br>
<br>
5. make lustre rpms:<br>
<font face="Courier
New"> # make rpms</font><br>
This gave me below
compilation error <br>
I looked online for
this error and found
bug registered on the
same: <a
moz-do-not-send="true"
href="https://urldefense.proofpoint.com/v2/url?u=https-3A__jira.hpdd.intel.com_browse_LU-2D4266&d=AAMCAw&c=IGDlg0lD0b-nebmJJ0Kp8A&r=c-1Cg_VH2lcYI_JXS3gypPA6xWmYsO2Md6-EoqjeIzk&m=q_uNuYFdGrDiFyB8x0KjRuPV4TbYGJf20PKQKambrfE&s=Gu0enSN8vm3fdyqEtx0cJjPMhWf9o_TCXmJhHez9HKE&e="
target="_blank">https://jira.hpdd.intel.com/browse/LU-4266</a><br>
Below patch from above
link solved the
problem and hence I
could build lustre
rpms:<br>
<a
moz-do-not-send="true"
href="https://urldefense.proofpoint.com/v2/url?u=http-3A__review.whamcloud.com_-23_c_8451_1&d=AAMCAw&c=IGDlg0lD0b-nebmJJ0Kp8A&r=c-1Cg_VH2lcYI_JXS3gypPA6xWmYsO2Md6-EoqjeIzk&m=q_uNuYFdGrDiFyB8x0KjRuPV4TbYGJf20PKQKambrfE&s=BqWJdkdWSRVMHWQkLWAhYaV0yfRwJZDUb61TfAgRss0&e="
target="_blank">http://review.whamcloud.com/#/c/8451/1</a><br>
<br>
Now first I want to do
the Infiniband setup
for mgs and mdt on
single machine which
also has Ethernet IP.
Then I want to format
and mount mgs and mdt.<br>
So I installed above
created lustre rpms
and then added below
line in <span
style="color:rgb(0,0,0);font-family:monospace;font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;display:inline!important;float:none">/etc/modprobe.d/lustre.conf</span><span
style="color:rgb(0,0,0);font-family:monospace;font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;display:inline!important;float:none"><br>
</span>options lnet
networks=o2ib(ib0)<br>
<br>
Then I rebooted the
machine to remove all
lustre related modules
including lnet and
then ran<font
face="Courier New">
modprobe lnet</font>
command to add above
parameters and the ran
<font face="Courier
New">lctl network up</font>
which is giving me
below error:<br>
LNET configure error
100: Network is down<br>
<br>
I looked online and
found below discussion
on same error:<br>
<a
moz-do-not-send="true"
href="https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.lustre.org_pipermail_lustre-2Ddiscuss_2010-2DJune_013510.html&d=AAMCAw&c=IGDlg0lD0b-nebmJJ0Kp8A&r=c-1Cg_VH2lcYI_JXS3gypPA6xWmYsO2Md6-EoqjeIzk&m=q_uNuYFdGrDiFyB8x0KjRuPV4TbYGJf20PKQKambrfE&s=aCgXfqCUyJ7IXVRJHjqpk2HCS1_dsKDuaKJrDPmWp4I&e="
target="_blank">http://lists.lustre.org/pipermail/lustre-discuss/2010-June/013510.html</a><br>
<br>
As per suggestion in
above mail I tried
with below line in <span
style="color:rgb(0,0,0);font-family:monospace;font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;display:inline!important;float:none">/etc/modprobe.d/lustre.conf</span>.
In below command for
IB_IP, I have given
infiniband IP.<br>
options lnet <b>networks=o2ib(ib0)</b>
routes="tcp0
IB_IP@o2ib"<br>
This command hangs for
around 2 to 3 minutes
and then gives error:
Write failed: Broken
pipe. Same is the case
for "options lnet <b>networks=o2ib(ib0)</b>"<br>
But if I set: options
lnet <b>networks=tcp0(eth0),o2ib(ib0)</b>
routes="tcp1
IB_IP@o2ib" then it
gives LNET configure
error 100: Network is
down.<br>
<br>
It seems that for
network=o2ib(ibo) I am
getting error Write
failed: Broken pipe.<br>
Am I missing anything
while following above
steps? Or how do I
resolve above error? <br>
<br>
Thanks,<br>
Aayush. <br>
</div>
</div>
<br>
<html><br>
_______________________________________________<br>
HPDD-discuss mailing list<br>
<a moz-do-not-send="true"
href="mailto:HPDD-discuss@lists.01.org" target="_blank">HPDD-discuss@lists.01.org</a><br>
<a moz-do-not-send="true"
href="https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.01.org_mailman_listinfo_hpdd-2Ddiscuss&d=AAICAg&c=IGDlg0lD0b-nebmJJ0Kp8A&r=c-1Cg_VH2lcYI_JXS3gypPA6xWmYsO2Md6-EoqjeIzk&m=q_uNuYFdGrDiFyB8x0KjRuPV4TbYGJf20PKQKambrfE&s=0hW3r7x0NhgbZ7zgaZKr9K_fk7_E8bs0f-GAlH89rgM&e="
target="_blank">https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.01.org_mailman_listinfo_hpdd-2Ddiscuss&d=AAICAg&c=IGDlg0lD0b-nebmJJ0Kp8A&r=c-1Cg_VH2lcYI_JXS3gypPA6xWmYsO2Md6-EoqjeIzk&m=q_uNuYFdGrDiFyB8x0KjRuPV4TbYGJf20PKQKambrfE&s=0hW3r7x0NhgbZ7zgaZKr9K_fk7_E8bs0f-GAlH89rgM&e=</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
<br>
</div>
</div>
_______________________________________________<br>
Lustre-discuss mailing list<br>
<a moz-do-not-send="true"
href="mailto:Lustre-discuss@lists.lustre.org"
target="_blank">Lustre-discuss@lists.lustre.org</a><br>
<a moz-do-not-send="true"
href="http://lists.lustre.org/mailman/listinfo/lustre-discuss"
target="_blank">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
</body>
</html>