<div dir="ltr">opps, sorry, Sohamm, not sure why I missed your reply. <div><br></div><div>Thanks for sharing, so lnet is not needed? then how then connect to each other? will check doc more carefully. </div><div><br></div><div>Chen</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Jun 26, 2016 at 12:02 AM, sohamm <span dir="ltr"><<a href="mailto:sohamm@gmail.com" target="_blank">sohamm@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small;color:rgb(153,51,153)">Hi YuChen, </div><div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small;color:rgb(153,51,153)"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small;color:rgb(153,51,153)">My setup also uses zfs on lustre and below is the output log from my system. </div><div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small;color:rgb(153,51,153)">The only difference is i did not setup the lnet ( as the lustre doc says its optional ). Not sure if thats the reason you see the error and i do not. I am also new to lustre and trying the config on bunch of VM's. </div><div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small;color:rgb(153,51,153)"><br></div><div class="gmail_default"><div class="gmail_default"><font color="#993399" face="tahoma, sans-serif">[root@lustre_mgs01_vm03 ~]# systemctl status lustre</font></div><div class="gmail_default"><font color="#993399" face="tahoma, sans-serif">● lustre.service - SYSV: Part of the lustre file system.</font></div><div class="gmail_default"><font color="#993399" face="tahoma, sans-serif"> Loaded: loaded (/etc/rc.d/init.d/lustre)</font></div><div class="gmail_default"><font color="#993399" face="tahoma, sans-serif"> Active: inactive (dead)</font></div><div class="gmail_default"><font color="#993399" face="tahoma, sans-serif"> Docs: man:systemd-sysv-generator(8)</font></div><div class="gmail_default"><font color="#993399" face="tahoma, sans-serif"><br></font></div><div class="gmail_default"><font color="#993399" face="tahoma, sans-serif">[root@lustre_mgs01_vm03 ~]# systemctl start lustre</font></div><div class="gmail_default"><font color="#993399" face="tahoma, sans-serif">[root@lustre_mgs01_vm03 ~]# systemctl status lustre</font></div><div class="gmail_default"><font color="#993399" face="tahoma, sans-serif">● lustre.service - SYSV: Part of the lustre file system.</font></div><div class="gmail_default"><font color="#993399" face="tahoma, sans-serif"> Loaded: loaded (/etc/rc.d/init.d/lustre)</font></div><div class="gmail_default"><font color="#993399" face="tahoma, sans-serif"> Active: active (exited) since Sat 2016-06-25 23:49:45 EDT; 2s ago</font></div><div class="gmail_default"><font color="#993399" face="tahoma, sans-serif"> Docs: man:systemd-sysv-generator(8)</font></div><div class="gmail_default"><font color="#993399" face="tahoma, sans-serif"> Process: 17751 ExecStart=/etc/rc.d/init.d/lustre start (code=exited, status=0/SUCCESS)</font></div><div class="gmail_default"><font color="#993399" face="tahoma, sans-serif"><br></font></div><div class="gmail_default"><font color="#993399" face="tahoma, sans-serif">Jun 25 23:49:45 lustre_mgs01_vm03 systemd[1]: Starting SYSV: Part of the lustre file system....</font></div><div class="gmail_default"><font color="#993399" face="tahoma, sans-serif">Jun 25 23:49:45 lustre_mgs01_vm03 systemd[1]: Started SYSV: Part of the lustre file system..</font></div><div class="gmail_default"><font color="#993399" face="tahoma, sans-serif"><br></font></div><div class="gmail_default"><font color="#993399" face="tahoma, sans-serif">[root@lustre_mgs01_vm03 ~]# ls /etc/modprobe.d/</font></div><div class="gmail_default"><font color="#993399" face="tahoma, sans-serif">ko2iblnd.conf mlx4.conf</font></div><div class="gmail_default"><font color="#993399" face="tahoma, sans-serif"><br></font></div><div class="gmail_default"><font color="#993399" face="tahoma, sans-serif">[root@lustre_mgs01_vm03 ~]# yum list installed | grep zfs</font></div><div class="gmail_default"><font color="#993399" face="tahoma, sans-serif">Repodata is over 2 weeks old. Install yum-cron? Or run: yum makecache fast</font></div><div class="gmail_default"><font color="#993399" face="tahoma, sans-serif">libnvpair1.x86_64 0.6.5.7-1.el7.centos @zfs</font></div><div class="gmail_default"><font color="#993399" face="tahoma, sans-serif">libuutil1.x86_64 0.6.5.7-1.el7.centos @zfs</font></div><div class="gmail_default"><font color="#993399" face="tahoma, sans-serif">libzfs2.x86_64 0.6.5.7-1.el7.centos @zfs</font></div><div class="gmail_default"><font color="#993399" face="tahoma, sans-serif">libzpool2.x86_64 0.6.5.7-1.el7.centos @zfs</font></div><div class="gmail_default"><font color="#993399" face="tahoma, sans-serif">lustre-osd-zfs.x86_64 2.8.54_1_g5f03bf9-3.10.0_327.18.2.el7_lustre.x86_64</font></div><div class="gmail_default"><font color="#993399" face="tahoma, sans-serif">lustre-osd-zfs-mount.x86_64 2.8.54_1_g5f03bf9-3.10.0_327.18.2.el7_lustre.x86_64</font></div><div class="gmail_default"><font color="#993399" face="tahoma, sans-serif">spl.x86_64 0.6.5.7-1.el7.centos @zfs</font></div><div class="gmail_default"><font color="#993399" face="tahoma, sans-serif">spl-dkms.noarch 0.6.5.7-1.el7.centos @zfs</font></div><div class="gmail_default"><font color="#993399" face="tahoma, sans-serif">zfs.x86_64 0.6.5.7-1.el7.centos @zfs</font></div><div class="gmail_default"><font color="#993399" face="tahoma, sans-serif">zfs-dkms.noarch 0.6.5.7-1.el7.centos @zfs</font></div><div class="gmail_default"><font color="#993399" face="tahoma, sans-serif">zfs-release.noarch 1-2.el7.centos @/zfs-release.el7.noarch</font></div><div class="gmail_default"><font color="#993399" face="tahoma, sans-serif"><br></font></div><div class="gmail_default"><font color="#993399" face="tahoma, sans-serif">[root@lustre_mgs01_vm03 ~]# yum list installed | grep lustre</font></div><div class="gmail_default"><font color="#993399" face="tahoma, sans-serif">Repodata is over 2 weeks old. Install yum-cron? Or run: yum makecache fast</font></div><div class="gmail_default"><font color="#993399" face="tahoma, sans-serif">kernel.x86_64 3.10.0-327.13.1.el7_lustre @hpddLustreserver</font></div><div class="gmail_default"><font color="#993399" face="tahoma, sans-serif">kernel.x86_64 3.10.0-327.18.2.el7_lustre @hpddLustreserver</font></div><div class="gmail_default"><font color="#993399" face="tahoma, sans-serif">kernel-devel.x86_64 3.10.0-327.13.1.el7_lustre @hpddLustreserver</font></div><div class="gmail_default"><font color="#993399" face="tahoma, sans-serif">kernel-headers.x86_64 3.10.0-327.13.1.el7_lustre @hpddLustreserver</font></div><div class="gmail_default"><font color="#993399" face="tahoma, sans-serif">lustre.x86_64 2.8.54_1_g5f03bf9-3.10.0_327.18.2.el7_lustre.x86_64</font></div><div class="gmail_default"><font color="#993399" face="tahoma, sans-serif">lustre-iokit.x86_64 2.8.53_11_gfd4ab6e-3.10.0_327.13.1.el7.x86_64</font></div><div class="gmail_default"><font color="#993399" face="tahoma, sans-serif">lustre-modules.x86_64 2.8.54_1_g5f03bf9-3.10.0_327.18.2.el7_lustre.x86_64</font></div><div class="gmail_default"><font color="#993399" face="tahoma, sans-serif">lustre-osd-ldiskfs.x86_64 2.8.54_1_g5f03bf9-3.10.0_327.18.2.el7_lustre.x86_64</font></div><div class="gmail_default"><font color="#993399" face="tahoma, sans-serif">lustre-osd-ldiskfs-mount.x86_64 2.8.54_1_g5f03bf9-3.10.0_327.18.2.el7_lustre.x86_64</font></div><div class="gmail_default"><font color="#993399" face="tahoma, sans-serif">lustre-osd-zfs.x86_64 2.8.54_1_g5f03bf9-3.10.0_327.18.2.el7_lustre.x86_64</font></div><div class="gmail_default"><font color="#993399" face="tahoma, sans-serif">lustre-osd-zfs-mount.x86_64 2.8.54_1_g5f03bf9-3.10.0_327.18.2.el7_lustre.x86_64</font></div><div class="gmail_default"><font color="#993399" face="tahoma, sans-serif">lustre-tests.x86_64 2.8.54_1_g5f03bf9-3.10.0_327.18.2.el7_lustre.x86_64</font></div><div style="color:rgb(153,51,153);font-family:tahoma,sans-serif;font-size:small"><br></div><div style="color:rgb(153,51,153);font-family:tahoma,sans-serif;font-size:small">Thanks</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jun 24, 2016 at 1:07 PM, <span dir="ltr"><<a href="mailto:lustre-discuss-request@lists.lustre.org" target="_blank">lustre-discuss-request@lists.lustre.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Send lustre-discuss mailing list submissions to<br>
<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
or, via email, send a message with subject or body 'help' to<br>
<a href="mailto:lustre-discuss-request@lists.lustre.org" target="_blank">lustre-discuss-request@lists.lustre.org</a><br>
<br>
You can reach the person managing the list at<br>
<a href="mailto:lustre-discuss-owner@lists.lustre.org" target="_blank">lustre-discuss-owner@lists.lustre.org</a><br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of lustre-discuss digest..."<br>
<br>
<br>
Today's Topics:<br>
<br>
1. Re: Lustre Client Package for Rocks 6.2 (Yasir Israr)<br>
2. Can't load module 'osd-zfs' (Yu Chen)<br>
<br>
<br>
----------------------------------------------------------------------<br>
<br>
Message: 1<br>
Date: Fri, 24 Jun 2016 08:24:36 +0530<br>
From: Yasir Israr <<a href="mailto:yasir@orionsolutions.co.in" target="_blank">yasir@orionsolutions.co.in</a>><br>
To: "Mannthey, Keith" <<a href="mailto:keith.mannthey@intel.com" target="_blank">keith.mannthey@intel.com</a>><br>
Cc: "<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a>"<br>
<<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a>><br>
Subject: Re: [lustre-discuss] Lustre Client Package for Rocks 6.2<br>
Message-ID:<br>
<<a href="mailto:F43ED9E5-1716-4A3F-B0DF-1FE79B7FCA75@orionsolutions.co.in" target="_blank">F43ED9E5-1716-4A3F-B0DF-1FE79B7FCA75@orionsolutions.co.in</a>><br>
Content-Type: text/plain; charset="utf-8"<br>
<br>
Hi Keith<br>
Running below src it's asking again to downgrade existing kernel. & in rocks compute I couldn't able to amend kernel because of rocks issue<br>
<br>
Thanks & Regards<br>
Yasir Israr<br>
<br>
> On 24-Jun-2016, at 4:12 AM, Mannthey, Keith <<a href="mailto:keith.mannthey@intel.com" target="_blank">keith.mannthey@intel.com</a>> wrote:<br>
><br>
> Yasir,<br>
> Did you try rebuilding from the src.rpm against your current kernel?<br>
> <a href="https://wiki.hpdd.intel.com/display/PUB/Rebuilding+the+Lustre-client+rpms+for+a+new+kernel" rel="noreferrer" target="_blank">https://wiki.hpdd.intel.com/display/PUB/Rebuilding+the+Lustre-client+rpms+for+a+new+kernel</a><br>
><br>
> Thanks,<br>
> Keith<br>
><br>
><br>
> From: lustre-discuss [mailto:<a href="mailto:lustre-discuss-bounces@lists.lustre.org" target="_blank">lustre-discuss-bounces@lists.lustre.org</a>] On Behalf Of Yasir Israr<br>
> Sent: Thursday, June 23, 2016 3:16 PM<br>
> To: E.S. Rosenberg <<a href="mailto:esr%2Blustre@mail.hebrew.edu" target="_blank">esr+lustre@mail.hebrew.edu</a>><br>
> Cc: <a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
> Subject: Re: [lustre-discuss] Lustre Client Package for Rocks 6.2<br>
><br>
> Hi Rosenberg<br>
> I've tried using source code but problem not resolved every lustre client package asking to downgrade below existing kernel but that I can't do because of many application will get an issue.<br>
><br>
> Thanks & Regards<br>
> Yasir Israr<br>
><br>
> On 23-Jun-2016, at 4:57 PM, E.S. Rosenberg <<a href="mailto:esr%2Blustre@mail.hebrew.edu" target="_blank">esr+lustre@mail.hebrew.edu</a>> wrote:<br>
><br>
> Have you tried doing the lustre install from source?<br>
><br>
> On Thu, Jun 23, 2016 at 10:45 AM, <<a href="mailto:yasir@orionsolutions.co.in" target="_blank">yasir@orionsolutions.co.in</a>> wrote:<br>
> Dear Rocks User,<br>
><br>
> Please help in getting correct Lustre Client Package for Rocks 6.2 with Kernel 2.6.32-504.16.2.el6.x86_64<br>
> Currently I?m trying to install Client Package of Lustre 2.7 & 2.8 but it?s not working & asking to downgrade existing kernel.<br>
><br>
> I couldn?t able to downgrade because we are running many packages which is installed & working fine. So please help if we can do it without downgrade kernel.<br>
><br>
> Thanks & Regards,<br>
> Yasir Israr<br>
><br>
><br>
> _______________________________________________<br>
> lustre-discuss mailing list<br>
> <a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
> <a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
><br>
><br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20160624/29ba2ee6/attachment.html" rel="noreferrer" target="_blank">http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20160624/29ba2ee6/attachment.html</a>><br>
<br>
------------------------------<br>
<br>
Message: 2<br>
Date: Fri, 24 Jun 2016 10:48:20 -0400<br>
From: Yu Chen <<a href="mailto:chenyu@umbc.edu" target="_blank">chenyu@umbc.edu</a>><br>
To: <a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
Subject: [lustre-discuss] Can't load module 'osd-zfs'<br>
Message-ID:<br>
<<a href="mailto:CAO0EFYMN-0f54mP_oQXtXRNv-nK8rqrzrLqxQwfqKUA5axXgfA@mail.gmail.com" target="_blank">CAO0EFYMN-0f54mP_oQXtXRNv-nK8rqrzrLqxQwfqKUA5axXgfA@mail.gmail.com</a>><br>
Content-Type: text/plain; charset="utf-8"<br>
<br>
Hello,<br>
<br>
I am new to lustre system. Was trying to setup a lustre on Centos 7 with<br>
zfs.<br>
<br>
When came to the step to start lustre.service from systemctl, got the<br>
errors:<br>
....<br>
<br>
-------<br>
<br>
osd_zfs: disagrees about version of symbol dmu_objset_space<br>
<br>
osd_zfs: Unknown symbol dmu_objset_space (err -22)<br>
<br>
------<br>
<br>
LustreError: 158-c: Can't load module 'osd-zfs'<br>
<br>
LustreError: 48041:0:(genops.c:318:class_newdev()) OBD: unknown type: osd<br>
-zfs<br>
<br>
.....<br>
<br>
<br>
Here is what I have done:<br>
<br>
<br>
- I installed kernel-3.10.0-327.3.1.el7_lustre.x86_64,<br>
- Then configured and installed zfs from <a href="http://archive.zfsonlinux.org" rel="noreferrer" target="_blank">archive.zfsonlinux.org</a><br>
(0.6.5.7-1)<br>
- Then downloaded and installed lustre rpms from el7.2.1511 from:<br>
<a href="https://downloads.hpdd.intel.com/public/lustre/latest-feature-release/" rel="noreferrer" target="_blank">https://downloads.hpdd.intel.com/public/lustre/latest-feature-release/</a><br>
- Then *zpool* create mdt01-pool /dev/sdb<br>
- mkfs.lustre --reformat --fsname=lustre01 --mgs --mdt --backfstype=zfs<br>
--index=0 --mgsnode=10.1.12.1@o2ib --servicenode=10.1.12.1@o2ib<br>
mdt01-pool/mgsmdt01<br>
- echo "mds1 - lustre01-MDT0000 zfs:mdt01-pool/mgsmdt01" > /etc/ldev.conf<br>
- echo "options lnet networks=o2ib(ib0)" > /etc/modprobe.d/lustre.conf<br>
- systemctl start lustre.service<br>
<br>
This is where it failed.<br>
<br>
Then I think it might be because of the zfs version. So downloaded<br>
lustre-2.8.0-3.10.0_327.3.1.el7_lustre.x86_64.src.rpm, and tried to rebuild<br>
from it. But after rebuilding, there were no lustre-osd-zfs and<br>
lustre-osd-zfs-mount rpms?<br>
<br>
<br>
Would greatly appreciate any suggestions!<br>
<br>
<br>
Thanks in advance!<br>
<br>
<br>
yc<br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20160624/ed3d8deb/attachment-0001.htm" rel="noreferrer" target="_blank">http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20160624/ed3d8deb/attachment-0001.htm</a>><br>
<br>
------------------------------<br>
<br>
Subject: Digest Footer<br>
<br>
_______________________________________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
<br>
<br>
------------------------------<br>
<br>
End of lustre-discuss Digest, Vol 123, Issue 35<br>
***********************************************<br>
</blockquote></div><br></div></div>
<br>_______________________________________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
<br></blockquote></div><br></div>