[Lustre-discuss] Latest RHEL kernel: won't make mgs module, and ko2iblnd not built for right OFED modules

[Chris Worley]

On Fri, Feb 29, 2008 at 1:49 PM, Canon, Richard Shane <canonrs at ornl.gov> wrote:
>
>  Chris,
>
>  Try using /usr/local/ofed/current/src/ofa_kernel/ instead of the version
>  specific one.  We were seeing Oops when compiling against the version
>  specific tree.  I've been meaning to post to the list about this.
Shane,

I'm amazed that the above path works on our system too, as our setup
differs from standard, where we always put OFED into /usr/local, and
"current" points to the OFED we've built for the current kernel:

# ls -l /usr/local/ofed/
total 4
drwxr-xr-x  9 root root 4096 Mar  3 21:52 1.2.5.5-lustre
lrwxrwxrwx  1 root root   31 Feb 28 16:45 current ->
/usr/local/ofed/1.2.5.5-lustre/

So, I'm not sure what you mean by "current" vs. "version specific" in
your system.  Is "current" not 1.2.5.5... maybe something distributed
w/ RHEL (or whatever distro you're using)?

Do you remember the specific panic you got?  I posted my panic in a
separate thread (as I have resolved the issues of this thread... as
Andreas said, I had patched the kernel incorrectly).

Thanks,

Chris
>
>
>
>  --Shane
>
>  -----Original Message-----
>  From: lustre-discuss-bounces at lists.lustre.org
>
>
> [mailto:lustre-discuss-bounces at lists.lustre.org] On Behalf Of Chris
>  Worley
>  Sent: Friday, February 29, 2008 1:32 PM
>  To: lustre-discuss
>  Subject: Re: [Lustre-discuss] Latest RHEL kernel: won't make mgs
>  module,and ko2iblnd not built for right OFED modules
>
>  On Fri, Feb 29, 2008 at 8:50 AM, Canon, Richard Shane <canonrs at ornl.gov>
>  wrote:
>  >
>  >
>  >  Yes.  We also had to fix that.  Sorry I left that off.  In our case,
>  we
>  >  created a modified Module.symvers by grep'ing out anything with
>  >  drivers/infiniband.  Also, OFED will create a Module.symvers for its
>  >  symbols.  You can append that to the (fixed) Module.symvers so that
>  the
>  >  symbols will still be checked.
>
>  Okay, I found it:
>
>  /usr/share/doc/ofed-docs-1.2.5.5/create_Module.symvers.sh
>
>  I've also changed my config line in Lustre to point to the proper
>  modules:
>
>  ./configure --with-o2ib=/usr/local/ofed/current/src/ofa_kernel-1.2.5.5/
>  --with-ldiskfs
>
>  ... that caused a few other Lustre build errors that were easy to
>  fix/hack.
>
>  To fix the mgs.ko issue, I'm just manually copying the module into
>  /lib/modules.
>
>  ... now it just panics when I "modprobe mgs" ;)
>
>  Chris
>  >
>  >
>  >  --Shane
>  >
>  >  -----Original Message-----
>  >  From: lustre-discuss-bounces at lists.lustre.org
>  >
>  > [mailto:lustre-discuss-bounces at lists.lustre.org] On Behalf Of Craig
>  >  Prescott
>  >  Sent: Friday, February 29, 2008 10:41 AM
>  >  To: Chris Worley
>  >  Cc: lustre-discuss at lists.lustre.org
>  >  Subject: Re: [Lustre-discuss] Latest RHEL kernel: won't make mgs
>  module,
>  >  and ko2iblnd not built for right OFED modules
>  >
>  >
>  >
>  >
>  > Nod - we ran into this issue as well.  Your Module.symvers
>  >  file for probably has entries in it for the IB modules
>  >  included with the RHEL kernel (and not from OFED).
>  >
>  >  Here's what we do to avoid this issue (and others) when
>  >  building a patched Lustre kernel with OFED:
>  >
>  >  1) Get the source for that kernel (rpmbuild -bp the
>  >     src.rpm and copy it somewhere).
>  >  2) Configure that kernel *without* infiniband
>  >     (edit the config for that kernel to say
>  >     "#CONFIG_INFINIBAND is not set" and get rid
>  >     of all the other IB stuff).
>  >  3) patch that kernel for Lustre
>  >  4) build and install that kernel
>  >  5) build ofa-kernel against that kernel and install it
>  >  6) cat <ofa-kernel-dir>/Module.symvers >>
>  >     <linux-kernel-dir>/Module.symvers
>  >  7) Build and install Lustre against this kernel and
>  >     ofa-kernel.
>  >
>  >  Hope that helps,
>  >  Craig Prescott
>  >  UF HPC Center
>  >
>  >  Chris Worley wrote:
>  >  > Lustre 1.6.4.2, OFED 1.2.5.5, RHEL kernel 2.6.9.67.0.4.
>  >  >
>  >  > Configured with: --with-o2ib --with-ldiskfs
>  >  >
>  >  > mgs.ko is built but not wrapped into modules rpm, with an undefined
>  >  > external that should be from the lvfs module (lvfs_check_rdonly).
>  >  >
>  >  > When ko2iblnd loads, it gets lots of "disagrees about version of
>  >  > symbol <some OFED symbol>" messages.  I'm guessing this is building
>  >  > wrt the ib modules in the kernel rather than the OFED modules I
>  have
>  >  > loaded.
>  >  >
>  >  > There were other patches I needed to make for this kernel (to
>  resolve
>  >  > mutex_lock/unlock, __d_rehash, and __d_move... but those were easy
>  to
>  >  > fix).
>  >  >
>  >  > What am I doing wrong?
>  >  >
>  >  > Thanks,
>  >  >
>  >  > Chris
>  >  > _______________________________________________
>  >  > Lustre-discuss mailing list
>  >  > Lustre-discuss at lists.lustre.org
>  >  > http://lists.lustre.org/mailman/listinfo/lustre-discuss
>  >
>  >  _______________________________________________
>  >  Lustre-discuss mailing list
>  >  Lustre-discuss at lists.lustre.org
>  >  http://lists.lustre.org/mailman/listinfo/lustre-discuss
>  >
>  _______________________________________________
>  Lustre-discuss mailing list
>  Lustre-discuss at lists.lustre.org
>  http://lists.lustre.org/mailman/listinfo/lustre-discuss
>