[Lustre-discuss] problem with installing lustre and OFED

[Ms. Megan Larko]

Greetings Jason,

As you have most likely discovered, Mellanox (MLNX) needs to be built
into the lustre linux kernel to use InfiniBand.

I worked on such an issue recently.   The Whamcloud linux kernel
2.1.2-2.6.32_220.17.1.el6_lustre would not work with our Mellanox
InfiniBand (IB) drivers optimally.  We got the MLXN version 1.8.5 to
match our Mellanox hardware and had to do the dance already described
to you in this list of...
1.   downloading all of the appropriate (Whamcloud) lustre linux
kernels, header and devel rpms
2.   boot into the lustre kernel
3.   in our /usr/src/lustre-2.1.2 directory built lustre against the
Mellanox "Module.symvers" information (which is why you see the
"Input/Output" errors on fid.ko, mdc.ko, osc.ko, lov.ko and because of
the aforementioned items, the lustre.ko.   The MLNX version 1.8.5 that
we needed was in the /usr/src/ofa_kernel directory (with the
Module.symvers etc....)  We used the defaults other than the o2ib so
our command in the /usr/src/lustre-2.1.2 directory looked like
"./configure --with-o2ib=/usr/src/ofa_kernel"
4.   next we issued "make"
5.   next we chose to run a "make rpms" command so that we could have
rpms for our system for cluster re-building

We had to do this for *both* our lustre servers and lustre clients
(using the lustre-client Whamcloud kernel, headers, ...   So we had
the servers and the clients communicating properly over the MLNX ib
fabric.

In /etc/modprobe.d  we used a lustre.conf file to explicitly direct
the system to use the o2ib network when starting lustre at boot.

Without the above actions the ko2iblnd would not load.

Just confirming that you need to build Mellanox on servers and clients
to use MLNX IB with Lustre cluster file system.

megan