[Lustre-discuss] struggling just get started

[Robin Humble]

On Sat, Jan 19, 2008 at 11:18:15AM -0500, William G Wichser wrote:
>The first item of business appears that I can download a kernel already built for my machine.  Talk on the network seems to indicate that this will not run over an IB interface unless I compile a new kernel from sources.  Yet I don't quite understand why this might be as I don't have toi recompile running over a gigE connection, regardless of the actual interface. Whether it be an Intel or a Broadcom, it doesn't appear to matter and I don't need to recompile against those sources.  Which makes me question why I'd need to recompile when using IB.

the ko2iblnd.ko Lustre kernel module needs to be built in order to use
the IB native protocols, and Sun doesn't seem to do this at the moment.
I don't know why. it's not there for sles10 or for rhel5, I haven't
checked the rest.

>It would seem that if I chose TCPoIB that the actual kernel drivers should handle this just fine, running over a TCP stack and the actual device driver should do "the right thing" by making these packets available over the correct physical device.

IPoIB will work, and it'll be faster than GigE, but nowhere near as fast
as using native IB (verbs etc.)

>Now maybe there is a performance issue in doing it this way.  and maybe it just doesn't work that way at all.

>So I've read some things and downloaded the sources, both the modified kernel and the lustre-source RPMs.  Now it want me to supply some link to the openib source directory.  How do I determine which OFED was installed with the kernel so I know which sources to download?  Aren't these drivers a part of the kernel now?  And if so, houldn't they actually be already there in the kernel sources?

unless you have a specific reason not to then it's simplest to just
point Lustre's configure at the kernel's IB implementation. eg.
  ./configure --with-linux=<kpath> --with-o2ib=<kpath>
where <kpath> might be /usr/src/linux-2.6.5-7.199/ or
/lib/modules/2.6.5-7.199-sn2/build/ or something like that. you'll need
the sles10 devel or source rpm or whatever it's called installed to get
this tree.

>Thanks for any help anyone can supply with helping me to understand how to even start this build process.

kinda depends if you want a patched server or patchless.

if patchless then just the above will suffice.
if patching then you need to patch (and preferably build) your kernel
tree prior to doing the above configure. I think the Sun recommended
way to patch is via quilt, or you can apply each of the patches in
  lustre/kernel_patches/series/2.6-sles10.series

IMHO the patched kernels are usually more stable than the patchless,
but you need to rebuild the kernel so they're a tad more of a pain.

cheers,
robin