[Lustre-discuss] diskless booting & Debian

[Jeff Darcy]

Troy Benjegerdes wrote:
> I see a relatively blank diskless booting wiki page at
>
> http://wiki.lustre.org/index.php?title=Diskless_Booting
>
> Are there any details on this, other than "University of Colorado at
> Boulder has performed interesting work?"
>
> I have a couple of interests here.. 
>
> 1) compare lustre netboot to AFS netboot
> 2) install Lustre MDS & ODS servers on Debian booted from AFS.
> 3) understand enough about Lustre to know if netbooting a Lustre server
>    from a lustre filesystem is completely wacked, or not.
>
> FYI, I haven't tried it yet, but I'd be comfortable booting an AFS
> server from a replicated AFS read-only volume as the root filesystem..
> I believe I understand the failure modes of AFS well enough that this
> would work.
>   
We've actually experimented a bit with this, since most of the nodes in
our system are disk-less.  In fact, they're -less just about everything
except for processors, memory, and a connection to our internal fabric. 
We've tried loading "canned" MDT/OST images into memory on some nodes
and serving from there, and it does seem to work.  There are two
downsides, though.  One is that the Linux loopback driver is a real
performance bottleneck, ever since some bright person had the idea to
make it less multi-threaded than it had been.  Another is that booting
tends to involve metadata-heavy access patterns which are not exactly
Lustre's strength - a situation made worse when you have nearly a
thousand clients doing it at the same time and your MDS is a relatively
small node like the others.  So far we've found that NBD serves us
better in the boot/root filesystem role, though that means a read-only
root which involves its own complexity.  Your mileage will almost
certainly vary.