[Lustre-discuss] How to install Lustre onto LVM?

[Daire Byrne]

Andrea,

Interesting configuration. Is there a particular reason why you decided
on using Xen VMs? Is failover better with Xen instances? I'm guessing 
you don't have hundreds of clients hammering the hardware.

----- "Andrea Rucks" <andrea.rucks at us.lawson.com> wrote:

> In your thread discussing "e2scan MDT backup", I was very interested
> in finding out more about how you initially setup your OST's and MDT's
> with LVM. We're implementing a new set of production Lustre servers
> and had Sun come on-site last fall to help get us started. Our Sun
> Lustre consultant was very knowledgeable and helpful and he setup our
> environment as follows:

<SNIP!>

> As you can see, our Sun consultant set our production environment up
> to use standard disk partitions that are striped across two sets of
> servers for our five Lustre filesystems (which is working fine and we
> can grow it no problems). We're running RHEL 5.2 XEN MGS/MDT/OSS
> servers using Linux Heartbeat for STONITH and failover. The XEN domU's
> are LVM-backed on EMC SAN. Lustre was compiled into the XEN kernel and
> we've compiled and are using the patchless Lustre client (Lustre
> 1.6.6).

I'm curious as to why you created 5 filesystems on the same "hardware" 
instead of one big filesystem?

> What I'm really curious about is how to go about setting up this
> environment under LVM2. Is it as simple as taking the "xvd?" disk on
> the OSS and MDS servers, doing a pvcreate, vgcreate, lvcreate, mkfs?
> Do you have to be careful about anything (striping as seen above)? May
> I ask if you might be able to give an example? I'd like to test LVM in
> our Lustre certification environment, but would like to make sure I
> have an idea of what I'm doing before I mess up my working cert
> environment. Are there any pros / cons for LVM2 versus disk
> partitions? Backup issues, speed, performance, etc?

The write performance does suffer as you add snapshots but on the MDT
it is unlikely to be that noticeable and you can remove the snapshot
when you have grabbed what you need for the EA and file backups. Also 
you shouldn't need to put the OSTs on LVM if you just want to do MDT 
backups. OST snapshots are only useful if you want to mount a snapshot
of the entire lustre filesystem on a client. Beware though that this is
likely to slow down the production filesystem permanently. We run this
configuration on a backup filesystem that is not accessed by production
so the slowdown is not important.

I'm not sure it is possible to migrate an existing filesystem to LVM 
easily - you would need to do a file backup of your MDT first and restore 
to the LVM device (section 15.1.3.1 of the manual). So in your case to 
wipe (!) a single MDT and create a new one I'd do something like:

  pvcreate /dev/xvdj
  vgcreate lusfs01 /dev/xvdj
  vgchange -a y lusfs01
  lvcreate -L3G -nmdt lusfs01

Now you do the mkfs.lustre stuff on it and the corresponding OST devices.
You can then create and remove a snapshot at anytime with something like:

  lvcreate -Ay -nmdt-snap -L50G -s -prw /dev/lusfs01/mdt
  lvremove /dev/lusfs01/mdt-snap

I hope this helps.

Daire