[lustre-discuss] Advice: Upgrading / Replacing servers

[Phill Harvey-Smith]

Hi all,

We will shortly be replacing the hardware that our Lustre servers run on 
as they are both over 5 years old and are due for retirement.

Our current setup is this :

Note that since this was largely setup before I joined and since it's in 
service at the moment I can't really take the machines down to confirm 
the raid configurations :(



Our OSS is a Dell server with 4 x 750GB drives internally as sda and 2 
banks of 6 x 2TB drives externally as sdb and sdc.

The internal drives are either raid 0+1 or raid 1+0 giving a total of 1.5TB.

The external drives are setup as two banks of 6 drives raided to give 
7.3TB each.

The Lustre specific parts are :

/dev/sda7             279G   66G  199G  25% /lustre/home/OST0000
/dev/sda11            765G  235G  492G  33% /lustre/scratch/OST0000
/dev/sdc              7.3T  2.4T  4.6T  34% /lustre/storage/OST0001
/dev/sdb              7.3T  2.6T  4.4T  37% /lustre/storage/OST0000


Our MDS / MDT are on the same server again a Dell server with 6 drives 
internally, setup as :

2x500GB mirrored, for a total of 500GB
4x600GB again either raid 0+1 or raid 1+0 for a total of 1.2TB, this is 
managed by lvm.

Details are :
/dev/sda6              46G  453M   44G   2% /lustre/mgs
/dev/mapper/mdt00vg-home_mdt00v
                        75G  843M   70G   2% /lustre/home-mdt
/dev/mapper/mdt00vg-scratch_mdt00v
                       300G 1001M  279G   1% /lustre/scratch-mdt
/dev/mapper/mdt00vg-storage_mdt00v
                       300G  4.0G  276G   2% /lustre/storage-mdt

The MDS/MDT is also a client though I am not sure why.

The servers are both running Lustre servers versions 2.1, using a Redhat 
kernel on top of an Ubuntu 10.04 base (don't ask!, this was again setup 
by a previous admin)



Our plans are to completely overhaul this, the replacement hardware will 
have close to the same disk configuration as the Lustre filesystems are 
currently only at about 35% utilization so we don't need a capacity 
increase.

One thing I *WILL* be doing is migrating the servers to the latest 
CentOS & Lustre version 2.8 as this is a supported server platform.

One thing I am wondering about is what would be the best underlying 
filesystem to use ? I see that Lustre now supports ZFS, I obviously need 
to make this decision now as reading around the subject it seems that 
ZFS works best where the disks are non raided and ZFS handles the 
striping / mirroring etc.

What do people think?

Cheers.

Phill.