[Lustre-discuss] Speeding up configuration log regeneration?

[Dilger, Andreas]

On 2013/10/17 5:34 AM, "Olli Lounela" <olli.lounela at helsinki.fi> wrote:

>Hi,
>
>We run four-node Lustre 2.3, and I needed to both change hardware
>under MGS/MDS and reassign an OSS ip. Just the same, I added a brand
>new 10GE network to the system, which was the reason for MDS hardware
>change.

Note that in Lustre 2.4 there is a "lctl replace_nids" command that
allows you to change the NIDs without running --writeconf.  That doesn't
help you now, but possibly in the future.

>I ran tunefs.lustre --writeconf as per chapter 14.4 in Lustre Manual,
>and everything mounts fine. Log regeneration apparently works, since
>it seems to do something, but exceedingly slowly. Disks show all but
>no activity, CPU utilization is zero across the board, and memory
>should be no issue. I believe it works, but currently it seems the
>1,5*10^9 files (some 55 TiB of data) won't be indexed in a week. My
>boss isn't happy when I can't even predict how long this will take, or
>even say for sure that it really works.

The --writeconf information is at most a few kB and should only take
seconds to complete.  What "reindexing" operation are you referencing?
It should be possible to mount the filesystem immediately (MGS first,
then MDS and OSSes) after running --writeconf.

You didn't really explain what is preventing you from using the filesystem,
since you said it mounted properly?

>Two questions: is there a way to know how fast it is progressing
>and/or where it is at, or even that it really works, and is there a
>way to speed up whatever is slowing it down? Seems all diagnostic
>/proc entries have been removed from 2.3.  I have tried mounting the
>Lustre partitions with -o nobarrier (yes, I know it's dangerous, but
>I'd really need to speed things up) but I don't know if that does
>anything at all.

I doubt that the "-o nobarrier" is helping you much.

>We run Centos 6.x in Lustre servers, where Lustre has been installed
>from rpm's from Whamcloud/Intel build bot, and Ubuntu 10.04 in clients
>with hand compiled kernel and Lustre. One MGC/MGS with twelve 15k-RPM
>SAS disks in RAID-10 as MDT that is all but empty, and six variously
>build RAID-6's in SAS-attached shelves in three OSS's.
>
>ATdhvaannkcse for any help,
>
>-- 
>     Olli Lounela
>     IT specialist and administrator
>     DNA sequencing and genomics
>     Institute of Biotechnology
>     University of Helsinki
>
>_______________________________________________
>Lustre-discuss mailing list
>Lustre-discuss at lists.lustre.org
>http://lists.lustre.org/mailman/listinfo/lustre-discuss
>


Cheers, Andreas
-- 
Andreas Dilger

Lustre Software Architect
Intel High Performance Data Division