[Lustre-discuss] Lustre::LFS + Lustre::Info (inc. lustre-info.pl) available on the CPAN

[Frederik Ferner]

Hi Adrian,

thanks for sharing these with us.

Adrian Ulrich wrote:
> I uploaded two lustre-related modules to the CPAN:
> 
> #1: Lustre::Info provides easy access to information located
>     at /proc/fs/lustre, it also comes with a 'performance monitoring'
>     script called 'lustre-info.pl'

I did have a bit of a play with the lustre-info.pl script on our test 
file system and it seems to work nicely. If you've got a lot of OSTs on 
your server you need a wide monitor for some of the options like 
--monitor=ost-patterns for all OSTs...

We are currently running Lustre 1.6.7.2 (+ a few patches) on our OSTs, 
in case this makes a difference for my issues below.

[snip]
> Examples and details:
> 
> Lustre::Info and lustre-info.pl
> ---------------------------------------
[snip]
> The module also includes a script called 'lustre-info.pl' that can
> be used to gather some live performance statistics:
> 
> Use `--ost-stats' to get a quick overview on what's going on:
> $ lustre-info.pl --ost-stats

In our case this looks like this (on a very quiet file system):
>   play01-OST0000 (@ /dev/sdb) :  write=   0.000 MB/s, read=   0.000 MB/s, create=  0.0 R/s, destroy=  0.0 R/s, setattr=  0.0 R/s, preprw=  0.0 R/s
>   play01-OST0001 (@ /dev/sdc) :  write=   0.000 MB/s, read=   0.000 MB/s, create=  0.0 R/s, destroy=  0.0 R/sUse of uninitialized value in division (/) at /usr/local/bin/lustre-info.pl line 187.
> , setattr=  0.0 R/s, preprw=  0.0 R/s
>   play01-OST0002 (@ /dev/sdd) :  write=   0.000 MB/s, read=   0.000 MB/s, create=  0.0 R/s, destroy=  0.0 R/s, setattr=  0.0 R/s, preprw=  0.0 R/s
>   play01-OST0003 (@ /dev/sde) :  write=   0.000 MB/s, read=   0.000 MB/s, create=  0.0 R/s, destroy=  0.0 R/s, setattr=  0.0 R/s, preprw=  0.0 R/s
>   play01-OST0004 (@ /dev/sdf) :  write=   0.000 MB/s, read=   0.000 MB/s, create=  0.0 R/s, destroy=  0.0 R/sUse of uninitialized value in division (/) at /usr/local/bin/lustre-info.pl line 187.
> , setattr=  0.0 R/s, preprw=  0.0 R/s
>   play01-OST0005 (@ /dev/sdg) :  write=   0.000 MB/s, read=   0.000 MB/s, create=  0.0 R/s, destroy=  0.0 R/sUse of uninitialized value in division (/) at /usr/local/bin/lustre-info.pl line 187.
> , setattr=  0.0 R/s, preprw=  0.0 R/s


Note the 'Use of uninitialized value in division...' errors. Looking at 
the code it seems the value for 'setattr' is missing from the stats file 
for some of our OSTs. Looking at the stats file, indeed the setattr line 
is missing for some OSTs.

Has anyone seen this before? What could have caused this?

> You can also get client<->ost details via `--monitor=MODE'
> 
> $ lustre-info.pl --monitor=ost --as-list  # this will only show clients where read+write >= 1MB/s
>> client nid       | lustre1-OST0006    | lustre1-OST000e    | lustre1-OST0016    | lustre1-OST001e    | +++ TOTALS +++ (MB/s)
> 10.201.46.25 at o2ib  | r=   0.0, w=   0.0 | r=   0.0, w=   0.0 | r=   0.0, w=   0.0 | r=   0.0, w=   1.1 | read=   0.0, write=   1.1
> 10.201.47.27 at o2ib  | r=   0.0, w=   0.0 | r=   0.0, w=   1.2 | r=   0.0, w=   2.0 | r=   0.0, w=   0.0 | read=   0.0, write=   3.2

'lustre-info.pl --monitor=io-size' seems to sit at "collecting data, 
please wait..." for a very long time until I killed it, I have not had 
the time to debug this yet.

Kind regards,
Frederik
-- 
Frederik Ferner
Computer Systems Administrator		phone: +44 1235 77 8624
Diamond Light Source Ltd.		mob:   +44 7917 08 5110
(Apologies in advance for the lines below. Some bits are a legal
requirement and I have no control over them.)