[lustre-discuss] Metrics Gathering into ELK stack

[Nathan Smith]

Sid Young writes:
> Are there any reliable/solid Lustre Specific metrics tools that can
> push data to ELK OR Can generate JSON strings of metrics I can push
> into more bespoke monitoring solutions...
>
> I am more interested in I/O metrics from the lustre side of things as
> I can gather Disk/CPU/memory metrics with Metricbeat as needed already
> in the legacy HPC.

Lustre Job Stats [0] may provide some or all of what you are looking
for. The job stats data are output in yaml format, which are fairly easy
to transform to inputs for Elasticsearch (or more generally as JSON for
other systems). In our case we used Python. The yaml inputs are imported
as Python dictionaries, which can then be used directly as input data to
the Elasticsearch Python module.

We happen to add some additional entries to the dictionary objects
before submitting them to Elasticsearch. We also find it useful to clear
the job stats after each read to simplify analysis. I do not have any
publicly-available code to share, but I think the Python implementation
is not overly complex.

[0] https://doc.lustre.org/lustre_manual.xhtml#dbdoclet.jobstats

-- 
Nathan Smith
Research Systems Engineer
Advanced Computing Center
Oregon Health & Science University