[lustre-discuss] osc and llite stats files are not exported to clients

[Chris Horn]

The stats files are in /sys/kernel/debug. Elevated privileges are required to read them. See https://jira.whamcloud.com/browse/LU-11850

# find /sys/kernel/debug/lustre -name stats
/sys/kernel/debug/lustre/ldlm/services/ldlm_canceld/stats
/sys/kernel/debug/lustre/ldlm/services/ldlm_cbd/stats
/sys/kernel/debug/lustre/ldlm/namespaces/lustre-OST0001-osc-MDT0000/pool/stats
/sys/kernel/debug/lustre/ldlm/namespaces/lustre-OST0000-osc-MDT0000/pool/stats
/sys/kernel/debug/lustre/ldlm/namespaces/lustre-OST0001-osc-ffff92312e7e2800/pool/stats
/sys/kernel/debug/lustre/ldlm/namespaces/lustre-OST0000-osc-ffff92312e7e2800/pool/stats
/sys/kernel/debug/lustre/ldlm/namespaces/lustre-MDT0000-mdc-ffff92312e7e2800/pool/stats
/sys/kernel/debug/lustre/ldlm/namespaces/lustre-MDT0000-lwp-OST0001/pool/stats
/sys/kernel/debug/lustre/ldlm/namespaces/filter-lustre-OST0001_UUID/pool/stats
/sys/kernel/debug/lustre/ldlm/namespaces/lustre-MDT0000-lwp-OST0000/pool/stats
/sys/kernel/debug/lustre/ldlm/namespaces/filter-lustre-OST0000_UUID/pool/stats
/sys/kernel/debug/lustre/ldlm/namespaces/lustre-MDT0000-lwp-MDT0000/pool/stats
/sys/kernel/debug/lustre/ldlm/namespaces/mdt-lustre-MDT0000_UUID/pool/stats
/sys/kernel/debug/lustre/ldlm/namespaces/MGC192.168.2.30 at tcp/pool/stats
/sys/kernel/debug/lustre/ldlm/namespaces/MGS/pool/stats
/sys/kernel/debug/lustre/llite/lustre-ffff92312e7e2800/stats
/sys/kernel/debug/lustre/mdc/lustre-MDT0000-mdc-ffff92312e7e2800/stats
/sys/kernel/debug/lustre/ost/OSS/ost_out/stats
/sys/kernel/debug/lustre/ost/OSS/ost_seq/stats
/sys/kernel/debug/lustre/ost/OSS/ost_io/stats
/sys/kernel/debug/lustre/ost/OSS/ost_create/stats
/sys/kernel/debug/lustre/ost/OSS/ost/stats
/sys/kernel/debug/lustre/osc/lustre-OST0001-osc-ffff92312e7e2800/stats
/sys/kernel/debug/lustre/osc/lustre-OST0000-osc-ffff92312e7e2800/stats
/sys/kernel/debug/lustre/lwp/lustre-MDT0000-lwp-OST0001/stats
/sys/kernel/debug/lustre/lwp/lustre-MDT0000-lwp-OST0000/stats
/sys/kernel/debug/lustre/lwp/lustre-MDT0000-lwp-MDT0000/stats
/sys/kernel/debug/lustre/osp/lustre-OST0001-osc-MDT0000/stats
/sys/kernel/debug/lustre/osp/lustre-OST0000-osc-MDT0000/stats
/sys/kernel/debug/lustre/mds/MDS/mdt_io/stats
/sys/kernel/debug/lustre/mds/MDS/mdt_fld/stats
/sys/kernel/debug/lustre/mds/MDS/mdt_seqm/stats
/sys/kernel/debug/lustre/mds/MDS/mdt_seqs/stats
/sys/kernel/debug/lustre/mds/MDS/mdt_out/stats
/sys/kernel/debug/lustre/mds/MDS/mdt_setattr/stats
/sys/kernel/debug/lustre/mds/MDS/mdt_readpage/stats
/sys/kernel/debug/lustre/mds/MDS/mdt/stats
/sys/kernel/debug/lustre/mgs/MGS/mgs/stats

Chris Horn

From: lustre-discuss <lustre-discuss-bounces at lists.lustre.org> on behalf of U Sawangwit <u.sawangwit at gmail.com>
Date: Thursday, July 25, 2019 at 9:58 PM
To: "Degremont, Aurelien" <degremoa at amazon.com>
Cc: "lustre-discuss at lists.lustre.org" <lustre-discuss at lists.lustre.org>
Subject: Re: [lustre-discuss] osc and llite stats files are not exported to clients

Hi

    Thank you very much for your reply.

I looked in the /sys but also did not find the osc/*/stats or llite/*/stats files.

I understand what you are saying about the lctl becoming the new standard way.
But I just want to make sure that this is not due to sth. wrong with my system or setup.

FYI, I am using CentOS 7.6 kernel 3.10.0-957.el7.x86_64 and *_lustre* for my clients and
lustre servers with Lustre version 2.12.0 using IB FDR as the LNET.

Would the missing stats files in /sys/fs/lustre or /proc/fs/lustre considered normal?

 Any monitoring tool you would suggest to use instead of lustre gmond_python_modules in order
to avoid reading stats files? My initial review of LMT3 also indicate that it too reading data from
stats file rather than using lctl.

Cheers,
Utane

---------------------------------------------------------------------------------------
Utane Sawangwit, PhD.
Research Astronomer
National Astronomical Research Institute of Thailand (NARIT)
Sirindhon AstroPark, 260, Mue 4, Donkaew
Mae-Rim, Chiang Mai, 50180, Thailand
----------------------------------------------------------------------------------------
ดร. อุเทน แสวงวิทย์
นักวิจัยชำนาญการ
สถาบันวิจัยดาราศาสตร์แห่งชาติ(องค์การมหาชน)
อุทยานดาราศาสตร์สิรินธร 260 หมู่ 4 ต.ดอนแก้ว
อ.แม่ริม จ.เชียงใหม่ 50180
---------------------------------------------------------------------------------------



On Thu, 25 Jul 2019 at 22:05, Degremont, Aurelien <degremoa at amazon.com<mailto:degremoa at amazon.com>> wrote:
Lustre is transitionning /proc files to /sys per kernel developer requests.

Instead of chasing where these files are located now depending on your Lustre version, 'lctl get_param' is the official way to read them.
This command will figure out where to properly read the files.

De : lustre-discuss <lustre-discuss-bounces at lists.lustre.org<mailto:lustre-discuss-bounces at lists.lustre.org>> au nom de U Sawangwit <u.sawangwit at gmail.com<mailto:u.sawangwit at gmail.com>>
Date : jeudi 25 juillet 2019 à 13:15
À : "lustre-discuss at lists.lustre.org<mailto:lustre-discuss at lists.lustre.org>" <lustre-discuss at lists.lustre.org<mailto:lustre-discuss at lists.lustre.org>>
Objet : [lustre-discuss] osc and llite stats files are not exported to clients

Dear all,

   I am having trouble with lustre monitoring using Ganglia gmond_python_modules
(https://github.com/ganglia/gmond_python_modules/tree/master/lustre).

    Upon closer inspections, I found that the stats files in /proc/fs/lustre/osc/*/
and /proc/fs/lustre/llite are missing (among other similar files in these directories).

     However, lctl get_param osc.*.stats and lctl get_param llite.*.stats return
parameters and values as you would expect. The stats files are what the
gmond_python_modules wants hence the problem monitoring with Ganglia.

    After many searching around, could not figure out what I could have done to
have stats export normally to clients and remounting did not help. On MDS and OSS,
I can see the obdfiler/*/export/*/stats or mdt/*/export/*/stats

    The lustre manual also said that the exporting of these info to client should be by
default.

  Thank you in advance.

Cheers,
Utane

---------------------------------------------------------------------------------------
Utane Sawangwit, PhD.
Research Astronomer
National Astronomical Research Institute of Thailand (NARIT)
Sirindhon AstroPark, 260, Mue 4, Donkaew
Mae-Rim, Chiang Mai, 50180, Thailand
----------------------------------------------------------------------------------------
ดร. อุเทน แสวงวิทย์
นักวิจัยชำนาญการ
สถาบันวิจัยดาราศาสตร์แห่งชาติ(องค์การมหาชน)
อุทยานดาราศาสตร์สิรินธร 260 หมู่ 4 ต.ดอนแก้ว
อ.แม่ริม จ.เชียงใหม่ 50180
---------------------------------------------------------------------------------------

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20190726/822bd02e/attachment-0001.html>