[lustre-discuss] Lustre Monitoring metrics

[Andreas Dilger]

On Feb 24, 2019, at 18:13, Masudul Hasan Masud Bhuiyan <masud.hasan at nevada.unr.edu> wrote:
> 
> Thank you very much. Unfortunately, as I am doing my research on  3rd party sc cluster, I don't have  access to these tools. Also, I am kind of developing something like LMT. So, it will be great if I can get some idea about these  metrics. I will be grateful if you can give me some direction about where can I get idea about these metrics. 

There is some documentation about the various stats files in the Lustre Operations Manual.  You could also look at lustre/utils/{llstat,llobdstat} to get some idea how they interpret the stats.

However, while I don't want to be negative about your work, my suggestion would be to spend your time to improve LMT or IML or even make "llstat" and "llobdstat" more useful, rather than develop yet another tool that implements some marginal new functionality but is not used by very many people.

This is the true nature of open source - leveraging and improving work that others have done, and contributing something of value back to the community.  That way you don't spend a lot of time to implement basic infrastructure, but instead you can add something interesting or improve the existing functionality, and your development efforts will be used in the future by a wide variety of users.  

You also get (IMHO) more valuable development experience, because very few real-world projects are "invent everything from scratch", and rather "improve some existing code", and becoming adept at understanding existing code and improving it is valuable in the industry.

Cheers, Andreas

> On Sun, Feb 24, 2019 at 11:06 AM Andreas Dilger <adilger at whamcloud.com> wrote:
>> Probably for a new user it doesn't make sense to use the Lustre stats in /proc directly. There are a number of different tools that present these stats in a more useful manner, such as IML (GUI Web front end), LMT, lltop, etc. 
>> 
>> Cheers, Andreas
>> 
>> On Feb 24, 2019, at 02:09, Masudul Hasan Masud Bhuiyan <masud.hasan at nevada.unr.edu> wrote:
>> 
>>> Hi,
>>> I am very new to Lustre and  I am trying to get idea about the lustre monitoring system. I have collected the stats from "/proc/fs/lustre". Unfortunately I couldn't understand what does these data are actually mean. I have tried to go through Lustre documentation and user guide to get understanding of the data. But had no luck.
>>> 
>>> I will be grateful if you can help me to understand these log files. This is a sample of stats for metadat server. But I am not sure how can I interpret these stat. what does "mds_getattr"/mds_get_root/mds_quotactl means?
>>> 
>>> Fri Feb 8 00:52:13 2019
>>> snapshot_time 1549615933.091769380 secs.nsecs
>>> req_waittime 540323978 samples [usec] 58 689541437 1767362743217 12488011030933825609
>>> req_active 540323988 samples [reqs] 1 142007 60970174330 3247243705183378
>>> mds_getattr 168893 samples [usec] 82 165621 390916699 1522734684023
>>> mds_close 110379799 samples [usec] 64 1231540 17796322782 62011789734542
>>> mds_readpage 1236125 samples [usec] 245 10402461 10527744335 1995909318950381
>>> mds_connect 6 samples [usec] 186 868 1900 969558
>>> mds_get_root 1 samples [usec] 85 85 85 7225
>>> mds_statfs 795609 samples [usec] 67 1109562 226674777 4390097174469
>>> mds_sync 6769 samples [usec] 361624 689541437 80448645395 7168190473271866793
>>> mds_quotactl 1449 samples [usec] 69 18051 551878 2621695252
>>> mds_getxattr 1886 samples [usec] 93 13792 419449 525089597
>>> mds_hsm_state_set 234556 samples [usec] 77 893155 96605623 1595746930715
>>> ldlm_cancel 115821035 samples [usec] 58 5627198 1149359370420 1270415775256810056
>>> obd_ping 23932 samples [usec] 87 34096 7061304 6296645618
>>> seq_query 38 samples [usec] 87 767 6885 1688653
>>> fld_query 3 samples [usec] 84 370 789 256181
>>> 
>>> Sincerely yours,
>>> Masudul Hasan Masud
>>> 

Cheers, Andreas
---
Andreas Dilger
Principal Lustre Architect
Whamcloud