[Lustre-discuss] How to detect process owner on client

[Michael Kluge]

But it does not give you PIDs or user names? Or is there a way to find 
these with standard lustre tools?

Michael

Am 11.02.2011 17:34, schrieb Andreas Dilger:
> On 2011-02-10, at 23:18, Michael Kluge wrote:
>> I am not aware of any possibility to map the current statistics in /proc
>> to UIDs. But I might be wrong. We had a script like this a while ago
>> which did not kill the I/O intensive processes but told us the PIDs.
>>
>> What we did is collecting for ~30 seconds the number of I/O operations
>> per node via /proc on all nodes. Then we attached an strace process to
>> each process on nodes with heavy I/O load. This strace intercepted only
>> the I/O calls and wrote one log file per process. If this strace is
>> running for the same amount of time for each process on a host, you just
>> need to sort the log files for size.
>
> On the OSS and MDS nodes there are per-client statistics that allow this kind of tracking.  They can be seen in /proc/fs/lustre/obdfilter/*/exports/*/stats for detailed information (e.g. broken down by RPC type, bytes read/written), or /proc/fs/lustre/ost/OSS/*/req_history to just get a dump of the recent RPCs sent by each client.
>
> A little script was discussed in the thread "How to determine which lustre clients are loading filesystem" (2010-07-08):
>
>> Another way that I heard some sites were doing this is to use the "rpc history".  They may already have a script to do this, but the basics are below:
>>
>> oss# lctl set_param ost.OSS.*.req_buffer_history_max=10240
>> {wait a few seconds to collect some history}
>> oss# lctl get_param ost.OSS.*.req_history
>>
>> This will give you a list of the past (up to) 10240 RPCs for the "ost_io" RPC service, which is what you are observing the high load on:
>>
>> 3436037:192.168.20.1 at tcp:12345-192.168.20.159 at tcp:x1340648957534353:448:Complete:1278612656:0s(-6s) opc 3
>> 3436038:192.168.20.1 at tcp:12345-192.168.20.159 at tcp:x1340648957536190:448:Complete:1278615489:1s(-41s) opc 3
>> 3436039:192.168.20.1 at tcp:12345-192.168.20.159 at tcp:x1340648957536193:448:Complete:1278615490:0s(-6s) opc 3
>>
>> This output is in the format:
>>
>> identifier:target_nid:source_nid:rpc_xid:rpc_size:rpc_status:arrival_time:service_time(deadline) opcode
>>
>> Using some shell scripting, one can find the clients sending the most RPC requests:
>>
>> oss# lctl get_param ost.OSS.*.req_history | tr ":" " " | cut -d" " -f3,9,10 | sort | uniq -c | sort -nr | head -20
>>
>>
>>    3443 12345-192.168.20.159 at tcp opc 3
>>    1215 12345-192.168.20.157 at tcp opc 3
>>     121 12345-192.168.20.157 at tcp opc 4
>>
>> This will give you a sorted list of the top 20 clients that are sending the most RPCs to the ost and ost_io services, along with the operation being done (3 = OST_READ, 4 = OST_WRITE, etc. see lustre/include/lustre/lustre_idl.h).
>
>
>> Am Donnerstag, den 10.02.2011, 21:16 -0600 schrieb Satoshi Isono:
>>> Dear members,
>>>
>>> I am looking into the way which can detect userid or jobid on the Lustre client. Assumed the following condition;
>>>
>>> 1) Any users run any jobs through scheduler like PBS Pro, LSF or SGE.
>>> 2) A users processes occupy Lustre I/O.
>>> 3) Some Lustre servers (MDS?/OSS?) can detect high I/O stress on each server.
>>> 4) But Lustre server cannot make the mapping between jobid/userid and Lustre I/O processes having heavy stress, because there aren't userid on Lustre servers.
>>> 5) I expect that Lustre can monitor and can make the mapping.
>>> 6) If possible for (5), we can make a script which launches scheduler command like as qdel.
>>> 7) Heavy users job will be killed by job scheduler.
>>>
>>> I want (5) for Lustre capability, but I guess current Lustre 1.8 cannot perform (5). On the other hand, in order to map Lustre process to userid/jobid, are there any ways using like rpctrace or nid stats? Can you please your advice or comments?
>>>
>>> Regards,
>>> Satoshi Isono
>>> _______________________________________________
>>> Lustre-discuss mailing list
>>> Lustre-discuss at lists.lustre.org
>>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>>>
>>
>> --
>>
>> Michael Kluge, M.Sc.
>>
>> Technische Universität Dresden
>> Center for Information Services and
>> High Performance Computing (ZIH)
>> D-01062 Dresden
>> Germany
>>
>> Contact:
>> Willersbau, Room A 208
>> Phone:  (+49) 351 463-34217
>> Fax:    (+49) 351 463-37773
>> e-mail: michael.kluge at tu-dresden.de
>> WWW:    http://www.tu-dresden.de/zih
>> _______________________________________________
>> Lustre-discuss mailing list
>> Lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>
>
> Cheers, Andreas
> --
> Andreas Dilger
> Principal Engineer
> Whamcloud, Inc.
>
>
>
>


-- 
Michael Kluge, M.Sc.

Technische Universität Dresden
Center for Information Services and
High Performance Computing (ZIH)
D-01062 Dresden
Germany

Contact:
Willersbau, Room WIL A 208
Phone:  (+49) 351 463-34217
Fax:    (+49) 351 463-37773
e-mail: michael.kluge at tu-dresden.de
WWW:    http://www.tu-dresden.de/zih