[Lustre-discuss] How to detect process owner on client

Andreas Dilger adilger at whamcloud.com
Fri Feb 11 08:34:25 PST 2011


On 2011-02-10, at 23:18, Michael Kluge wrote:
> I am not aware of any possibility to map the current statistics in /proc
> to UIDs. But I might be wrong. We had a script like this a while ago
> which did not kill the I/O intensive processes but told us the PIDs. 
> 
> What we did is collecting for ~30 seconds the number of I/O operations
> per node via /proc on all nodes. Then we attached an strace process to
> each process on nodes with heavy I/O load. This strace intercepted only
> the I/O calls and wrote one log file per process. If this strace is
> running for the same amount of time for each process on a host, you just
> need to sort the log files for size.

On the OSS and MDS nodes there are per-client statistics that allow this kind of tracking.  They can be seen in /proc/fs/lustre/obdfilter/*/exports/*/stats for detailed information (e.g. broken down by RPC type, bytes read/written), or /proc/fs/lustre/ost/OSS/*/req_history to just get a dump of the recent RPCs sent by each client.

A little script was discussed in the thread "How to determine which lustre clients are loading filesystem" (2010-07-08):

> Another way that I heard some sites were doing this is to use the "rpc history".  They may already have a script to do this, but the basics are below:
> 
> oss# lctl set_param ost.OSS.*.req_buffer_history_max=10240
> {wait a few seconds to collect some history}
> oss# lctl get_param ost.OSS.*.req_history
> 
> This will give you a list of the past (up to) 10240 RPCs for the "ost_io" RPC service, which is what you are observing the high load on:
> 
> 3436037:192.168.20.1 at tcp:12345-192.168.20.159 at tcp:x1340648957534353:448:Complete:1278612656:0s(-6s) opc 3
> 3436038:192.168.20.1 at tcp:12345-192.168.20.159 at tcp:x1340648957536190:448:Complete:1278615489:1s(-41s) opc 3
> 3436039:192.168.20.1 at tcp:12345-192.168.20.159 at tcp:x1340648957536193:448:Complete:1278615490:0s(-6s) opc 3
> 
> This output is in the format:
> 
> identifier:target_nid:source_nid:rpc_xid:rpc_size:rpc_status:arrival_time:service_time(deadline) opcode
> 
> Using some shell scripting, one can find the clients sending the most RPC requests:
> 
> oss# lctl get_param ost.OSS.*.req_history | tr ":" " " | cut -d" " -f3,9,10 | sort | uniq -c | sort -nr | head -20
> 
> 
>   3443 12345-192.168.20.159 at tcp opc 3
>   1215 12345-192.168.20.157 at tcp opc 3
>    121 12345-192.168.20.157 at tcp opc 4
> 
> This will give you a sorted list of the top 20 clients that are sending the most RPCs to the ost and ost_io services, along with the operation being done (3 = OST_READ, 4 = OST_WRITE, etc. see lustre/include/lustre/lustre_idl.h).


> Am Donnerstag, den 10.02.2011, 21:16 -0600 schrieb Satoshi Isono: 
>> Dear members,
>> 
>> I am looking into the way which can detect userid or jobid on the Lustre client. Assumed the following condition;
>> 
>> 1) Any users run any jobs through scheduler like PBS Pro, LSF or SGE.
>> 2) A users processes occupy Lustre I/O.
>> 3) Some Lustre servers (MDS?/OSS?) can detect high I/O stress on each server.
>> 4) But Lustre server cannot make the mapping between jobid/userid and Lustre I/O processes having heavy stress, because there aren't userid on Lustre servers.
>> 5) I expect that Lustre can monitor and can make the mapping.
>> 6) If possible for (5), we can make a script which launches scheduler command like as qdel.
>> 7) Heavy users job will be killed by job scheduler.
>> 
>> I want (5) for Lustre capability, but I guess current Lustre 1.8 cannot perform (5). On the other hand, in order to map Lustre process to userid/jobid, are there any ways using like rpctrace or nid stats? Can you please your advice or comments?
>> 
>> Regards,
>> Satoshi Isono
>> _______________________________________________
>> Lustre-discuss mailing list
>> Lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>> 
> 
> -- 
> 
> Michael Kluge, M.Sc.
> 
> Technische Universität Dresden
> Center for Information Services and
> High Performance Computing (ZIH)
> D-01062 Dresden
> Germany
> 
> Contact:
> Willersbau, Room A 208
> Phone:  (+49) 351 463-34217
> Fax:    (+49) 351 463-37773
> e-mail: michael.kluge at tu-dresden.de
> WWW:    http://www.tu-dresden.de/zih
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss


Cheers, Andreas
--
Andreas Dilger 
Principal Engineer
Whamcloud, Inc.






More information about the lustre-discuss mailing list