<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Apr 25, 2017 at 7:41 PM, Oucharek, Doug S <span dir="ltr"><<a target="_blank" href="mailto:doug.s.oucharek@intel.com">doug.s.oucharek@intel.com</a>></span> wrote:<br><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote">That specific message happens when the \u201cmagic\u201d u32 field at the start of a message does not match what we are expecting. We do check if the message was transmitted as a different endian from us so when you see this error, we assume that message has been corrupted or the sender is using an invalid magic value. I don\u2019t believe this value has changed in the history of the LND so this is more likely corruption of some sort.<br></blockquote><div><br>OT: this information should probably be added to LU-2977 which specifically includes the question: What does "consumer defined fatal error" mean and why is this connection rejected?<br><br></div><div><br> </div><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote">
<span class="gmail-HOEnZb"><font color="#888888"><br>
Doug<br>
</font></span><div class="gmail-HOEnZb"><div class="gmail-h5"><br>
> On Apr 25, 2017, at 2:29 AM, Dilger, Andreas <<a href="mailto:andreas.dilger@intel.com">andreas.dilger@intel.com</a>> wrote:<br>
><br>
> I'm not an LNet expert, but I think the critical issue to focus on is:<br>
><br>
> Lustre: Lustre: Build Version: 2.5.3-RC1--PRISTINE-2.6.32-<wbr>573.el6.x86_64<br>
> LNet: Added LNI 172.23.54.51@o2ib [8/256/0/180]<br>
> LNetError: 2878:0:(o2iblnd_cb.c:2587:<wbr>kiblnd_rejected()) 172.23.55.211@o2ib rejected: consumer defined fatal error<br>
><br>
> This means that the LND didn't connect at startup time, but I don't know what the cause is.<br>
> The error that generates this message is IB_CM_REJ_CONSUMER_DEFINED, but I don't know enough about IB to tell you what that means. Some of the later code is checking for mismatched Lustre versions, but it doesn't even get that far.<br>
><br>
> Cheers, Andreas<br>
><br>
>> On Apr 25, 2017, at 02:21, Strikwerda, Ger <<a href="mailto:g.j.c.strikwerda@rug.nl">g.j.c.strikwerda@rug.nl</a>> wrote:<br>
>><br>
>> Hi Raj,<br>
>><br>
>> [root@pg-gpu01 ~]# lustre_rmmod<br>
>><br>
>> [root@pg-gpu01 ~]# modprobe -v lustre<br>
>> insmod /lib/modules/2.6.32-642.6.2.<wbr>el6.x86_64/weak-updates/<wbr>kernel/net/lustre/libcfs.ko<br>
>> insmod /lib/modules/2.6.32-642.6.2.<wbr>el6.x86_64/weak-updates/<wbr>kernel/fs/lustre/lvfs.ko<br>
>> insmod /lib/modules/2.6.32-642.6.2.<wbr>el6.x86_64/weak-updates/<wbr>kernel/net/lustre/lnet.ko networks=o2ib(ib0)<br>
>> insmod /lib/modules/2.6.32-642.6.2.<wbr>el6.x86_64/weak-updates/<wbr>kernel/fs/lustre/obdclass.ko<br>
>> insmod /lib/modules/2.6.32-642.6.2.<wbr>el6.x86_64/weak-updates/<wbr>kernel/fs/lustre/ptlrpc.ko<br>
>> insmod /lib/modules/2.6.32-642.6.2.<wbr>el6.x86_64/weak-updates/<wbr>kernel/fs/lustre/fid.ko<br>
>> insmod /lib/modules/2.6.32-642.6.2.<wbr>el6.x86_64/weak-updates/<wbr>kernel/fs/lustre/mdc.ko<br>
>> insmod /lib/modules/2.6.32-642.6.2.<wbr>el6.x86_64/weak-updates/<wbr>kernel/fs/lustre/osc.ko<br>
>> insmod /lib/modules/2.6.32-642.6.2.<wbr>el6.x86_64/weak-updates/<wbr>kernel/fs/lustre/lov.ko<br>
>> insmod /lib/modules/2.6.32-642.6.2.<wbr>el6.x86_64/weak-updates/<wbr>kernel/fs/lustre/lustre.ko<br>
>><br>
>> dmesg:<br>
>><br>
>> LNet: HW CPU cores: 24, npartitions: 4<br>
>> alg: No test for crc32 (crc32-table)<br>
>> alg: No test for adler32 (adler32-zlib)<br>
>> alg: No test for crc32 (crc32-pclmul)<br>
>> Lustre: Lustre: Build Version: 2.5.3-RC1--PRISTINE-2.6.32-<wbr>573.el6.x86_64<br>
>> LNet: Added LNI 172.23.54.51@o2ib [8/256/0/180]<br>
>><br>
>> But no luck,<br>
>><br>
>> [root@pg-gpu01 ~]# lctl ping 172.23.55.211@o2ib<br>
>> failed to ping 172.23.55.211@o2ib: Input/output error<br>
>><br>
>> [root@pg-gpu01 ~]# mount /home<br>
>> mount.lustre: mount 172.23.55.211@o2ib:172.23.55.<wbr>212@o2ib:/pghome01 at /home failed: Input/output error<br>
>> Is the MGS running?<br>
>><br>
>><br>
>><br>
>><br>
>><br>
>><br>
>> On Mon, Apr 24, 2017 at 7:53 PM, Raj <<a href="mailto:rajgautam@gmail.com">rajgautam@gmail.com</a>> wrote:<br>
>> Yes, this is strange. Normally, I have seen that credits mismatch results this scenario but it doesn't look like this is the case.<br>
>><br>
>> You wouldn't want to put mgs into capture debug messages as there will be a lot of data.<br>
>><br>
>> I guess you already tried removing the lustre drivers and adding it again ?<br>
>> lustre_rmmod<br>
>> modprobe -v lustre<br>
>><br>
>> And check dmesg for any errors...<br>
>><br>
>><br>
>> On Mon, Apr 24, 2017 at 12:43 PM Strikwerda, Ger <<a href="mailto:g.j.c.strikwerda@rug.nl">g.j.c.strikwerda@rug.nl</a>> wrote:<br>
>> Hi Raj,<br>
>><br>
>> When i do a lctl ping on a MGS server i do not see any logs at all. Also not when i do a sucessfull ping from a working node. Is there a way to verbose the Lustre logging to see more detail on the LNET level?<br>
>><br>
>> It is very strange that a rebooted node is able to lctl ping compute nodes, but fails to lctl ping metadata and storage nodes.<br>
>><br>
>><br>
>><br>
>><br>
>> On Mon, Apr 24, 2017 at 7:35 PM, Raj <<a href="mailto:rajgautam@gmail.com">rajgautam@gmail.com</a>> wrote:<br>
>> Ger,<br>
>> It looks like default configuration of lustre.<br>
>><br>
>> Do you see any error message on the MGS side while you are doing lctl ping from the rebooted clients?<br>
>> On Mon, Apr 24, 2017 at 12:27 PM Strikwerda, Ger <<a href="mailto:g.j.c.strikwerda@rug.nl">g.j.c.strikwerda@rug.nl</a>> wrote:<br>
>> Hi Eli,<br>
>><br>
>> Nothing can be mounted on the Lustre filesystems so the output is:<br>
>><br>
>> [root@pg-gpu01 ~]# lfs df /home/ger/<br>
>> [root@pg-gpu01 ~]#<br>
>><br>
>> Empty..<br>
>><br>
>><br>
>><br>
>> On Mon, Apr 24, 2017 at 7:24 PM, E.S. Rosenberg <<a href="mailto:esr@cs.huji.ac.il">esr@cs.huji.ac.il</a>> wrote:<br>
>><br>
>><br>
>> On Mon, Apr 24, 2017 at 8:19 PM, Strikwerda, Ger <<a href="mailto:g.j.c.strikwerda@rug.nl">g.j.c.strikwerda@rug.nl</a>> wrote:<br>
>> Hallo Eli,<br>
>><br>
>> Logfile/syslog on the client-side:<br>
>><br>
>> Lustre: Lustre: Build Version: 2.5.3-RC1--PRISTINE-2.6.32-<wbr>573.el6.x86_64<br>
>> LNet: Added LNI 172.23.54.51@o2ib [8/256/0/180]<br>
>> LNetError: 2878:0:(o2iblnd_cb.c:2587:<wbr>kiblnd_rejected()) 172.23.55.211@o2ib rejected: consumer defined fatal error<br>
>><br>
>> lctl df /path/to/some/file<br>
>><br>
>> gives nothing useful? (the second one will dump *a lot*)<br>
>><br>
>><br>
>><br>
>><br>
>> On Mon, Apr 24, 2017 at 7:16 PM, E.S. Rosenberg <<a href="mailto:esr%2Blustre@mail.hebrew.edu">esr+lustre@mail.hebrew.edu</a>> wrote:<br>
>><br>
>><br>
>> On Mon, Apr 24, 2017 at 8:13 PM, Strikwerda, Ger <<a href="mailto:g.j.c.strikwerda@rug.nl">g.j.c.strikwerda@rug.nl</a>> wrote:<br>
>> Hi Raj (and others),<br>
>><br>
>> In which file should i state the credits/peer_credits stuff?<br>
>><br>
>> Perhaps relevant config-files:<br>
>><br>
>> [root@pg-gpu01 ~]# cd /etc/modprobe.d/<br>
>><br>
>> [root@pg-gpu01 modprobe.d]# ls<br>
>> anaconda.conf blacklist-kvm.conf dist-alsa.conf dist-oss.conf ib_ipoib.conf lustre.conf openfwwf.conf<br>
>> blacklist.conf blacklist-nouveau.conf dist.conf freeipmi-modalias.conf ib_sdp.conf mlnx.conf truescale.conf<br>
>><br>
>> [root@pg-gpu01 modprobe.d]# cat ./ib_ipoib.conf<br>
>> alias netdev-ib* ib_ipoib<br>
>><br>
>> [root@pg-gpu01 modprobe.d]# cat ./mlnx.conf<br>
>> # Module parameters for MLNX_OFED kernel modules<br>
>><br>
>> [root@pg-gpu01 modprobe.d]# cat ./lustre.conf<br>
>> options lnet networks=o2ib(ib0)<br>
>><br>
>> Are there more Lustre/LNET options that could help in this situation?<br>
>><br>
>> What about the logfiles?<br>
>> Any error messages in syslog? lctl debug options?<br>
>> Veel geluk,<br>
>> Eli<br>
>><br>
>><br>
>><br>
>><br>
>> On Mon, Apr 24, 2017 at 7:02 PM, Raj <<a href="mailto:rajgautam@gmail.com">rajgautam@gmail.com</a>> wrote:<br>
>> May be worth checking your lnet credits and peer_credits in /etc/modprobe.d ?<br>
>> You can compare between working hosts and non working hosts.<br>
>> Thanks<br>
>> _Raj<br>
>><br>
>> On Mon, Apr 24, 2017 at 10:10 AM Strikwerda, Ger <<a href="mailto:g.j.c.strikwerda@rug.nl">g.j.c.strikwerda@rug.nl</a>> wrote:<br>
>> Hi Rick,<br>
>><br>
>> Even without iptables rules and loading the correct modules afterwards, we get the same results:<br>
>><br>
>> [root@pg-gpu01 sysconfig]# iptables --list<br>
>> Chain INPUT (policy ACCEPT)<br>
>> target prot opt source destination<br>
>><br>
>> Chain FORWARD (policy ACCEPT)<br>
>> target prot opt source destination<br>
>><br>
>> Chain OUTPUT (policy ACCEPT)<br>
>> target prot opt source destination<br>
>><br>
>> Chain LOGDROP (0 references)<br>
>> target prot opt source destination<br>
>> LOG all -- anywhere anywhere LOG level warning<br>
>> DROP all -- anywhere anywhere<br>
>><br>
>> [root@pg-gpu01 sysconfig]# modprobe lnet<br>
>><br>
>> [root@pg-gpu01 sysconfig]# modprobe lustre<br>
>><br>
>> [root@pg-gpu01 sysconfig]# lctl ping 172.23.55.211@o2ib<br>
>><br>
>> failed to ping 172.23.55.211@o2ib: Input/output error<br>
>><br>
>><br>
>><br>
>><br>
>><br>
>><br>
>><br>
>> On Mon, Apr 24, 2017 at 4:59 PM, Mohr Jr, Richard Frank (Rick Mohr) <<a href="mailto:rmohr@utk.edu">rmohr@utk.edu</a>> wrote:<br>
>> This might be a long shot, but have you checked for possible firewall rules that might be causing the issue? I\u2019m wondering if there is a chance that some rules were added after the nodes were up to allow Lustre access, and when a node got rebooted, it lost the rules.<br>
>><br>
>> --<br>
>> Rick Mohr<br>
>> Senior HPC System Administrator<br>
>> National Institute for Computational Sciences<br>
>> <a target="_blank" rel="noreferrer" href="http://www.nics.tennessee.edu">http://www.nics.tennessee.edu</a><br>
>><br>
>><br>
>>> On Apr 24, 2017, at 10:19 AM, Strikwerda, Ger <<a href="mailto:g.j.c.strikwerda@rug.nl">g.j.c.strikwerda@rug.nl</a>> wrote:<br>
>>><br>
>>> Hi Russell,<br>
>>><br>
>>> Thanks for the IB subnet clues:<br>
>>><br>
>>> [root@pg-gpu01 ~]# ibv_devinfo<br>
>>> hca_id: mlx4_0<br>
>>> transport: InfiniBand (0)<br>
>>> fw_ver: 2.32.5100<br>
>>> node_guid: f452:1403:00f5:4620<br>
>>> sys_image_guid: f452:1403:00f5:4623<br>
>>> vendor_id: 0x02c9<br>
>>> vendor_part_id: 4099<br>
>>> hw_ver: 0x1<br>
>>> board_id: MT_1100120019<br>
>>> phys_port_cnt: 1<br>
>>> port: 1<br>
>>> state: PORT_ACTIVE (4)<br>
>>> max_mtu: 4096 (5)<br>
>>> active_mtu: 4096 (5)<br>
>>> sm_lid: 1<br>
>>> port_lid: 185<br>
>>> port_lmc: 0x00<br>
>>> link_layer: InfiniBand<br>
>>><br>
>>> [root@pg-gpu01 ~]# sminfo<br>
>>> sminfo: sm lid 1 sm guid 0xf452140300f62320, activity count 80878098 priority 0 state 3 SMINFO_MASTER<br>
>>><br>
>>> Looks like the rebooted node is able to connect/contact IB/IB subnetmanager<br>
>>><br>
>>><br>
>>><br>
>>><br>
>>> On Mon, Apr 24, 2017 at 4:14 PM, Russell Dekema <<a href="mailto:dekemar@umich.edu">dekemar@umich.edu</a>> wrote:<br>
>>> At first glance, this sounds like your Infiniband subnet manager may<br>
>>> be down or malfunctioning. In this case, nodes which were already up<br>
>>> when the subnet manager was working will continue to be able to<br>
>>> communicate over IB, but nodes which reboot after the SM goes down<br>
>>> will not.<br>
>>><br>
>>> You can test this theory by running the 'ibv_devinfo' command on one<br>
>>> of your rebooted nodes. If the relevant IB port is in state PORT_INIT,<br>
>>> this confirms there is a problem with your subnet manager.<br>
>>><br>
>>> Sincerely,<br>
>>> Rusty Dekema<br>
>>><br>
>>><br>
>>><br>
>>><br>
>>> On Mon, Apr 24, 2017 at 9:57 AM, Strikwerda, Ger<br>
>>> <<a href="mailto:g.j.c.strikwerda@rug.nl">g.j.c.strikwerda@rug.nl</a>> wrote:<br>
>>>> Hi everybody,<br>
>>>><br>
>>>> Here at the university of Groningen we are now experiencing a strange Lustre<br>
>>>> error. If a client reboots, it fails to mount the Lustre storage. The client<br>
>>>> is not able to reach the MSG service. The storage and nodes are<br>
>>>> communicating over IB and unitil now without any problems. It looks like an<br>
>>>> issue inside LNET. Clients cannot LNET ping/connect the metadata and or<br>
>>>> storage. But the clients are able to LNET ping each other. Clients which not<br>
>>>> have been rebooted, are working fine and have their mounts on our Lustre<br>
>>>> filesystem.<br>
>>>><br>
>>>> Lustre client log:<br>
>>>><br>
>>>> Lustre: Lustre: Build Version: 2.5.3-RC1--PRISTINE-2.6.32-<wbr>573.el6.x86_64<br>
>>>> LNet: Added LNI 172.23.54.51@o2ib [8/256/0/180]<br>
>>>><br>
>>>> LustreError: 15c-8: MGC172.23.55.211@o2ib: The configuration from log<br>
>>>> 'pgdata01-client' failed (-5). This may be the result of communication<br>
>>>> errors between this node and the MGS, a bad configuration, or other errors.<br>
>>>> See the syslog for more information.<br>
>>>> LustreError: 3812:0:(llite_lib.c:1046:ll_<wbr>fill_super()) Unable to process<br>
>>>> log: -5<br>
>>>> Lustre: Unmounted pgdata01-client<br>
>>>> LustreError: 3812:0:(obd_mount.c:1325:<wbr>lustre_fill_super()) Unable to mount<br>
>>>> (-5)<br>
>>>> LNetError: 2882:0:(o2iblnd_cb.c:2587:<wbr>kiblnd_rejected()) 172.23.55.212@o2ib<br>
>>>> rejected: consumer defined fatal error<br>
>>>> LNetError: 2882:0:(o2iblnd_cb.c:2587:<wbr>kiblnd_rejected()) Skipped 1 previous<br>
>>>> similar message<br>
>>>> Lustre: 3765:0:(client.c:1918:ptlrpc_<wbr>expire_one_request()) @@@ Request sent<br>
>>>> has failed due to network error: [sent 1492789626/real 1492789626]<br>
>>>> req@ffff88105af2cc00 x1565303228072004/t0(0)<br>
>>>> o250->MGC172.23.55.211@o2ib@<wbr>172.23.55.212@o2ib:26/25 lens 400/544 e 0 to 1<br>
>>>> dl 1492789631 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1<br>
>>>> Lustre: 3765:0:(client.c:1918:ptlrpc_<wbr>expire_one_request()) Skipped 1<br>
>>>> previous similar message<br>
>>>> LustreError: 3826:0:(client.c:1083:ptlrpc_<wbr>import_delay_req()) @@@ send limit<br>
>>>> expired req@ffff882041ffc000 x1565303228071996/t0(0)<br>
>>>> o101->MGC172.23.55.211@o2ib@<wbr>172.23.55.211@o2ib:26/25 lens 328/344 e 0 to 0<br>
>>>> dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1<br>
>>>> LustreError: 3826:0:(client.c:1083:ptlrpc_<wbr>import_delay_req()) Skipped 2<br>
>>>> previous similar messages<br>
>>>> LustreError: 15c-8: MGC172.23.55.211@o2ib: The configuration from log<br>
>>>> 'pghome01-client' failed (-5). This may be the result of communication<br>
>>>> errors between this node and the MGS, a bad configuration, or other errors.<br>
>>>> See the syslog for more information.<br>
>>>> LustreError: 3826:0:(llite_lib.c:1046:ll_<wbr>fill_super()) Unable to process<br>
>>>> log: -5<br>
>>>><br>
>>>> LNetError: 2882:0:(o2iblnd_cb.c:2587:<wbr>kiblnd_rejected()) 172.23.55.212@o2ib<br>
>>>> rejected: consumer defined fatal error<br>
>>>> LNetError: 2882:0:(o2iblnd_cb.c:2587:<wbr>kiblnd_rejected()) Skipped 1 previous<br>
>>>> similar message<br>
>>>> LNet: 3755:0:(o2iblnd_cb.c:475:<wbr>kiblnd_rx_complete()) Rx from<br>
>>>> 172.23.55.211@o2ib failed: 5<br>
>>>> LNetError: 2882:0:(o2iblnd_cb.c:2587:<wbr>kiblnd_rejected()) 172.23.55.211@o2ib<br>
>>>> rejected: consumer defined fatal error<br>
>>>> LNetError: 2882:0:(o2iblnd_cb.c:2587:<wbr>kiblnd_rejected()) Skipped 1 previous<br>
>>>> similar message<br>
>>>> LNet: 2882:0:(o2iblnd_cb.c:2072:<wbr>kiblnd_peer_connect_failed()) Deleting<br>
>>>> messages for 172.23.55.211@o2ib: connection failed<br>
>>>> LNet: 2882:0:(o2iblnd_cb.c:2072:<wbr>kiblnd_peer_connect_failed()) Deleting<br>
>>>> messages for 172.23.55.212@o2ib: connection failed<br>
>>>> LNet: 3754:0:(o2iblnd_cb.c:475:<wbr>kiblnd_rx_complete()) Rx from<br>
>>>> 172.23.55.212@o2ib failed: 5<br>
>>>> LNet: 3754:0:(o2iblnd_cb.c:475:<wbr>kiblnd_rx_complete()) Skipped 17 previous<br>
>>>> similar messages<br>
>>>> LNet: 2882:0:(o2iblnd_cb.c:2072:<wbr>kiblnd_peer_connect_failed()) Deleting<br>
>>>> messages for 172.23.55.211@o2ib: connection failed<br>
>>>> LNet: 3754:0:(o2iblnd_cb.c:475:<wbr>kiblnd_rx_complete()) Rx from<br>
>>>> 172.23.55.212@o2ib failed: 5<br>
>>>> LNet: 2882:0:(o2iblnd_cb.c:2072:<wbr>kiblnd_peer_connect_failed()) Deleting<br>
>>>> messages for 172.23.55.212@o2ib: connection failed<br>
>>>><br>
>>>> LNET ping of a metadata-node:<br>
>>>><br>
>>>> [root@pg-gpu01 ~]# lctl ping 172.23.55.211@o2ib<br>
>>>> failed to ping 172.23.55.211@o2ib: Input/output error<br>
>>>><br>
>>>> LNET ping of the number 2 metadata-node:<br>
>>>><br>
>>>> [root@pg-gpu01 ~]# lctl ping 172.23.55.212@o2ib<br>
>>>> failed to ping 172.23.55.212@o2ib: Input/output error<br>
>>>><br>
>>>> LNET ping of a random compute-node:<br>
>>>><br>
>>>> [root@pg-gpu01 ~]# lctl ping 172.23.52.5@o2ib<br>
>>>> 12345-0@lo<br>
>>>> 12345-172.23.52.5@o2ib<br>
>>>><br>
>>>> LNET to OST01:<br>
>>>><br>
>>>> [root@pg-gpu01 ~]# lctl ping 172.23.55.201@o2ib<br>
>>>> failed to ping 172.23.55.201@o2ib: Input/output error<br>
>>>><br>
>>>> LNET to OST02:<br>
>>>><br>
>>>> [root@pg-gpu01 ~]# lctl ping 172.23.55.202@o2ib<br>
>>>> failed to ping 172.23.55.202@o2ib: Input/output error<br>
>>>><br>
>>>> 'normal' pings (on ip level) works fine:<br>
>>>><br>
>>>> [root@pg-gpu01 ~]# ping 172.23.55.201<br>
>>>> PING 172.23.55.201 (172.23.55.201) 56(84) bytes of data.<br>
>>>> 64 bytes from <a target="_blank" rel="noreferrer" href="http://172.23.55.201">172.23.55.201</a>: icmp_seq=1 ttl=64 time=0.741 ms<br>
>>>><br>
>>>> [root@pg-gpu01 ~]# ping 172.23.55.202<br>
>>>> PING 172.23.55.202 (172.23.55.202) 56(84) bytes of data.<br>
>>>> 64 bytes from <a target="_blank" rel="noreferrer" href="http://172.23.55.202">172.23.55.202</a>: icmp_seq=1 ttl=64 time=0.704 ms<br>
>>>><br>
>>>> lctl on a rebooted node:<br>
>>>><br>
>>>> [root@pg-gpu01 ~]# lctl dl<br>
>>>><br>
>>>> lctl on a not rebooted node:<br>
>>>><br>
>>>> [root@pg-node005 ~]# lctl dl<br>
>>>> 0 UP mgc MGC172.23.55.211@o2ib 94bd1c8a-512f-b920-9a4e-<wbr>a6aced3d386d 5<br>
>>>> 1 UP lov pgtemp01-clilov-<wbr>ffff88206906d400<br>
>>>> 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 4<br>
>>>> 2 UP lmv pgtemp01-clilmv-<wbr>ffff88206906d400<br>
>>>> 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 4<br>
>>>> 3 UP mdc pgtemp01-MDT0000-mdc-<wbr>ffff88206906d400<br>
>>>> 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
>>>> 4 UP osc pgtemp01-OST0001-osc-<wbr>ffff88206906d400<br>
>>>> 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
>>>> 5 UP osc pgtemp01-OST0003-osc-<wbr>ffff88206906d400<br>
>>>> 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
>>>> 6 UP osc pgtemp01-OST0005-osc-<wbr>ffff88206906d400<br>
>>>> 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
>>>> 7 UP osc pgtemp01-OST0007-osc-<wbr>ffff88206906d400<br>
>>>> 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
>>>> 8 UP osc pgtemp01-OST0009-osc-<wbr>ffff88206906d400<br>
>>>> 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
>>>> 9 UP osc pgtemp01-OST000b-osc-<wbr>ffff88206906d400<br>
>>>> 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
>>>> 10 UP osc pgtemp01-OST000d-osc-<wbr>ffff88206906d400<br>
>>>> 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
>>>> 11 UP osc pgtemp01-OST000f-osc-<wbr>ffff88206906d400<br>
>>>> 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
>>>> 12 UP osc pgtemp01-OST0011-osc-<wbr>ffff88206906d400<br>
>>>> 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
>>>> 13 UP osc pgtemp01-OST0002-osc-<wbr>ffff88206906d400<br>
>>>> 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
>>>> 14 UP osc pgtemp01-OST0004-osc-<wbr>ffff88206906d400<br>
>>>> 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
>>>> 15 UP osc pgtemp01-OST0006-osc-<wbr>ffff88206906d400<br>
>>>> 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
>>>> 16 UP osc pgtemp01-OST0008-osc-<wbr>ffff88206906d400<br>
>>>> 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
>>>> 17 UP osc pgtemp01-OST000a-osc-<wbr>ffff88206906d400<br>
>>>> 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
>>>> 18 UP osc pgtemp01-OST000c-osc-<wbr>ffff88206906d400<br>
>>>> 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
>>>> 19 UP osc pgtemp01-OST000e-osc-<wbr>ffff88206906d400<br>
>>>> 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
>>>> 20 UP osc pgtemp01-OST0010-osc-<wbr>ffff88206906d400<br>
>>>> 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
>>>> 21 UP osc pgtemp01-OST0012-osc-<wbr>ffff88206906d400<br>
>>>> 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
>>>> 22 UP osc pgtemp01-OST0013-osc-<wbr>ffff88206906d400<br>
>>>> 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
>>>> 23 UP osc pgtemp01-OST0015-osc-<wbr>ffff88206906d400<br>
>>>> 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
>>>> 24 UP osc pgtemp01-OST0017-osc-<wbr>ffff88206906d400<br>
>>>> 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
>>>> 25 UP osc pgtemp01-OST0014-osc-<wbr>ffff88206906d400<br>
>>>> 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
>>>> 26 UP osc pgtemp01-OST0016-osc-<wbr>ffff88206906d400<br>
>>>> 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
>>>> 27 UP osc pgtemp01-OST0018-osc-<wbr>ffff88206906d400<br>
>>>> 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
>>>> 28 UP lov pgdata01-clilov-<wbr>ffff88204bab6400<br>
>>>> 996b1742-82eb-281c-c322-<wbr>e244672d5225 4<br>
>>>> 29 UP lmv pgdata01-clilmv-<wbr>ffff88204bab6400<br>
>>>> 996b1742-82eb-281c-c322-<wbr>e244672d5225 4<br>
>>>> 30 UP mdc pgdata01-MDT0000-mdc-<wbr>ffff88204bab6400<br>
>>>> 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
>>>> 31 UP osc pgdata01-OST0001-osc-<wbr>ffff88204bab6400<br>
>>>> 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
>>>> 32 UP osc pgdata01-OST0003-osc-<wbr>ffff88204bab6400<br>
>>>> 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
>>>> 33 UP osc pgdata01-OST0005-osc-<wbr>ffff88204bab6400<br>
>>>> 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
>>>> 34 UP osc pgdata01-OST0007-osc-<wbr>ffff88204bab6400<br>
>>>> 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
>>>> 35 UP osc pgdata01-OST0009-osc-<wbr>ffff88204bab6400<br>
>>>> 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
>>>> 36 UP osc pgdata01-OST000b-osc-<wbr>ffff88204bab6400<br>
>>>> 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
>>>> 37 UP osc pgdata01-OST000d-osc-<wbr>ffff88204bab6400<br>
>>>> 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
>>>> 38 UP osc pgdata01-OST000f-osc-<wbr>ffff88204bab6400<br>
>>>> 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
>>>> 39 UP osc pgdata01-OST0002-osc-<wbr>ffff88204bab6400<br>
>>>> 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
>>>> 40 UP osc pgdata01-OST0004-osc-<wbr>ffff88204bab6400<br>
>>>> 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
>>>> 41 UP osc pgdata01-OST0006-osc-<wbr>ffff88204bab6400<br>
>>>> 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
>>>> 42 UP osc pgdata01-OST0008-osc-<wbr>ffff88204bab6400<br>
>>>> 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
>>>> 43 UP osc pgdata01-OST000a-osc-<wbr>ffff88204bab6400<br>
>>>> 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
>>>> 44 UP osc pgdata01-OST000c-osc-<wbr>ffff88204bab6400<br>
>>>> 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
>>>> 45 UP osc pgdata01-OST000e-osc-<wbr>ffff88204bab6400<br>
>>>> 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
>>>> 46 UP osc pgdata01-OST0010-osc-<wbr>ffff88204bab6400<br>
>>>> 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
>>>> 47 UP osc pgdata01-OST0013-osc-<wbr>ffff88204bab6400<br>
>>>> 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
>>>> 48 UP osc pgdata01-OST0015-osc-<wbr>ffff88204bab6400<br>
>>>> 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
>>>> 49 UP osc pgdata01-OST0017-osc-<wbr>ffff88204bab6400<br>
>>>> 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
>>>> 50 UP osc pgdata01-OST0014-osc-<wbr>ffff88204bab6400<br>
>>>> 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
>>>> 51 UP osc pgdata01-OST0016-osc-<wbr>ffff88204bab6400<br>
>>>> 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
>>>> 52 UP osc pgdata01-OST0018-osc-<wbr>ffff88204bab6400<br>
>>>> 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
>>>> 53 UP osc pgdata01-OST0019-osc-<wbr>ffff88204bab6400<br>
>>>> 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
>>>> 54 UP osc pgdata01-OST001a-osc-<wbr>ffff88204bab6400<br>
>>>> 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
>>>> 55 UP osc pgdata01-OST001b-osc-<wbr>ffff88204bab6400<br>
>>>> 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
>>>> 56 UP lov pghome01-clilov-<wbr>ffff88204bb50000<br>
>>>> 9ae8f2a9-1cdf-901f-160c-<wbr>66f70e4c10d1 4<br>
>>>> 57 UP lmv pghome01-clilmv-<wbr>ffff88204bb50000<br>
>>>> 9ae8f2a9-1cdf-901f-160c-<wbr>66f70e4c10d1 4<br>
>>>> 58 UP mdc pghome01-MDT0000-mdc-<wbr>ffff88204bb50000<br>
>>>> 9ae8f2a9-1cdf-901f-160c-<wbr>66f70e4c10d1 5<br>
>>>> 59 UP osc pghome01-OST0011-osc-<wbr>ffff88204bb50000<br>
>>>> 9ae8f2a9-1cdf-901f-160c-<wbr>66f70e4c10d1 5<br>
>>>> 60 UP osc pghome01-OST0012-osc-<wbr>ffff88204bb50000<br>
>>>> 9ae8f2a9-1cdf-901f-160c-<wbr>66f70e4c10d1 5<br>
>>>><br>
>>>> Please help, any clues/advice/hints/tips are appricated<br>
>>>><br>
>>>> --<br>
>>>><br>
>>>> Vriendelijke groet,<br>
>>>><br>
>>>> Ger Strikwerda<br>
>>>> Chef Special<br>
>>>> Rijksuniversiteit Groningen<br>
>>>> Centrum voor Informatie Technologie<br>
>>>> Unit Pragmatisch Systeembeheer<br>
>>>><br>
>>>> Smitsborg<br>
>>>> Nettelbosje 1<br>
>>>> 9747 AJ Groningen<br>
>>>> Tel. 050 363 9276<br>
>>>><br>
>>>> "God is hard, God is fair<br>
>>>> some men he gave brains, others he gave hair"<br>
>>>><br>
>>>><br>
>>>> ______________________________<wbr>_________________<br>
>>>> lustre-discuss mailing list<br>
>>>> <a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.<wbr>org</a><br>
>>>> <a target="_blank" rel="noreferrer" href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org">http://lists.lustre.org/<wbr>listinfo.cgi/lustre-discuss-<wbr>lustre.org</a><br>
>>>><br>
>>><br>
>>><br>
>>><br>
>>> --<br>
>>> Vriendelijke groet,<br>
>>><br>
>>> Ger Strikwerda<br>
>>><br>
>>> Chef Special<br>
>>> Rijksuniversiteit Groningen<br>
>>> Centrum voor Informatie Technologie<br>
>>> Unit Pragmatisch Systeembeheer<br>
>>><br>
>>> Smitsborg<br>
>>> Nettelbosje 1<br>
>>> 9747 AJ Groningen<br>
>>> Tel. 050 363 9276<br>
>>><br>
>>><br>
>>> "God is hard, God is fair<br>
>>> some men he gave brains, others he gave hair"<br>
>>> ______________________________<wbr>_________________<br>
>>> lustre-discuss mailing list<br>
>>> <a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.<wbr>org</a><br>
>>> <a target="_blank" rel="noreferrer" href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org">http://lists.lustre.org/<wbr>listinfo.cgi/lustre-discuss-<wbr>lustre.org</a><br>
>><br>
>><br>
>><br>
>><br>
>><br>
>> --<br>
>> Vriendelijke groet,<br>
>><br>
>> Ger Strikwerda<br>
>><br>
>> Chef Special<br>
>> Rijksuniversiteit Groningen<br>
>> Centrum voor Informatie Technologie<br>
>> Unit Pragmatisch Systeembeheer<br>
>><br>
>> Smitsborg<br>
>> Nettelbosje 1<br>
>> 9747 AJ Groningen<br>
>> Tel. 050 363 9276<br>
>><br>
>><br>
>> "God is hard, God is fair<br>
>> some men he gave brains, others he gave hair"<br>
>> ______________________________<wbr>_________________<br>
>> lustre-discuss mailing list<br>
>> <a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.<wbr>org</a><br>
>> <a target="_blank" rel="noreferrer" href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org">http://lists.lustre.org/<wbr>listinfo.cgi/lustre-discuss-<wbr>lustre.org</a><br>
>><br>
>><br>
>><br>
>> --<br>
>> Vriendelijke groet,<br>
>><br>
>> Ger Strikwerda<br>
>><br>
>> Chef Special<br>
>> Rijksuniversiteit Groningen<br>
>> Centrum voor Informatie Technologie<br>
>> Unit Pragmatisch Systeembeheer<br>
>><br>
>> Smitsborg<br>
>> Nettelbosje 1<br>
>> 9747 AJ Groningen<br>
>> Tel. 050 363 9276<br>
>><br>
>><br>
>> "God is hard, God is fair<br>
>> some men he gave brains, others he gave hair"<br>
>><br>
>> ______________________________<wbr>_________________<br>
>> lustre-discuss mailing list<br>
>> <a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.<wbr>org</a><br>
>> <a target="_blank" rel="noreferrer" href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org">http://lists.lustre.org/<wbr>listinfo.cgi/lustre-discuss-<wbr>lustre.org</a><br>
>><br>
>><br>
>><br>
>><br>
>><br>
>> --<br>
>> Vriendelijke groet,<br>
>><br>
>> Ger Strikwerda<br>
>><br>
>> Chef Special<br>
>> Rijksuniversiteit Groningen<br>
>> Centrum voor Informatie Technologie<br>
>> Unit Pragmatisch Systeembeheer<br>
>><br>
>> Smitsborg<br>
>> Nettelbosje 1<br>
>> 9747 AJ Groningen<br>
>> Tel. 050 363 9276<br>
>><br>
>><br>
>> "God is hard, God is fair<br>
>> some men he gave brains, others he gave hair"<br>
>><br>
>><br>
>><br>
>><br>
>> --<br>
>> Vriendelijke groet,<br>
>><br>
>> Ger Strikwerda<br>
>><br>
>> Chef Special<br>
>> Rijksuniversiteit Groningen<br>
>> Centrum voor Informatie Technologie<br>
>> Unit Pragmatisch Systeembeheer<br>
>><br>
>> Smitsborg<br>
>> Nettelbosje 1<br>
>> 9747 AJ Groningen<br>
>> Tel. 050 363 9276<br>
>><br>
>><br>
>> "God is hard, God is fair<br>
>> some men he gave brains, others he gave hair"<br>
>><br>
>><br>
>><br>
>> --<br>
>> Vriendelijke groet,<br>
>><br>
>> Ger Strikwerda<br>
>><br>
>> Chef Special<br>
>> Rijksuniversiteit Groningen<br>
>> Centrum voor Informatie Technologie<br>
>> Unit Pragmatisch Systeembeheer<br>
>><br>
>> Smitsborg<br>
>> Nettelbosje 1<br>
>> 9747 AJ Groningen<br>
>> Tel. 050 363 9276<br>
>><br>
>><br>
>> "God is hard, God is fair<br>
>> some men he gave brains, others he gave hair"<br>
>><br>
>><br>
>><br>
>> --<br>
>> Vriendelijke groet,<br>
>><br>
>> Ger Strikwerda<br>
>><br>
>> Chef Special<br>
>> Rijksuniversiteit Groningen<br>
>> Centrum voor Informatie Technologie<br>
>> Unit Pragmatisch Systeembeheer<br>
>><br>
>> Smitsborg<br>
>> Nettelbosje 1<br>
>> 9747 AJ Groningen<br>
>> Tel. 050 363 9276<br>
>><br>
>><br>
>> "God is hard, God is fair<br>
>> some men he gave brains, others he gave hair"<br>
>> ______________________________<wbr>_________________<br>
>> lustre-discuss mailing list<br>
>> <a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.<wbr>org</a><br>
>> <a target="_blank" rel="noreferrer" href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org">http://lists.lustre.org/<wbr>listinfo.cgi/lustre-discuss-<wbr>lustre.org</a><br>
><br>
> Cheers, Andreas<br>
> --<br>
> Andreas Dilger<br>
> Lustre Principal Architect<br>
> Intel Corporation<br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
> ______________________________<wbr>_________________<br>
> lustre-discuss mailing list<br>
> <a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.<wbr>org</a><br>
> <a target="_blank" rel="noreferrer" href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org">http://lists.lustre.org/<wbr>listinfo.cgi/lustre-discuss-<wbr>lustre.org</a><br>
<br>
______________________________<wbr>_________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.<wbr>org</a><br>
<a target="_blank" rel="noreferrer" href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org">http://lists.lustre.org/<wbr>listinfo.cgi/lustre-discuss-<wbr>lustre.org</a><br>
</div></div></blockquote></div><br></div></div>