<div dir="ltr"><div>Hi Russell,<br><br></div>On a rebooted node:<br><div><br>[root@pg-gpu01 ~]# ibhosts | wc -l<br>183<br><br></div><div>On a not-rebooted node:<br></div><div><br>[root@pg-gpu02 ~]# ibhosts | wc -l<br>183<br><br></div><div>No diffence and all our lustre storage nodes seems to be present:<br><br>Ca : 0xf45214030062eb50 ports 2 "pg-ost01 HCA-1"<br>Ca : 0xf45214030062eb30 ports 2 "pg-mds02 HCA-1"<br>Ca : 0xf45214030062ee60 ports 2 "pg-ost03 HCA-1"<br>Ca : 0xf45214030062ee90 ports 2 "pg-mds01 HCA-1"<br>Ca : 0xf45214030062eb00 ports 2 "pg-ost04 HCA-1"<br>Ca : 0xe41d2d030001ce80 ports 2 "pg-ost02 HCA-1"<br><br><br></div><div><br><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Apr 24, 2017 at 4:46 PM, Russell Dekema <span dir="ltr"><<a href="mailto:dekemar@umich.edu" target="_blank">dekemar@umich.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I'm not sure this is likely to help either, but if you run the command<br>
'ibhosts' on one of the non-working Lustre client nodes, do you see<br>
all of your Lustre servers in the printed list?<br>
<br>
-Rusty<br>
<div class="HOEnZb"><div class="h5"><br>
On Mon, Apr 24, 2017 at 10:39 AM, Russell Dekema <<a href="mailto:dekemar@umich.edu">dekemar@umich.edu</a>> wrote:<br>
> I can't rule it out, but it seems unlikely to me that an out of date<br>
> IB HCA firmware version would cause a problem like this, especially<br>
> when everything was working before on that same version, and when IB<br>
> communication over the device seems to be working in general (as shown<br>
> by your pings over your IPoIB interfaces).<br>
><br>
> If you decide to update your HCA (or switch, in the case of<br>
> switch-f0cc8e/U1) firmware, you may want to check with the vendor<br>
> before doing so. In the past, they have sometimes told me that the<br>
> "latest FW version available for this device" reported by ibdiagnet is<br>
> incorrect and should be ignored. Of course, in other cases, newer<br>
> firmware versions were in fact available and they did recommend<br>
> upgrading.<br>
><br>
> Sincerely,<br>
> Rusty D.<br>
><br>
><br>
><br>
> On Mon, Apr 24, 2017 at 10:32 AM, Strikwerda, Ger<br>
> <<a href="mailto:g.j.c.strikwerda@rug.nl">g.j.c.strikwerda@rug.nl</a>> wrote:<br>
>> Hi Russell/*,<br>
>><br>
>> If we run ibdiagnet we get errors/warnings about some (newer) nodes which<br>
>> happen to have a new firmware on the IB interface:<br>
>><br>
>> Nodes Information<br>
>> -E- FW Check finished with errors<br>
>> -W- pg-gpu01/U1 - Node has FW version 2.32.5100 while the latest FW version,<br>
>> for the same device available on this fabric is 2.36.5150<br>
>> -W- pg-gpu05/U1 - Node has FW version 2.35.5100 while the latest FW version,<br>
>> for the same device available on this fabric is 2.36.5150<br>
>> -W- pg-gpu06/U1 - Node has FW version 2.35.5100 while the latest FW version,<br>
>> for the same device available on this fabric is 2.36.5150<br>
>> -W- pg-gpu02/U1 - Node has FW version 2.32.5100 while the latest FW version,<br>
>> for the same device available on this fabric is 2.36.5150<br>
>> -W- pg-gpu03/U1 - Node has FW version 2.32.5100 while the latest FW version,<br>
>> for the same device available on this fabric is 2.36.5150<br>
>> -W- pg-gpu04/U1 - Node has FW version 2.32.5100 while the latest FW version,<br>
>> for the same device available on this fabric is 2.36.5150<br>
>> -W- switch-f0cc8e/U1 - Node has FW version 9.2.7300 while the latest FW<br>
>> version, for the same device available on this fabric is 9.2.8000<br>
>> -W- pg-memory02/U1 - Node has FW version 2.32.5100 while the latest FW<br>
>> version, for the same device available on this fabric is 2.36.5150<br>
>> -W- pg-memory01/U1 - Node has FW version 2.32.5100 while the latest FW<br>
>> version, for the same device available on this fabric is 2.36.5150<br>
>> -W- pg-memory03/U1 - Node has FW version 2.32.5100 while the latest FW<br>
>> version, for the same device available on this fabric is 2.36.5150<br>
>> -W- pg-memory04/U1 - Node has FW version 2.32.5100 while the latest FW<br>
>> version, for the same device available on this fabric is 2.36.5150<br>
>> -W- pg-memory06/U1 - Node has FW version 2.32.5100 while the latest FW<br>
>> version, for the same device available on this fabric is 2.36.5150<br>
>> -W- pg-memory05/U1 - Node has FW version 2.32.5100 while the latest FW<br>
>> version, for the same device available on this fabric is 2.36.5150<br>
>> -W- pg-node163/U1 - Node has FW version 2.36.5000 while the latest FW<br>
>> version, for the same device available on this fabric is 2.36.5150<br>
>> -W- pg-node164/U1 - Node has FW version 2.36.5000 while the latest FW<br>
>> version, for the same device available on this fabric is 2.36.5150<br>
>> -W- pg-node001/U1 - Node has FW version 2.33.5100 while the latest FW<br>
>> version, for the same device available on this fabric is 2.36.5150<br>
>> -W- pg-interactive/U1 - Node has FW version 2.32.5100 while the latest FW<br>
>> version, for the same device available on this fabric is 2.36.5150<br>
>> -W- peregrine/U1 - Node has FW version 2.32.5100 while the latest FW<br>
>> version, for the same device available on this fabric is 2.36.5150<br>
>> -W- pg-node004/U1 - Node has FW version 2.32.5100 while the latest FW<br>
>> version, for the same device available on this fabric is 2.36.5150<br>
>> -W- pg-node005/U1 - Node has FW version 2.32.5100 while the latest FW<br>
>> version, for the same device available on this fabric is 2.36.5150<br>
>> -W- pg-node006/U1 - Node has FW version 2.32.5100 while the latest FW<br>
>> version, for the same device available on this fabric is 2.36.5150<br>
>> -W- pg-node007/U1 - Node has FW version 2.32.5100 while the latest FW<br>
>> version, for the same device available on this fabric is 2.36.5150<br>
>> -W- pg-node008/U1 - Node has FW version 2.32.5100 while the latest FW<br>
>> version, for the same device available on this fabric is 2.36.5150<br>
>> -W- pg-node009/U1 - Node has FW version 2.32.5100 while the latest FW<br>
>> version, for the same device available on this fabric is 2.36.5150<br>
>> -W- pg-node010/U1 - Node has FW version 2.32.5100 while the latest FW<br>
>> version, for the same device available on this fabric is 2.36.5150<br>
>> -W- pg-node011/U1 - Node has FW version 2.32.5100 while the latest FW<br>
>> version, for the same device available on this fabric is 2.36.5150<br>
>> -W- pg-node012/U1 - Node has FW version 2.32.5100 while the latest FW<br>
>> version, for the same device available on this fabric is 2.36.5150<br>
>> -W- pg-node013/U1 - Node has FW version 2.32.5100 while the latest FW<br>
>> version, for the same device available on this fabric is 2.36.5150<br>
>> -W- pg-node014/U1 - Node has FW version 2.32.5100 while the latest FW<br>
>> version, for the same device available on this fabric is 2.36.5150<br>
>> -W- pg-node015/U1 - Node has FW version 2.32.5100 while the latest FW<br>
>> version, for the same device available on this fabric is 2.36.5150<br>
>><br>
>> Could that be problematic/an issue?<br>
>><br>
>><br>
>><br>
>><br>
>> On Mon, Apr 24, 2017 at 4:27 PM, Russell Dekema <<a href="mailto:dekemar@umich.edu">dekemar@umich.edu</a>> wrote:<br>
>>><br>
>>> Oh, ok, that seems to rule the subnet manager out.<br>
>>><br>
>>> I mis-read your IP network numbers earlier and thought you had not<br>
>>> tried regular IP-ping across your IPoIB interfaces, but, upon<br>
>>> re-reading your initial message, it seems you have tried this and it<br>
>>> does work, even between a client with non-working Lustre and your<br>
>>> MGS/MDS.<br>
>>><br>
>>> In this case, I have no further suggestions.<br>
>>><br>
>>> Best of luck,<br>
>>> Rusty D.<br>
>>><br>
>>> On Mon, Apr 24, 2017 at 10:19 AM, Strikwerda, Ger<br>
>>> <<a href="mailto:g.j.c.strikwerda@rug.nl">g.j.c.strikwerda@rug.nl</a>> wrote:<br>
>>> > Hi Russell,<br>
>>> ><br>
>>> > Thanks for the IB subnet clues:<br>
>>> ><br>
>>> > [root@pg-gpu01 ~]# ibv_devinfo<br>
>>> > hca_id: mlx4_0<br>
>>> > transport: InfiniBand (0)<br>
>>> > fw_ver: 2.32.5100<br>
>>> > node_guid: f452:1403:00f5:4620<br>
>>> > sys_image_guid: f452:1403:00f5:4623<br>
>>> > vendor_id: 0x02c9<br>
>>> > vendor_part_id: 4099<br>
>>> > hw_ver: 0x1<br>
>>> > board_id: MT_1100120019<br>
>>> > phys_port_cnt: 1<br>
>>> > port: 1<br>
>>> > state: PORT_ACTIVE (4)<br>
>>> > max_mtu: 4096 (5)<br>
>>> > active_mtu: 4096 (5)<br>
>>> > sm_lid: 1<br>
>>> > port_lid: 185<br>
>>> > port_lmc: 0x00<br>
>>> > link_layer: InfiniBand<br>
>>> ><br>
>>> > [root@pg-gpu01 ~]# sminfo<br>
>>> > sminfo: sm lid 1 sm guid 0xf452140300f62320, activity count 80878098<br>
>>> > priority 0 state 3 SMINFO_MASTER<br>
>>> ><br>
>>> > Looks like the rebooted node is able to connect/contact IB/IB<br>
>>> > subnetmanager<br>
>>> ><br>
>>> ><br>
>>> ><br>
>>> ><br>
>>> > On Mon, Apr 24, 2017 at 4:14 PM, Russell Dekema <<a href="mailto:dekemar@umich.edu">dekemar@umich.edu</a>><br>
>>> > wrote:<br>
>>> >><br>
>>> >> At first glance, this sounds like your Infiniband subnet manager may<br>
>>> >> be down or malfunctioning. In this case, nodes which were already up<br>
>>> >> when the subnet manager was working will continue to be able to<br>
>>> >> communicate over IB, but nodes which reboot after the SM goes down<br>
>>> >> will not.<br>
>>> >><br>
>>> >> You can test this theory by running the 'ibv_devinfo' command on one<br>
>>> >> of your rebooted nodes. If the relevant IB port is in state PORT_INIT,<br>
>>> >> this confirms there is a problem with your subnet manager.<br>
>>> >><br>
>>> >> Sincerely,<br>
>>> >> Rusty Dekema<br>
>>> >><br>
>>> >><br>
>>> >><br>
>>> >><br>
>>> >> On Mon, Apr 24, 2017 at 9:57 AM, Strikwerda, Ger<br>
>>> >> <<a href="mailto:g.j.c.strikwerda@rug.nl">g.j.c.strikwerda@rug.nl</a>> wrote:<br>
>>> >> > Hi everybody,<br>
>>> >> ><br>
>>> >> > Here at the university of Groningen we are now experiencing a strange<br>
>>> >> > Lustre<br>
>>> >> > error. If a client reboots, it fails to mount the Lustre storage. The<br>
>>> >> > client<br>
>>> >> > is not able to reach the MSG service. The storage and nodes are<br>
>>> >> > communicating over IB and unitil now without any problems. It looks<br>
>>> >> > like<br>
>>> >> > an<br>
>>> >> > issue inside LNET. Clients cannot LNET ping/connect the metadata and<br>
>>> >> > or<br>
>>> >> > storage. But the clients are able to LNET ping each other. Clients<br>
>>> >> > which<br>
>>> >> > not<br>
>>> >> > have been rebooted, are working fine and have their mounts on our<br>
>>> >> > Lustre<br>
>>> >> > filesystem.<br>
>>> >> ><br>
>>> >> > Lustre client log:<br>
>>> >> ><br>
>>> >> > Lustre: Lustre: Build Version:<br>
>>> >> > 2.5.3-RC1--PRISTINE-2.6.32-<wbr>573.el6.x86_64<br>
>>> >> > LNet: Added LNI 172.23.54.51@o2ib [8/256/0/180]<br>
>>> >> ><br>
>>> >> > LustreError: 15c-8: MGC172.23.55.211@o2ib: The configuration from log<br>
>>> >> > 'pgdata01-client' failed (-5). This may be the result of<br>
>>> >> > communication<br>
>>> >> > errors between this node and the MGS, a bad configuration, or other<br>
>>> >> > errors.<br>
>>> >> > See the syslog for more information.<br>
>>> >> > LustreError: 3812:0:(llite_lib.c:1046:ll_<wbr>fill_super()) Unable to<br>
>>> >> > process<br>
>>> >> > log: -5<br>
>>> >> > Lustre: Unmounted pgdata01-client<br>
>>> >> > LustreError: 3812:0:(obd_mount.c:1325:<wbr>lustre_fill_super()) Unable to<br>
>>> >> > mount<br>
>>> >> > (-5)<br>
>>> >> > LNetError: 2882:0:(o2iblnd_cb.c:2587:<wbr>kiblnd_rejected())<br>
>>> >> > 172.23.55.212@o2ib<br>
>>> >> > rejected: consumer defined fatal error<br>
>>> >> > LNetError: 2882:0:(o2iblnd_cb.c:2587:<wbr>kiblnd_rejected()) Skipped 1<br>
>>> >> > previous<br>
>>> >> > similar message<br>
>>> >> > Lustre: 3765:0:(client.c:1918:ptlrpc_<wbr>expire_one_request()) @@@<br>
>>> >> > Request<br>
>>> >> > sent<br>
>>> >> > has failed due to network error: [sent 1492789626/real 1492789626]<br>
>>> >> > req@ffff88105af2cc00 x1565303228072004/t0(0)<br>
>>> >> > o250->MGC172.23.55.211@o2ib@<wbr>172.23.55.212@o2ib:26/25 lens 400/544 e 0<br>
>>> >> > to<br>
>>> >> > 1<br>
>>> >> > dl 1492789631 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1<br>
>>> >> > Lustre: 3765:0:(client.c:1918:ptlrpc_<wbr>expire_one_request()) Skipped 1<br>
>>> >> > previous similar message<br>
>>> >> > LustreError: 3826:0:(client.c:1083:ptlrpc_<wbr>import_delay_req()) @@@<br>
>>> >> > send<br>
>>> >> > limit<br>
>>> >> > expired req@ffff882041ffc000 x1565303228071996/t0(0)<br>
>>> >> > o101->MGC172.23.55.211@o2ib@<wbr>172.23.55.211@o2ib:26/25 lens 328/344 e 0<br>
>>> >> > to<br>
>>> >> > 0<br>
>>> >> > dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1<br>
>>> >> > LustreError: 3826:0:(client.c:1083:ptlrpc_<wbr>import_delay_req()) Skipped<br>
>>> >> > 2<br>
>>> >> > previous similar messages<br>
>>> >> > LustreError: 15c-8: MGC172.23.55.211@o2ib: The configuration from log<br>
>>> >> > 'pghome01-client' failed (-5). This may be the result of<br>
>>> >> > communication<br>
>>> >> > errors between this node and the MGS, a bad configuration, or other<br>
>>> >> > errors.<br>
>>> >> > See the syslog for more information.<br>
>>> >> > LustreError: 3826:0:(llite_lib.c:1046:ll_<wbr>fill_super()) Unable to<br>
>>> >> > process<br>
>>> >> > log: -5<br>
>>> >> ><br>
>>> >> > LNetError: 2882:0:(o2iblnd_cb.c:2587:<wbr>kiblnd_rejected())<br>
>>> >> > 172.23.55.212@o2ib<br>
>>> >> > rejected: consumer defined fatal error<br>
>>> >> > LNetError: 2882:0:(o2iblnd_cb.c:2587:<wbr>kiblnd_rejected()) Skipped 1<br>
>>> >> > previous<br>
>>> >> > similar message<br>
>>> >> > LNet: 3755:0:(o2iblnd_cb.c:475:<wbr>kiblnd_rx_complete()) Rx from<br>
>>> >> > 172.23.55.211@o2ib failed: 5<br>
>>> >> > LNetError: 2882:0:(o2iblnd_cb.c:2587:<wbr>kiblnd_rejected())<br>
>>> >> > 172.23.55.211@o2ib<br>
>>> >> > rejected: consumer defined fatal error<br>
>>> >> > LNetError: 2882:0:(o2iblnd_cb.c:2587:<wbr>kiblnd_rejected()) Skipped 1<br>
>>> >> > previous<br>
>>> >> > similar message<br>
>>> >> > LNet: 2882:0:(o2iblnd_cb.c:2072:<wbr>kiblnd_peer_connect_failed())<br>
>>> >> > Deleting<br>
>>> >> > messages for 172.23.55.211@o2ib: connection failed<br>
>>> >> > LNet: 2882:0:(o2iblnd_cb.c:2072:<wbr>kiblnd_peer_connect_failed())<br>
>>> >> > Deleting<br>
>>> >> > messages for 172.23.55.212@o2ib: connection failed<br>
>>> >> > LNet: 3754:0:(o2iblnd_cb.c:475:<wbr>kiblnd_rx_complete()) Rx from<br>
>>> >> > 172.23.55.212@o2ib failed: 5<br>
>>> >> > LNet: 3754:0:(o2iblnd_cb.c:475:<wbr>kiblnd_rx_complete()) Skipped 17<br>
>>> >> > previous<br>
>>> >> > similar messages<br>
>>> >> > LNet: 2882:0:(o2iblnd_cb.c:2072:<wbr>kiblnd_peer_connect_failed())<br>
>>> >> > Deleting<br>
>>> >> > messages for 172.23.55.211@o2ib: connection failed<br>
>>> >> > LNet: 3754:0:(o2iblnd_cb.c:475:<wbr>kiblnd_rx_complete()) Rx from<br>
>>> >> > 172.23.55.212@o2ib failed: 5<br>
>>> >> > LNet: 2882:0:(o2iblnd_cb.c:2072:<wbr>kiblnd_peer_connect_failed())<br>
>>> >> > Deleting<br>
>>> >> > messages for 172.23.55.212@o2ib: connection failed<br>
>>> >> ><br>
>>> >> > LNET ping of a metadata-node:<br>
>>> >> ><br>
>>> >> > [root@pg-gpu01 ~]# lctl ping 172.23.55.211@o2ib<br>
>>> >> > failed to ping 172.23.55.211@o2ib: Input/output error<br>
>>> >> ><br>
>>> >> > LNET ping of the number 2 metadata-node:<br>
>>> >> ><br>
>>> >> > [root@pg-gpu01 ~]# lctl ping 172.23.55.212@o2ib<br>
>>> >> > failed to ping 172.23.55.212@o2ib: Input/output error<br>
>>> >> ><br>
>>> >> > LNET ping of a random compute-node:<br>
>>> >> ><br>
>>> >> > [root@pg-gpu01 ~]# lctl ping 172.23.52.5@o2ib<br>
>>> >> > 12345-0@lo<br>
>>> >> > 12345-172.23.52.5@o2ib<br>
>>> >> ><br>
>>> >> > LNET to OST01:<br>
>>> >> ><br>
>>> >> > [root@pg-gpu01 ~]# lctl ping 172.23.55.201@o2ib<br>
>>> >> > failed to ping 172.23.55.201@o2ib: Input/output error<br>
>>> >> ><br>
>>> >> > LNET to OST02:<br>
>>> >> ><br>
>>> >> > [root@pg-gpu01 ~]# lctl ping 172.23.55.202@o2ib<br>
>>> >> > failed to ping 172.23.55.202@o2ib: Input/output error<br>
>>> >> ><br>
>>> >> > 'normal' pings (on ip level) works fine:<br>
>>> >> ><br>
>>> >> > [root@pg-gpu01 ~]# ping 172.23.55.201<br>
>>> >> > PING 172.23.55.201 (172.23.55.201) 56(84) bytes of data.<br>
>>> >> > 64 bytes from <a href="http://172.23.55.201" rel="noreferrer" target="_blank">172.23.55.201</a>: icmp_seq=1 ttl=64 time=0.741 ms<br>
>>> >> ><br>
>>> >> > [root@pg-gpu01 ~]# ping 172.23.55.202<br>
>>> >> > PING 172.23.55.202 (172.23.55.202) 56(84) bytes of data.<br>
>>> >> > 64 bytes from <a href="http://172.23.55.202" rel="noreferrer" target="_blank">172.23.55.202</a>: icmp_seq=1 ttl=64 time=0.704 ms<br>
>>> >> ><br>
>>> >> > lctl on a rebooted node:<br>
>>> >> ><br>
>>> >> > [root@pg-gpu01 ~]# lctl dl<br>
>>> >> ><br>
>>> >> > lctl on a not rebooted node:<br>
>>> >> ><br>
>>> >> > [root@pg-node005 ~]# lctl dl<br>
>>> >> > 0 UP mgc MGC172.23.55.211@o2ib 94bd1c8a-512f-b920-9a4e-<wbr>a6aced3d386d<br>
>>> >> > 5<br>
>>> >> > 1 UP lov pgtemp01-clilov-<wbr>ffff88206906d400<br>
>>> >> > 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 4<br>
>>> >> > 2 UP lmv pgtemp01-clilmv-<wbr>ffff88206906d400<br>
>>> >> > 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 4<br>
>>> >> > 3 UP mdc pgtemp01-MDT0000-mdc-<wbr>ffff88206906d400<br>
>>> >> > 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
>>> >> > 4 UP osc pgtemp01-OST0001-osc-<wbr>ffff88206906d400<br>
>>> >> > 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
>>> >> > 5 UP osc pgtemp01-OST0003-osc-<wbr>ffff88206906d400<br>
>>> >> > 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
>>> >> > 6 UP osc pgtemp01-OST0005-osc-<wbr>ffff88206906d400<br>
>>> >> > 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
>>> >> > 7 UP osc pgtemp01-OST0007-osc-<wbr>ffff88206906d400<br>
>>> >> > 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
>>> >> > 8 UP osc pgtemp01-OST0009-osc-<wbr>ffff88206906d400<br>
>>> >> > 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
>>> >> > 9 UP osc pgtemp01-OST000b-osc-<wbr>ffff88206906d400<br>
>>> >> > 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
>>> >> > 10 UP osc pgtemp01-OST000d-osc-<wbr>ffff88206906d400<br>
>>> >> > 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
>>> >> > 11 UP osc pgtemp01-OST000f-osc-<wbr>ffff88206906d400<br>
>>> >> > 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
>>> >> > 12 UP osc pgtemp01-OST0011-osc-<wbr>ffff88206906d400<br>
>>> >> > 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
>>> >> > 13 UP osc pgtemp01-OST0002-osc-<wbr>ffff88206906d400<br>
>>> >> > 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
>>> >> > 14 UP osc pgtemp01-OST0004-osc-<wbr>ffff88206906d400<br>
>>> >> > 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
>>> >> > 15 UP osc pgtemp01-OST0006-osc-<wbr>ffff88206906d400<br>
>>> >> > 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
>>> >> > 16 UP osc pgtemp01-OST0008-osc-<wbr>ffff88206906d400<br>
>>> >> > 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
>>> >> > 17 UP osc pgtemp01-OST000a-osc-<wbr>ffff88206906d400<br>
>>> >> > 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
>>> >> > 18 UP osc pgtemp01-OST000c-osc-<wbr>ffff88206906d400<br>
>>> >> > 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
>>> >> > 19 UP osc pgtemp01-OST000e-osc-<wbr>ffff88206906d400<br>
>>> >> > 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
>>> >> > 20 UP osc pgtemp01-OST0010-osc-<wbr>ffff88206906d400<br>
>>> >> > 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
>>> >> > 21 UP osc pgtemp01-OST0012-osc-<wbr>ffff88206906d400<br>
>>> >> > 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
>>> >> > 22 UP osc pgtemp01-OST0013-osc-<wbr>ffff88206906d400<br>
>>> >> > 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
>>> >> > 23 UP osc pgtemp01-OST0015-osc-<wbr>ffff88206906d400<br>
>>> >> > 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
>>> >> > 24 UP osc pgtemp01-OST0017-osc-<wbr>ffff88206906d400<br>
>>> >> > 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
>>> >> > 25 UP osc pgtemp01-OST0014-osc-<wbr>ffff88206906d400<br>
>>> >> > 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
>>> >> > 26 UP osc pgtemp01-OST0016-osc-<wbr>ffff88206906d400<br>
>>> >> > 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
>>> >> > 27 UP osc pgtemp01-OST0018-osc-<wbr>ffff88206906d400<br>
>>> >> > 281c441f-8aa3-ab56-8812-<wbr>e459d308f47c 5<br>
>>> >> > 28 UP lov pgdata01-clilov-<wbr>ffff88204bab6400<br>
>>> >> > 996b1742-82eb-281c-c322-<wbr>e244672d5225 4<br>
>>> >> > 29 UP lmv pgdata01-clilmv-<wbr>ffff88204bab6400<br>
>>> >> > 996b1742-82eb-281c-c322-<wbr>e244672d5225 4<br>
>>> >> > 30 UP mdc pgdata01-MDT0000-mdc-<wbr>ffff88204bab6400<br>
>>> >> > 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
>>> >> > 31 UP osc pgdata01-OST0001-osc-<wbr>ffff88204bab6400<br>
>>> >> > 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
>>> >> > 32 UP osc pgdata01-OST0003-osc-<wbr>ffff88204bab6400<br>
>>> >> > 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
>>> >> > 33 UP osc pgdata01-OST0005-osc-<wbr>ffff88204bab6400<br>
>>> >> > 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
>>> >> > 34 UP osc pgdata01-OST0007-osc-<wbr>ffff88204bab6400<br>
>>> >> > 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
>>> >> > 35 UP osc pgdata01-OST0009-osc-<wbr>ffff88204bab6400<br>
>>> >> > 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
>>> >> > 36 UP osc pgdata01-OST000b-osc-<wbr>ffff88204bab6400<br>
>>> >> > 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
>>> >> > 37 UP osc pgdata01-OST000d-osc-<wbr>ffff88204bab6400<br>
>>> >> > 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
>>> >> > 38 UP osc pgdata01-OST000f-osc-<wbr>ffff88204bab6400<br>
>>> >> > 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
>>> >> > 39 UP osc pgdata01-OST0002-osc-<wbr>ffff88204bab6400<br>
>>> >> > 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
>>> >> > 40 UP osc pgdata01-OST0004-osc-<wbr>ffff88204bab6400<br>
>>> >> > 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
>>> >> > 41 UP osc pgdata01-OST0006-osc-<wbr>ffff88204bab6400<br>
>>> >> > 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
>>> >> > 42 UP osc pgdata01-OST0008-osc-<wbr>ffff88204bab6400<br>
>>> >> > 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
>>> >> > 43 UP osc pgdata01-OST000a-osc-<wbr>ffff88204bab6400<br>
>>> >> > 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
>>> >> > 44 UP osc pgdata01-OST000c-osc-<wbr>ffff88204bab6400<br>
>>> >> > 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
>>> >> > 45 UP osc pgdata01-OST000e-osc-<wbr>ffff88204bab6400<br>
>>> >> > 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
>>> >> > 46 UP osc pgdata01-OST0010-osc-<wbr>ffff88204bab6400<br>
>>> >> > 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
>>> >> > 47 UP osc pgdata01-OST0013-osc-<wbr>ffff88204bab6400<br>
>>> >> > 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
>>> >> > 48 UP osc pgdata01-OST0015-osc-<wbr>ffff88204bab6400<br>
>>> >> > 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
>>> >> > 49 UP osc pgdata01-OST0017-osc-<wbr>ffff88204bab6400<br>
>>> >> > 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
>>> >> > 50 UP osc pgdata01-OST0014-osc-<wbr>ffff88204bab6400<br>
>>> >> > 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
>>> >> > 51 UP osc pgdata01-OST0016-osc-<wbr>ffff88204bab6400<br>
>>> >> > 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
>>> >> > 52 UP osc pgdata01-OST0018-osc-<wbr>ffff88204bab6400<br>
>>> >> > 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
>>> >> > 53 UP osc pgdata01-OST0019-osc-<wbr>ffff88204bab6400<br>
>>> >> > 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
>>> >> > 54 UP osc pgdata01-OST001a-osc-<wbr>ffff88204bab6400<br>
>>> >> > 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
>>> >> > 55 UP osc pgdata01-OST001b-osc-<wbr>ffff88204bab6400<br>
>>> >> > 996b1742-82eb-281c-c322-<wbr>e244672d5225 5<br>
>>> >> > 56 UP lov pghome01-clilov-<wbr>ffff88204bb50000<br>
>>> >> > 9ae8f2a9-1cdf-901f-160c-<wbr>66f70e4c10d1 4<br>
>>> >> > 57 UP lmv pghome01-clilmv-<wbr>ffff88204bb50000<br>
>>> >> > 9ae8f2a9-1cdf-901f-160c-<wbr>66f70e4c10d1 4<br>
>>> >> > 58 UP mdc pghome01-MDT0000-mdc-<wbr>ffff88204bb50000<br>
>>> >> > 9ae8f2a9-1cdf-901f-160c-<wbr>66f70e4c10d1 5<br>
>>> >> > 59 UP osc pghome01-OST0011-osc-<wbr>ffff88204bb50000<br>
>>> >> > 9ae8f2a9-1cdf-901f-160c-<wbr>66f70e4c10d1 5<br>
>>> >> > 60 UP osc pghome01-OST0012-osc-<wbr>ffff88204bb50000<br>
>>> >> > 9ae8f2a9-1cdf-901f-160c-<wbr>66f70e4c10d1 5<br>
>>> >> ><br>
>>> >> > Please help, any clues/advice/hints/tips are appricated<br>
>>> >> ><br>
>>> >> > --<br>
>>> >> ><br>
>>> >> > Vriendelijke groet,<br>
>>> >> ><br>
>>> >> > Ger Strikwerda<br>
>>> >> > Chef Special<br>
>>> >> > Rijksuniversiteit Groningen<br>
>>> >> > Centrum voor Informatie Technologie<br>
>>> >> > Unit Pragmatisch Systeembeheer<br>
>>> >> ><br>
>>> >> > Smitsborg<br>
>>> >> > Nettelbosje 1<br>
>>> >> > 9747 AJ Groningen<br>
>>> >> > Tel. 050 363 9276<br>
>>> >> ><br>
>>> >> > "God is hard, God is fair<br>
>>> >> > some men he gave brains, others he gave hair"<br>
>>> >> ><br>
>>> >> ><br>
>>> >> > ______________________________<wbr>_________________<br>
>>> >> > lustre-discuss mailing list<br>
>>> >> > <a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.<wbr>org</a><br>
>>> >> > <a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer" target="_blank">http://lists.lustre.org/<wbr>listinfo.cgi/lustre-discuss-<wbr>lustre.org</a><br>
>>> >> ><br>
>>> ><br>
>>> ><br>
>>> ><br>
>>> ><br>
>>> > --<br>
>>> ><br>
>>> > Vriendelijke groet,<br>
>>> ><br>
>>> > Ger Strikwerda<br>
>>> > Chef Special<br>
>>> > Rijksuniversiteit Groningen<br>
>>> > Centrum voor Informatie Technologie<br>
>>> > Unit Pragmatisch Systeembeheer<br>
>>> ><br>
>>> > Smitsborg<br>
>>> > Nettelbosje 1<br>
>>> > 9747 AJ Groningen<br>
>>> > Tel. 050 363 9276<br>
>>> ><br>
>>> > "God is hard, God is fair<br>
>>> > some men he gave brains, others he gave hair"<br>
>><br>
>><br>
>><br>
>><br>
>> --<br>
>><br>
>> Vriendelijke groet,<br>
>><br>
>> Ger Strikwerda<br>
>> Chef Special<br>
>> Rijksuniversiteit Groningen<br>
>> Centrum voor Informatie Technologie<br>
>> Unit Pragmatisch Systeembeheer<br>
>><br>
>> Smitsborg<br>
>> Nettelbosje 1<br>
>> 9747 AJ Groningen<br>
>> Tel. 050 363 9276<br>
>><br>
>> "God is hard, God is fair<br>
>> some men he gave brains, others he gave hair"<br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><pre cols="72"><span style="font-family:arial,helvetica,sans-serif"><font size="2">Vriendelijke groet, <br><br>Ger Strikwerda
</font></span><span style="font-family:arial,helvetica,sans-serif"><font size="2">Chef Special
Rijksuniversiteit Groningen
Centrum voor Informatie Technologie
Unit Pragmatisch Systeembeheer
Smitsborg
Nettelbosje 1
9747 AJ Groningen
Tel. 050 363 9276
<br>"God is hard, God is fair<br> some men he gave brains, others he gave hair"<br></font></span></pre></div></div></div></div></div></div></div></div></div></div></div></div>
</div>