[lustre-discuss] client fails to mount

Raj rajgautam at gmail.com
Mon Apr 24 10:53:17 PDT 2017


Yes, this is strange. Normally, I have seen that credits mismatch results
this scenario but it doesn't look like this is the case.

You wouldn't want to put mgs into capture debug messages as there will be a
lot of data.

I guess you already tried removing the lustre drivers and adding it again ?
lustre_rmmod
modprobe -v lustre

And check dmesg for any errors...


On Mon, Apr 24, 2017 at 12:43 PM Strikwerda, Ger <g.j.c.strikwerda at rug.nl>
wrote:

> Hi Raj,
>
> When i do a lctl ping on a MGS server i do not see any logs at all. Also
> not when i do a sucessfull ping from a working node. Is there a way to
> verbose the Lustre logging to see more detail on the LNET level?
>
> It is very strange that a rebooted node is able to lctl ping compute
> nodes, but fails to lctl ping metadata and storage nodes.
>
>
>
>
> On Mon, Apr 24, 2017 at 7:35 PM, Raj <rajgautam at gmail.com> wrote:
>
>> Ger,
>> It looks like default configuration of lustre.
>>
>> Do you see any error message on the MGS side while you are doing lctl
>> ping from the rebooted clients?
>> On Mon, Apr 24, 2017 at 12:27 PM Strikwerda, Ger <g.j.c.strikwerda at rug.nl>
>> wrote:
>>
>>> Hi Eli,
>>>
>>> Nothing can be mounted on the Lustre filesystems so the output is:
>>>
>>> [root at pg-gpu01 ~]# lfs df /home/ger/
>>> [root at pg-gpu01 ~]#
>>>
>>> Empty..
>>>
>>>
>>>
>>> On Mon, Apr 24, 2017 at 7:24 PM, E.S. Rosenberg <esr at cs.huji.ac.il>
>>> wrote:
>>>
>>>>
>>>>
>>>> On Mon, Apr 24, 2017 at 8:19 PM, Strikwerda, Ger <
>>>> g.j.c.strikwerda at rug.nl> wrote:
>>>>
>>>>> Hallo Eli,
>>>>>
>>>>> Logfile/syslog on the client-side:
>>>>>
>>>>> Lustre: Lustre: Build Version:
>>>>> 2.5.3-RC1--PRISTINE-2.6.32-573.el6.x86_64
>>>>> LNet: Added LNI 172.23.54.51 at o2ib [8/256/0/180]
>>>>> LNetError: 2878:0:(o2iblnd_cb.c:2587:kiblnd_rejected())
>>>>> 172.23.55.211 at o2ib rejected: consumer defined fatal error
>>>>>
>>>>
>>>> lctl df /path/to/some/file
>>>>
>>>> gives nothing useful? (the second one will dump *a lot*)
>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Apr 24, 2017 at 7:16 PM, E.S. Rosenberg <
>>>>> esr+lustre at mail.hebrew.edu> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Apr 24, 2017 at 8:13 PM, Strikwerda, Ger <
>>>>>> g.j.c.strikwerda at rug.nl> wrote:
>>>>>>
>>>>>>> Hi Raj (and others),
>>>>>>>
>>>>>>> In which file should i state the credits/peer_credits stuff?
>>>>>>>
>>>>>>> Perhaps relevant config-files:
>>>>>>>
>>>>>>> [root at pg-gpu01 ~]# cd /etc/modprobe.d/
>>>>>>>
>>>>>>> [root at pg-gpu01 modprobe.d]# ls
>>>>>>> anaconda.conf   blacklist-kvm.conf      dist-alsa.conf
>>>>>>> dist-oss.conf           ib_ipoib.conf  lustre.conf  openfwwf.conf
>>>>>>> blacklist.conf  blacklist-nouveau.conf  dist.conf
>>>>>>> freeipmi-modalias.conf  ib_sdp.conf    mlnx.conf    truescale.conf
>>>>>>>
>>>>>>> [root at pg-gpu01 modprobe.d]# cat ./ib_ipoib.conf
>>>>>>> alias netdev-ib* ib_ipoib
>>>>>>>
>>>>>>> [root at pg-gpu01 modprobe.d]# cat ./mlnx.conf
>>>>>>> # Module parameters for MLNX_OFED kernel modules
>>>>>>>
>>>>>>> [root at pg-gpu01 modprobe.d]# cat ./lustre.conf
>>>>>>> options lnet networks=o2ib(ib0)
>>>>>>>
>>>>>>> Are there more Lustre/LNET options that could help in this situation?
>>>>>>>
>>>>>>
>>>>>> What about the logfiles?
>>>>>> Any error messages in syslog? lctl debug options?
>>>>>> Veel geluk,
>>>>>> Eli
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Apr 24, 2017 at 7:02 PM, Raj <rajgautam at gmail.com> wrote:
>>>>>>>
>>>>>>>> May be worth checking your lnet credits and peer_credits in
>>>>>>>> /etc/modprobe.d ?
>>>>>>>> You can compare between working hosts and non working hosts.
>>>>>>>> Thanks
>>>>>>>> _Raj
>>>>>>>>
>>>>>>>> On Mon, Apr 24, 2017 at 10:10 AM Strikwerda, Ger <
>>>>>>>> g.j.c.strikwerda at rug.nl> wrote:
>>>>>>>>
>>>>>>>>> Hi Rick,
>>>>>>>>>
>>>>>>>>> Even without iptables rules and loading the correct modules
>>>>>>>>> afterwards, we get the same results:
>>>>>>>>>
>>>>>>>>> [root at pg-gpu01 sysconfig]# iptables --list
>>>>>>>>> Chain INPUT (policy ACCEPT)
>>>>>>>>> target     prot opt source               destination
>>>>>>>>>
>>>>>>>>> Chain FORWARD (policy ACCEPT)
>>>>>>>>> target     prot opt source               destination
>>>>>>>>>
>>>>>>>>> Chain OUTPUT (policy ACCEPT)
>>>>>>>>> target     prot opt source               destination
>>>>>>>>>
>>>>>>>>> Chain LOGDROP (0 references)
>>>>>>>>> target     prot opt source               destination
>>>>>>>>> LOG        all  --  anywhere             anywhere            LOG
>>>>>>>>> level warning
>>>>>>>>> DROP       all  --  anywhere             anywhere
>>>>>>>>>
>>>>>>>>> [root at pg-gpu01 sysconfig]# modprobe lnet
>>>>>>>>>
>>>>>>>>> [root at pg-gpu01 sysconfig]# modprobe lustre
>>>>>>>>>
>>>>>>>>> [root at pg-gpu01 sysconfig]# lctl ping 172.23.55.211 at o2ib
>>>>>>>>>
>>>>>>>>> failed to ping 172.23.55.211 at o2ib: Input/output error
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Apr 24, 2017 at 4:59 PM, Mohr Jr, Richard Frank (Rick
>>>>>>>>> Mohr) <rmohr at utk.edu> wrote:
>>>>>>>>>
>>>>>>>>>> This might be a long shot, but have you checked for possible
>>>>>>>>>> firewall rules that might be causing the issue?  I’m wondering if there is
>>>>>>>>>> a chance that some rules were added after the nodes were up to allow Lustre
>>>>>>>>>> access, and when a node got rebooted, it lost the rules.
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Rick Mohr
>>>>>>>>>> Senior HPC System Administrator
>>>>>>>>>> National Institute for Computational Sciences
>>>>>>>>>> http://www.nics.tennessee.edu
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> > On Apr 24, 2017, at 10:19 AM, Strikwerda, Ger <
>>>>>>>>>> g.j.c.strikwerda at rug.nl> wrote:
>>>>>>>>>> >
>>>>>>>>>> > Hi Russell,
>>>>>>>>>> >
>>>>>>>>>> > Thanks for the IB subnet clues:
>>>>>>>>>> >
>>>>>>>>>> > [root at pg-gpu01 ~]# ibv_devinfo
>>>>>>>>>> > hca_id: mlx4_0
>>>>>>>>>> >         transport:                      InfiniBand (0)
>>>>>>>>>> >         fw_ver:                         2.32.5100
>>>>>>>>>> >         node_guid:                      f452:1403:00f5:4620
>>>>>>>>>> >         sys_image_guid:                 f452:1403:00f5:4623
>>>>>>>>>> >         vendor_id:                      0x02c9
>>>>>>>>>> >         vendor_part_id:                 4099
>>>>>>>>>> >         hw_ver:                         0x1
>>>>>>>>>> >         board_id:                       MT_1100120019
>>>>>>>>>> >         phys_port_cnt:                  1
>>>>>>>>>> >                 port:   1
>>>>>>>>>> >                         state:                  PORT_ACTIVE (4)
>>>>>>>>>> >                         max_mtu:                4096 (5)
>>>>>>>>>> >                         active_mtu:             4096 (5)
>>>>>>>>>> >                         sm_lid:                 1
>>>>>>>>>> >                         port_lid:               185
>>>>>>>>>> >                         port_lmc:               0x00
>>>>>>>>>> >                         link_layer:             InfiniBand
>>>>>>>>>> >
>>>>>>>>>> > [root at pg-gpu01 ~]# sminfo
>>>>>>>>>> > sminfo: sm lid 1 sm guid 0xf452140300f62320, activity count
>>>>>>>>>> 80878098 priority 0 state 3 SMINFO_MASTER
>>>>>>>>>> >
>>>>>>>>>> > Looks like the rebooted node is able to connect/contact IB/IB
>>>>>>>>>> subnetmanager
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> > On Mon, Apr 24, 2017 at 4:14 PM, Russell Dekema <
>>>>>>>>>> dekemar at umich.edu> wrote:
>>>>>>>>>> > At first glance, this sounds like your Infiniband subnet
>>>>>>>>>> manager may
>>>>>>>>>> > be down or malfunctioning. In this case, nodes which were
>>>>>>>>>> already up
>>>>>>>>>> > when the subnet manager was working will continue to be able to
>>>>>>>>>> > communicate over IB, but nodes which reboot after the SM goes
>>>>>>>>>> down
>>>>>>>>>> > will not.
>>>>>>>>>> >
>>>>>>>>>> > You can test this theory by running the 'ibv_devinfo' command
>>>>>>>>>> on one
>>>>>>>>>> > of your rebooted nodes. If the relevant IB port is in state
>>>>>>>>>> PORT_INIT,
>>>>>>>>>> > this confirms there is a problem with your subnet manager.
>>>>>>>>>> >
>>>>>>>>>> > Sincerely,
>>>>>>>>>> > Rusty Dekema
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> > On Mon, Apr 24, 2017 at 9:57 AM, Strikwerda, Ger
>>>>>>>>>> > <g.j.c.strikwerda at rug.nl> wrote:
>>>>>>>>>> > > Hi everybody,
>>>>>>>>>> > >
>>>>>>>>>> > > Here at the university of Groningen we are now experiencing a
>>>>>>>>>> strange Lustre
>>>>>>>>>> > > error. If a client reboots, it fails to mount the Lustre
>>>>>>>>>> storage. The client
>>>>>>>>>> > > is not able to reach the MSG service. The storage and nodes
>>>>>>>>>> are
>>>>>>>>>> > > communicating over IB and unitil now without any problems. It
>>>>>>>>>> looks like an
>>>>>>>>>> > > issue inside LNET. Clients cannot LNET ping/connect the
>>>>>>>>>> metadata and or
>>>>>>>>>> > > storage. But the clients are able to LNET ping each other.
>>>>>>>>>> Clients which not
>>>>>>>>>> > > have been rebooted, are working fine and have their mounts on
>>>>>>>>>> our Lustre
>>>>>>>>>> > > filesystem.
>>>>>>>>>> > >
>>>>>>>>>> > > Lustre client log:
>>>>>>>>>> > >
>>>>>>>>>> > > Lustre: Lustre: Build Version:
>>>>>>>>>> 2.5.3-RC1--PRISTINE-2.6.32-573.el6.x86_64
>>>>>>>>>> > > LNet: Added LNI 172.23.54.51 at o2ib [8/256/0/180]
>>>>>>>>>> > >
>>>>>>>>>> > > LustreError: 15c-8: MGC172.23.55.211 at o2ib: The configuration
>>>>>>>>>> from log
>>>>>>>>>> > > 'pgdata01-client' failed (-5). This may be the result of
>>>>>>>>>> communication
>>>>>>>>>> > > errors between this node and the MGS, a bad configuration, or
>>>>>>>>>> other errors.
>>>>>>>>>> > > See the syslog for more information.
>>>>>>>>>> > > LustreError: 3812:0:(llite_lib.c:1046:ll_fill_super()) Unable
>>>>>>>>>> to process
>>>>>>>>>> > > log: -5
>>>>>>>>>> > > Lustre: Unmounted pgdata01-client
>>>>>>>>>> > > LustreError: 3812:0:(obd_mount.c:1325:lustre_fill_super())
>>>>>>>>>> Unable to mount
>>>>>>>>>> > > (-5)
>>>>>>>>>> > > LNetError: 2882:0:(o2iblnd_cb.c:2587:kiblnd_rejected())
>>>>>>>>>> 172.23.55.212 at o2ib
>>>>>>>>>> > > rejected: consumer defined fatal error
>>>>>>>>>> > > LNetError: 2882:0:(o2iblnd_cb.c:2587:kiblnd_rejected())
>>>>>>>>>> Skipped 1 previous
>>>>>>>>>> > > similar message
>>>>>>>>>> > > Lustre: 3765:0:(client.c:1918:ptlrpc_expire_one_request())
>>>>>>>>>> @@@ Request sent
>>>>>>>>>> > > has failed due to network error: [sent 1492789626/real
>>>>>>>>>> 1492789626]
>>>>>>>>>> > > req at ffff88105af2cc00 x1565303228072004/t0(0)
>>>>>>>>>> > > o250->MGC172.23.55.211 at o2ib@172.23.55.212 at o2ib:26/25 lens
>>>>>>>>>> 400/544 e 0 to 1
>>>>>>>>>> > > dl 1492789631 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
>>>>>>>>>> > > Lustre: 3765:0:(client.c:1918:ptlrpc_expire_one_request())
>>>>>>>>>> Skipped 1
>>>>>>>>>> > > previous similar message
>>>>>>>>>> > > LustreError: 3826:0:(client.c:1083:ptlrpc_import_delay_req())
>>>>>>>>>> @@@ send limit
>>>>>>>>>> > > expired   req at ffff882041ffc000 x1565303228071996/t0(0)
>>>>>>>>>> > > o101->MGC172.23.55.211 at o2ib@172.23.55.211 at o2ib:26/25 lens
>>>>>>>>>> 328/344 e 0 to 0
>>>>>>>>>> > > dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
>>>>>>>>>> > > LustreError: 3826:0:(client.c:1083:ptlrpc_import_delay_req())
>>>>>>>>>> Skipped 2
>>>>>>>>>> > > previous similar messages
>>>>>>>>>> > > LustreError: 15c-8: MGC172.23.55.211 at o2ib: The configuration
>>>>>>>>>> from log
>>>>>>>>>> > > 'pghome01-client' failed (-5). This may be the result of
>>>>>>>>>> communication
>>>>>>>>>> > > errors between this node and the MGS, a bad configuration, or
>>>>>>>>>> other errors.
>>>>>>>>>> > > See the syslog for more information.
>>>>>>>>>> > > LustreError: 3826:0:(llite_lib.c:1046:ll_fill_super()) Unable
>>>>>>>>>> to process
>>>>>>>>>> > > log: -5
>>>>>>>>>> > >
>>>>>>>>>> > > LNetError: 2882:0:(o2iblnd_cb.c:2587:kiblnd_rejected())
>>>>>>>>>> 172.23.55.212 at o2ib
>>>>>>>>>> > > rejected: consumer defined fatal error
>>>>>>>>>> > > LNetError: 2882:0:(o2iblnd_cb.c:2587:kiblnd_rejected())
>>>>>>>>>> Skipped 1 previous
>>>>>>>>>> > > similar message
>>>>>>>>>> > > LNet: 3755:0:(o2iblnd_cb.c:475:kiblnd_rx_complete()) Rx from
>>>>>>>>>> > > 172.23.55.211 at o2ib failed: 5
>>>>>>>>>> > > LNetError: 2882:0:(o2iblnd_cb.c:2587:kiblnd_rejected())
>>>>>>>>>> 172.23.55.211 at o2ib
>>>>>>>>>> > > rejected: consumer defined fatal error
>>>>>>>>>> > > LNetError: 2882:0:(o2iblnd_cb.c:2587:kiblnd_rejected())
>>>>>>>>>> Skipped 1 previous
>>>>>>>>>> > > similar message
>>>>>>>>>> > > LNet: 2882:0:(o2iblnd_cb.c:2072:kiblnd_peer_connect_failed())
>>>>>>>>>> Deleting
>>>>>>>>>> > > messages for 172.23.55.211 at o2ib: connection failed
>>>>>>>>>> > > LNet: 2882:0:(o2iblnd_cb.c:2072:kiblnd_peer_connect_failed())
>>>>>>>>>> Deleting
>>>>>>>>>> > > messages for 172.23.55.212 at o2ib: connection failed
>>>>>>>>>> > > LNet: 3754:0:(o2iblnd_cb.c:475:kiblnd_rx_complete()) Rx from
>>>>>>>>>> > > 172.23.55.212 at o2ib failed: 5
>>>>>>>>>> > > LNet: 3754:0:(o2iblnd_cb.c:475:kiblnd_rx_complete()) Skipped
>>>>>>>>>> 17 previous
>>>>>>>>>> > > similar messages
>>>>>>>>>> > > LNet: 2882:0:(o2iblnd_cb.c:2072:kiblnd_peer_connect_failed())
>>>>>>>>>> Deleting
>>>>>>>>>> > > messages for 172.23.55.211 at o2ib: connection failed
>>>>>>>>>> > > LNet: 3754:0:(o2iblnd_cb.c:475:kiblnd_rx_complete()) Rx from
>>>>>>>>>> > > 172.23.55.212 at o2ib failed: 5
>>>>>>>>>> > > LNet: 2882:0:(o2iblnd_cb.c:2072:kiblnd_peer_connect_failed())
>>>>>>>>>> Deleting
>>>>>>>>>> > > messages for 172.23.55.212 at o2ib: connection failed
>>>>>>>>>> > >
>>>>>>>>>> > > LNET ping of a metadata-node:
>>>>>>>>>> > >
>>>>>>>>>> > > [root at pg-gpu01 ~]# lctl ping 172.23.55.211 at o2ib
>>>>>>>>>> > > failed to ping 172.23.55.211 at o2ib: Input/output error
>>>>>>>>>> > >
>>>>>>>>>> > > LNET ping of the number 2 metadata-node:
>>>>>>>>>> > >
>>>>>>>>>> > > [root at pg-gpu01 ~]# lctl ping 172.23.55.212 at o2ib
>>>>>>>>>> > > failed to ping 172.23.55.212 at o2ib: Input/output error
>>>>>>>>>> > >
>>>>>>>>>> > > LNET ping of a random compute-node:
>>>>>>>>>> > >
>>>>>>>>>> > > [root at pg-gpu01 ~]# lctl ping 172.23.52.5 at o2ib
>>>>>>>>>> > > 12345-0 at lo
>>>>>>>>>> > > 12345-172.23.52.5 at o2ib
>>>>>>>>>> > >
>>>>>>>>>> > > LNET to OST01:
>>>>>>>>>> > >
>>>>>>>>>> > > [root at pg-gpu01 ~]# lctl ping 172.23.55.201 at o2ib
>>>>>>>>>> > > failed to ping 172.23.55.201 at o2ib: Input/output error
>>>>>>>>>> > >
>>>>>>>>>> > > LNET to OST02:
>>>>>>>>>> > >
>>>>>>>>>> > > [root at pg-gpu01 ~]# lctl ping 172.23.55.202 at o2ib
>>>>>>>>>> > > failed to ping 172.23.55.202 at o2ib: Input/output error
>>>>>>>>>> > >
>>>>>>>>>> > > 'normal' pings (on ip level) works fine:
>>>>>>>>>> > >
>>>>>>>>>> > > [root at pg-gpu01 ~]# ping 172.23.55.201
>>>>>>>>>> > > PING 172.23.55.201 (172.23.55.201) 56(84) bytes of data.
>>>>>>>>>> > > 64 bytes from 172.23.55.201: icmp_seq=1 ttl=64 time=0.741 ms
>>>>>>>>>> > >
>>>>>>>>>> > > [root at pg-gpu01 ~]# ping 172.23.55.202
>>>>>>>>>> > > PING 172.23.55.202 (172.23.55.202) 56(84) bytes of data.
>>>>>>>>>> > > 64 bytes from 172.23.55.202: icmp_seq=1 ttl=64 time=0.704 ms
>>>>>>>>>> > >
>>>>>>>>>> > > lctl on a rebooted node:
>>>>>>>>>> > >
>>>>>>>>>> > > [root at pg-gpu01 ~]# lctl dl
>>>>>>>>>> > >
>>>>>>>>>> > > lctl on a not rebooted node:
>>>>>>>>>> > >
>>>>>>>>>> > > [root at pg-node005 ~]# lctl dl
>>>>>>>>>> > >   0 UP mgc MGC172.23.55.211 at o2ib
>>>>>>>>>> 94bd1c8a-512f-b920-9a4e-a6aced3d386d 5
>>>>>>>>>> > >   1 UP lov pgtemp01-clilov-ffff88206906d400
>>>>>>>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 4
>>>>>>>>>> > >   2 UP lmv pgtemp01-clilmv-ffff88206906d400
>>>>>>>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 4
>>>>>>>>>> > >   3 UP mdc pgtemp01-MDT0000-mdc-ffff88206906d400
>>>>>>>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5
>>>>>>>>>> > >   4 UP osc pgtemp01-OST0001-osc-ffff88206906d400
>>>>>>>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5
>>>>>>>>>> > >   5 UP osc pgtemp01-OST0003-osc-ffff88206906d400
>>>>>>>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5
>>>>>>>>>> > >   6 UP osc pgtemp01-OST0005-osc-ffff88206906d400
>>>>>>>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5
>>>>>>>>>> > >   7 UP osc pgtemp01-OST0007-osc-ffff88206906d400
>>>>>>>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5
>>>>>>>>>> > >   8 UP osc pgtemp01-OST0009-osc-ffff88206906d400
>>>>>>>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5
>>>>>>>>>> > >   9 UP osc pgtemp01-OST000b-osc-ffff88206906d400
>>>>>>>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5
>>>>>>>>>> > >  10 UP osc pgtemp01-OST000d-osc-ffff88206906d400
>>>>>>>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5
>>>>>>>>>> > >  11 UP osc pgtemp01-OST000f-osc-ffff88206906d400
>>>>>>>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5
>>>>>>>>>> > >  12 UP osc pgtemp01-OST0011-osc-ffff88206906d400
>>>>>>>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5
>>>>>>>>>> > >  13 UP osc pgtemp01-OST0002-osc-ffff88206906d400
>>>>>>>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5
>>>>>>>>>> > >  14 UP osc pgtemp01-OST0004-osc-ffff88206906d400
>>>>>>>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5
>>>>>>>>>> > >  15 UP osc pgtemp01-OST0006-osc-ffff88206906d400
>>>>>>>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5
>>>>>>>>>> > >  16 UP osc pgtemp01-OST0008-osc-ffff88206906d400
>>>>>>>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5
>>>>>>>>>> > >  17 UP osc pgtemp01-OST000a-osc-ffff88206906d400
>>>>>>>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5
>>>>>>>>>> > >  18 UP osc pgtemp01-OST000c-osc-ffff88206906d400
>>>>>>>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5
>>>>>>>>>> > >  19 UP osc pgtemp01-OST000e-osc-ffff88206906d400
>>>>>>>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5
>>>>>>>>>> > >  20 UP osc pgtemp01-OST0010-osc-ffff88206906d400
>>>>>>>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5
>>>>>>>>>> > >  21 UP osc pgtemp01-OST0012-osc-ffff88206906d400
>>>>>>>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5
>>>>>>>>>> > >  22 UP osc pgtemp01-OST0013-osc-ffff88206906d400
>>>>>>>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5
>>>>>>>>>> > >  23 UP osc pgtemp01-OST0015-osc-ffff88206906d400
>>>>>>>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5
>>>>>>>>>> > >  24 UP osc pgtemp01-OST0017-osc-ffff88206906d400
>>>>>>>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5
>>>>>>>>>> > >  25 UP osc pgtemp01-OST0014-osc-ffff88206906d400
>>>>>>>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5
>>>>>>>>>> > >  26 UP osc pgtemp01-OST0016-osc-ffff88206906d400
>>>>>>>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5
>>>>>>>>>> > >  27 UP osc pgtemp01-OST0018-osc-ffff88206906d400
>>>>>>>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5
>>>>>>>>>> > >  28 UP lov pgdata01-clilov-ffff88204bab6400
>>>>>>>>>> > > 996b1742-82eb-281c-c322-e244672d5225 4
>>>>>>>>>> > >  29 UP lmv pgdata01-clilmv-ffff88204bab6400
>>>>>>>>>> > > 996b1742-82eb-281c-c322-e244672d5225 4
>>>>>>>>>> > >  30 UP mdc pgdata01-MDT0000-mdc-ffff88204bab6400
>>>>>>>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5
>>>>>>>>>> > >  31 UP osc pgdata01-OST0001-osc-ffff88204bab6400
>>>>>>>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5
>>>>>>>>>> > >  32 UP osc pgdata01-OST0003-osc-ffff88204bab6400
>>>>>>>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5
>>>>>>>>>> > >  33 UP osc pgdata01-OST0005-osc-ffff88204bab6400
>>>>>>>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5
>>>>>>>>>> > >  34 UP osc pgdata01-OST0007-osc-ffff88204bab6400
>>>>>>>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5
>>>>>>>>>> > >  35 UP osc pgdata01-OST0009-osc-ffff88204bab6400
>>>>>>>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5
>>>>>>>>>> > >  36 UP osc pgdata01-OST000b-osc-ffff88204bab6400
>>>>>>>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5
>>>>>>>>>> > >  37 UP osc pgdata01-OST000d-osc-ffff88204bab6400
>>>>>>>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5
>>>>>>>>>> > >  38 UP osc pgdata01-OST000f-osc-ffff88204bab6400
>>>>>>>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5
>>>>>>>>>> > >  39 UP osc pgdata01-OST0002-osc-ffff88204bab6400
>>>>>>>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5
>>>>>>>>>> > >  40 UP osc pgdata01-OST0004-osc-ffff88204bab6400
>>>>>>>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5
>>>>>>>>>> > >  41 UP osc pgdata01-OST0006-osc-ffff88204bab6400
>>>>>>>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5
>>>>>>>>>> > >  42 UP osc pgdata01-OST0008-osc-ffff88204bab6400
>>>>>>>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5
>>>>>>>>>> > >  43 UP osc pgdata01-OST000a-osc-ffff88204bab6400
>>>>>>>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5
>>>>>>>>>> > >  44 UP osc pgdata01-OST000c-osc-ffff88204bab6400
>>>>>>>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5
>>>>>>>>>> > >  45 UP osc pgdata01-OST000e-osc-ffff88204bab6400
>>>>>>>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5
>>>>>>>>>> > >  46 UP osc pgdata01-OST0010-osc-ffff88204bab6400
>>>>>>>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5
>>>>>>>>>> > >  47 UP osc pgdata01-OST0013-osc-ffff88204bab6400
>>>>>>>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5
>>>>>>>>>> > >  48 UP osc pgdata01-OST0015-osc-ffff88204bab6400
>>>>>>>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5
>>>>>>>>>> > >  49 UP osc pgdata01-OST0017-osc-ffff88204bab6400
>>>>>>>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5
>>>>>>>>>> > >  50 UP osc pgdata01-OST0014-osc-ffff88204bab6400
>>>>>>>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5
>>>>>>>>>> > >  51 UP osc pgdata01-OST0016-osc-ffff88204bab6400
>>>>>>>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5
>>>>>>>>>> > >  52 UP osc pgdata01-OST0018-osc-ffff88204bab6400
>>>>>>>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5
>>>>>>>>>> > >  53 UP osc pgdata01-OST0019-osc-ffff88204bab6400
>>>>>>>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5
>>>>>>>>>> > >  54 UP osc pgdata01-OST001a-osc-ffff88204bab6400
>>>>>>>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5
>>>>>>>>>> > >  55 UP osc pgdata01-OST001b-osc-ffff88204bab6400
>>>>>>>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5
>>>>>>>>>> > >  56 UP lov pghome01-clilov-ffff88204bb50000
>>>>>>>>>> > > 9ae8f2a9-1cdf-901f-160c-66f70e4c10d1 4
>>>>>>>>>> > >  57 UP lmv pghome01-clilmv-ffff88204bb50000
>>>>>>>>>> > > 9ae8f2a9-1cdf-901f-160c-66f70e4c10d1 4
>>>>>>>>>> > >  58 UP mdc pghome01-MDT0000-mdc-ffff88204bb50000
>>>>>>>>>> > > 9ae8f2a9-1cdf-901f-160c-66f70e4c10d1 5
>>>>>>>>>> > >  59 UP osc pghome01-OST0011-osc-ffff88204bb50000
>>>>>>>>>> > > 9ae8f2a9-1cdf-901f-160c-66f70e4c10d1 5
>>>>>>>>>> > >  60 UP osc pghome01-OST0012-osc-ffff88204bb50000
>>>>>>>>>> > > 9ae8f2a9-1cdf-901f-160c-66f70e4c10d1 5
>>>>>>>>>> > >
>>>>>>>>>> > > Please help, any clues/advice/hints/tips are appricated
>>>>>>>>>> > >
>>>>>>>>>> > > --
>>>>>>>>>> > >
>>>>>>>>>> > > Vriendelijke groet,
>>>>>>>>>> > >
>>>>>>>>>> > > Ger Strikwerda
>>>>>>>>>> > > Chef Special
>>>>>>>>>> > > Rijksuniversiteit Groningen
>>>>>>>>>> > > Centrum voor Informatie Technologie
>>>>>>>>>> > > Unit Pragmatisch Systeembeheer
>>>>>>>>>> > >
>>>>>>>>>> > > Smitsborg
>>>>>>>>>> > > Nettelbosje 1
>>>>>>>>>> > > 9747 AJ Groningen
>>>>>>>>>> > > Tel. 050 363 9276
>>>>>>>>>> > >
>>>>>>>>>> > > "God is hard, God is fair
>>>>>>>>>> > >  some men he gave brains, others he gave hair"
>>>>>>>>>> > >
>>>>>>>>>> > >
>>>>>>>>>> > > _______________________________________________
>>>>>>>>>> > > lustre-discuss mailing list
>>>>>>>>>> > > lustre-discuss at lists.lustre.org
>>>>>>>>>> > >
>>>>>>>>>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>>>>>>>>>> > >
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> > --
>>>>>>>>>> > Vriendelijke groet,
>>>>>>>>>> >
>>>>>>>>>> > Ger Strikwerda
>>>>>>>>>> >
>>>>>>>>>> > Chef Special
>>>>>>>>>> > Rijksuniversiteit Groningen
>>>>>>>>>> > Centrum voor Informatie Technologie
>>>>>>>>>> > Unit Pragmatisch Systeembeheer
>>>>>>>>>> >
>>>>>>>>>> > Smitsborg
>>>>>>>>>> > Nettelbosje 1
>>>>>>>>>> > 9747 AJ Groningen
>>>>>>>>>> > Tel. 050 363 9276
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> > "God is hard, God is fair
>>>>>>>>>> >  some men he gave brains, others he gave hair"
>>>>>>>>>> > _______________________________________________
>>>>>>>>>> > lustre-discuss mailing list
>>>>>>>>>> > lustre-discuss at lists.lustre.org
>>>>>>>>>> > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>>
>>>>>>>>> Vriendelijke groet,
>>>>>>>>>
>>>>>>>>> Ger StrikwerdaChef Special
>>>>>>>>> Rijksuniversiteit Groningen
>>>>>>>>> Centrum voor Informatie Technologie
>>>>>>>>> Unit Pragmatisch Systeembeheer
>>>>>>>>>
>>>>>>>>> Smitsborg
>>>>>>>>> Nettelbosje 1
>>>>>>>>> 9747 AJ Groningen
>>>>>>>>> Tel. 050 363 9276
>>>>>>>>> "God is hard, God is fair
>>>>>>>>>  some men he gave brains, others he gave hair"
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> lustre-discuss mailing list
>>>>>>>>> lustre-discuss at lists.lustre.org
>>>>>>>>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> Vriendelijke groet,
>>>>>>>
>>>>>>> Ger StrikwerdaChef Special
>>>>>>> Rijksuniversiteit Groningen
>>>>>>> Centrum voor Informatie Technologie
>>>>>>> Unit Pragmatisch Systeembeheer
>>>>>>>
>>>>>>> Smitsborg
>>>>>>> Nettelbosje 1
>>>>>>> 9747 AJ Groningen
>>>>>>> Tel. 050 363 9276
>>>>>>> "God is hard, God is fair
>>>>>>>  some men he gave brains, others he gave hair"
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> lustre-discuss mailing list
>>>>>>> lustre-discuss at lists.lustre.org
>>>>>>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Vriendelijke groet,
>>>>>
>>>>> Ger StrikwerdaChef Special
>>>>> Rijksuniversiteit Groningen
>>>>> Centrum voor Informatie Technologie
>>>>> Unit Pragmatisch Systeembeheer
>>>>>
>>>>> Smitsborg
>>>>> Nettelbosje 1
>>>>> 9747 AJ Groningen
>>>>> Tel. 050 363 9276
>>>>> "God is hard, God is fair
>>>>>  some men he gave brains, others he gave hair"
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>>
>>> Vriendelijke groet,
>>>
>>> Ger StrikwerdaChef Special
>>> Rijksuniversiteit Groningen
>>> Centrum voor Informatie Technologie
>>> Unit Pragmatisch Systeembeheer
>>>
>>> Smitsborg
>>> Nettelbosje 1
>>> 9747 AJ Groningen
>>> Tel. 050 363 9276
>>> "God is hard, God is fair
>>>  some men he gave brains, others he gave hair"
>>>
>>>
>
>
> --
>
> Vriendelijke groet,
>
> Ger StrikwerdaChef Special
> Rijksuniversiteit Groningen
> Centrum voor Informatie Technologie
> Unit Pragmatisch Systeembeheer
>
> Smitsborg
> Nettelbosje 1
> 9747 AJ Groningen
> Tel. 050 363 9276
> "God is hard, God is fair
>  some men he gave brains, others he gave hair"
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20170424/a9a13160/attachment-0001.htm>


More information about the lustre-discuss mailing list