[lustre-discuss] client fails to mount

Mon Apr 24 07:51:58 PDT 2017

Hi Russell,

On a rebooted node:

[root at pg-gpu01 ~]# ibhosts | wc -l
183

On a not-rebooted node:

[root at pg-gpu02 ~]# ibhosts | wc -l
183

No diffence and all our lustre storage nodes seems to be present:

Ca      : 0xf45214030062eb50 ports 2 "pg-ost01 HCA-1"
Ca      : 0xf45214030062eb30 ports 2 "pg-mds02 HCA-1"
Ca      : 0xf45214030062ee60 ports 2 "pg-ost03 HCA-1"
Ca      : 0xf45214030062ee90 ports 2 "pg-mds01 HCA-1"
Ca      : 0xf45214030062eb00 ports 2 "pg-ost04 HCA-1"
Ca      : 0xe41d2d030001ce80 ports 2 "pg-ost02 HCA-1"

On Mon, Apr 24, 2017 at 4:46 PM, Russell Dekema <dekemar at umich.edu> wrote:

> I'm not sure this is likely to help either, but if you run the command
> 'ibhosts' on one of the non-working Lustre client nodes, do you see
> all of your Lustre servers in the printed list?
>
> -Rusty
>
> On Mon, Apr 24, 2017 at 10:39 AM, Russell Dekema <dekemar at umich.edu>
> wrote:
> > I can't rule it out, but it seems unlikely to me that an out of date
> > IB HCA firmware version would cause a problem like this, especially
> > when everything was working before on that same version, and when IB
> > communication over the device seems to be working in general (as shown
> > by your pings over your IPoIB interfaces).
> >
> > If you decide to update your HCA (or switch, in the case of
> > switch-f0cc8e/U1) firmware, you may want to check with the vendor
> > before doing so. In the past, they have sometimes told me that the
> > "latest FW version available for this device" reported by ibdiagnet is
> > incorrect and should be ignored. Of course, in other cases, newer
> > firmware versions were in fact available and they did recommend
> > upgrading.
> >
> > Sincerely,
> > Rusty D.
> >
> >
> >
> > On Mon, Apr 24, 2017 at 10:32 AM, Strikwerda, Ger
> > <g.j.c.strikwerda at rug.nl> wrote:
> >> Hi Russell/*,
> >>
> >> If we run ibdiagnet we get errors/warnings about some (newer) nodes
> which
> >> happen to have a new firmware on the IB interface:
> >>
> >> Nodes Information
> >> -E- FW Check finished with errors
> >> -W- pg-gpu01/U1 - Node has FW version 2.32.5100 while the latest FW
> version,
> >> for the same device available on this fabric is 2.36.5150
> >> -W- pg-gpu05/U1 - Node has FW version 2.35.5100 while the latest FW
> version,
> >> for the same device available on this fabric is 2.36.5150
> >> -W- pg-gpu06/U1 - Node has FW version 2.35.5100 while the latest FW
> version,
> >> for the same device available on this fabric is 2.36.5150
> >> -W- pg-gpu02/U1 - Node has FW version 2.32.5100 while the latest FW
> version,
> >> for the same device available on this fabric is 2.36.5150
> >> -W- pg-gpu03/U1 - Node has FW version 2.32.5100 while the latest FW
> version,
> >> for the same device available on this fabric is 2.36.5150
> >> -W- pg-gpu04/U1 - Node has FW version 2.32.5100 while the latest FW
> version,
> >> for the same device available on this fabric is 2.36.5150
> >> -W- switch-f0cc8e/U1 - Node has FW version 9.2.7300 while the latest FW
> >> version, for the same device available on this fabric is 9.2.8000
> >> -W- pg-memory02/U1 - Node has FW version 2.32.5100 while the latest FW
> >> version, for the same device available on this fabric is 2.36.5150
> >> -W- pg-memory01/U1 - Node has FW version 2.32.5100 while the latest FW
> >> version, for the same device available on this fabric is 2.36.5150
> >> -W- pg-memory03/U1 - Node has FW version 2.32.5100 while the latest FW
> >> version, for the same device available on this fabric is 2.36.5150
> >> -W- pg-memory04/U1 - Node has FW version 2.32.5100 while the latest FW
> >> version, for the same device available on this fabric is 2.36.5150
> >> -W- pg-memory06/U1 - Node has FW version 2.32.5100 while the latest FW
> >> version, for the same device available on this fabric is 2.36.5150
> >> -W- pg-memory05/U1 - Node has FW version 2.32.5100 while the latest FW
> >> version, for the same device available on this fabric is 2.36.5150
> >> -W- pg-node163/U1 - Node has FW version 2.36.5000 while the latest FW
> >> version, for the same device available on this fabric is 2.36.5150
> >> -W- pg-node164/U1 - Node has FW version 2.36.5000 while the latest FW
> >> version, for the same device available on this fabric is 2.36.5150
> >> -W- pg-node001/U1 - Node has FW version 2.33.5100 while the latest FW
> >> version, for the same device available on this fabric is 2.36.5150
> >> -W- pg-interactive/U1 - Node has FW version 2.32.5100 while the latest
> FW
> >> version, for the same device available on this fabric is 2.36.5150
> >> -W- peregrine/U1 - Node has FW version 2.32.5100 while the latest FW
> >> version, for the same device available on this fabric is 2.36.5150
> >> -W- pg-node004/U1 - Node has FW version 2.32.5100 while the latest FW
> >> version, for the same device available on this fabric is 2.36.5150
> >> -W- pg-node005/U1 - Node has FW version 2.32.5100 while the latest FW
> >> version, for the same device available on this fabric is 2.36.5150
> >> -W- pg-node006/U1 - Node has FW version 2.32.5100 while the latest FW
> >> version, for the same device available on this fabric is 2.36.5150
> >> -W- pg-node007/U1 - Node has FW version 2.32.5100 while the latest FW
> >> version, for the same device available on this fabric is 2.36.5150
> >> -W- pg-node008/U1 - Node has FW version 2.32.5100 while the latest FW
> >> version, for the same device available on this fabric is 2.36.5150
> >> -W- pg-node009/U1 - Node has FW version 2.32.5100 while the latest FW
> >> version, for the same device available on this fabric is 2.36.5150
> >> -W- pg-node010/U1 - Node has FW version 2.32.5100 while the latest FW
> >> version, for the same device available on this fabric is 2.36.5150
> >> -W- pg-node011/U1 - Node has FW version 2.32.5100 while the latest FW
> >> version, for the same device available on this fabric is 2.36.5150
> >> -W- pg-node012/U1 - Node has FW version 2.32.5100 while the latest FW
> >> version, for the same device available on this fabric is 2.36.5150
> >> -W- pg-node013/U1 - Node has FW version 2.32.5100 while the latest FW
> >> version, for the same device available on this fabric is 2.36.5150
> >> -W- pg-node014/U1 - Node has FW version 2.32.5100 while the latest FW
> >> version, for the same device available on this fabric is 2.36.5150
> >> -W- pg-node015/U1 - Node has FW version 2.32.5100 while the latest FW
> >> version, for the same device available on this fabric is 2.36.5150
> >>
> >> Could that be problematic/an issue?
> >>
> >>
> >>
> >>
> >> On Mon, Apr 24, 2017 at 4:27 PM, Russell Dekema <dekemar at umich.edu>
> wrote:
> >>>
> >>> Oh, ok, that seems to rule the subnet manager out.
> >>>
> >>> I mis-read your IP network numbers earlier and thought you had not
> >>> tried regular IP-ping across your IPoIB interfaces, but, upon
> >>> re-reading your initial message, it seems you have tried this and it
> >>> does work, even between a client with non-working Lustre and your
> >>> MGS/MDS.
> >>>
> >>> In this case, I have no further suggestions.
> >>>
> >>> Best of luck,
> >>> Rusty D.
> >>>
> >>> On Mon, Apr 24, 2017 at 10:19 AM, Strikwerda, Ger
> >>> <g.j.c.strikwerda at rug.nl> wrote:
> >>> > Hi Russell,
> >>> >
> >>> > Thanks for the IB subnet clues:
> >>> >
> >>> > [root at pg-gpu01 ~]# ibv_devinfo
> >>> > hca_id: mlx4_0
> >>> >         transport:                      InfiniBand (0)
> >>> >         fw_ver:                         2.32.5100
> >>> >         node_guid:                      f452:1403:00f5:4620
> >>> >         sys_image_guid:                 f452:1403:00f5:4623
> >>> >         vendor_id:                      0x02c9
> >>> >         vendor_part_id:                 4099
> >>> >         hw_ver:                         0x1
> >>> >         board_id:                       MT_1100120019
> >>> >         phys_port_cnt:                  1
> >>> >                 port:   1
> >>> >                         state:                  PORT_ACTIVE (4)
> >>> >                         max_mtu:                4096 (5)
> >>> >                         active_mtu:             4096 (5)
> >>> >                         sm_lid:                 1
> >>> >                         port_lid:               185
> >>> >                         port_lmc:               0x00
> >>> >                         link_layer:             InfiniBand
> >>> >
> >>> > [root at pg-gpu01 ~]# sminfo
> >>> > sminfo: sm lid 1 sm guid 0xf452140300f62320, activity count 80878098
> >>> > priority 0 state 3 SMINFO_MASTER
> >>> >
> >>> > Looks like the rebooted node is able to connect/contact IB/IB
> >>> > subnetmanager
> >>> >
> >>> >
> >>> >
> >>> >
> >>> > On Mon, Apr 24, 2017 at 4:14 PM, Russell Dekema <dekemar at umich.edu>
> >>> > wrote:
> >>> >>
> >>> >> At first glance, this sounds like your Infiniband subnet manager may
> >>> >> be down or malfunctioning. In this case, nodes which were already up
> >>> >> when the subnet manager was working will continue to be able to
> >>> >> communicate over IB, but nodes which reboot after the SM goes down
> >>> >> will not.
> >>> >>
> >>> >> You can test this theory by running the 'ibv_devinfo' command on one
> >>> >> of your rebooted nodes. If the relevant IB port is in state
> PORT_INIT,
> >>> >> this confirms there is a problem with your subnet manager.
> >>> >>
> >>> >> Sincerely,
> >>> >> Rusty Dekema
> >>> >>
> >>> >>
> >>> >>
> >>> >>
> >>> >> On Mon, Apr 24, 2017 at 9:57 AM, Strikwerda, Ger
> >>> >> <g.j.c.strikwerda at rug.nl> wrote:
> >>> >> > Hi everybody,
> >>> >> >
> >>> >> > Here at the university of Groningen we are now experiencing a
> strange
> >>> >> > Lustre
> >>> >> > error. If a client reboots, it fails to mount the Lustre storage.
> The
> >>> >> > client
> >>> >> > is not able to reach the MSG service. The storage and nodes are
> >>> >> > communicating over IB and unitil now without any problems. It
> looks
> >>> >> > like
> >>> >> > an
> >>> >> > issue inside LNET. Clients cannot LNET ping/connect the metadata
> and
> >>> >> > or
> >>> >> > storage. But the clients are able to LNET ping each other. Clients
> >>> >> > which
> >>> >> > not
> >>> >> > have been rebooted, are working fine and have their mounts on our
> >>> >> > Lustre
> >>> >> > filesystem.
> >>> >> >
> >>> >> > Lustre client log:
> >>> >> >
> >>> >> > Lustre: Lustre: Build Version:
> >>> >> > 2.5.3-RC1--PRISTINE-2.6.32-573.el6.x86_64
> >>> >> > LNet: Added LNI 172.23.54.51 at o2ib [8/256/0/180]
> >>> >> >
> >>> >> > LustreError: 15c-8: MGC172.23.55.211 at o2ib: The configuration
> from log
> >>> >> > 'pgdata01-client' failed (-5). This may be the result of
> >>> >> > communication
> >>> >> > errors between this node and the MGS, a bad configuration, or
> other
> >>> >> > errors.
> >>> >> > See the syslog for more information.
> >>> >> > LustreError: 3812:0:(llite_lib.c:1046:ll_fill_super()) Unable to
> >>> >> > process
> >>> >> > log: -5
> >>> >> > Lustre: Unmounted pgdata01-client
> >>> >> > LustreError: 3812:0:(obd_mount.c:1325:lustre_fill_super())
> Unable to
> >>> >> > mount
> >>> >> > (-5)
> >>> >> > LNetError: 2882:0:(o2iblnd_cb.c:2587:kiblnd_rejected())
> >>> >> > 172.23.55.212 at o2ib
> >>> >> > rejected: consumer defined fatal error
> >>> >> > LNetError: 2882:0:(o2iblnd_cb.c:2587:kiblnd_rejected()) Skipped 1
> >>> >> > previous
> >>> >> > similar message
> >>> >> > Lustre: 3765:0:(client.c:1918:ptlrpc_expire_one_request()) @@@
> >>> >> > Request
> >>> >> > sent
> >>> >> > has failed due to network error: [sent 1492789626/real 1492789626]
> >>> >> > req at ffff88105af2cc00 x1565303228072004/t0(0)
> >>> >> > o250->MGC172.23.55.211 at o2ib@172.23.55.212 at o2ib:26/25 lens
> 400/544 e 0
> >>> >> > to
> >>> >> > 1
> >>> >> > dl 1492789631 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
> >>> >> > Lustre: 3765:0:(client.c:1918:ptlrpc_expire_one_request())
> Skipped 1
> >>> >> > previous similar message
> >>> >> > LustreError: 3826:0:(client.c:1083:ptlrpc_import_delay_req()) @@@
> >>> >> > send
> >>> >> > limit
> >>> >> > expired   req at ffff882041ffc000 x1565303228071996/t0(0)
> >>> >> > o101->MGC172.23.55.211 at o2ib@172.23.55.211 at o2ib:26/25 lens
> 328/344 e 0
> >>> >> > to
> >>> >> > 0
> >>> >> > dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
> >>> >> > LustreError: 3826:0:(client.c:1083:ptlrpc_import_delay_req())
> Skipped
> >>> >> > 2
> >>> >> > previous similar messages
> >>> >> > LustreError: 15c-8: MGC172.23.55.211 at o2ib: The configuration
> from log
> >>> >> > 'pghome01-client' failed (-5). This may be the result of
> >>> >> > communication
> >>> >> > errors between this node and the MGS, a bad configuration, or
> other
> >>> >> > errors.
> >>> >> > See the syslog for more information.
> >>> >> > LustreError: 3826:0:(llite_lib.c:1046:ll_fill_super()) Unable to
> >>> >> > process
> >>> >> > log: -5
> >>> >> >
> >>> >> > LNetError: 2882:0:(o2iblnd_cb.c:2587:kiblnd_rejected())
> >>> >> > 172.23.55.212 at o2ib
> >>> >> > rejected: consumer defined fatal error
> >>> >> > LNetError: 2882:0:(o2iblnd_cb.c:2587:kiblnd_rejected()) Skipped 1
> >>> >> > previous
> >>> >> > similar message
> >>> >> > LNet: 3755:0:(o2iblnd_cb.c:475:kiblnd_rx_complete()) Rx from
> >>> >> > 172.23.55.211 at o2ib failed: 5
> >>> >> > LNetError: 2882:0:(o2iblnd_cb.c:2587:kiblnd_rejected())
> >>> >> > 172.23.55.211 at o2ib
> >>> >> > rejected: consumer defined fatal error
> >>> >> > LNetError: 2882:0:(o2iblnd_cb.c:2587:kiblnd_rejected()) Skipped 1
> >>> >> > previous
> >>> >> > similar message
> >>> >> > LNet: 2882:0:(o2iblnd_cb.c:2072:kiblnd_peer_connect_failed())
> >>> >> > Deleting
> >>> >> > messages for 172.23.55.211 at o2ib: connection failed
> >>> >> > LNet: 2882:0:(o2iblnd_cb.c:2072:kiblnd_peer_connect_failed())
> >>> >> > Deleting
> >>> >> > messages for 172.23.55.212 at o2ib: connection failed
> >>> >> > LNet: 3754:0:(o2iblnd_cb.c:475:kiblnd_rx_complete()) Rx from
> >>> >> > 172.23.55.212 at o2ib failed: 5
> >>> >> > LNet: 3754:0:(o2iblnd_cb.c:475:kiblnd_rx_complete()) Skipped 17
> >>> >> > previous
> >>> >> > similar messages
> >>> >> > LNet: 2882:0:(o2iblnd_cb.c:2072:kiblnd_peer_connect_failed())
> >>> >> > Deleting
> >>> >> > messages for 172.23.55.211 at o2ib: connection failed
> >>> >> > LNet: 3754:0:(o2iblnd_cb.c:475:kiblnd_rx_complete()) Rx from
> >>> >> > 172.23.55.212 at o2ib failed: 5
> >>> >> > LNet: 2882:0:(o2iblnd_cb.c:2072:kiblnd_peer_connect_failed())
> >>> >> > Deleting
> >>> >> > messages for 172.23.55.212 at o2ib: connection failed
> >>> >> >
> >>> >> > LNET ping of a metadata-node:
> >>> >> >
> >>> >> > [root at pg-gpu01 ~]# lctl ping 172.23.55.211 at o2ib
> >>> >> > failed to ping 172.23.55.211 at o2ib: Input/output error
> >>> >> >
> >>> >> > LNET ping of the number 2 metadata-node:
> >>> >> >
> >>> >> > [root at pg-gpu01 ~]# lctl ping 172.23.55.212 at o2ib
> >>> >> > failed to ping 172.23.55.212 at o2ib: Input/output error
> >>> >> >
> >>> >> > LNET ping of a random compute-node:
> >>> >> >
> >>> >> > [root at pg-gpu01 ~]# lctl ping 172.23.52.5 at o2ib
> >>> >> > 12345-0 at lo
> >>> >> > 12345-172.23.52.5 at o2ib
> >>> >> >
> >>> >> > LNET to OST01:
> >>> >> >
> >>> >> > [root at pg-gpu01 ~]# lctl ping 172.23.55.201 at o2ib
> >>> >> > failed to ping 172.23.55.201 at o2ib: Input/output error
> >>> >> >
> >>> >> > LNET to OST02:
> >>> >> >
> >>> >> > [root at pg-gpu01 ~]# lctl ping 172.23.55.202 at o2ib
> >>> >> > failed to ping 172.23.55.202 at o2ib: Input/output error
> >>> >> >
> >>> >> > 'normal' pings (on ip level) works fine:
> >>> >> >
> >>> >> > [root at pg-gpu01 ~]# ping 172.23.55.201
> >>> >> > PING 172.23.55.201 (172.23.55.201) 56(84) bytes of data.
> >>> >> > 64 bytes from 172.23.55.201: icmp_seq=1 ttl=64 time=0.741 ms
> >>> >> >
> >>> >> > [root at pg-gpu01 ~]# ping 172.23.55.202
> >>> >> > PING 172.23.55.202 (172.23.55.202) 56(84) bytes of data.
> >>> >> > 64 bytes from 172.23.55.202: icmp_seq=1 ttl=64 time=0.704 ms
> >>> >> >
> >>> >> > lctl on a rebooted node:
> >>> >> >
> >>> >> > [root at pg-gpu01 ~]# lctl dl
> >>> >> >
> >>> >> > lctl on a not rebooted node:
> >>> >> >
> >>> >> > [root at pg-node005 ~]# lctl dl
> >>> >> >   0 UP mgc MGC172.23.55.211 at o2ib 94bd1c8a-512f-b920-9a4e-
> a6aced3d386d
> >>> >> > 5
> >>> >> >   1 UP lov pgtemp01-clilov-ffff88206906d400
> >>> >> > 281c441f-8aa3-ab56-8812-e459d308f47c 4
> >>> >> >   2 UP lmv pgtemp01-clilmv-ffff88206906d400
> >>> >> > 281c441f-8aa3-ab56-8812-e459d308f47c 4
> >>> >> >   3 UP mdc pgtemp01-MDT0000-mdc-ffff88206906d400
> >>> >> > 281c441f-8aa3-ab56-8812-e459d308f47c 5
> >>> >> >   4 UP osc pgtemp01-OST0001-osc-ffff88206906d400
> >>> >> > 281c441f-8aa3-ab56-8812-e459d308f47c 5
> >>> >> >   5 UP osc pgtemp01-OST0003-osc-ffff88206906d400
> >>> >> > 281c441f-8aa3-ab56-8812-e459d308f47c 5
> >>> >> >   6 UP osc pgtemp01-OST0005-osc-ffff88206906d400
> >>> >> > 281c441f-8aa3-ab56-8812-e459d308f47c 5
> >>> >> >   7 UP osc pgtemp01-OST0007-osc-ffff88206906d400
> >>> >> > 281c441f-8aa3-ab56-8812-e459d308f47c 5
> >>> >> >   8 UP osc pgtemp01-OST0009-osc-ffff88206906d400
> >>> >> > 281c441f-8aa3-ab56-8812-e459d308f47c 5
> >>> >> >   9 UP osc pgtemp01-OST000b-osc-ffff88206906d400
> >>> >> > 281c441f-8aa3-ab56-8812-e459d308f47c 5
> >>> >> >  10 UP osc pgtemp01-OST000d-osc-ffff88206906d400
> >>> >> > 281c441f-8aa3-ab56-8812-e459d308f47c 5
> >>> >> >  11 UP osc pgtemp01-OST000f-osc-ffff88206906d400
> >>> >> > 281c441f-8aa3-ab56-8812-e459d308f47c 5
> >>> >> >  12 UP osc pgtemp01-OST0011-osc-ffff88206906d400
> >>> >> > 281c441f-8aa3-ab56-8812-e459d308f47c 5
> >>> >> >  13 UP osc pgtemp01-OST0002-osc-ffff88206906d400
> >>> >> > 281c441f-8aa3-ab56-8812-e459d308f47c 5
> >>> >> >  14 UP osc pgtemp01-OST0004-osc-ffff88206906d400
> >>> >> > 281c441f-8aa3-ab56-8812-e459d308f47c 5
> >>> >> >  15 UP osc pgtemp01-OST0006-osc-ffff88206906d400
> >>> >> > 281c441f-8aa3-ab56-8812-e459d308f47c 5
> >>> >> >  16 UP osc pgtemp01-OST0008-osc-ffff88206906d400
> >>> >> > 281c441f-8aa3-ab56-8812-e459d308f47c 5
> >>> >> >  17 UP osc pgtemp01-OST000a-osc-ffff88206906d400
> >>> >> > 281c441f-8aa3-ab56-8812-e459d308f47c 5
> >>> >> >  18 UP osc pgtemp01-OST000c-osc-ffff88206906d400
> >>> >> > 281c441f-8aa3-ab56-8812-e459d308f47c 5
> >>> >> >  19 UP osc pgtemp01-OST000e-osc-ffff88206906d400
> >>> >> > 281c441f-8aa3-ab56-8812-e459d308f47c 5
> >>> >> >  20 UP osc pgtemp01-OST0010-osc-ffff88206906d400
> >>> >> > 281c441f-8aa3-ab56-8812-e459d308f47c 5
> >>> >> >  21 UP osc pgtemp01-OST0012-osc-ffff88206906d400
> >>> >> > 281c441f-8aa3-ab56-8812-e459d308f47c 5
> >>> >> >  22 UP osc pgtemp01-OST0013-osc-ffff88206906d400
> >>> >> > 281c441f-8aa3-ab56-8812-e459d308f47c 5
> >>> >> >  23 UP osc pgtemp01-OST0015-osc-ffff88206906d400
> >>> >> > 281c441f-8aa3-ab56-8812-e459d308f47c 5
> >>> >> >  24 UP osc pgtemp01-OST0017-osc-ffff88206906d400
> >>> >> > 281c441f-8aa3-ab56-8812-e459d308f47c 5
> >>> >> >  25 UP osc pgtemp01-OST0014-osc-ffff88206906d400
> >>> >> > 281c441f-8aa3-ab56-8812-e459d308f47c 5
> >>> >> >  26 UP osc pgtemp01-OST0016-osc-ffff88206906d400
> >>> >> > 281c441f-8aa3-ab56-8812-e459d308f47c 5
> >>> >> >  27 UP osc pgtemp01-OST0018-osc-ffff88206906d400
> >>> >> > 281c441f-8aa3-ab56-8812-e459d308f47c 5
> >>> >> >  28 UP lov pgdata01-clilov-ffff88204bab6400
> >>> >> > 996b1742-82eb-281c-c322-e244672d5225 4
> >>> >> >  29 UP lmv pgdata01-clilmv-ffff88204bab6400
> >>> >> > 996b1742-82eb-281c-c322-e244672d5225 4
> >>> >> >  30 UP mdc pgdata01-MDT0000-mdc-ffff88204bab6400
> >>> >> > 996b1742-82eb-281c-c322-e244672d5225 5
> >>> >> >  31 UP osc pgdata01-OST0001-osc-ffff88204bab6400
> >>> >> > 996b1742-82eb-281c-c322-e244672d5225 5
> >>> >> >  32 UP osc pgdata01-OST0003-osc-ffff88204bab6400
> >>> >> > 996b1742-82eb-281c-c322-e244672d5225 5
> >>> >> >  33 UP osc pgdata01-OST0005-osc-ffff88204bab6400
> >>> >> > 996b1742-82eb-281c-c322-e244672d5225 5
> >>> >> >  34 UP osc pgdata01-OST0007-osc-ffff88204bab6400
> >>> >> > 996b1742-82eb-281c-c322-e244672d5225 5
> >>> >> >  35 UP osc pgdata01-OST0009-osc-ffff88204bab6400
> >>> >> > 996b1742-82eb-281c-c322-e244672d5225 5
> >>> >> >  36 UP osc pgdata01-OST000b-osc-ffff88204bab6400
> >>> >> > 996b1742-82eb-281c-c322-e244672d5225 5
> >>> >> >  37 UP osc pgdata01-OST000d-osc-ffff88204bab6400
> >>> >> > 996b1742-82eb-281c-c322-e244672d5225 5
> >>> >> >  38 UP osc pgdata01-OST000f-osc-ffff88204bab6400
> >>> >> > 996b1742-82eb-281c-c322-e244672d5225 5
> >>> >> >  39 UP osc pgdata01-OST0002-osc-ffff88204bab6400
> >>> >> > 996b1742-82eb-281c-c322-e244672d5225 5
> >>> >> >  40 UP osc pgdata01-OST0004-osc-ffff88204bab6400
> >>> >> > 996b1742-82eb-281c-c322-e244672d5225 5
> >>> >> >  41 UP osc pgdata01-OST0006-osc-ffff88204bab6400
> >>> >> > 996b1742-82eb-281c-c322-e244672d5225 5
> >>> >> >  42 UP osc pgdata01-OST0008-osc-ffff88204bab6400
> >>> >> > 996b1742-82eb-281c-c322-e244672d5225 5
> >>> >> >  43 UP osc pgdata01-OST000a-osc-ffff88204bab6400
> >>> >> > 996b1742-82eb-281c-c322-e244672d5225 5
> >>> >> >  44 UP osc pgdata01-OST000c-osc-ffff88204bab6400
> >>> >> > 996b1742-82eb-281c-c322-e244672d5225 5
> >>> >> >  45 UP osc pgdata01-OST000e-osc-ffff88204bab6400
> >>> >> > 996b1742-82eb-281c-c322-e244672d5225 5
> >>> >> >  46 UP osc pgdata01-OST0010-osc-ffff88204bab6400
> >>> >> > 996b1742-82eb-281c-c322-e244672d5225 5
> >>> >> >  47 UP osc pgdata01-OST0013-osc-ffff88204bab6400
> >>> >> > 996b1742-82eb-281c-c322-e244672d5225 5
> >>> >> >  48 UP osc pgdata01-OST0015-osc-ffff88204bab6400
> >>> >> > 996b1742-82eb-281c-c322-e244672d5225 5
> >>> >> >  49 UP osc pgdata01-OST0017-osc-ffff88204bab6400
> >>> >> > 996b1742-82eb-281c-c322-e244672d5225 5
> >>> >> >  50 UP osc pgdata01-OST0014-osc-ffff88204bab6400
> >>> >> > 996b1742-82eb-281c-c322-e244672d5225 5
> >>> >> >  51 UP osc pgdata01-OST0016-osc-ffff88204bab6400
> >>> >> > 996b1742-82eb-281c-c322-e244672d5225 5
> >>> >> >  52 UP osc pgdata01-OST0018-osc-ffff88204bab6400
> >>> >> > 996b1742-82eb-281c-c322-e244672d5225 5
> >>> >> >  53 UP osc pgdata01-OST0019-osc-ffff88204bab6400
> >>> >> > 996b1742-82eb-281c-c322-e244672d5225 5
> >>> >> >  54 UP osc pgdata01-OST001a-osc-ffff88204bab6400
> >>> >> > 996b1742-82eb-281c-c322-e244672d5225 5
> >>> >> >  55 UP osc pgdata01-OST001b-osc-ffff88204bab6400
> >>> >> > 996b1742-82eb-281c-c322-e244672d5225 5
> >>> >> >  56 UP lov pghome01-clilov-ffff88204bb50000
> >>> >> > 9ae8f2a9-1cdf-901f-160c-66f70e4c10d1 4
> >>> >> >  57 UP lmv pghome01-clilmv-ffff88204bb50000
> >>> >> > 9ae8f2a9-1cdf-901f-160c-66f70e4c10d1 4
> >>> >> >  58 UP mdc pghome01-MDT0000-mdc-ffff88204bb50000
> >>> >> > 9ae8f2a9-1cdf-901f-160c-66f70e4c10d1 5
> >>> >> >  59 UP osc pghome01-OST0011-osc-ffff88204bb50000
> >>> >> > 9ae8f2a9-1cdf-901f-160c-66f70e4c10d1 5
> >>> >> >  60 UP osc pghome01-OST0012-osc-ffff88204bb50000
> >>> >> > 9ae8f2a9-1cdf-901f-160c-66f70e4c10d1 5
> >>> >> >
> >>> >> > Please help, any clues/advice/hints/tips are appricated
> >>> >> >
> >>> >> > --
> >>> >> >
> >>> >> > Vriendelijke groet,
> >>> >> >
> >>> >> > Ger Strikwerda
> >>> >> > Chef Special
> >>> >> > Rijksuniversiteit Groningen
> >>> >> > Centrum voor Informatie Technologie
> >>> >> > Unit Pragmatisch Systeembeheer
> >>> >> >
> >>> >> > Smitsborg
> >>> >> > Nettelbosje 1
> >>> >> > 9747 AJ Groningen
> >>> >> > Tel. 050 363 9276
> >>> >> >
> >>> >> > "God is hard, God is fair
> >>> >> >  some men he gave brains, others he gave hair"
> >>> >> >
> >>> >> >
> >>> >> > _______________________________________________
> >>> >> > lustre-discuss mailing list
> >>> >> > lustre-discuss at lists.lustre.org
> >>> >> > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
> >>> >> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> > --
> >>> >
> >>> > Vriendelijke groet,
> >>> >
> >>> > Ger Strikwerda
> >>> > Chef Special
> >>> > Rijksuniversiteit Groningen
> >>> > Centrum voor Informatie Technologie
> >>> > Unit Pragmatisch Systeembeheer
> >>> >
> >>> > Smitsborg
> >>> > Nettelbosje 1
> >>> > 9747 AJ Groningen
> >>> > Tel. 050 363 9276
> >>> >
> >>> > "God is hard, God is fair
> >>> >  some men he gave brains, others he gave hair"
> >>
> >>
> >>
> >>
> >> --
> >>
> >> Vriendelijke groet,
> >>
> >> Ger Strikwerda
> >> Chef Special
> >> Rijksuniversiteit Groningen
> >> Centrum voor Informatie Technologie
> >> Unit Pragmatisch Systeembeheer
> >>
> >> Smitsborg
> >> Nettelbosje 1
> >> 9747 AJ Groningen
> >> Tel. 050 363 9276
> >>
> >> "God is hard, God is fair
> >>  some men he gave brains, others he gave hair"
>

-- 

Vriendelijke groet,

Ger StrikwerdaChef Special
Rijksuniversiteit Groningen
Centrum voor Informatie Technologie
Unit Pragmatisch Systeembeheer

Smitsborg
Nettelbosje 1
9747 AJ Groningen
Tel. 050 363 9276
"God is hard, God is fair
 some men he gave brains, others he gave hair"
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20170424/6103b513/attachment-0001.htm>