[lustre-discuss] client fails to mount

E.S. Rosenberg esr+lustre at mail.hebrew.edu
Mon Apr 24 10:16:34 PDT 2017


On Mon, Apr 24, 2017 at 8:13 PM, Strikwerda, Ger <g.j.c.strikwerda at rug.nl>
wrote:

> Hi Raj (and others),
>
> In which file should i state the credits/peer_credits stuff?
>
> Perhaps relevant config-files:
>
> [root at pg-gpu01 ~]# cd /etc/modprobe.d/
>
> [root at pg-gpu01 modprobe.d]# ls
> anaconda.conf   blacklist-kvm.conf      dist-alsa.conf
> dist-oss.conf           ib_ipoib.conf  lustre.conf  openfwwf.conf
> blacklist.conf  blacklist-nouveau.conf  dist.conf
> freeipmi-modalias.conf  ib_sdp.conf    mlnx.conf    truescale.conf
>
> [root at pg-gpu01 modprobe.d]# cat ./ib_ipoib.conf
> alias netdev-ib* ib_ipoib
>
> [root at pg-gpu01 modprobe.d]# cat ./mlnx.conf
> # Module parameters for MLNX_OFED kernel modules
>
> [root at pg-gpu01 modprobe.d]# cat ./lustre.conf
> options lnet networks=o2ib(ib0)
>
> Are there more Lustre/LNET options that could help in this situation?
>

What about the logfiles?
Any error messages in syslog? lctl debug options?
Veel geluk,
Eli

>
>
>
>
> On Mon, Apr 24, 2017 at 7:02 PM, Raj <rajgautam at gmail.com> wrote:
>
>> May be worth checking your lnet credits and peer_credits in
>> /etc/modprobe.d ?
>> You can compare between working hosts and non working hosts.
>> Thanks
>> _Raj
>>
>> On Mon, Apr 24, 2017 at 10:10 AM Strikwerda, Ger <g.j.c.strikwerda at rug.nl>
>> wrote:
>>
>>> Hi Rick,
>>>
>>> Even without iptables rules and loading the correct modules afterwards,
>>> we get the same results:
>>>
>>> [root at pg-gpu01 sysconfig]# iptables --list
>>> Chain INPUT (policy ACCEPT)
>>> target     prot opt source               destination
>>>
>>> Chain FORWARD (policy ACCEPT)
>>> target     prot opt source               destination
>>>
>>> Chain OUTPUT (policy ACCEPT)
>>> target     prot opt source               destination
>>>
>>> Chain LOGDROP (0 references)
>>> target     prot opt source               destination
>>> LOG        all  --  anywhere             anywhere            LOG level
>>> warning
>>> DROP       all  --  anywhere             anywhere
>>>
>>> [root at pg-gpu01 sysconfig]# modprobe lnet
>>>
>>> [root at pg-gpu01 sysconfig]# modprobe lustre
>>>
>>> [root at pg-gpu01 sysconfig]# lctl ping 172.23.55.211 at o2ib
>>>
>>> failed to ping 172.23.55.211 at o2ib: Input/output error
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Mon, Apr 24, 2017 at 4:59 PM, Mohr Jr, Richard Frank (Rick Mohr) <
>>> rmohr at utk.edu> wrote:
>>>
>>>> This might be a long shot, but have you checked for possible firewall
>>>> rules that might be causing the issue?  I’m wondering if there is a chance
>>>> that some rules were added after the nodes were up to allow Lustre access,
>>>> and when a node got rebooted, it lost the rules.
>>>>
>>>> --
>>>> Rick Mohr
>>>> Senior HPC System Administrator
>>>> National Institute for Computational Sciences
>>>> http://www.nics.tennessee.edu
>>>>
>>>>
>>>> > On Apr 24, 2017, at 10:19 AM, Strikwerda, Ger <
>>>> g.j.c.strikwerda at rug.nl> wrote:
>>>> >
>>>> > Hi Russell,
>>>> >
>>>> > Thanks for the IB subnet clues:
>>>> >
>>>> > [root at pg-gpu01 ~]# ibv_devinfo
>>>> > hca_id: mlx4_0
>>>> >         transport:                      InfiniBand (0)
>>>> >         fw_ver:                         2.32.5100
>>>> >         node_guid:                      f452:1403:00f5:4620
>>>> >         sys_image_guid:                 f452:1403:00f5:4623
>>>> >         vendor_id:                      0x02c9
>>>> >         vendor_part_id:                 4099
>>>> >         hw_ver:                         0x1
>>>> >         board_id:                       MT_1100120019
>>>> >         phys_port_cnt:                  1
>>>> >                 port:   1
>>>> >                         state:                  PORT_ACTIVE (4)
>>>> >                         max_mtu:                4096 (5)
>>>> >                         active_mtu:             4096 (5)
>>>> >                         sm_lid:                 1
>>>> >                         port_lid:               185
>>>> >                         port_lmc:               0x00
>>>> >                         link_layer:             InfiniBand
>>>> >
>>>> > [root at pg-gpu01 ~]# sminfo
>>>> > sminfo: sm lid 1 sm guid 0xf452140300f62320, activity count 80878098
>>>> priority 0 state 3 SMINFO_MASTER
>>>> >
>>>> > Looks like the rebooted node is able to connect/contact IB/IB
>>>> subnetmanager
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > On Mon, Apr 24, 2017 at 4:14 PM, Russell Dekema <dekemar at umich.edu>
>>>> wrote:
>>>> > At first glance, this sounds like your Infiniband subnet manager may
>>>> > be down or malfunctioning. In this case, nodes which were already up
>>>> > when the subnet manager was working will continue to be able to
>>>> > communicate over IB, but nodes which reboot after the SM goes down
>>>> > will not.
>>>> >
>>>> > You can test this theory by running the 'ibv_devinfo' command on one
>>>> > of your rebooted nodes. If the relevant IB port is in state PORT_INIT,
>>>> > this confirms there is a problem with your subnet manager.
>>>> >
>>>> > Sincerely,
>>>> > Rusty Dekema
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > On Mon, Apr 24, 2017 at 9:57 AM, Strikwerda, Ger
>>>> > <g.j.c.strikwerda at rug.nl> wrote:
>>>> > > Hi everybody,
>>>> > >
>>>> > > Here at the university of Groningen we are now experiencing a
>>>> strange Lustre
>>>> > > error. If a client reboots, it fails to mount the Lustre storage.
>>>> The client
>>>> > > is not able to reach the MSG service. The storage and nodes are
>>>> > > communicating over IB and unitil now without any problems. It looks
>>>> like an
>>>> > > issue inside LNET. Clients cannot LNET ping/connect the metadata
>>>> and or
>>>> > > storage. But the clients are able to LNET ping each other. Clients
>>>> which not
>>>> > > have been rebooted, are working fine and have their mounts on our
>>>> Lustre
>>>> > > filesystem.
>>>> > >
>>>> > > Lustre client log:
>>>> > >
>>>> > > Lustre: Lustre: Build Version: 2.5.3-RC1--PRISTINE-2.6.32-573
>>>> .el6.x86_64
>>>> > > LNet: Added LNI 172.23.54.51 at o2ib [8/256/0/180]
>>>> > >
>>>> > > LustreError: 15c-8: MGC172.23.55.211 at o2ib: The configuration from
>>>> log
>>>> > > 'pgdata01-client' failed (-5). This may be the result of
>>>> communication
>>>> > > errors between this node and the MGS, a bad configuration, or other
>>>> errors.
>>>> > > See the syslog for more information.
>>>> > > LustreError: 3812:0:(llite_lib.c:1046:ll_fill_super()) Unable to
>>>> process
>>>> > > log: -5
>>>> > > Lustre: Unmounted pgdata01-client
>>>> > > LustreError: 3812:0:(obd_mount.c:1325:lustre_fill_super()) Unable
>>>> to mount
>>>> > > (-5)
>>>> > > LNetError: 2882:0:(o2iblnd_cb.c:2587:kiblnd_rejected())
>>>> 172.23.55.212 at o2ib
>>>> > > rejected: consumer defined fatal error
>>>> > > LNetError: 2882:0:(o2iblnd_cb.c:2587:kiblnd_rejected()) Skipped 1
>>>> previous
>>>> > > similar message
>>>> > > Lustre: 3765:0:(client.c:1918:ptlrpc_expire_one_request()) @@@
>>>> Request sent
>>>> > > has failed due to network error: [sent 1492789626/real 1492789626]
>>>> > > req at ffff88105af2cc00 x1565303228072004/t0(0)
>>>> > > o250->MGC172.23.55.211 at o2ib@172.23.55.212 at o2ib:26/25 lens 400/544
>>>> e 0 to 1
>>>> > > dl 1492789631 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
>>>> > > Lustre: 3765:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped
>>>> 1
>>>> > > previous similar message
>>>> > > LustreError: 3826:0:(client.c:1083:ptlrpc_import_delay_req()) @@@
>>>> send limit
>>>> > > expired   req at ffff882041ffc000 x1565303228071996/t0(0)
>>>> > > o101->MGC172.23.55.211 at o2ib@172.23.55.211 at o2ib:26/25 lens 328/344
>>>> e 0 to 0
>>>> > > dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
>>>> > > LustreError: 3826:0:(client.c:1083:ptlrpc_import_delay_req())
>>>> Skipped 2
>>>> > > previous similar messages
>>>> > > LustreError: 15c-8: MGC172.23.55.211 at o2ib: The configuration from
>>>> log
>>>> > > 'pghome01-client' failed (-5). This may be the result of
>>>> communication
>>>> > > errors between this node and the MGS, a bad configuration, or other
>>>> errors.
>>>> > > See the syslog for more information.
>>>> > > LustreError: 3826:0:(llite_lib.c:1046:ll_fill_super()) Unable to
>>>> process
>>>> > > log: -5
>>>> > >
>>>> > > LNetError: 2882:0:(o2iblnd_cb.c:2587:kiblnd_rejected())
>>>> 172.23.55.212 at o2ib
>>>> > > rejected: consumer defined fatal error
>>>> > > LNetError: 2882:0:(o2iblnd_cb.c:2587:kiblnd_rejected()) Skipped 1
>>>> previous
>>>> > > similar message
>>>> > > LNet: 3755:0:(o2iblnd_cb.c:475:kiblnd_rx_complete()) Rx from
>>>> > > 172.23.55.211 at o2ib failed: 5
>>>> > > LNetError: 2882:0:(o2iblnd_cb.c:2587:kiblnd_rejected())
>>>> 172.23.55.211 at o2ib
>>>> > > rejected: consumer defined fatal error
>>>> > > LNetError: 2882:0:(o2iblnd_cb.c:2587:kiblnd_rejected()) Skipped 1
>>>> previous
>>>> > > similar message
>>>> > > LNet: 2882:0:(o2iblnd_cb.c:2072:kiblnd_peer_connect_failed())
>>>> Deleting
>>>> > > messages for 172.23.55.211 at o2ib: connection failed
>>>> > > LNet: 2882:0:(o2iblnd_cb.c:2072:kiblnd_peer_connect_failed())
>>>> Deleting
>>>> > > messages for 172.23.55.212 at o2ib: connection failed
>>>> > > LNet: 3754:0:(o2iblnd_cb.c:475:kiblnd_rx_complete()) Rx from
>>>> > > 172.23.55.212 at o2ib failed: 5
>>>> > > LNet: 3754:0:(o2iblnd_cb.c:475:kiblnd_rx_complete()) Skipped 17
>>>> previous
>>>> > > similar messages
>>>> > > LNet: 2882:0:(o2iblnd_cb.c:2072:kiblnd_peer_connect_failed())
>>>> Deleting
>>>> > > messages for 172.23.55.211 at o2ib: connection failed
>>>> > > LNet: 3754:0:(o2iblnd_cb.c:475:kiblnd_rx_complete()) Rx from
>>>> > > 172.23.55.212 at o2ib failed: 5
>>>> > > LNet: 2882:0:(o2iblnd_cb.c:2072:kiblnd_peer_connect_failed())
>>>> Deleting
>>>> > > messages for 172.23.55.212 at o2ib: connection failed
>>>> > >
>>>> > > LNET ping of a metadata-node:
>>>> > >
>>>> > > [root at pg-gpu01 ~]# lctl ping 172.23.55.211 at o2ib
>>>> > > failed to ping 172.23.55.211 at o2ib: Input/output error
>>>> > >
>>>> > > LNET ping of the number 2 metadata-node:
>>>> > >
>>>> > > [root at pg-gpu01 ~]# lctl ping 172.23.55.212 at o2ib
>>>> > > failed to ping 172.23.55.212 at o2ib: Input/output error
>>>> > >
>>>> > > LNET ping of a random compute-node:
>>>> > >
>>>> > > [root at pg-gpu01 ~]# lctl ping 172.23.52.5 at o2ib
>>>> > > 12345-0 at lo
>>>> > > 12345-172.23.52.5 at o2ib
>>>> > >
>>>> > > LNET to OST01:
>>>> > >
>>>> > > [root at pg-gpu01 ~]# lctl ping 172.23.55.201 at o2ib
>>>> > > failed to ping 172.23.55.201 at o2ib: Input/output error
>>>> > >
>>>> > > LNET to OST02:
>>>> > >
>>>> > > [root at pg-gpu01 ~]# lctl ping 172.23.55.202 at o2ib
>>>> > > failed to ping 172.23.55.202 at o2ib: Input/output error
>>>> > >
>>>> > > 'normal' pings (on ip level) works fine:
>>>> > >
>>>> > > [root at pg-gpu01 ~]# ping 172.23.55.201
>>>> > > PING 172.23.55.201 (172.23.55.201) 56(84) bytes of data.
>>>> > > 64 bytes from 172.23.55.201: icmp_seq=1 ttl=64 time=0.741 ms
>>>> > >
>>>> > > [root at pg-gpu01 ~]# ping 172.23.55.202
>>>> > > PING 172.23.55.202 (172.23.55.202) 56(84) bytes of data.
>>>> > > 64 bytes from 172.23.55.202: icmp_seq=1 ttl=64 time=0.704 ms
>>>> > >
>>>> > > lctl on a rebooted node:
>>>> > >
>>>> > > [root at pg-gpu01 ~]# lctl dl
>>>> > >
>>>> > > lctl on a not rebooted node:
>>>> > >
>>>> > > [root at pg-node005 ~]# lctl dl
>>>> > >   0 UP mgc MGC172.23.55.211 at o2ib 94bd1c8a-512f-b920-9a4e-a6aced3d386d
>>>> 5
>>>> > >   1 UP lov pgtemp01-clilov-ffff88206906d400
>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 4
>>>> > >   2 UP lmv pgtemp01-clilmv-ffff88206906d400
>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 4
>>>> > >   3 UP mdc pgtemp01-MDT0000-mdc-ffff88206906d400
>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5
>>>> > >   4 UP osc pgtemp01-OST0001-osc-ffff88206906d400
>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5
>>>> > >   5 UP osc pgtemp01-OST0003-osc-ffff88206906d400
>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5
>>>> > >   6 UP osc pgtemp01-OST0005-osc-ffff88206906d400
>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5
>>>> > >   7 UP osc pgtemp01-OST0007-osc-ffff88206906d400
>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5
>>>> > >   8 UP osc pgtemp01-OST0009-osc-ffff88206906d400
>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5
>>>> > >   9 UP osc pgtemp01-OST000b-osc-ffff88206906d400
>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5
>>>> > >  10 UP osc pgtemp01-OST000d-osc-ffff88206906d400
>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5
>>>> > >  11 UP osc pgtemp01-OST000f-osc-ffff88206906d400
>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5
>>>> > >  12 UP osc pgtemp01-OST0011-osc-ffff88206906d400
>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5
>>>> > >  13 UP osc pgtemp01-OST0002-osc-ffff88206906d400
>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5
>>>> > >  14 UP osc pgtemp01-OST0004-osc-ffff88206906d400
>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5
>>>> > >  15 UP osc pgtemp01-OST0006-osc-ffff88206906d400
>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5
>>>> > >  16 UP osc pgtemp01-OST0008-osc-ffff88206906d400
>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5
>>>> > >  17 UP osc pgtemp01-OST000a-osc-ffff88206906d400
>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5
>>>> > >  18 UP osc pgtemp01-OST000c-osc-ffff88206906d400
>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5
>>>> > >  19 UP osc pgtemp01-OST000e-osc-ffff88206906d400
>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5
>>>> > >  20 UP osc pgtemp01-OST0010-osc-ffff88206906d400
>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5
>>>> > >  21 UP osc pgtemp01-OST0012-osc-ffff88206906d400
>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5
>>>> > >  22 UP osc pgtemp01-OST0013-osc-ffff88206906d400
>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5
>>>> > >  23 UP osc pgtemp01-OST0015-osc-ffff88206906d400
>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5
>>>> > >  24 UP osc pgtemp01-OST0017-osc-ffff88206906d400
>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5
>>>> > >  25 UP osc pgtemp01-OST0014-osc-ffff88206906d400
>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5
>>>> > >  26 UP osc pgtemp01-OST0016-osc-ffff88206906d400
>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5
>>>> > >  27 UP osc pgtemp01-OST0018-osc-ffff88206906d400
>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5
>>>> > >  28 UP lov pgdata01-clilov-ffff88204bab6400
>>>> > > 996b1742-82eb-281c-c322-e244672d5225 4
>>>> > >  29 UP lmv pgdata01-clilmv-ffff88204bab6400
>>>> > > 996b1742-82eb-281c-c322-e244672d5225 4
>>>> > >  30 UP mdc pgdata01-MDT0000-mdc-ffff88204bab6400
>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5
>>>> > >  31 UP osc pgdata01-OST0001-osc-ffff88204bab6400
>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5
>>>> > >  32 UP osc pgdata01-OST0003-osc-ffff88204bab6400
>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5
>>>> > >  33 UP osc pgdata01-OST0005-osc-ffff88204bab6400
>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5
>>>> > >  34 UP osc pgdata01-OST0007-osc-ffff88204bab6400
>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5
>>>> > >  35 UP osc pgdata01-OST0009-osc-ffff88204bab6400
>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5
>>>> > >  36 UP osc pgdata01-OST000b-osc-ffff88204bab6400
>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5
>>>> > >  37 UP osc pgdata01-OST000d-osc-ffff88204bab6400
>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5
>>>> > >  38 UP osc pgdata01-OST000f-osc-ffff88204bab6400
>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5
>>>> > >  39 UP osc pgdata01-OST0002-osc-ffff88204bab6400
>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5
>>>> > >  40 UP osc pgdata01-OST0004-osc-ffff88204bab6400
>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5
>>>> > >  41 UP osc pgdata01-OST0006-osc-ffff88204bab6400
>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5
>>>> > >  42 UP osc pgdata01-OST0008-osc-ffff88204bab6400
>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5
>>>> > >  43 UP osc pgdata01-OST000a-osc-ffff88204bab6400
>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5
>>>> > >  44 UP osc pgdata01-OST000c-osc-ffff88204bab6400
>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5
>>>> > >  45 UP osc pgdata01-OST000e-osc-ffff88204bab6400
>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5
>>>> > >  46 UP osc pgdata01-OST0010-osc-ffff88204bab6400
>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5
>>>> > >  47 UP osc pgdata01-OST0013-osc-ffff88204bab6400
>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5
>>>> > >  48 UP osc pgdata01-OST0015-osc-ffff88204bab6400
>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5
>>>> > >  49 UP osc pgdata01-OST0017-osc-ffff88204bab6400
>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5
>>>> > >  50 UP osc pgdata01-OST0014-osc-ffff88204bab6400
>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5
>>>> > >  51 UP osc pgdata01-OST0016-osc-ffff88204bab6400
>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5
>>>> > >  52 UP osc pgdata01-OST0018-osc-ffff88204bab6400
>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5
>>>> > >  53 UP osc pgdata01-OST0019-osc-ffff88204bab6400
>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5
>>>> > >  54 UP osc pgdata01-OST001a-osc-ffff88204bab6400
>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5
>>>> > >  55 UP osc pgdata01-OST001b-osc-ffff88204bab6400
>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5
>>>> > >  56 UP lov pghome01-clilov-ffff88204bb50000
>>>> > > 9ae8f2a9-1cdf-901f-160c-66f70e4c10d1 4
>>>> > >  57 UP lmv pghome01-clilmv-ffff88204bb50000
>>>> > > 9ae8f2a9-1cdf-901f-160c-66f70e4c10d1 4
>>>> > >  58 UP mdc pghome01-MDT0000-mdc-ffff88204bb50000
>>>> > > 9ae8f2a9-1cdf-901f-160c-66f70e4c10d1 5
>>>> > >  59 UP osc pghome01-OST0011-osc-ffff88204bb50000
>>>> > > 9ae8f2a9-1cdf-901f-160c-66f70e4c10d1 5
>>>> > >  60 UP osc pghome01-OST0012-osc-ffff88204bb50000
>>>> > > 9ae8f2a9-1cdf-901f-160c-66f70e4c10d1 5
>>>> > >
>>>> > > Please help, any clues/advice/hints/tips are appricated
>>>> > >
>>>> > > --
>>>> > >
>>>> > > Vriendelijke groet,
>>>> > >
>>>> > > Ger Strikwerda
>>>> > > Chef Special
>>>> > > Rijksuniversiteit Groningen
>>>> > > Centrum voor Informatie Technologie
>>>> > > Unit Pragmatisch Systeembeheer
>>>> > >
>>>> > > Smitsborg
>>>> > > Nettelbosje 1
>>>> > > 9747 AJ Groningen
>>>> > > Tel. 050 363 9276
>>>> > >
>>>> > > "God is hard, God is fair
>>>> > >  some men he gave brains, others he gave hair"
>>>> > >
>>>> > >
>>>> > > _______________________________________________
>>>> > > lustre-discuss mailing list
>>>> > > lustre-discuss at lists.lustre.org
>>>> > > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>>>> > >
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > Vriendelijke groet,
>>>> >
>>>> > Ger Strikwerda
>>>> >
>>>> > Chef Special
>>>> > Rijksuniversiteit Groningen
>>>> > Centrum voor Informatie Technologie
>>>> > Unit Pragmatisch Systeembeheer
>>>> >
>>>> > Smitsborg
>>>> > Nettelbosje 1
>>>> > 9747 AJ Groningen
>>>> > Tel. 050 363 9276
>>>> >
>>>> >
>>>> > "God is hard, God is fair
>>>> >  some men he gave brains, others he gave hair"
>>>> > _______________________________________________
>>>> > lustre-discuss mailing list
>>>> > lustre-discuss at lists.lustre.org
>>>> > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>>
>>> Vriendelijke groet,
>>>
>>> Ger StrikwerdaChef Special
>>> Rijksuniversiteit Groningen
>>> Centrum voor Informatie Technologie
>>> Unit Pragmatisch Systeembeheer
>>>
>>> Smitsborg
>>> Nettelbosje 1
>>> 9747 AJ Groningen
>>> Tel. 050 363 9276
>>> "God is hard, God is fair
>>>  some men he gave brains, others he gave hair"
>>>
>>> _______________________________________________
>>> lustre-discuss mailing list
>>> lustre-discuss at lists.lustre.org
>>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>>>
>>
>
>
> --
>
> Vriendelijke groet,
>
> Ger StrikwerdaChef Special
> Rijksuniversiteit Groningen
> Centrum voor Informatie Technologie
> Unit Pragmatisch Systeembeheer
>
> Smitsborg
> Nettelbosje 1
> 9747 AJ Groningen
> Tel. 050 363 9276
> "God is hard, God is fair
>  some men he gave brains, others he gave hair"
>
>
> _______________________________________________
> lustre-discuss mailing list
> lustre-discuss at lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20170424/1be73b20/attachment-0001.htm>


More information about the lustre-discuss mailing list