[lustre-discuss] how to set max_pages_per_rpc (I have done something wrong and need help)

[Harald van Pee]

Hi,

I have done 
lctl set_param -d <param>
lctl conf_param -d <param>

on the mgs/mdt
after this all problems are gone and all clients are able to mount the lustre 
filesystem again.

Best
Harald

On Thursday 16 November 2017 15:10:43 Harald van Pee wrote:
> Thank you,
> 
> I will start next week and report what helps.
> Unfortunately in this cluster we have to wait with the upgrade to 2.10.1
> until next year.
> 
> Best whishes
> Harald
> 
> On Wednesday 15 November 2017 23:09:49 Dilger, Andreas wrote:
> > On Nov 15, 2017, at 12:56, Harald van Pee <pee at hiskp.uni-bonn.de> wrote:
> > > Hello Andreas,
> > > 
> > > thanks for your information, now I have the feeling I'm not completly
> > > lost. With erasing configuration parameters do you mean the
> > > writeconf procedure? (chapter 14.4)
> > 
> > Yes.
> > 
> > > Or is it possible to erase the unknown parameter?
> > 
> > You could try "lctl conf_param -d <param>" to delete the parameter.
> > 
> > Cheers, Andreas
> > 
> > > On Wednesday 15 November 2017 20:37:20 Dilger, Andreas wrote:
> > >> The problem that Lustre clients fail to mount when they get an unknown
> > >> parameter is fixed in newer Lustre releases (2.9+) via patch
> > >> https://review.whamcloud.com/21112 .
> > >> 
> > >> The current maintenance release is 2.10.1.
> > >> 
> > >> You could also work around this by erasing the configuration
> > >> parameters (see Lustre manual).
> > >> 
> > >> Cheers, Andreas
> > >> 
> > >> On Nov 15, 2017, at 09:26, Harald van Pee
> > >> <pee at hiskp.uni-bonn.de<mailto:pee at hiskp.uni-bonn.de>> wrote:
> > >> 
> > >> Here are more informations:
> > >> 
> > >> if I try to mount the filesystem on the client I get similar messages
> > >> as from the failing conf_param command. It seems one have to remove
> > >> this failed configuration but how?
> > >> Here the syslog output on the client:
> > >> 
> > >> kernel: [ 4203.506437] LustreError: 3698:0:
> > >> (obd_mount.c:1340:lustre_fill_super()) Unable to mount  (-2)
> > >> kernel: [ 5028.547095] LustreError: 3830:0:
> > >> (obd_config.c:1202:class_process_config()) no device for:
> > >> hiskp3-OST0000-osc- ffff880416680800
> > >> kernel: [ 5028.547105] LustreError: 3830:0:
> > >> (obd_config.c:1606:class_config_llog_handler())
> > >> MGC192.168.128.200 at o2ib: cfg command failed: rc = -22
> > >> kernel: [ 5028.547112] Lustre:    cmd=cf00f 0:hiskp3-OST0000-osc
> > >> 1:osc.max_pages_per_rpc=256
> > >> kernel: [ 5028.547112]
> > >> kernel: [ 5028.547156] LustreError: 15b-f: MGC192.168.128.200 at o2ib:
> > >> The configuration from log 'hiskp3-client'failed from the MGS (-22). 
> > >> Make sure this client and the MGS are running compatible versions of
> > >> Lustre. kernel: [ 5028.547407] LustreError:
> > >> 1680:0:(lov_obd.c:946:lov_cleanup()) hiskp3-clilov-ffff880416680800:
> > >> lov tgt 1 not cleaned! deathrow=0, lovrc=1 kernel: [ 5028.547415]
> > >> LustreError: 1680:0:(lov_obd.c:946:lov_cleanup()) Skipped 3 previous
> > >> similar messages
> > >> kernel: [ 5028.550906] Lustre: Unmounted hiskp3-client
> > >> kernel: [ 5028.551407] LustreError: 3815:0:
> > >> (obd_mount.c:1340:lustre_fill_super()) Unable to mount  (-22)
> > >> 
> > >> 
> > >> 
> > >> On Wednesday 15 November 2017 16:06:29 Harald van Pee wrote:
> > >> Dear all,
> > >> 
> > >> I want to set max_pages_per_rpc to 64 instead of 256
> > >> lustre mgs/mdt version 2.53
> > >> lustre oss version 2.53
> > >> lustre client 2.6
> > >> 
> > >> on client I have done:
> > >> lctl get_param osc.hiskp3-OST*.max_pages_per_rpc
> > >> osc.hiskp3-OST0001-osc-ffff88105dba4800.max_pages_per_rpc=256
> > >> osc.hiskp3-OST0002-osc-ffff88105dba4800.max_pages_per_rpc=256
> > >> osc.hiskp3-OST0003-osc-ffff88105dba4800.max_pages_per_rpc=256
> > >> osc.hiskp3-OST0004-osc-ffff88105dba4800.max_pages_per_rpc=256
> > >> lctl set_param osc.hiskp3-OST*.max_pages_per_rpc=64
> > >> 
> > >> this works, but after remount I get again 256 therefore I want to make
> > >> it permant with
> > >> lctl conf_param hiskp3-OST*.osc.max_pages_per_rpc=64
> > >> 
> > >> But I get the message, that this command have to be given on mdt
> > >> unfortunately I go to our combined mgs/mdt and get
> > >> 
> > >> Lustre: Setting parameter hiskp3-OST0000-osc.osc.max_pages_per_rpc in
> > >> log hiskp3-client
> > >> LustreError: 956:0:(obd_config.c:1221:class_process_config()) no
> > >> device for: hiskp3-OST0000-osc-MDT0000
> > >> LustreError: 956:0:(obd_config.c:1591:class_config_llog_handler())
> > >> MGC192.168.128.200 at o2ib: cfg command failed: rc = -22
> > >> Lustre:    cmd=cf00f 0:hiskp3-OST0000-osc-MDT0000
> > >> 1:osc.max_pages_per_rpc=64
> > >> 
> > >> than I can not mount client and want to go back
> > >> lctl set_param osc.hiskp3-OST*.max_pages_per_rpc=64
> > >> 
> > >> Lustre: Modifying parameter hiskp3-OST0000-osc.osc.max_pages_per_rpc
> > >> in log hiskp3-client
> > >> Lustre: Skipped 1 previous similar message
> > >> LustreError: 966:0:(obd_config.c:1221:class_process_config()) no
> > >> device for: hiskp3-OST0000-osc-MDT0000
> > >> LustreError: 966:0:(obd_config.c:1591:class_config_llog_handler())
> > >> MGC192.168.128.200 at o2ib: cfg command failed: rc = -22
> > >> Lustre:    cmd=cf00f 0:hiskp3-OST0000-osc-MDT0000
> > >> 1:osc.max_pages_per_rpc=256
> > >> 
> > >> obviously what I have done was completly wrong and I can no longer
> > >> mount a client, mounted clients are working.
> > >> How can I get it back working?
> > >> hiskp3-MDT0000 ist the label of the mgs/mdt but
> > >> hiskp3-OST0000-osc-MDT0000 seems to be incorrect
> > >> 
> > >> What I have to do to get the mgs/mdt working again?
> > >> Its your production cluster
> > >> Any help is welcome
> > >> 
> > >> Best
> > >> Harald
> > >> 
> > >> 
> > >> 
> > >> 
> > >> 
> > >> 
> > >> _______________________________________________
> > >> lustre-discuss mailing list
> > >> lustre-discuss at lists.lustre.org<mailto:lustre-discuss at lists.lustre.org
> > >> > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
> > >> 
> > >> --
> > >> Harald van Pee
> > >> 
> > >> Helmholtz-Institut fuer Strahlen- und Kernphysik der Universitaet Bonn
> > >> Nussallee 14-16 - 53115 Bonn - Tel +49-228-732213 - Fax +49-228-732505
> > >> mail: pee at hiskp.uni-bonn.de<mailto:pee at hiskp.uni-bonn.de>
> > >> _______________________________________________
> > >> lustre-discuss mailing list
> > >> lustre-discuss at lists.lustre.org<mailto:lustre-discuss at lists.lustre.org
> > >> > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
> > 
> > Cheers, Andreas
> > --
> > Andreas Dilger
> > Lustre Principal Architect
> > Intel Corporation
> 
> _______________________________________________
> lustre-discuss mailing list
> lustre-discuss at lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

-- 
Harald van Pee

Helmholtz-Institut fuer Strahlen- und Kernphysik der Universitaet Bonn
Nussallee 14-16 - 53115 Bonn - Tel +49-228-732213 - Fax +49-228-732505
mail: pee at hiskp.uni-bonn.de