[lustre-discuss] mgs stops working after accidently setting a non existing paramter (no device for)

[Harald van Pee]

Dear all, 

I changed the subject, because its most important for us to get the mgs 
running again and that it would be possible to allow mount a client.

Why ever I have managed to set a paramter for which no device exists,
indeed there is no ost0000.
But obviously the hiskp3-client log has logged a cmd to set
parameter for device hiskp3-OST0000-osc
which was not my intention.

The mgs/mdt is not used as a client, and probably therefore
lctl get_param osc.hiskp3-OST*.max_pages_per_rpc
error: get_param: /proc/{fs,sys}/{lnet,lustre}/osc/hiskp3-
OST*/max_pages_per_rpc: Found no match
and
lctl set_param osc.hiskp3-OST*.max_pages_per_rpc=256
does not work there.
but indeed
ls /proc/fs/lustre/osc/
shows:
hiskp3-OST0001-osc-MDT0000  hiskp3-OST0002-osc-MDT0000  hiskp3-OST0003-osc-
MDT0000  hiskp3-OST0004-osc-MDT0000  num_refs

(but no OST0000).
Can I get ride of this misconfiguration by setting
lctl conf_param hiskp3-OST0000.osc.max_pages_per_rpc=0
??

And if so, how I have to proceed after that command?

Thanks in advance
Harald



On Wednesday 15 November 2017 17:26:25 Harald van Pee wrote:
> Here are more informations:
> 
> if I try to mount the filesystem on the client I get similar messages as
> from the failing conf_param command. It seems one have to remove this
> failed configuration but how?
> Here the syslog output on the client:
> 
> kernel: [ 4203.506437] LustreError: 3698:0:
> (obd_mount.c:1340:lustre_fill_super()) Unable to mount  (-2)
>  kernel: [ 5028.547095] LustreError: 3830:0:
> (obd_config.c:1202:class_process_config()) no device for:
> hiskp3-OST0000-osc- ffff880416680800
>  kernel: [ 5028.547105] LustreError: 3830:0:
> (obd_config.c:1606:class_config_llog_handler()) MGC192.168.128.200 at o2ib:
> cfg command failed: rc = -22
>  kernel: [ 5028.547112] Lustre:    cmd=cf00f 0:hiskp3-OST0000-osc
> 1:osc.max_pages_per_rpc=256
>  kernel: [ 5028.547112]
>  kernel: [ 5028.547156] LustreError: 15b-f: MGC192.168.128.200 at o2ib: The
> configuration from log 'hiskp3-client'failed from the MGS (-22).  Make sure
> this client and the MGS are running compatible versions of Lustre.
>  kernel: [ 5028.547407] LustreError: 1680:0:(lov_obd.c:946:lov_cleanup())
> hiskp3-clilov-ffff880416680800: lov tgt 1 not cleaned! deathrow=0, lovrc=1
>  kernel: [ 5028.547415] LustreError: 1680:0:(lov_obd.c:946:lov_cleanup())
> Skipped 3 previous similar messages
>  kernel: [ 5028.550906] Lustre: Unmounted hiskp3-client
>  kernel: [ 5028.551407] LustreError: 3815:0:
> (obd_mount.c:1340:lustre_fill_super()) Unable to mount  (-22)
> 
> On Wednesday 15 November 2017 16:06:29 Harald van Pee wrote:
> > Dear all,
> > 
> > I want to set max_pages_per_rpc to 64 instead of 256
> > lustre mgs/mdt version 2.53
> > lustre oss version 2.53
> > lustre client 2.6
> > 
> > on client I have done:
> > lctl get_param osc.hiskp3-OST*.max_pages_per_rpc
> > osc.hiskp3-OST0001-osc-ffff88105dba4800.max_pages_per_rpc=256
> > osc.hiskp3-OST0002-osc-ffff88105dba4800.max_pages_per_rpc=256
> > osc.hiskp3-OST0003-osc-ffff88105dba4800.max_pages_per_rpc=256
> > osc.hiskp3-OST0004-osc-ffff88105dba4800.max_pages_per_rpc=256
> > lctl set_param osc.hiskp3-OST*.max_pages_per_rpc=64
> > 
> > this works, but after remount I get again 256 therefore I want to make it
> > permant with
> > 
> >  lctl conf_param hiskp3-OST*.osc.max_pages_per_rpc=64
> > 
> > But I get the message, that this command have to be given on mdt
> > unfortunately I go to our combined mgs/mdt and get
> > 
> > Lustre: Setting parameter hiskp3-OST0000-osc.osc.max_pages_per_rpc in log
> > hiskp3-client
> > LustreError: 956:0:(obd_config.c:1221:class_process_config()) no device
> > for: hiskp3-OST0000-osc-MDT0000
> > LustreError: 956:0:(obd_config.c:1591:class_config_llog_handler())
> > MGC192.168.128.200 at o2ib: cfg command failed: rc = -22
> > Lustre:    cmd=cf00f 0:hiskp3-OST0000-osc-MDT0000
> > 1:osc.max_pages_per_rpc=64
> > 
> > than I can not mount client and want to go back
> > lctl set_param osc.hiskp3-OST*.max_pages_per_rpc=64
> > 
> > Lustre: Modifying parameter hiskp3-OST0000-osc.osc.max_pages_per_rpc in
> > log hiskp3-client
> > Lustre: Skipped 1 previous similar message
> > LustreError: 966:0:(obd_config.c:1221:class_process_config()) no device
> > for: hiskp3-OST0000-osc-MDT0000
> > LustreError: 966:0:(obd_config.c:1591:class_config_llog_handler())
> > MGC192.168.128.200 at o2ib: cfg command failed: rc = -22
> > Lustre:    cmd=cf00f 0:hiskp3-OST0000-osc-MDT0000
> > 1:osc.max_pages_per_rpc=256
> > 
> > obviously what I have done was completly wrong and I can no longer mount
> > a client, mounted clients are working.
> > How can I get it back working?
> > hiskp3-MDT0000 ist the label of the mgs/mdt but
> > hiskp3-OST0000-osc-MDT0000 seems to be incorrect
> > 
> > What I have to do to get the mgs/mdt working again?
> > Its your production cluster
> > Any help is welcome
> > 
> > Best
> > Harald
> > 
> > 
> > 
> > 
> > 
> > 
> > _______________________________________________
> > lustre-discuss mailing list
> > lustre-discuss at lists.lustre.org
> > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org