[Lustre-discuss] Question about setting max service threads

[Mike Hanby]

Howdy,

Our OSS servers are logging quite a few "heavy IO load" combined with system load (via 'uptime') being reported in the 100's to several 100's range.

Lustre: lustre-OST0004: slow commitrw commit 191s due to heavy IO load
Aug 15 13:00:38 lustre-oss-0-2 kernel: Lustre: Service thread pid 17651 completed after 236.04s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).
Lustre: Skipped 1 previous similar message
Lustre: lustre-OST0004: slow commitrw commit 191s due to heavy IO load
Lustre: Service thread pid 16436 completed after 210.17s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).

I'd like to test setting the ost_io.threads_max to values lower than 512.

Question 1: Will this command survive a reboot "lctl set_param ost.OSS.ost_io.threads_max=256" or do I need to also run "lctl conf_param ost.OSS.ost_io.threads_max=256"?

Question 2: Since Lustre "does not reduce the number of service threads in use", is there any way I can force the extra running service threads to exit, or is a reboot of the OSS servers the only clean way?

Thanks,

Mike