[Lustre-discuss] Question about setting max service threads

[Andreas Dilger]

On 2011-08-15, at 3:58 PM, Mike Hanby wrote:
> Our OSS servers are logging quite a few "heavy IO load" combined with system load (via 'uptime') being reported in the 100's to several 100's range.
> 
> Lustre: lustre-OST0004: slow commitrw commit 191s due to heavy IO load
> Aug 15 13:00:38 lustre-oss-0-2 kernel: Lustre: Service thread pid 17651 completed after 236.04s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).
> Lustre: Skipped 1 previous similar message
> Lustre: lustre-OST0004: slow commitrw commit 191s due to heavy IO load
> Lustre: Service thread pid 16436 completed after 210.17s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).
> 
> I'd like to test setting the ost_io.threads_max to values lower than 512.
> 
> Question 1: Will this command survive a reboot "lctl set_param ost.OSS.ost_io.threads_max=256"

This is only a temporary setting.

> or do I need to also run "lctl conf_param ost.OSS.ost_io.threads_max=256"?

The conf_param syntax is (unfortunately) slightly different than the set_param syntax.  You can also set this in /etc/modprobe.d/lustre.conf:

options ost oss_num_threads=256
options mds mds_num_threads=256

> Question 2: Since Lustre "does not reduce the number of service threads in use", is there any way I can force the extra running service threads to exit, or is a reboot of the OSS servers the only clean way?

I had written a patch to do this, but it wasn't landed yet.  Currently the only way to limit the thread count is to set this before the number of running threads has exceeded the maximum thread count.

Cheers, Andreas
--
Andreas Dilger 
Principal Engineer
Whamcloud, Inc.