[Lustre-devel] tunables

[Andreas Dilger]

On Jan 31, 2008  11:47 -0800, Nathaniel Rutman wrote:
> 'Andreas Dilger' wrote:
>> The patch as it stands now will set any tunables in
>> /proc/{fs,sys}/{lustre,lnet}/ that match the path regexp supplied.
>>
>> This works on both clients and servers.  On 1.6 it just creates a
>> path regexp to find the various files.  On 1.8 it will eventually
>> communicate with the uOSS/uMDS process via a socket or something,
>> but the goal right now is putting an interface in place with the
>> existing release so that scripts/documentation can be compatible.
>>
>> It doesn't necessarily have anything to do with a mountpoint, but it
>> does require the Lustre devices (that need to be tuned) to be set up.
>   
> When I originally filed 14471, it was a grand plan of getting rid of /proc/ 
> and other Linuxisms by putting everything under Lustre control in a special
> <lustremountpt>/.lustre/param directory.  14471 has now morphed into 
> "implement lctl {get,set}_param", which, along with 14691, allows a
> consistent way of accessing the items formerly under proc.  This is fine
> for a first step, but I still think it doesn't go far enough. We'll
> eventually need  snapshots: <mntpt>/.lustre/snapshot/ and sooner feeds:
> <mntpt>/.lustre/feed/

The "move everything to .../.lustre/param" functionality was moved to
bug 14687 "Move /proc/{fs,sys}/{lustre,lnet} to /mntpt/.lustre" in
order to make the separate tasks more manageable.  The work in bug
14471 and 14691 are required for userspace servers, and as such for
the 1.8 release, while 14687 is a "nice to have" and doesn't really
bear any relationship to snapshots and feeds except that they share
the ".lustre" virtual directory.

The "lctl {get,set}_param" interfaces are intended to provide a common
userland interface to tunables and stats regardless of whether the
servers are running in userspace or the kernel.  I think similar changes
can easily be made to llstat and collectl to extract stats directly
from the userspace server processes as well.

> This idea works fine for current clients and servers (which all use 
> mountpoints), but not for userspace servers.  I don't think we have come
> up with a good answer for that.

In fact, the move of the parameters to the lustre mountpoint is
irrelevant for userspace servers, because they won't have any mountpoint.
The work in bug 14691 will address the getting and setting of Lustre
tunables for userspace servers, and access to stats.

> For params that are shared between all mounts, I think its fine to have 
> access to them from each one; a change anywhere is reflected server-wide.

This might be considered a security risk, as any malicious client could
change parameters for the whole system.

> For tunables "relevant without anything at all mounted" say e.g. LNET,
> or uOSS: I don't know.  Also for tunables that must be set before
> services are started: I don't know.  FWIW, right now, servers get
> permanent params out of their config files, but those are read while
> the services are starting (not before).  Module parameters are used to
> set pre-startup params. I really don't like the idea of using modules
> params either (does Solaris use the same API?).  Maybe we need a plain
> old text config file for portability.

> Really, I want something contained entirely within Lustre, but I don't see 
> how to do that unless Lustre is running.
>
> I do think the uOSS somehow needs to have a mountpoint.  How is a uOSS 
> started?
> Maybe by mounting a stub Lustre client that just provides access to 
> /.lustre?

That's just the point - I think you are putting extra requirements in
place to handle the change to move tunables into .../.lustre.  With
the work in bug 14691 to communicate directly with the userspace server
over a socket (a natural process-to-process communication method) there
is no need to have a mountpoint.  Otherwise, we get into some overly
complex framework like FUSE to have a userspace process providing services
to the kernel to provide a mechanism for another userspace process to
talk to it.  Ugh.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.