#LyX 1.5.3 created this file. For more info see http://www.lyx.org/
\lyxformat 276
\begin_document
\begin_header
\textclass newbook
\language english
\inputencoding auto
\font_roman default
\font_sans default
\font_typewriter default
\font_default_family default
\font_sc false
\font_osf false
\font_sf_scale 100
\font_tt_scale 100
\graphics default
\paperfontsize default
\spacing single
\papersize default
\use_geometry false
\use_amsmath 1
\use_esint 0
\cite_engine basic
\use_bibtopic false
\paperorientation portrait
\secnumdepth 3
\tocdepth 3
\paragraph_separation indent
\defskip medskip
\quotes_language english
\papercolumns 1
\papersides 1
\paperpagestyle fancy
\tracking_changes false
\output_changes false
\author "" 
\author "" 
\end_header

\begin_body

\begin_layout Title
High Level Design for MDS and OSS Capability
\end_layout

\begin_layout Date
2008-05-30
\end_layout

\begin_layout Author
Fan Yong
\end_layout

\begin_layout Section
Introduction
\end_layout

\begin_layout Standard
Capability is a piece of data generated by one service - 
\series bold
master service
\series default
, passed to the client and presented by the client to another service -
 
\series bold
slave service,
\series default
 to authorize an action.
 Capability is independent from file permission.
 Lustre introduces capability for the following reasons:
\end_layout

\begin_layout Itemize
Baleful user (client) maybe access the lower items directly by fid guessing
 to skip file permission check which is done for normal path parsing.
\end_layout

\begin_layout Itemize
Baleful user (client) maybe read/write/truncate file object(s) on OSS directly
 without communicating with MDS to bypass the file permission check on MDS.
\end_layout

\begin_layout Itemize
For cross-MDS case (clustered MDSes), baleful user (client) maybe cheat
 one MDS to perform bottom partial operation(s) directly to skip file permission
 check which is done in top partial operation(s) on another MDS(es).
\end_layout

\begin_layout Section
Requirements
\end_layout

\begin_layout Itemize
Client can not modify or fabricate the capability.
\end_layout

\begin_layout Itemize
Each operation on MDS should present the related capabilities which are
 generated in previous path parsing, and the MDS will authorize their validity.
\end_layout

\begin_layout Itemize
When read/write/truncate file object(s) on OSS, the client should present
 the related capability which is generated by MDS, and the OSS will authorize
 its validity.
\end_layout

\begin_layout Section
Functional specification
\end_layout

\begin_layout Subsection
HMAC-SHA1 function
\end_layout

\begin_layout Standard
Capability is generated by master service, client can not modify or fabricate
 the capability by itself.
 A MAC is used to prove that the capability was not modified since it was
 generated by the master service.
 We need a kernel level implementation of this function to compute hashes
 of capability, to protect its integrity.
 It seems that HMAC-MD5 or HMAC-SHA1 would be acceptable for this purpose.
 We use HMAC-SHA1 for that.
\end_layout

\begin_layout Subsection
capability encryption
\end_layout

\begin_layout Standard
Capability needs to be encrypted on the wire so that they cannot be snooped.
 There are two mechanisms can be used for encrypting capability: 
\end_layout

\begin_layout Enumerate
Use the GSS infrastructure.
\end_layout

\begin_layout Enumerate
Build a special purpose encryption mechanism for capability.
\end_layout

\begin_layout Standard
It seems that using the existing GSS infrastructure is some convenient which
 was done in CMD project.
\end_layout

\begin_layout Subsection
capability renew
\end_layout

\begin_layout Standard
To prevent baleful user (client) from decrypting the capability KEY, the
 master service should update its KEY periodically.
\end_layout

\begin_layout Subsection
Feature control
\end_layout

\begin_layout Standard
For each MDT and OST, the feature of capability generating and checking
 can be turned on/off with mount options as 
\begin_inset Quotes eld
\end_inset

-o mds_capa
\begin_inset Quotes erd
\end_inset

, 
\begin_inset Quotes eld
\end_inset

-o nomds_capa
\begin_inset Quotes erd
\end_inset

, 
\begin_inset Quotes eld
\end_inset

-o oss_capa
\begin_inset Quotes erd
\end_inset

, 
\begin_inset Quotes eld
\end_inset

-o nooss_capa
\begin_inset Quotes erd
\end_inset

.
\end_layout

\begin_layout Section
Use cases
\end_layout

\begin_layout Standard
sanity-sec.sh test_5 and test_6.
 
\end_layout

\begin_layout Section
Logic specification
\end_layout

\begin_layout Subsection
capability renew
\end_layout

\begin_layout Standard
The master service maintains three KEYs whose lifecycle is 3N seconds (usually
 N is about half of day), and each capability KEY is assigned with a key
 sequence number, as following:
\end_layout

\begin_layout Itemize
For the master service (on MDS(es)), KEY1 is used to both generate new capabilit
y and check capability.
 KEY2 and KEY3 are used for checking capability only.
 Every N seconds, a new KEY is generated by the master service, and the
 sequence number is increased, the original KEY3 is discarded, the original
 KEY2 becomes KEY3, the original KEY1 becomes KEY2, the new generated KEY
 becomes KEY1.
 The master service sends the new generated KEY with its sequence number
 to the slave service for updating.
\end_layout

\begin_layout Itemize
For the slave service (on MDS(es) and OSS(es)), all the three KEYs are used
 for capability checking.
 Every N seconds, the master service generates a new KEY, and sends it to
 the slave service with its sequence number.
 The slave service update its KEYs: the original KEY3 is discarded, the
 original KEY2 becomes KEY3, the original KEY1 becomes KEY2, the new generated
 KEY becomes KEY1.
\end_layout

\begin_layout Itemize
For the client, it gets capability from the master service which only contains
 KEY's sequence number to indicate with KEY should be used for checking,
 but not the KEY itself.
 The client holds the capability until unused.
 If the holding time is longer than N seconds, it needs to renew the capability
 from the master service.
 When renew on the master service, the KEY for generating such capability
 has been changed from KEY1 to KEY2.
 Consider some network delay, it maybe changed from KEY1 to KEY3.
 But anyway, the capability can be renewed so long as the generating KEY
 exists on the master service.
 The master service generates a new capability with the current KEY1 for
 the renewed one.
\end_layout

\begin_layout Subsection
capability timeout 
\end_layout

\begin_layout Standard
Unfortunately, there maybe some reasons cause the generating KEY has been
 discarded on the master service when the client renews the capability:
\end_layout

\begin_layout Itemize
Long time more than 2N seconds of network failure or the master service
 crash delays the client to renew the capability from the master service.
\end_layout

\begin_layout Itemize
The baleful user (client) uses the old capability to cheat the master service
 for getting valid capability.
\end_layout

\begin_layout Standard
For the first case, we should allow the client to renew the expired capabilities
; for the latter one, it should be forbidden.
 But it is difficult for the master service to distinguish the two cases.
 A baleful user maybe control the client completely, it can fabricate any
 request to cheat the MDS as an open replay or capability renew.
 That means, the two cases can appear the same behavior to the MDS, and
 the MDS can not distinguish them, which maybe cause:
\end_layout

\begin_layout Itemize
If the real expired capabilities renew are denied, maybe cause unexpected
 EIO/EPERM error, then recovery failed.
\end_layout

\begin_layout Itemize
If the fake capabilities renew are approved, will cause capability mechanism
 partially or completely failed.
\end_layout

\begin_layout Standard
Up to now, we have no effective automatical solution for that.
 So here, lustre introduces some factitious control from administrator as
 following: lustre administrator can divide the clients into three sorts
 based on their reliability, and the master service makes decision based
 on that.
\end_layout

\begin_layout Itemize
trusted client
\end_layout

\begin_deeper
\begin_layout Standard
These clients can renew expired capabilities.
\end_layout

\end_deeper
\begin_layout Itemize
untrusted client
\end_layout

\begin_deeper
\begin_layout Standard
These clients can not renew expired capabilities.
\end_layout

\end_deeper
\begin_layout Itemize
baleful client
\end_layout

\begin_deeper
\begin_layout Standard
These clients capabilities check and renew always failed.
\end_layout

\end_deeper
\begin_layout Standard
Lustre administrator can set client reliability through mount parameter
 or by /proc interface.
 For example:
\end_layout

\begin_layout LyX-Code
mount -t lustre -o mds_capa,oss_capa --param mdt.capa_untrusted=10.10.10.1@tcp
 --param mdt.capa_baleful=10.10.10.2@tcp /dev/sda1 /mnt/mdt1
\end_layout

\begin_layout LyX-Code
mount -t lustre -o oss_capa --param obdfilter.capa_baleful=10.10.10.2@tcp /dev/sdb1
 /mnt/ost1
\end_layout

\begin_layout LyX-Code
echo + 10.10.10.3@tcp > /proc/fs/lustre/mdt/*MDT*/capa_untrusted
\end_layout

\begin_layout LyX-Code
echo + 10.10.10.4@tcp > /proc/fs/lustre/mdt/*MDT*/capa_baleful
\end_layout

\begin_layout LyX-Code
echo + 10.10.10.4@tcp > /proc/fs/lustre/obdfilter/*OST*/capa_baleful
\end_layout

\begin_layout LyX-Code
echo - 10.10.10.1@tcp > /proc/fs/lustre/mdt/*MDT*/capa_untrusted
\end_layout

\begin_layout LyX-Code
echo - 10.10.10.2@tcp > /proc/fs/lustre/mdt/*MDT*/capa_baleful
\end_layout

\begin_layout LyX-Code
echo - 10.10.10.2@tcp > /proc/fs/lustre/obdfilter/*OST*/capa_baleful
\end_layout

\begin_layout Subsection
cross-MDS capability
\end_layout

\begin_layout Standard
Some operations on MDS are related with more than one targets, e.g.
 lookup (two), open (two), rename (three or four).
 If the parent target is on MDS1 and the child target is on MDS2, then partial
 operation on MDS2 need to know whether the partial operation on MDS1 has
 been done or not.
 Only send parent target capability to MDS2 is not enough, for MDS2 do not
 know which is the real parent target (on MDS1) and so do not know whether
 the capability is really corresponding to the parent target or not.
\end_layout

\begin_layout Standard
Under such case, the master service on MDS1 should generate the cross-MDS
 capability for the child target, then such cross-MDS capability will be
 sent to MDS2 with its parent target capability together.
 The cross-MDS capability contains parent target fid, the MDS2 can compare
 the fid in parent target capability and in child target cross-MDS capability
 to verify its validity.
 When such partial operation on MDS2 is finished, a normal capability for
 child target is generated by master service on MDS2 and sent back to client.
\end_layout

\begin_layout Standard
The cross-MDS capability just can be used in cross-MDS operations, but can
 not be used in non-cross-MDS operations to avoid baleful user (client)
 to gaining capability by cheating.
 For example, lookup 
\begin_inset Quotes eld
\end_inset

a1/a2/a3
\begin_inset Quotes erd
\end_inset

, a1 and a3 on MDS1, a2 on MDS2, client has a1's capability already.
 The operations sequence is as following:
\end_layout

\begin_layout Enumerate
client sends lookup 
\begin_inset Quotes eld
\end_inset

a1/a2
\begin_inset Quotes erd
\end_inset

 to MDS1 with a1's capability
\end_layout

\begin_layout Enumerate
MDS1 check a1's capability, generates a2's cross-MDS capability, sends it
 back to client
\end_layout

\begin_layout Enumerate
client sends lookup 
\begin_inset Quotes eld
\end_inset

a1/a2
\begin_inset Quotes erd
\end_inset

 to MDS2 with a1's capability and a2's cross-MDS capability
\end_layout

\begin_layout Enumerate
MDS2 check a1's capability and a2's cross-MDS capability, generates a2's
 capability, sends it back to client
\end_layout

\begin_layout Enumerate
client sends lookup 
\begin_inset Quotes eld
\end_inset

a2/a3
\begin_inset Quotes erd
\end_inset

 to MDS2 with a2's capability
\end_layout

\begin_layout Enumerate
MDS2 check a2's capability, generates a3's cross-MDS capability, sends it
 back to client
\end_layout

\begin_layout Enumerate
client sends lookup 
\begin_inset Quotes eld
\end_inset

a2/a3
\begin_inset Quotes erd
\end_inset

 to MDS1 with a2's capability and a3's cross-MDS capability
\end_layout

\begin_layout Enumerate
MDS1 check a2's capability and a3's cross-MDS capability, generates a3's
 capability, sends it back to client
\end_layout

\begin_layout Standard
In the above operations sequence, the client can not use a2's cross-MDS
 capability in the step (7), otherwise, the step (5) and (6) maybe skipped
\end_layout

\begin_layout Subsection
capability cache
\end_layout

\begin_layout Standard
Signing capability is time consuming operation.
 For master service, it needs to generate and check capability, for slave
 service, it needs to check capability.
 They both cache the signed non-timeout capabilities for reducing the performanc
e overhead.
\end_layout

\begin_layout Section
State management
\end_layout

\begin_layout Subsection
Scalability & performance
\end_layout

\begin_layout Standard
There are two factors maybe performance affected:
\end_layout

\begin_layout Itemize
sign capability
\end_layout

\begin_deeper
\begin_layout Standard
HMAC-SHA1 is used to make sure the capability integrality.
 It is time consuming operation.
 Both master service and slave service cache the signed non-timeout capabilities
 for reducing the performance overhead.
 But the cache is limited, and creating a lot of files continuously maybe
 cause the cache failed.
\end_layout

\end_deeper
\begin_layout Itemize
encrypt capability
\end_layout

\begin_deeper
\begin_layout Standard
GSS infrastructure is used to encrypt the capability on wire to avoid being
 snooped.
 But the use of GSS connections is associated with serious overhead.
 This needs to be profiled.
\end_layout

\end_deeper
\begin_layout Standard

\end_layout

\begin_layout Subsection
Protocol changes
\end_layout

\begin_layout Itemize
Each reply for lookup or open request from MDS contains capabilities (or
 cross-MDS capability) associated with the target fid.
\end_layout

\begin_layout Itemize
Each request from client to MDS includes the capability or capabilities
 (or cross-MDS capability) associated with the fid(s) in that request.
\end_layout

\begin_layout Itemize
When open file or set file size successfully, related capability is sent
 from MDS to client allowing to access (read/write/truncate) the objects
 of the file on OSS.
\end_layout

\begin_layout Itemize
Each request from client to OSS includes the capability associated with
 the fid(s) in that request.
\end_layout

\begin_layout Itemize
Client uses 
\begin_inset Quotes eld
\end_inset

MDS_GETATTR
\begin_inset Quotes erd
\end_inset

 RPC to renew its capabilities from MDS with the command flags (mdt_body.valid)
 of 
\begin_inset Quotes eld
\end_inset

OBD_MD_FLOSSCAPA
\begin_inset Quotes erd
\end_inset

.
\end_layout

\begin_layout Itemize
The master service uses obd-ioctl or set info command with 
\begin_inset Quotes eld
\end_inset

CAPA_KEYS
\begin_inset Quotes erd
\end_inset

 to tell slave services to update capability KEY(s).
\end_layout

\begin_layout Subsection
Disk format changes
\end_layout

\begin_layout Standard
The capability KEY, the sequence number, and the last KEY generating time
 are written to master service DISK file (
\begin_inset Quotes eld
\end_inset

capa_keys
\begin_inset Quotes erd
\end_inset

).
 When the master service restarts, such information are loaded for recovery.
\end_layout

\begin_layout Subsection
Recovery changes
\end_layout

\begin_layout Subsubsection
OSS and MDS connection crash
\end_layout

\begin_layout Standard
If the connection crash lasts too long time than one-third (N seconds) of
 the capability KEY lifecycle (3N seconds), the new generated capability
 KEY by the master service can not be sent to the slave service.
 If client gets new OSS capability generated with the new KEY from the master
 service, when it performs IO from/to OSS with such OSS capability, the
 slave service on OSS will find that such OSS capability's KEY sequence
 number is newer than the current KEY1's sequence number, then it knows
 that it has missed some capability KEYs update from the master service,
 and then it should wait for the connection with MDS to be recovered.
 After the connection is recovered, the master service will send the new
 generated capability KEY(s) with sequence number(s) to it for updating.
\end_layout

\begin_layout Subsubsection
client and MDS connection crash
\end_layout

\begin_layout Standard
If the connection crash lasts too long time than two-thirds (2N seconds)
 of the capability KEY lifecycle (3N seconds), the capabilities in the requests
 from client to MDS (including the ones want to be renewed), and the IO
 requests to OSS, maybe expired.
 After the connection is recovered (the client maybe evicted by MDS), the
 non-renew requests which contain expired capabilities will get EPERM from
 MDS or EIO from OSS.
 Then such operations will wait such capabilities to be renewed and be resent.
 On the mean while, client will renew all the expired and close to expired
 capabilities from the master service.
 The master service takes action based on client's reliability to process
 its capabilities renew (for expired one).
\end_layout

\begin_layout Subsubsection
OSS crash
\end_layout

\begin_layout Standard
When the slave service restarts, the master service will send the current
 three capability KEYs and sequence numbers to it for initializing.
\end_layout

\begin_layout Subsubsection
MDS crash
\end_layout

\begin_layout Standard
When the master service restarts, it loads capability KEYs, sequence number
 and the last KEY generating time from the DISK file (
\begin_inset Quotes eld
\end_inset

capa_keys
\begin_inset Quotes erd
\end_inset

).
 If MDS crash lasts too long time than one-third (N seconds) of the capability
 KEY lifecycle (3N seconds), a new capabilities KEY is generated and sent
 to slave service.
 But the original capability KEYs are not discarded for capabilities renew
 and check until the recovery interval is over.
 When recovery finishes, all the expired capability KEY(s) are discarded,
 and the master service sends current capability KEY(s) and sequence numbers
 to slave service for reinitializing.
\end_layout

\begin_layout Section
Alternatives
\end_layout

\begin_layout Standard
The master service divides clients based on their reliability and make decision
 for capability check and expired capability renew.
 Sometimes luster administrator can not know which client is trusted, and
 which one is untrusted or baleful.
 So there maybe some incorrect processes which cause unexpected EIO/EPERM
 error or operation permission out-of-control.
 Whether some more automatical means can be found to replace administrator's
 participation? If we can not resolve expired capability renew issue, can
 we drop the capability KEY timeout mechanism? That means only one capability
 KEY is generated and never to be updated, so the client need not to renew
 the capabilities.
\end_layout

\begin_layout Section
Focus for inspections
\end_layout

\begin_layout Itemize
Processing for expired capability.
\end_layout

\begin_layout Itemize
Once client gains a valid capability, it can be used many times by capability
 renew.
 If the administrator does not set the client as baleful one, the baleful
 user (client) maybe use such capability for cheating to skip some normal
 permission check regardless the up layer directory permission have been
 changed or not.
 For example:
\end_layout

\begin_deeper
\begin_layout Enumerate
client lookup 
\begin_inset Quotes eld
\end_inset

a1/a2/a3/
\begin_inset Quotes erd
\end_inset

 successfully, and gains a3's capability
\end_layout

\begin_layout Enumerate
administrator changes a1's file mode as 
\begin_inset Quotes eld
\end_inset

000
\begin_inset Quotes erd
\end_inset


\end_layout

\begin_layout Enumerate
client lookup 
\begin_inset Quotes eld
\end_inset

a1/a2/a3/a4
\begin_inset Quotes erd
\end_inset

.
 In the normal case, the operation should fail for a1's permission check
 in path parsing.
 But baleful user (client) can send request lookup 
\begin_inset Quotes eld
\end_inset

a3/a4
\begin_inset Quotes erd
\end_inset

 with valid a3's capability directly to MDS, and operation succeeds.
\end_layout

\begin_layout Standard
In such case, the capability take little effect to prevent such abnormal
 operation.
 How to resolve it?
\end_layout

\end_body
\end_document