<div id=":2lp" class="ArwC7c ckChnd"><span style="font-family: courier new,monospace;">Greetings,</span><br style="font-family: courier new,monospace;"><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">We (System Fabric Works) have been retained by Sun to prove concept on</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">integrating the lustre client filesystem with a local disk cache</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">(specifically fscache from Red Hat). </span><br style="font-family: courier new,monospace;">
<br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">Eric Barton and I discussed this several days ago, but I would appreciate <br>others' feedback on the requirements and approach documented</span> <span style="font-family: courier new,monospace;">below. I'm <br>
relatively new to Lustre, so there is a very real possibility</span> <span style="font-family: courier new,monospace;">that I "don't <br>know what I don't know". </span><br style="font-family: courier new,monospace;">
<br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">Motivation</span><br style="font-family: courier new,monospace;"><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">This work is primarily motivated by the need to improve the performance</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">of lustre clients as SMB servers to windows nodes. As I understand it,</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">this need is primarily for file readers.</span><br style="font-family: courier new,monospace;">
<br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">Requirements</span><br style="font-family: courier new,monospace;"><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">1. Enabling fscache should be a mount option, and there should be ioctl</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> support for enabling, disabling and querying a file's fscache usage.</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">2. Data read into the page cache will be asynchronously copied to the</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> disk-based fscache upon arrival.</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">3. If requested data is not present in the page cache, it will be retrieved</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> preferentially from the fscache. If not present in the fscache, data</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> will be read via RPC.</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">4. When pages are reclaimed due to memory pressure, they should remain in</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> the fscache.</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">5. When a user writes a page (if we support fscache for non-read-only opens),</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> the corresponding fscache page must either be invalidated or</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> (more likely) rewritten.</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">6. When a DLM lock is revoked, the entire extent of the lock must be </span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> dropped from the fscache (in addition to dropping any page cache</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> resident pages) - regardless of whether any pages are currently resident</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> in the page cache.</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">7. As sort-of a corollary to #6, DLM locks must not be canceled by the owner</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> as long as pages are resident in the fscache, even if memory pressure</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> reclamation has emptied the page cache for a given file.</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">8. Utilities and test programs will be needed, of course. </span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">9. The fscache must be cleared upon mount or dismount.</span><br style="font-family: courier new,monospace;">
<br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">High Level Design Points</span><br style="font-family: courier new,monospace;"><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">The following is written based primarily on review of the <a href="http://1.6.5.1/" target="_blank">1.6.5.1</a> code.</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">I'm aware that this is not the place for new development, but it was</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">deemed a stable place for initial experimentation. </span><br style="font-family: courier new,monospace;"><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">Req. Notes</span><br style="font-family: courier new,monospace;">
<br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> 1. In current Redhat distributions, fscache is included and</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> NFS includes fscache support, enabled by a mount option.</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> We don't see any problems with doing something similar.</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> A per-file ioctl to enable/disable fscache usage is also seen</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> as straightforward.</span><br style="font-family: courier new,monospace;"><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> 2. When an RPC read (into the page cache) completes, in the</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> ll_ap_completion() function, an asynchronous read to the</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> same offset in the file's fscache object will be initiated.</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> This should not materially impact access time (think dirty page</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> to fscache filesystem).</span><br style="font-family: courier new,monospace;">
<br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> 3. When the readpage method is invoked because a page is not</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> already resident in the page cache, the page will be read</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> first from the fscache. This is non-blocking and (presumably)</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> fast for the non-resident case. If available, the fscache</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> read will proceed asynchronously, after which the page will be</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> valid in the page cache. If not available in the fscache,</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> the RPC read will proceed normally.</span><br style="font-family: courier new,monospace;"><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> 4. Page removal due to memory pressure is triggered by a call to</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> the llap_shrink_cache function. This function should not require</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> any material change, since pages can be removed from the page</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> cache without removal from the fscache in this case. In fact,</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> if this doesn't happen, the fscache will never be read.</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> (note: test coverage will be important here)</span><br style="font-family: courier new,monospace;"><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> 5. It may be reasonable in early code to enable fscache only</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> for read-only opens. However, we don't see any inherent problems</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> with running an asynchronous write to the fscache concurrently</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> with a Lustre RPC write. Note that this approach would *never*</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> have dirty pages exist only in the fscache; if it's dirty it</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> stays in the page cache until it's written via RPC (or RPC</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> AND fscache if we're writing to both places)..</span><br style="font-family: courier new,monospace;">
<br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> 6 & 7 This is where it gets a little more tedious. Let me revert to</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> paragraph form to address these cases below.</span><br style="font-family: courier new,monospace;"><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> 8 Testing will require the following:</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> * ability to query and punch holes in the page cache (already done).</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> * ability to query and punch holes in the fscache (nearly done).</span><br style="font-family: courier new,monospace;">
<br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> 9 I presume that all locks are canceled when a client dismounts</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> a filesystem, in which case it would never be safe to use data</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> in the fscache from a prior mount.</span><br style="font-family: courier new,monospace;"><br style="font-family: courier new,monospace;"><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Lock Revocation</span><br style="font-family: courier new,monospace;"><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">Please apply that "it looks to me like this is how things work" filter here;</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">I am still pretty new to Lustre (thanks). My questions are summarized</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">after the the text of this section.</span><br style="font-family: courier new,monospace;">
<br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">As of <a href="http://1.6.5.1/" target="_blank">1.6.5.1</a>, DLM locks keep a list of page-cached pages</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">(lock->l_extents_list contains osc_async_page structs for all currently</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">cached pages - and I think the word extent is used both for each page cached</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">under a lock, and to describe a locked region...is this right?). If a lock</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">is revoked, that list is torn down and the pages are freed. Pages are also</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">removed from that list when they are freed due to memory pressure, making</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">that list sparse with regard to the actual region of the lock. </span><br style="font-family: courier new,monospace;">
<br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">Adding fscache, there will be zero or more page-cache pages in the extent</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">list, as well as zero or more pages in the file object in the fscache. </span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">The primary question, then, is whether a lock will remain valid (i.e. not be</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">voluntarily released) if all of the page-cache pages are freed for</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">non-lock-related reasons (see question 3 below).</span><br style="font-family: courier new,monospace;">
<br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">The way I foresee cleaning up the fscache is by looking at the overall</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">extent of the lock (at release or revocation time), and punching a</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">lock-extent-sized hole in the fscache object prior to looping through</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">the page list (possibly in cache_remove_lock() prior to calling</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">cache_remove_extents_from_lock()). </span><br style="font-family: courier new,monospace;">
<br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">However, that would require finding the inode, which (AFAICS) is not</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">available in that context (ironically, unless the l_extents_list is non-</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">empty, in which case the inode can be found via any of the page structs in</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">the list). I have put in a hack to solve this, but see question 6 below.</span><br style="font-family: courier new,monospace;"><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Summarized questions:</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">Q1: Where can I read up on the unit testing infrastructure for Lustre?</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Q2: Is stale cache already covered by existing unit tests?</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">Q3: Will a DLM lock remain valid (i.e. not be canceled) even if its page</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> list is empty (i.e. all pages have been freed due to memory pressure)?</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">Q4: Will there *always* be a call to cache_remove_lock() when a lock is</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> canceled or revoked? (i.e. is this the place to punch a hole in the</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> fscache object?)</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Q5: for the purpose of punching a hole in a cache object upon lock</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> revocation, can I rely on the lock->l_req_extent structure as the</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> actual extent of the lock?</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">Q6: a) is there a way to find the inode that I've missed?, and</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> b) if not what is the preferred way of giving that function a way to</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> find the inode?</span><br style="font-family: courier new,monospace;">
<br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">...</span><br style="font-family: courier new,monospace;"><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">FYI we have done some experimenting and we have the read path in a</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">demonstrable state, including crude code to effect lock revocation on the</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">fscache contents. The NFS code modularized the fscache hooks pretty nicely,</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">and we have followed that example.</span><br style="font-family: courier new,monospace;"><br style="font-family: courier new,monospace;">Thanks,<br><font color="#888888"><font color="#888888">John Groves<br>
John@SystemFabricWorks.com<br>
+1-512-302-4005</font>
</font></div>