[Lustre-discuss] Caching gone crazy

[Daniel Kulinski]

I have a bit of a caching issue I was hoping I could get some help with.
Let me start off with a quick synopsis of our currently installed Lustre
system.

 

3 Sun x4500s running RHEL4.5 with Lustre 1.6.5.  

 

Most clients run patchless kernels with version 1.6.5 of the client.

 

The data this came from was a client with the patched kernel running version
1.6.7.1.

 

All data is served over gigabit connections and in the case of the x4500s
they are serving via 3 bonded GigE connections.

 

Nightly we run our backups starting at 8pm.  Within the next hour to 3 hours
we see the following behavior.  A client that was happily reading along
around 4 to 5MB sec will suddenly start reading around 20 MB per second and
the packet rate skyrockets at this time too.  At this point the application
is reading about a tenth to a fifteenth as slow as previous rates.  

 

Using the tool collect I have noticed some interesting statistics:

The lustre reads drop in a similar proportion to the application performance

The readahead cache hits quadruple or more

The readahead cache discards go from 0 to over 1500

 

Once the application enters this state it stays in such a pattern.  Killing
the job and restarting gives us normal operation.  

 

I am attaching the performance data from a large time period.  The specific
event starts at 20:13:36 and ends about 1.5 hours afterward.  

 

Please let me know if there is something I can tune on the client to change
this behavior or if the backup process will spike this behavior constantly.

 

Thanks,

  Dan Kulinski

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20090605/47c330a2/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: perfdata.zip
Type: application/x-zip-compressed
Size: 970038 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20090605/47c330a2/attachment.bin>