[Lustre-discuss] Caching gone crazy
Daniel Kulinski
dank at weinmangeoscience.com
Fri Jun 5 10:20:48 PDT 2009
I have a bit of a caching issue I was hoping I could get some help with.
Let me start off with a quick synopsis of our currently installed Lustre
system.
3 Sun x4500s running RHEL4.5 with Lustre 1.6.5.
Most clients run patchless kernels with version 1.6.5 of the client.
The data this came from was a client with the patched kernel running version
1.6.7.1.
All data is served over gigabit connections and in the case of the x4500s
they are serving via 3 bonded GigE connections.
Nightly we run our backups starting at 8pm. Within the next hour to 3 hours
we see the following behavior. A client that was happily reading along
around 4 to 5MB sec will suddenly start reading around 20 MB per second and
the packet rate skyrockets at this time too. At this point the application
is reading about a tenth to a fifteenth as slow as previous rates.
Using the tool collect I have noticed some interesting statistics:
The lustre reads drop in a similar proportion to the application performance
The readahead cache hits quadruple or more
The readahead cache discards go from 0 to over 1500
Once the application enters this state it stays in such a pattern. Killing
the job and restarting gives us normal operation.
I am attaching the performance data from a large time period. The specific
event starts at 20:13:36 and ends about 1.5 hours afterward.
Please let me know if there is something I can tune on the client to change
this behavior or if the backup process will spike this behavior constantly.
Thanks,
Dan Kulinski
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20090605/47c330a2/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: perfdata.zip
Type: application/x-zip-compressed
Size: 970038 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20090605/47c330a2/attachment.bin>
More information about the lustre-discuss
mailing list