The statahead_max is actually at llite.*.statahead_max not mdc.<br><br>Jeremy<br><br><div class="gmail_quote">On Tue, Sep 6, 2011 at 10:13 PM, Jeremy Filizetti <span dir="ltr"><<a href="mailto:jeremy.filizetti@gmail.com">jeremy.filizetti@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">You'll see a small benefit from mdc.*.statahead_max which will grab most of the information for the stat from the MDT, but you still pay a penalty because the glimpse for each OSC will be synchronous (but issued simultaneously to each OSC) IIRC. There was some patches mentioned quite a while back for asynchronous glimpses but I didn't see much of a difference and never tried to narrow down why that was. It also wasn't production ready.<br>
<br>Hopefully you have extended attributes disabled for your Samba servers as well because it could be much worse if you have to listxattrs and then fetch each one. Every file in Lustre also has a trusted.lov or lustre.lov xattr that stores the striping information, so for every file you'd be doing a listxattr and getxattr. Quite a while back I wrote a samba module to filter out Lustre striping xattrs but not others, which helped a little.<br>
<br>The last thing which doesn't sound like it is affecting you is how Lustre clients read directory pages. Today in 1.8 and 2.0 each directory is read one page at a time (4k on x86_64). If you have hundreds of thousands of files your directories they will be larger then a single page. There are some patches already done by Whamcloud (<a href="http://jira.whamcloud.com/browse/LU-5" target="_blank">http://jira.whamcloud.com/browse/LU-5</a>) that allows had the directory readpage readahead a full bulk RPC which is 1 MB. Depending on your environment this could be almost a 256x speed up. This had a huge impact for me, but only seems significant if your not doing a stat on every file.<br>
<font color="#888888">
<br>Jeremy <br></font><div><div></div><div class="h5"><br><div class="gmail_quote">On Tue, Sep 6, 2011 at 8:21 PM, Indivar Nair <span dir="ltr"><<a href="mailto:indivar.nair@techterra.in" target="_blank">indivar.nair@techterra.in</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I have 1 OST Per OSS.<br>I have set the stripe to Lustre default. Should I still change the stripe count to 1?<br>Yes, the reads are sequential.<br><br>1. Will increasing / decreasing the number of max_rpcs_in_flight help?<br>
It is set to 32 now.<br><br>2. Just for experimenting, would it speed up listing if 1 OSS served 2 OSTs, thereby reducing the number of OSS and RPCs?<br><br>Regards,<br><font color="#888888"><br><br>Indivar Nair</font><div>
<div></div><div><br><br><div class="gmail_quote">
On Wed, Sep 7, 2011 at 1:28 AM, Adrian Ulrich <span dir="ltr"><<a href="mailto:adrian@blinkenlights.ch" target="_blank">adrian@blinkenlights.ch</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div><br>
> While normal file access works fine, the directory listing is extremely<br>
> slow.<br>
> Depending on the number of files in a directory, the listing takes around 5<br>
> - 15 secs.<br>
><br>
> I tried 'ls --color=none' and it worked fine; listed the contents<br>
> immediately.<br>
<br>
</div>That's because 'ls --color=always|auto' does an lstat() of each file (--color=none doesn't) which causes Lustre to send:<br>
<br>
- 1 RPC to the MDS per file<br>
- 1 RPC (per file) to EACH OSS where the file is stored to get the filesize<br>
<br>
Some time ago i've created a patch to speed up 'ls' while keeping (most) of the colors<br>
(<a href="https://github.com/adrian-bl/patchwork/blob/master/coreutils/ls/ls-lustre.diff" target="_blank">https://github.com/adrian-bl/patchwork/blob/master/coreutils/ls/ls-lustre.diff</a>)<br>
<br>
But patching samba will not be possible in your case as it really needs the information returned by stat().<br>
<div><br>
<br>
> Double clicking on directory takes a long long time to display.<br>
<br>
</div>Attach `strace' to samba: It will probably be busy doing lstat() which is a 'slow' operation on Lustre in any case.<br>
<div><br>
<br>
<br>
> The cluster consist of -<br>
> - two DRBD Mirrored MDS Servers (Dell R610s) with 10K RPM disks<br>
> - four OSS Nodes (2 Node Cluster (Dell R710s) with a common storage (Dell<br>
> MD3200))<br>
<br>
</div>How many OSTs do you have per OSS?<br>
What's your stripe setting? Setting the stripe to 1 could give you a huge speedup (without affecting normal I/O as i assume that the 9MB files are read/written sequentially)<br>
<br>
<br>
Regards,<br>
Adrian<br>
<br>
<br>
--<br>
RFC 1925:<br>
(11) Every old idea will be proposed again with a different name and<br>
a different presentation, regardless of whether it works.<br>
<br>
_______________________________________________<br>
Lustre-discuss mailing list<br>
<a href="mailto:Lustre-discuss@lists.lustre.org" target="_blank">Lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss" target="_blank">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a><br>
</blockquote></div><br>
</div></div><br>_______________________________________________<br>
Lustre-discuss mailing list<br>
<a href="mailto:Lustre-discuss@lists.lustre.org" target="_blank">Lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss" target="_blank">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a><br>
<br></blockquote></div><br>
</div></div></blockquote></div><br>