Hi ...,<br><br>Thanks for the inputs Adrian, Jeremy, Michael.<br><br>From Adrian's explanation, I gather that the OSC generates 1 RPC to each OST for each file. Since there is only 1 OST in each of the 4 OSS, we only get 128 simultaneous RPCs. So Listing 2000 files would only get us that much speed, right?<br>
<br>Now, each of the OST is around 4.5 TB in size. So say, we reduce the disk size 1.125TB, but increase the number to 4, then we would get 4OSTx32RPCs=128 RPC connections to each OSS, and 512 simultaneous RPCs across the Lustre storage. Wouldn't this increase the listing speed four times over?<br>
<br>Currently, we have around 12GB of RAM on each OSS. I belive, we will have increase this to accommodate the extra 3 OSTs and another 4 OSTs in case of failover. We will also require a proportionate increase in MDS RAM too.<br>
<br>Is my theory right? Is there any catch to it?<br><br>Also, can 1 RPC consume more than 1 I/O thread, say like, it reads from the buffer of one I/O and then moves to the next I/O buffer? Or is it strictly 1 RPC = 1 I/O?<br>
<br>Regards,<br><br><br><br>Indivar Nair<br><br><br><div class="gmail_quote">On Wed, Sep 7, 2011 at 6:40 PM, Michael Barnes <span dir="ltr"><<a href="mailto:Michael.Barnes@jlab.org">Michael.Barnes@jlab.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><br>
Another thing to try is setting vfs_cache_pressure=0 on the OSSes and<br>
periodically setting it to nonzero to reclaim memory. More details<br>
here:<br>
<br>
<a href="http://www.olcf.ornl.gov/wp-content/events/lug2011/4-14-2011/830-900_Robin_Humble_rjh.lug2011.pdf" target="_blank">http://www.olcf.ornl.gov/wp-content/events/lug2011/4-14-2011/830-900_Robin_Humble_rjh.lug2011.pdf</a><br>
<br>
-mb<br>
<div><div></div><div class="h5"><br>
On Sep 6, 2011, at 2:43 AM, Indivar Nair wrote:<br>
<br>
> Hi ...,<br>
><br>
> I have a lustre storage that stores lots of small files i.e. hundreds to<br>
> thousand of 9MB image files.<br>
> While normal file access works fine, the directory listing is extremely<br>
> slow.<br>
> Depending on the number of files in a directory, the listing takes around 5<br>
> - 15 secs.<br>
><br>
> I tried 'ls --color=none' and it worked fine; listed the contents<br>
> immediately.<br>
><br>
> But that doesn't help my cause. I have Samba Gateway Servers, and all users<br>
> access the storage through the gateway. Double clicking on directory takes a<br>
> long long time to display.<br>
><br>
> The cluster consist of -<br>
> - two DRBD Mirrored MDS Servers (Dell R610s) with 10K RPM disks<br>
> - four OSS Nodes (2 Node Cluster (Dell R710s) with a common storage (Dell<br>
> MD3200))<br>
><br>
> The storage consists of 12 x 1TB HDDs on both arrays, in RAID 6<br>
> Configuration.<br>
><br>
> What actually happens when one does a listing like this?<br>
> What can I do to make the listing faster?<br>
> Could it be an MDS issue?<br>
> Some site suggested that this could be caused due to '-o flock' switch. Is<br>
> it so?<br>
><br>
> Kindly Help.<br>
> The storage is in Production, and this is causing a lot of issues.<br>
><br>
> Regards,<br>
><br>
><br>
> Indivar Nair<br>
</div></div><div class="im">> _______________________________________________<br>
> Lustre-discuss mailing list<br>
> <a href="mailto:Lustre-discuss@lists.lustre.org">Lustre-discuss@lists.lustre.org</a><br>
> <a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss" target="_blank">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a><br>
<br>
</div>--<br>
+-----------------------------------------------<br>
<font color="#888888">| Michael Barnes<br>
|<br>
| Thomas Jefferson National Accelerator Facility<br>
| Scientific Computing Group<br>
| 12000 Jefferson Ave.<br>
| Newport News, VA 23606<br>
| (757) 269-7634<br>
+-----------------------------------------------<br>
<br>
<br>
<br>
<br>
</font></blockquote></div><br>