[Lustre-discuss] tuning for small I/O

[Jay Christopherson]

I'm attempting to run a pair of ActiveMQ java instances, using a shared
Lustre filesystem mounted with flock for failover purposes.  There's lots of
ways to do ActiveMQ failover and shared filesystem just happens to be the
easiest.

ActiveMQ, at least the way we are using it, does a lot of small I/O's, like
600 - 800 IOPS worth of 6K I/O's.  When I attempt to use Lustre as the
shared filesystem, I see major IO wait time on the cpu's, like 40 - 50%.  My
OSS's and MDS don't seem to be particularly busy being 90% idle or more
while this is running.  If I remove Lustre from the equation and simply
write to local disk OR to an iSCSI mounted SAN disk, my ActiveMQ instances
don't seem to have any problems.

The disk that is backing the OSS's are all SAS 15K disks in a RAID1 config.
The OSS's (2 of them) each have 8GB of memory and 4 cpu cores and are doing
nothing else except being OSS's.  The MDS has one cpu and 4G of memory and
is 98% idle while under this ActiveMQ load.  The network I am using for
Lustre is dedicated gigabit ethernet and there are 8 clients, two of which
are these ActiveMQ clients.

So, my question is:

1.  What should I be looking at to tune my Lustre FS for this type of IO?
I've tried upping the lru_size of the MDT and OST namespaces in
/proc/fs/lustre/ldlm to 5000 and 2000 respectively, but I don't really see
much difference.  I have also ensured that striping is disabled (lfs
setstripe -d) on the shared directory.

I guess I am just not experienced enough yet with Lustre to know how to
track down and resolve this issue.  I would think Lustre should be able to
handle this load, but I must be missing something.  For the record, NFS was
not able to handle this load either, at least with default export settings
(async was improved, but async is not an option).

- Jay
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20100107/3636a407/attachment.htm>