[Lustre-discuss] Distributed Object storage lookup of small files

[tao.a.wu at nokia.com]

If you serve 30000 req/s, you will likely need a distributed in-memory cache.  Things like Terracotta or Coherence may work well for your dataset (500 GB), although I haven't used either.

-Tao


________________________________
From: lustre-discuss-bounces at lists.lustre.org [mailto:lustre-discuss-bounces at lists.lustre.org] On Behalf Of ext Pranas Baliuka
Sent: Monday, August 10, 2009 12:11 AM
To: lustre-discuss at lists.lustre.org
Subject: [Lustre-discuss] Distributed Object storage lookup of small files

Dear Lustre experts/users,

I looking for optimal solution of the task:
Internet-scale applications must be designed to process high volumes of transactions.
Describe a design for a system that must process on average 30,000 HTTP requests per second.
For each request, the system must perform a lookup into a dictionary of 50 million words, using a key word passed in via the URL query string.
Each response will consist of a string containing the definition of the word (10 KB or less).

My initial though was using MySQL/Berkeley DB pointing to SAN, but probably lower level solution would be more affordable.
Can I use e.g. QFS storage via Java without DB severer instead. Can SAN be avoided and local HDDs joined to Lustre system?

Task is hypothetical, but would be nice to get feedback from specific technology experts...
Some ideas ;)

I've send similar request to QFS forum and really not sure which product would fit better. Both works as distributed file systems ... and both sounds as convenient storage for particular task.

Thanks,
Pranas

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20090812/ebcbbb53/attachment.htm>