[Lustre-discuss] Distributed Object storage lookup of small files

Peter Grandi pg_lus at lus.for.sabi.co.UK
Sat Aug 15 12:03:17 PDT 2009


>From Pranas Baliuka:

pranas> Sent: Monday, August 10, 2009 12:11 AM
pranas> To: lustre-discuss at lists.lustre.org
pranas> Subject: [Lustre-discuss] Distributed Object storage lookup of small files

pranas> Dear Lustre experts/users,
pranas> I looking for optimal solution of the task:

pranas> Internet-scale applications must be designed to process
pranas> high volumes of transactions.  Describe a design for a
pranas> system that must process on average 30,000 HTTP requests
pranas> per second. For each request, the system must perform a
pranas> lookup into a dictionary of 50 million words, using a
pranas> key word passed in via the URL query string.  Each
pranas> response will consist of a string containing the
pranas> definition of the word (10 KB or less).

pranas> Task is hypothetical, but would be nice to get feedback
pranas> from specific technology experts...  Some ideas ;)

This looks like to me an attempt at cheating on a university
assignment or a job interview challenge.

Especially given that something like Lustre looks ridiculous
overkill for such a task (50m word, each on average 7-10 chars
long => only 500MB table, and read-only too), so posting the
question here makes little sense.

But then I have seen a number of ignoramuses happy to use
filesystems instead of incore or storage databases (typical
questions about having several million or hundred million files
almost all of which less than a block long, or directories with
hundreds of thousands or millions of files, especially on the
XFS mailing list).



More information about the lustre-discuss mailing list