[Lustre-discuss] High Performance Data warehouse

[Malcolm Cowe]

You might want to look at Greenplum, a DW database designed for parallel 
execution, based on PostGres. This is a commercial application stack and 
isn't exactly cheap, but it addresses data warehousing scalability using 
a highly parallelised model and is worth looking at, since this appears 
to be similar to the problem space that you are interested in.

Sun sells a solution based on the Greenplum DW. The Sun DW Appliance 
comprises a front-end query loader / optimiser using X4100 and one or 
more X4500s as the segment hosts storing the actual data. Queries are 
submitted to the X4100, which compiles the query and creates an 
execution plan which is distributed to each of the segment hosts for 
parallel execution. It's pretty decent.

More information can be found at:

    * http://sun.com/greenplum
    * http://www.greenplum.com

There are a number of presentations and white papers on both sites that 
outline the approach and the interaction between the PostGres developer 
community and GP.

Getting back to Lustre, using a massively parallel file system to host 
one "standard" DB instance that is striped across all the OSTs 
represents an interesting proposal. I'd be careful to ensure that you 
have appropriate controls to ensure data integrity and availability -- 
replication or mirroring for example, in order to protect your database 
from OST failures and so on. I guess you could create an environment 
with many Lustre file systems, each hosting a separate DB instance, then 
relying upon the cluster / availability mechanisms in the DB for 
ensuring reliability. Sounds like fun.

Regards,

Malcolm.



Kiran Patil wrote:
> Hello All,
>
> I want to build massively scalable high-speed clustered database for
> datawarehouse.
>
> Is it possible with Lustre ?
>
> Are there any production datawarehouse implementation hosted on
> cluster filesystem ?
>
> Is it the right approach to host data warehouse ?
>
> Thanks,
> Kiran.
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>   

-- 
<http://www.sun.com> 	
*Malcolm Cowe*
/Solutions Integration Engineer/

*Sun Microsystems, Inc.*
Blackness Road
Linlithgow, West Lothian EH49 7LR UK
Phone: x73602 / +44 1506 673 602
Email: Malcolm.Cowe at Sun.COM