[Lustre-discuss] Understanding lustre setup ..

[linux freaker]

Hi Colin,

Thanks for the response. So you mean to say I need to install Hadoop
on just one lustre client.
Is it enough to setup Hadoop?  But how come I will start
datanode.Where will it run?
Shall I need to keep OSS and MDS untouched for Hadoop?

I have wordcount application testing to be done. Can you suggest me
the steps to configure Hadoop under this setup?


On Tue, Mar 12, 2013 at 12:23 AM, Colin Faber <colin_faber at xyratex.com> wrote:
> Hi,
>
> In general the clients are isolated nodes which only act as single client
> nodes accessing the greater file system (MGS, MDS, and OSS (OSTS))
>
> You can run a client mount on any of these nodes however it's not
> recommended as it can lead to deadlock and memory contention problems.
>
> Ideally your hadoop data source from lustre would be a lustre client. This
> can consist of a network boot machine which then mounts Lustre FS via it's
> interconnect.
>
>
> So think of it like this:
>
> (LUSTRE: [MGS <-> MDS] <-> [OSS { OST OST OST }]) <-> Client A (mount
> /mnt/lustre)
>                |______________________________________^
>
> Then Client A acts as data store exporting /mnt/lustre to your hadoop
> cluster.
>
> I hope this makes sense =)
>
> -cf
>
>
>
>
> On 03/11/2013 11:33 AM, linux freaker wrote:
>>
>> Hello,
>>
>> I have been reading
>> http://wiki.lustre.org/images/1/1b/Hadoop_wp_v0.4.2.pdf for setting up
>> Hadoop over lustre.
>> Generally in hadoop setup, we have 1 Namenode and various number of
>> datanodes.
>> If I want to setup the same keeping Lustre as backend, in the document
>> it is mentioned that:
>>
>> ".............Our experiments run on cluster with 8 nodes in total,
>> one is mds/namenode, the rest are
>> OSS/DataNode".
>>
>> I wonder where does the Lustre Client fit here?
>>
>> For Hadoop to work , we mention filesystem parameter esp /lustre here.
>> We dont have /lustre on OSS.
>> How is it possible?
>> _______________________________________________
>> Lustre-discuss mailing list
>> Lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>
>