[Lustre-discuss] Understanding lustre setup ..

linux freaker linuxfreaker at gmail.com
Mon Mar 18 03:33:09 PDT 2013


Can you please suggest?

On Mon, Mar 18, 2013 at 1:26 PM, linux freaker <linuxfreaker at gmail.com> wrote:
> Hi Colin,
>
> Thanks for the response. So you mean to say I need to install Hadoop
> on just one lustre client.
> Is it enough to setup Hadoop?  But how come I will start
> datanode.Where will it run?
> Shall I need to keep OSS and MDS untouched for Hadoop?
>
> I have wordcount application testing to be done. Can you suggest me
> the steps to configure Hadoop under this setup?
>
>
> On Tue, Mar 12, 2013 at 12:23 AM, Colin Faber <colin_faber at xyratex.com> wrote:
>> Hi,
>>
>> In general the clients are isolated nodes which only act as single client
>> nodes accessing the greater file system (MGS, MDS, and OSS (OSTS))
>>
>> You can run a client mount on any of these nodes however it's not
>> recommended as it can lead to deadlock and memory contention problems.
>>
>> Ideally your hadoop data source from lustre would be a lustre client. This
>> can consist of a network boot machine which then mounts Lustre FS via it's
>> interconnect.
>>
>>
>> So think of it like this:
>>
>> (LUSTRE: [MGS <-> MDS] <-> [OSS { OST OST OST }]) <-> Client A (mount
>> /mnt/lustre)
>>                |______________________________________^
>>
>> Then Client A acts as data store exporting /mnt/lustre to your hadoop
>> cluster.
>>
>> I hope this makes sense =)
>>
>> -cf
>>
>>
>>
>>
>> On 03/11/2013 11:33 AM, linux freaker wrote:
>>>
>>> Hello,
>>>
>>> I have been reading
>>> http://wiki.lustre.org/images/1/1b/Hadoop_wp_v0.4.2.pdf for setting up
>>> Hadoop over lustre.
>>> Generally in hadoop setup, we have 1 Namenode and various number of
>>> datanodes.
>>> If I want to setup the same keeping Lustre as backend, in the document
>>> it is mentioned that:
>>>
>>> ".............Our experiments run on cluster with 8 nodes in total,
>>> one is mds/namenode, the rest are
>>> OSS/DataNode".
>>>
>>> I wonder where does the Lustre Client fit here?
>>>
>>> For Hadoop to work , we mention filesystem parameter esp /lustre here.
>>> We dont have /lustre on OSS.
>>> How is it possible?
>>> _______________________________________________
>>> Lustre-discuss mailing list
>>> Lustre-discuss at lists.lustre.org
>>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>>
>>



More information about the lustre-discuss mailing list