[Lustre-discuss] Fwd: Understanding lustre setup ..

[linux freaker]

---------- Forwarded message ----------
From: linux freaker
Date: Monday, March 18, 2013
Subject: Understanding lustre setup ..
To: Colin Faber <colin_faber at xyratex.com>




On Monday, March 18, 2013, Colin Faber <colin_faber at xyratex.com> wrote:
> Hi,
>
> On 03/15/2013 10:56 AM, linux freaker wrote:
>>
>> Let me explain in brief the clear picture I would need to understand:
>>
>>
>> Aim: Comparing Hadoop over HDFS Vs Hadoop over Lustre
>>
>> Things I tried:
>>
>> I took 5 machines: 1 MDS, 2 OSS/OST and 2 Lustre Client.(each of 4GB RAM
and 700 GB hard disk).
>> I created around 6 OST on each OSS for size 6GB each through LVM.
>
> Using LVM based OST's can slow things down a lot. Really for best
performance you should be using unpartitioned raw disk for ldiskfs
formating.
>
>>
>>
>>
>> On Wed, Mar 13, 2013 at 8:01 AM, linux freaker <linuxfreaker at gmail.com<mailto:
linuxfreaker at gmail.com>> wrote:
>> >
>> >
>> > On Wednesday, March 13, 2013, Colin Faber <colin_faber at xyratex.com<mailto:
colin_faber at xyratex.com>> wrote:
>> >> Hi,
>> >>
>> >> I'm sorry, It was a busy day for me. I will try and respond
appropriately
>> >> to your questions tomorrow.
>> >>
>> >> -cf
>> >>
>> >> On 03/12/2013 07:45 PM, linux freaker wrote:
>> >>>
>> >>>
>> >>> On Tuesday, March 12, 2013, linux freaker <linuxfreaker at gmail.com<mailto:
linuxfreaker at gmail.com>
>> >>> <mailto:linuxfreaker at gmail.com <mailto:linuxfreaker at gmail.com>>>
wrote:
>> >>> >
>> >>> >
>> >>> > On Tuesday, March 12, 2013, linux freaker <linuxfreaker at gmail.com<mailto:
linuxfreaker at gmail.com>
>> >>> > <mailto:linuxfreaker at gmail.com <mailto:linuxfreaker at gmail.com>>>
wrote:
>> >>> >>
>> >>> >>
>> >>> >> On Tuesday, March 12, 2013, Colin Faber <colin_faber at xyratex.com<mailto:
colin_faber at xyratex.com>
>> >>> >> <mailto:colin_faber at xyratex.com <mailto:colin_faber at xyratex.com>>>
wrote:
>> >>> >>> Hi,
>> >>> >>>
>> >>> >>> On 03/11/2013 08:13 PM, linux freaker wrote:
>> >>> >>>>
>> >>> >>>>
>> >>> >>>> On Tuesday, March 12, 2013, linux freaker <
linuxfreaker at gmail.com <mailto:linuxfreaker at gmail.com>
>> >>> >>>> <mailto:linuxfreaker at gmail.com <mailto:linuxfreaker at gmail.com>>
<mailto:Thanks for response.
I got what you suggested.
so i will have 1 mds, 2 oss and 2 lustre client.I will install hadoop on
both the lustre client.


May I know what are the steps to configure hadoop over lustre? I tried to
configure core-site.xml but no idea what other files chinch I need.
Can you share the steps?
Also, i am trying to run wordcount example over lustre.Please suggest.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20130318/229a7fe2/attachment.htm>