[Lustre-discuss] OST - low MB/s

[Dr. Hung-Sheng Tsao (LaoTsao)]

not sure I understand Ur setup
which one is the raid6 lun?
which are the individual HD?


Rafael David Tinoco wrote:
> 216MB/s using 8*128 (1024k) as bs. Too low for 8 active disks.. right ? Arround 27MB/s.. from 50MB/s in the "real" disk.
>
> -----Original Message-----
> From: lustre-discuss-bounces at lists.lustre.org [mailto:lustre-discuss-bounces at lists.lustre.org] On Behalf Of Dr. Hung-Sheng Tsao
> (LaoTsao)
> Sent: Thursday, September 10, 2009 5:50 PM
> To: Rafael David Tinoco
> Cc: lustre-discuss at lists.lustre.org
> Subject: Re: [Lustre-discuss] OST - low MB/s
>
> raid 6 chunk size=128k the full strip size will be 128k*8 (for 10 disks 
> 8+2 raid 6)
> in Ur dd test one should use bs=128k*8, then each 8 HDD will be busy
> regards
>
>
> Rafael David Tinoco wrote:
>   
>> With this RAID5 configuration Im getting:
>>
>> root at a02n00:~# dd if=/dev/zero of=/dev/md20 bs=128k count=10000
>>
>> 10000+0 records in
>>
>> 10000+0 records out
>>
>> 1310720000 bytes (1.3 GB) copied, 5.20774 seconds, 252 MB/s
>>
>> root at a02n00:~# dd if=/dev/zero of=/dev/md20 bs=128k count=10000
>>
>> 10000+0 records in
>>
>> 10000+0 records out
>>
>> 1310720000 bytes (1.3 GB) copied, 5.12 seconds, 256 MB/s
>>
>> So, 80MB/s using these md20 as OSTs isnt quite right .
>>
>> *From:* lustre-discuss-bounces at lists.lustre.org 
>> [mailto:lustre-discuss-bounces at lists.lustre.org] *On Behalf Of *Rafael 
>> David Tinoco
>> *Sent:* Thursday, September 10, 2009 4:26 PM
>> *To:* lustre-discuss at lists.lustre.org
>> *Subject:* [Lustre-discuss] OST - low MB/s
>>
>> Hello,
>>
>> I'm having problems now with my "OSTs" throughput.
>>
>> I have 4 OSS each one with 2 OSTs. These OSTs are RAID6 with 10 disks, 
>> chunk size of 128k.
>>
>> These disks are from J4400 (JBOD) connected in multipath using multipathd.
>>
>> Each disk speed is giving me 50MB/s with dd.
>>
>> With lustre, using IOR or DD I can get only arround 80MB/s. I was 
>> expecting for 8 active disks in raid 8*50 = something between 300 and 
>> 400MB/s.
>>
>> avg-cpu: %user %nice %system %iowait %steal %idle
>>
>> 0.00 0.00 6.00 9.06 0.00 84.94
>>
>> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await 
>> svctm %util
>>
>> md10 0.00 0.00 0.00 398.00 0.00 1.55 8.00 0.00 0.00 0.00 0.00
>>
>> md11 0.00 0.00 0.00 380.00 0.00 1.48 8.00 0.00 0.00 0.00 0.00
>>
>> md20 0.00 0.00 0.00 158.00 0.00 79.00 1024.00 0.00 0.00 0.00 0.00
>>
>> md21 0.00 0.00 0.00 159.00 0.00 79.50 1024.00 0.00 0.00 0.00 0.00
>>
>> avg-cpu: %user %nice %system %iowait %steal %idle
>>
>> 0.00 0.00 5.94 9.32 0.00 84.74
>>
>> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await 
>> svctm %util
>>
>> md10 0.00 0.00 0.00 407.50 0.00 1.59 8.00 0.00 0.00 0.00 0.00
>>
>> md11 0.00 0.00 0.00 394.00 0.00 1.54 8.00 0.00 0.00 0.00 0.00
>>
>> md20 0.00 0.00 0.00 159.00 0.00 79.50 1024.00 0.00 0.00 0.00 0.00
>>
>> md21 0.00 0.00 0.00 158.00 0.00 79.00 1024.00 0.00 0.00 0.00 0.00
>>
>> avg-cpu: %user %nice %system %iowait %steal %idle
>>
>> 0.00 0.00 6.37 9.43 0.00 84.21
>>
>> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await 
>> svctm %util
>>
>> md10 0.00 0.00 0.00 410.50 0.00 1.60 8.00 0.00 0.00 0.00 0.00
>>
>> md11 0.00 0.00 0.00 376.00 0.00 1.47 8.00 0.00 0.00 0.00 0.00
>>
>> md20 0.00 0.00 0.00 165.00 0.00 82.50 1024.00 0.00 0.00 0.00 0.00
>>
>> md21 0.00 0.00 0.00 165.00 0.00 82.50 1024.00 0.00 0.00 0.00 0.00
>>
>> Any clues ?
>>
>> Rafael David Tinoco - Sun Microsystems
>>
>> Systems Engineer - High Performance Computing
>>
>> Rafael.Tinoco at Sun.COM - 55.11.5187.2194
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> Lustre-discuss mailing list
>> Lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>>   
>>     
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>
>