<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META http-equiv=Content-Type content="text/html; charset=us-ascii">
<META content="MSHTML 6.00.6000.16735" name=GENERATOR></HEAD>
<BODY>
<DIV dir=ltr align=left><SPAN class=921312720-28012009><FONT face=Arial
color=#0000ff size=2>Hi Arden,</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=921312720-28012009><FONT face=Arial
color=#0000ff size=2></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=921312720-28012009><FONT face=Arial
color=#0000ff size=2>Are you obtaining more than 100 MB/sec from one client
to one OST? Given that you are using 802.3ad link aggregation, it will determine
the physical NIC by the other party's MAC address. So having multiple OST and
multiple clients will improve the chances of using more than one NIC of the
bonding.</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=921312720-28012009><FONT face=Arial
color=#0000ff size=2></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=921312720-28012009><FONT face=Arial
color=#0000ff size=2>What is the maximum performance you obtain on the client
with two 1GbE?</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=921312720-28012009><FONT face=Arial
color=#0000ff size=2></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=921312720-28012009><FONT face=Arial
color=#0000ff size=2>jeff</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=921312720-28012009><FONT face=Arial
color=#0000ff size=2></FONT></SPAN> </DIV>
<DIV><FONT face=Arial color=#0000ff size=2></FONT> </DIV>
<DIV><FONT size=2></FONT> </DIV><BR>
<BLOCKQUOTE
style="PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #0000ff 2px solid; MARGIN-RIGHT: 0px">
<DIV class=OutlookMessageHeader lang=en-us dir=ltr align=left>
<HR tabIndex=-1>
<FONT face=Tahoma size=2><B>From:</B> lustre-discuss-bounces@lists.lustre.org
[mailto:lustre-discuss-bounces@lists.lustre.org] <B>On Behalf Of </B>Arden
Wiebe<BR><B>Sent:</B> Sunday, January 25, 2009 12:08 AM<BR><B>To:</B>
lustre-discuss@lists.lustre.org<BR><B>Subject:</B> Re: [Lustre-discuss]
Plateau around 200MiB/s bond0<BR></FONT><BR></DIV>
<DIV></DIV><FONT face=Arial color=#0000ff size=2></FONT>
<TABLE cellSpacing=0 cellPadding=0 border=0>
<TBODY>
<TR>
<TD vAlign=top>So if one OST gets 200MiB/s and another OST gets 200MiB/s
does that make 400 MiB/s or this is not how to calculate
throughput? I will eventually plug the right sequence into iozone
to measure it. <BR><BR>From my perspective it looks like
ioio.ca/ioio.jpg ioio.ca/lustreone.png ioio.ca/lustretwo.png
ioio.ca/lustrethree.png ioio.ca/lustrefour.png<BR><BR>--- On <B>Sat,
1/24/09, Arden Wiebe <I><albert682@yahoo.com></I></B> wrote:<BR>
<BLOCKQUOTE
style="PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: rgb(16,16,255) 2px solid"><BR>From:
Arden Wiebe <albert682@yahoo.com><BR>Subject: [Lustre-discuss]
Plateau around 200MiB/s bond0<BR>To:
lustre-discuss@lists.lustre.org<BR>Date: Saturday, January 24, 2009,
6:04 PM<BR><BR>
<DIV id=yiv426997758>
<TABLE cellSpacing=0 cellPadding=0 border=0>
<TBODY>
<TR>
<TD
style="FONT-FAMILY: inherit; font-size-adjust: inherit; font-stretch: inherit; -x-system-font: none"
vAlign=top>1-2948-SFP Plus Baseline 3Com Switch<BR>1-MGS
bond0(eth0,eth1,eth2,eth3,eth4,eth5) raid1<BR>1-MDT
bond0(eth0,eth1,eth2,eth3,eth4,eth5) raid1<BR>2-OSS
bond0(eth0,eth1,eth2,eth3,eth4,eth5) raid6<BR>1-MGS-CLIENT
bond0(eth0,eth1,eth2,eth3,eth4,eth5)<BR>1-CLIENT
bond0(eth0,eth1)<BR>1-CLIENT eth0<BR>1-CLIENT eth0<BR><BR>I fail
so far creating external journal for MDT, MGS and OSSx2.
How to add the external journal to /etc/fstab specifically the
output of e2label /dev/sdb followed by what options for
fstab?<BR><BR>[root@lustreone ~]# cat
/proc/fs/lustre/devices<BR> 0 UP mgs MGS MGS 17<BR>
1 UP mgc MGC192.168.0.7@tcp 876c20af-aaec-1da0-5486-1fc61ec8cd15
5<BR> 2 UP lov ioio-clilov-ffff810209363c00
7307490a-4a12-4e8c-56ea-448e030a82e4 4<BR> 3 UP mdc
ioio-MDT0000-mdc-ffff810209363c00
7307490a-4a12-4e8c-56ea-448e030a82e4 5<BR> 4 UP osc
ioio-OST0000-osc-ffff810209363c00
7307490a-4a12-4e8c-56ea-448e030a82e4 5<BR> 5 UP osc
ioio-OST0001-osc-ffff810209363c00
7307490a-4a12-4e8c-56ea-448e030a82e4 5<BR>[root@lustreone ~]#
lfs df
-h<BR>UUID
bytes Used Available Use%
Mounted
on<BR>ioio-MDT0000_UUID
815.0G 534.0M
767.9G 0%
/mnt/ioio[MDT:0]<BR>ioio-OST0000_UUID
3.6T 28.4G
3.4T 0%
/mnt/ioio[OST:0]<BR>ioio-OST0001_UUID
3.6T 18.0G
3.4T 0% /mnt/ioio[OST:1]<BR><BR>filesystem
summary:
7.2T 46.4G
6.8T 0% /mnt/ioio<BR><BR>[root@lustreone ~]#
cat /proc/net/bonding/bond0<BR>Ethernet Channel Bonding Driver:
v3.2.4 (January 28, 2008)<BR><BR>Bonding Mode: IEEE 802.3ad
Dynamic link aggregation<BR>Transmit Hash Policy: layer2
(0)<BR>MII Status: up<BR>MII Polling Interval (ms): 100<BR>Up
Delay (ms): 0<BR>Down Delay (ms): 0<BR><BR>802.3ad info<BR>LACP
rate: slow<BR>Active Aggregator
Info:<BR> Aggregator
ID: 1<BR> Number of
ports: 1<BR> Actor
Key: 17<BR> Partner
Key: 1<BR> Partner Mac
Address: 00:00:00:00:00:00<BR><BR>Slave Interface: eth0<BR>MII
Status: up<BR>Link Failure Count: 1<BR>Permanent HW addr:
00:1b:21:28:77:db<BR>Aggregator ID: 1<BR><BR>Slave Interface:
eth1<BR>MII Status: up<BR>Link Failure Count: 1<BR>Permanent HW
addr: 00:1b:21:28:77:6c<BR>Aggregator ID: 2<BR><BR>Slave
Interface: eth3<BR>MII Status: up<BR>Link Failure Count:
0<BR>Permanent HW addr: 00:22:15:06:3a:94<BR>Aggregator ID:
3<BR><BR>Slave Interface: eth2<BR>MII Status: up<BR>Link Failure
Count: 0<BR>Permanent HW addr: 00:22:15:06:3a:93<BR>Aggregator
ID: 4<BR><BR>Slave Interface: eth4<BR>MII Status: up<BR>Link
Failure Count: 0<BR>Permanent HW addr:
00:22:15:06:3a:95<BR>Aggregator ID: 5<BR><BR>Slave Interface:
eth5<BR>MII Status: up<BR>Link Failure Count: 0<BR>Permanent HW
addr: 00:22:15:06:3a:96<BR>Aggregator ID: 6<BR>[root@lustreone
~]# cat /proc/mdstat<BR>Personalities : [raid1]<BR>md0 : active
raid1 sdb[0] sdc[1]<BR> 976762496
blocks [2/2] [UU]<BR><BR>unused devices:
<none><BR>[root@lustreone ~]# cat
/etc/fstab<BR>LABEL=/
/
ext3
defaults 1
1<BR>tmpfs
/dev/shm
tmpfs
defaults 0
0<BR>devpts
/dev/pts
devpts gid=5,mode=620 0
0<BR>sysfs
/sys
sysfs
defaults 0
0<BR>proc
/proc
proc
defaults 0
0<BR>LABEL=MGS
/mnt/mgs
lustre defaults,_netdev 0
0<BR>192.168.0.7@tcp0:/ioio
/mnt/ioio
lustre defaults,_netdev,noauto 0 0<BR><BR>[root@lustreone
~]# ifconfig<BR>bond0 Link
encap:Ethernet HWaddr
00:1B:21:28:77:DB<BR>
inet addr:192.168.0.7 Bcast:192.168.0.255
Mask:255.255.255.0<BR>
inet6 addr: fe80::21b:21ff:fe28:77db/64
Scope:Link<BR>
UP BROADCAST RUNNING MASTER MULTICAST MTU:9000
Metric:1<BR>
RX packets:5457486 errors:0 dropped:0 overruns:0
frame:0<BR>
TX packets:4665580 errors:0 dropped:0 overruns:0
carrier:0<BR>
collisions:0
txqueuelen:0<BR>
RX bytes:12376680079 (11.5 GiB) TX bytes:34438742885 (32.0
GiB)<BR><BR>eth0 Link
encap:Ethernet HWaddr
00:1B:21:28:77:DB<BR>
inet6 addr: fe80::21b:21ff:fe28:77db/64
Scope:Link<BR>
UP BROADCAST RUNNING SLAVE MULTICAST MTU:9000
Metric:1<BR>
RX packets:3808615 errors:0 dropped:0 overruns:0
frame:0<BR>
TX packets:4664270 errors:0 dropped:0 overruns:0
carrier:0<BR>
collisions:0
txqueuelen:1000<BR>
RX bytes:12290700380 (11.4 GiB) TX bytes:34438581771 (32.0
GiB)<BR>
Base address:0xec00 Memory:febe0000-fec00000<BR><BR>From what I
have read not having an external journal configured for the
OST's is a sure recipie for slowness which I would rather not
have considering the goal is around 350MiB/s or more which
should be obtainable. <BR><BR>Here is how I formated the
raid6 device on both OSS's that have identical
<BR>[root@lustrefour ~]# fdisk -l<BR><BR>Disk /dev/sda: 1000.2
GB, 1000204886016 bytes<BR>255 heads, 63 sectors/track, 121601
cylinders<BR>Units = cylinders of 16065 * 512 = 8225280
bytes<BR><BR> Device
Boot
Start
End Blocks Id
System<BR>/dev/sda1
*
1 121601
976760001 83 Linux<BR><BR>Disk /dev/sdb:
1000.2 GB, 1000204886016 bytes<BR>255 heads, 63 sectors/track,
121601 cylinders<BR>Units = cylinders of 16065 * 512 = 8225280
bytes<BR><BR>Disk /dev/sdb doesn't contain a valid partition
table<BR><BR>Disk /dev/sdc: 1000.2 GB, 1000204886016
bytes<BR>255 heads, 63 sectors/track, 121601 cylinders<BR>Units
= cylinders of 16065 * 512 = 8225280 bytes<BR><BR>Disk /dev/sdc
doesn't contain a valid partition table<BR><BR>Disk /dev/sdd:
1000.2 GB, 1000204886016 bytes<BR>255 heads, 63 sectors/track,
121601 cylinders<BR>Units = cylinders of 16065 * 512 = 8225280
bytes<BR><BR>Disk /dev/sdd doesn't contain a valid partition
table<BR><BR>Disk /dev/sde: 1000.2 GB, 1000204886016
bytes<BR>255 heads, 63 sectors/track, 121601 cylinders<BR>Units
= cylinders of 16065 * 512 = 8225280 bytes<BR><BR>Disk /dev/sde
doesn't contain a valid partition table<BR><BR>Disk /dev/sdf:
1000.2 GB, 1000204886016 bytes<BR>255 heads, 63 sectors/track,
121601 cylinders<BR>Units = cylinders of 16065 * 512 = 8225280
bytes<BR><BR>Disk /dev/sdf doesn't contain a valid partition
table<BR><BR>Disk /dev/sdg: 1000.2 GB, 1000204886016
bytes<BR>255 heads, 63 sectors/track, 121601 cylinders<BR>Units
= cylinders of 16065 * 512 = 8225280 bytes<BR><BR>Disk /dev/sdg
doesn't contain a valid partition table<BR><BR>Disk /dev/sdh:
1000.2 GB, 1000204886016 bytes<BR>255 heads, 63 sectors/track,
121601 cylinders<BR>Units = cylinders of 16065 * 512 = 8225280
bytes<BR><BR>Disk /dev/sdh doesn't contain a valid partition
table<BR><BR>Disk /dev/md0: 4000.8 GB, 4000819183616 bytes<BR>2
heads, 4 sectors/track, 976762496 cylinders<BR>Units = cylinders
of 8 * 512 = 4096 bytes<BR><BR>Disk /dev/md0 doesn't contain a
valid partition table<BR>[root@lustrefour ~]#
<BR><BR>[root@lustrefour ~]# mdadm --create --assume-clean
/dev/md0 --level=6 --chunk=128 --raid-devices=6
/dev/sd[cdefgh]<BR>[root@lustrefour ~]# cat
/proc/mdstat<BR>Personalities : [raid6] [raid5] [raid4]<BR>md0 :
active raid6 sdc[0] sdh[5] sdg[4] sdf[3] sde[2]
sdd[1]<BR> 3907049984 blocks level
6, 128k chunk, algorithm 2 [6/6]
[UUUUUU]<BR>
in: 16674 reads, 16217479 writes; out: 3022788 reads, 32865192
writes<BR>
7712698 in raid5d, 8264 out of stripes, 25661224 handle
called<BR>
reads: 0 for rmw, 1710975 for rcw. zcopy writes: 4864584, copied
writes:
16115932<BR>
0 delayed, 0 bit delayed, 0 active, queues: 0 in, 0
out<BR>
0 expanding overlap<BR><BR><BR>unused devices:
<none><BR><BR>Followed with:<BR><BR>[root@lustrefour ~]#
mkfs.lustre --ost --fsname=ioio --mgsnode=192.168.0.7@tcp0
--mkfsoptions="-J device=/dev/sdb1" --reformat
/dev/md0<BR><BR>[root@lustrefour ~]# mke2fs -b 4096 -O
journal_dev /dev/sdb1<BR><BR>But that is hard to reassemble on
the reboot or at least was before I use e2label and label things
right. Question how to label the external journal in fstab
if at all? Right now only running <BR><BR>[root@lustrefour
~]# mkfs.lustre --fsname=ioio --ost --mgsnode=192.168.0.7@tcp0
--reformat /dev/md0<BR><BR>So just raid6 no external
journal.<BR><BR>[root@lustrefour ~]# cat
/etc/fstab<BR>LABEL=/
/
ext3
defaults 1
1<BR>tmpfs
/dev/shm
tmpfs
defaults 0
0<BR>devpts
/dev/pts
devpts gid=5,mode=620 0
0<BR>sysfs
/sys
sysfs
defaults 0
0<BR>proc
/proc
proc
defaults 0
0<BR>LABEL=ioio-OST0001
/mnt/ost00
lustre defaults,_netdev 0
0<BR>192.168.0.7@tcp0:/ioio
/mnt/ioio
lustre defaults,_netdev,noauto 0 0<BR><BR>[root@lustrefour
~]#<BR><BR><BR>[root@lustreone bin]# ./ost-survey -s 4096
/mnt/ioio<BR>./ost-survey: 01/24/09 OST speed survey on
/mnt/ioio from 192.168.0.7@tcp<BR>Number of Active OST devices :
2<BR>Worst Read OST indx: 0 speed:
38.789337<BR>Best Read OST indx: 1 speed:
40.017201<BR>Read Average: 39.403269 +/- 0.613932
MB/s<BR>Worst Write OST indx: 0 speed:
49.227064<BR>Best Write OST indx: 1 speed:
78.673564<BR>Write Average: 63.950314 +/- 14.723250
MB/s<BR>Ost# Read(MB/s) Write(MB/s)
Read-time
Write-time<BR>----------------------------------------------------<BR>0
38.789
49.227
105.596
83.206<BR>1
40.017
78.674
102.356 52.063<BR>[root@lustreone
bin]# ./ost-survey -s 1024 /mnt/ioio<BR>./ost-survey: 01/24/09
OST speed survey on /mnt/ioio from 192.168.0.7@tcp<BR>Number of
Active OST devices : 2<BR>Worst Read OST indx: 0 speed:
38.559620<BR>Best Read OST indx: 1 speed:
40.053787<BR>Read Average: 39.306704 +/- 0.747083
MB/s<BR>Worst Write OST indx: 0 speed:
71.623744<BR>Best Write OST indx: 1 speed:
82.764897<BR>Write Average: 77.194320 +/- 5.570577
MB/s<BR>Ost# Read(MB/s) Write(MB/s)
Read-time
Write-time<BR>----------------------------------------------------<BR>0
38.560
71.624
26.556
14.297<BR>1
40.054
82.765
25.566 12.372<BR>[root@lustreone
bin]# dd of=/mnt/ioio/bigfileMGS if=/dev/zero
bs=1048576<BR>3536+0 records in<BR>3536+0 records
out<BR>3707764736 bytes (3.7 GB) copied, 38.4775 seconds, 96.4
MB/s<BR><BR>lustreonetwothreefour all have the same for
modprobe.conf<BR><BR>[root@lustrefour ~]# cat
/etc/modprobe.conf<BR>alias eth0 e1000<BR>alias eth1
e1000<BR>alias scsi_hostadapter pata_marvell<BR>alias
scsi_hostadapter1 ata_piix<BR>options lnet networks=tcp<BR>alias
eth2 sky2<BR>alias eth3 sky2<BR>alias eth4 sky2<BR>alias eth5
sky2<BR>alias bond0 bonding<BR>options bonding miimon=100
mode=4<BR>[root@lustrefour ~]# <BR><BR>When do the
same from all clients I can watch ./usr/bin/gnome-system-monitor
and the send and recieve from the various nodes reaches a 209
MiB/s plateau?
Uggh<BR></TD></TR></TBODY></TABLE><BR></DIV><BR>-----Inline Attachment
Follows-----<BR><BR>
<DIV
class=plainMail>_______________________________________________<BR>Lustre-discuss
mailing list<BR><A
href="/mc/compose?to=Lustre-discuss@lists.lustre.org"
ymailto="mailto:Lustre-discuss@lists.lustre.org">Lustre-discuss@lists.lustre.org</A><BR><A
href="http://lists.lustre.org/mailman/listinfo/lustre-discuss"
target=_blank>http://lists.lustre.org/mailman/listinfo/lustre-discuss</A><BR></DIV></BLOCKQUOTE></TD></TR></TBODY></TABLE><BR></BLOCKQUOTE></BODY></HTML>