<table cellspacing="0" cellpadding="0" border="0" ><tr><td valign="top" style="font: inherit;">1-2948-SFP Plus Baseline 3Com Switch<br>1-MGS bond0(eth0,eth1,eth2,eth3,eth4,eth5) raid1<br>1-MDT bond0(eth0,eth1,eth2,eth3,eth4,eth5) raid1<br>2-OSS bond0(eth0,eth1,eth2,eth3,eth4,eth5) raid6<br>1-MGS-CLIENT bond0(eth0,eth1,eth2,eth3,eth4,eth5)<br>1-CLIENT bond0(eth0,eth1)<br>1-CLIENT eth0<br>1-CLIENT eth0<br><br>I fail so far creating external journal for MDT, MGS and OSSx2. How to add the external journal to /etc/fstab specifically the output of e2label /dev/sdb followed by what options for fstab?<br><br>[root@lustreone ~]# cat /proc/fs/lustre/devices<br> 0 UP mgs MGS MGS 17<br> 1 UP mgc MGC192.168.0.7@tcp 876c20af-aaec-1da0-5486-1fc61ec8cd15 5<br> 2 UP lov ioio-clilov-ffff810209363c00 7307490a-4a12-4e8c-56ea-448e030a82e4 4<br> 3 UP mdc ioio-MDT0000-mdc-ffff810209363c00 7307490a-4a12-4e8c-56ea-448e030a82e4 5<br> 4 UP osc
ioio-OST0000-osc-ffff810209363c00 7307490a-4a12-4e8c-56ea-448e030a82e4 5<br> 5 UP osc ioio-OST0001-osc-ffff810209363c00 7307490a-4a12-4e8c-56ea-448e030a82e4 5<br>[root@lustreone ~]# lfs df -h<br>UUID bytes Used Available Use% Mounted on<br>ioio-MDT0000_UUID 815.0G 534.0M 767.9G 0% /mnt/ioio[MDT:0]<br>ioio-OST0000_UUID 3.6T 28.4G 3.4T 0% /mnt/ioio[OST:0]<br>ioio-OST0001_UUID 3.6T 18.0G 3.4T 0% /mnt/ioio[OST:1]<br><br>filesystem summary:
7.2T 46.4G 6.8T 0% /mnt/ioio<br><br>[root@lustreone ~]# cat /proc/net/bonding/bond0<br>Ethernet Channel Bonding Driver: v3.2.4 (January 28, 2008)<br><br>Bonding Mode: IEEE 802.3ad Dynamic link aggregation<br>Transmit Hash Policy: layer2 (0)<br>MII Status: up<br>MII Polling Interval (ms): 100<br>Up Delay (ms): 0<br>Down Delay (ms): 0<br><br>802.3ad info<br>LACP rate: slow<br>Active Aggregator Info:<br> Aggregator ID: 1<br> Number of ports: 1<br> Actor Key: 17<br> Partner Key: 1<br> Partner Mac Address: 00:00:00:00:00:00<br><br>Slave Interface: eth0<br>MII Status: up<br>Link Failure Count: 1<br>Permanent HW addr: 00:1b:21:28:77:db<br>Aggregator ID: 1<br><br>Slave Interface:
eth1<br>MII Status: up<br>Link Failure Count: 1<br>Permanent HW addr: 00:1b:21:28:77:6c<br>Aggregator ID: 2<br><br>Slave Interface: eth3<br>MII Status: up<br>Link Failure Count: 0<br>Permanent HW addr: 00:22:15:06:3a:94<br>Aggregator ID: 3<br><br>Slave Interface: eth2<br>MII Status: up<br>Link Failure Count: 0<br>Permanent HW addr: 00:22:15:06:3a:93<br>Aggregator ID: 4<br><br>Slave Interface: eth4<br>MII Status: up<br>Link Failure Count: 0<br>Permanent HW addr: 00:22:15:06:3a:95<br>Aggregator ID: 5<br><br>Slave Interface: eth5<br>MII Status: up<br>Link Failure Count: 0<br>Permanent HW addr: 00:22:15:06:3a:96<br>Aggregator ID: 6<br>[root@lustreone ~]# cat /proc/mdstat<br>Personalities : [raid1]<br>md0 : active raid1 sdb[0] sdc[1]<br> 976762496 blocks [2/2] [UU]<br><br>unused devices: <none><br>[root@lustreone ~]# cat
/etc/fstab<br>LABEL=/ / ext3 defaults 1 1<br>tmpfs /dev/shm tmpfs defaults 0 0<br>devpts /dev/pts devpts gid=5,mode=620 0
0<br>sysfs /sys sysfs defaults 0 0<br>proc /proc proc defaults 0 0<br>LABEL=MGS /mnt/mgs lustre defaults,_netdev 0 0<br>192.168.0.7@tcp0:/ioio
/mnt/ioio lustre defaults,_netdev,noauto 0 0<br><br>[root@lustreone ~]# ifconfig<br>bond0 Link encap:Ethernet HWaddr 00:1B:21:28:77:DB<br> inet addr:192.168.0.7 Bcast:192.168.0.255 Mask:255.255.255.0<br> inet6 addr: fe80::21b:21ff:fe28:77db/64 Scope:Link<br> UP BROADCAST RUNNING MASTER MULTICAST MTU:9000 Metric:1<br> RX packets:5457486 errors:0 dropped:0 overruns:0 frame:0<br> TX packets:4665580 errors:0 dropped:0 overruns:0 carrier:0<br> collisions:0
txqueuelen:0<br> RX bytes:12376680079 (11.5 GiB) TX bytes:34438742885 (32.0 GiB)<br><br>eth0 Link encap:Ethernet HWaddr 00:1B:21:28:77:DB<br> inet6 addr: fe80::21b:21ff:fe28:77db/64 Scope:Link<br> UP BROADCAST RUNNING SLAVE MULTICAST MTU:9000 Metric:1<br> RX packets:3808615 errors:0 dropped:0 overruns:0 frame:0<br> TX packets:4664270 errors:0 dropped:0 overruns:0 carrier:0<br> collisions:0 txqueuelen:1000<br> RX bytes:12290700380 (11.4 GiB) TX bytes:34438581771 (32.0
GiB)<br> Base address:0xec00 Memory:febe0000-fec00000<br><br>From what I have read not having an external journal configured for the OST's is a sure recipie for slowness which I would rather not have considering the goal is around 350MiB/s or more which should be obtainable. <br><br>Here is how I formated the raid6 device on both OSS's that have identical <br>[root@lustrefour ~]# fdisk -l<br><br>Disk /dev/sda: 1000.2 GB, 1000204886016 bytes<br>255 heads, 63 sectors/track, 121601 cylinders<br>Units = cylinders of 16065 * 512 = 8225280 bytes<br><br> Device Boot Start End Blocks Id System<br>/dev/sda1 * 1 121601 976760001 83
Linux<br><br>Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes<br>255 heads, 63 sectors/track, 121601 cylinders<br>Units = cylinders of 16065 * 512 = 8225280 bytes<br><br>Disk /dev/sdb doesn't contain a valid partition table<br><br>Disk /dev/sdc: 1000.2 GB, 1000204886016 bytes<br>255 heads, 63 sectors/track, 121601 cylinders<br>Units = cylinders of 16065 * 512 = 8225280 bytes<br><br>Disk /dev/sdc doesn't contain a valid partition table<br><br>Disk /dev/sdd: 1000.2 GB, 1000204886016 bytes<br>255 heads, 63 sectors/track, 121601 cylinders<br>Units = cylinders of 16065 * 512 = 8225280 bytes<br><br>Disk /dev/sdd doesn't contain a valid partition table<br><br>Disk /dev/sde: 1000.2 GB, 1000204886016 bytes<br>255 heads, 63 sectors/track, 121601 cylinders<br>Units = cylinders of 16065 * 512 = 8225280 bytes<br><br>Disk /dev/sde doesn't contain a valid partition table<br><br>Disk /dev/sdf: 1000.2 GB, 1000204886016 bytes<br>255 heads, 63 sectors/track, 121601
cylinders<br>Units = cylinders of 16065 * 512 = 8225280 bytes<br><br>Disk /dev/sdf doesn't contain a valid partition table<br><br>Disk /dev/sdg: 1000.2 GB, 1000204886016 bytes<br>255 heads, 63 sectors/track, 121601 cylinders<br>Units = cylinders of 16065 * 512 = 8225280 bytes<br><br>Disk /dev/sdg doesn't contain a valid partition table<br><br>Disk /dev/sdh: 1000.2 GB, 1000204886016 bytes<br>255 heads, 63 sectors/track, 121601 cylinders<br>Units = cylinders of 16065 * 512 = 8225280 bytes<br><br>Disk /dev/sdh doesn't contain a valid partition table<br><br>Disk /dev/md0: 4000.8 GB, 4000819183616 bytes<br>2 heads, 4 sectors/track, 976762496 cylinders<br>Units = cylinders of 8 * 512 = 4096 bytes<br><br>Disk /dev/md0 doesn't contain a valid partition table<br>[root@lustrefour ~]# <br><br>[root@lustrefour ~]# mdadm --create --assume-clean /dev/md0 --level=6 --chunk=128 --raid-devices=6 /dev/sd[cdefgh]<br>[root@lustrefour ~]# cat
/proc/mdstat<br>Personalities : [raid6] [raid5] [raid4]<br>md0 : active raid6 sdc[0] sdh[5] sdg[4] sdf[3] sde[2] sdd[1]<br> 3907049984 blocks level 6, 128k chunk, algorithm 2 [6/6] [UUUUUU]<br> in: 16674 reads, 16217479 writes; out: 3022788 reads, 32865192 writes<br> 7712698 in raid5d, 8264 out of stripes, 25661224 handle called<br> reads: 0 for rmw, 1710975 for rcw. zcopy writes: 4864584, copied writes: 16115932<br> 0 delayed, 0 bit delayed, 0 active, queues: 0 in, 0 out<br> 0
expanding overlap<br><br><br>unused devices: <none><br><br>Followed with:<br><br>[root@lustrefour ~]# mkfs.lustre --ost --fsname=ioio --mgsnode=192.168.0.7@tcp0 --mkfsoptions="-J device=/dev/sdb1" --reformat /dev/md0<br><br>[root@lustrefour ~]# mke2fs -b 4096 -O journal_dev /dev/sdb1<br><br>But that is hard to reassemble on the reboot or at least was before I use e2label and label things right. Question how to label the external journal in fstab if at all? Right now only running <br><br>[root@lustrefour ~]# mkfs.lustre --fsname=ioio --ost --mgsnode=192.168.0.7@tcp0 --reformat /dev/md0<br><br>So just raid6 no external journal.<br><br>[root@lustrefour ~]# cat /etc/fstab<br>LABEL=/ / ext3
defaults 1 1<br>tmpfs /dev/shm tmpfs defaults 0 0<br>devpts /dev/pts devpts gid=5,mode=620 0 0<br>sysfs /sys sysfs defaults 0
0<br>proc /proc proc defaults 0 0<br>LABEL=ioio-OST0001 /mnt/ost00 lustre defaults,_netdev 0 0<br>192.168.0.7@tcp0:/ioio /mnt/ioio lustre defaults,_netdev,noauto 0 0<br><br>[root@lustrefour ~]#<br><br><br>[root@lustreone bin]# ./ost-survey -s 4096 /mnt/ioio<br>./ost-survey: 01/24/09 OST speed survey on /mnt/ioio from 192.168.0.7@tcp<br>Number of Active OST devices : 2<br>Worst Read OST indx: 0 speed: 38.789337<br>Best Read OST indx: 1 speed:
40.017201<br>Read Average: 39.403269 +/- 0.613932 MB/s<br>Worst Write OST indx: 0 speed: 49.227064<br>Best Write OST indx: 1 speed: 78.673564<br>Write Average: 63.950314 +/- 14.723250 MB/s<br>Ost# Read(MB/s) Write(MB/s) Read-time Write-time<br>----------------------------------------------------<br>0 38.789 49.227 105.596 83.206<br>1 40.017 78.674 102.356 52.063<br>[root@lustreone bin]# ./ost-survey -s 1024 /mnt/ioio<br>./ost-survey: 01/24/09 OST speed survey on /mnt/ioio from 192.168.0.7@tcp<br>Number of Active OST devices : 2<br>Worst Read OST indx: 0 speed: 38.559620<br>Best Read OST indx: 1 speed: 40.053787<br>Read Average:
39.306704 +/- 0.747083 MB/s<br>Worst Write OST indx: 0 speed: 71.623744<br>Best Write OST indx: 1 speed: 82.764897<br>Write Average: 77.194320 +/- 5.570577 MB/s<br>Ost# Read(MB/s) Write(MB/s) Read-time Write-time<br>----------------------------------------------------<br>0 38.560 71.624 26.556 14.297<br>1 40.054 82.765 25.566 12.372<br>[root@lustreone bin]# dd of=/mnt/ioio/bigfileMGS if=/dev/zero bs=1048576<br>3536+0 records in<br>3536+0 records out<br>3707764736 bytes (3.7 GB) copied, 38.4775 seconds, 96.4 MB/s<br><br>lustreonetwothreefour all have the same for modprobe.conf<br><br>[root@lustrefour ~]# cat /etc/modprobe.conf<br>alias eth0
e1000<br>alias eth1 e1000<br>alias scsi_hostadapter pata_marvell<br>alias scsi_hostadapter1 ata_piix<br>options lnet networks=tcp<br>alias eth2 sky2<br>alias eth3 sky2<br>alias eth4 sky2<br>alias eth5 sky2<br>alias bond0 bonding<br>options bonding miimon=100 mode=4<br>[root@lustrefour ~]# <br><br>When do the same from all clients I can watch ./usr/bin/gnome-system-monitor and the send and recieve from the various nodes reaches a 209 MiB/s plateau? Uggh<br></td></tr></table><br>