[Lustre-discuss] RAID usage?
Arden Wiebe
albert682 at yahoo.com
Tue Jan 22 13:18:20 PST 2013
Jon;
Found this one today and noticed no response so I thought I would give it a go. From the manual it states to use zero partitions when possible and to deal entirely with the block device itself. The manual also states that the creation of an external journal is recommended. I used a separate 1TB disk for the creation of the external journal and in some instances swap space.
Here I outline a small capable robust Configuration for under $40000.00:
[root at ns1 ~]# cat /proc/mdstat
Personalities : [raid1] [raid10]
md3 : active raid10 sdc[0] sdh[5] sdg[4] sdf[3] sde[2] sdd[1]
2930287488 blocks 64K chunks 2 near-copies [6/6] [UUUUUU]
md1 : active raid1 sda3[0] sdb3[1]
891520 blocks [2/2] [UU]
md0 : active raid1 sdb1[1] sda1[0]
965634880 blocks [2/2] [UU]
unused devices: <none>
[root at ns1 ~]# lctl dl
0 UP mgc MGC192.168.0.10 at tcp f038d71d-792a-16cd-9909-e24dbe6dcfd2 5
1 UP ost OSS OSS_uuid 3
2 UP obdfilter ioio-OST0002 ioio-OST0002_UUID 13
[root at ns1 ~]# cat /etc/fstab
/dev/md0 / ext3 defaults 1 1
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
LABEL=SWAP-sdb2 swap swap defaults 0 0
LABEL=SWAP-sda2 swap swap defaults 0 0
/dev/md3 /mnt/data/ost02 lustre defaults,_netdev 0 0
#/dev/md4 /mnt/flash/ost03 lustre defaults,_netdev 0 0
[root at ns1 ~]# cat /proc/net/bonding
cat: /proc/net/bonding: Is a directory
[root at ns1 ~]# cat /proc/net/bond
cat: /proc/net/bond: No such file or directory
[root at ns1 ~]# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.4.0-1 (October 7, 2008)
Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer3+4 (1)
MII Status: up
MII Polling Interval (ms): 10
Up Delay (ms): 0
Down Delay (ms): 0
802.3ad info
LACP rate: slow
Active Aggregator Info:
Aggregator ID: 1
Number of ports: 5
Actor Key: 17
Partner Key: 37823
Partner Mac Address: 00:22:57:9a:ea:00
Slave Interface: eth0
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:1b:21:28:74:d8
Aggregator ID: 1
Slave Interface: eth1
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 1
Permanent HW addr: 00:1b:21:28:76:d2
Aggregator ID: 1
Slave Interface: eth2
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:22:15:06:39:2f
Aggregator ID: 1
Slave Interface: eth3
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:22:15:06:39:30
Aggregator ID: 1
Slave Interface: eth4
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:22:15:06:39:31
Aggregator ID: 1
[root at ns1 ~]# ifconfig
bond0 Link encap:Ethernet HWaddr 00:1B:21:28:74:D8
inet addr:192.168.0.19 Bcast:192.168.0.255 Mask:255.255.255.0
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:89680747 errors:0 dropped:493712 overruns:0 frame:0
TX packets:54703009 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:87798998004 (81.7 GiB) TX bytes:137015921248 (127.6 GiB)
eth0 Link encap:Ethernet HWaddr 00:1B:21:28:74:D8
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:6158033 errors:0 dropped:0 overruns:0 frame:0
TX packets:4754443 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2572939264 (2.3 GiB) TX bytes:6124172876 (5.7 GiB)
eth1 Link encap:Ethernet HWaddr 00:1B:21:28:74:D8
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:1715604 errors:0 dropped:0 overruns:0 frame:0
TX packets:3837069 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:105078923 (100.2 MiB) TX bytes:2128255017 (1.9 GiB)
eth2 Link encap:Ethernet HWaddr 00:1B:21:28:74:D8
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:141703 errors:0 dropped:0 overruns:0 frame:0
TX packets:3999225 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:14455587 (13.7 MiB) TX bytes:2440673445 (2.2 GiB)
Interrupt:74
eth3 Link encap:Ethernet HWaddr 00:1B:21:28:74:D8
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:63575447 errors:0 dropped:493696 overruns:0 frame:0
TX packets:14850445 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:83036911085 (77.3 GiB) TX bytes:122278775783 (113.8 GiB)
Interrupt:98
eth4 Link encap:Ethernet HWaddr 00:1B:21:28:74:D8
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:18089968 errors:0 dropped:16 overruns:0 frame:0
TX packets:27261838 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2069613561 (1.9 GiB) TX bytes:4044047045 (3.7 GiB)
Interrupt:169
eth5 Link encap:Ethernet HWaddr 00:22:15:06:39:32
inet addr:XX.XX.XXX.XXX Bcast:96.53.120.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:8650228 errors:0 dropped:4716 overruns:0 frame:0
TX packets:4278085 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:7047416207 (6.5 GiB) TX bytes:1721168584 (1.6 GiB)
Interrupt:177
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:211949 errors:0 dropped:0 overruns:0 frame:0
TX packets:211949 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:48836875 (46.5 MiB) TX bytes:48836875 (46.5 MiB)
[root at ns1 ~]# uptime
12:25:59 up 7 days, 16:49, 5 users, load average: 0.00, 0.02, 0.00]#
**Firewall/DNS External IP eth5 OSS/OST
[root at ns2 ~]# cat /proc/mdstat
Personalities : [raid1] [raid10]
md3 : active raid10 sdc[0] sdh[5] sdg[4] sdf[3] sde[2] sdd[1]
2930287488 blocks 64K chunks 2 near-copies [6/6] [UUUUUU]
md1 : active raid1 sda3[0] sdb3[1]
891520 blocks [2/2] [UU]
md0 : active raid1 sdb1[1] sda1[0]
965634880 blocks [2/2] [UU]
unused devices: <none>
[root at ns2 ~]# lctl dl
0 UP mgc MGC192.168.0.10 at tcp 0e72db5f-9495-e31e-85c2-8d31a009d130 5
1 UP ost OSS OSS_uuid 3
2 UP obdfilter ioio-OST0001 ioio-OST0001_UUID 13
[root at ns2 ~]# cat /etc/fstab
/dev/md0 / ext3 defaults 1 1
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
LABEL=SWAP-sdb2 swap swap defaults 0 0
LABEL=SWAP-sda2 swap swap defaults 0 0
/dev/md3 /mnt/data/ost01 lustre defaults,_netdev 0 0
#/dev/md4 /mnt/flash/ost02 lustre defaults,_netdev 0 0
[root at ns2 ~]# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.4.0-1 (October 7, 2008)
Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer3+4 (1)
MII Status: up
MII Polling Interval (ms): 10
Up Delay (ms): 0
Down Delay (ms): 0
802.3ad info
LACP rate: slow
Active Aggregator Info:
Aggregator ID: 1
Number of ports: 5
Actor Key: 17
Partner Key: 37823
Partner Mac Address: 00:22:57:9a:ea:00
Slave Interface: eth0
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:1b:21:2a:1c:3c
Aggregator ID: 1
Slave Interface: eth1
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:1b:21:2a:1c:34
Aggregator ID: 1
Slave Interface: eth2
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:22:15:06:38:b3
Aggregator ID: 1
Slave Interface: eth3
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:22:15:06:38:b4
Aggregator ID: 1
Slave Interface: eth4
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:22:15:06:38:b5
Aggregator ID: 1
[root at ns2 ~]# ifconfig
bond0 Link encap:Ethernet HWaddr 00:1B:21:2A:1C:3C
inet addr:192.168.0.13 Bcast:192.168.0.255 Mask:255.255.255.0
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:73883791 errors:0 dropped:405040 overruns:0 frame:0
TX packets:89400059 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:74258769521 (69.1 GiB) TX bytes:116179989862 (108.2 GiB)
eth0 Link encap:Ethernet HWaddr 00:1B:21:2A:1C:3C
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:53224496 errors:0 dropped:2 overruns:0 frame:0
TX packets:37354638 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:70370076932 (65.5 GiB) TX bytes:51859648442 (48.2 GiB)
eth1 Link encap:Ethernet HWaddr 00:1B:21:2A:1C:3C
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:20567731 errors:0 dropped:0 overruns:0 frame:0
TX packets:41842603 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:3880185151 (3.6 GiB) TX bytes:33224132469 (30.9 GiB)
eth2 Link encap:Ethernet HWaddr 00:1B:21:2A:1C:3C
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:46143 errors:0 dropped:0 overruns:0 frame:0
TX packets:2055392 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:3573296 (3.4 MiB) TX bytes:487616806 (465.0 MiB)
Interrupt:74
eth3 Link encap:Ethernet HWaddr 00:1B:21:2A:1C:3C
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:22228 errors:0 dropped:0 overruns:0 frame:0
TX packets:2406691 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2445080 (2.3 MiB) TX bytes:675387313 (644.0 MiB)
Interrupt:98
eth4 Link encap:Ethernet HWaddr 00:1B:21:2A:1C:3C
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:23207 errors:0 dropped:405038 overruns:0 frame:0
TX packets:5740755 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2489986 (2.3 MiB) TX bytes:29933208120 (27.8 GiB)
Interrupt:169
eth5 Link encap:Ethernet HWaddr 00:22:15:06:38:B6
inet addr:XX.XX.XXX.XXX Bcast:96.53.120.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:5679458 errors:0 dropped:124836 overruns:0 frame:0
TX packets:1640529 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:329507734 (314.2 MiB) TX bytes:1288265410 (1.1 GiB)
Interrupt:177
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:33230 errors:0 dropped:0 overruns:0 frame:0
TX packets:33230 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:24813551 (23.6 MiB) TX bytes:24813551 (23.6 MiB)
[root at ns2 ~]# uptime
12:28:08 up 7 days, 16:52, 1 user, load average: 0.37, 0.25, 0.19
** Firewall/DNS External IP OSS/OST
[root at ns3 ~]# cat /proc/mdstat
Personalities : [raid1] [raid10]
md3 : active raid10 sdc[0] sdh[5] sdg[4] sdf[3] sde[2] sdd[1]
2930287488 blocks 64K chunks 2 near-copies [6/6] [UUUUUU]
md1 : active raid1 sda3[0] sdb3[1]
891520 blocks [2/2] [UU]
md0 : active raid1 sdb1[1] sda1[0]
965634880 blocks [2/2] [UU]
unused devices: <none>
[root at ns3 ~]# lctl dl
0 UP mgc MGC192.168.0.10 at tcp 75c57524-4f99-6424-f78a-eaff14d115bd 5
1 UP ost OSS OSS_uuid 3
2 UP obdfilter ioio-OST0000 ioio-OST0000_UUID 13
[root at ns3 ~]# cat /etc/fstab
/dev/md0 / ext3 defaults 1 1
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
LABEL=SWAP-sdb2 swap swap defaults 0 0
LABEL=SWAP-sda2 swap swap defaults 0 0
/dev/md3 /mnt/data/ost00 lustre defaults,_netdev 0 0
#/dev/md4 /mnt/flash/ost01 lustre defaults,_netdev 0 0
[root at ns3 ~]# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.4.0-1 (October 7, 2008)
Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer3+4 (1)
MII Status: up
MII Polling Interval (ms): 10
Up Delay (ms): 0
Down Delay (ms): 0
802.3ad info
LACP rate: slow
Active Aggregator Info:
Aggregator ID: 5
Number of ports: 6
Actor Key: 17
Partner Key: 37823
Partner Mac Address: 00:22:57:9a:ea:00
Slave Interface: eth0
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:22:15:06:3a:0f
Aggregator ID: 5
Slave Interface: eth1
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:22:15:06:3a:10
Aggregator ID: 5
Slave Interface: eth2
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:22:15:06:3a:11
Aggregator ID: 5
Slave Interface: eth3
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:22:15:06:3a:12
Aggregator ID: 5
Slave Interface: eth4
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:1b:21:2a:17:76
Aggregator ID: 5
Slave Interface: eth5
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:1b:21:2a:1c:dc
Aggregator ID: 5
[root at ns3 ~]# ifconfig
bond0 Link encap:Ethernet HWaddr 00:22:15:06:3A:0F
inet addr:192.168.0.7 Bcast:192.168.0.255 Mask:255.255.255.0
inet6 addr: fe80::222:15ff:fe06:3a0f/64 Scope:Link
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:91510283 errors:0 dropped:364706 overruns:0 frame:0
TX packets:69302068 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:55958008995 (52.1 GiB) TX bytes:223016561848 (207.7 GiB)
eth0 Link encap:Ethernet HWaddr 00:22:15:06:3A:0F
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:42433 errors:0 dropped:2 overruns:0 frame:0
TX packets:9926913 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:3374702 (3.2 MiB) TX bytes:851799191 (812.3 MiB)
Interrupt:74
eth1 Link encap:Ethernet HWaddr 00:22:15:06:3A:0F
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:1712310 errors:0 dropped:0 overruns:0 frame:0
TX packets:266655 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:80908257 (77.1 MiB) TX bytes:208988501 (199.3 MiB)
Interrupt:98
eth2 Link encap:Ethernet HWaddr 00:22:15:06:3A:0F
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:8953421 errors:0 dropped:364702 overruns:0 frame:0
TX packets:27700345 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:952280437 (908.1 MiB) TX bytes:182598635220 (170.0 GiB)
Interrupt:169
eth3 Link encap:Ethernet HWaddr 00:22:15:06:3A:0F
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:23273 errors:0 dropped:2 overruns:0 frame:0
TX packets:5192617 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2493150 (2.3 MiB) TX bytes:1880125062 (1.7 GiB)
Interrupt:177
eth4 Link encap:Ethernet HWaddr 00:22:15:06:3A:0F
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:38058348 errors:0 dropped:0 overruns:0 frame:0
TX packets:1653728 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:3642127587 (3.3 GiB) TX bytes:759108694 (723.9 MiB)
eth5 Link encap:Ethernet HWaddr 00:22:15:06:3A:0F
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:42720504 errors:0 dropped:0 overruns:0 frame:0
TX packets:24561812 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:51276825174 (47.7 GiB) TX bytes:36717905520 (34.1 GiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:2798 errors:0 dropped:0 overruns:0 frame:0
TX packets:2798 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:4431535 (4.2 MiB) TX bytes:4431535 (4.2 MiB)
[root at ns3 ~]# uptime
12:30:58 up 7 days, 16:55, 1 user, load average: 0.00, 0.00, 0.00
** OSS/OST LAMP webroot=/var/www/html ie: not Lustre mounted
[root at ten ~]# cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 sdc[0] sdb[1]
976762496 blocks [2/2] [UU]
unused devices: <none>
[root at ten ~]# lctl dl
0 UP mgs MGS MGS 17
1 UP mgc MGC192.168.0.10 at tcp 24fb4ce0-ea3f-1324-5d61-ea9f45f0baa8 5
2 UP mdt MDS MDS_uuid 3
3 UP lov ioio-mdtlov ioio-mdtlov_UUID 4
4 UP mds ioio-MDT0000 ioio-MDT0000_UUID 11
5 UP osc ioio-OST0000-osc ioio-mdtlov_UUID 5
6 UP osc ioio-OST0001-osc ioio-mdtlov_UUID 5
7 UP osc ioio-OST0002-osc ioio-mdtlov_UUID 5
8 UP lov ioio-clilov-ffff8102179aa000 b47f5fbf-f9dc-7660-3488-6a0d3420f386 4
9 UP mdc ioio-MDT0000-mdc-ffff8102179aa000 b47f5fbf-f9dc-7660-3488-6a0d3420f386 5
10 UP osc ioio-OST0000-osc-ffff8102179aa000 b47f5fbf-f9dc-7660-3488-6a0d3420f386 5
11 UP osc ioio-OST0001-osc-ffff8102179aa000 b47f5fbf-f9dc-7660-3488-6a0d3420f386 5
12 UP osc ioio-OST0002-osc-ffff8102179aa000 b47f5fbf-f9dc-7660-3488-6a0d3420f386 5
[root at ten ~]# cat /etc/fstab
LABEL=/ / ext3 defaults 1 1
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
LABEL=SWAP-sda2 swap swap defaults 0 0
/dev/md0 /mnt/data/mdt lustre defaults,_netdev 0 0
#/dev/sdd1 /mnt/data/flash lustre defaults,_netdev 0 0
[root at ten ~]# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.4.0-1 (October 7, 2008)
Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer3+4 (1)
MII Status: up
MII Polling Interval (ms): 10
Up Delay (ms): 0
Down Delay (ms): 0
802.3ad info
LACP rate: slow
Active Aggregator Info:
Aggregator ID: 5
Number of ports: 6
Actor Key: 17
Partner Key: 37823
Partner Mac Address: 00:22:57:9a:ea:00
Slave Interface: eth0
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:22:15:06:3a:93
Aggregator ID: 5
Slave Interface: eth1
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:22:15:06:3a:94
Aggregator ID: 5
Slave Interface: eth2
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:22:15:06:3a:95
Aggregator ID: 5
Slave Interface: eth3
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:00:00:00:00:00
Aggregator ID: 5
Slave Interface: eth4
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:1b:21:28:77:db
Aggregator ID: 5
Slave Interface: eth5
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 1
Permanent HW addr: 00:1b:21:28:77:6c
Aggregator ID: 5
[root at ten ~]# uptime
12:36:03 up 7 days, 16:59, 1 user, load average: 0.00, 0.01, 0.00
[root at ten ~]# lfs df -i
UUID Inodes IUsed IFree IUse% Mounted on
ioio-MDT0000_UUID 215600857 2046153 213554704 1% /mnt/ioio[MDT:0]
ioio-OST0000_UUID 183148544 656907 182491637 0% /mnt/ioio[OST:0]
ioio-OST0001_UUID 183148544 657442 182491102 0% /mnt/ioio[OST:1]
ioio-OST0002_UUID 183148544 668586 182479958 0% /mnt/ioio[OST:2]
filesystem summary: 215600857 2046153 213554704 1% /mnt/ioio
** MGS/MDT Machine. Redundant Lustre Client. Redundant LAMP Server.
Last login: Mon Jan 21 21:37:31 2013
[root at three ~]# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdb[1] sdd[2] sdc[0] sde[4]
5860538880 blocks super 1.1 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
bitmap: 0/15 pages [0KB], 65536KB chunk
unused devices: <none>
[root at three ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 459G 18G 418G 5% /
tmpfs 1004M 37M 968M 4% /dev/shm
/dev/md0 5.4T 3.2T 2.0T 62% /mnt/rv
192.168.0.10 at tcp0:/ioio
8.1T 4.8T 3.0T 62% /mnt/ioio
[root at three ~]# lctl dl
0 UP mgc MGC192.168.0.10 at tcp 4aee8c8e-ab6f-d288-9cff-7953dff76905 5
1 UP lov ioio-clilov-ffff880075023000 06e9543a-5540-afe2-8e6a-5f21d8085076 4
2 UP mdc ioio-MDT0000-mdc-ffff880075023000 06e9543a-5540-afe2-8e6a-5f21d8085076 5
3 UP osc ioio-OST0000-osc-ffff880075023000 06e9543a-5540-afe2-8e6a-5f21d8085076 5
4 UP osc ioio-OST0001-osc-ffff880075023000 06e9543a-5540-afe2-8e6a-5f21d8085076 5
5 UP osc ioio-OST0002-osc-ffff880075023000 06e9543a-5540-afe2-8e6a-5f21d8085076 5
[root at three ~]# cat /etc/fstab
#
# /etc/fstab
# Created by anaconda on Mon Sep 3 07:44:32 2012
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=06e1bb0d-d7c8-42a4-bad1-6d12f931b77a / ext4 defaults 1 1
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
/dev/md0 /mnt/rv ext4 defaults 0 0
[root at three ~]# ifconfig
bond0 Link encap:Ethernet HWaddr 00:22:15:06:3B:73
inet addr:192.168.0.3 Bcast:192.168.0.255 Mask:255.255.255.0
inet6 addr: fe80::222:15ff:fe06:3b73/64 Scope:Link
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:2084539 errors:0 dropped:0 overruns:0 frame:0
TX packets:1840604 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1353887185 (1.2 GiB) TX bytes:990151888 (944.2 MiB)
eth0 Link encap:Ethernet HWaddr 00:22:15:06:3B:73
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:186644 errors:0 dropped:0 overruns:0 frame:0
TX packets:1611744 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:165778866 (158.0 MiB) TX bytes:950467922 (906.4 MiB)
Interrupt:18
eth1 Link encap:Ethernet HWaddr 00:22:15:06:3B:73
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:1896003 errors:0 dropped:0 overruns:0 frame:0
TX packets:34762 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1187880303 (1.1 GiB) TX bytes:14394977 (13.7 MiB)
Interrupt:19
eth2 Link encap:Ethernet HWaddr 00:22:15:06:3B:73
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:1892 errors:0 dropped:0 overruns:0 frame:0
TX packets:194098 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:228016 (222.6 KiB) TX bytes:25288989 (24.1 MiB)
Interrupt:16
eth3 Link encap:Ethernet HWaddr 00:22:15:06:3B:73
UP BROADCAST SLAVE MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
Interrupt:17
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:6544 errors:0 dropped:0 overruns:0 frame:0
TX packets:6544 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1409100 (1.3 MiB) TX bytes:1409100 (1.3 MiB)
[root at three ~]# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.5.0 (November 4, 2008)
Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer3+4 (1)
MII Status: up
MII Polling Interval (ms): 10
Up Delay (ms): 0
Down Delay (ms): 0
802.3ad info
LACP rate: slow
Aggregator selection policy (ad_select): stable
Active Aggregator Info:
Aggregator ID: 1
Number of ports: 3
Actor Key: 17
Partner Key: 37823
Partner Mac Address: 00:22:57:9a:ea:00
Slave Interface: eth0
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:22:15:06:3b:73
Aggregator ID: 1
Slave Interface: eth1
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:22:15:06:3b:74
Aggregator ID: 1
Slave Interface: eth2
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:22:15:06:3b:75
Aggregator ID: 1
Slave Interface: eth3
MII Status: down
Link Failure Count: 0
Permanent HW addr: 00:22:15:06:3b:76
Aggregator ID: 4
[root at three ~]# ifconfig
bond0 Link encap:Ethernet HWaddr 00:22:15:06:3B:73
inet addr:192.168.0.3 Bcast:192.168.0.255 Mask:255.255.255.0
inet6 addr: fe80::222:15ff:fe06:3b73/64 Scope:Link
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:2085357 errors:0 dropped:0 overruns:0 frame:0
TX packets:1841322 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1354333514 (1.2 GiB) TX bytes:990593774 (944.7 MiB)
eth0 Link encap:Ethernet HWaddr 00:22:15:06:3B:73
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:186719 errors:0 dropped:0 overruns:0 frame:0
TX packets:1612447 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:165785794 (158.1 MiB) TX bytes:950893930 (906.8 MiB)
Interrupt:18
eth1 Link encap:Ethernet HWaddr 00:22:15:06:3B:73
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:1896745 errors:0 dropped:0 overruns:0 frame:0
TX packets:34765 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1188319580 (1.1 GiB) TX bytes:14395233 (13.7 MiB)
Interrupt:19
eth2 Link encap:Ethernet HWaddr 00:22:15:06:3B:73
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:1893 errors:0 dropped:0 overruns:0 frame:0
TX packets:194110 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:228140 (222.7 KiB) TX bytes:25304611 (24.1 MiB)
Interrupt:16
eth3 Link encap:Ethernet HWaddr 00:22:15:06:3B:73
UP BROADCAST SLAVE MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
Interrupt:17
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:6544 errors:0 dropped:0 overruns:0 frame:0
TX packets:6544 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1409100 (1.3 MiB) TX bytes:1409100 (1.3 MiB)
[root at three ~]# uptime
12:37:32 up 15:06, 7 users, load average: 0.00, 0.02, 0.00
** Mobile Mythtv backend. Raid Storage/Lustre/Mythtv Backup. Zoneminder /dev/video0-3 /mnt/ioio/html/zm Lustre Mounted LAMP webroot=/mnt/ioio/html Lustre mounted.
[mythtv at twentythree bin]$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 35G 9.2G 24G 28% /
tmpfs 1004M 12K 1004M 1% /dev/shm
192.168.0.10 at tcp0:/ioio
8.1T 4.8T 3.0T 62% /mnt/ioio
[mythtv at twentythree bin]$ cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.5.0 (November 4, 2008)
Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer3+4 (1)
MII Status: up
MII Polling Interval (ms): 10
Up Delay (ms): 0
Down Delay (ms): 0
802.3ad info
LACP rate: slow
Aggregator selection policy (ad_select): stable
Active Aggregator Info:
Aggregator ID: 1
Number of ports: 2
Actor Key: 17
Partner Key: 37823
Partner Mac Address: 00:22:57:9a:ea:00
Slave Interface: eth0
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:22:15:95:2d:23
Aggregator ID: 1
Slave Interface: eth1
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:04:e2:d7:33:c3
Aggregator ID: 1
[mythtv at twentythree bin]$ ifconfig
bond0 Link encap:Ethernet HWaddr 00:22:15:95:2D:23
inet addr:192.168.0.23 Bcast:192.168.0.255 Mask:255.255.255.0
inet6 addr: fe80::222:15ff:fe95:2d23/64 Scope:Link
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:591449 errors:0 dropped:0 overruns:0 frame:0
TX packets:238413 errors:0 dropped:0 overruns:0 carrier:2
collisions:0 txqueuelen:0
RX bytes:324872352 (309.8 MiB) TX bytes:33161781 (31.6 MiB)
eth0 Link encap:Ethernet HWaddr 00:22:15:95:2D:23
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:135740 errors:0 dropped:0 overruns:0 frame:0
TX packets:130875 errors:0 dropped:0 overruns:0 carrier:2
collisions:0 txqueuelen:1000
RX bytes:8407741 (8.0 MiB) TX bytes:13738521 (13.1 MiB)
Interrupt:28
eth1 Link encap:Ethernet HWaddr 00:22:15:95:2D:23
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:455709 errors:0 dropped:0 overruns:0 frame:0
TX packets:107538 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:316464611 (301.8 MiB) TX bytes:19423260 (18.5 MiB)
Interrupt:17
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:262 errors:0 dropped:0 overruns:0 frame:0
TX packets:262 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:15508 (15.1 KiB) TX bytes:15508 (15.1 KiB)
[mythtv at twentythree bin]$ uptime
12:47:20 up 13:43, 14 users, load average: 1.12, 1.14, 1.11
[mythtv at twentythree bin]$ lfs df -h
UUID bytes Used Available Use% Mounted on
ioio-MDT0000_UUID 815.0G 374.0M 768.1G 0% /mnt/ioio[MDT:0]
ioio-OST0000_UUID 2.7T 1.7T 915.7G 65% /mnt/ioio[OST:0]
ioio-OST0001_UUID 2.7T 1.8T 796.3G 70% /mnt/ioio[OST:1]
ioio-OST0002_UUID 2.7T 1.3T 1.3T 50% /mnt/ioio[OST:2]
filesystem summary: 8.1T 4.7T 2.9T 62% /mnt/ioio
** Lustre Client. Mythtv Frontend. BzFlag 2.4.2 Enabled. Can be e17 driven.
[root at sixteen ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 143G 9.0G 127G 7% /
tmpfs 1.3G 88K 1.3G 1% /dev/shm
192.168.0.10 at tcp0:/ioio
8.1T 4.8T 3.0T 62% /mnt/ioio
[root at sixteen ~]# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.5.0 (November 4, 2008)
Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer3+4 (1)
MII Status: up
MII Polling Interval (ms): 10
Up Delay (ms): 0
Down Delay (ms): 0
802.3ad info
LACP rate: slow
Aggregator selection policy (ad_select): stable
Active Aggregator Info:
Aggregator ID: 1
Number of ports: 1
Actor Key: 17
Partner Key: 1
Partner Mac Address: 00:00:00:00:00:00
Slave Interface: eth0
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:14:85:18:83:7a
Aggregator ID: 1
Slave Interface: eth1
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:0e:0c:dd:3a:c7
Aggregator ID: 2
[root at sixteen ~]# ifconfig
bond0 Link encap:Ethernet HWaddr 00:14:85:18:83:7A
inet addr:192.168.0.16 Bcast:192.168.0.255 Mask:255.255.255.0
inet6 addr: fe80::214:85ff:fe18:837a/64 Scope:Link
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:531666451 errors:42 dropped:0 overruns:0 frame:42
TX packets:155532069 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:561475010002 (522.9 GiB) TX bytes:650261730391 (605.6 GiB)
eth0 Link encap:Ethernet HWaddr 00:14:85:18:83:7A
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:525163826 errors:42 dropped:0 overruns:0 frame:42
TX packets:155449633 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:561052441951 (522.5 GiB) TX bytes:650251508327 (605.5 GiB)
Interrupt:22
eth1 Link encap:Ethernet HWaddr 00:14:85:18:83:7A
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:6502625 errors:0 dropped:0 overruns:0 frame:0
TX packets:82436 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:422568051 (402.9 MiB) TX bytes:10222064 (9.7 MiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:205748 errors:0 dropped:0 overruns:0 frame:0
TX packets:205748 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:56913227 (54.2 MiB) TX bytes:56913227 (54.2 MiB)
[root at sixteen ~]# uptime
12:49:51 up 28 days, 14:59, 1 user, load average: 0.08, 0.05, 0.01
[root at sixteen ~]# lfs df -h
UUID bytes Used Available Use% Mounted on
ioio-MDT0000_UUID 815.0G 374.0M 768.1G 0% /mnt/ioio[MDT:0]
ioio-OST0000_UUID 2.7T 1.7T 915.7G 65% /mnt/ioio[OST:0]
ioio-OST0001_UUID 2.7T 1.8T 796.3G 70% /mnt/ioio[OST:1]
ioio-OST0002_UUID 2.7T 1.3T 1.3T 50% /mnt/ioio[OST:2]
filesystem summary: 8.1T 4.7T 2.9T 62% /mnt/ioio
** Lustre Client. Mythtv Backend.
So there are reasons for using raid10 on specific parts of your Lustre deployment. I had a few bad drives/sata errors on one box and raid10 saved everything. I was able to watch the drive fail after time and finally when it died I was able to replace while still having Lustre interoperablility. Still observing for that same box with another bad drive. Procurring 1TB 32mb sata II drives market driven.
Rebuild time was tolerable but from recollection less than a couple of hours. Just one of those tasks every raided machine does once a week anyhow via the cron resync.
I haven't lost data with this layout for years and have been able to adapt, maintain and implement different technologies and updates along the way.
https://groups.google.com/forum/?fromgroups#!msg/lustre-discuss-list/NfNilGsVYxA/KVCR8nixXbMJ
Would smaller disks be better? Have you ever tried to make a large raid 5 array out of 2+TB Sata III drives? There are some definate changes regarding the partitioning structure. I eventually had to just let the installer do all the work for me when I made a 6TB raid 5 Lustre Backup array. Smaller disks are easier from this perspective.
From this base I attached all HDMI cables. My LM7600 deploys 3d stereoscopic images and I've even take to researching the production and conversion of that medium. Beamsplitter on the RV, lol. The HDMI is upscaled to 4K by the Pioneer 1527K and it's all fed into a dozen KEF 101's
with 3 subs for 9.1 Stereo Sound utilizing neoX soundtracks from the newest blu rays. Waiting on 4K OLED sets to become available.
I routinely get 90+ days of uptime on my Lustre machines and find I'm more like the Maytag guy when it comes to maintaining the network. It is just there and for the most part enjoyable to use. Not saying that there haven't been some challenges when everything hangs in the balance of a few keystrokes but for the most part it works period.
I would like to expand it all again with Sata III 4TB storage disks and 120 MB SSD boot drives. Grab some 10GigE interconnect cards and plum some Bell Axia into the house but time will tell. Having a purpose for the storage is the best question to ask. 4x8=32TB per OST x 3 OST = 96TB linear raid. That would be personal use future proofed with room for offering storage services.
Perhaps the smallest clients of all are rooted androids and tablets accessing the network via apps to the SMB/NMB shared Lustre Mount. This ties the entire configuration together even driving down the road rebooting servers or restarting apache. Some apps like mythfrontend and mythmote are far superior to anything produced by commercial vendors for products. Most commercial apps are chalk full of thoughtless design much like smart TV's and for the most part commercialy available PVRs.
>________________________________
> From: Jon Yeargers <yeargers at ohsu.edu>
>To: "lustre-discuss at lists.lustre.org" <lustre-discuss at lists.lustre.org>
>Sent: Friday, December 28, 2012 11:37:06 AM
>Subject: [Lustre-discuss] RAID usage?
>
>
>Im curious to hear what RAID configurations people are using in existing Lustre installations.
>
>I’ve been collecting bids to build an array and assuming RAID6/1Tb but the more I read dire warnings about losing data and ‘days to rebuild an array’ the more I’m questioning my approach.
>
>Would RAID 10 be better? Smaller disks?
>_______________________________________________
>Lustre-discuss mailing list
>Lustre-discuss at lists.lustre.org
>http://lists.lustre.org/mailman/listinfo/lustre-discuss
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20130122/66945790/attachment.htm>
More information about the lustre-discuss
mailing list