[Lustre-discuss] disk fragmented I/Os
Lu Wang
wanglu at ihep.ac.cn
Wed Mar 31 04:12:48 PDT 2010
Dear list,
We have got a brw_stat result for one OST since it was up. According to this statistic, 50% percent disk I/Os are fragmented. I find a earlier discussion in this list referred to this qestion:
http://lists.lustre.org/pipermail/lustre-discuss/2009-August/011433.html
It seems it is ideal to have 100% disk I/Os with fragment "1" or "0". I don't know why the I/Os are fragmented, since I found the max_sectors_kb is big enough(16MB?)for biggest disk I/O size( according to brw_stat, it is 1MB)
# cat /sys/block/sda/queue/max_sectors_kb
32767
# cat /sys/block/sda/queue/max_hw_sectors_kb
32767
read | write
pages per bulk r/w rpcs % cum % | rpcs % cum %
1: 1624391 6 6 | 133752 3 3
2: 87238 0 6 | 13637 0 3
4: 175534 0 7 | 9266 0 3
8: 357057 1 9 | 15245 0 4
16: 638252 2 11 | 26643 0 4
32: 1249181 5 16 | 49749 1 6
64: 2266806 9 25 | 98010 2 8
128: 4976083 20 45 | 198864 4 13
256: 13457144 54 100 | 3522333 86 100
read | write
discontiguous pages rpcs % cum % | rpcs % cum %
0: 24628983 99 99 | 3921862 96 96
1: 197041 0 99 | 143302 3 99
2: 4679 0 99 | 1592 0 99
3: 692 0 99 | 523 0 99
4: 133 0 99 | 100 0 99
5: 38 0 99 | 120 0 100
6: 26 0 99 | 0 0 100
7: 17 0 99 | 0 0 100
8: 14 0 99 | 0 0 100
9: 5 0 99 | 0 0 100
10: 7 0 99 | 0 0 100
11: 8 0 99 | 0 0 100
12: 2 0 99 | 0 0 100
13: 6 0 99 | 0 0 100
14: 6 0 99 | 0 0 100
15: 4 0 99 | 0 0 100
16: 4 0 99 | 0 0 100
17: 4 0 99 | 0 0 100
18: 2 0 99 | 0 0 100
19: 2 0 99 | 0 0 100
20: 3 0 99 | 0 0 100
21: 1 0 99 | 0 0 100
22: 1 0 99 | 0 0 100
23: 3 0 99 | 0 0 100
24: 1 0 99 | 0 0 100
25: 0 0 99 | 0 0 100
26: 0 0 99 | 0 0 100
27: 1 0 99 | 0 0 100
28: 0 0 99 | 0 0 100
29: 0 0 99 | 0 0 100
30: 0 0 99 | 0 0 100
31: 3 0 100 | 0 0 100
read | write
discontiguous blocks rpcs % cum % | rpcs % cum %
0: 24616522 99 99 | 3908288 96 96
1: 205785 0 99 | 156444 3 99
2: 5200 0 99 | 1805 0 99
3: 797 0 99 | 733 0 99
4: 233 0 99 | 109 0 99
5: 331 0 99 | 120 0 100
6: 381 0 99 | 0 0 100
7: 346 0 99 | 0 0 100
8: 253 0 99 | 0 0 100
9: 369 0 99 | 0 0 100
10: 124 0 99 | 0 0 100
11: 172 0 99 | 0 0 100
12: 53 0 99 | 0 0 100
13: 113 0 99 | 0 0 100
14: 226 0 99 | 0 0 100
15: 69 0 99 | 0 0 100
16: 63 0 99 | 0 0 100
17: 195 0 99 | 0 0 100
18: 29 0 99 | 0 0 100
19: 152 0 99 | 0 0 100
20: 148 0 99 | 0 0 100
21: 7 0 99 | 0 0 100
22: 7 0 99 | 0 0 100
23: 3 0 99 | 0 0 100
24: 10 0 99 | 0 0 100
25: 2 0 99 | 0 0 100
26: 8 0 99 | 0 0 100
27: 11 0 99 | 0 0 100
28: 4 0 99 | 0 0 100
29: 2 0 99 | 0 0 100
30: 2 0 99 | 0 0 100
31: 69 0 100 | 0 0 100
read | write
disk fragmented I/Os ios % cum % | ios % cum %
0: 9821 0 0 | 0 0 0
1: 11933478 48 48 | 630964 15 15
2: 12726392 51 99 | 3350479 82 97
3: 155476 0 99 | 84465 2 99
4: 2962 0 99 | 1103 0 99
5: 393 0 99 | 364 0 99
6: 341 0 99 | 123 0 99
7: 384 0 99 | 1 0 100
8: 345 0 99 | 0 0 100
9: 254 0 99 | 0 0 100
10: 369 0 99 | 0 0 100
11: 124 0 99 | 0 0 100
12: 172 0 99 | 0 0 100
13: 53 0 99 | 0 0 100
14: 113 0 99 | 0 0 100
15: 226 0 99 | 0 0 100
16: 69 0 99 | 0 0 100
17: 63 0 99 | 0 0 100
18: 195 0 99 | 0 0 100
19: 29 0 99 | 0 0 100
20: 152 0 99 | 0 0 100
21: 148 0 99 | 0 0 100
22: 7 0 99 | 0 0 100
23: 7 0 99 | 0 0 100
24: 3 0 99 | 0 0 100
25: 10 0 99 | 0 0 100
26: 2 0 99 | 0 0 100
27: 8 0 99 | 0 0 100
28: 11 0 99 | 0 0 100
29: 4 0 99 | 0 0 100
30: 2 0 99 | 0 0 100
31: 71 0 100 | 0 0 100
read | write
disk I/Os in flight ios % cum % | ios % cum %
1: 10954265 28 28 | 3781021 49 49
2: 9217023 24 53 | 3329128 43 93
3: 6063548 15 69 | 272981 3 97
4: 4147809 10 80 | 121808 1 98
5: 2974531 7 87 | 30924 0 99
6: 1985323 5 93 | 19276 0 99
7: 866099 2 95 | 10509 0 99
8: 568797 1 97 | 7813 0 99
9: 373967 0 98 | 5110 0 99
10: 225063 0 98 | 3844 0 99
11: 158800 0 99 | 2594 0 99
12: 112305 0 99 | 1997 0 99
13: 71542 0 99 | 1351 0 99
14: 55104 0 99 | 1055 0 99
15: 44688 0 99 | 721 0 99
16: 29819 0 99 | 527 0 99
17: 14854 0 99 | 369 0 99
18: 10595 0 99 | 286 0 99
19: 7301 0 99 | 203 0 99
20: 5284 0 99 | 154 0 99
21: 3859 0 99 | 113 0 99
22: 2766 0 99 | 76 0 99
23: 1887 0 99 | 56 0 99
24: 1386 0 99 | 44 0 99
25: 1086 0 99 | 39 0 99
26: 877 0 99 | 24 0 99
27: 708 0 99 | 18 0 99
28: 559 0 99 | 17 0 99
29: 455 0 99 | 15 0 99
30: 390 0 99 | 13 0 99
31: 7047 0 100 | 208 0 100
read | write
I/O time (1/1000s) ios % cum % | ios % cum %
1: 139469 0 0 | 279599 6 6
2: 241078 0 1 | 280009 6 13
4: 2422008 9 11 | 2710995 66 80
8: 4039296 16 27 | 601942 14 95
16: 4139106 16 44 | 146443 3 98
32: 11264485 45 89 | 39174 0 99
64: 2161652 8 98 | 7816 0 99
128: 369479 1 99 | 1294 0 99
256: 47904 0 99 | 187 0 99
512: 6368 0 99 | 29 0 99
1K: 408 0 99 | 4 0 99
2K: 316 0 99 | 6 0 99
4K: 71 0 99 | 1 0 100
8K: 32 0 99 | 0 0 100
16K: 12 0 100 | 0 0 100
read | write
disk I/O size ios % cum % | ios % cum %
4K: 1737019 4 4 | 180540 2 2
8K: 192978 0 5 | 34454 0 2
16K: 339763 0 5 | 37064 0 3
32K: 674382 1 7 | 47011 0 3
64K: 1252812 3 11 | 90929 1 5
128K: 2449225 6 17 | 168972 2 7
256K: 4464264 11 29 | 288737 3 11
512K: 24846133 65 94 | 5997373 78 90
1M: 1951161 5 100 | 747214 9 100
I have 2 questions:
1. Could any one explain what dose these parameters exactly mean?
/sys/block/sda/queue/max_sectors_kb /sys/block/sda/queue/max_hw_sectors_kb ,disk fragmented I/Os, disk I/O size of brw_stats
2. In which case, the disk I/O will be fragemented?
Thanks a lot in advance!
Best Regards
Lu Wang
More information about the lustre-discuss
mailing list