From Vitaly.Fertman at Sun.COM Mon Mar 2 17:04:54 2009 From: Vitaly.Fertman at Sun.COM (Vitaly Fertman) Date: Mon, 02 Mar 2009 20:04:54 +0300 Subject: [Lustre-devel] LustreFS performance In-Reply-To: <02FEAA2B-8D98-4C2D-9CE8-FF6E1EB135A2@sun.com> References: <3376C558-E29A-4BB5-8C4C-3E8F4537A195@sun.com> <02FEAA2B-8D98-4C2D-9CE8-FF6E1EB135A2@sun.com> Message-ID: <8AD540D2-0B50-4630-B794-E65443352696@Sun.COM> **************************************************** LustreFS benchmarking methodology. **************************************************** The document aims to describe the benchmarking methodology which helps to understand the LustreFS performance and reveal LustreFS bottlenecks in different configurations on different hardware, to ensure the next LustreFS release does not downgrade comparing with a previous one. In other words: Goal1. Understand the HEAD performance. Goal2. Compare HEAD and b1_6 (b1_8) performance. To achieve the Goal1, the methodology suggests to test different layers of software in the bottom-top direction, i.e. the underlying back-end, the target server sitting on this back-end, the network connected to this target and how the target performs through this network, etc up to the whole cluster. Each next step has only 1 change over the previous one, it is either a new layer added or 1 parameter in the configuration is changed (probably another network type or another back-end). Comparing the results of each test with the previous test, we get the overhead of the added layer or the performance impact of changing this parameter. To achieve the Goal2, the methodology suggests to go in the reverse top-bottom direction, i.e. to test some large sub-systems first and, if a downgrade vs. a previous LustreFS version is detected, to perform more detailed tests. (This is considered as the primary goal of the 2.0 Performance Team). The document does not cover the way of fixing revealed problems, probably some special purpose test needs to be run or oprofile needs to be compiled in -- it is our of scope of the document. Obviously, it is not possible to perform all the thousands of tests in all the configurations, running all the special purpose tests, etc, the document tries to prepare: 1) all the essential and sufficient tests to see how the system performs in general; 2) some minimal amount of essential tests to see how the system scales in different conditions. Therefore, the plan does not guarantee we will not miss a bottleneck or a bug, it just tries to cover maximum possible scenarios in most interesting conditions/environment states. The amount of tests described below is already about 2K, and there will be definitely more, and it will take a lot of time to perform all of them and to analyze the results. So one of the major concerns here is how to minimize the amount of test so that we would not miss some interesting case and would be able to get all the results within a reasonable amount of time. Please keep it in mind while looking at the tests below. **** Hardware Requirements. **** The test plan implies that we change only 1 parameter (cpu or disk or network) on each step. Thus, the HW requirements are: -- at least 1 node with: CPU:32; RAM: enough to have a tmpfs for MDS; DISK: raid, regular. NET: both GiGe and IB installed. -- besides that: 8 clients, 4 other servers. -- the other servers include: DISK: raid, regular. NET: both GiGe and IB installed. -- client includes: NET: both GiGe and IB installed. **** Software requirements **** 1. Short term. 1.1 mdsrate to be completed to test all the operations listed in MDST3 (see below). 1.2 mdsrate-**.sh to be fixed/written to run mdsrate properly and test all the operations listed in MDST3 (see below). 1.3. fake disk implement FAIL flag and report 'done' without doing anything in obdfilter to get a low-latency disk. 1.4. MT. add more tests here and implement them. 2. Long term. 2.1. mdtstack-survey - an echo client-server is to be written for mds similar to ost. - a test script similar to obdfilter-survey.sh is to be written. **** Different configurations **** Configuration of Node: RAM. Amount of RAM on nodes (?) CPU. Count of CPUs on nodes (1..32) DISK. Disk type (regular, raid, tmpfs, fake) JOUR. Journal type (internal, external, ram) Q: which raid? A: raid5, as it seems to be the most popular. fake: to get a low-latency disk, it is preferable to report 'done' without doing anything in obdfilter once some FAIL flag is set. It is useful for OST testing, because first of all, it does not have a CPU overhead of memcpy of using tmpfs and it lets to test large amount of data in contrast to tmpfs. As a drawback, it skips the localfs code paths. Configuration of Cluster: CL. Amount of clients (1,2,4,8) OSS. Amount of OSS nodes (1,2,4) NET. Network type (GiGe, IB) OSTN. Amount of OST per nodes (1,2,4) Configuration of test. TH. Amount of threads per client (1,2,4,8) VER. Lustre version (b1_6, HEAD. later b1_8). FEAT. Lustre features to turn off (COS, SA, RA, debug messages) TEST. Specific test parameters. **** Testing **** Low Layers Testing (LLT) LLT1. Raw disk (lustre-iokit:sgpdd-survey) LLT2. Local filesystem (lustre-iokit: ior-survey, is fs mounted synchronously?) Network Testing (NETT). NETT1. lnet: lnetself test. NETT2. OBD: lustre-iokit: (obdfilter-survey, echo_client-osc-..- net-..-ost-echo_server) NETT3. MD: (not ready) OST Testing (OSTT). OSTT1. Isolated OST (lustre-iokit: obdfilter-survey, echo_client- obdfilter-..-disk) OSTT2. Remote OST (lustre-iokit: obdfilter-survey, echo_client-osc-..- ost-obdfilter-..-disk) OSTT3. Client-OST IO (lustre-iokit: ost-survey, client-ost-disk). MDS Testing (MDST). MDST1. Isolated MDS test (not ready) MDST2. Remote MDS test (not ready) MDST3. Simple Client-MDS operation test Mixed testing (MT) (not ready) **** Statistics **** During all the tests the following is supposed to be running on all the servers: 1) vmstat 2) iostat, if there is some disk activity. smth else? *** Goal1. Understand the HEAD performance. *** The Goal1 describes the testing methodology in the bottom-top direction, from the lower layers (disk) to the complete Lustre cluster. LLT1. Raw disk (lustre-iokit:sgpdd-survey) RAM: fixed CPU: 1 DISK: regular,raid,tmpfs (default=raid) JOUR:- CL: 1 OSS:1 NET: - OSTN:- TH: 1,2,4,8 (default=1) F: debug TEST: *)bulk size is specified as rszlo/rszhi=[1,4,64,1024K] *)TH is specified as thrlo/thrhi=[1,2,4,8] *)the amount of objects to work on in parallel: crglo=crghi=[1;TH] i.e. test only cases when all the threads work on the same file and when all of them work on a separate file. [bulk;separate or commin dir]=8 tests; Test matrix(TESTxTHxDISK): Run TESTs with different amount of threads for each DISK. TESTxTHxDISK=(8x4 - 1)x3=93 tests. "-1" because TH=1 is already covered. Total:93 tests. *** NETT1. lnetself test.*** RAM: fixed CPU: 1,2,8,32 (default=1) DISK: - JOUR:- CL: 1,2,4,8 (default=1) OSS:1 NET: GiGe, IB (default=IB) OSTN:1 TH: 1,2,4,8 (default=1) F: debug TEST: *) test type: PING,READ,WRITE tests *) bulk size for READ/WRITE: 1k,4k,64k,1M [1 ping + 4 reads + 4 writes] = 9 tests Test matrix (TESTxCLxTHxNETxCPU): 1. Multi-thread test. Run TESTs on CL=1 with different amount of threads. TESTxTH=[1+4+4]x4=36 tests. 2. Multi-client test 2.1. Let's check how clients scale vs. threads per client (TH=1). 2.2. Let's check how the system scale with many clients and threads (TH=8). Note: to be more demonstrative, the maximum amount of threads could be taken <8, if TH=8 reaches the maximum network throughput with small amount of clients. [CL>1;TH=1,8]. TESTxCLxTH=9x3x2=54 tests. 3. Network test As the nature of IB is different from GiGe, we need to repeat all the tests from (1,2) here. 36+54=90 tests. 4. CPU test Note: lnet fixes from Liang to be applied here. Run TESTs on different amount of CPU. It is mostly interesting to look at large amount of threads, as we are going to benefit from handling them in parallel. At the same time, if some HW (network) limit is reached, the result will not be very demonstrative, so test with 1 small & 1 large bulk size only:[1k; 1024K]: [CL=8,TH=1,2,4,8]. TESTxCLxTHxCPU=5x1x4x(4-1)=60. Total: 240 tests. *** NETT2. OBD performance *** lustre-iokit: obdfilter-survey, case=network. The results of this tests are to be compared with lnet results to get the osc+ost+ptlrpc overhead. RAM: fixed CPU: 1 DISK: - JOUR:- CL: 1,2,4,8 (default=1) OSS:1 NET: IB OSTN:1 TH: 1,2,4,8 (default=1) F: debug TEST: *) bulk size: rszlo=rszhi=N (1,4,64,1024) *) TH is specified through: thrlo=1, thrhi=8 (thread count, 1,2,4,8) *) the amount of objects is: nobjlo=nobjhi=[1;TH] i.e. test only cases when all the threads work on the same file and when all of them work on a separate file. [4 bulks; common or separate dir]=8 tests Test matrix(TESTxTHxCLxNET): 1. Multi-thread test. Run TESTs on CL=1 with different amount of threads. TESTxTH=8x4=32 tests 2. Multi-client test 2.1. Let's check how clients scale vs. threads per client (TH=1). 2.2. Let's check how the system scale with many clients and threads (TH=8). Note: to be more demonstrative, the maximum amount of threads should be taken <8, if TH=8 reaches the maximum network throughput with small amount of clients. [CL>1;TH=1,8]. TESTxCLxTH=8x3x2=48 tests. 3. Network test. Having IB results in hand after (1,2) and these results from NETT1, we already see how osc+ost+ptlrpc changes the behavior. There is no reason to repeat them for GiGe, it seems. 4.CPU test Note: lnet fixes from Liang to be applied here. Run TESTs on different amount of CPU. It is mostly interesting to look at large amount of threads, as we are going to benefit from handling them in parallel. At the same time, if some HW (network) limit is reached, the result will not be very demonstrative, so test with 1 small & 1 large bulk size only:[1k; 1024K]: [CL=8,TH=1,2,4,8]. TESTxCLxTHxCPU=4x1x4x(4-1)=48. Total: 128 tests. *** OSTT1. Isolated OST *** lustre-iokit: obdfilter-survey, case=disk The results of this tests are to be compared with LLT results to get the OST stack overhead. RAM: fixed CPU: 1,2,8,32 (default=1) DISK: regular, raid, fake (default=fake) JOUR: int, ext, ram, (default=int) CL: 1 OSS:1 NET: - OSTN:1,2,4 (default=1) TH: 1,2,4,8 (default=1) F: debug TEST: *) bulk size: rszlo=rszhi=N (1,4,64,1024K) *) TH is specified through: thrlo=1, thrhi=8 (1,2,4,8) *) each OST is supposed to be configured on a separate disk. *) the amount of objects is: nobjlo=nobjhi=[1;TH] i.e. test only cases when all the threads work on the same file and when all of them work on a separate file. [4 bulks; common of separate dir]=8 tests Test matrix(TESTxTHxOSTNxDISKxCPU): 1. Multi-thread test. Run TESTs on OSTN=1 with different amount of threads. TESTxTH=8x4=32 tests 2. Multi-OST test 2.1. Let's check how OSTs vs. threads per OST scale (TH=OSTN). 2.2. Let's check how the system scale with many OSTs and threads (TH=8*OSTN). [OSTN>1;TH=OSTN,8*OSTN]. TESTxOSTNxTH=8x2x2=32 tests. 3. DISK test As other disks are completely different, so lets repeat most of the (1,2) for 2 others: [TH=OSTN;8*OSTN]: TESTxOSTNxTHxDISK=8x3x2x2=96 4. JOURNAL test. Limit the tests with only raid-disk. Limit the test with only 1 large and 1 small bulk:[1,1024K]. TESTxOSTNxTHxJOUR: 4x3x2x2=48 5. CPU test Note: lnet fixes from Liang to be applied here. Run TESTs on different amount of CPU. It is better to perform it on a fast backend (DISK=fake) to see how CPU really matters. It is mostly interesting to look at large amount of threads, as we are going to benefit from handling them in parallel. Also, run with a small & a large bulk only:[1,1024K] [OSTN=4,TH=1,2,4,8]: TESTxOSTNxTHxCPU=4x1x4x3=48 Total: 256 tests. *** OSTT2. Real OST test *** lustre-iokit: obdfilter-survey, case=netdisk This test is a composition of OBD performance and Isolated OST tests, so its results are to be compared with NETT2 & OSTT1 results. RAM: fixed CPU: 1,2,8,32 (default=1) DISK: fake JOUR:int CL: 1,2,4,8 (default=1) OSS:1,2,4 (default=1) NET: IB OSTN:1,2,4 (default=1) TH: 1,2,4,8 (default=1) F: debug TEST: *) bulk size: rszlo=rszhi=N (1,4,64,1024) *) TH is specified through: thrlo=1, thrhi=8 (thread count, 1,2,4,8) *) each OST is supposed to be configured on a separate disk. *) the amount of objects is: nobjlo=nobjhi=[1;TH] i.e. test only cases when all the threads work on the same file and when all of them work on a separate file. [4 bulks; common of separate dir]=8 tests Test matrix(TESTxTHxCLxCPUxNETxOSSxOSTN): 1. Multi-thread test. Run TESTs on CL=1 with different amount of threads. TESTxTH=8x4=32 tests 2. Multi-client test 2.1. Let's check how clients scale vs. threads per client (TH=1). 2.2. Let's check how the system scale with many clients and threads (TH=8). Note: to be more demonstrative, the maximum amount of threads could be taken <8, if TH=8 reaches the maximum network throughput with small amount of clients. [CL>1;TH=1,8]. TESTxCLxTH=8x3x2=48 tests. 3. Network test Having IB results in hand after (1,2) and these results from NETT2, we already see how osc+ost+ptlrpc+obdfilter changes the behavior. Thus, there is no reason to repeat them for GiGe, it seems. 4.CPU test Note: lnet fixes from Liang to be applied here. Run TESTs on different amount of CPU. It is mostly interesting to look at large amount of threads, as we are going to benefit from handling them in parallel. At the same time, if some HW (network) limit is reached, the result will not be very demonstrative, so test with 1 small & 1 large bulk size only:[1k; 1024K]: [CL=8,TH=1,2,4,8]. TESTxCLxTHxCPU=4x1x4x(4-1)=48. 5. OSTN test. The same OSC, network, CPU, disk, just check how OST stack (see 1,2 tests) is scalable. 5.1. Let's check how N threads per 1 OST vs. 1 thread per N OST scales (CL=OSTN). 5.2. Let's check how the system scale with many clients and threads (CL=8) Note: to be more demonstrative, the maximum amount of threads could be taken <8, if TH=8 reaches the maximum network throughput with small amount of clients. It seems enough to look at 1 small & 1 large bulk only: [1,1024K] [CL=OSTN,8;TH=1,8]. TESTxCLxTHxOSTN=4x2x2x2=32 tests 6. OSS test. 6.1. Let's check how 1 thread per N OST vs. 1 thread per N OSS scales (CL=OSS). 6.2. Let's check how the system scale with many clients and threads (CL=8) Note: to be more demonstrative, the maximum amount of threads could be taken <8, if TH=8 reaches the maximum network throughput with small amount of clients. It seems enough to look at 1 small & 1 large bulk only: [1,1024K] [CL=OSS,8;TH=1,8]. TESTxCLxTHxOSTN=4x2x2x2=32 tests Total:192 tests *** OSTT3. Client-OST test *** lustre-iokit: ior-survey. The test results are to be compared with OSTT2 results to get the overhead for Lustre Client: client stack, distributed locking, etc. RAM: fixed CPU: 1,2,8,32 (default=1) DISK: fake JOUR:int CL: 1,2,4,8 (default=1) OSS:1,2,4 (default=1) NET: IB OSTN:1,2,4 (default=1) TH: 1,2,4,8 (default=1) F: debug TEST: *) CL is specified through $clients_hi *) TH is specified through $tasks_per_client_hi *) bulk is specified through rsize_lo/hi (1,4,64,1028K) *) file_per_task=[0;1] i.e. test only cases when all the threads work on the same file and when all of them work on a separate file. [4 bulks; common of separate dir]=8 tests Test matrix(TESTxTHxCLxCPU): absolutely the same as for OSTT2. NETT3. MD: (not ready) MDST1. Isolated MDS (not ready) MDST2. Remote MDS (not ready) This set of tests need to be implemented in a utility similar to obdfilter-survey but for MDS testing. MDST3. Simple Client-MDS operation tests 1. create,mknod,mkdir (symlink, link??) RAM: fixed CPU: 1,2,8,32 (default=1) DISK(MDS): tmpfs, raid, regular (default=tmpfs) DISK(OST): tmpfs JOUR: int,ext,ram (default=int) CL: 1,2,4,8 (default=1) OSS:1,2,4 (default=1) NET: IB OSTN:1 TH: 1,2,4,8 (default=1) F: debug TEST: it will be probably mdsrate/mdsrate-create-small.sh, but it needs to be fixed to support all of these operations, not only create. If so: *) TH could be specified through THREADS_PER_CLIENT=[1,2,4,8] *) CL is specified through CLIENTS or NODES_TO_USE. *) NOSINGLE should be provided *) add --dirnum option to COMMAND *) DIRNUM=[1,TH*CL], so we test a case when all the threads work in the same dir and when each works in a separate one. *) nfiles is files-per-dir * DIRNUM [common or separate dir]=2tests; Note: we should probably limit the amount of files in 1 directory with 2M, otherwise the performance will definitely downgrade. Test matrix(TESTxTHxCLxCPUxNETxOSS): 1. Multi-thread test. Run TESTs on CL=1 with different amount of threads. TESTxTH=2x4-1=7 tests (not 8 as if TH=1, DIRNUM=1, and this is already covered). 2. Multi-client test 2.1. Let's check how clients scale vs. threads per client (TH=1). 2.2. Let's check how the system scale with many clients and threads (TH=8). Note: to be more demonstrative, the maximum amount of threads could be taken <8, if TH=8 reaches the maximum network throughput with small amount of clients. [CL>1;TH=1,8]. TESTxCLxTH=2x3x2=12 tests. 3. Striped test. 2.1. Let's check how multi-client system scales (TH=1). 2.2. Let's check how large load system scales (TH=8) Test (only) create with different stripeness. TESTxCLxTHxOSS=[2x4x2-1]x2=30 4. Network test Having IB results in hand after (1,2,3) and these results from NETT1, we already see how mdc+mdt-stack+ptlrpc changes the behavior. There is no reason to repeat them for GiGe, it seems. 5. DISK test. Unlink the OST testing, we do not have echo-md client (MDTT1), thus we have not checked how different disks impact the performance, so we need to check it here. Limit this test with only couple of operations: create, mknod. As different disks are of completely different nature we need to repeat most of (1,2) here [TH=1,8]: TESTxCLxTHxDISK=(2x4x2-1)x2=30 6. JOURNAL test. Repeat (5) for different journals, but limit the test with raid-disk only. TESTxCLxTHxDISKxJOUR=(2x4x2-1)x1x2=30 7.CPU test Note: lnet fixes from Liang to be applied here. Run TESTs on different amount of CPU. Limit this test with only couple of operations: create, mknod. It is mostly interesting to look at large amount of threads, as we are going to benefit from handling them in parallel, so run it for CL=max only: [CL=8,TH=1,2,4,8]. TESTxCLxTHxCPU=2x1x4x(4-1)=24 Total: 19 tests for mkdir, 103 for mknod, 133 for create. 2. lookup (mdsrate-lookup-1dir.sh => mdsrate-lookup.sh) RAM: fixed CPU: 1,2,8,32 (default=1) DISK: tmpfs JOUR: int CL: 1,2,4,8 (default=1) OSS:1 NET: GiGe,IB (default=IB) OSTN:1 TH: 1,2,4,8 (default=1) F: debug TEST: it will be probably better to work out mdsrate-lookup-1dir.sh, which could work in several directories in parallel. *) TH could be specified through THREADS_PER_CLIENT=[1,2,4,8] (to be added into the script) *) CL is an amount of nodes specified in CLIENTS or NODES_TO_USE. *) NOSINGLE should be provided *) add --dirnum option to COMMAND *) DIRNUM=[1,TH*CL], so we test a case when all the threads work in the same dir and when each works in a separate one. *) nfiles is files-per-dir * DIRNUM *) add READDIR_ORDER to test both random and readdir order lookups. [common or separate di; readdir,random order]=4 tests. Q: it seems this test does md_getattr_name(), instead of lookup, thus no lock enqueue is involved. A: what about to replace it with access(2)?? Test matrix(TESTxTHxCLxCPUxNET): the same as for (1, mknod), but 4 tests instead of 2: 19x2=38 tests. 3. stat RAM: fixed CPU: 1,2,8,32 (default=1) DISK(MDS): tmpfs, raid, regular (default=tmpfs) DISK(OST): tmpfs JOUR: int,ext,ram (default=int) CL: 1,2,4 (default=1) OSS:1,2,4 (default=1) NET: GiGe,IB (default=IB) OSTN:1 TH: 1,2,4,8 (default=1) F: debug TEST: mdsrate/mdsrate-stat-small.sh *) add THREADS_PER_CLIENT to the script to specify TH *) CL is specified through CLIENTS or NODES_TO_USE. *) NOSINGLE should be provided *) add --dirnum option to COMMAND *) DIRNUM=[1,TH*CL], so we test a case when all the threads work in the same dir and when each works in a separate one. *) nfiles is files-per-dir * DIRNUM *) add READDIR_ORDER to test both random and readdir order lookups. [common or separate dir; readdir,random order]=4 tests. Q: do we want to test stat(2) with other then tmpfs disk on OST? what journal should it have if so? Test matrix(TESTxTHxCLxCPUxNETxDISKxJOUR): the same as for (1, create), but 4 tests instead of 2: 133x2=266 tests. 4. unlink (mdsrate-create-small.sh, run twice??) it should be run (and it is run in mdsrate-create-small.sh) for all the operations in (1), i.e. create, mkdir, mknod. The test matrix is the same and the total: 19 tests for mkdir, 103 for mknod, 133 for create. 5. chmod (mdsrate-chmod.sh, new one, fix mdsrate) The same as (1, mkdir) and the total: 19 tests. 6. utime (mdsrate-utime.sh, new one, fix mdsrate) The same as (1, mkdir) and the total: 19 tests. 7. chown (mdsrate-chown.sh, new one, fix mdsrate) The same as (1, create), but skip different DISKs&JOURNALs: 19 + 30 + 24=73 tests. 8. rename (mdsrate-rename.sh, new one, fix mdsrate) The same as (1, mkdir) and the total: 19 tests. 9. find Q: despite the fact we currently have a large downgrade with "find -f type", do we want to have this test in the general test set? **** MT. Mixed testing. **** MT1. Create-write test. RAM: fixed CPU: 32 DISK(MDS): tmpfs, raid (default=tmpfs) DISK(OST): raid JOUR: int CL: 1,2,4,8 (default=1) OSS:1 (default=1) NET: IB OSTN:1 TH: 1,2,4,8 (default=1) F: debug TEST: must be a new one. Each thread creates files in a loop, writes 1 bulk to each and closes it. *) it is enough to test with a small bulk only: [1k] *) [common or separate dir]=2tests; Test matrix(TESTxTHxCLxCPUxNETxOSS): 1. Multi-thread test. Run TESTs on CL=1 with different amount of threads. TESTxTH=2x4-1=7 tests (not 8 as if TH=1, it is always in 1 dir, and this is already covered). 2. Multi-client test 2.1. Let's check how clients scale vs. threads per client (TH=1). 2.2. Let's check how the system scale with many clients and threads (TH=8). Note: to be more demonstrative, the maximum amount of threads could be taken <8, if TH=8 reaches the maximum network throughput with small amount of clients. [CL>1;TH=1,8]. TESTxCLxTH=2x3x2=12 tests. 3. DISK test. Check how different disks impact on the performance. As different disks are of completely different nature we need to repeat most of (1,2) here [TH=1,8]: TESTxCLxTHxDISK=(2x4x2-1)x1=15 Total: 34 tests. MT2. Create-Readdir test. RAM: fixed CPU: 32 DISK(MDS): tmpfs, raid (default=tmpfs) DISK(OST): raid JOUR: int CL: 1,2,4,8 (default=1) (1 extra client does "ls -U") OSS:1 (default=1) NET: IB OSTN:1 TH: 1,2,4,8 (default=1) F: debug TEST: must be a new one. Each thread creates files in a loop and immediately closes them. 1 thread on another client does "ls -U". It is done in 1 directory. The test matrix is exactly the same as for MT1. Total: 34 tests. MT3. ??? Some more tests ???? **** Goal2. Compare HEAD and b1_6 (b1_8) performance. **** This paragraph describes the testing methodology in the reverse order of testing, i.e. in the top-bottom direction, making sure new LustreFS (HEAD) version does not downgrade comparing with the previous ones (b1_6/b1_8). Therefore, the first testing cycle includes: MT, MDST3, OSTT3, NETT1. from the above tests. In the case a downgrade is detected, lower layer tests are to be run until the downgrade disappear. -- Vitaly From magawake at gmail.com Sun Mar 1 00:20:20 2009 From: magawake at gmail.com (Mag Gam) Date: Sat, 28 Feb 2009 19:20:20 -0500 Subject: [Lustre-devel] OST remount problem. Message-ID: <1cbd6f830902281620h2c87055aqdc58e80e114336af@mail.gmail.com> I have posted a very similar question in lustre-discuss but the lack of responses made me move to the developers. I created an OST (while clients were still active). I then mounted the OST, and it states the OST's services are already running but nothing is showing up on /etc/mtab. I suspect this is a bug. Therefore, I submitted a bug report. https://bugzilla.lustre.org/show_bug.cgi?id=18651 But, meanwhile, is there a way to override this by changing a flag in ost's configuration (writeconf) to say, its not mouted up? This will hopefully enable me to mount up partition as intended. TIA From adilger at sun.com Mon Mar 2 20:45:01 2009 From: adilger at sun.com (Andreas Dilger) Date: Mon, 02 Mar 2009 13:45:01 -0700 Subject: [Lustre-devel] LustreFS performance In-Reply-To: <8AD540D2-0B50-4630-B794-E65443352696@Sun.COM> References: <3376C558-E29A-4BB5-8C4C-3E8F4537A195@sun.com> <02FEAA2B-8D98-4C2D-9CE8-FF6E1EB135A2@sun.com> <8AD540D2-0B50-4630-B794-E65443352696@Sun.COM> Message-ID: <20090302204501.GQ3199@webber.adilger.int> On Mar 02, 2009 20:04 +0300, Vitaly Fertman wrote: > RAM: enough to have a tmpfs for MDS; Note that strictly speaking we need to use ldiskfs on a ramdisk, not tmpfs, because we don't have an fsfilt_tmpfs. > Q: which raid? > A: raid5, as it seems to be the most popular. I would propose: - for MDT it needs to be RAID-1+0, because of small, random IO sizes - for OST it needs to be RAID-6, because of double-failure risk (see Lustre Manual "RAID" section for discussion) > **** Statistics **** > > During all the tests the following is supposed to be running on all > the servers: > 1) vmstat > 2) iostat, if there is some disk activity. > smth else? I would propose either LLNL's LMT or HP's collectl, which both also collect Lustre stats. Those both provide more information than the above, and having the IO/CPU load correlated to Lustre RPC counts is very useful. > MDST3 > > Q: do we want to test stat(2) with other then tmpfs disk on OST? > what journal should it have if so? I would be quite interested in the performance numbers from just the ramdisk MDT+OST, to see what the upper limit of the protocol and network are. > 9. find > Q: despite the fact we currently have a large downgrade with > "find -f type", do we want to have this test in the general test set? Some of that performance loss should have been fixed recently. We should continue to test it. Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc. From Oleg.Drokin at Sun.COM Wed Mar 4 17:14:17 2009 From: Oleg.Drokin at Sun.COM (Oleg Drokin) Date: Wed, 04 Mar 2009 12:14:17 -0500 Subject: [Lustre-devel] Fast checkpoints in Lustre today, at essentially zero cost. Message-ID: <7636AEA6-3D56-491E-A085-C85CA4D96020@sun.com> Hello! I was having a discussion with Mike Booth just now about how application programmers are willing to sacrifice robustness of their checkpoint files and go the the previous version if the last one did not made it to the disk if only the checkpointing itself could be done quick (even if not persistent). I also remembered how we discussed using of ramdisks on the compute nodes and then have some background agent to actually copy the data to Lustre later on when the application is happily computing still. It now suddenly appeared to me (duh!) that we can get that all for free with buffer cache right now. We are artificially limiting our dirty memory to 32M/osc on every node (which is probably way to low today too), but if we lift the limit significantly (or remove it altogether) such checkpointing applications (and I verified that many of them use 10-20% of RAM for checkpointing) would benefit tremendously (as long as they do not do fsync at the end of checkpoint, of course). Currently there is a way to achieve this same goal, but to do this we essentially need to select a very suboptimal stripe pattern (essentially drag your file to be striped across as many OSTs as possible with small stripe size to maximize allowed dirty memory cache in use at the expense of a lot of seeking at the OSTs since that would essentially mean every client would be writing to every OST). The old justification for the small dirty memory limit was to do with lock timeouts and stuff, but now that we have lock prolonging and indefinite waiting for locks on clients on the other, there is no reason to limit ourselves anymore, I think. I plan to speak with ORNL later today to conduct an experiment (if they would agree) and to lift their dirty memory limit on some of the scratch filesystems often used for scratch files to see what the effect would be. I expect it to be very positive myself. At the same time Mike is doing a test right now with some real world applications + the above-mentioned suboptimal striping pattern to see the effects as well. I think that allows us to take advantage of huge amounts of wasted memory on compute nodes today for this caching and benefit many application with this checkpointing model (essentially superseding old "flash-cache" idea at a fraction of the cost and effort). Bye, Oleg From Oleg.Drokin at Sun.COM Wed Mar 4 17:19:53 2009 From: Oleg.Drokin at Sun.COM (Oleg Drokin) Date: Wed, 04 Mar 2009 12:19:53 -0500 Subject: [Lustre-devel] LustreFS performance In-Reply-To: <20090302204501.GQ3199@webber.adilger.int> References: <3376C558-E29A-4BB5-8C4C-3E8F4537A195@sun.com> <02FEAA2B-8D98-4C2D-9CE8-FF6E1EB135A2@sun.com> <8AD540D2-0B50-4630-B794-E65443352696@Sun.COM> <20090302204501.GQ3199@webber.adilger.int> Message-ID: <5BC7F78A-7FD4-4F54-A1C9-F7A632EA7200@sun.com> Hello! On Mar 2, 2009, at 3:45 PM, Andreas Dilger wrote: > On Mar 02, 2009 20:04 +0300, Vitaly Fertman wrote: >> RAM: enough to have a tmpfs for MDS; > Note that strictly speaking we need to use ldiskfs on a ramdisk, not > tmpfs, because we don't have an fsfilt_tmpfs. The idea was loop device on tmpfs, I think. Bye, Oleg From jeffd at sicortex.com Wed Mar 4 17:28:08 2009 From: jeffd at sicortex.com (Jeff Darcy) Date: Wed, 04 Mar 2009 12:28:08 -0500 Subject: [Lustre-devel] LustreFS performance In-Reply-To: <5BC7F78A-7FD4-4F54-A1C9-F7A632EA7200@sun.com> References: <3376C558-E29A-4BB5-8C4C-3E8F4537A195@sun.com> <02FEAA2B-8D98-4C2D-9CE8-FF6E1EB135A2@sun.com> <8AD540D2-0B50-4630-B794-E65443352696@Sun.COM> <20090302204501.GQ3199@webber.adilger.int> <5BC7F78A-7FD4-4F54-A1C9-F7A632EA7200@sun.com> Message-ID: <49AEBA28.5050700@sicortex.com> Oleg Drokin wrote: > On Mar 2, 2009, at 3:45 PM, Andreas Dilger wrote: > > >> On Mar 02, 2009 20:04 +0300, Vitaly Fertman wrote: >> >>> RAM: enough to have a tmpfs for MDS; >>> >> Note that strictly speaking we need to use ldiskfs on a ramdisk, not >> tmpfs, because we don't have an fsfilt_tmpfs. >> > > The idea was loop device on tmpfs, I think. > FYI, this is exactly what we do with our FabriCache feature - i.e. both MDT and OSTs are actually loopback files on tmpfs. Modulo a few issues with preallocated write space eating all storage leaving none for actual data, it works rather well producing high performance numbers and giving LNDs a good workout. BTW, the loopback driver does copies and is disturbingly single-threaded, which can create a bottleneck. This can be worked around with multiple instances per node, though. From Nathan.Rutman at Sun.COM Thu Mar 5 18:05:22 2009 From: Nathan.Rutman at Sun.COM (Nathaniel Rutman) Date: Thu, 05 Mar 2009 10:05:22 -0800 Subject: [Lustre-devel] Security configuration In-Reply-To: <014901c9913f$94f0b560$bed22020$@com> References: <014901c9913f$94f0b560$bed22020$@com> Message-ID: <49B01462.1020001@sun.com> Eric Barton wrote: > Nathan, > > We'd like to be able to describe a set of nodes and say that > as far as security is concerned, they are all equivalent - i.e. if > an MDT authorizes eeb at node1 to perform a certain action, then > eeb at nodex is implicitly authorized provided node1 and nodex are in > the same set. > > Leaving aside for now, the question of how the sets are described > (they could be whole LNETs or whole Kerberos realms, or NID lists), > is the MGS the right place to stash this config? > Yes, I think the MGS is the right place to stash any config. FWIW we're pretty seriously thinking about removing all the distributed configuration we can (mkfs/tunefs.lustre settings and module parameters) and concentrating it all on the MGS node in a text-based config file. Exceptions would have to be made for the network setup, so that everyone could talk to the MGS -- so lnet networks and MGS nids would still have to be stored locally. From adilger at sun.com Thu Mar 5 20:00:36 2009 From: adilger at sun.com (Andreas Dilger) Date: Thu, 05 Mar 2009 13:00:36 -0700 Subject: [Lustre-devel] Fast checkpoints in Lustre today, at essentially zero cost. In-Reply-To: <7636AEA6-3D56-491E-A085-C85CA4D96020@sun.com> References: <7636AEA6-3D56-491E-A085-C85CA4D96020@sun.com> Message-ID: <20090305200035.GF3199@webber.adilger.int> On Mar 04, 2009 12:14 -0500, Oleg Drokin wrote: > It now suddenly appeared to me (duh!) that we can get that all for > free with buffer cache right now. We are artificially limiting our > dirty memory to 32M/osc on every node (which is probably way to low > today too), but if we lift the limit significantly (or remove it > altogether) such checkpointing applications (and I verified that many > of them use 10-20% of RAM for checkpointing) would benefit tremendously > (as long as they do not do fsync at the end of checkpoint, of course). > Currently there is a way to achieve this same goal, but to do this > we essentially need to select a very suboptimal stripe pattern > (essentially drag your file to be striped across as many OSTs as > possible with small stripe size to maximize allowed dirty memory cache > in use at the expense of a lot of seeking at the OSTs since that would > essentially mean every client would be writing to every OST). We don't need to go to the sub-optimal striping to get this result, as that causes not only lots of seeking on the OSTs, but also requires the clients to get locks on every OST. Instead it is possible today to just increase this limit to be much larger via /proc tunings on the client for testing (assume 1/2 of RAM is large enough): client# lctl set_param osc.*.max_dirty_mb=${ramsize/2} or the cache limit can be increased permanently on all of the clients via conf_param (sorry, syntax may not be 100% correct): mgs# lctl --device ${mgsdevno} conf_param fsname.osc.max_dirty_mb=${ramsize/2} > The old justification for the small dirty memory limit was to do > with lock timeouts and stuff, but now that we have lock prolonging and > indefinite waiting for locks on clients on the other, there is no reason > to limit ourselves anymore, I think. You are probably correct. At times we have discussed using the client grant to manage the amount of dirty data that clients can have, so that we don't get 5TB of dirty data on the clients for a single 100MB/s OST before trying to flush the data. You may be right that with lock extension and glimpse ASTs we may just be better off to allow the clients to fill the cache as needed. One possible downfall is that when multiple clients are writing to the same file, if the first client to get the lock (full [0-EOF] lock) can dump a huge amount of dirty data under the lock, all of the other clients will not even be able to get a lock and start writing until the first client is finished. I think this shows up in SSF IOR testing today when the write chunk size is 4MB being slower than when it is 1MB, because the clients need to flush 4MB of data before their lock can be revoked and split, instead of just 1MB. Having lock conversion allow the client to shrink or split the lock would avoid this contention. > At the same time Mike is doing a test right now with some real > world applications + the above-mentioned suboptimal striping pattern > to see the effects as well. > > I think that allows us to take advantage of huge amounts of wasted > memory on compute nodes today for this caching and benefit many > application with this checkpointing model (essentially superseding > old "flash-cache" idea at a fraction of the cost and effort). Sure, as long as apps are not impacted by the increased memory usage, since client nodes generally do not have swap, nor would we want to swap out the application to cache the checkpoint data. Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc. From Oleg.Drokin at Sun.COM Thu Mar 5 20:35:58 2009 From: Oleg.Drokin at Sun.COM (Oleg Drokin) Date: Thu, 05 Mar 2009 15:35:58 -0500 Subject: [Lustre-devel] Fast checkpoints in Lustre today, at essentially zero cost. In-Reply-To: <20090305200035.GF3199@webber.adilger.int> References: <7636AEA6-3D56-491E-A085-C85CA4D96020@sun.com> <20090305200035.GF3199@webber.adilger.int> Message-ID: <47835A78-9713-4C6B-B65C-DA243115C656@Sun.COM> Hello! On Mar 5, 2009, at 3:00 PM, Andreas Dilger wrote: > We don't need to go to the sub-optimal striping to get this result, > as that causes not only lots of seeking on the OSTs, but also requires > the clients to get locks on every OST. Instead it is possible today > to just increase this limit to be much larger via /proc tunings on > the client for testing (assume 1/2 of RAM is large enough): > client# lctl set_param osc.*.max_dirty_mb=${ramsize/2} Of course! But I am speaking of a situation like say ORNL, where users cannot control this setting directly. > One possible downfall is that when multiple clients are writing to the > same file, if the first client to get the lock (full [0-EOF] lock) can > dump a huge amount of dirty data under the lock, all of the other > clients > will not even be able to get a lock and start writing until the first > client is finished. This is unlikely case, since as soon as lock is cancelled, it cannot be rematched. > I think this shows up in SSF IOR testing today when the write chunk > size > is 4MB being slower than when it is 1MB, because the clients need to > flush > 4MB of data before their lock can be revoked and split, instead of > just 1MB. > Having lock conversion allow the client to shrink or split the lock > would > avoid this contention. I would think the reason is different since essentially 4M memory copy is very small and chances are client was ably to issue write syscall many times per lock acquisition. But this is pure speculation from my side. Bye, Oleg From adilger at sun.com Thu Mar 5 21:16:53 2009 From: adilger at sun.com (Andreas Dilger) Date: Thu, 05 Mar 2009 14:16:53 -0700 Subject: [Lustre-devel] Fast checkpoints in Lustre today, at essentially zero cost. In-Reply-To: <47835A78-9713-4C6B-B65C-DA243115C656@Sun.COM> References: <7636AEA6-3D56-491E-A085-C85CA4D96020@sun.com> <20090305200035.GF3199@webber.adilger.int> <47835A78-9713-4C6B-B65C-DA243115C656@Sun.COM> Message-ID: <20090305211653.GH3199@webber.adilger.int> On Mar 05, 2009 15:35 -0500, Oleg Drokin wrote: > On Mar 5, 2009, at 3:00 PM, Andreas Dilger wrote: >> We don't need to go to the sub-optimal striping to get this result, >> as that causes not only lots of seeking on the OSTs, but also requires >> the clients to get locks on every OST. Instead it is possible today >> to just increase this limit to be much larger via /proc tunings on >> the client for testing (assume 1/2 of RAM is large enough): >> client# lctl set_param osc.*.max_dirty_mb=${ramsize/2} > > Of course! But I am speaking of a situation like say ORNL, where > users cannot control this setting directly. But if this is just testing, then we may as well avoid the extra overhead and apples-to-oranges comparison of single-stripe vs. many-stripe files and allow single-stripe files to cache a lot of data. Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc. From adilger at sun.com Thu Mar 5 21:27:00 2009 From: adilger at sun.com (Andreas Dilger) Date: Thu, 05 Mar 2009 14:27:00 -0700 Subject: [Lustre-devel] LustreFS performance In-Reply-To: <49AEBA28.5050700@sicortex.com> References: <3376C558-E29A-4BB5-8C4C-3E8F4537A195@sun.com> <02FEAA2B-8D98-4C2D-9CE8-FF6E1EB135A2@sun.com> <8AD540D2-0B50-4630-B794-E65443352696@Sun.COM> <20090302204501.GQ3199@webber.adilger.int> <5BC7F78A-7FD4-4F54-A1C9-F7A632EA7200@sun.com> <49AEBA28.5050700@sicortex.com> Message-ID: <20090305212700.GI3199@webber.adilger.int> On Mar 04, 2009 12:28 -0500, Jeff Darcy wrote: > Oleg Drokin wrote: >> On Mar 2, 2009, at 3:45 PM, Andreas Dilger wrote: >>> Note that strictly speaking we need to use ldiskfs on a ramdisk, not >>> tmpfs, because we don't have an fsfilt_tmpfs. >> >> The idea was loop device on tmpfs, I think. > > FYI, this is exactly what we do with our FabriCache feature - i.e. both > MDT and OSTs are actually loopback files on tmpfs. The problem with using a loop device instead of a ramdisk is that you now have 2 layers of indirection - MDS->ldiskfs->loop->tmpfs->RAM instead of MDS->ldiskfs->RAM. The drawback (or possibly benefit) is that ramdisks consume a fixed amount of RAM and are not "sparse" (AFAIK, that may have changed since I last looked into this). That said, once a block is written to by mke2fs or by ldiskfs in the loop->tmpfs case it will also never be freed again, so you only get some marginal benefit. > Modulo a few issues > with preallocated write space eating all storage leaving none for actual > data, it works rather well producing high performance numbers and giving > LNDs a good workout. BTW, the loopback driver does copies and is > disturbingly single-threaded, which can create a bottleneck. This can > be worked around with multiple instances per node, though. Even better, if you have some development skills, would be to implement (or possibly resurrect) an fsfilt-tmpfs layer. Since tmpfs isn't going to be recoverable anyways (I assume you just reformat from scratch when there is a crash), then you can make all of the transaction handling as no-ops, and just implement the minimal interfaces needed to work. That would allow unlinked files to release space from tmpfs, and also avoid the fixed allocation overhead and journaling of ldiskfs, probably saving you 5% of RAM (more on the MDS) and a LOT of memcpy() overhead. Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc. From eeb at sun.com Fri Mar 6 13:25:49 2009 From: eeb at sun.com (Eric Barton) Date: Fri, 06 Mar 2009 13:25:49 +0000 Subject: [Lustre-devel] create performance In-Reply-To: <20090305214620.GJ3199@webber.adilger.int> References: <49AEE953.2030401@sun.com> <49B00E4D.2080806@sun.com> <20090305214620.GJ3199@webber.adilger.int> Message-ID: <049301c99e5f$10ee3c60$32cab520$@com> Moving this discussion to lustre-devel... OST object placement is a hard problem with conflicting requirements including... 1. Even server space balance 2. Even server load balance 3. Minimal network congestion 4. Scalable ultra-wide file layout descriptor 5. Scalable placement algorithm Implementing a placement algorithm with a centralized server clearly isn't scalable and will have to be reworked for CMD. A starting point might be to explore how to ensure CROW goes some way to satisfy requirements 1-3 above. BTW, I've long believed that it's a mistake not to give Lustre any inkling that all the creates done by a FPP parallel application are somehow related - e.g. via a cluster-wide job identifier. Surely file-per-process placement is very close to shared file placement (minus extent locking conflicts :)? I recognize that fixing this still leaves the problem of how to get best F/S utilization when different applications share a cluster - but I don't think they are necessarily the same problem and trying to address them both with the same solution seems wrong. Cheers, Eric > -----Original Message----- > From: Andreas.Dilger at Sun.COM [mailto:Andreas.Dilger at Sun.COM] On Behalf Of Andreas Dilger > Sent: 05 March 2009 9:46 PM > To: Nathaniel Rutman > Cc: lustre-tech-leads at sun.com > Subject: Re: create performance > > On Mar 05, 2009 09:39 -0800, Nathaniel Rutman wrote: > > Alex Zhuravlav wrote: > >>>>>>> Nathaniel Rutman (NR) writes: > >> NR> What about preallocating objects per client, on the clients? > >> NR> Client still needs to get a namespace entry from MDT, but could then > >> NR> hold a write layout lock > >> NR> and do it's own round-robin allocation. For clients with subtree > >> NR> locks this could avoid any need to talk to the MDT and wouldn't need > >> NR> the writeback cache. > >> > >> I thought "avoid any need to talk to MDT" implies "writeback cache" > > > > Hmm, well, maybe you consider this a limited version of writeback cache? > > It would be kind of a notification of "here is the layout/objects of my > > new file, with my new fid." Fid ranges and object numbers would be > > granted to clients for their own use, and the MDT would only have to do > > the namespace entry, asynchronously. I suppose there's recovery issues > > we have to worry about then. > > > > What I was really trying to get at was to avoid the two step process of > > client -> MDT -> OST stripe allocation, which includes an extra network > > hop in some precreation starvation cases, and always includes some (a > > little?) cpu on the MDT: > > 1. clients get object grants for every OST. > > 2. clients assign objects to new files and send in reqs to MDT, which > > just records the objects in the LOV EA > > 3. MDT batches up the assigned objects and sends to OSTs for orphan > > cleanup llog. > > The main problem with having many clients do precreation themselves is > that this will invariably cause load imbalance on the OSTs, which will > cause long-term file IO performance problems (much in excess of the > performance problems hit during precreate). > > Cray recently filed a bug on the read performance of files being noticably > hurt by QOS object allocation due to space imbalance, even thoguh the MDS > is trying to balance across OSTs locally, but is using random numbers to > do this and is not selecting OSTs evenly. > > In a file-per-process checkpoint (say 100 processes/files on 100 OSTs) > the MDS round-robin will allocate 1 object per OST evenly across all > OSTs (excluding the case where an OSC is out of preallocated objects). > If clients are doing the allocation (or in the past when the MDS did > "random" OST selection) then the chance of all 100 clients allocating > on 100 OSTs is vanishingly small. Instead it is likely that some OSTs > will have no objects used, and some will have 2 or 3 or 4, and the > aggregate write performance will FOREVER be 50% or 33% or 25% of the > MDS-round-robin allocated objects for that set of files. That is far > worse than waiting 1s for the MDS to allocate the objects. > > IMHO, if we are doing WBC on the client, then there is no _requirement_ > that the client has to allocate objects for the files at all, and any > write data could just be in the client page cache. Until the new file > is visible on the MDS to another client nobody can even try to access > the data. Once the WBC cache is flushed to the MDS then objects can > be allocated by the MDS evenly (granting an exclusive layout lock to the > client in the process) until the cached client data is either flushed to > disk or at least protected by extent locks and can be partially flushed > as needed. > > Note that I don't totally object to WBC clients doing object allocations > if they are creating a large number of files, in essence becoming an > MDS that is tracking the load on the OSTs and balancing object creation > appropriately. What I object to is the more common case where each > client is creating a single file for a large FPP checkpoint, and the > clients all selecting the OSTs separately. > > Cheers, Andreas > -- > Andreas Dilger > Sr. Staff Engineer, Lustre Group > Sun Microsystems of Canada, Inc. From adilger at sun.com Fri Mar 6 19:16:20 2009 From: adilger at sun.com (Andreas Dilger) Date: Fri, 06 Mar 2009 12:16:20 -0700 Subject: [Lustre-devel] layout lock / extent lock interaction In-Reply-To: <49B16509.9000409@sun.com> References: <20090130233253.GK3652@webber.adilger.int> <63CF2DAF-59F7-4124-8AE9-188927051AAB@sun.com> <5882C1BC-E01E-42FF-A728-08885E997075@sun.com> <20090221002109.GA3863@webber.adilger.int> <49B16509.9000409@sun.com> Message-ID: <20090306191620.GB3199@webber.adilger.int> On Mar 06, 2009 10:01 -0800, Nathaniel Rutman wrote: > I think we need to explicitly list the extent / layout lock interactions > so we don't miss anything in the implementation: > 1. Create > > * MDT generates new layout lock at open > * client gets Common Reader layout lock > * client can get new extents read/write locks as long it holds CR > layout lock > > 2. Layout change > > * MDT takes PW layout lock, revoking all client CR locks > * in parallel, MDT takes PW lock on all extents on all OSTs for this > file > * Clients drop layout lock and requeue > * Clients flush cache and drop their extent locks > * MDT changes layout > * MDT releases layout lock and extents locks > * Clients get CR layout lock and can now requeue their extent locks > > 3. Client / MDT network partition > > * client can continue reading/writing to currently held extents > * when client determines it has been disconnected from MDT it drops > layout lock > * client can't get new extent locks, but can continue writing to > currently held extents > * if MDT changes layout, it first PW locks all extents, causing OSTs > to revoke client's extents locks > * Client must requeue layout lock before requeueing extents locks > > What if client hasn't noticed it's been disconnected from the MDT by > the time it tries to requeue extent locks? It doesn't know that the > layout lock its holding is invalid... That is a thorny problem. I'll go through several partial solutions and see why they do not work, then hopefully a safe solution at the end. One possibility is that the AST sent to the clients during the extent lock revocation would contain a flag that indicates "the layout is changing" (similar to the truncate/discard data flag), so the clients get notified even if disconnected from the MDS. It still isn't enough, however, as the clients will only get this AST if they currently have an extent lock, and it isn't always true. A second option is in case a client holding a layout lock is evicted AND the layout is being changed then the MDS can't release the extent locks until at least one ping interval (assuming any still-alive client would have detected this and try reconnecting). This is also not 100% safe because the client might have been evicted moments earlier due to some other lock and the "wait for one ping interval" heuristic would no longer apply. We cannot depend on the layout change to be drastic and the objects would no longer exist to be written to (CROW issues aside). If we are changing the layout to add a mirror that wouldn't help and we would now have inconsistent data on each half of the mirror. Another option is something like "imperative eviction" so that clients being evicted are actively told they are being evicted, but that has the issue of the "you are evicted" RPC will normally be sent to a node which is already dead and slow down the MDS and/or block all of its LNET credits so isn't really even a usable option. A safe option (AFAICS) is to have MDS eviction force OST eviction (via obd_set_info_async(EVICT_BY_NID). That would also resolve some other recovery problems, but might be overly drastic if e.g. the client is being evicted from the MDS due to router failure or simple network partition. Having a proper health network and also server-side RPC resending would help avoid such problems. This is one of the main reasons why having DLM servers on one node controlling resources on another node is a bad idea. We had similar issues in the past when we locked all objects via the OST only on stripe index 0, and we might have similar problems with subtree locks in the future with CMD or any SNS RAID that is only locking a subset of objects. Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc. From Nathan.Rutman at Sun.COM Fri Mar 6 22:59:58 2009 From: Nathan.Rutman at Sun.COM (Nathaniel Rutman) Date: Fri, 06 Mar 2009 14:59:58 -0800 Subject: [Lustre-devel] layout lock / extent lock interaction In-Reply-To: <20090306191620.GB3199@webber.adilger.int> References: <20090130233253.GK3652@webber.adilger.int> <63CF2DAF-59F7-4124-8AE9-188927051AAB@sun.com> <5882C1BC-E01E-42FF-A728-08885E997075@sun.com> <20090221002109.GA3863@webber.adilger.int> <49B16509.9000409@sun.com> <20090306191620.GB3199@webber.adilger.int> Message-ID: <49B1AAEE.2080104@sun.com> Andreas Dilger wrote: > On Mar 06, 2009 10:01 -0800, Nathaniel Rutman wrote: > >> I think we need to explicitly list the extent / layout lock interactions >> so we don't miss anything in the implementation: >> 1. Create >> >> * MDT generates new layout lock at open >> * client gets Common Reader layout lock >> * client can get new extents read/write locks as long it holds CR >> layout lock >> >> 2. Layout change >> >> * MDT takes PW layout lock, revoking all client CR locks >> * in parallel, MDT takes PW lock on all extents on all OSTs for this >> file >> * Clients drop layout lock and requeue >> * Clients flush cache and drop their extent locks >> * MDT changes layout >> * MDT releases layout lock and extents locks >> * Clients get CR layout lock and can now requeue their extent locks >> >> 3. Client / MDT network partition >> >> * client can continue reading/writing to currently held extents >> * when client determines it has been disconnected from MDT it drops >> layout lock >> * client can't get new extent locks, but can continue writing to >> currently held extents >> * if MDT changes layout, it first PW locks all extents, causing OSTs >> to revoke client's extents locks >> * Client must requeue layout lock before requeueing extents locks >> >> What if client hasn't noticed it's been disconnected from the MDT by >> the time it tries to requeue extent locks? It doesn't know that the >> layout lock its holding is invalid... >> > > That is a thorny problem. I'll go through several partial solutions > and see why they do not work, then hopefully a safe solution at the end. > > One possibility is that the AST sent to the clients during the extent lock > revocation would contain a flag that indicates "the layout is changing" > (similar to the truncate/discard data flag), so the clients get notified > even if disconnected from the MDS. It still isn't enough, however, > as the clients will only get this AST if they currently have an extent > lock, and it isn't always true. > How about if we introduce the concept of a layout generation? The generation is stored in the layout and also with each OST object. When the MDT takes the extent locks it sends the new generation to the OSTs. Clients send the layout generation along with any extent lock enqueue. The OSTs only grant extents to clients that match the current generation. Maybe "match or exceed" in case OST dies before new gen can be recorded. And OST increases gen to latest seen whenever any (MDT or client) extent lock is enqueued. > A second option is in case a client holding a layout lock is evicted AND > the layout is being changed then the MDS can't release the extent locks > until at least one ping interval (assuming any still-alive client would > have detected this and try reconnecting). This is also not 100% safe because > the client might have been evicted moments earlier due to some other lock > and the "wait for one ping interval" heuristic would no longer apply. > > We cannot depend on the layout change to be drastic and the objects would > no longer exist to be written to (CROW issues aside). If we are changing > the layout to add a mirror that wouldn't help and we would now have > inconsistent data on each half of the mirror. > > Another option is something like "imperative eviction" so that clients > being evicted are actively told they are being evicted, but that has > the issue of the "you are evicted" RPC will normally be sent to a node > which is already dead and slow down the MDS and/or block all of its > LNET credits so isn't really even a usable option. > > > A safe option (AFAICS) is to have MDS eviction force OST eviction (via > obd_set_info_async(EVICT_BY_NID). That would also resolve some other > recovery problems, but might be overly drastic if e.g. the client is > being evicted from the MDS due to router failure or simple network > partition. Having a proper health network and also server-side RPC > resending would help avoid such problems. > This is drastic, but on the other hand we only need to do this if the layout is being changed. Of course, since eviction would happen before layout change we would need to remember who was evicted and hasn't reconnected... > This is one of the main reasons why having DLM servers on one node > controlling resources on another node is a bad idea. We had similar > issues in the past when we locked all objects via the OST only on > stripe index 0, and we might have similar problems with subtree locks > in the future with CMD or any SNS RAID that is only locking a subset > of objects. > From adilger at sun.com Fri Mar 6 23:48:59 2009 From: adilger at sun.com (Andreas Dilger) Date: Fri, 06 Mar 2009 16:48:59 -0700 Subject: [Lustre-devel] layout lock / extent lock interaction In-Reply-To: <49B1AAEE.2080104@sun.com> References: <20090130233253.GK3652@webber.adilger.int> <63CF2DAF-59F7-4124-8AE9-188927051AAB@sun.com> <5882C1BC-E01E-42FF-A728-08885E997075@sun.com> <20090221002109.GA3863@webber.adilger.int> <49B16509.9000409@sun.com> <20090306191620.GB3199@webber.adilger.int> <49B1AAEE.2080104@sun.com> Message-ID: <20090306234859.GH3199@webber.adilger.int> On Mar 06, 2009 14:59 -0800, Nathaniel Rutman wrote: > How about if we introduce the concept of a layout generation? The > generation is stored in the layout and also with each OST object. When > the MDT takes the extent locks it sends the new generation to the OSTs. > Clients send the layout generation along with any extent lock enqueue. > The OSTs only grant extents to clients that match the current > generation. Maybe "match or exceed" in case OST dies before new gen can > be recorded. And OST increases gen to latest seen whenever any (MDT or > client) extent lock is enqueued. I like this idea. We would need some place to store this information in the LOV EA on the MDT and pass it to the client, and to/on the OST. We already have: - inode versions (VBR; change on each file modification) - IO epochs (SOM; change slowly as files are written, not persistent) - recovery epochs (CMD/WBC; change frequently as global epochs are committed) We could concievably use the space in "l_ost_gen" in the first stripe, as we have never implemented OST generations. Those were intended for OST replacement, and/or OST snapshots, but have never been implemented. It also has the drawback that it is per-stripe, and we would likely be wasting the additional l_ost_gen values in later stripes in addition to breaking their intended use. Maybe we just bite the bullet and add another LOV EA type? >> A safe option (AFAICS) is to have MDS eviction force OST eviction (via >> obd_set_info_async(EVICT_BY_NID). That would also resolve some other >> recovery problems, but might be overly drastic if e.g. the client is >> being evicted from the MDS due to router failure or simple network >> partition. Having a proper health network and also server-side RPC >> resending would help avoid such problems. > > This is drastic, but on the other hand we only need to do this if the > layout is being changed. Of course, since eviction would happen before > layout change we would need to remember who was evicted and hasn't > reconnected... No, I don't think we need to remember recently-evicted clients, since the MDS would also evict clients from all OSTs immediately. The goal to avoid this drastic action would be to avoid evicting the client from the MDS in the first place (e.g. by request resend, health net), which is a double win. >> This is one of the main reasons why having DLM servers on one node >> controlling resources on another node is a bad idea. We had similar >> issues in the past when we locked all objects via the OST only on >> stripe index 0, and we might have similar problems with subtree locks >> in the future with CMD or any SNS RAID that is only locking a subset >> of objects. >> Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc. From Oleg.Drokin at Sun.COM Mon Mar 9 02:50:10 2009 From: Oleg.Drokin at Sun.COM (Oleg Drokin) Date: Sun, 08 Mar 2009 22:50:10 -0400 Subject: [Lustre-devel] LustreFS performance In-Reply-To: <20090305212700.GI3199@webber.adilger.int> References: <3376C558-E29A-4BB5-8C4C-3E8F4537A195@sun.com> <02FEAA2B-8D98-4C2D-9CE8-FF6E1EB135A2@sun.com> <8AD540D2-0B50-4630-B794-E65443352696@Sun.COM> <20090302204501.GQ3199@webber.adilger.int> <5BC7F78A-7FD4-4F54-A1C9-F7A632EA7200@sun.com> <49AEBA28.5050700@sicortex.com> <20090305212700.GI3199@webber.adilger.int> Message-ID: <80354020-87CB-4704-8020-D775AEE09A6D@Sun.COM> Hello! On Mar 5, 2009, at 4:27 PM, Andreas Dilger wrote: > Even better, if you have some development skills, would be to > implement > (or possibly resurrect) an fsfilt-tmpfs layer. Since tmpfs isn't > going > to be recoverable anyways (I assume you just reformat from scratch > when > there is a crash), then you can make all of the transaction handling > as no-ops, and just implement the minimal interfaces needed to work. > That would allow unlinked files to release space from tmpfs, and also > avoid the fixed allocation overhead and journaling of ldiskfs, > probably > saving you 5% of RAM (more on the MDS) and a LOT of memcpy() overhead. This is exactly what I was trying to avoid. I tried to measure things as if I had an infinitely fast disk only, and I still needed all the journal/blockdevice and other such things to take the CPU they would normally take. After all we cannot expect people to actually run real MDSes on tmpfs unless they have some means to replicate that MDS somewhere else. Bye, Oleg From adilger at sun.com Mon Mar 9 06:05:34 2009 From: adilger at sun.com ('Andreas Dilger') Date: Mon, 09 Mar 2009 00:05:34 -0600 Subject: [Lustre-devel] create performance In-Reply-To: <049301c99e5f$10ee3c60$32cab520$@com> References: <49AEE953.2030401@sun.com> <49B00E4D.2080806@sun.com> <20090305214620.GJ3199@webber.adilger.int> <049301c99e5f$10ee3c60$32cab520$@com> Message-ID: <20090309060534.GK3199@webber.adilger.int> On Mar 06, 2009 13:25 +0000, Eric Barton wrote: > OST object placement is a hard problem with conflicting requirements > including... > > 1. Even server space balance > 2. Even server load balance > 3. Minimal network congestion > 4. Scalable ultra-wide file layout descriptor > 5. Scalable placement algorithm > > Implementing a placement algorithm with a centralized server clearly > isn't scalable and will have to be reworked for CMD. A starting > point might be to explore how to ensure CROW goes some way to > satisfy requirements 1-3 above. While CROW can help avoid latency for precreating objects (which can avoid some of the object allocation imbalances hit today when OSTs are slow precreating objects), it doesn't really fundamentally help to balance space and performance of the OSTs. With any filesystem with more than a handful of OSTs there shouldn't be any reason why the OSTs precreating can't keep up with the MDS create rate. Johann and I were discussing this problem and I suspect it is only a defect in the object precreation code and not a fundamental problem int the design. I definitely agree that for CMD we will have distributed object allocation, but so far it isn't clear whether having more than the MDSes and/or WBC clients doing the allocation will improve the situation or make it worse. > BTW, I've long believed that it's a mistake not to give Lustre any > inkling that all the creates done by a FPP parallel application are > somehow related - e.g. via a cluster-wide job identifier. Surely > file-per-process placement is very close to shared file placement > (minus extent locking conflicts :)? Yes, I agree. In theory it should be possible to extract this kind of information from the client processes themselves, either by examining the process environment (some MPI job launchers store the MPI rank there for pre-launch shell scripts) or by comparing the filenames being created by the clients. Any file-per-process job will invariably create filenames with the rank in the filename. > I recognize that fixing this > still leaves the problem of how to get best F/S utilization when > different applications share a cluster - but I don't think they are > necessarily the same problem and trying to address them both with the > same solution seems wrong. > > Cheers, > Eric > > > -----Original Message----- > > From: Andreas.Dilger at Sun.COM [mailto:Andreas.Dilger at Sun.COM] On Behalf Of Andreas Dilger > > Sent: 05 March 2009 9:46 PM > > To: Nathaniel Rutman > > Cc: lustre-tech-leads at sun.com > > Subject: Re: create performance > > > > On Mar 05, 2009 09:39 -0800, Nathaniel Rutman wrote: > > > Alex Zhuravlav wrote: > > >>>>>>> Nathaniel Rutman (NR) writes: > > >> NR> What about preallocating objects per client, on the clients? > > >> NR> Client still needs to get a namespace entry from MDT, but could then > > >> NR> hold a write layout lock > > >> NR> and do it's own round-robin allocation. For clients with subtree > > >> NR> locks this could avoid any need to talk to the MDT and wouldn't need > > >> NR> the writeback cache. > > >> > > >> I thought "avoid any need to talk to MDT" implies "writeback cache" > > > > > > Hmm, well, maybe you consider this a limited version of writeback cache? > > > It would be kind of a notification of "here is the layout/objects of my > > > new file, with my new fid." Fid ranges and object numbers would be > > > granted to clients for their own use, and the MDT would only have to do > > > the namespace entry, asynchronously. I suppose there's recovery issues > > > we have to worry about then. > > > > > > What I was really trying to get at was to avoid the two step process of > > > client -> MDT -> OST stripe allocation, which includes an extra network > > > hop in some precreation starvation cases, and always includes some (a > > > little?) cpu on the MDT: > > > 1. clients get object grants for every OST. > > > 2. clients assign objects to new files and send in reqs to MDT, which > > > just records the objects in the LOV EA > > > 3. MDT batches up the assigned objects and sends to OSTs for orphan > > > cleanup llog. > > > > The main problem with having many clients do precreation themselves is > > that this will invariably cause load imbalance on the OSTs, which will > > cause long-term file IO performance problems (much in excess of the > > performance problems hit during precreate). > > > > Cray recently filed a bug on the read performance of files being noticably > > hurt by QOS object allocation due to space imbalance, even thoguh the MDS > > is trying to balance across OSTs locally, but is using random numbers to > > do this and is not selecting OSTs evenly. > > > > In a file-per-process checkpoint (say 100 processes/files on 100 OSTs) > > the MDS round-robin will allocate 1 object per OST evenly across all > > OSTs (excluding the case where an OSC is out of preallocated objects). > > If clients are doing the allocation (or in the past when the MDS did > > "random" OST selection) then the chance of all 100 clients allocating > > on 100 OSTs is vanishingly small. Instead it is likely that some OSTs > > will have no objects used, and some will have 2 or 3 or 4, and the > > aggregate write performance will FOREVER be 50% or 33% or 25% of the > > MDS-round-robin allocated objects for that set of files. That is far > > worse than waiting 1s for the MDS to allocate the objects. > > > > IMHO, if we are doing WBC on the client, then there is no _requirement_ > > that the client has to allocate objects for the files at all, and any > > write data could just be in the client page cache. Until the new file > > is visible on the MDS to another client nobody can even try to access > > the data. Once the WBC cache is flushed to the MDS then objects can > > be allocated by the MDS evenly (granting an exclusive layout lock to the > > client in the process) until the cached client data is either flushed to > > disk or at least protected by extent locks and can be partially flushed > > as needed. > > > > Note that I don't totally object to WBC clients doing object allocations > > if they are creating a large number of files, in essence becoming an > > MDS that is tracking the load on the OSTs and balancing object creation > > appropriately. What I object to is the more common case where each > > client is creating a single file for a large FPP checkpoint, and the > > clients all selecting the OSTs separately. > > > > Cheers, Andreas > > -- > > Andreas Dilger > > Sr. Staff Engineer, Lustre Group > > Sun Microsystems of Canada, Inc. > Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc. From adilger at sun.com Mon Mar 9 08:29:58 2009 From: adilger at sun.com (Andreas Dilger) Date: Mon, 09 Mar 2009 02:29:58 -0600 Subject: [Lustre-devel] LustreFS performance In-Reply-To: <80354020-87CB-4704-8020-D775AEE09A6D@Sun.COM> References: <3376C558-E29A-4BB5-8C4C-3E8F4537A195@sun.com> <02FEAA2B-8D98-4C2D-9CE8-FF6E1EB135A2@sun.com> <8AD540D2-0B50-4630-B794-E65443352696@Sun.COM> <20090302204501.GQ3199@webber.adilger.int> <5BC7F78A-7FD4-4F54-A1C9-F7A632EA7200@sun.com> <49AEBA28.5050700@sicortex.com> <20090305212700.GI3199@webber.adilger.int> <80354020-87CB-4704-8020-D775AEE09A6D@Sun.COM> Message-ID: <20090309082826.GL3199@webber.adilger.int> On Mar 08, 2009 22:50 -0400, Oleg Drokin wrote: > On Mar 5, 2009, at 4:27 PM, Andreas Dilger wrote: >> Even better, if you have some development skills, would be to >> implement (or possibly resurrect) an fsfilt-tmpfs layer. Since >> tmpfs isn't going to be recoverable anyways (I assume you just >> reformat from scratch when there is a crash), then you can make >> all of the transaction handling as no-ops, and just implement >> the minimal interfaces needed to work. >> >> That would allow unlinked files to release space from tmpfs, and also >> avoid the fixed allocation overhead and journaling of ldiskfs, probably >> saving you 5% of RAM (more on the MDS) and a LOT of memcpy() overhead. > > This is exactly what I was trying to avoid. > I tried to measure things as if I had an infinitely fast disk only, and I > still needed all the journal/blockdevice and other such things to take > the CPU they would normally take. > After all we cannot expect people to actually run real MDSes on tmpfs > unless they have some means to replicate that MDS somewhere else. In fact, it wasn't my proposal for the metadata performance testing itself, but rather SiCortex are apparently running with RAM-backed filesystems for some kind of fast cache filesystem (e.g. distributed shared memory or similar). I was only proposing their implementation could be more efficient. I could imagine that for flash-cache type applications that storing checkpoints for a short time in a RAM-backed OST pool before migrating it to persistent storage. Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc. From eeb at sun.com Mon Mar 9 14:39:23 2009 From: eeb at sun.com (Eric Barton) Date: Mon, 09 Mar 2009 14:39:23 +0000 Subject: [Lustre-devel] statahead_performance.pdf - Adobe Reader Message-ID: <070201c9a0c4$d6fe8b50$84fba1f0$@com> Fan Yong, The attached PDF shows the results you gathered in a form that makes it easier to understand the results and helps comparison. Please could you review it for correctness. Cheers, Eric -------------- next part -------------- A non-text attachment was scrubbed... Name: statahead_performance.pdf Type: application/pdf Size: 80910 bytes Desc: not available URL: From Yong.Fan at Sun.COM Mon Mar 9 16:34:02 2009 From: Yong.Fan at Sun.COM (Yong Fan) Date: Tue, 10 Mar 2009 00:34:02 +0800 Subject: [Lustre-devel] statahead_performance.pdf - Adobe Reader In-Reply-To: <070201c9a0c4$d6fe8b50$84fba1f0$@com> References: <070201c9a0c4$d6fe8b50$84fba1f0$@com> Message-ID: <49B544FA.9020009@sun.com> Eric Barton 写道: > Fan Yong, > > The attached PDF shows the results you gathered in a form > that makes it easier to understand the results and helps > comparison. Please could you review it for correctness. > > Yes, it is much help to understand the performance improvement by statahead with the chart. But I think the first chart for "Statahead Dirents/second" maybe some misguide, since the TCP result and IB result are from different hardware environment (I mean the CPU and memory), that the speed of stating under IB is faster than TCP case is meaningless. Relatively, the second chart for "Statahead Speedup" is more useful, which is relatively independent from hardware (CPU and memory). It gives the trend of upper bound on performance improvement from statahead under different network configure, and different dir/file ratio. Thanks! -- Fan Yong > Cheers, > Eric > > From eeb at sun.com Mon Mar 9 17:33:17 2009 From: eeb at sun.com (Eric Barton) Date: Mon, 09 Mar 2009 17:33:17 +0000 Subject: [Lustre-devel] statahead_performance.pdf - Adobe Reader In-Reply-To: <49B544FA.9020009@sun.com> References: <070201c9a0c4$d6fe8b50$84fba1f0$@com> <49B544FA.9020009@sun.com> Message-ID: <002901c9a0dd$22249850$666dc8f0$@com> Fan Yong, I don't think the factor of ~2 can be explained simply by node performance. But why debate the point when it will be simpler to repeat the measurements (i.e. repeating the tests exactly with the same parameters e.g. # dirents) using 1GigE on the same nodes that were used to measure IB. Cheers, Eric > -----Original Message----- > From: Yong.Fan at Sun.COM [mailto:Yong.Fan at Sun.COM] > Sent: 09 March 2009 4:34 PM > To: Eric Barton > Cc: lustre-devel at lists.lustre.org > Subject: Re: statahead_performance.pdf - Adobe Reader > > Eric Barton 写道: > > Fan Yong, > > > > The attached PDF shows the results you gathered in a form > > that makes it easier to understand the results and helps > > comparison. Please could you review it for correctness. > > > > > Yes, it is much help to understand the performance improvement > by statahead with the chart. > > But I think the first chart for "Statahead Dirents/second" maybe > some misguide, since the TCP result and IB result are from different > hardware environment (I mean the CPU and memory), that the speed > of stating under IB is faster than TCP case is meaningless. > > Relatively, the second chart for "Statahead Speedup" is more useful, > which is relatively independent from hardware (CPU and memory). > It gives the trend of upper bound on performance improvement from > statahead under different network configure, and different dir/file ratio. > > > Thanks! > -- > Fan Yong > > Cheers, > > Eric > > > > From nic at cray.com Mon Mar 9 20:26:25 2009 From: nic at cray.com (Nicholas Henke) Date: Mon, 09 Mar 2009 15:26:25 -0500 Subject: [Lustre-devel] moving obd_fail_check to libcfs In-Reply-To: <49A18841.7030200@cray.com> References: <49A18841.7030200@cray.com> Message-ID: <49B57B71.2060603@cray.com> Nic Henke wrote: > Would there be any objection to a patch that'd move the current > obdclass-centric obd_fail_check and friends to libcfs ? I'd like to be > able to use the same logic inside LNET and our new LND without > replicating the code. > > I'm thinking it would need to be renamed to CFS_FAIL_CHECK as well. > > Thoughts or suggestions? > > Nic For those of you playing the at-home game, bug 18750 is filed for this. Nic From bzzz at sun.com Tue Mar 10 06:41:05 2009 From: bzzz at sun.com (Alex Zhuravlev) Date: Tue, 10 Mar 2009 09:41:05 +0300 Subject: [Lustre-devel] some observations about metadata writeback cache Message-ID: Hello, I spent quite amount time thinking of wbc problem and I'd like to share the thoughts. for wbc we store metadata in local memory for two purposes: 1) later reintegration 2) read access (lookup, getattr, readdir) w/o server involvement for (2) it makes sense to store everything as "state". e.g. directory contains all alive entries, inode contains last valid attributes, etc. let's call it state cache. in theory reintegration can be done from the state cache and this is probably the most efficient way (in terms of network traffic and memory footprint). but for simpler implementation we can introduce log of changes for (1). in turn, the log can be per-object or just global log for given filesystem. it's hard to implement state cache in terms of operations because usual operation involves more than one object (e.g. parent directory + file). it's much simpler when state cache is per-object. literally the best example is linux's dcache and inode cache. it's also fairly simple to maintain such cache at level where single object is being modified. for our purposes this matches layer implementing OSD API - because all operations in OSD API are per single object. the same applies to reintegration because: * we need to break complex operations to be sent to different servers anyway * if we'd need to optimize log (i.e., create/unlink), then it's simpler to collapse log entries when they are basic operations * when we'd want to reintegrate from state cache we also need a layer to take metadata operations and translate them into per-object basic operations (updates). responsbility of this layer is: * to grab all required ldlm locks as the layer understands operation's nature, locking rules, etc * to check current state whether name exists alread (for create), permissions * to apply updates to state cache (and reintegration backend, if required) * to release ldlm locks essentially this is what current metadata server does. the difference is * locks to be acquired on remote node * current state can be on remote node (not in local state cache) * updates can be stored in local memory for later reintegration (perhaps this applies to usual mds) it looks quite obvious that it'd make sense to use metadata server code to implement wbc: * ldlm hides where lock is being mastered * dedicated osd layer below metadata server can maintain state cache needed to check existing names, attributes, permissions, etc * dedicated osd layer below metadata server can take care of reintegration implementation would look like set of the following modules: * mdf - metadata filter this is location-free metadata server operating on top of osd api, grabs ldlm locks, check current state, apply changes. * cosd - caching osd this is dedicated layer with osd api, it maintains state cache and all data needed for reintegration. it also tries to use network efficient: regular lookup can be implemented via underlying readdir, etc. * gosd - global osd very specific module allowing node to talk to remote storage over osd api, it's stateless, something similar to current mdc, but using different apis. some obvious cons of this approach: * implementation doesn't rely on any system specific thing like dcache/icache * we can unify the code and re-use it to implement regular metadata server, wbc and metadata proxy server * overall simplicity inter-layer interaction is well defined and simple, same about layer's functionality * clustered metadata fits this model very well because metadata server doesn't need to know whether some update local or remote any comments and suggestions are very welcome! thanks, Alex From Yong.Fan at Sun.COM Tue Mar 10 09:28:46 2009 From: Yong.Fan at Sun.COM (Yong Fan) Date: Tue, 10 Mar 2009 17:28:46 +0800 Subject: [Lustre-devel] statahead_performance.pdf - Adobe Reader In-Reply-To: <002901c9a0dd$22249850$666dc8f0$@com> References: <070201c9a0c4$d6fe8b50$84fba1f0$@com> <49B544FA.9020009@sun.com> <002901c9a0dd$22249850$666dc8f0$@com> Message-ID: <49B632CE.4040500@sun.com> Sorry, some test resources budgeting limit delayed the test a little. This is the last test result: TCP2, IB and Routed IB were tested on the same cluster with the same scale. TPC1 was tested on another cluster. Regards, -- Fan Yong > Fan Yong, > > I don't think the factor of ~2 can be explained simply by node > performance. But why debate the point when it will be simpler > to repeat the measurements (i.e. repeating the tests exactly > with the same parameters e.g. # dirents) using 1GigE on the > same nodes that were used to measure IB. > > Cheers, > Eric > > >> -----Original Message----- >> From: Yong.Fan at Sun.COM [mailto:Yong.Fan at Sun.COM] >> Sent: 09 March 2009 4:34 PM >> To: Eric Barton >> Cc: lustre-devel at lists.lustre.org >> Subject: Re: statahead_performance.pdf - Adobe Reader >> >> Eric Barton 写道: >> >>> Fan Yong, >>> >>> The attached PDF shows the results you gathered in a form >>> that makes it easier to understand the results and helps >>> comparison. Please could you review it for correctness. >>> >>> >>> >> Yes, it is much help to understand the performance improvement >> by statahead with the chart. >> >> But I think the first chart for "Statahead Dirents/second" maybe >> some misguide, since the TCP result and IB result are from different >> hardware environment (I mean the CPU and memory), that the speed >> of stating under IB is faster than TCP case is meaningless. >> >> Relatively, the second chart for "Statahead Speedup" is more useful, >> which is relatively independent from hardware (CPU and memory). >> It gives the trend of upper bound on performance improvement from >> statahead under different network configure, and different dir/file ratio. >> >> >> Thanks! >> -- >> Fan Yong >> >>> Cheers, >>> Eric >>> >>> >>> > > > -------------- next part -------------- A non-text attachment was scrubbed... Name: statahead_performance.ods Type: application/vnd.oasis.opendocument.spreadsheet Size: 32448 bytes Desc: not available URL: From Mallikarjunarao.Ragampudi at Sun.COM Tue Mar 10 11:55:16 2009 From: Mallikarjunarao.Ragampudi at Sun.COM (Mallik Ragampudi) Date: Tue, 10 Mar 2009 06:55:16 -0500 Subject: [Lustre-devel] LustreFS performance In-Reply-To: <8AD540D2-0B50-4630-B794-E65443352696@Sun.COM> References: <3376C558-E29A-4BB5-8C4C-3E8F4537A195@sun.com> <02FEAA2B-8D98-4C2D-9CE8-FF6E1EB135A2@sun.com> <8AD540D2-0B50-4630-B794-E65443352696@Sun.COM> Message-ID: <49B65524.6080707@sun.com> Vitaly, This is very comprehensive. Few comments: 1) I think it would be good to start with LLT2 (lustre-iokit: ior-survey) and get an out-of-the-box performance picture/comparison (1.6 and 2.0) from the whole cluster before testing the individual layers. 2) Can this plan be extended to include CMD performance testing as well ? I would expect that most of your test cases apply for the CMD as well ? 3) I assume the "CPU" parameter in your methodology refers to # of CPUs in MDS, right ? 4) I am not sure if we can test 32 cores without hitting other bottlenecks beyond 8 coress on the servers. This will cut down some combinations in the matrix. Thanks, Mallik Vitaly Fertman wrote: > **************************************************** > LustreFS benchmarking methodology. > **************************************************** > > The document aims to describe the benchmarking methodology which helps > to understand the LustreFS performance and reveal LustreFS bottlenecks in > different configurations on different hardware, to ensure the next > LustreFS > release does not downgrade comparing with a previous one. In other words: > Goal1. Understand the HEAD performance. > Goal2. Compare HEAD and b1_6 (b1_8) performance. > > To achieve the Goal1, the methodology suggests to test different > layers of > software in the bottom-top direction, i.e. the underlying back-end, > the target > server sitting on this back-end, the network connected to this target > and how > the target performs through this network, etc up to the whole cluster. > Each next step has only 1 change over the previous one, it is either a > new layer > added or 1 parameter in the configuration is changed (probably another > network > type or another back-end). Comparing the results of each test with the > previous > test, we get the overhead of the added layer or the performance impact of > changing this parameter. > > To achieve the Goal2, the methodology suggests to go in the reverse > top-bottom > direction, i.e. to test some large sub-systems first and, if a > downgrade vs. a previous > LustreFS version is detected, to perform more detailed tests. (This is > considered as > the primary goal of the 2.0 Performance Team). > > The document does not cover the way of fixing revealed problems, > probably some > special purpose test needs to be run or oprofile needs to be compiled > in -- it is our > of scope of the document. > > Obviously, it is not possible to perform all the thousands of tests in > all the configurations, > running all the special purpose tests, etc, the document tries to > prepare: > 1) all the essential and sufficient tests to see how the system > performs in general; > 2) some minimal amount of essential tests to see how the system scales > in different > conditions. > Therefore, the plan does not guarantee we will not miss a bottleneck > or a bug, it just > tries to cover maximum possible scenarios in most interesting > conditions/environment > states. > > The amount of tests described below is already about 2K, and there > will be definitely more, > and it will take a lot of time to perform all of them and to analyze > the results. So one of > the major concerns here is how to minimize the amount of test so that > we would not miss > some interesting case and would be able to get all the results within > a reasonable amount of > time. Please keep it in mind while looking at the tests below. > > **** Hardware Requirements. **** > > The test plan implies that we change only 1 parameter (cpu or disk or > network) > on each step. Thus, the HW requirements are: > > -- at least 1 node with: > CPU:32; > RAM: enough to have a tmpfs for MDS; > DISK: raid, regular. > NET: both GiGe and IB installed. > -- besides that: 8 clients, 4 other servers. > -- the other servers include: > DISK: raid, regular. > NET: both GiGe and IB installed. > -- client includes: > NET: both GiGe and IB installed. > > **** Software requirements **** > > 1. Short term. > 1.1 mdsrate > to be completed to test all the operations listed in MDST3 (see below). > 1.2 mdsrate-**.sh > to be fixed/written to run mdsrate properly and test all the > operations listed in > MDST3 (see below). > 1.3. fake disk > implement FAIL flag and report 'done' without doing anything in > obdfilter to get > a low-latency disk. > 1.4. MT. > add more tests here and implement them. > > 2. Long term. > 2.1. mdtstack-survey > - an echo client-server is to be written for mds similar to ost. > - a test script similar to obdfilter-survey.sh is to be written. > > **** Different configurations **** > > Configuration of Node: > RAM. Amount of RAM on nodes (?) > CPU. Count of CPUs on nodes (1..32) > DISK. Disk type (regular, raid, tmpfs, fake) > JOUR. Journal type (internal, external, ram) > > Q: which raid? > A: raid5, as it seems to be the most popular. > > fake: to get a low-latency disk, it is preferable to report 'done' > without doing anything > in obdfilter once some FAIL flag is set. It is useful for OST testing, > because first of all, > it does not have a CPU overhead of memcpy of using tmpfs and it lets > to test large > amount of data in contrast to tmpfs. As a drawback, it skips the > localfs code paths. > > Configuration of Cluster: > CL. Amount of clients (1,2,4,8) > OSS. Amount of OSS nodes (1,2,4) > NET. Network type (GiGe, IB) > OSTN. Amount of OST per nodes (1,2,4) > > Configuration of test. > TH. Amount of threads per client (1,2,4,8) > VER. Lustre version (b1_6, HEAD. later b1_8). > FEAT. Lustre features to turn off (COS, SA, RA, debug messages) > TEST. Specific test parameters. > > **** Testing **** > Low Layers Testing (LLT) > LLT1. Raw disk (lustre-iokit:sgpdd-survey) > LLT2. Local filesystem (lustre-iokit: ior-survey, is fs mounted > synchronously?) > > Network Testing (NETT). > NETT1. lnet: lnetself test. > NETT2. OBD: lustre-iokit: (obdfilter-survey, > echo_client-osc-..-net-..-ost-echo_server) > NETT3. MD: (not ready) > > OST Testing (OSTT). > OSTT1. Isolated OST (lustre-iokit: obdfilter-survey, > echo_client-obdfilter-..-disk) > OSTT2. Remote OST (lustre-iokit: obdfilter-survey, > echo_client-osc-..-ost-obdfilter-..-disk) > OSTT3. Client-OST IO (lustre-iokit: ost-survey, client-ost-disk). > > MDS Testing (MDST). > MDST1. Isolated MDS test (not ready) > MDST2. Remote MDS test (not ready) > MDST3. Simple Client-MDS operation test > > Mixed testing (MT) (not ready) > > **** Statistics **** > > During all the tests the following is supposed to be running on all > the servers: > 1) vmstat > 2) iostat, if there is some disk activity. > smth else? > > *** Goal1. Understand the HEAD performance. *** > > The Goal1 describes the testing methodology in the bottom-top direction, > from the lower layers (disk) to the complete Lustre cluster. > > LLT1. Raw disk (lustre-iokit:sgpdd-survey) > RAM: fixed > CPU: 1 > DISK: regular,raid,tmpfs (default=raid) > JOUR:- > CL: 1 > OSS:1 > NET: - > OSTN:- > TH: 1,2,4,8 (default=1) > F: debug > TEST: > *)bulk size is specified as rszlo/rszhi=[1,4,64,1024K] > *)TH is specified as thrlo/thrhi=[1,2,4,8] > *)the amount of objects to work on in parallel: crglo=crghi=[1;TH] > i.e. test only cases when all the threads work on the same file and > when all of them work on a separate file. > [bulk;separate or commin dir]=8 tests; > > Test matrix(TESTxTHxDISK): > Run TESTs with different amount of threads for each DISK. > TESTxTHxDISK=(8x4 - 1)x3=93 tests. > "-1" because TH=1 is already covered. > > Total:93 tests. > > *** NETT1. lnetself test.*** > > RAM: fixed > CPU: 1,2,8,32 (default=1) > DISK: - > JOUR:- > CL: 1,2,4,8 (default=1) > OSS:1 > NET: GiGe, IB (default=IB) > OSTN:1 > TH: 1,2,4,8 (default=1) > F: debug > TEST: > *) test type: PING,READ,WRITE tests > *) bulk size for READ/WRITE: 1k,4k,64k,1M > [1 ping + 4 reads + 4 writes] = 9 tests > > Test matrix (TESTxCLxTHxNETxCPU): > 1. Multi-thread test. > Run TESTs on CL=1 with different amount of threads. > TESTxTH=[1+4+4]x4=36 tests. > 2. Multi-client test > 2.1. Let's check how clients scale vs. threads per client (TH=1). > 2.2. Let's check how the system scale with many clients and threads > (TH=8). > Note: to be more demonstrative, the maximum amount of threads could be > taken > <8, if TH=8 reaches the maximum network throughput with small amount > of clients. > [CL>1;TH=1,8]. TESTxCLxTH=9x3x2=54 tests. > 3. Network test > As the nature of IB is different from GiGe, we need to repeat all the > tests > from (1,2) here. 36+54=90 tests. > 4. CPU test > Note: lnet fixes from Liang to be applied here. > Run TESTs on different amount of CPU. > It is mostly interesting to look at large amount of threads, as we are > going to benefit from handling them in parallel. > At the same time, if some HW (network) limit is reached, the result > will not be > very demonstrative, so test with 1 small & 1 large bulk size > only:[1k;1024K]: > [CL=8,TH=1,2,4,8]. TESTxCLxTHxCPU=5x1x4x(4-1)=60. > > Total: 240 tests. > > *** NETT2. OBD performance *** > lustre-iokit: obdfilter-survey, case=network. > > The results of this tests are to be compared with lnet results to get > the osc+ost+ptlrpc overhead. > > RAM: fixed > CPU: 1 > DISK: - > JOUR:- > CL: 1,2,4,8 (default=1) > OSS:1 > NET: IB > OSTN:1 > TH: 1,2,4,8 (default=1) > F: debug > TEST: > *) bulk size: rszlo=rszhi=N (1,4,64,1024) > *) TH is specified through: thrlo=1, thrhi=8 (thread count, 1,2,4,8) > *) the amount of objects is: nobjlo=nobjhi=[1;TH] > i.e. test only cases when all the threads work on the same file and > when all of them work on a separate file. > [4 bulks; common or separate dir]=8 tests > > Test matrix(TESTxTHxCLxNET): > 1. Multi-thread test. > Run TESTs on CL=1 with different amount of threads. TESTxTH=8x4=32 tests > 2. Multi-client test > 2.1. Let's check how clients scale vs. threads per client (TH=1). > 2.2. Let's check how the system scale with many clients and threads > (TH=8). > Note: to be more demonstrative, the maximum amount of threads should > be taken > <8, if TH=8 reaches the maximum network throughput with small amount > of clients. > [CL>1;TH=1,8]. TESTxCLxTH=8x3x2=48 tests. > 3. Network test. > Having IB results in hand after (1,2) and these results from NETT1, we > already see how > osc+ost+ptlrpc changes the behavior. There is no reason to repeat them > for GiGe, it seems. > 4.CPU test > Note: lnet fixes from Liang to be applied here. > Run TESTs on different amount of CPU. > It is mostly interesting to look at large amount of threads, as we are > going to benefit from handling them in parallel. > At the same time, if some HW (network) limit is reached, the result > will not be > very demonstrative, so test with 1 small & 1 large bulk size > only:[1k;1024K]: > [CL=8,TH=1,2,4,8]. TESTxCLxTHxCPU=4x1x4x(4-1)=48. > > Total: 128 tests. > > *** OSTT1. Isolated OST *** > lustre-iokit: obdfilter-survey, case=disk > > The results of this tests are to be compared with LLT results to get > the OST > stack overhead. > > RAM: fixed > CPU: 1,2,8,32 (default=1) > DISK: regular, raid, fake (default=fake) > JOUR: int, ext, ram, (default=int) > CL: 1 > OSS:1 > NET: - > OSTN:1,2,4 (default=1) > TH: 1,2,4,8 (default=1) > F: debug > TEST: > *) bulk size: rszlo=rszhi=N (1,4,64,1024K) > *) TH is specified through: thrlo=1, thrhi=8 (1,2,4,8) > *) each OST is supposed to be configured on a separate disk. > *) the amount of objects is: nobjlo=nobjhi=[1;TH] > i.e. test only cases when all the threads work on the same file and > when all of them work on a separate file. > [4 bulks; common of separate dir]=8 tests > > Test matrix(TESTxTHxOSTNxDISKxCPU): > 1. Multi-thread test. > Run TESTs on OSTN=1 with different amount of threads. TESTxTH=8x4=32 > tests > 2. Multi-OST test > 2.1. Let's check how OSTs vs. threads per OST scale (TH=OSTN). > 2.2. Let's check how the system scale with many OSTs and threads > (TH=8*OSTN). > [OSTN>1;TH=OSTN,8*OSTN]. TESTxOSTNxTH=8x2x2=32 tests. > 3. DISK test > As other disks are completely different, so lets repeat most of the > (1,2) for 2 others: > [TH=OSTN;8*OSTN]: TESTxOSTNxTHxDISK=8x3x2x2=96 > 4. JOURNAL test. > Limit the tests with only raid-disk. > Limit the test with only 1 large and 1 small bulk:[1,1024K]. > TESTxOSTNxTHxJOUR: 4x3x2x2=48 > 5. CPU test > Note: lnet fixes from Liang to be applied here. > Run TESTs on different amount of CPU. It is better to perform it on a > fast > backend (DISK=fake) to see how CPU really matters. > It is mostly interesting to look at large amount of threads, as we are > going > to benefit from handling them in parallel. > Also, run with a small & a large bulk only:[1,1024K] > [OSTN=4,TH=1,2,4,8]: TESTxOSTNxTHxCPU=4x1x4x3=48 > > Total: 256 tests. > > *** OSTT2. Real OST test *** > lustre-iokit: obdfilter-survey, case=netdisk > > This test is a composition of OBD performance and Isolated OST tests, > so its results are to be compared with NETT2 & OSTT1 results. > > RAM: fixed > CPU: 1,2,8,32 (default=1) > DISK: fake > JOUR:int > CL: 1,2,4,8 (default=1) > OSS:1,2,4 (default=1) > NET: IB > OSTN:1,2,4 (default=1) > TH: 1,2,4,8 (default=1) > F: debug > TEST: > *) bulk size: rszlo=rszhi=N (1,4,64,1024) > *) TH is specified through: thrlo=1, thrhi=8 (thread count, 1,2,4,8) > *) each OST is supposed to be configured on a separate disk. > *) the amount of objects is: nobjlo=nobjhi=[1;TH] > i.e. test only cases when all the threads work on the same file and > when all of them work on a separate file. > [4 bulks; common of separate dir]=8 tests > > Test matrix(TESTxTHxCLxCPUxNETxOSSxOSTN): > 1. Multi-thread test. > Run TESTs on CL=1 with different amount of threads. TESTxTH=8x4=32 tests > 2. Multi-client test > 2.1. Let's check how clients scale vs. threads per client (TH=1). > 2.2. Let's check how the system scale with many clients and threads > (TH=8). > Note: to be more demonstrative, the maximum amount of threads could be > taken > <8, if TH=8 reaches the maximum network throughput with small amount > of clients. > [CL>1;TH=1,8]. TESTxCLxTH=8x3x2=48 tests. > 3. Network test > Having IB results in hand after (1,2) and these results from NETT2, we > already see how > osc+ost+ptlrpc+obdfilter changes the behavior. Thus, there is no > reason to repeat them > for GiGe, it seems. > 4.CPU test > Note: lnet fixes from Liang to be applied here. > Run TESTs on different amount of CPU. > It is mostly interesting to look at large amount of threads, as we are > going to benefit from handling them in parallel. > At the same time, if some HW (network) limit is reached, the result > will not be > very demonstrative, so test with 1 small & 1 large bulk size > only:[1k;1024K]: > [CL=8,TH=1,2,4,8]. TESTxCLxTHxCPU=4x1x4x(4-1)=48. > 5. OSTN test. > The same OSC, network, CPU, disk, just check how OST stack (see 1,2 > tests) is scalable. > 5.1. Let's check how N threads per 1 OST vs. 1 thread per N OST scales > (CL=OSTN). > 5.2. Let's check how the system scale with many clients and threads > (CL=8) > Note: to be more demonstrative, the maximum amount of threads could be > taken > <8, if TH=8 reaches the maximum network throughput with small amount > of clients. > It seems enough to look at 1 small & 1 large bulk only: [1,1024K] > [CL=OSTN,8;TH=1,8]. TESTxCLxTHxOSTN=4x2x2x2=32 tests > 6. OSS test. > 6.1. Let's check how 1 thread per N OST vs. 1 thread per N OSS scales > (CL=OSS). > 6.2. Let's check how the system scale with many clients and threads > (CL=8) > Note: to be more demonstrative, the maximum amount of threads could be > taken > <8, if TH=8 reaches the maximum network throughput with small amount > of clients. > It seems enough to look at 1 small & 1 large bulk only: [1,1024K] > [CL=OSS,8;TH=1,8]. TESTxCLxTHxOSTN=4x2x2x2=32 tests > > Total:192 tests > > *** OSTT3. Client-OST test *** > lustre-iokit: ior-survey. > > The test results are to be compared with OSTT2 results to get the > overhead > for Lustre Client: client stack, distributed locking, etc. > > RAM: fixed > CPU: 1,2,8,32 (default=1) > DISK: fake > JOUR:int > CL: 1,2,4,8 (default=1) > OSS:1,2,4 (default=1) > NET: IB > OSTN:1,2,4 (default=1) > TH: 1,2,4,8 (default=1) > F: debug > TEST: > *) CL is specified through $clients_hi > *) TH is specified through $tasks_per_client_hi > *) bulk is specified through rsize_lo/hi (1,4,64,1028K) > *) file_per_task=[0;1] > i.e. test only cases when all the threads work on the same file and > when all of them work on a separate file. > [4 bulks; common of separate dir]=8 tests > > Test matrix(TESTxTHxCLxCPU): absolutely the same as for OSTT2. > > NETT3. MD: (not ready) > MDST1. Isolated MDS (not ready) > MDST2. Remote MDS (not ready) > This set of tests need to be implemented in a utility similar to > obdfilter-survey > but for MDS testing. > > MDST3. Simple Client-MDS operation tests > > 1. create,mknod,mkdir (symlink, link??) > RAM: fixed > CPU: 1,2,8,32 (default=1) > DISK(MDS): tmpfs, raid, regular (default=tmpfs) > DISK(OST): tmpfs > JOUR: int,ext,ram (default=int) > CL: 1,2,4,8 (default=1) > OSS:1,2,4 (default=1) > NET: IB > OSTN:1 > TH: 1,2,4,8 (default=1) > F: debug > TEST: it will be probably mdsrate/mdsrate-create-small.sh, but it > needs to be > fixed to support all of these operations, not only create. If so: > *) TH could be specified through THREADS_PER_CLIENT=[1,2,4,8] > *) CL is specified through CLIENTS or NODES_TO_USE. > *) NOSINGLE should be provided > *) add --dirnum option to COMMAND > *) DIRNUM=[1,TH*CL], so we test a case when all the threads work in the > same dir and when each works in a separate one. > *) nfiles is files-per-dir * DIRNUM > [common or separate dir]=2tests; > > Note: we should probably limit the amount of files in 1 directory with > 2M, > otherwise the performance will definitely downgrade. > > Test matrix(TESTxTHxCLxCPUxNETxOSS): > 1. Multi-thread test. > Run TESTs on CL=1 with different amount of threads. TESTxTH=2x4-1=7 tests > (not 8 as if TH=1, DIRNUM=1, and this is already covered). > 2. Multi-client test > 2.1. Let's check how clients scale vs. threads per client (TH=1). > 2.2. Let's check how the system scale with many clients and threads > (TH=8). > Note: to be more demonstrative, the maximum amount of threads could be > taken > <8, if TH=8 reaches the maximum network throughput with small amount > of clients. > [CL>1;TH=1,8]. TESTxCLxTH=2x3x2=12 tests. > 3. Striped test. > 2.1. Let's check how multi-client system scales (TH=1). > 2.2. Let's check how large load system scales (TH=8) > Test (only) create with different stripeness. > TESTxCLxTHxOSS=[2x4x2-1]x2=30 > 4. Network test > Having IB results in hand after (1,2,3) and these results from NETT1, > we already see > how mdc+mdt-stack+ptlrpc changes the behavior. There is no reason to > repeat them > for GiGe, it seems. > 5. DISK test. > Unlink the OST testing, we do not have echo-md client (MDTT1), thus we > have not checked > how different disks impact the performance, so we need to check it here. > Limit this test with only couple of operations: create, mknod. > As different disks are of completely different nature we need to > repeat most of (1,2) here > [TH=1,8]: TESTxCLxTHxDISK=(2x4x2-1)x2=30 > 6. JOURNAL test. > Repeat (5) for different journals, but limit the test with raid-disk > only. TESTxCLxTHxDISKxJOUR=(2x4x2-1)x1x2=30 > 7.CPU test > Note: lnet fixes from Liang to be applied here. > Run TESTs on different amount of CPU. > Limit this test with only couple of operations: create, mknod. > It is mostly interesting to look at large amount of threads, as we are > going to benefit from handling them in parallel, so run it for CL=max > only: [CL=8,TH=1,2,4,8]. TESTxCLxTHxCPU=2x1x4x(4-1)=24 > > Total: 19 tests for mkdir, 103 for mknod, 133 for create. > > 2. lookup (mdsrate-lookup-1dir.sh => mdsrate-lookup.sh) > RAM: fixed > CPU: 1,2,8,32 (default=1) > DISK: tmpfs > JOUR: int > CL: 1,2,4,8 (default=1) > OSS:1 > NET: GiGe,IB (default=IB) > OSTN:1 > TH: 1,2,4,8 (default=1) > F: debug > TEST: it will be probably better to work out mdsrate-lookup-1dir.sh, > which > could work in several directories in parallel. > *) TH could be specified through THREADS_PER_CLIENT=[1,2,4,8] > (to be added into the script) > *) CL is an amount of nodes specified in CLIENTS or NODES_TO_USE. > *) NOSINGLE should be provided > *) add --dirnum option to COMMAND > *) DIRNUM=[1,TH*CL], so we test a case when all the threads work in the > same dir and when each works in a separate one. > *) nfiles is files-per-dir * DIRNUM > *) add READDIR_ORDER to test both random and readdir order lookups. > [common or separate di; readdir,random order]=4 tests. > > Q: it seems this test does md_getattr_name(), instead of lookup, thus > no lock > enqueue is involved. > A: what about to replace it with access(2)?? > > Test matrix(TESTxTHxCLxCPUxNET): the same as for (1, mknod), but 4 tests > instead of 2: 19x2=38 tests. > > 3. stat > > RAM: fixed > CPU: 1,2,8,32 (default=1) > DISK(MDS): tmpfs, raid, regular (default=tmpfs) > DISK(OST): tmpfs > JOUR: int,ext,ram (default=int) > CL: 1,2,4 (default=1) > OSS:1,2,4 (default=1) > NET: GiGe,IB (default=IB) > OSTN:1 > TH: 1,2,4,8 (default=1) > F: debug > TEST: mdsrate/mdsrate-stat-small.sh > *) add THREADS_PER_CLIENT to the script to specify TH > *) CL is specified through CLIENTS or NODES_TO_USE. > *) NOSINGLE should be provided > *) add --dirnum option to COMMAND > *) DIRNUM=[1,TH*CL], so we test a case when all the threads work in the > same dir and when each works in a separate one. > *) nfiles is files-per-dir * DIRNUM > *) add READDIR_ORDER to test both random and readdir order lookups. > [common or separate dir; readdir,random order]=4 tests. > > Q: do we want to test stat(2) with other then tmpfs disk on OST? > what journal should it have if so? > > Test matrix(TESTxTHxCLxCPUxNETxDISKxJOUR): the same as for (1, create), > but 4 tests instead of 2: 133x2=266 tests. > > 4. unlink (mdsrate-create-small.sh, run twice??) > it should be run (and it is run in mdsrate-create-small.sh) for all > the operations in (1), i.e. create, mkdir, mknod. > The test matrix is the same and the total: > 19 tests for mkdir, 103 for mknod, 133 for create. > > 5. chmod (mdsrate-chmod.sh, new one, fix mdsrate) > The same as (1, mkdir) and the total: 19 tests. > > 6. utime (mdsrate-utime.sh, new one, fix mdsrate) > The same as (1, mkdir) and the total: 19 tests. > > 7. chown (mdsrate-chown.sh, new one, fix mdsrate) > The same as (1, create), but skip different DISKs&JOURNALs: > 19 + 30 + 24=73 tests. > > 8. rename (mdsrate-rename.sh, new one, fix mdsrate) > The same as (1, mkdir) and the total: 19 tests. > > 9. find > Q: despite the fact we currently have a large downgrade with > "find -f type", do we want to have this test in the general test set? > > **** MT. Mixed testing. **** > > MT1. Create-write test. > RAM: fixed > CPU: 32 > DISK(MDS): tmpfs, raid (default=tmpfs) > DISK(OST): raid > JOUR: int > CL: 1,2,4,8 (default=1) > OSS:1 (default=1) > NET: IB > OSTN:1 > TH: 1,2,4,8 (default=1) > F: debug > TEST: must be a new one. Each thread creates files in a loop, writes 1 > bulk to each and closes it. > *) it is enough to test with a small bulk only: [1k] > *) [common or separate dir]=2tests; > > Test matrix(TESTxTHxCLxCPUxNETxOSS): > 1. Multi-thread test. > Run TESTs on CL=1 with different amount of threads. TESTxTH=2x4-1=7 tests > (not 8 as if TH=1, it is always in 1 dir, and this is already covered). > 2. Multi-client test > 2.1. Let's check how clients scale vs. threads per client (TH=1). > 2.2. Let's check how the system scale with many clients and threads > (TH=8). > Note: to be more demonstrative, the maximum amount of threads could be > taken > <8, if TH=8 reaches the maximum network throughput with small amount > of clients. > [CL>1;TH=1,8]. TESTxCLxTH=2x3x2=12 tests. > 3. DISK test. > Check how different disks impact on the performance. > As different disks are of completely different nature we need to > repeat most of (1,2) here > [TH=1,8]: TESTxCLxTHxDISK=(2x4x2-1)x1=15 > > Total: 34 tests. > > MT2. Create-Readdir test. > RAM: fixed > CPU: 32 > DISK(MDS): tmpfs, raid (default=tmpfs) > DISK(OST): raid > JOUR: int > CL: 1,2,4,8 (default=1) (1 extra client does "ls -U") > OSS:1 (default=1) > NET: IB > OSTN:1 > TH: 1,2,4,8 (default=1) > F: debug > TEST: must be a new one. Each thread creates files in a loop and > immediately closes them. > 1 thread on another client does "ls -U". It is done in 1 directory. > > The test matrix is exactly the same as for MT1. Total: 34 tests. > > MT3. ??? Some more tests ???? > > **** Goal2. Compare HEAD and b1_6 (b1_8) performance. **** > > This paragraph describes the testing methodology in the reverse order > of testing, > i.e. in the top-bottom direction, making sure new LustreFS (HEAD) > version does > not downgrade comparing with the previous ones (b1_6/b1_8). > > Therefore, the first testing cycle includes: > MT, MDST3, OSTT3, NETT1. > from the above tests. In the case a downgrade is detected, lower layer > tests are > to be run until the downgrade disappear. > > -- > Vitaly -- Mallik Ragampudi (877)860-5044 Lustre Engineering x52907 Sun Microsystems Mallikarjunarao.Ragampudi at sun.com From nic at cray.com Tue Mar 10 14:39:13 2009 From: nic at cray.com (Nicholas Henke) Date: Tue, 10 Mar 2009 09:39:13 -0500 Subject: [Lustre-devel] LustreFS performance In-Reply-To: <20090302204501.GQ3199@webber.adilger.int> References: <3376C558-E29A-4BB5-8C4C-3E8F4537A195@sun.com> <02FEAA2B-8D98-4C2D-9CE8-FF6E1EB135A2@sun.com> <8AD540D2-0B50-4630-B794-E65443352696@Sun.COM> <20090302204501.GQ3199@webber.adilger.int> Message-ID: <49B67B91.9080605@cray.com> Andreas Dilger wrote: > On Mar 02, 2009 20:04 +0300, Vitaly Fertman wrote: >> RAM: enough to have a tmpfs for MDS; >> **** Statistics **** >> >> During all the tests the following is supposed to be running on all >> the servers: >> 1) vmstat >> 2) iostat, if there is some disk activity. >> smth else? > > I would propose either LLNL's LMT or HP's collectl, which both also > collect Lustre stats. Those both provide more information than the > above, and having the IO/CPU load correlated to Lustre RPC counts is > very useful. It would be great if we could standardize on a set of tools for performance issues. I've got to think a set of tools like this would make it easier for customer & partners to gather the correct data the first time. Cray has been using lstats, a package of scripts we got from Sun a while back. We've added things like AT timeout and sar per-cpu usage to it (see bug 18574 att 22140 for complete set of scripts). I'm all for using collectl, but I think the requirements and setup for LMT makes it a tough sell. Does Sun have a set of customizations for collectl or does the standard collectl collect enough information? Nic From Mark.Seger at hp.com Tue Mar 10 15:07:36 2009 From: Mark.Seger at hp.com (Mark Seger) Date: Tue, 10 Mar 2009 11:07:36 -0400 Subject: [Lustre-devel] LustreFS performance In-Reply-To: <49B67B91.9080605@cray.com> References: <3376C558-E29A-4BB5-8C4C-3E8F4537A195@sun.com> <02FEAA2B-8D98-4C2D-9CE8-FF6E1EB135A2@sun.com> <8AD540D2-0B50-4630-B794-E65443352696@Sun.COM> <20090302204501.GQ3199@webber.adilger.int> <49B67B91.9080605@cray.com> Message-ID: <49B68238.4000901@hp.com> >>> **** Statistics **** >>> >>> During all the tests the following is supposed to be running on all >>> the servers: >>> 1) vmstat >>> 2) iostat, if there is some disk activity. >>> smth else? >>> >> I would propose either LLNL's LMT or HP's collectl, which both also >> collect Lustre stats. Those both provide more information than the >> above, and having the IO/CPU load correlated to Lustre RPC counts is >> very useful. >> > > It would be great if we could standardize on a set of tools for performance > issues. I've got to think a set of tools like this would make it easier for > customer & partners to gather the correct data the first time. > > Cray has been using lstats, a package of scripts we got from Sun a while back. > We've added things like AT timeout and sar per-cpu usage to it (see bug 18574 > att 22140 for complete set of scripts). > > I'm all for using collectl, but I think the requirements and setup for LMT makes > it a tough sell. Does Sun have a set of customizations for collectl or does > the standard collectl collect enough information? > My goal when I wrote collectl was to provide one-stop shopping for as much system performance data as seemed relevant and view lustre as only one of many data sources. To that end, if you do a merge of all the data collected by the *stat utilities, sar, perfquery (for IB), many of the lustre stats (but not all) and maybe a few others you'll get closer to understanding what collectl can collect. On the output side you can pick and choose what to display - when used interactively only those data elements are collected but when run as a daemon you can collect them all and replay the data as ofter as you like looking at different slices. As for LMT I haven't played with it as my interests are in dealing with all data. However, as an exercise left to the reader, there are a number of switches for changing collectl's display as well as --home which moves the cursor the terminal's home position before displaying the output, giving a display similar to the feel of top. If you want to display what's happening to lustre and your disks, cpu, etc all at the same time on a refreshing display, --top is definitely the way to go. And finally, if you want something totally different and are feeling creative, just write your own print routines in perl and tell collectl to use them with the --export switch. -mark From Vitaly.Fertman at Sun.COM Tue Mar 10 16:40:53 2009 From: Vitaly.Fertman at Sun.COM (Vitaly Fertman) Date: Tue, 10 Mar 2009 19:40:53 +0300 Subject: [Lustre-devel] LustreFS performance In-Reply-To: <49B65524.6080707@sun.com> References: <3376C558-E29A-4BB5-8C4C-3E8F4537A195@sun.com> <02FEAA2B-8D98-4C2D-9CE8-FF6E1EB135A2@sun.com> <8AD540D2-0B50-4630-B794-E65443352696@Sun.COM> <49B65524.6080707@sun.com> Message-ID: <861E8C51-4422-4C51-9062-DC3916DB96D6@Sun.COM> On Mar 10, 2009, at 2:55 PM, Mallik Ragampudi wrote: > Vitaly, > > This is very comprehensive. Few comments: > > 1) I think it would be good to start with LLT2 (lustre-iokit: ior- > survey) and get an out-of-the-box performance > picture/comparison (1.6 and 2.0) from the whole cluster before > testing the individual layers. LLT2 is about underlying local fs on servers, not Lustre. the overall ior-survey is OSTT3 (it was wrongly mentioned as ost-survey in the document header, sorry, it must be ior-survey as written in the detailed section) > 2) Can this plan be extended to include CMD performance testing as > well ? I would expect that most > of your test cases apply for the CMD as well ? sure, I will extend it. > 3) I assume the "CPU" parameter in your methodology refers to # of > CPUs in MDS, right ? to the server which is being tested at the moment, i.e. OST for OST* tests, MDS fro MDS* tests. In the mixed section it refers to MDS indeed. > 4) I am not sure if we can test 32 cores without hitting other > bottlenecks beyond 8 coress on the servers. if we hit some bottleneck, we will get a problem to solve, what is good ;) however, to see the bottleneck from these results, it is better to run MT tests with different amount of CPUs (the same as for all the CPU tests, 1,2,8,32). > This will cut down some combinations in the matrix. > > Thanks, > Mallik > -- Vitaly From adilger at sun.com Tue Mar 10 19:07:42 2009 From: adilger at sun.com (Andreas Dilger) Date: Tue, 10 Mar 2009 13:07:42 -0600 Subject: [Lustre-devel] statahead_performance.pdf In-Reply-To: <49B632CE.4040500@sun.com> References: <070201c9a0c4$d6fe8b50$84fba1f0$@com> <49B544FA.9020009@sun.com> <002901c9a0dd$22249850$666dc8f0$@com> <49B632CE.4040500@sun.com> Message-ID: <20090310190742.GS3199@webber.adilger.int> On Mar 10, 2009 17:28 +0800, Yong Fan wrote: > Sorry, some test resources budgeting limit delayed the test > a little. This is the last test result: > TCP2, IB and Routed IB were tested on the same cluster > with the same scale. TPC1 was tested on another cluster. Fan Yong, thanks for the updated numbers. These are MUCH easier to understand than without the graphs. Minor note - the graphs should be "X-Y" graphs, with the first column (% files) as the X axis instead of a label. That makes it clear in the first case that the files vs. directories performance is in fact linear with the percentage of files in the directory. What else this exposes is that the statahead performance is really limited by the lack of OST statahead and/or SOM. The 100% files case (as I expected) is limited by the need to do the OST stat before returning anything to userspace, so Amdahl's law says it can take at minimum 50% as long as without statahead. For the 0% files case (all subdirs), and when SOM is available, there is no such limitation and the stat rate can increase significantly more. > Regards, > -- > Fan Yong > > Fan Yong, > > > > I don't think the factor of ~2 can be explained simply by node > > performance. But why debate the point when it will be simpler > > to repeat the measurements (i.e. repeating the tests exactly > > with the same parameters e.g. # dirents) using 1GigE on the > > same nodes that were used to measure IB. > > > > Cheers, > > Eric > > > > > >> -----Original Message----- > >> From: Yong.Fan at Sun.COM [mailto:Yong.Fan at Sun.COM] > >> Sent: 09 March 2009 4:34 PM > >> To: Eric Barton > >> Cc: lustre-devel at lists.lustre.org > >> Subject: Re: statahead_performance.pdf - Adobe Reader > >> > >> Eric Barton 写道: > >> > >>> Fan Yong, > >>> > >>> The attached PDF shows the results you gathered in a form > >>> that makes it easier to understand the results and helps > >>> comparison. Please could you review it for correctness. > >>> > >>> > >>> > >> Yes, it is much help to understand the performance improvement > >> by statahead with the chart. > >> > >> But I think the first chart for "Statahead Dirents/second" maybe > >> some misguide, since the TCP result and IB result are from different > >> hardware environment (I mean the CPU and memory), that the speed > >> of stating under IB is faster than TCP case is meaningless. > >> > >> Relatively, the second chart for "Statahead Speedup" is more useful, > >> which is relatively independent from hardware (CPU and memory). > >> It gives the trend of upper bound on performance improvement from > >> statahead under different network configure, and different dir/file ratio. > >> > >> > >> Thanks! > >> -- > >> Fan Yong > >> > >>> Cheers, > >>> Eric > >>> > >>> > >>> > > > > > > > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc. From steve.allen at usherbrooke.ca Wed Mar 11 15:13:11 2009 From: steve.allen at usherbrooke.ca (Steve Allen) Date: Wed, 11 Mar 2009 11:13:11 -0400 Subject: [Lustre-devel] kernel including perfctr Message-ID: <49B7D507.4080101@usherbrooke.ca> Since we had troubles compiling the Lustre source on our CentOS distribution with Infiniband, we installed the prepatched kernel. Poorly this kernel does not have the Perfctr patch enabling measure of hardware performance with PAPI (http://icl.cs.utk.edu/papi/) Can we expect, in a near future, that the Perfctr patch will be added to the kernel available from Lustre? Steve Allen HPC Analyst CCS, RQCHP Universite de Sherbrooke From Vitaly.Fertman at Sun.COM Fri Mar 13 15:32:00 2009 From: Vitaly.Fertman at Sun.COM (Vitaly Fertman) Date: Fri, 13 Mar 2009 18:32:00 +0300 Subject: [Lustre-devel] SOM Recovery of open files Message-ID: Hi All, this is the summary of our following discussion with Andreas from Feb24, it includes the problem use cases of SOM recovery and interoperability, describes problems of each use case and suggests possible solutions. UseCase1. Client is evicted from MDS and re-connects. UseCase2. MDS failover. UseCase3. Client eviction and following MDS failover. Problem1 File opened for write is not re-opened; Problem2. File opened (precisely, with IOEpoch opened) for truncate is not re-opened; Problem3. Client is able to write (new syscall) to a not re-opened file (MDS has no control over IO happened in the cluster). Problem4. Client is able to flush dirty data to OST for a not re- opened file. Problem5. Client is able to re-send a write RPC to OST for a not re- opened file. Solution1: New OPEN rpc on recovery. Problem1.1: does not work for client eviction so far, when no MDS recovery is involved. Problem1.2. even if recovery is involved but the client is already evicted, it does not work. Problem1.2: does not work for truncate. The situation is pretty rare as client does not cache punches, but what if at the time of the client eviction from MDS, the connection between this client and an OST is unstable so that punches will hang in the re-send list for a while, enough for another client to modify the file -- MDS gets a new SOM cache, and later punch will modify the file. Solution2: LOV EA lock, client blocks new IO if absent. Problem2.1: LOV EA works for new syscalls only, not for (Problem4,Problem5). Solution3: SOM cache is removed upon client eviction for all the opened IOEpochs Problem3.1. works until SOM cache is re-validated before some later IO happens from a lost client. Solution4: client's dirty cache is controlled from OST through extent locks. MDS removes SOM cache for inode on client eviction, next file writer sees there is no cache on MDS upon file open, thus the cache is re-obtained under [0;EOF] extent lock, what flushes all the data on OST. Problem4.1: Lockless IO (write,truncate) is not handled this way. rpc may be sitting in the re-send list enough for another client to modify the file, SOM cache is re-obtained to MDS, and delayed write/punch makes it invalid. Problem4.2: Locked truncate is not handled this way. Enqueue may be sitting in re-send list similar to 4.1. Solution5: Cluster flush on MDS-OST synchronization. SOM is disabled for a file until all the OSTs from its stripe are synchronized with MDS; Synchronization includes: all the clients flushes their dirty cache to OST, llog cookie is sent to MDS, MDS removes SOM cache for files involved. It means, a new IOEpoch opens but cache is not re-validated at the end; getattr does not obtain SOM cache. Problem5.1. Lockless IO (write,truncate) is not handled this way. Problem5.2. Locked truncate is not handled this way (see Problem4.2) UseCase4. Upgrade to SOM-enabled Lustre. All the above problems exists. Problem6. No IOEpoch has been opened (however, the SOM cache is removed synchronously on open) truncate does not close "opened" file at all, i.e. MDS has no control over IO happened in the cluster and later punch may destroy the SOM cache on MDS. Solution6. Send done_writing even if SOM is disabled. UseCase5. MDS fails over, a client has dirty cache but does not participate in the recovery. Solution7. Invalidate SOM cache on MDS on close. I.e. instead of blocking IO on the client, remove SOM cache in advance. Problem7.1. the solution does not work, thus we cannot depend on client as it may be evicted (UseCase5) if close is not committed yet, lockless IO still may happen with a delay.. Solution8. Evict client from OST once client is evicted from MDS (via MDS->OSS connection and set_info(KEY_EVICT_BY_NID)). Or cancel this client extent locks only. Therefore, prevent any IO happen since then. Problem8.1. if 2 mds failovers happen right one after another, it seems mds is already not able to tell which clients are lost over failover -- after the first failover it lets the clients to re-connect and overwrites the previous info about connected clients but does not succeeds to tell ost to evict the client -- and here the 2nd mds failure happens. This solution could be probably done in some different way -- (a) client itself informs OST it is evicted; (b) MDS provides the full list of connected clients to OST on boot and then informs OSTs about client evictions; Solution9. Invalidate SOM cache on open for write. Problem1. Unless written synchronously, MDS may fail before open gets committed. Problem2. Even if committed, a new IOEPOCH may re-validate SOM cache which could become wrong due to a later lockless IO reached OST or a such. Still missed solutions: (*) block truncate and lockless IO (either it is a new syscall, an enqueue rpc is in re-send list; lockless IO (write,truncate) is in the re-send list) until the connection to MDS is restored. Problem: a possible race between time MDS restarts and client detects it is evicted. I.e. client may continue to send IO to OST but it must detect the time (and block its IO until file is re-opened) when MDS is up, MDS-OST synchronization is completed; Solution10. Cause the OST to get an ioepoch for the file and invalidate the SOM cache on the MDS by itself BEFORE allowing the lockless operation to complete. Solution11. Client could wait for an open+SOM invalidate to commit before sending the lockless operation. Solution12. OST may send a new RPC to all the clients once MDS-OST synchronization starts. Clients re-validate its connection to MDS and re-opens files on MDS is so, client blocks its IO to OST until done. If client re-connects to OST, it must re-validate MDS connection as well right before that. -- Vitaly From eeb at sun.com Mon Mar 16 12:56:45 2009 From: eeb at sun.com (Eric Barton) Date: Mon, 16 Mar 2009 12:56:45 +0000 Subject: [Lustre-devel] Oleg/Mike Work on Apps Metrics - FW: Mike Booth week ending 2009.03.15 In-Reply-To: <48A6CF99-F630-4581-9FFC-05E3AFF66FA0@Sun.COM> References: <015c01c9a5a7$cb2f83a0$618e8ae0$@com> <52E982EB-3ADE-46FA-A7BC-D1645961DF0B@Sun.COM> <018101c9a5b6$25dbc300$71934900$@com> <48A6CF99-F630-4581-9FFC-05E3AFF66FA0@Sun.COM> Message-ID: <023201c9a636$a987e4b0$fc97ae10$@com> Mike, Yes, it would be fun to discuss - but I'm probably not going to be available for a discussion like that for a week or 2. BTW, I'm cc-ing lustre-devel since this is of general interest. I _do_ agree that for some apps, if there was sufficient memory on the app node to buffer the local component of a checkpoint and let it "dribble" out to disk would achieve better utilization of the compute resource. However parallel apps can be very sensitive to "noise" on the network they're using for inter- process communication - i.e. the checkpoint data has either to be written all the way to disk, or at least buffered somewhere so that moving it to disk will not interfere with the app's own communications. This latter concept is the basis for the "flash cache" concept. Actually, I think it's worth exploring the economics of it in more detail. The variables are aggregate network bandwidth into the distributed checkpoint cache, which determines the checkpoint time, and aggregate path-minimum bandwidth (i.e. lesser of network and disk bandwidth) from the cache to disk, which determines how soon the cache can be ready for the next checkpoint. The cache could be dedicated nodes and storage (e.g. flash) or additional storage on the OSSes, or any combination of either. And the interesting relationship is how compute cluster utilisation varies with the cost of the server and cache subsystems. -- Cheers, Eric > -----Original Message----- > From: Michael.Booth at Sun.COM [mailto:Michael.Booth at Sun.COM] > Sent: 16 March 2009 3:06 AM > To: Eric Barton > Subject: Re: Oleg/Mike Work on Apps Metrics - FW: Mike Booth week ending 2009.03.15 > > Eric, > > This is too bad. I should run the test on my laptop and see if I get > the same behavior. > > The huge bandwidth requirements (30+ gbyes/sec) that I see for > checkpoint-style I/O is driven in burst that last about 1/10 of the > time of the following computation. There is not a desire to assure > that everything is on disk before resuming computations. If while the > computations proceeded the system cleared out the cache, the next > write would go to cache at memory speed if the previous clean pages > could be reused for the next write. The bandwidth requirement to > achieve what appears to be memory speed I/o could be achieved in this > case with 3 gbytes/sec. > > There are middleware schemes being developed to do asynchronous I/O > on "other" nodes to transfer the checkpoint data out to the other > nodes so they write it all out. To me this is the middleware working > at odds with what the system software should naturally do for the > application. > > I think it is safe to say it is a minority of scientific applications > that are writing out and quickly reading it back like a typical linux > application, like web browsers. This type of I/O is usually limited > to codes that are larger than the sum of the nodes memory,, which is > rarer and rarer these days. > > I believe that making this work for these codes is a win in three ways;, > > One: reduces the need for high burst rate I/O to disk for many > programs while giving the perception of much faster I/O to the > application. > > Two: helps to reduce the impact of filesystem performance > variability, > > Three: Overall in the system, not having the system being hit with > huge burst of I/O by tens of thousands of cores at seemingly random > times, could reduce the variability of the complete file system. > > Should we discuss on the phone, with Oleg? > > Thanks,, this is fun, > > Mike > > > Michael Booth > michael.booth at sun.com > mobile 512-289-3805 > From acuselton at lbl.gov Mon Mar 16 20:41:40 2009 From: acuselton at lbl.gov (Andrew C. Uselton) Date: Mon, 16 Mar 2009 13:41:40 -0700 Subject: [Lustre-devel] protocol backofs In-Reply-To: <029901c9a66b$d7107020$85315060$@com> References: <49BEA192.2050701@lbl.gov> <029901c9a66b$d7107020$85315060$@com> Message-ID: <49BEB984.5030206@lbl.gov> Howdy Isaac, Nice to meet you. As Eric suggested I am also cc:ing Nick Henke, since he might find this an interesting discussion. For all you lustre-devel dwellers out there, feel free to chime in. I have been running a few tests on the Franklin Cray XT at NERSC and also on Jaguar (Cray XT at ORNL) and on Jacquard (Opteron/Infiniband w/GPFS at NERSC). You can see a lot of what I have done here: http://www.nersc.gov/~uselton/ipm-io.html In particular, this link shows something of interest: http://www.nersc.gov/~uselton/frank_jag/ These tests use Madbench, which has a somewhat unusual I/O pattern. It is implementing an out-of-core solution to a series of very large matrix operations. The third row of graphs gives an idea of the aggregate I/O emerging from the application over the course of the run. It has a pattern of writes, reads and writes, then reads. Each of the I/O spikes is from every task writing or reading a single 300 MB buffer. The last row of graphs gives a sense of the task by task behavior. The "frank_jag" page shows data collected during 4 test with 256 tasks (4 tasks per node on 64 nodes). The target is a single file striped across all OSTs of the Lustre file system. Two tests are on Franklin and two on Jaguar. Each machine runs a test using the POSIX I/O interface and another using the MPI-I/O interface. In the third column the Franklin, MPI-I/O test has extremely long delays in the reads in the middle phase, but not during the other reads or any of the writes. This does not happen for POSIX, nor does it happen for Jaguar using MPI-I/O. The results shown are entirely reproducible and not due to interference from other jobs on the system. The only difference between the Franklin and Jaguar configurations is that Jaguar has 144 OSTs on 72 OSSs instead of 80 OSTs on 20 OSSs. Eric put the notion in my head that that we may be looking at a contention issue in the Sea-Star network. Since the I/O is being necked down to 20 OSSs in the case of Franklin, this seems plausible. If you guys have a moment to consider the subject I'd like to think about: a) Why would contention introduce the catastrophic delays rather than just slow things down generally and more or less evenly? Is there some form of back-off in the protocol(s) that could occasionally get kicked up to tens of seconds? b) Why is the contention introduced only in the MPI-I/O test and not in the POSIX test? Does the MPI-I/O from Cray's xt-mpt/3.1.0 divert I/O to a subset of nodes so that all the I/O is going through a smaller section of the torus? If I have been too terse in this note feel free to ask questions and I'll try to add more detail. Cheers, Andrew From robl at mcs.anl.gov Mon Mar 16 22:13:18 2009 From: robl at mcs.anl.gov (Robert Latham) Date: Mon, 16 Mar 2009 17:13:18 -0500 Subject: [Lustre-devel] protocol backofs In-Reply-To: <49BEB984.5030206@lbl.gov> References: <49BEA192.2050701@lbl.gov> <029901c9a66b$d7107020$85315060$@com> <49BEB984.5030206@lbl.gov> Message-ID: <20090316221316.GM1408@mcs.anl.gov> On Mon, Mar 16, 2009 at 01:41:40PM -0700, Andrew C. Uselton wrote: > Howdy Isaac, > Nice to meet you. As Eric suggested I am also cc:ing Nick Henke, > since he might find this an interesting discussion. For all you > lustre-devel dwellers out there, feel free to chime in. Hi Andrew. Yes, there is no way to avoid me... I don't have too much information about Lustre but I can tell you a bit about Madbench and MPI-IO. > b) Why is the contention introduced only in the MPI-I/O test and not in > the POSIX test? Does the MPI-I/O from Cray's xt-mpt/3.1.0 divert I/O to > a subset of nodes so that all the I/O is going through a smaller section > of the torus? Cray's MPI-IO is old enough that it's doing "generic unix" file system operations. (I've committed the optimized Lustre driver, but it will take some time for it to end up on a Cray). Madbench is doing independent I/O, though, so optimized or no, there is no "aggregation" -- it's a shame, too, as it sounds like aggregation would at least rule out your contention theory. You've essentially written this up on your website already, but for the wider lustre-devel audience, The MPI-IO in Madbench is dead simple: MPI_File_seek MPI_File_read or MPI_File_write (or the nonblocking versions) MPI_Barrier This is *almost* an exact correspondance to the POSIX case: fseeko64 fread or fwrite fclose Did you see the difference? I know you did because you wrote http://www.nersc.gov/~uselton/sf-mpi.html How big is an individual madbench I/O operation for you? We ran some I/O tests with madbench on our bluegene that showed about 20 MB per operation -- large enough that i'd be surprised if the libc buffering was having much effect. So, off the top of my head I don't have too many ideas from an MPI-IO perspective. Your graphs suggest irregular performance on franklin for both reads and writes (http://www.nersc.gov/~uselton/frank_jag/20090215183709/rate.png), so that kind of rules out interference from the lock manager. to me, your contention idea is still in play. ==rob -- Rob Latham Mathematics and Computer Science Division A215 0178 EA2D B059 8CDF Argonne National Lab, IL USA B29D F333 664A 4280 315B From acuselton at lbl.gov Mon Mar 16 22:41:44 2009 From: acuselton at lbl.gov (Andrew C. Uselton) Date: Mon, 16 Mar 2009 15:41:44 -0700 Subject: [Lustre-devel] protocol backofs In-Reply-To: <20090316221316.GM1408@mcs.anl.gov> References: <49BEA192.2050701@lbl.gov> <029901c9a66b$d7107020$85315060$@com> <49BEB984.5030206@lbl.gov> <20090316221316.GM1408@mcs.anl.gov> Message-ID: <49BED5A8.4060703@lbl.gov> Robert Latham wrote: > On Mon, Mar 16, 2009 at 01:41:40PM -0700, Andrew C. Uselton wrote: >> Howdy Isaac, ... > > Hi Andrew. Yes, there is no way to avoid me... I don't have too much > information about Lustre but I can tell you a bit about Madbench and > MPI-IO. > Glad to hear from you :) ... > Cray's MPI-IO is old enough that it's doing "generic unix" file system > operations. (I've committed the optimized Lustre driver, but it will > take some time for it to end up on a Cray). > I am looking over David Knaak's shoulder even as we speak (electron?). > Madbench is doing independent I/O, though, so optimized or no, there > is no "aggregation" -- it's a shame, too, as it sounds like > aggregation would at least rule out your contention theory. When you say "independent" you mean it isn't using MPI "collective" I/O, yes? That is true, just making sure I understand your comment. > > How big is an individual madbench I/O operation for you? We ran some I usually run madbench "as large as possible". That ends up with the target buffer for I/O in the 300 MB range. > > So, off the top of my head I don't have too many ideas from an MPI-IO > perspective. Your graphs suggest irregular performance on franklin > for both reads and writes > (http://www.nersc.gov/~uselton/frank_jag/20090215183709/rate.png), so > that kind of rules out interference from the lock manager. There is some variability in the writes (and reads in other tests), but the MPI-I/O, middle-phase reads seem to be a special case. Those delays are an order of magnitude higher and do not seem to correspond to any I/O activity. That's why I'm hoping for a protocol backoff induced by congestion. Also note that in that phase, and only in that phase, each node has been given 1.2 GB to send to the file and immediately asked to read that much back in from a different offset. I've looked quite carefully and none of the I/O is outside its locked range as established in the first "writes" phase, so there should be no lock traffic during this phase. So in this middle phase there may be extra resource contention in kernel space on each node. So an alternative might be a low-probability near-deadlock on those resources where writes are still being drained but reads are already demanding attention. > > to me, your contention idea is still in play. > > ==rob > I think I forgot to mention: NERSC is soon planning to extend the Franklin I/O resources so they look a lot more like Jaguar's. When they do we'll be able to "do the experiment", in that if the delay disappears that argues for contention in the torus getting to the OSSs or in the OSSs themselves. I'm still stumped for why it would only happen in the MPI-I/O case, though. Cheers, Andrew From He.Huang at Sun.COM Tue Mar 17 15:28:44 2009 From: He.Huang at Sun.COM (Isaac Huang) Date: Tue, 17 Mar 2009 11:28:44 -0400 Subject: [Lustre-devel] protocol backofs In-Reply-To: <49BEB984.5030206@lbl.gov> References: <49BEA192.2050701@lbl.gov> <029901c9a66b$d7107020$85315060$@com> <49BEB984.5030206@lbl.gov> Message-ID: <20090317152844.GG17185@sun.com> On Mon, Mar 16, 2009 at 01:41:40PM -0700, Andrew C. Uselton wrote: > Howdy Isaac, > Nice to meet you. As Eric suggested I am also cc:ing Nick Henke, > since he might find this an interesting discussion. For all you > lustre-devel dwellers out there, feel free to chime in. Hello Andrew, please see my comments inline. > ...... > The "frank_jag" page shows data collected during 4 test with 256 tasks > (4 tasks per node on 64 nodes). The target is a single file striped > across all OSTs of the Lustre file system. Two tests are on Franklin > and two on Jaguar. Each machine runs a test using the POSIX I/O > interface and another using the MPI-I/O interface. In the third column > the Franklin, MPI-I/O test has extremely long delays in the reads in the > middle phase, but not during the other reads or any of the writes. This I've got zero knowledge on MPI-IO. Could you please elaborate for a bit on how this "delays in the reads" are measured and what "the middle phase" is? > does not happen for POSIX, nor does it happen for Jaguar using MPI-I/O. > The results shown are entirely reproducible and not due to interference > from other jobs on the system. The only difference between the Franklin > and Jaguar configurations is that Jaguar has 144 OSTs on 72 OSSs instead > of 80 OSTs on 20 OSSs. Not sure about Franklin, but on Jaguar, depending on the file-system in use, the OSSs could reside in either the Sea-Star network or an IB network (accessed via lnet routers). I think it might be worthwhile to double check what server network had been used. > Eric put the notion in my head that that we may be looking at a > contention issue in the Sea-Star network. Since the I/O is being necked > down to 20 OSSs in the case of Franklin, this seems plausible. If you > guys have a moment to consider the subject I'd like to think about: > a) Why would contention introduce the catastrophic delays rather than > just slow things down generally and more or less evenly? Is there some > form of back-off in the protocol(s) that could occasionally get kicked > up to tens of seconds? It involves many layers: 1. At Lustre/PTLRPC layer, there is a limit on the number of in-flight RPCs to a server. This is end-to-end, and the limit could change at runtime. 2. At lnet/lnd layer, for ptllnd and o2iblnd, there's a credit-based mechanism to prevent a sending node from overrunning buffers at the remote end. This is not end-to-end, and the number of pre-granted credits doesn't change over runtime. 3. Cray Portals and the Sea-Star network runs beneath lnet/ptllnd, and I'd think that there could also be some similar mechanisms. Thanks, Isaac From He.Huang at Sun.COM Tue Mar 17 18:13:29 2009 From: He.Huang at Sun.COM (Isaac Huang) Date: Tue, 17 Mar 2009 14:13:29 -0400 Subject: [Lustre-devel] protocol backofs In-Reply-To: <49BEB984.5030206@lbl.gov> References: <49BEA192.2050701@lbl.gov> <029901c9a66b$d7107020$85315060$@com> <49BEB984.5030206@lbl.gov> Message-ID: <20090317181329.GO17185@sun.com> On Mon, Mar 16, 2009 at 01:41:40PM -0700, Andrew C. Uselton wrote: > ...... > The "frank_jag" page shows data collected during 4 test with 256 tasks > (4 tasks per node on 64 nodes). The target is a single file striped > across all OSTs of the Lustre file system. Two tests are on Franklin > and two on Jaguar. Each machine runs a test using the POSIX I/O > interface and another using the MPI-I/O interface. In the third column > the Franklin, MPI-I/O test has extremely long delays in the reads in the > middle phase, but not during the other reads or any of the writes. This > does not happen for POSIX, nor does it happen for Jaguar using MPI-I/O. > The results shown are entirely reproducible and not due to interference > from other jobs on the system. The only difference between the Franklin > and Jaguar configurations is that Jaguar has 144 OSTs on 72 OSSs instead > of 80 OSTs on 20 OSSs. I just happened to have a talk with an ORNL folk and was told that, when compared with the other Cray XT system, it's relatively easier to hit congestion in Sea-Star network on Jaguar where the servers are less distributed with regard to the network topology. So I wonder whether there could be a similar difference between Franklin and Jaguar? On the other hand, were the POSIX test and the MPI-IO test on Franklin run over the same set of client nodes? Thanks, Isaac From pauln at psc.edu Tue Mar 17 19:50:52 2009 From: pauln at psc.edu (Paul Nowoczynski) Date: Tue, 17 Mar 2009 15:50:52 -0400 Subject: [Lustre-devel] Deadlock in usocklnd Message-ID: <49BFFF1C.8080203@psc.edu> Hi, I've got some code which uses ptlrpc and lnet and a few weeks ago we integrated usocklnd into our environment. This lnd has been working quite well and seems very efficient. The only issue so far is that we're hitting a clear case of deadlock when trying to handle a failed connection. I'm not sure if this code is actually in use by any lustre component as of yet but I thought it best to bring this to your attention. paul (gdb) bt #0 0x00002b3872336888 in __lll_mutex_lock_wait () from /lib64/libpthread.so.0 #1 0x00002b38723328a5 in _L_mutex_lock_107 () from /lib64/libpthread.so.0 #2 0x00002b3872332333 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x000000000049c9f5 in usocklnd_find_or_create_conn (peer=0x2695270, type=0, connp=0x42803d30, tx=0x2aaad46f5d10, zc_ack=0x0, send_immediately=0x42803d2c) at ..//../..//lnet-lite/ulnds/socklnd/conn.c:841 #4 0x00000000004a2701 in usocklnd_send (ni=0x25e1810, private=0x0, lntmsg=0x2aaacca1ace0) at ..//../..//lnet-lite/ulnds/socklnd/usocklnd_cb.c:124 #5 0x000000000048d886 in lnet_ni_send (ni=0x25e1810, msg=0x2aaacca1ace0) at ..//../..//lnet-lite/lnet/lib-move.c:863 #6 0x000000000048dd20 in lnet_post_send_locked (msg=0x2aaacca1ace0, do_send=1) at ..//../..//lnet-lite/lnet/lib-move.c:951 #7 0x000000000048dfb4 in lnet_return_credits_locked (msg=0x2aaacca1ae40) at ..//../..//lnet-lite/lnet/lib-move.c:1108 #8 0x0000000000494230 in lnet_complete_msg_locked (msg=0x2aaacca1ae40) at ..//../..//lnet-lite/lnet/lib-msg.c:143 #9 0x00000000004944f5 in lnet_finalize (ni=0x25e1810, msg=0x2aaacca1ae40, status=-5) at ..//../..//lnet-lite/lnet/lib-msg.c:242 #10 0x000000000049bff8 in usocklnd_destroy_tx (ni=0x25e1810, tx=0x26950c0) at ..//../..//lnet-lite/ulnds/socklnd/conn.c:562 #11 0x000000000049c02d in usocklnd_destroy_txlist (ni=0x25e1810, txlist=0x2693650) at ..//../..//lnet-lite/ulnds/socklnd/conn.c:574 #12 0x000000000049b09c in usocklnd_tear_peer_conn (conn=0x2692550) at ..//../..//lnet-lite/ulnds/socklnd/conn.c:148 #13 0x000000000049f5c8 in usocklnd_process_stale_list (pt_data=0x25e1c98) at ..//../..//lnet-lite/ulnds/socklnd/poll.c:112 #14 0x000000000049f7b2 in usocklnd_poll_thread (arg=0x25e1c98) at ..//../..//lnet-lite/ulnds/socklnd/poll.c:169 #15 0x0000000000450871 in psc_usklndthr_begin (arg=0x25e4370) at ..//../..//psc_fsutil_libs/psc_util/usklndthr.c:16 #16 0x000000000044f000 in _pscthr_begin (arg=0x7fff389980b0) at ..//../..//psc_fsutil_libs/psc_util/thread.c:237 #17 0x00002b38723302f7 in start_thread () from /lib64/libpthread.so.0 #18 0x00002b387281fe3d in clone () from /lib64/libc.so.6 (gdb) up #4 0x00000000004a2701 in usocklnd_send (ni=0x25e1810, private=0x0, lntmsg=0x2aaacca1ace0) at ..//../..//lnet-lite/ulnds/socklnd/usocklnd_cb.c:124 124 rc = usocklnd_find_or_create_conn(peer, type, &conn, tx, NULL, (gdb) print *peer $4 = {up_list = {next = 0x6e1b08, prev = 0x6e1b08}, up_peerid = {nid = 562954416842067, pid = 2147520945}, up_conns = {0x2692550, 0x0, 0x0}, up_ni = 0x25e1810, up_incarnation = 0, up_incrn_is_set = 0, up_refcount = {counter = 3}, up_lock = {__data = {__lock = 2, __count = 0, __owner = 23283, __nusers = 1, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = "\002\000\000\000\000\000\000\000�Z\000\000\001", '\0' , __align = 2}, up_errored = 0, up_last_alive = 0} (gdb) ## We're waiting for a lock which we already hold! * 6 Thread 1115703616 (LWP 23283) 0x00002b3872336888 in __lll_mutex_lock_wait () from /lib64/libpthread.so.0 From He.Huang at Sun.COM Tue Mar 17 21:17:28 2009 From: He.Huang at Sun.COM (Isaac Huang) Date: Tue, 17 Mar 2009 17:17:28 -0400 Subject: [Lustre-devel] Deadlock in usocklnd In-Reply-To: <49BFFF1C.8080203@psc.edu> References: <49BFFF1C.8080203@psc.edu> Message-ID: <20090317211728.GV17185@sun.com> On Tue, Mar 17, 2009 at 03:50:52PM -0400, Paul Nowoczynski wrote: > Hi, > I've got some code which uses ptlrpc and lnet and a few weeks ago we > integrated usocklnd into our environment. This lnd has been working > quite well and seems very efficient. The only issue so far is that we're > hitting a clear case of deadlock when trying to handle a failed > connection. I'm not sure if this code is actually in use by any lustre > component as of yet but I thought it best to bring this to your attention. > paul The usocklnd was completely rewritten some time ago, and the new code hasn't been used at any Lustre major site as far as I know. It appeared to me that you were using the new usocklnd. It seemed to me that the deadlock had been triggered like: 1. usocklnd_poll_thread -> usocklnd_process_stale_list -> usocklnd_tear_peer_conn: acquired &peer->up_lock. 2. Then with &peer->up_lock locked, usocklnd_tear_peer_conn -> usocklnd_destroy_txlist -> usocklnd_destroy_tx -> lnet_finalize -> lnet_complete_msg_locked -> lnet_return_credits_locked -> lnet_post_send_locked -> lnet_ni_send -> usocklnd_send -> usocklnd_find_or_create_conn 3. And usocklnd_find_or_create_conn would try to acquire the same &peer->up_lock again. I'm not 100% certain about the sequence above, but generally it should be considered unsafe for a LND to call into LNet with a LND lock held since LNet would likely call back into the LND again. Paul, please file a bug and assign it to lustre-spider-team at sun.com directly. Thanks, Isaac > (gdb) bt > #0 0x00002b3872336888 in __lll_mutex_lock_wait () from > /lib64/libpthread.so.0 > #1 0x00002b38723328a5 in _L_mutex_lock_107 () from /lib64/libpthread.so.0 > #2 0x00002b3872332333 in pthread_mutex_lock () from /lib64/libpthread.so.0 > #3 0x000000000049c9f5 in usocklnd_find_or_create_conn (peer=0x2695270, > type=0, connp=0x42803d30, tx=0x2aaad46f5d10, zc_ack=0x0, > send_immediately=0x42803d2c) at > ..//../..//lnet-lite/ulnds/socklnd/conn.c:841 > #4 0x00000000004a2701 in usocklnd_send (ni=0x25e1810, private=0x0, > lntmsg=0x2aaacca1ace0) at > ..//../..//lnet-lite/ulnds/socklnd/usocklnd_cb.c:124 > #5 0x000000000048d886 in lnet_ni_send (ni=0x25e1810, msg=0x2aaacca1ace0) > at ..//../..//lnet-lite/lnet/lib-move.c:863 > #6 0x000000000048dd20 in lnet_post_send_locked (msg=0x2aaacca1ace0, > do_send=1) at ..//../..//lnet-lite/lnet/lib-move.c:951 > #7 0x000000000048dfb4 in lnet_return_credits_locked (msg=0x2aaacca1ae40) > at ..//../..//lnet-lite/lnet/lib-move.c:1108 > #8 0x0000000000494230 in lnet_complete_msg_locked (msg=0x2aaacca1ae40) > at ..//../..//lnet-lite/lnet/lib-msg.c:143 > #9 0x00000000004944f5 in lnet_finalize (ni=0x25e1810, > msg=0x2aaacca1ae40, status=-5) at ..//../..//lnet-lite/lnet/lib-msg.c:242 > #10 0x000000000049bff8 in usocklnd_destroy_tx (ni=0x25e1810, > tx=0x26950c0) at ..//../..//lnet-lite/ulnds/socklnd/conn.c:562 > #11 0x000000000049c02d in usocklnd_destroy_txlist (ni=0x25e1810, > txlist=0x2693650) at ..//../..//lnet-lite/ulnds/socklnd/conn.c:574 > #12 0x000000000049b09c in usocklnd_tear_peer_conn (conn=0x2692550) at > ..//../..//lnet-lite/ulnds/socklnd/conn.c:148 > #13 0x000000000049f5c8 in usocklnd_process_stale_list > (pt_data=0x25e1c98) at ..//../..//lnet-lite/ulnds/socklnd/poll.c:112 > #14 0x000000000049f7b2 in usocklnd_poll_thread (arg=0x25e1c98) at > ..//../..//lnet-lite/ulnds/socklnd/poll.c:169 > #15 0x0000000000450871 in psc_usklndthr_begin (arg=0x25e4370) at > ..//../..//psc_fsutil_libs/psc_util/usklndthr.c:16 > #16 0x000000000044f000 in _pscthr_begin (arg=0x7fff389980b0) at > ..//../..//psc_fsutil_libs/psc_util/thread.c:237 > #17 0x00002b38723302f7 in start_thread () from /lib64/libpthread.so.0 > #18 0x00002b387281fe3d in clone () from /lib64/libc.so.6 > (gdb) up > #4 0x00000000004a2701 in usocklnd_send (ni=0x25e1810, private=0x0, > lntmsg=0x2aaacca1ace0) at > ..//../..//lnet-lite/ulnds/socklnd/usocklnd_cb.c:124 > 124 rc = usocklnd_find_or_create_conn(peer, type, &conn, tx, NULL, > (gdb) print *peer > $4 = {up_list = {next = 0x6e1b08, prev = 0x6e1b08}, up_peerid = {nid = > 562954416842067, pid = 2147520945}, up_conns = {0x2692550, 0x0, 0x0}, > up_ni = 0x25e1810, up_incarnation = 0, up_incrn_is_set = 0, up_refcount > = {counter = 3}, up_lock = {__data = {__lock = 2, __count = 0, __owner = > 23283, > __nusers = 1, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = > 0x0}}, > __size = "\002\000\000\000\000\000\000\000???Z\000\000\001", '\0' 26 times>, __align = 2}, up_errored = 0, up_last_alive = 0} > (gdb) > > ## We're waiting for a lock which we already hold! > * 6 Thread 1115703616 (LWP 23283) 0x00002b3872336888 in > __lll_mutex_lock_wait () from /lib64/libpthread.so.0 > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel From pauln at psc.edu Tue Mar 17 21:31:37 2009 From: pauln at psc.edu (Paul Nowoczynski) Date: Tue, 17 Mar 2009 17:31:37 -0400 Subject: [Lustre-devel] Deadlock in usocklnd In-Reply-To: <20090317211728.GV17185@sun.com> References: <49BFFF1C.8080203@psc.edu> <20090317211728.GV17185@sun.com> Message-ID: <49C016B9.9030600@psc.edu> Isaac, Your description accurately describes what I'm seeing. I'll file a bug. thanks, paul Isaac Huang wrote: > On Tue, Mar 17, 2009 at 03:50:52PM -0400, Paul Nowoczynski wrote: > >> Hi, >> I've got some code which uses ptlrpc and lnet and a few weeks ago we >> integrated usocklnd into our environment. This lnd has been working >> quite well and seems very efficient. The only issue so far is that we're >> hitting a clear case of deadlock when trying to handle a failed >> connection. I'm not sure if this code is actually in use by any lustre >> component as of yet but I thought it best to bring this to your attention. >> paul >> > > The usocklnd was completely rewritten some time ago, and the new code > hasn't been used at any Lustre major site as far as I know. It appeared > to me that you were using the new usocklnd. > > It seemed to me that the deadlock had been triggered like: > 1. usocklnd_poll_thread -> usocklnd_process_stale_list -> > usocklnd_tear_peer_conn: acquired &peer->up_lock. > 2. Then with &peer->up_lock locked, usocklnd_tear_peer_conn -> > usocklnd_destroy_txlist -> usocklnd_destroy_tx -> lnet_finalize -> > lnet_complete_msg_locked -> lnet_return_credits_locked -> > lnet_post_send_locked -> lnet_ni_send -> usocklnd_send -> > usocklnd_find_or_create_conn > 3. And usocklnd_find_or_create_conn would try to acquire the same > &peer->up_lock again. > > I'm not 100% certain about the sequence above, but generally it should > be considered unsafe for a LND to call into LNet with a LND lock held > since LNet would likely call back into the LND again. > > Paul, please file a bug and assign it to lustre-spider-team at sun.com > directly. > > Thanks, > Isaac > > >> (gdb) bt >> #0 0x00002b3872336888 in __lll_mutex_lock_wait () from >> /lib64/libpthread.so.0 >> #1 0x00002b38723328a5 in _L_mutex_lock_107 () from /lib64/libpthread.so.0 >> #2 0x00002b3872332333 in pthread_mutex_lock () from /lib64/libpthread.so.0 >> #3 0x000000000049c9f5 in usocklnd_find_or_create_conn (peer=0x2695270, >> type=0, connp=0x42803d30, tx=0x2aaad46f5d10, zc_ack=0x0, >> send_immediately=0x42803d2c) at >> ..//../..//lnet-lite/ulnds/socklnd/conn.c:841 >> #4 0x00000000004a2701 in usocklnd_send (ni=0x25e1810, private=0x0, >> lntmsg=0x2aaacca1ace0) at >> ..//../..//lnet-lite/ulnds/socklnd/usocklnd_cb.c:124 >> #5 0x000000000048d886 in lnet_ni_send (ni=0x25e1810, msg=0x2aaacca1ace0) >> at ..//../..//lnet-lite/lnet/lib-move.c:863 >> #6 0x000000000048dd20 in lnet_post_send_locked (msg=0x2aaacca1ace0, >> do_send=1) at ..//../..//lnet-lite/lnet/lib-move.c:951 >> #7 0x000000000048dfb4 in lnet_return_credits_locked (msg=0x2aaacca1ae40) >> at ..//../..//lnet-lite/lnet/lib-move.c:1108 >> #8 0x0000000000494230 in lnet_complete_msg_locked (msg=0x2aaacca1ae40) >> at ..//../..//lnet-lite/lnet/lib-msg.c:143 >> #9 0x00000000004944f5 in lnet_finalize (ni=0x25e1810, >> msg=0x2aaacca1ae40, status=-5) at ..//../..//lnet-lite/lnet/lib-msg.c:242 >> #10 0x000000000049bff8 in usocklnd_destroy_tx (ni=0x25e1810, >> tx=0x26950c0) at ..//../..//lnet-lite/ulnds/socklnd/conn.c:562 >> #11 0x000000000049c02d in usocklnd_destroy_txlist (ni=0x25e1810, >> txlist=0x2693650) at ..//../..//lnet-lite/ulnds/socklnd/conn.c:574 >> #12 0x000000000049b09c in usocklnd_tear_peer_conn (conn=0x2692550) at >> ..//../..//lnet-lite/ulnds/socklnd/conn.c:148 >> #13 0x000000000049f5c8 in usocklnd_process_stale_list >> (pt_data=0x25e1c98) at ..//../..//lnet-lite/ulnds/socklnd/poll.c:112 >> #14 0x000000000049f7b2 in usocklnd_poll_thread (arg=0x25e1c98) at >> ..//../..//lnet-lite/ulnds/socklnd/poll.c:169 >> #15 0x0000000000450871 in psc_usklndthr_begin (arg=0x25e4370) at >> ..//../..//psc_fsutil_libs/psc_util/usklndthr.c:16 >> #16 0x000000000044f000 in _pscthr_begin (arg=0x7fff389980b0) at >> ..//../..//psc_fsutil_libs/psc_util/thread.c:237 >> #17 0x00002b38723302f7 in start_thread () from /lib64/libpthread.so.0 >> #18 0x00002b387281fe3d in clone () from /lib64/libc.so.6 >> (gdb) up >> #4 0x00000000004a2701 in usocklnd_send (ni=0x25e1810, private=0x0, >> lntmsg=0x2aaacca1ace0) at >> ..//../..//lnet-lite/ulnds/socklnd/usocklnd_cb.c:124 >> 124 rc = usocklnd_find_or_create_conn(peer, type, &conn, tx, NULL, >> (gdb) print *peer >> $4 = {up_list = {next = 0x6e1b08, prev = 0x6e1b08}, up_peerid = {nid = >> 562954416842067, pid = 2147520945}, up_conns = {0x2692550, 0x0, 0x0}, >> up_ni = 0x25e1810, up_incarnation = 0, up_incrn_is_set = 0, up_refcount >> = {counter = 3}, up_lock = {__data = {__lock = 2, __count = 0, __owner = >> 23283, >> __nusers = 1, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = >> 0x0}}, >> __size = "\002\000\000\000\000\000\000\000???Z\000\000\001", '\0' > 26 times>, __align = 2}, up_errored = 0, up_last_alive = 0} >> (gdb) >> >> ## We're waiting for a lock which we already hold! >> * 6 Thread 1115703616 (LWP 23283) 0x00002b3872336888 in >> __lll_mutex_lock_wait () from /lib64/libpthread.so.0 >> >> _______________________________________________ >> Lustre-devel mailing list >> Lustre-devel at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-devel >> From acuselton at lbl.gov Tue Mar 17 21:45:59 2009 From: acuselton at lbl.gov (Andrew C. Uselton) Date: Tue, 17 Mar 2009 14:45:59 -0700 Subject: [Lustre-devel] protocol backofs In-Reply-To: <20090317152844.GG17185@sun.com> References: <49BEA192.2050701@lbl.gov> <029901c9a66b$d7107020$85315060$@com> <49BEB984.5030206@lbl.gov> <20090317152844.GG17185@sun.com> Message-ID: <49C01A17.6070108@lbl.gov> Isaac Huang wrote: > On Mon, Mar 16, 2009 at 01:41:40PM -0700, Andrew C. Uselton wrote: >> Howdy Isaac, ... > Hello Andrew, please see my comments inline. > >> ...... >> The "frank_jag" page shows data collected during 4 test with 256 tasks >> (4 tasks per node on 64 nodes). The target is a single file striped >> across all OSTs of the Lustre file system. Two tests are on Franklin >> and two on Jaguar. Each machine runs a test using the POSIX I/O >> interface and another using the MPI-I/O interface. In the third column >> the Franklin, MPI-I/O test has extremely long delays in the reads in the >> middle phase, but not during the other reads or any of the writes. This > > I've got zero knowledge on MPI-IO. Could you please elaborate for a > bit on how this "delays in the reads" are measured and what "the > middle phase" is? > All discussion is related to figures in: http://www.nersc.gov/~uselton/frank_jag/ The application in question is MADbench. I can send a reference or two if you want detail on how MADbench works. In short it is an MPI application that solves a very large matrix problem with an out-of-core algorithm. That is, It works on a matrix problem that fills all the memory on all the nodes, 64 nodes/256 tasks in this case. It must write out intermediate results and the read them back in. As such, every task must execute a write of 300 MB at each step in "phase 1". In our example phase 1 has eight steps, so eight 300 MB writes from each of 256 tasks. In "phase 2", each of the eight matrices must be read in turn, a result calculated, and the result written out - for(i=0;i<8i++){read(300 MB); compute(); write(300 MB);}. In "phase 3" the eight results are again read back in and a final value calculated. So the reads in the middle phase take a long time when using an MPI-I/O interface and a single-file I/O model. If you follow along in the graphs you should be able ot pick out the above actions and see where the slow reads are. The data for identifying this behavior comes from augmenting the application with the "Integrated Performance Monitoring" library (IPM). That tool provides an event trace across the whole application of library call, result, and timeing information. Whith that one may reconstruct the trace graphs see in the web page. Other interesting manipulations of that data also appear, for instance a histogram of frequency of occurence versus bandwidth exibited by individual I/Os. > > Not sure about Franklin, but on Jaguar, depending on the file-system in > use, the OSSs could reside in either the Sea-Star network or an IB > network (accessed via lnet routers). I think it might be worthwhile to > double check what server network had been used. > I was using /lustre/scr144 on Jaguar. I believe that is SeaStar. > > It involves many layers: > 1. At Lustre/PTLRPC layer, there is a limit on the number of in-flight > RPCs to a server. This is end-to-end, and the limit could change at > runtime. The amount of I/O (1.2 GB per node, per step) is large enough I'd assume we hit steady state in the RPC mechanism. Most of the time all available system "cache" is full and RPCs are being issued as quickly as they can be completed. > 2. At lnet/lnd layer, for ptllnd and o2iblnd, there's a credit-based > mechanism to prevent a sending node from overrunning buffers at the > remote end. This is not end-to-end, and the number of pre-granted > credits doesn't change over runtime. I am only vaguely familiar with the credit mechanism. That would be relevant for the writes, yes? Is it possible to exhaust the available credits and get blocked trying to clear "cache" such that the reads (which got started after) can't complete until the writes are drained from "cache". that would certain address why the delays only occur in the read,write,read,write... (middle) phase. > 3. Cray Portals and the Sea-Star network runs beneath lnet/ptllnd, > and I'd think that there could also be some similar mechanisms. Yes, I'm shopping for an understanding of how things can get bogged down this way, and why it only appears to happen for MPI-I/O not POSIX. > > Thanks, > Isaac Your follow-up note about congestion is consistent with Eric's comment. It may be that the cross-section bandwidth to the region with the OSSs is not high enough to forestall congestion. This could be worse on Franklin (20 OSSs) than on Jaguar (72 OSSs) even if Jaguar does have a problem with it. Cheers, Andrew From Oleg.Drokin at Sun.COM Wed Mar 18 20:31:50 2009 From: Oleg.Drokin at Sun.COM (Oleg Drokin) Date: Wed, 18 Mar 2009 16:31:50 -0400 Subject: [Lustre-devel] Oleg/Mike Work on Apps Metrics - FW: Mike Booth week ending 2009.03.15 In-Reply-To: <023201c9a636$a987e4b0$fc97ae10$@com> References: <015c01c9a5a7$cb2f83a0$618e8ae0$@com> <52E982EB-3ADE-46FA-A7BC-D1645961DF0B@Sun.COM> <018101c9a5b6$25dbc300$71934900$@com> <48A6CF99-F630-4581-9FFC-05E3AFF66FA0@Sun.COM> <023201c9a636$a987e4b0$fc97ae10$@com> Message-ID: <5CE22E46-771C-47EE-9CAC-26A37C87C466@sun.com> Hello! On Mar 16, 2009, at 8:56 AM, Eric Barton wrote: > I _do_ agree that for some apps, if there was sufficient memory on the > app node to buffer the local component of a checkpoint and let it > "dribble" out to disk would achieve better utilization of the compute > resource. However parallel apps can be very sensitive to "noise" on > the network they're using for inter- process communication - i.e. the > checkpoint data has either to be written all the way to disk, or at > least buffered somewhere so that moving it to disk will not interfere > with the app's own communications. > This latter concept is the basis for the "flash cache" concept. > Actually, I think it's worth exploring the economics of it in more > detail. This turns out to be a very true assertion. We (I) do see a huge delay in e.g. MPI barriers done immediately after write. > The variables are aggregate network bandwidth into the distributed > checkpoint cache, which determines the checkpoint time, and aggregate > path-minimum bandwidth (i.e. lesser of network and disk bandwidth) > from the cache to disk, which determines how soon the cache can be > ready for the next checkpoint. The cache could be dedicated nodes and > storage (e.g. flash) or additional storage on the OSSes, or any > combination of either. And the interesting relationship is how > compute cluster utilisation varies with the cost of the server and > cache subsystems. The thing is, if we can just flush out data from the cache at the moment when there is no network-latency critical activity on the app side (somehow signaled by the app), why would we need the flash storage at all? We can write nice sequential chunks to normal disks just as fast, I presume. It is the random i/o patterns that make flash shine. Bye, Oleg From Vitaly.Fertman at Sun.COM Thu Mar 19 19:34:55 2009 From: Vitaly.Fertman at Sun.COM (Vitaly Fertman) Date: Thu, 19 Mar 2009 22:34:55 +0300 Subject: [Lustre-devel] LustreFS performance (update) In-Reply-To: <02FEAA2B-8D98-4C2D-9CE8-FF6E1EB135A2@sun.com> References: <3376C558-E29A-4BB5-8C4C-3E8F4537A195@sun.com> <02FEAA2B-8D98-4C2D-9CE8-FF6E1EB135A2@sun.com> Message-ID: **************************************************** LustreFS benchmarking methodology. **************************************************** The document aims to describe the benchmarking methodology which helps to understand the LustreFS performance and reveal LustreFS bottlenecks in different configurations on different hardware, to ensure the next LustreFS release does not downgrade comparing with a previous one. In other words: Goal1. Understand the HEAD performance. Goal2. Compare HEAD and b1_6 (b1_8) performance. To achieve the Goal1, the methodology suggests to test different layers of software in the bottom-top direction, i.e. the underlying back-end, the target server sitting on this back-end, the network connected to this target and how the target performs through this network, etc up to the whole cluster. Each next step has only 1 change over the previous one, it is either a new layer added or 1 parameter in the configuration is changed (probably another network type or another back-end). Comparing the results of each test with the previous test, we get the overhead of the added layer or the performance impact of changing this parameter. To achieve the Goal2, the methodology suggests to go in the reverse top-bottom direction, i.e. to test some large sub-systems first and, if a downgrade vs. a previous LustreFS version is detected, to perform more detailed tests. (This is considered as the primary goal of the 2.0 Performance Team). The document does not cover the way of fixing revealed problems, probably some special purpose test needs to be run or oprofile needs to be compiled in -- it is our of scope of the document. Obviously, it is not possible to perform all the thousands of tests in all the configurations, running all the special purpose tests, etc, the document tries to prepare: 1) all the essential and sufficient tests to see how the system performs in general; 2) some minimal amount of essential tests to see how the system scales in different conditions. Therefore, the plan does not guarantee we will not miss a bottleneck or a bug, it just tries to cover maximum possible scenarios in most interesting conditions/environment states. The amount of tests described below is already about 2K, and there will be definitely more, and it will take a lot of time to perform all of them and to analyze the results. So one of the major concerns here is how to minimize the amount of test so that we would not miss some interesting case and would be able to get all the results within a reasonable amount of time. Please keep it in mind while looking at the tests below. **** Hardware Requirements. **** The test plan implies that we change only 1 parameter (cpu or disk or network) on each step. The HW requirements are: -- at least 1 node with: CPU:32; RAM: enough to have a ramdisk for MDS; DISK: enough disks for raid6 or raid1+0 (as this node could be mds or ost); an extra disk for external journal; NET: both GiGe and IB installed. -- at least 1 another node includes: DISK: enough disks for raid6 or raid1+0 (as this node could be mds or ost); an extra disk for external journal; -- besides that: 8 clients, 3 other servers. -- the other servers include: DISK: raid6 NET: IB installed. -- client includes: NET: both GiGe and IB installed. **** Software requirements **** 1. Short term. 1.1 mdsrate to be completed to test all the operations listed in MDST3 (see below). 1.2 mdsrate-**.sh to be fixed/written to run mdsrate properly and test all the operations listed in MDST3 (see below). 1.3. fake disk implement FAIL flag and report 'done' without doing anything in obdfilter to get a low-latency disk. 1.4. MT. add more tests here and implement them. 2. Long term. 2.1. mdtstack-survey - an echo client-server is to be written for mds similar to ost. - a test script similar to obdfilter-survey.sh is to be written. **** Different configurations **** Configuration of Node: RAM. Amount of RAM on nodes (?) CPU. Count of CPUs on nodes (1..32) DISK. Disk type (raid, ramdisk, fake) JOUR. Journal type (internal, external, ram) Q: which raid? A: use RAID 1+0 for MDS; RAID6 for OST. fake: to get a low-latency disk, it is preferable to report 'done' without doing anything in obdfilter once some FAIL flag is set. It is useful for OST testing, because first of all, it does not have a CPU overhead of memcpy of using ramdisk and it lets to test large amount of data in contrast to ramdisk. As a drawback, it skips the localfs code paths. Note: OSS back-end has write through cache; MDS back-end has write- back cache. Configuration of Cluster: CL. Amount of clients (1,2,4,8) MDS. Amount of MDS nodes (1,2,4). OSS. Amount of OSS nodes (1,2,4) NET. Network type (GiGe, IB) OSTN. Amount of OST per nodes (1,2,4) Configuration of test. TH. Amount of threads per client (1,2,4,8) VER. Lustre version (b1_6, HEAD. later b1_8). FEAT. Lustre features to turn off (COS, SA, RA, debug messages) TEST. Specific test parameters. **** Testing **** Low Layers Testing (LLT) LLT1. Raw disk (lustre-iokit:sgpdd-survey) LLT2. Local filesystem (lustre-iokit: ior-survey, is fs mounted synchronously?) Network Testing (NETT). NETT1. lnet: lnetself test. NETT2. OBD: lustre-iokit: (obdfilter-survey, echo_client-osc-..- net-..-ost-echo_server) NETT3. MD: (not ready) OST Testing (OSTT). OSTT1. Isolated OST (lustre-iokit: obdfilter-survey, echo_client- obdfilter-..-disk) OSTT2. Remote OST (lustre-iokit: obdfilter-survey, echo_client-osc-..- ost-obdfilter-..-disk) OSTT3. Client-OST IO (lustre-iokit: ior-survey, client-ost-disk). MDS Testing (MDST). MDST1. Isolated MDS test (not ready) MDST2. Remote MDS test (not ready) MDST3. Simple Client-MDS operation test Mixed testing (MT) (not ready) **** Statistics **** During all the tests the following is supposed to be running on all the servers: 1) HP collectl or LLNL's LMT; 2) smth else? *** Goal1. Understand the HEAD performance. *** The Goal1 describes the testing methodology in the bottom-top direction, from the lower layers (disk) to the complete Lustre cluster. LLT1. Raw disk (lustre-iokit:sgpdd-survey) RAM: fixed CPU: 1 DISK: raid,ramdisk,fake (default=raid) JOUR:- CL: 1 OSS:1 NET: - OSTN:- TH: 1,2,4,8 (default=1) F: debug TEST: *)bulk size is specified as rszlo/rszhi=[1,4,64,1024K] *)TH is specified as thrlo/thrhi=[1,2,4,8] *)the amount of objects to work on in parallel: crglo=crghi=[1;TH] i.e. test only cases when all the threads work on the same file and when all of them work on a separate file. TEST=[bulk;separate or commin file]=8 tests; Test matrix(TESTxTHxDISK): Run TESTs with different amount of threads for each DISK. TESTxTHxDISK=(8x4 - 1)x3=93 tests. "-1" because TH=1 is already covered. Total:93 tests. *** NETT1. lnetself test.*** RAM: fixed CPU: 1,8,32 (default=1) DISK: - JOUR:- CL: 1,8 (default=1) OSS:1 NET: GiGe, IB (default=IB) OSTN:1 TH: 1,2,4,8 (default=1) F: debug TEST: *) test type: PING,READ,WRITE tests *) bulk size for READ/WRITE: 1k,4k,64k,1M [1 ping + 4 reads + 4 writes] = 9 tests Test matrix (TESTxCLxTHxNETxCPU): 1. Multi-thread test. Run TESTs on CL=1 with different amount of threads. TESTxTH=[1+4+4]x4=36 tests. 2. Multi-client test Note: to be more demonstrative, the maximum amount of threads could be taken <8, if TH=8 reaches the maximum network throughput with small amount of clients. TESTxCLxTH=9x1x4=36 tests. 3. Network test As the nature of IB is different from GiGe, we need to repeat all the tests from (1,2) here. 36+36=72 tests. 4. CPU test Note: lnet fixes from Liang to be applied here. Run TESTs on different amount of CPU. It is mostly interesting to look at large amount of threads, as we are going to benefit from handling them in parallel. At the same time, if some HW (network) limit is reached, the result will not be very demonstrative, so test with 1 small & 1 large bulk size only:[1k; 1024K]: [CL=8,TH=1,2,4,8]. TESTxCLxTHxCPU=5x1x4x(3-1)=40. Total: 184 tests. *** NETT2. OBD performance *** lustre-iokit: obdfilter-survey, case=network. The results of this tests are to be compared with lnet results to get the osc+ost+ptlrpc overhead. RAM: fixed CPU: 1 DISK: - JOUR:- CL: 1,8 (default=1) OSS:1 NET: IB OSTN:1 TH: 1,2,4,8 (default=1) F: debug TEST: *) bulk size: rszlo=rszhi=N (1,4,64,1024) *) TH is specified through: thrlo=1, thrhi=8 (thread count, 1,2,4,8) *) the amount of objects is: nobjlo=nobjhi=[1;TH] i.e. test only cases when all the threads work on the same file and when all of them work on a separate file. TEST=[4 bulks; common or separate file]=8 tests Test matrix(TESTxTHxCL): 1. Multi-thread test. Run TESTs on CL=1 with different amount of threads. TESTxTH=8x4=32 tests 2. Multi-client test Note: to be more demonstrative, the maximum amount of threads should be taken <8, if TH=8 reaches the maximum network throughput with small amount of clients. TESTxCLxTH=8x1x4=32 tests. 3. Network test. Having IB results in hand after (1,2) and these results from NETT1, we already see how osc+ost+ptlrpc changes the behavior. There is no reason to repeat them for GiGe, it seems. 4.CPU test Note: lnet fixes from Liang to be applied here. Run TESTs on different amount of CPU. It is mostly interesting to look at large amount of threads, as we are going to benefit from handling them in parallel. At the same time, if some HW (network) limit is reached, the result will not be very demonstrative, so test with 1 small & 1 large bulk size only:[1k; 1024K]: [CL=8,TH=1,2,4,8]. TESTxCLxTHxCPU=4x1x4x(4-1)=48. Total: 112 tests. *** OSTT1. Isolated OST *** lustre-iokit: obdfilter-survey, case=disk The results of this tests are to be compared with LLT results to get the OST stack overhead. RAM: fixed CPU: 1,8,32 (default=1) DISK: raid, fake (default=fake) JOUR: int, ext, ram, (default=ext) CL: 1 OSS:1 NET: - OSTN:1,2,4 (default=1) TH: 1,2,4,8 (default=1) F: debug TEST: *) bulk size: rszlo=rszhi=N (1,4,64,1024K) *) TH is specified through: thrlo=1, thrhi=8 (1,2,4,8) *) each OST is supposed to be configured on a separate disk. *) the amount of objects is: nobjlo=nobjhi=[1;TH] i.e. test only cases when all the threads work on the same file and when all of them work on a separate file. TEST=[4 bulks; common of separate file]=8 tests Test matrix(TESTxTHxOSTNxDISKxJOURxCPU): 1. Multi-thread test. Run TESTs on OSTN=1 with different amount of threads. TESTxTH=8x4=32 tests 2. Multi-OST test 2.1. Let's check how OSTs vs. threads per OST scale (TH=OSTN). 2.2. Let's check how the system scale with many OSTs and threads (TH=8*OSTN). [OSTN>1;TH=OSTN,8*OSTN]. TESTxOSTNxTH=8x2x2=32 tests. 3. DISK test As other disks are completely different, so lets repeat most of the (1,2) for 2 others: [TH=OSTN;8*OSTN]: TESTxOSTNxTHxDISK=8x3x2x1=48 4. JOURNAL test. Limit the tests with only raid-disk. Limit the test with only 1 large and 1 small bulk:[1,1024K]. TESTxOSTNxTHxJOUR: 4x3x2x2=48 5. CPU test Note: lnet fixes from Liang to be applied here. Run TESTs on different amount of CPU. It is better to perform it on a fast backend (DISK=fake) to see how CPU really matters. It is mostly interesting to look at large amount of threads, as we are going to benefit from handling them in parallel. Also, run with a small & a large bulk only:[1,1024K] [OSTN=4,TH=1,2,4,8]: TESTxOSTNxTHxCPU=4x1x4x2=32 Total: 192 tests. *** OSTT2. Real OST test *** lustre-iokit: obdfilter-survey, case=netdisk This test is a composition of OBD performance and Isolated OST tests, so its results are to be compared with NETT2 & OSTT1 results. RAM: fixed CPU: 1,8,32 (default=1) DISK: fake JOUR: ext CL: 1,8 (default=1) OSS:1,2,4 (default=1) NET: IB OSTN:1,2,4 (default=1) TH: 1,2,4,8 (default=1) F: debug TEST: *) bulk size: rszlo=rszhi=N (1,4,64,1024) *) TH is specified through: thrlo=1, thrhi=8 (thread count, 1,2,4,8) *) each OST is supposed to be configured on a separate disk. *) the amount of objects is: nobjlo=nobjhi=[1;TH] i.e. test only cases when all the threads work on the same file and when all of them work on a separate file. TEST=[4 bulks; common of separate file]=8 tests Test matrix(TESTxTHxCLxCPUxOSSxOSTN): 1. Multi-thread test. Run TESTs on CL=1 with different amount of threads. TESTxTH=8x4=32 tests 2. Multi-client test Note: to be more demonstrative, the maximum amount of threads could be taken <8, if TH=8 reaches the maximum network throughput with small amount of clients. TESTxCLxTH=8x1x4=32 tests. 3. Network test Having IB results in hand after (1,2) and these results from NETT2, we already see how osc+ost+ptlrpc+obdfilter changes the behavior. Thus, there is no reason to repeat them for GiGe, it seems. 4.CPU test Note: lnet fixes from Liang to be applied here. Run TESTs on different amount of CPU. It is mostly interesting to look at large amount of threads, as we are going to benefit from handling them in parallel. At the same time, if some HW (network) limit is reached, the result will not be very demonstrative, so test with 1 small & 1 large bulk size only:[1k; 1024K]: [CL=8,TH=1,2,4,8]. TESTxCLxTHxCPU=4x1x4x(3-1)=32. 5. OSTN test. The same OSC, network, CPU, disk, just check how OST stack (see 1,2 tests) is scalable. 5.1. Let's check how N threads per 1 OST vs. 1 thread per N OST scales (CL=OSTN). 5.2. Let's check how the system scale with many clients and threads (CL=8) Note: to be more demonstrative, the maximum amount of threads could be taken <8, if TH=8 reaches the maximum network throughput with small amount of clients. As the different with (1,2) on the OSS part only, it is enough to test in separate directories only. It seems enough to look at 1 small & 1 large bulk only: [1,1024K] [CL=OSTN,8;TH=1,8]. TEST=2. TESTxCLxTHxOSTN=2x2x2x2=16 tests 6. OSS test. 6.1. Let's check how 1 thread per N OST vs. 1 thread per N OSS scales (CL=OSS). 6.2. Let's check how the system scale with many clients and threads (CL=8) Note: to be more demonstrative, the maximum amount of threads could be taken <8, if TH=8 reaches the maximum network throughput with small amount of clients. As the different with (1,2) on the OSS part only, it is enough to test in separate directories only. It seems enough to look at 1 small & 1 large bulk only: [1,1024K] [CL=OSS,8;TH=1,8]. TEST=2. TESTxCLxTHxOSTN=2x2x2x2=16 tests Total:128 tests *** OSTT3. Client-OST test *** lustre-iokit: ior-survey. The test results are to be compared with OSTT2 results to get the overhead for Lustre Client: client stack, distributed locking, etc. RAM: fixed CPU: 1,8,32 (default=1) DISK: fake JOUR: ext CL: 1,8 (default=1) OSS:1,2,4 (default=1) NET: IB OSTN:1,2,4 (default=1) TH: 1,2,4,8 (default=1) F: debug TEST: *) CL is specified through $clients_hi *) TH is specified through $tasks_per_client_hi *) bulk is specified through rsize_lo/hi (1,4,64,1028K) *) file_per_task=[0;1] i.e. test only cases when all the threads work on the same file and when all of them work on a separate file. TEST=[4 bulks; common of separate file]=8 tests Test matrix(TESTxTHxCLxCPUxOSSxOSTN): absolutely the same as for OSTT2. NETT3. MD: (not ready) MDST1. Isolated MDS (not ready) MDST2. Remote MDS (not ready) This set of tests need to be implemented in a utility similar to obdfilter-survey but for MDS testing. MDST3. Simple Client-MDS operation tests 1. create,mknod,mkdir (symlink, link??) RAM: fixed CPU(MDS): 1,8,32 (default=1) DISK(MDS): ramdisk, raid (default=ramdisk) DISK(OST): raid JOUR(MDS): int,ext,ram (default=ext) CL: 1,8 (default=1) MDS:1,2,4 (default=1) OSS:1,2,4 (default=1) NET: IB OSTN:1 TH: 1,2,4,8 (default=1) F: debug TEST: it will be probably mdsrate/mdsrate-create-small.sh, but it needs to be fixed to support all of these operations, not only create. If so: *) TH could be specified through THREADS_PER_CLIENT=[1,2,4,8] *) CL is specified through CLIENTS or NODES_TO_USE. *) NOSINGLE should be provided *) add --dirnum option to COMMAND *) DIRNUM=[1,TH*CL], so we test a case when all the threads work in the same dir and when each works in a separate one. *) nfiles is files-per-dir * DIRNUM [common or separate dir]=2tests; Note: we should probably limit the amount of files in 1 directory with 2M, otherwise the performance will definitely downgrade. Test matrix(TESTxTHxCLxCPUxMDSxOSSxDISKxJOUR): 1. Multi-thread test. (mknod) Run TESTs on CL=1 with different amount of threads. TESTxTH=2x4-1=7 tests (not 8 as if TH=1, DIRNUM=1, and this is already covered). 2. Multi-client test (mknod) Note: to be more demonstrative, the maximum amount of threads could be taken <8, if TH=8 reaches the maximum network throughput with small amount of clients. TESTxCLxTH=2x1x4=8 tests. 3. OSS (create) 3.1. Let's check how multi-client system scales (TH=OSS). 3.2. Let's check how large load system scales (TH=8) As the different with (1,2) on the OSS part only, it is enough to test in separate directories only. Stripeness is [1, -1]. TEST=2. TESTxCLxTHxOSS=[2x2x2]x2 + [2x2x2]x1(1OSS case)=24 4. Network test Having IB results in hand after (1,2,3) and these results from NETT1, we already see how mdc+mdt-stack+ptlrpc changes the behavior. There is no reason to repeat them for GiGe, it seems. 5. DISK test. (mknod) Unlike the OST testing, we do not have echo-md client (MDTT1), thus we have not checked how different disks impact the performance, so we need to check it here. As difference disks are of completely different nature we need to repeat most of (1,2) here [TH=1,8]: TESTxCLxTHxDISK=(2x2x2-1)x1=7 6. JOURNAL test. (mknod) Repeat (5) for different journals, but limit the test with raid-disk only. TESTxCLxTHxDISKxJOUR=(2x2x2-1)x1x2=15 7.CPU test (mknod) Note: lnet fixes from Liang to be applied here. Run TESTs on different amount of CPU. It is mostly interesting to look at large amount of threads, as we are going to benefit from handling them in parallel, so run it for CL=8 only TESTxCLxTHxCPU=2x1x4x(3-1)=16 8. CMD test. (mkdir) 8.1. Let's check how N threads per 1 MDS vs. 1 thread per N MDS scales (CL=MDS). 8.2. Let's check how the system scale with many clients and threads (CL=8) Note: to be more demonstrative, the maximum amount of threads could be taken <8, if TH=8 reaches the maximum network throughput with small amount of clients These test happens in a separate directory (for each thread) only, enough to test with nid creation policy only. TEST=1. TESTxCLxTHxMDS=1x2x4x2=16 tests. Total: 16 tests for mkdir, 53 for mknod, 24 for create. 2. stat RAM: fixed CPU(MDS): 1,8,32 (default=1) DISK(MDS): ramdisk DISK(OST): raid JOUR: ext CL: 1,8 (default=1) MDS: 1,2,4 (default=1) OSS:1,2,4 (default=1) NET: IB OSTN:1 TH: 1,2,4,8 (default=1) F: debug TEST: mdsrate/mdsrate-stat-small.sh *) add THREADS_PER_CLIENT to the script to specify TH *) CL is specified through CLIENTS or NODES_TO_USE. *) NOSINGLE should be provided *) add --dirnum option to COMMAND *) DIRNUM=[1,TH*CL], so we test a case when all the threads work in the same dir and when each works in a separate one. *) nfiles is files-per-dir * DIRNUM *) add READDIR_ORDER to test readdir access order (random order is not very interesting for stat). [common or separate dir; readdir order]=2 tests. Test matrix(TESTxTHxCLxCPUxMDSxOSS): 1. Multi-thread test. Run TESTs on CL=1 with different amount of threads. TESTxTH=2x4-1=7 tests (not 16 as if TH=1, DIRNUM=1, and this is already covered). 2. Multi-client test Note: to be more demonstrative, the maximum amount of threads could be taken <8, if TH=8 reaches the maximum network throughput with small amount of clients. TESTxCLxTH=2x1x4=8 tests. 3. OSS. 3.1. Let's check how multi-client system scales (TH=OSS). 3.2. Let's check how large load system scales (TH=8) As the difference with (1,2) on the OSS part only, it is enough to test in separate directories only. Test must be done for create with different stripeness: [1, -1]. TEST=2. TESTxCLxTHxOSS=[2x2x2]x2=16 4.CPU test Note: lnet fixes from Liang to be applied here. Run TESTs on different amount of CPU. It is mostly interesting to look at large amount of threads, as we are going to benefit from handling them in parallel, so run it for CL=8 only. TESTxCLxTHxCPU=2x1x4x(3-1)=16 5. CMD test. (mkdir) 5.1. Let's check how N threads per 1 MDS vs. 1 thread per N MDS scales (CL=MDS). 5.2. Let's check how the system scale with many clients and threads (CL=8) Note: to be more demonstrative, the maximum amount of threads could be taken <8, if TH=8 reaches the maximum network throughput with small amount of clients. 1 creation policy (nid) is enough: TESTxCLxTHxMDS=2x2x4x2=32 tests. Total: 79 tests. 3. unlink (mdsrate-create-small.sh) RAM: fixed CPU(MDS): 1,8,32 (default=1) DISK(MDS): ramdisk, raid (default=ramdisk) DISK(OST): raid JOUR(MDS): int,ext,ram (default=ext) CL: 1,8 (default=1) MDS:1,2,4 (default=1) OSS:1,2,4 (default=1) NET: IB OSTN:1 TH: 1,2,4,8 (default=1) F: debug TEST: it will be probably mdsrate/mdsrate-create-small.sh, but it needs to be fixed to support all of these operations, not only create. If so: *) TH could be specified through THREADS_PER_CLIENT=[1,2,4,8] *) CL is specified through CLIENTS or NODES_TO_USE. *) NOSINGLE should be provided *) add --dirnum option to COMMAND *) DIRNUM=[1,TH*CL], so we test a case when all the threads work in the same dir and when each works in a separate one. *) nfiles is files-per-dir * DIRNUM *) add an ability to remove in readdir order to mdsrate test and its script. [readdir or _create_ order; common or separate dir]=3 (skip readdir/ common dir). Note: we should probably limit the amount of files in 1 directory with 2M, otherwise the performance will definitely downgrade. Test matrix(TESTxTHxCLxCPUxMDSxOSSxDISKxJOUR): 1. Multi-thread test. (mknod) Run TESTs on CL=1 with different amount of threads. TESTxTH=3x4-2=10 tests 2. Multi-client test (mknod) Note: to be more demonstrative, the maximum amount of threads could be taken <8, if TH=8 reaches the maximum network throughput with small amount of clients. TESTxCLxTH=3x1x4=12 tests. 3. OSS (create) 3.1. Let's check how multi-client system scales (TH=OSS). 3.2. Let's check how large load system scales (TH=8) As the difference with (1,2) on the OSS part only, it is enough to test in separate directories only. Stripeness is [1, -1]. TEST=4. TESTxCLxTHxOSS=[4x2x2]x2 + [4x2x2]x1(1OSS case)=48 4. Network test Having IB results in hand after (1,2,3) and these results from NETT1, we already see how mdc+mdt-stack+ptlrpc changes the behavior. There is no reason to repeat them for GiGe, it seems. 5. DISK test. (mknod) Unlike the OST testing, we do not have echo-md client (MDTT1), thus we have not checked how different disks impact the performance, so we need to check it here. As different disks are of completely different nature we need to repeat most of (1,2) here [TH=1,8]: TESTxCLxTHxDISK=(3x2x2-2)x1=10 6. JOURNAL test. (mknod) Repeat (5) for different journals, but limit the test with raid-disk only. TESTxCLxTHxDISKxJOUR=(3x2x2-2)x2=20 7.CPU test (mknod) Note: lnet fixes from Liang to be applied here. Run TESTs on different amount of CPU. It is mostly interesting to look at large amount of threads, as we are going to benefit from handling them in parallel, so run it for CL=8 only TESTxCLxTHxCPU=3x1x4x(3-1)=24 8. CMD test. (mkdir) 8.1. Let's check how N threads per 1 MDS vs. 1 thread per N MDS scales (CL=MDS). 8.2. Let's check how the system scale with many clients and threads (CL=8) Note: to be more demonstrative, the maximum amount of threads could be taken <8, if TH=8 reaches the maximum network throughput with small amount of clients These test happens in a separate directory (for each thread) only, enough to test with nid creation policy only. TEST=2. TESTxCLxTHxMDS=2x2x4x2=32 tests. Total: 32 tests for mkdir, 76 for mknod, 48 for create. 4. find (not ready) **** MT. Mixed testing. **** MT1. Create-write test. RAM: fixed CPU(MDS): 1,8,32 (default=32) DISK(MDS): ramdisk, raid (default=ramdisk) DISK(OST): raid JOUR: ext CL: 1,8 (default=1) MDS: 1,2,4 (default=1) OSS:1 NET: IB OSTN:1 TH: 1,2,4,8 (default=1) F: debug TEST: must be a new one. Each thread creates files in a loop, writes 1 bulk to each and closes it. *) it is enough to test with a small bulk only: [1k] *) [common or separate dir]=2tests; Test matrix(TESTxTHxCLxCPUxMDSxDISK): 1. Multi-thread test. Run TESTs on CL=1 with different amount of threads. TESTxTH=2x4-1=7 tests (not 8 as if TH=1, it is always in 1 dir, and this is already covered). 2. Multi-client test Note: to be more demonstrative, the maximum amount of threads could be taken <8, if TH=8 reaches the maximum network throughput with small amount of clients. TESTxCLxTH=2x1x4=8 tests. 3. DISK test. Check how different disks impact on the performance. As different disks are of completely different nature we need to repeat most of (1,2) here [TH=1,8]: TESTxCLxTHxDISK=(2x2x2-1)x1=7 4.CPU test Note: lnet fixes from Liang to be applied here. Run TESTs on different amount of CPU. It is mostly interesting to look at large amount of threads, as we are going to benefit from handling them in parallel, so run it for CL=max only: [CL=8,TH=1,2,4,8]. TESTxCLxTHxCPU=2x1x4x(4-1)=24 5. CMD test. 5.1. Let's check how N threads per 1 MDS vs. 1 thread per N MDS scales (CL=MDS). 5.2. Let's check how the system scale with many clients and threads (CL=8) Note: to be more demonstrative, the maximum amount of threads could be taken <8, if TH=8 reaches the maximum network throughput with small amount of clients These test happens in a separate directory (for each thread) only, creation policy=[nid,name]. TEST=2. TESTxCLxTHxMDS=2x2x4x2=32 tests. Total: 78 tests. MT2. Create-Readdir test. RAM: fixed CPU(MDS): 1,8,32 (default=32) DISK(MDS): ramdisk, raid (default=ramdisk) DISK(OST): raid JOUR: ext CL: 1,8 (default=1) (1 extra client does "ls -U") MDS:1,2,4 (default=1) OSS:1 NET: IB OSTN:1 TH: 1,2,4,8 (default=1) F: debug TEST: must be a new one. Each thread creates files in a loop and immediately closes them. 1 thread on another client does "ls -U". It is done in 1 directory. The test matrix is exactly the same as for MT1. Total: 78 tests. MT3. untar a kernel. MT4. pmake (compile a kernel). RAM: fixed CPU(MDS): 1,8,32 (default=32) DISK(MDS): ramdisk, raid (default=ramdisk) DISK(OST): raid JOUR: ext CL: 1 MDS:1,2,4 (default=1) OSS:1,2,4 (default=1) NET: IB OSTN:1 TH: 1 F: debug TEST: a new one. Test matrix(TESTxCPUxMDSxOSSxDISK): 1. DISK test. Check how different disks impact on the performance. TESTxDISK=1 2.CPU test Note: lnet fixes from Liang to be applied here. Run TESTs on different amount of CPU. It is mostly interesting to look at large amount of threads, as we are going to benefit from handling them in parallel, so run it for CL=max only: TESTxCPU=1x(3-1)=2 3. CMD test. Creation policy=name. TESTxMDS=1x2=2 tests. 4. OSS As most of the files are small, stripeness does not play any role (=1) TESTxOSS=1x(3-1)=2. Total: 7 tests. MT5. ??? Some more tests ???? **** Goal2. Compare HEAD and b1_6 (b1_8) performance. **** This paragraph describes the testing methodology in the reverse order of testing, i.e. in the top-bottom direction, making sure new LustreFS (HEAD) version does not downgrade comparing with the previous ones (b1_6/b1_8). Therefore, the first testing cycle includes: 1) MT, MDST3, OSTT3, NETT1 from the above tests. 2) no CMD tests In the case a downgrade is detected, lower layer tests are to be run until the downgrade disappear. **** Goal3. CMD testing. **** MT, MDST3 tests, their CMD sections. **** Goal4. Quick weekly MD performance test. **** 1) It covers tests described in MT,MDST sections. 2) MDST: No CPU,OSS,OSTN tests 3) MT: no MT1,MT2 tests 4) Only 1 node configuration: MDS on RAID1+0 with write back cache OSS on RAID6 with write through cache JOUR: external for both servers; 5) Only 1 network: IB; 6) Minimal amount of cluster configurations: MDS=1; OST=1; [CL,TH]=[1,1],[1,8],[8,8]; MDST1.1: perform only create (not mkdir,mknod) for [common or separate dir]=2. 1. Multi-thread test. TESTxTH=2x2-1=3 tests 2. Multi-client test. TESTxCLxTH=2x1x1=2 tests. Total: 5 tests. MDST1.2. stat for [common or separate dir; readdir order]=2 tests. 1. Multi-thread test. TESTxTH=2x2-1=3 tests 2. Multi-client test. TESTxCLxTH=2x1x1=2 tests. Total: 5 tests. MDST1.3 unlink for [readdir or _create_ order; common or separate dir]=3 (skip readdir/common dir). All tests are done against create (not mkdir,mknod). 1. Multi-thread test. TESTxTH=3x2-2=4 tests 2. Multi-client test (mknod) TESTxCLxTH=3x1x1=3 tests. Total: 7 tests. MT3. untar a kernel. MT4. pmake (compile a kernel). Total: 1 tests. Total: 19 tests. -- Vitaly From acuselton at lbl.gov Thu Mar 19 20:16:09 2009 From: acuselton at lbl.gov (Andrew C. Uselton) Date: Thu, 19 Mar 2009 13:16:09 -0700 Subject: [Lustre-devel] LustreFS performance (update) In-Reply-To: References: <3376C558-E29A-4BB5-8C4C-3E8F4537A195@sun.com> <02FEAA2B-8D98-4C2D-9CE8-FF6E1EB135A2@sun.com> Message-ID: <49C2A809.7030900@lbl.gov> Howdy Vitaly, I like this. It is quite comprehensive and detailed. I'd like to offer a few constructive criticisms in hope that you will better achieve your goals. Mostly I'll stick them in-line where they seem relevant, but I'll start with: 1) Your write up is quite dense and terse. I could follow the overall structure, but found it pretty tough going to understand any specific detail. It really helps to work with someone who will write up the same information, but in a form with whole sentences and a minimum of acronyms or special symbols. Define the acronyms you do use in a clear way in one place that I can refer back to. Vitaly Fertman wrote: > **************************************************** > LustreFS benchmarking methodology. > **************************************************** > > The document aims to describe the benchmarking methodology which helps > to understand the LustreFS performance and reveal LustreFS bottlenecks > in > different configurations on different hardware, to ensure the next > LustreFS > release does not downgrade comparing with a previous one. In other > words: > Goal1. Understand the HEAD performance. > Goal2. Compare HEAD and b1_6 (b1_8) performance. > > To achieve the Goal1, the methodology suggests to test different > layers of > software in the bottom-top direction, i.e. the underlying back-end, > the target > server sitting on this back-end, the network connected to this target > and how > the target performs through this network, etc up to the whole cluster. I like this approach. My own efforts tend to be at-scale testing at the whole-cluster end of the range, often in the presence of other cluster activity. It is good to have the details of the underlying components documented. ... > Obviously, it is not possible to perform all the thousands of tests in > all the configurations, > running all the special purpose tests, etc, the document tries to > prepare: > 1) all the essential and sufficient tests to see how the system > performs in general; > 2) some minimal amount of essential tests to see how the system scales > in different > conditions. In some cases it's obvious, but in many it is not clear what exactly you mean to be testing. It is a good extension to your methodology to state clearly not only the mechanics of the test itself, but what you think you are testing with the given experiment. Spend a little time and describe what the system is under examination, how it responds or should respond to the proposed test, and what tunables and parameters you think might be relevant. For instance, if the test is supposed to saturate the target server, then how much I/O do you expect will be required and why? What timeout or other tunable may determine the observed saturation point. Your goal should be to have, not only a test, but a real expectation about its results even before you run the test. Once you have that expectation then you can evaluate the results. The bottom up approach helps with this, since you can use the performance of the individual pieces to help establish your expectation about the larger assemblies. ... > **** Hardware Requirements. **** > > The test plan implies that we change only 1 parameter (cpu or disk or > network) > on each step. The HW requirements are: > > -- at least 1 node with: > CPU:32; > RAM: enough to have a ramdisk for MDS; > DISK: enough disks for raid6 or raid1+0 (as this node could be mds > or ost); > an extra disk for external journal; > NET: both GiGe and IB installed. > -- at least 1 another node includes: > DISK: enough disks for raid6 or raid1+0 (as this node could be mds > or ost); > an extra disk for external journal; > -- besides that: 8 clients, 3 other servers. > -- the other servers include: > DISK: raid6 > NET: IB installed. > -- client includes: > NET: both GiGe and IB installed. > > **** Software requirements **** > You might provide links to these tests for those not familiar with them. > 1. Short term. > 1.1 mdsrate > to be completed to test all the operations listed in MDST3 (see below). > 1.2 mdsrate-**.sh > to be fixed/written to run mdsrate properly and test all the > operations listed in > MDST3 (see below). > 1.3. fake disk > implement FAIL flag and report 'done' without doing anything in > obdfilter to get > a low-latency disk. > 1.4. MT. > add more tests here and implement them. > > 2. Long term. > 2.1. mdtstack-survey > - an echo client-server is to be written for mds similar to ost. > - a test script similar to obdfilter-survey.sh is to be written. > > **** Different configurations **** > ... I'll cut it short here, but in general, I think you might be surprised that if you organize this document so that anyone else could come along behind you and perform all the same tests in the same way, you might get a lot of others doing these experiments along side you. That would make your job a lot easier and increase the likelihood that bugs and regressions would be caught quickly. > -- > Vitaly > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel Cheers, Andrew From Vitaly.Fertman at Sun.COM Fri Mar 20 13:15:56 2009 From: Vitaly.Fertman at Sun.COM (Vitaly Fertman) Date: Fri, 20 Mar 2009 16:15:56 +0300 Subject: [Lustre-devel] LustreFS performance (update) In-Reply-To: <49C2A809.7030900@lbl.gov> References: <3376C558-E29A-4BB5-8C4C-3E8F4537A195@sun.com> <02FEAA2B-8D98-4C2D-9CE8-FF6E1EB135A2@sun.com> <49C2A809.7030900@lbl.gov> Message-ID: <96F3B9B8-65F6-412F-8F81-241101293D63@Sun.COM> Hi Andrew, thanks for you feedback, indeed, this still looks more like a raw test list than a ready for publishing document, but this is a continuous work and I am still working on it, so I will try to address you suggestions. On Mar 19, 2009, at 11:16 PM, Andrew C. Uselton wrote: > Howdy Vitaly, > I like this. It is quite comprehensive and detailed. I'd like to > offer a few constructive criticisms in hope that you will better > achieve your goals. Mostly I'll stick them in-line where they seem > relevant, but I'll start with: > 1) Your write up is quite dense and terse. I could follow the > overall structure, but found it pretty tough going to understand any > specific detail. It really helps to work with someone who will > write up the same information, but in a form with whole sentences > and a minimum of acronyms or special symbols. Define the acronyms > you do use in a clear way in one place that I can refer back to. > > > Vitaly Fertman wrote: >> **************************************************** >> LustreFS benchmarking methodology. >> **************************************************** >> The document aims to describe the benchmarking methodology which >> helps >> to understand the LustreFS performance and reveal LustreFS >> bottlenecks in >> different configurations on different hardware, to ensure the next >> LustreFS >> release does not downgrade comparing with a previous one. In other >> words: >> Goal1. Understand the HEAD performance. >> Goal2. Compare HEAD and b1_6 (b1_8) performance. >> To achieve the Goal1, the methodology suggests to test different >> layers of >> software in the bottom-top direction, i.e. the underlying back- >> end, the target >> server sitting on this back-end, the network connected to this >> target and how >> the target performs through this network, etc up to the whole >> cluster. > > I like this approach. My own efforts tend to be at-scale testing at > the whole-cluster end of the range, often in the presence of other > cluster activity. It is good to have the details of the underlying > components documented. > > ... >> Obviously, it is not possible to perform all the thousands of tests >> in all the configurations, >> running all the special purpose tests, etc, the document tries to >> prepare: >> 1) all the essential and sufficient tests to see how the system >> performs in general; >> 2) some minimal amount of essential tests to see how the system >> scales in different >> conditions. > > In some cases it's obvious, but in many it is not clear what exactly > you mean to be testing. It is a good extension to your methodology > to state clearly not only the mechanics of the test itself, but what > you think you are testing with the given experiment. Spend a little > time and describe what the system is under examination, how it > responds or should respond to the proposed test, and what tunables > and parameters you think might be relevant. For instance, if the > test is supposed to saturate the target server, then how much I/O do > you expect will be required and why? What timeout or other tunable > may determine the observed saturation point. Your goal should be to > have, not only a test, but a real expectation about its results even > before you run the test. Once you have that expectation then you > can evaluate the results. The bottom up approach helps with this, > since you can use the performance of the individual pieces to help > establish your expectation about the larger assemblies. > > ... >> **** Hardware Requirements. **** >> The test plan implies that we change only 1 parameter (cpu or disk >> or network) >> on each step. The HW requirements are: >> -- at least 1 node with: >> CPU:32; >> RAM: enough to have a ramdisk for MDS; >> DISK: enough disks for raid6 or raid1+0 (as this node could be >> mds or ost); >> an extra disk for external journal; >> NET: both GiGe and IB installed. >> -- at least 1 another node includes: >> DISK: enough disks for raid6 or raid1+0 (as this node could be >> mds or ost); >> an extra disk for external journal; >> -- besides that: 8 clients, 3 other servers. >> -- the other servers include: >> DISK: raid6 >> NET: IB installed. >> -- client includes: >> NET: both GiGe and IB installed. >> **** Software requirements **** > You might provide links to these tests for those not familiar with > them. >> 1. Short term. >> 1.1 mdsrate >> to be completed to test all the operations listed in MDST3 (see >> below). >> 1.2 mdsrate-**.sh >> to be fixed/written to run mdsrate properly and test all the >> operations listed in >> MDST3 (see below). >> 1.3. fake disk >> implement FAIL flag and report 'done' without doing anything in >> obdfilter to get >> a low-latency disk. >> 1.4. MT. >> add more tests here and implement them. >> 2. Long term. >> 2.1. mdtstack-survey >> - an echo client-server is to be written for mds similar to ost. >> - a test script similar to obdfilter-survey.sh is to be written. >> **** Different configurations **** > ... > > I'll cut it short here, but in general, I think you might be > surprised that if you organize this document so that anyone else > could come along behind you and perform all the same tests in the > same way, you might get a lot of others doing these experiments > along side you. That would make your job a lot easier and increase > the likelihood that bugs and regressions would be caught quickly. > >> -- >> Vitaly >> _______________________________________________ >> Lustre-devel mailing list >> Lustre-devel at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-devel > > Cheers, > Andrew -- Vitaly From Parinay.Kondekar at Sun.COM Fri Mar 20 05:47:10 2009 From: Parinay.Kondekar at Sun.COM (parinay kondekar) Date: Fri, 20 Mar 2009 11:17:10 +0530 Subject: [Lustre-devel] LustreFS performance (update) In-Reply-To: References: <3376C558-E29A-4BB5-8C4C-3E8F4537A195@sun.com> <02FEAA2B-8D98-4C2D-9CE8-FF6E1EB135A2@sun.com> Message-ID: <49C32DDE.1030705@sun.com> The wiki :: https://wikis.clusterfs.com/intra/index.php/LustreFS_performance ~p Vitaly Fertman wrote: > **************************************************** > LustreFS benchmarking methodology. > **************************************************** > > From rkrishnakumar at gmail.com Sat Mar 21 07:49:24 2009 From: rkrishnakumar at gmail.com (Krishnakumar R) Date: Sat, 21 Mar 2009 13:19:24 +0530 Subject: [Lustre-devel] Regarding Server Driven QOS for lustre Message-ID: <89c400ad0903210049o489ba04ax38f9198d57543e9a@mail.gmail.com> Hi Folks, What is the current status of the LRS and EH implementation on Lustre (mentioned in the mail: http://groups.google.com/group/lustre-devel-list/msg/c921d5b7121c50eb) ? Any further documents/links on the server driven QOS on lustre ? -- Thanks and Regards, KK. From Alexander.Zarochentsev at Sun.COM Mon Mar 23 21:58:30 2009 From: Alexander.Zarochentsev at Sun.COM (Alexander Zarochentsev) Date: Tue, 24 Mar 2009 00:58:30 +0300 Subject: [Lustre-devel] WBC HLD outline Message-ID: <200903240058.30343.alexander.zarochentsev@sun.com> Hello, here is a wbc hld outline. Please take a look. =============================================== WBC HLD OUTLINE * Definitions WBC (MD WBC): (Meta Data) Write Back Cache. MD operation: whole MD operation over an object: rename/create/open/close/setattr/getattr/link/unlink/mkdir/rmdir + readdir. Reintegration: The process of applying accumulated MD operation to the MD servers. MDS/RAW: MDS API extension to do "raw" fs operations: inserting of a dir entry w/o creating inode and so. MD update: a part of MD operation to be executed on one server, contains one or more MDS/RAW operations. MD batch: a collection of per-server MD updates. MDTR: MD translator: translates MD operations into MD/Raw ones. * Requirements Client application is able to create 64k files/second. Reintergration moves fs from one consistent state to another consistent state. Non-WBC client support w/o visible overhead. Avoid MDS code rewrite if possible. * Design outline ** Overall picture [Application] | =syscalls= | V [VFS] | =vfs hooks= | V [LLITE/MDC] | =MD (non-WBC) proto= | V [MD CACHE MANAGER] ---> [LDLM] | V [MDTR] +-----------+----------+ | | | =======WBC proto========== | | | V V V [MDS1/RAW] [MDS2/RAW] [MDS3/RAW] ** WBC WBC client has a MDTR running on client side, it also can be a proxy server, acting as a server for non-WBC clients and as a client for MD servers. *** WBC vs non-WBC Processing MD operation request (lock enqueue + op intent, by Alex suggestion), MD server may decide to execute it by itself, or grant a only a lock (subtree one) and allow client to continue in WBC mode. *** Locks needed LDLM locks are taken before operation starts and held until the corresponded batch is re-integrated. *** Local cache management WBC client executes operations locally, modifying local in-memory objects. WBC client has a (redo-)log of all operations. The cache manager controls process of MD cache re-integration. *** MDS/RAW operations Managing directory entries and inodes, without maintaining fs consistency automatically. create/update/delete methods for directory entries and inodes. *** MDTR MDTR is responsible for converting MD operations into set of per-server MD/RAW operations. *** Client re-integration Periodically, or because of (sub-)lock releasing, dirty memory flushing or so, WBC client submits batches to all MD servers involved into the operations. Process of re-integration is protected by LDLM locks. MD servers are updated using WBC protocol. *** WBC protocol WBC request contains a set of MD/RAW operations, tagged with one epoch number. Bulk transfers are used. *** File data Flushing file data to the OST servers is delayed until file creation is re-integrated. *** Recovery The redo-log preserved until it is not needed in recovery (i.e. epoch gets stable) Client replay the log and re-execute all operations from it, repeating MDTR processing (dispatching the operation between MD servers). **** WBC client eviction, uncompleted updates If client dies until re-integration is completed, there are three choices: a) Cluster-wide rollback, all servers roll back to the last globally stable epoch, then clients to replay heir redo-logs. This scenario should be avoided because a single client failure may may stop whole cluster for recovery. b) All servers participating in re-integration coordinate to undo uncompleted updates. c) The servers have all information needed to complete re-integration w/o client. The recovery strategy is a subject of CMD Recovery Design document, but a possibility of (c) need a support in the WBC protocol. ** non-WBC *** MD protocol MD (non-WBC) protocol remains the same as now. ** Use cases *** WBC / non-WBC decision 1. Check whether server and client can operate in WBC-mode through connect flags. 2. I they can, a lock enqueue request may contain a request for WBC-mode, the server may respond with granting WBC-mode and STL or PW lock on the directory. MD server accepts or rejects WBC-mode request depending on server rules and per-object access statistics. *** File creation client gets a PW lock on directory. client fetches directory content. client does file creation locally, in cache, the operation record is added to the client redo-log. Another client want to read the directory, lock conflict triggers re-integration. MD Cache manager processes the redo-log, prepares batches with MDS/RAW operations and submits them to the MD servers. The MD servers integrate the batches. MD Cache manager frees local cache content and cancels the directory lock. ** Questions Q: Can several wbc clients work in one directory simultaneously? A: If extent locks for directories are implemented, each WBC client can take a lock on a hash interval. Q: can wbc clients do massive file creation in one directory efficiently? A: the idea that may help: if we can guess that the file names created by a client are lexicographically ordered, a special hash function may reduce lock conflicts between clients holding locks on directory extents. Thanks, -- Alexander "Zam" Zarochentsev Staff Engineer Lustre Group, Sun Microsystems From rread at sun.com Mon Mar 23 23:17:33 2009 From: rread at sun.com (Robert Read) Date: Mon, 23 Mar 2009 16:17:33 -0700 Subject: [Lustre-devel] WBC HLD outline In-Reply-To: <200903240058.30343.alexander.zarochentsev@sun.com> References: <200903240058.30343.alexander.zarochentsev@sun.com> Message-ID: <0B02BFB6-2461-48E7-A572-946898BAC9B0@sun.com> Hi Zam, On Mar 23, 2009, at 14:58 , Alexander Zarochentsev wrote: > Hello, > > here is a wbc hld outline. > Please take a look. > > =============================================== > WBC HLD OUTLINE > > * Definitions > WBC (MD WBC): (Meta Data) Write Back Cache. > > MD operation: whole MD operation over an object: > rename/create/open/close/setattr/getattr/link/unlink/mkdir/rmdir + > readdir. > > Reintegration: The process of applying accumulated MD operation to the > MD servers. > > MDS/RAW: MDS API extension to do "raw" fs operations: inserting of a > dir entry w/o creating inode and so. > > MD update: a part of MD operation to be executed on one server, > contains one or more MDS/RAW operations. Why does the client need to to be more granular than an update? It seems MDS/Raw and update should be the same. > > MD batch: a collection of per-server MD updates. > > MDTR: MD translator: translates MD operations into MD/Raw ones. Isn't this essentially what the cmm is doing today? (Breaking down distributed operations into per-node updates?) Are you expanding on Alex's idea of creating a new generic MD server stack? > > * Requirements > > Client application is able to create 64k files/second. > > Reintergration moves fs from one consistent state to another > consistent state. > > Non-WBC client support w/o visible overhead. > > Avoid MDS code rewrite if possible. > > * Design outline > > ** Overall picture > > [Application] > | > =syscalls= > | > V > [VFS] > | > =vfs hooks= > | > V > [LLITE/MDC] > | > =MD (non-WBC) proto= > | > V > [MD CACHE MANAGER] ---> [LDLM] > | > V > [MDTR] > +-----------+----------+ > | | | > =======WBC proto========== > | | | > V V V > [MDS1/RAW] [MDS2/RAW] [MDS3/RAW] > > ** WBC > > WBC client has a MDTR running on client side, > it also can be a proxy server, acting as a server for > non-WBC clients and as a client for MD servers. > > *** WBC vs non-WBC > > Processing MD operation request (lock enqueue + op intent, by Alex > suggestion), MD server may decide to execute it by itself, or grant a > only a lock (subtree one) and allow client to continue in WBC mode. > > *** Locks > > needed LDLM locks are taken before operation starts and held until the > corresponded batch is re-integrated. > > *** Local cache management > > WBC client executes operations locally, modifying local in-memory > objects. WBC client has a (redo-)log of all operations. > > The cache manager controls process of MD cache re-integration. > > *** MDS/RAW operations > > Managing directory entries and inodes, without maintaining > fs consistency automatically. > > create/update/delete methods for directory entries and inodes. > > *** MDTR > > MDTR is responsible for converting MD operations into set of > per-server MD/RAW operations. > > *** Client re-integration > > Periodically, or because of (sub-)lock releasing, dirty memory > flushing or so, WBC client submits batches to all MD servers involved > into the operations. > > Process of re-integration is protected by LDLM locks. MD servers are > updated > using WBC protocol. > > *** WBC protocol > > WBC request contains a set of MD/RAW operations, tagged with one epoch > number. Bulk transfers are used. All the updates in a single operation must have the same epoch, but I don't think we can guarantee that all the operations in a batch will be in the same epoch, unless we stop exchanging messages with all the MD servers. I don't see a need for them to be in the same epoch, either. > > *** File data > Flushing file data to the OST servers is delayed until file creation > is re-integrated. > > *** Recovery > > The redo-log preserved until it is not needed in recovery (i.e. epoch > gets stable) > > Client replay the log and re-execute all operations from it, repeating > MDTR processing (dispatching the operation between MD servers). Since the MD servers all roll back before recovery, recovery will be very similar to the original reintegration, with the exception of using versions. So we should try to keep the recovery (replay) code as similar to the normal code as possible, and move recovery higher into the stack. > > **** WBC client eviction, uncompleted updates > > If client dies until re-integration is completed, there are three > choices: > > a) Cluster-wide rollback, all servers roll back to the last globally > stable epoch, then clients to replay heir redo-logs. > > This scenario should be avoided because a single client failure may > may stop whole cluster for recovery. > > b) All servers participating in re-integration coordinate to undo > uncompleted updates. > > c) The servers have all information needed to complete re-integration > w/o client. You mean by keeping the original operation info in the undo logs? > > The recovery strategy is a subject of CMD Recovery Design document, > but a possibility of (c) need a support in the WBC protocol. > > ** non-WBC > > *** MD protocol > > MD (non-WBC) protocol remains the same as now. > > ** Use cases > > *** WBC / non-WBC decision > > 1. Check whether server and client can operate in WBC-mode through > connect flags. > > 2. I they can, a lock enqueue request may contain a request for > WBC-mode, the server may respond with granting WBC-mode and STL or PW > lock on the directory. MD server accepts or rejects WBC-mode request > depending on server rules and per-object access statistics. > > *** File creation > > client gets a PW lock on directory. > > client fetches directory content. > > client does file creation locally, in cache, the operation record is > added to the client redo-log. > > Another client want to read the directory, lock conflict triggers > re-integration. > > MD Cache manager processes the redo-log, prepares batches with MDS/RAW > operations and submits them to the MD servers. > > The MD servers integrate the batches. > > MD Cache manager frees local cache content and cancels the directory > lock. > > ** Questions > > Q: Can several wbc clients work in one directory simultaneously? > A: If extent locks for directories are implemented, each WBC client > can take a lock on a hash interval. > > Q: can wbc clients do massive file creation in one directory > efficiently? > A: the idea that may help: if we can guess that the file names created > by a client are lexicographically ordered, a special hash function > may reduce lock conflicts between clients holding locks on > directory extents. cheers, robert From eeb at sun.com Tue Mar 24 00:34:17 2009 From: eeb at sun.com (Eric Barton) Date: Mon, 23 Mar 2009 17:34:17 -0700 Subject: [Lustre-devel] LustreFS performance (update) In-Reply-To: <49C32DDE.1030705@sun.com> References: <3376C558-E29A-4BB5-8C4C-3E8F4537A195@sun.com> <02FEAA2B-8D98-4C2D-9CE8-FF6E1EB135A2@sun.com> <49C32DDE.1030705@sun.com> Message-ID: <005e01c9ac18$4f0107f0$ed0317d0$@com> Vitaly, I've been following this thread with great interest and I'd like to chat with you about this and also the MDS performance regression tests. Unfortunately, I'm unlikely to be able to do that this week and it will probably have to wait until I'm back in the UK next week. In the mean time... 1. Have you got a rough idea how much work it would be to write the software that could exercise the MDD directly? I'd just like to know if we're talking days or weeks or months - we need to know that before we decide whether to do it. 2. I think Andrew Uselton's comments are helpful. We cannot afford routinely to sample the whole performance space - there are just too many dimensions. So we need to develop a performance model that allows us to restrict the number of measurements we need to be confident that there are no surprises "in between" the points we have sampled. That means we have to start running tests as soon as possible over as wide a parameter range as possible, with as much hardware as possible. Then we'll start to get a feel how much variability there is all over the space and where the "edges" and asymptotes are. 3. It's worthwhile taking time to analyse and present results with care. I've attached a spreadsheet that compares ping performance of a single 8-core server with varying numbers of clients and client threads, measured using different LNET locking schemes - hp (HEAD ping), 2lp (HEAD modified to split the LNET global lock into 2) and 3lp (same, but splitting the LNET global lock into 3). The lower row of graphs shows ping throughput versus number of client nodes, with different numbers of threads per node in each series. The upper row of graphs shows the same ping throughput, but plotted against client threads totalled over all nodes, with different numbers of nodes in each series. Please note.... a) Set axis scaling correctly so that visual comparison is accurate. b) The upper row of graphs shows that it's the total number of threads exercising the server that's most important - and that how those threads are distributed over client nodes seems to matter most when there are 8 of them. That's absolutely _not_ obvious from looking at the lower row of graphs. Cheers, Eric > -----Original Message----- > From: lustre-devel-bounces at lists.lustre.org [mailto:lustre-devel-bounces at lists.lustre.org] On Behalf Of parinay > kondekar > Sent: 19 March 2009 10:47 PM > To: Vitaly Fertman > Cc: lustre-2.0-performance at sun.com; minh diep; Lustre Development Mailing List > Subject: Re: [Lustre-devel] LustreFS performance (update) > > The wiki :: https://wikis.clusterfs.com/intra/index.php/LustreFS_performance > > ~p > > Vitaly Fertman wrote: > > **************************************************** > > LustreFS benchmarking methodology. > > **************************************************** > > > > > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel -------------- next part -------------- A non-text attachment was scrubbed... Name: example graphs.ods Type: application/vnd.oasis.opendocument.spreadsheet Size: 51672 bytes Desc: not available URL: From di.wang at sun.com Tue Mar 24 01:56:29 2009 From: di.wang at sun.com (di wang) Date: Mon, 23 Mar 2009 21:56:29 -0400 Subject: [Lustre-devel] Regarding Server Driven QOS for lustre In-Reply-To: <89c400ad0903210049o489ba04ax38f9198d57543e9a@mail.gmail.com> References: <89c400ad0903210049o489ba04ax38f9198d57543e9a@mail.gmail.com> Message-ID: <49C83DCD.803@sun.com> Hello, The project has turned into NRS (network request scheduler). You can check this bug https://bugzilla.lustre.org/show_bug.cgi?id=13634 Thanks WangDi Krishnakumar R wrote: > Hi Folks, > > What is the current status of the LRS and EH implementation on Lustre > (mentioned in the mail: > http://groups.google.com/group/lustre-devel-list/msg/c921d5b7121c50eb) > ? > Any further documents/links on the server driven QOS on lustre ? > > -- > Thanks and Regards, > KK. > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel > From bzzz at sun.com Tue Mar 24 05:06:39 2009 From: bzzz at sun.com (Alex Zhuravlev) Date: Tue, 24 Mar 2009 08:06:39 +0300 Subject: [Lustre-devel] WBC HLD outline In-Reply-To: <200903240058.30343.alexander.zarochentsev@sun.com> References: <200903240058.30343.alexander.zarochentsev@sun.com> Message-ID: >>>>> Alexander Zarochentsev (AZ) writes: AZ> MDS/RAW: MDS API extension to do "raw" fs operations: inserting of a AZ> dir entry w/o creating inode and so. this seems to be duplication of OSD API's insert/delete/etc. AZ> MDTR: MD translator: translates MD operations into MD/Raw ones. and this one seems to duplicate MDD code. why would we want to duplicate these things? thanks, Alex From rread at sun.com Tue Mar 24 23:53:29 2009 From: rread at sun.com (Robert Read) Date: Tue, 24 Mar 2009 16:53:29 -0700 Subject: [Lustre-devel] some observations about metadata writeback cache In-Reply-To: References: Message-ID: <3AF28E58-CF74-49C9-AED7-86BDBB5EEC65@sun.com> Hi Alex, I'm trying to figure out how untrusted (what I'm calling simple) clients and trusted WBC-type clients will work together at the same time. Simple clients will need to participate in the oldest volatile epoch calculation, but will need to retain operations for replay. I've draw a simplified picture of how I think things are beginning to fit together, but more thought is needed here. Simple clients - don't participate in global epochs - don't have a node epoch or add epochs to messages - sends operations to MD server - replies include extended opaque "replay" data field - replayed operations the replay data is included - replay list is flushed based on "transno" (which may actually be the epoch and the replay data contains the actual transnos) - multiple operations can have the same "transno" Trusted clients - participate in global epochs - have a capability that allows them to participate - sends updates to OSD servers with epochs - replay-data contains only a single reply, could be same as today - when all update replies are received operation is placed on redo list - redo list flushed based on OVE MD server - MDT/MDD receives operations without epochs - sets the operation epoch to the node's current epoch - all updates executed for that operation will use same epoch. - replies are gathered and sent in "replay data" field - participates in OVE - how much state does it need to retain to do this on behalf of the clients? - OSD receives updates epochs - handled locally - normal reply returned robert -------------- next part -------------- A non-text attachment was scrubbed... Name: cmd-recovery.pdf Type: application/pdf Size: 50831 bytes Desc: not available URL: -------------- next part -------------- On Mar 9, 2009, at 23:41 , Alex Zhuravlev wrote: > > Hello, > > I spent quite amount time thinking of wbc problem and I'd like to > share > the thoughts. > > for wbc we store metadata in local memory for two purposes: > 1) later reintegration > 2) read access (lookup, getattr, readdir) w/o server involvement > > for (2) it makes sense to store everything as "state". e.g. directory > contains all alive entries, inode contains last valid attributes, etc. > let's call it state cache. > > in theory reintegration can be done from the state cache and this is > probably the most efficient way (in terms of network traffic and > memory > footprint). but for simpler implementation we can introduce log of > changes for (1). in turn, the log can be per-object or just global log > for given filesystem. > > it's hard to implement state cache in terms of operations because > usual > operation involves more than one object (e.g. parent directory + > file). > it's much simpler when state cache is per-object. literally the best > example is linux's dcache and inode cache. > > it's also fairly simple to maintain such cache at level where single > object is being modified. for our purposes this matches layer > implementing > OSD API - because all operations in OSD API are per single object. > > the same applies to reintegration because: > * we need to break complex operations to be sent to different > servers anyway > * if we'd need to optimize log (i.e., create/unlink), then it's > simpler > to collapse log entries when they are basic operations > * when we'd want to reintegrate from state cache > > we also need a layer to take metadata operations and translate them > into > per-object basic operations (updates). responsbility of this layer is: > * to grab all required ldlm locks > as the layer understands operation's nature, locking rules, etc > * to check current state > whether name exists alread (for create), permissions > * to apply updates to state cache (and reintegration backend, if > required) > * to release ldlm locks > > essentially this is what current metadata server does. the > difference is > * locks to be acquired on remote node > * current state can be on remote node (not in local state cache) > * updates can be stored in local memory for later reintegration > (perhaps this applies to usual mds) > > it looks quite obvious that it'd make sense to use metadata server > code to > implement wbc: > * ldlm hides where lock is being mastered > * dedicated osd layer below metadata server can maintain state cache > needed > to check existing names, attributes, permissions, etc > * dedicated osd layer below metadata server can take care of > reintegration > > > implementation would look like set of the following modules: > * mdf - metadata filter > this is location-free metadata server operating on top of osd api, > grabs > ldlm locks, check current state, apply changes. > * cosd - caching osd > this is dedicated layer with osd api, it maintains state cache and > all data > needed for reintegration. it also tries to use network efficient: > regular > lookup can be implemented via underlying readdir, etc. > * gosd - global osd > very specific module allowing node to talk to remote storage over > osd api, > it's stateless, something similar to current mdc, but using > different apis. > > > some obvious cons of this approach: > * implementation doesn't rely on any system specific thing like > dcache/icache > * we can unify the code and re-use it to implement regular metadata > server, > wbc and metadata proxy server > * overall simplicity > inter-layer interaction is well defined and simple, same about > layer's > functionality > * clustered metadata fits this model very well because metadata server > doesn't need to know whether some update local or remote > > any comments and suggestions are very welcome! > > > thanks, Alex > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel From Oleg.Drokin at Sun.COM Wed Mar 25 04:37:54 2009 From: Oleg.Drokin at Sun.COM (Oleg Drokin) Date: Wed, 25 Mar 2009 00:37:54 -0400 Subject: [Lustre-devel] ORNL metadata testing results from this Friday. Message-ID: Hello! Dave Dillow was kind enough to spend some time with me on some metadata tests this Friday, two main goals were to find out how much the patch from bug 18534 has on mkdir performance and how big of a drag is there from slow OST object creations (the test program was modified to use O_LOV_DELAY_CREATE on file open to completely skip object creation). The test was run with different number of clients, from 1 to 64, using mdtest in 2 modes, shared directory and separate directory for every client. MDS is 16-way SMP node (4x 4-core cpus) with 32G RAM. Infiniband interconnect. The good news - when we are not disk bound, the patch from 18534 doubles mkdir rate. Unfortunately at around 8 clients we seem to be overflowing journal rapidly and that drags performance down very significantly (visible in vmstat). Another set of runs was introduced where we used ext3 with 1G journal on a loop device in a file on tmpfs. Strange parts - shared directory file creates (with objects) seems to be faster than separate dir-client. Another strange thing is that actually not creating any objects did not have any dramatic impact on create performance and creates still remained 5-10 times slower than mkdirs. I tried to reproduce this on my home test machine and certainly do not observe anything like this, so this will be next thing I will try to figure out at ORNL when Dave gets back here. (Before this question arises - yes, we did check that files were created without any objects). mkdir create rate dives down significantly when we go from 32 to 64 clients even with journal on ramdisk, which is strange, since there should not be any significant overhead, I would think we should remain at roughly the same level of performance. CPU is at 100%. Unfortunately oprofile was not installed. (I am not sure that ext3 actually uses more than 400M of journal even with external journals, so it could be we overflown entire journal and were forced to write the changes to actual disk, though now that I think about it, I do not remember big spikes in activity on the disk, and we were trying to watch it.) Attached is an (openoffice) table and my feeble attempts at building some graphs in it with test results. There are numbers for creates and deletes. The test also gathers stat performance, but it behaves very similar to unlinks in separate dirs (except much faster, of course), so I did not entered the numbers into the table. Bye, Oleg -------------- next part -------------- A non-text attachment was scrubbed... Name: ORNL MDtest Results.ods Type: application/vnd.oasis.opendocument.spreadsheet Size: 61561 bytes Desc: not available URL: -------------- next part -------------- From Alexander.Zarochentsev at Sun.COM Wed Mar 25 08:17:48 2009 From: Alexander.Zarochentsev at Sun.COM (Alexander Zarochentsev) Date: Wed, 25 Mar 2009 11:17:48 +0300 Subject: [Lustre-devel] WBC HLD outline In-Reply-To: <0B02BFB6-2461-48E7-A572-946898BAC9B0@sun.com> References: <200903240058.30343.alexander.zarochentsev@sun.com> <0B02BFB6-2461-48E7-A572-946898BAC9B0@sun.com> Message-ID: <200903251117.48986.alexander.zarochentsev@sun.com> On 24 March 2009 02:17:33 Robert Read wrote: > Hi Zam, > > MD update: a part of MD operation to be executed on one server, > > contains one or more MDS/RAW operations. > > Why does the client need to to be more granular than an update? It > seems MDS/Raw and update should be the same. well, better to say an update is MDS op if the operation touch only one MD server and MDS/Raw op in case of distributed operation. > > MD batch: a collection of per-server MD updates. > > > > MDTR: MD translator: translates MD operations into MD/Raw ones. > > Isn't this essentially what the cmm is doing today? (Breaking down > distributed operations into per-node updates?) Are you expanding on > Alex's idea of creating a new generic MD server stack? I just doubt that cmm code reuse is worth MD stack relayering. Can it be done as a subtask later? > > *** WBC protocol > > > > WBC request contains a set of MD/RAW operations, tagged with one > > epoch number. Bulk transfers are used. > > All the updates in a single operation must have the same epoch, but I > don't think we can guarantee that all the operations in a batch will > be in the same epoch, unless we stop exchanging messages with all the > MD servers. I don't see a need for them to be in the same epoch, > either. you are right. > > *** File data > > Flushing file data to the OST servers is delayed until file > > creation is re-integrated. > > > > *** Recovery > > > > The redo-log preserved until it is not needed in recovery (i.e. > > epoch gets stable) > > > > Client replay the log and re-execute all operations from it, > > repeating MDTR processing (dispatching the operation between MD > > servers). > > Since the MD servers all roll back before recovery, recovery will be > very similar to the original reintegration, with the exception of > using versions. So we should try to keep the recovery (replay) code > as similar to the normal code as possible, and move recovery higher > into the stack. ok. > > **** WBC client eviction, uncompleted updates > > > > If client dies until re-integration is completed, there are three > > choices: > > > > a) Cluster-wide rollback, all servers roll back to the last > > globally stable epoch, then clients to replay heir redo-logs. > > > > This scenario should be avoided because a single client failure may > > may stop whole cluster for recovery. > > > > b) All servers participating in re-integration coordinate to undo > > uncompleted updates. > > > > c) The servers have all information needed to complete > > re-integration w/o client. > > You mean by keeping the original operation info in the undo logs? I meant the servers receive not updates but whole operations. If the client failed and didn't send an update to some of the servers, the operation can be completed w/o the client. It is an alternative to undoing of partial updates. Thanks, -- Alexander "Zam" Zarochentsev Staff Engineer Lustre Group, Sun Microsystems From bzzz at sun.com Wed Mar 25 08:33:12 2009 From: bzzz at sun.com (Alex Zhuravlev) Date: Wed, 25 Mar 2009 11:33:12 +0300 Subject: [Lustre-devel] WBC HLD outline In-Reply-To: <200903251117.48986.alexander.zarochentsev@sun.com> References: <200903240058.30343.alexander.zarochentsev@sun.com> <0B02BFB6-2461-48E7-A572-946898BAC9B0@sun.com> <200903251117.48986.alexander.zarochentsev@sun.com> Message-ID: >>>>> Alexander Zarochentsev (AZ) writes: AZ> On 24 March 2009 02:17:33 Robert Read wrote: >> Hi Zam, >> > MD update: a part of MD operation to be executed on one server, >> > contains one or more MDS/RAW operations. >> >> Why does the client need to to be more granular than an update? It >> seems MDS/Raw and update should be the same. AZ> well, better to say an update is MDS op if the operation touch only one AZ> MD server and MDS/Raw op in case of distributed operation. I think this just adds unneeded entity to the system. stating that we either have updates or operations is simpler. >> Isn't this essentially what the cmm is doing today? (Breaking down >> distributed operations into per-node updates?) Are you expanding on >> Alex's idea of creating a new generic MD server stack? AZ> I just doubt that cmm code reuse is worth MD stack relayering. Can it be AZ> done as a subtask later? I don't think CMM is right thing because it essentially breaks layering: instead of sending object creation request in terms of OSD API or index insert in terms of OSD API it introduces some intermediate thing which is neither operation nor update. >> You mean by keeping the original operation info in the undo logs? AZ> I meant the servers receive not updates but whole operations. If the AZ> client failed and didn't send an update to some of the servers, the AZ> operation can be completed w/o the client. It is an alternative to AZ> undoing of partial updates. same can be done with updates if you send them through single server. and you don't need to put additional cpu processing to parse operation into updates. thanks, Alex From Alexander.Zarochentsev at Sun.COM Wed Mar 25 16:17:29 2009 From: Alexander.Zarochentsev at Sun.COM (Alexander Zarochentsev) Date: Wed, 25 Mar 2009 19:17:29 +0300 Subject: [Lustre-devel] WBC HLD outline In-Reply-To: References: <200903240058.30343.alexander.zarochentsev@sun.com> <200903251117.48986.alexander.zarochentsev@sun.com> Message-ID: <200903251917.30178.alexander.zarochentsev@sun.com> On 25 March 2009 11:33:12 Alex Zhuravlev wrote: > >>>>> Alexander Zarochentsev (AZ) writes: > > AZ> On 24 March 2009 02:17:33 Robert Read wrote: > >> Hi Zam, > >> > >> > MD update: a part of MD operation to be executed on one server, > >> > contains one or more MDS/RAW operations. > >> > >> Why does the client need to to be more granular than an update? > >> It seems MDS/Raw and update should be the same. > > AZ> well, better to say an update is MDS op if the operation touch > only one AZ> MD server and MDS/Raw op in case of distributed > operation. > > > I think this just adds unneeded entity to the system. stating that > we either have updates or operations is simpler. > > >> Isn't this essentially what the cmm is doing today? (Breaking > >> down distributed operations into per-node updates?) Are you > >> expanding on Alex's idea of creating a new generic MD server > >> stack? > > AZ> I just doubt that cmm code reuse is worth MD stack relayering. > Can it be AZ> done as a subtask later? > > I don't think CMM is right thing because it essentially breaks > layering: instead of sending object creation request in terms of OSD > API or index insert in terms of OSD API it introduces some > intermediate thing which is neither operation nor update. Server MD stack has to support both WBC and non-WBC clients for the same objects. It is why I think MDT layer should handle MD ops as well as MDS/RAW ops. Then CMM only passes RAW operations to MDD layer, where raw ops are already supported. Thanks, -- Alexander "Zam" Zarochentsev Staff Engineer Lustre Group, Sun Microsystems From bzzz at sun.com Wed Mar 25 16:26:28 2009 From: bzzz at sun.com (Alex Zhuravlav) Date: Wed, 25 Mar 2009 19:26:28 +0300 Subject: [Lustre-devel] WBC HLD outline In-Reply-To: <200903251917.30178.alexander.zarochentsev@sun.com> References: <200903240058.30343.alexander.zarochentsev@sun.com> <200903251117.48986.alexander.zarochentsev@sun.com> <200903251917.30178.alexander.zarochentsev@sun.com> Message-ID: >>>>> Alexander Zarochentsev (AZ) writes: AZ> Server MD stack has to support both WBC and non-WBC clients for the same AZ> objects. It is why I think MDT layer should handle MD ops as well as AZ> MDS/RAW ops. Then CMM only passes RAW operations to MDD layer, where AZ> raw ops are already supported. then I don't understand what you mean by CMM. same about RAW operations. -- thanks, Alex From bzzz at sun.com Wed Mar 25 16:32:39 2009 From: bzzz at sun.com (Alex Zhuravlav) Date: Wed, 25 Mar 2009 19:32:39 +0300 Subject: [Lustre-devel] WBC HLD outline In-Reply-To: <200903251917.30178.alexander.zarochentsev@sun.com> References: <200903240058.30343.alexander.zarochentsev@sun.com> <200903251117.48986.alexander.zarochentsev@sun.com> <200903251917.30178.alexander.zarochentsev@sun.com> Message-ID: >>>>> Alexander Zarochentsev (AZ) writes: AZ> Server MD stack has to support both WBC and non-WBC clients for the same AZ> objects. It is why I think MDT layer should handle MD ops as well as AZ> MDS/RAW ops. Then CMM only passes RAW operations to MDD layer, where AZ> raw ops are already supported. btw, what's problem with supporting WBC and non-WBC clients for same objects? any time you access some object via short (MDT-OSD for WBC client) or long (MDT-MDD-OSD) for non-WBC client) it's initialized at all layers (MDT-MDD-OSD). -- thanks, Alex From bzzz at sun.com Wed Mar 25 16:59:48 2009 From: bzzz at sun.com (Alex Zhuravlev) Date: Wed, 25 Mar 2009 19:59:48 +0300 Subject: [Lustre-devel] some observations about metadata writeback cache In-Reply-To: <3AF28E58-CF74-49C9-AED7-86BDBB5EEC65@sun.com> References: <3AF28E58-CF74-49C9-AED7-86BDBB5EEC65@sun.com> Message-ID: >>>>> Robert Read (RR) writes: RR> Hi Alex, RR> I'm trying to figure out how untrusted (what I'm calling simple) RR> clients and trusted WBC-type clients will work together at the same RR> time. Simple clients will need to participate in the oldest volatile RR> epoch calculation, but will need to retain operations for replay. RR> I've draw a simplified picture of how I think things are beginning to RR> fit together, but more thought is needed here. RR> Simple clients RR> - don't participate in global epochs hmm. if committed (in terms of transno) request can be reverted during global recovery, then even simple client has to retain request on replay list till it's stable in terms of epochs? RR> - don't have a node epoch or add epochs to messages RR> - sends operations to MD server RR> - replies include extended opaque "replay" data field probably we could simplify code a lot if we don't need to put reply into request in order to do replay? IOW, make all request's fields client-generated? thanks, Alex From Oleg.Drokin at Sun.COM Wed Mar 25 17:48:09 2009 From: Oleg.Drokin at Sun.COM (Oleg Drokin) Date: Wed, 25 Mar 2009 13:48:09 -0400 Subject: [Lustre-devel] some observations about metadata writeback cache In-Reply-To: References: <3AF28E58-CF74-49C9-AED7-86BDBB5EEC65@sun.com> Message-ID: <878C772A-9A7F-4B1B-A7FB-31815306CA42@sun.com> Hello! On Mar 25, 2009, at 12:59 PM, Alex Zhuravlev wrote: > RR> Simple clients > RR> - don't participate in global epochs > hmm. if committed (in terms of transno) request can be reverted > during global recovery, then even simple client has to retain > request on replay list till it's stable in terms of epochs? Supposedly, server that performed the operation on behalf of the client can do this? So the simple client semantic does not change - the moment server has some stable record about hte operation, client can throw the data away (otherwise simple clients would need to know how to participate in rollback/replay even when the server the operation was sent to did not go down). Bye, Oleg From bzzz at sun.com Wed Mar 25 17:52:13 2009 From: bzzz at sun.com (Alex Zhuravlev) Date: Wed, 25 Mar 2009 20:52:13 +0300 Subject: [Lustre-devel] some observations about metadata writeback cache In-Reply-To: <878C772A-9A7F-4B1B-A7FB-31815306CA42@sun.com> References: <3AF28E58-CF74-49C9-AED7-86BDBB5EEC65@sun.com> <878C772A-9A7F-4B1B-A7FB-31815306CA42@sun.com> Message-ID: >>>>> Oleg Drokin (OD) writes: OD> Hello! OD> On Mar 25, 2009, at 12:59 PM, Alex Zhuravlev wrote: RR> Simple clients RR> - don't participate in global epochs >> hmm. if committed (in terms of transno) request can be reverted >> during global recovery, then even simple client has to retain >> request on replay list till it's stable in terms of epochs? OD> Supposedly, server that performed the operation on behalf of the client OD> can do this? So the simple client semantic does not change - the moment OD> server has some stable record about hte operation, client can throw the OD> data away (otherwise simple clients would need to know how to OD> participate OD> in rollback/replay even when the server the operation was sent to did OD> not OD> go down). hmm. then wouldn't be simpler to do replay before global recovery and then do global replay from server's undo logs? -- thanks, Alex From Oleg.Drokin at Sun.COM Wed Mar 25 17:59:25 2009 From: Oleg.Drokin at Sun.COM (Oleg Drokin) Date: Wed, 25 Mar 2009 13:59:25 -0400 Subject: [Lustre-devel] some observations about metadata writeback cache In-Reply-To: References: <3AF28E58-CF74-49C9-AED7-86BDBB5EEC65@sun.com> <878C772A-9A7F-4B1B-A7FB-31815306CA42@sun.com> Message-ID: <34C324F2-81EA-46CC-83FC-86D0265E4192@Sun.COM> Hello! On Mar 25, 2009, at 1:52 PM, Alex Zhuravlev wrote: > RR> Simple clients > RR> - don't participate in global epochs >>> hmm. if committed (in terms of transno) request can be reverted >>> during global recovery, then even simple client has to retain >>> request on replay list till it's stable in terms of epochs? > OD> Supposedly, server that performed the operation on behalf of the > client > OD> can do this? So the simple client semantic does not change - the > moment > OD> server has some stable record about hte operation, client can > throw the > OD> data away (otherwise simple clients would need to know how to > OD> participate > OD> in rollback/replay even when the server the operation was sent > to did > OD> not > OD> go down). > hmm. then wouldn't be simpler to do replay before global recovery > and then > do global replay from server's undo logs? Yes, but aside from that, losing a caching client leads to global recovery, but losing simple client is not, since server tracks its status. Bye, Oleg From rread at sun.com Wed Mar 25 18:03:54 2009 From: rread at sun.com (Robert Read) Date: Wed, 25 Mar 2009 11:03:54 -0700 Subject: [Lustre-devel] some observations about metadata writeback cache In-Reply-To: References: <3AF28E58-CF74-49C9-AED7-86BDBB5EEC65@sun.com> Message-ID: <6EEE8418-1284-4908-ABA8-CF9446308AAB@sun.com> On Mar 25, 2009, at 09:59 , Alex Zhuravlev wrote: >>>>>> Robert Read (RR) writes: > > RR> Hi Alex, > RR> I'm trying to figure out how untrusted (what I'm calling simple) > RR> clients and trusted WBC-type clients will work together at the > same > RR> time. Simple clients will need to participate in the oldest > volatile > RR> epoch calculation, but will need to retain operations for replay. > RR> I've draw a simplified picture of how I think things are > beginning to > RR> fit together, but more thought is needed here. > > RR> Simple clients > RR> - don't participate in global epochs > > hmm. if committed (in terms of transno) request can be reverted > during global recovery, then even simple client has to retain > request on replay list till it's stable in terms of epochs? Yes, this is why these clients won't see the actual transno(s). Those would only be in the replay data blob the server returns with the reply. Instead, the MD server would send the epoch as the transno in the reply to these clients. > > RR> - don't have a node epoch or add epochs to messages > RR> - sends operations to MD server > RR> - replies include extended opaque "replay" data field > > probably we could simplify code a lot if we don't need to put > reply into request in order to do replay? IOW, make all request's > fields client-generated? True, but a request could contain multiple replies (one for each update), and the client doesn't need to be aware of that. I was thinking it would be better if the server managed this field. This mans the client can replay the request as it was originally sent and include the additional data at the end so the server can replay the updates in the correct order. cheers, robert From Nathan.Rutman at Sun.COM Wed Mar 25 19:48:03 2009 From: Nathan.Rutman at Sun.COM (Nathaniel Rutman) Date: Wed, 25 Mar 2009 12:48:03 -0700 Subject: [Lustre-devel] Start the MGS first? Message-ID: <49CA8A73.9080103@sun.com> As of Lustre 1.6, servers can be started in any order (after the initial registration at first startup). Internally, this required significant bending of our connection rules, and with a move toward ZFS becomes even more burdensome. So my question to the Lustre community is this: would anyone strenuously object to a startup ordering requirement that the MGS must be started before any other servers? This would probably be in the Lustre 3.0 timeframe. It is also likely that we will have to divorce the MGS and MDT onto separate devices -- no more "combo" MDT/MGSes. > > NR> I think the only reason to have a local config file is to be able to > > NR> start a server in the absence of the MGS. How much effort do we want > > NR> to expend to be able to keep that ability? I don't think it's a huge > > NR> burden to say "MGS must be started first". From Brian.Murrell at Sun.COM Wed Mar 25 20:00:40 2009 From: Brian.Murrell at Sun.COM (Brian J. Murrell) Date: Wed, 25 Mar 2009 16:00:40 -0400 Subject: [Lustre-devel] Start the MGS first? In-Reply-To: <49CA8A73.9080103@sun.com> References: <49CA8A73.9080103@sun.com> Message-ID: <1238011240.6090.16.camel@pc.interlinx.bc.ca> On Wed, 2009-03-25 at 12:48 -0700, Nathaniel Rutman wrote: > As of Lustre 1.6, servers can be started in any order (after the initial registration at first startup). Internally, this required significant bending of our connection rules, and with a move toward ZFS becomes even more burdensome. > So my question to the Lustre community is this: would anyone strenuously object to a startup ordering requirement that the MGS must be started before any other servers? > This would probably be in the Lustre 3.0 timeframe. It is also likely that we will have to divorce the MGS and MDT onto separate devices -- no more "combo" MDT/MGSes. I'll ask before anyone else does... would this require the MGS be available at any time a server (or client?) needs to start rather than the optional behaviour (for anything but first time server starts) that we currently enjoy? IOW, does this make the MGS a much more critical component of the filesystem than it is currently? b. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part URL: From nic at cray.com Wed Mar 25 20:15:59 2009 From: nic at cray.com (Nicholas Henke) Date: Wed, 25 Mar 2009 15:15:59 -0500 Subject: [Lustre-devel] Start the MGS first? In-Reply-To: <49CA8A73.9080103@sun.com> References: <49CA8A73.9080103@sun.com> Message-ID: <49CA90FF.1070801@cray.com> Nathaniel Rutman wrote: > As of Lustre 1.6, servers can be started in any order (after the initial > registration at first startup). Internally, this required significant > bending of our connection rules, and with a move toward ZFS becomes even > more burdensome. So my question to the Lustre community is this: would > anyone strenuously object to a startup ordering requirement that the MGS > must be started before any other servers? This would probably be in the > Lustre 3.0 timeframe. It is also likely that we will have to divorce > the MGS and MDT onto separate devices -- no more "combo" MDT/MGSes. >> > NR> I think the only reason to have a local config file is to be >> able to >> > NR> start a server in the absence of the MGS. How much effort do >> we want >> > NR> to expend to be able to keep that ability? I don't think it's >> a huge >> > NR> burden to say "MGS must be started first". > This is virtually ensured today due to how the timeouts and ordering works. 1) It is a real PITA to script up different server start orders to deal with reformat and write_conf. It is easier to just script one correct way of doing this - KISS if you will. 2) At large scale, the timeout cascading on the OSTs (many OSTs per OSS) from a missing MGS requires it be started first. 3) With bug 14134 and --nomgs and --nosvc options for starting, it makes starting a combo MGS and MDS "correctly" much easier. Nic From Kevin.Fox at pnl.gov Thu Mar 26 21:35:14 2009 From: Kevin.Fox at pnl.gov (Kevin Fox) Date: Thu, 26 Mar 2009 14:35:14 -0700 Subject: [Lustre-devel] [Lustre-discuss] Start the MGS first? In-Reply-To: <49CA8A73.9080103@sun.com> References: <49CA8A73.9080103@sun.com> Message-ID: <1238103314.4115.11.camel@sledge.emsl.pnl.gov> On Wed, 2009-03-25 at 12:48 -0700, Nathaniel Rutman wrote: > As of Lustre 1.6, servers can be started in any order (after the > initial registration at first startup). Internally, this required > significant bending of our connection rules, and with a move toward > ZFS becomes even more burdensome. > So my question to the Lustre community is this: would anyone > strenuously object to a startup ordering requirement that the MGS must > be started before any other servers? > This would probably be in the Lustre 3.0 timeframe. It is also likely > that we will have to divorce the MGS and MDT onto separate devices -- > no more "combo" MDT/MGSes. > "server startup" being, things wont start working until the MGS is up, or, server startup commands will fail if the MGS is not up? The former, is much better then the latter. It allows the system to potentially bring itself back up automatically if things get rebooted (power bump, spike, etc). Kevin > > > NR> I think the only reason to have a local config file is to be > able to > > > NR> start a server in the absence of the MGS. How much effort do > we want > > > NR> to expend to be able to keep that ability? I don't think > it's a huge > > > NR> burden to say "MGS must be started first". > > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss > > From sethpn at gmail.com Fri Mar 27 05:23:17 2009 From: sethpn at gmail.com (sethpn at gmail.com) Date: Fri, 27 Mar 2009 05:23:17 +0000 Subject: [Lustre-devel] (no subject) Message-ID: <00163628360a279308046612ee63@google.com> Is there any Lenny packaged with Lustre 1.6.7? This will be great! -------------- next part -------------- An HTML attachment was scrubbed... URL: From Oleg.Drokin at Sun.COM Sat Mar 28 19:08:25 2009 From: Oleg.Drokin at Sun.COM (Oleg Drokin) Date: Sat, 28 Mar 2009 15:08:25 -0400 Subject: [Lustre-devel] Async close RPC Message-ID: <29A3C6ED-8132-4896-92CB-3518036D84BB@Sun.COM> Hello! It recently struck me as how unfair many file creation tests are to Lustre. The problem is only with tests because they portray Lustre creation rates lower than actual speed to be seen by application. (same goes for opens, btw). The problem is typical create-rate test does an open-close sequence in a loop. Now close is synchronous RPC on Lustre in majority of tests. By making close to be asynchronous we would make Lustre to appear more fast at creates by reducing all of that overhead. Of course no real application would benefit, because I am not aware of any where there is a tight open-close loop there. Real applications are opening some files at some point, then do i/o for some extended time and only then close would happen. I know in some of CMD cases this idea was considered, but did not pan out for some reason (I am not familiar with that implementation). Anyway I performed a test at ORNL Jaguar system running an application creating 10000 files (open-creat, with O_LOV_DELAY_CREATE flag to reduce OST influence, since we are working separately on addressing that) and then closing 10000 files, all in 2 timed loops. The app was run on a scale of 1 to 64 clients (in power of 2 increments). From the test it is easily observable that the closes easily bring in 50% penalty to overall creation rate. E.g. at a scale 1: 10k opens take 1.946946, 10k subsequent closes take 1.031471. (5136 real creates/sec vs 3357 "reported by usual test" creates sec) at a scale 8: 80k opens take 6.21 sec, 80k subsequent closes take 3.51 sec (12800 real creates/sec vs 8230 "reported by usual test" creates sec). Now of course if we make closes completely asynchronous, they would still be competing for CPU at MDS with opens, inducing some penalty still, so for this type of test ideally we would like all closes to go to some separate portal with only one handling thread to minimize cpu consumption, but this is not really idea for real workloads, of course, the real impact here could be made by NRS, where opens from same job would get prioritized ahead of closes from the same job. Anyway, I am thinking it is good idea to implement async closes if only to make us look better (read - more realistic) in these tests, and for proper implementation to work we need to get rid of close sending serialization (since spawning a separate close thread for every close would be stupid). I think the close serialization is not needed anyway. If the close reply was lost, it would be resent and we can just supress the resulting error seeing how resent close just tried to close nonexistent close handle. On recovery we care even less, there is nothing to close after server restart. (I am not sure what SOM implications that might have? But I suspect none - there is some extra state in mfd that could tell us if we already executed this close and we probably can reconstruct necessary reply state for resend from it, Vitaly?) Any comments or concerns from anyone? Bye, Oleg From Brian.Murrell at Sun.COM Mon Mar 30 14:42:21 2009 From: Brian.Murrell at Sun.COM (Brian J. Murrell) Date: Mon, 30 Mar 2009 10:42:21 -0400 Subject: [Lustre-devel] Async close RPC In-Reply-To: <29A3C6ED-8132-4896-92CB-3518036D84BB@Sun.COM> References: <29A3C6ED-8132-4896-92CB-3518036D84BB@Sun.COM> Message-ID: <1238424141.24213.195.camel@pc.interlinx.bc.ca> On Sat, 2009-03-28 at 15:08 -0400, Oleg Drokin wrote: > Hello! > > It recently struck me as how unfair many file creation tests are > to Lustre. The problem is only with tests because they portray Lustre > creation rates lower than actual speed to be seen by application. > (same goes for opens, btw). The only thing I would add/say Oleg, is have you looked at the Hendrix CMD specific metadata tests? I only mention this because from what I recall, we took advantage of every opportunity we could (within the requirements of the project) to show performance in the best light we could. I'd imagine that those tests (and the corresponding Lustre code_ would have considered some of the same things you are considering. It may be however that some of the solutions were CMD specific and not really applicable to you, but perhaps in the least, looking at some of those tests (i.e. mdsrate was one key test) might prevent you from re-inventing tests. Cheers, b. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part URL: From Oleg.Drokin at Sun.COM Mon Mar 30 14:52:17 2009 From: Oleg.Drokin at Sun.COM (Oleg Drokin) Date: Mon, 30 Mar 2009 10:52:17 -0400 Subject: [Lustre-devel] Async close RPC In-Reply-To: <1238424141.24213.195.camel@pc.interlinx.bc.ca> References: <29A3C6ED-8132-4896-92CB-3518036D84BB@Sun.COM> <1238424141.24213.195.camel@pc.interlinx.bc.ca> Message-ID: <00BFC60E-B3D5-44A0-AF25-FF8D9440E6D2@sun.com> Hello! On Mar 30, 2009, at 10:42 AM, Brian J. Murrell wrote: >> It recently struck me as how unfair many file creation tests are >> to Lustre. The problem is only with tests because they portray Lustre >> creation rates lower than actual speed to be seen by application. >> (same goes for opens, btw). > The only thing I would add/say Oleg, is have you looked at the Hendrix > CMD specific metadata tests? I only mention this because from what I > recall, we took advantage of every opportunity we could (within the > requirements of the project) to show performance in the best light we > could. I'd imagine that those tests (and the corresponding Lustre > code_ > would have considered some of the same things you are considering. Yes. As I said, I am aware CMD project tried this idea and found it to be of no help. I am not deeply aware of the details, though. > It may be however that some of the solutions were CMD specific and not > really applicable to you, but perhaps in the least, looking at some of > those tests (i.e. mdsrate was one key test) might prevent you from > re-inventing tests. I am not reinventing tests, I just want to make existing tests to show more fair number when run on Lustre. Bye, Oleg From adilger at sun.com Tue Mar 31 18:51:11 2009 From: adilger at sun.com (Andreas Dilger) Date: Tue, 31 Mar 2009 12:51:11 -0600 Subject: [Lustre-devel] Oleg/Mike Work on Apps Metrics - FW: Mike Booth week ending 2009.03.15 In-Reply-To: <5CE22E46-771C-47EE-9CAC-26A37C87C466@sun.com> References: <015c01c9a5a7$cb2f83a0$618e8ae0$@com> <52E982EB-3ADE-46FA-A7BC-D1645961DF0B@Sun.COM> <018101c9a5b6$25dbc300$71934900$@com> <48A6CF99-F630-4581-9FFC-05E3AFF66FA0@Sun.COM> <023201c9a636$a987e4b0$fc97ae10$@com> <5CE22E46-771C-47EE-9CAC-26A37C87C466@sun.com> Message-ID: <20090331185111.GL3199@webber.adilger.int> On Mar 18, 2009 16:31 -0400, Oleg Drokin wrote: > On Mar 16, 2009, at 8:56 AM, Eric Barton wrote: > > I _do_ agree that for some apps, if there was sufficient memory on the > > app node to buffer the local component of a checkpoint and let it > > "dribble" out to disk would achieve better utilization of the compute > > resource. However parallel apps can be very sensitive to "noise" on > > the network they're using for inter- process communication - i.e. the > > checkpoint data has either to be written all the way to disk, or at > > least buffered somewhere so that moving it to disk will not interfere > > with the app's own communications. > > This latter concept is the basis for the "flash cache" concept. > > Actually, I think it's worth exploring the economics of it in more > > detail. > > This turns out to be a very true assertion. We (I) do see a huge delay > in e.g. MPI barriers done immediately after write. While this is true, I still believe that the amount of delay seen by the application cannot possibly be worse than waiting for all of the IO to complete. Also, the question is whether you are measuring the FIRST MPI barrier after the write, vs e.g. the SECOND MPI barrier after the write? Since Lustre is currently aggressively flushing the write cache then the first MPI barrier is essentially waiting for all of the IO to complete, which is of course very slow. The real item of interest is how long the SECOND MPI barrier takes, which is what the overhead of Lustre IO is on the network performance. It is impossible that Lustre IO completely saturates the entire cross-sectional bandwidth of the system OR the client CPUs, so having some amount of computation for "free" during IO is still better than waiting for the IO to complete. For example, we have 1000 processes each doing a 1GB write to their own file, and the aggregate IO bandwidth is 10GB/s the IO will take about 100s to write if (as we currently do) limit the amount of dirty data on each client to avoid interfering with the application, and no computation can be done during this time. If we allowed clients to cache that 1GB of IO it might only take 1s to complete the "write" and then 99s to flush the IO to the OSTs. If each compute timestep takes 0.1s during IO vs 0.01s without IO and you would get 990 timesteps during the write flush in the second case until the cache was cleared, vs. none in the first case. I suspect that the overhead of the MPI communication on the Lustre IO is small, since the IO will be limited by the OST network and disk bandwidth, which is generally a small fraction of the cross-sectional bandwidth. This could be tested fairly easily with a real application that is doing computation between IO, instead of a benchmark that is only doing IO or only sleeping between IO, simply by increasing the per-OSC write cache limit from 32MB to e.g. 1GB in the above case (or 2GB to avoid the case where 2 processes on the same node are writing to the same OST). Then, measure the time taken for the application to do, say, 1M timesteps and 100 checkpoints with the 32MB and the 2GB write cache sizes. > > The variables are aggregate network bandwidth into the distributed > > checkpoint cache, which determines the checkpoint time, and aggregate > > path-minimum bandwidth (i.e. lesser of network and disk bandwidth) > > from the cache to disk, which determines how soon the cache can be > > ready for the next checkpoint. The cache could be dedicated nodes and > > storage (e.g. flash) or additional storage on the OSSes, or any > > combination of either. And the interesting relationship is how > > compute cluster utilisation varies with the cost of the server and > > cache subsystems. > > The thing is, if we can just flush out data from the cache at the moment > when there is no network-latency critical activity on the app side > (somehow signaled by the app), why would we need the flash storage at > all? We can write nice sequential chunks to normal disks just as fast, > I presume. Actually, an idea I had for clusters that are running many jobs at once was essentially having the apps poll for IO capacity when doing a checkpoint, so that they can avoid contending with other jobs that are doing checkpoints at the same time. That way, an app might skip an occasional checkpoint if the filesystem is busy, and instead compute until the filesystem is less busy. This would be equivalent to the client node being able to cache all of the write data and flushing it out in the background, so long as the time to flush a single checkpoint never took longer than the time between checkpoints. Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc. From Oleg.Drokin at Sun.COM Tue Mar 31 20:58:36 2009 From: Oleg.Drokin at Sun.COM (Oleg Drokin) Date: Tue, 31 Mar 2009 16:58:36 -0400 Subject: [Lustre-devel] Oleg/Mike Work on Apps Metrics - FW: Mike Booth week ending 2009.03.15 In-Reply-To: <20090331185111.GL3199@webber.adilger.int> References: <015c01c9a5a7$cb2f83a0$618e8ae0$@com> <52E982EB-3ADE-46FA-A7BC-D1645961DF0B@Sun.COM> <018101c9a5b6$25dbc300$71934900$@com> <48A6CF99-F630-4581-9FFC-05E3AFF66FA0@Sun.COM> <023201c9a636$a987e4b0$fc97ae10$@com> <5CE22E46-771C-47EE-9CAC-26A37C87C466@sun.com> <20090331185111.GL3199@webber.adilger.int> Message-ID: <179D81CC-3A2F-4D39-9A48-9BFA2F4D6BB0@Sun.COM> Hello! On Mar 31, 2009, at 2:51 PM, Andreas Dilger wrote: >>> This latter concept is the basis for the "flash cache" concept. >>> Actually, I think it's worth exploring the economics of it in more >>> detail. >> This turns out to be a very true assertion. We (I) do see a huge >> delay >> in e.g. MPI barriers done immediately after write. > While this is true, I still believe that the amount of delay seen by > the application cannot possibly be worse than waiting for all of the > IO to complete. Also, the question is whether you are measuring the Absolutely. > FIRST MPI barrier after the write, vs e.g. the SECOND MPI barrier > after the write? Since Lustre is currently aggressively flushing the > write cache then the first MPI barrier is essentially waiting for all > of the IO to complete, which is of course very slow. The real item > of interest is how long the SECOND MPI barrier takes, which is what > the overhead of Lustre IO is on the network performance. Second MPI takes 1.5 seconds for me. > It is impossible that Lustre IO completely saturates the entire > cross-sectional bandwidth of the system OR the client CPUs, so having > some amount of computation for "free" during IO is still better than > waiting for the IO to complete. No arguments about that from me, I am advocating this same thing from the very beginning Bye, Oleg