From tuve at swedisk.se Thu Sep 10 12:34:33 2009 From: tuve at swedisk.se (Tuve Nordius) Date: Thu, 10 Sep 2009 14:34:33 +0200 Subject: [Lustre-devel] Mac OS client Message-ID: <064780CA-3CD7-4FD6-8967-0871DA3D6A20@swedisk.se> Hi I am happy to announce that I have a pre alpha lustre client for mac os x, based on liblustre (lustre version 1.9.240) and mac fuse. From tuve at swedisk.se Thu Sep 10 12:52:35 2009 From: tuve at swedisk.se (Tuve Nordius) Date: Thu, 10 Sep 2009 14:52:35 +0200 Subject: [Lustre-devel] Multithreaded liblustre Message-ID: <68880AC1-3DDB-4B20-AE49-55227D64F295@swedisk.se> Hi Does anyone know what need to be done/ the main obstacles in order to make liblustre multithreaded? Best Regards Tuve Nordius From Oleg.Drokin at Sun.COM Thu Sep 10 18:39:58 2009 From: Oleg.Drokin at Sun.COM (Oleg Drokin) Date: Thu, 10 Sep 2009 14:39:58 -0400 Subject: [Lustre-devel] Multithreaded liblustre In-Reply-To: <68880AC1-3DDB-4B20-AE49-55227D64F295@swedisk.se> References: <68880AC1-3DDB-4B20-AE49-55227D64F295@swedisk.se> Message-ID: Hello! On Sep 10, 2009, at 8:52 AM, Tuve Nordius wrote: > Does anyone know what need to be done/ the main obstacles in order to > make liblustre multithreaded? What sort of multi-threadness are you looking at? Currently socklnd has separate threads if I remember right. There are no callbacks and other async stuff happening, but that is pretty much by design so would be hard to change. Bye, Oleg From lee at sandia.gov Thu Sep 10 19:03:05 2009 From: lee at sandia.gov (Lee Ward) Date: Thu, 10 Sep 2009 13:03:05 -0600 Subject: [Lustre-devel] Multithreaded liblustre In-Reply-To: References: <68880AC1-3DDB-4B20-AE49-55227D64F295@swedisk.se> Message-ID: <1252609385.3953.41.camel@wheel> On Thu, 2009-09-10 at 12:39 -0600, Oleg Drokin wrote: > Hello! > > On Sep 10, 2009, at 8:52 AM, Tuve Nordius wrote: > > Does anyone know what need to be done/ the main obstacles in order to > > make liblustre multithreaded? > > What sort of multi-threadness are you looking at? > Currently socklnd has separate threads if I remember right. > There are no callbacks and other async stuff happening, but that is > pretty much by design so would be hard to change. If you are talking about the API, the current head for SYSIO is thread-safe for all but initialization, mount, unmount, and shutdown. Don't think it works for liblustre at present, but rklundt at sandia.gov is currently doing a port, maybe done already. Re-entrancy is to be supported some time in 2010. But... I thought SUN had dropped support for liblustre entirely? Did I miss something? --Lee > > Bye, > Oleg > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel > From Oleg.Drokin at Sun.COM Thu Sep 10 19:06:48 2009 From: Oleg.Drokin at Sun.COM (Oleg Drokin) Date: Thu, 10 Sep 2009 15:06:48 -0400 Subject: [Lustre-devel] Multithreaded liblustre In-Reply-To: <1252609385.3953.41.camel@wheel> References: <68880AC1-3DDB-4B20-AE49-55227D64F295@swedisk.se> <1252609385.3953.41.camel@wheel> Message-ID: <86344224-0630-481D-B40F-87ED6D18D408@Sun.COM> Hello! On Sep 10, 2009, at 3:03 PM, Lee Ward wrote: >> On Sep 10, 2009, at 8:52 AM, Tuve Nordius wrote: >>> Does anyone know what need to be done/ the main obstacles in >>> order to >>> make liblustre multithreaded? >> What sort of multi-threadness are you looking at? >> Currently socklnd has separate threads if I remember right. >> There are no callbacks and other async stuff happening, but that is >> pretty much by design so would be hard to change. > If you are talking about the API, the current head for SYSIO is > thread-safe for all but initialization, mount, unmount, and shutdown. > Don't think it works for liblustre at present, but > rklundt at sandia.gov is > currently doing a port, maybe done already. Well, there are many sides to "multithreaded", that's why I am asking. Good to know. > But... I thought SUN had dropped support for liblustre entirely? Did I > miss something? Liblustre is still a first class citizen, fully supported in HEAD (2.0) codebase too, tested along with other stuff at scheduled intervals in our testsuite. I don't even remember anybody proposing dropping liblustre support in recent times. Bye, Oleg From lee at sandia.gov Thu Sep 10 20:10:12 2009 From: lee at sandia.gov (Lee Ward) Date: Thu, 10 Sep 2009 14:10:12 -0600 Subject: [Lustre-devel] Multithreaded liblustre In-Reply-To: <86344224-0630-481D-B40F-87ED6D18D408@Sun.COM> References: <68880AC1-3DDB-4B20-AE49-55227D64F295@swedisk.se> <1252609385.3953.41.camel@wheel> <86344224-0630-481D-B40F-87ED6D18D408@Sun.COM> Message-ID: <1252613412.3953.42.camel@wheel> On Thu, 2009-09-10 at 13:06 -0600, Oleg Drokin wrote: > Hello! > > On Sep 10, 2009, at 3:03 PM, Lee Ward wrote: > >> On Sep 10, 2009, at 8:52 AM, Tuve Nordius wrote: > >>> Does anyone know what need to be done/ the main obstacles in > >>> order to > >>> make liblustre multithreaded? > >> What sort of multi-threadness are you looking at? > >> Currently socklnd has separate threads if I remember right. > >> There are no callbacks and other async stuff happening, but that is > >> pretty much by design so would be hard to change. > > If you are talking about the API, the current head for SYSIO is > > thread-safe for all but initialization, mount, unmount, and shutdown. > > Don't think it works for liblustre at present, but > > rklundt at sandia.gov is > > currently doing a port, maybe done already. > > Well, there are many sides to "multithreaded", that's why I am asking. > Good to know. > > > But... I thought SUN had dropped support for liblustre entirely? Did I > > miss something? > > Liblustre is still a first class citizen, fully supported in HEAD (2.0) > codebase too, tested along with other stuff at scheduled intervals in > our testsuite. > > I don't even remember anybody proposing dropping liblustre support in > recent times. I did and, it turns out, I was mistaken. Thanks. --Lee > > Bye, > Oleg > From tuve at swedisk.se Thu Sep 10 19:59:47 2009 From: tuve at swedisk.se (Tuve Nordius) Date: Thu, 10 Sep 2009 21:59:47 +0200 Subject: [Lustre-devel] Multithreaded liblustre In-Reply-To: <86344224-0630-481D-B40F-87ED6D18D408@Sun.COM> References: <68880AC1-3DDB-4B20-AE49-55227D64F295@swedisk.se> <1252609385.3953.41.camel@wheel> <86344224-0630-481D-B40F-87ED6D18D408@Sun.COM> Message-ID: I guess what I am looking for is re-entrant. The problem is if I try to do any operation (stat, readdir or whatever) while occupied with a large read or write, liblustre does not respond, causing mac fuse to unmount the filesystem. I can of course make macfuse run in a single thread, but that is not what I want. //Tuve 10 sep 2009 kl. 21.06 skrev Oleg Drokin: > Hello! > > On Sep 10, 2009, at 3:03 PM, Lee Ward wrote: >>> On Sep 10, 2009, at 8:52 AM, Tuve Nordius wrote: >>>> Does anyone know what need to be done/ the main obstacles in >>>> order to >>>> make liblustre multithreaded? >>> What sort of multi-threadness are you looking at? >>> Currently socklnd has separate threads if I remember right. >>> There are no callbacks and other async stuff happening, but that is >>> pretty much by design so would be hard to change. >> If you are talking about the API, the current head for SYSIO is >> thread-safe for all but initialization, mount, unmount, and shutdown. >> Don't think it works for liblustre at present, but >> rklundt at sandia.gov is >> currently doing a port, maybe done already. > > Well, there are many sides to "multithreaded", that's why I am asking. > Good to know. > >> But... I thought SUN had dropped support for liblustre entirely? >> Did I >> miss something? > > Liblustre is still a first class citizen, fully supported in HEAD > (2.0) > codebase too, tested along with other stuff at scheduled intervals in > our testsuite. > > I don't even remember anybody proposing dropping liblustre support in > recent times. > > Bye, > Oleg From lee at sandia.gov Tue Sep 15 03:22:52 2009 From: lee at sandia.gov (Lee Ward) Date: Mon, 14 Sep 2009 21:22:52 -0600 Subject: [Lustre-devel] Multithreaded liblustre In-Reply-To: References: <68880AC1-3DDB-4B20-AE49-55227D64F295@swedisk.se> <1252609385.3953.41.camel@wheel> <86344224-0630-481D-B40F-87ED6D18D408@Sun.COM> Message-ID: <1252984972.3969.30.camel@wheel> On Thu, 2009-09-10 at 13:59 -0600, Tuve Nordius wrote: > I guess what I am looking for is re-entrant. > > The problem is if I try to do any operation (stat, readdir or > whatever) while occupied with a large read or write, liblustre does > not respond, causing mac fuse to unmount the filesystem. I can of > course make macfuse run in a single thread, but that is not what I want. The SYSIO IO functions all have asynchronous counterparts, in all releases. If those work, it sounds like you could deal with your issues, now. However, SYSIO is dependent on the client driver supporting asynchronous IO and last time I checked Lustre didn't. Maybe someone here can tell us if that is still true? If Lustre can do it now, you will need to be careful not to allow more than one thread into the SYSIO library at a time unless you use the current development head. Then, look at the IO functions that start with 'i', all returning an ioid_t. You should also look at iopoll and iowait, they are used to test for completion or wait for completion and give you the return code. It's almost always a good idea to call iopoll frequently. --Lee > > //Tuve > > > 10 sep 2009 kl. 21.06 skrev Oleg Drokin: > > > Hello! > > > > On Sep 10, 2009, at 3:03 PM, Lee Ward wrote: > >>> On Sep 10, 2009, at 8:52 AM, Tuve Nordius wrote: > >>>> Does anyone know what need to be done/ the main obstacles in > >>>> order to > >>>> make liblustre multithreaded? > >>> What sort of multi-threadness are you looking at? > >>> Currently socklnd has separate threads if I remember right. > >>> There are no callbacks and other async stuff happening, but that is > >>> pretty much by design so would be hard to change. > >> If you are talking about the API, the current head for SYSIO is > >> thread-safe for all but initialization, mount, unmount, and shutdown. > >> Don't think it works for liblustre at present, but > >> rklundt at sandia.gov is > >> currently doing a port, maybe done already. > > > > Well, there are many sides to "multithreaded", that's why I am asking. > > Good to know. > > > >> But... I thought SUN had dropped support for liblustre entirely? > >> Did I > >> miss something? > > > > Liblustre is still a first class citizen, fully supported in HEAD > > (2.0) > > codebase too, tested along with other stuff at scheduled intervals in > > our testsuite. > > > > I don't even remember anybody proposing dropping liblustre support in > > recent times. > > > > Bye, > > Oleg > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel > From nic at cray.com Thu Sep 24 20:33:18 2009 From: nic at cray.com (Nic Henke) Date: Thu, 24 Sep 2009 15:33:18 -0500 Subject: [Lustre-devel] using LST for performance testing Message-ID: <4ABBD78E.50506@cray.com> Hello, I'm hoping to get a few ideas on how we could modify LST to make doing performance testing easier. Right now we can use "lst stat" to get a rough idea of performance, but the timers are pretty rough and the data is a snapshot. Any ideas ? I've got cycles to do the coding, but not sure what would be the best way to fit this into the existing LST framework. BTW - the ability to dump CSV or some other text file with per-node and per-group data would also be nice. Nic From He.Huang at Sun.COM Mon Sep 28 17:35:54 2009 From: He.Huang at Sun.COM (Isaac Huang) Date: Mon, 28 Sep 2009 13:35:54 -0400 Subject: [Lustre-devel] using LST for performance testing In-Reply-To: <4ABBD78E.50506@cray.com> References: <4ABBD78E.50506@cray.com> Message-ID: <20090928173554.GA4911@sun.com> On Thu, Sep 24, 2009 at 03:33:18PM -0500, Nic Henke wrote: > Hello, > > I'm hoping to get a few ideas on how we could modify LST to make doing > performance testing easier. Right now we can use "lst stat" to get a > rough idea of performance, but the timers are pretty rough and the data > is a snapshot. > > Any ideas ? I've got cycles to do the coding, but not sure what would > be the best way to fit this into the existing LST framework. There's some rough edges in the stat gathering code. First, the LST console has no idea whether the tests have stopped, and that's why the 'lst stat' command by default loops until a ^C. Test clients could return a counter for active test batches and when it drops to 0 all tests on the client must have completed, but servers are passive and have no idea whether clients are done or not. The throughput calculation also could be inaccurate. IIRC, the console just take a snapshot of stat counters on test nodes at a fixed interval (1 second by default), and calculate the throughput as changes in the successive counter snapshots divided by the interval. But, apparently the interval at which the console sends 'get_stat' requests does not equal the interval at which snapshots are taken on test nodes - the 'get_stat' requests could be delayed on the path when the network is stressed (something LST was designed to do), and even worse they could be reordered in the presence of routers. One possible solution would be to include timestamp in the 'get_stat' replies, and calculate the throughput as diffs in counters divided by diffs in timestamps. Since the console only cares about the changes in timestamps, the test nodes clocks do not need to be in sync at all (but they do need to be monotonic and be of a same resolution). The test servers concurrently posts one passive buffer for each request, so for each test request there's one LNetMDAttach and one unlink operation and both operations need to grab the one big LNET_LOCK therefore it could be possible that the server CPU becomes a bottleneck before the network could be saturated. The solution is to, instead of one request per buffer, post one big buffer that could accommodate multiple requests to amortize the per buffer processing costs. Refining these rough edges might likely involve protocol changes. The LST is not a production service so strict backward compatibility is not necessary. I think it'd suffice to do a protocol version check at the time of 'add_node' command and simply refuse to add a test node whose protocol version is different than that of the console. > BTW - the ability to dump CSV or some other text file with per-node and > per-group data would also be nice. That's a good idea, then users could do whatever they'd like to the data. Thanks, Isaac From office at hailoo.com Fri Sep 25 19:49:03 2009 From: office at hailoo.com (office at hailoo.com) Date: Fri, 25 Sep 2009 15:49:03 -0400 Subject: [Lustre-devel] File offsets with multiple clients Message-ID: <20090925194903.F379E20593C1@hailoo.com> When a client obtains a file descriptor to a file on a Lustre file system, is the file offset pointer shared among any connected clients, or is it only meaningful to a single client? For example, given the file "foo.txt" on a Lustre file system, suppose there are 2 clients with an open file descriptor to "foo.txt". If Client1 calls lseek and then write, will this affect the file offset position of Client2? Or is the application responsible for synchronizing file access in this case? -------------- next part -------------- An HTML attachment was scrubbed... URL: From Oleg.Drokin at Sun.COM Tue Sep 29 01:04:41 2009 From: Oleg.Drokin at Sun.COM (Oleg Drokin) Date: Mon, 28 Sep 2009 21:04:41 -0400 Subject: [Lustre-devel] File offsets with multiple clients In-Reply-To: <20090925194903.F379E20593C1@hailoo.com> References: <20090925194903.F379E20593C1@hailoo.com> Message-ID: Hello! On Sep 25, 2009, at 3:49 PM, office at hailoo.com wrote: > When a client obtains a file descriptor to a file on a Lustre file > system, is the file offset pointer shared among any connected > clients, or is it only meaningful to a single client? It's even stricter than that. the pointer is per file-descriptor (the one you get from open(2) system call). > For example, given the file "foo.txt" on a Lustre file system, > suppose there are 2 clients with an open file descriptor to > "foo.txt". If Client1 calls lseek and then write, will this affect > the file offset position of Client2? Or is the application > responsible for synchronizing file access in this case? > No, since both applications on both clients did separate opens, their current file position pointer is separate. The answer does not change even if both processes were running on the same client. This is by design and is mandated by POSIX. Bye, Oleg From nic at cray.com Tue Sep 29 16:51:45 2009 From: nic at cray.com (Nic Henke) Date: Tue, 29 Sep 2009 11:51:45 -0500 Subject: [Lustre-devel] using LST for performance testing In-Reply-To: <20090928173554.GA4911@sun.com> References: <4ABBD78E.50506@cray.com> <20090928173554.GA4911@sun.com> Message-ID: <4AC23B21.2030207@cray.com> Isaac Huang wrote: > On Thu, Sep 24, 2009 at 03:33:18PM -0500, Nic Henke wrote: > >> Hello, >> >> I'm hoping to get a few ideas on how we could modify LST to make doing >> performance testing easier. Right now we can use "lst stat" to get a >> rough idea of performance, but the timers are pretty rough and the data >> is a snapshot. >> >> Any ideas ? I've got cycles to do the coding, but not sure what would >> be the best way to fit this into the existing LST framework. >> > > There's some rough edges in the stat gathering code. First, the LST > console has no idea whether the tests have stopped, and that's why the > 'lst stat' command by default loops until a ^C. Test clients could > return a counter for active test batches and when it drops to 0 all > tests on the client must have completed, but servers are passive and > have no idea whether clients are done or not. > I think the timing of the start/stop of each of the tests is probably the trickiest bit. To get really good end-to-end numbers, we'd need to be able to accurately time each of the tests. > The throughput calculation also could be inaccurate. IIRC, the console > just take a snapshot of stat counters on test nodes at a fixed > interval (1 second by default), and calculate the throughput as > changes in the successive counter snapshots divided by the interval. > But, apparently the interval at which the console sends 'get_stat' > requests does not equal the interval at which snapshots are taken on > test nodes - the 'get_stat' requests could be delayed on the path when > the network is stressed (something LST was designed to do), and even > worse they could be reordered in the presence of routers. One possible > solution would be to include timestamp in the 'get_stat' replies, and > calculate the throughput as diffs in counters divided by diffs in > timestamps. Since the console only cares about the changes in > timestamps, the test nodes clocks do not need to be in sync at all > (but they do need to be monotonic and be of a same resolution). > I'm wondering if we couldn't add a new 'batch_stat' command. The idea is that the client code will fill in the start/stop times for each test and then after the test is done, 'batch_stat' would collect this data. The collection would still be passive and a new command should minimize the protocol changes. The per-test data would allow us to get accurate perf numbers and also provide some data into how parallel the tests were, if there are any unfairness issues, etc. > The test servers concurrently posts one passive buffer for each > request, so for each test request there's one LNetMDAttach and one > unlink operation and both operations need to grab the one big > LNET_LOCK therefore it could be possible that the server CPU becomes a > bottleneck before the network could be saturated. The solution is to, > instead of one request per buffer, post one big buffer that could > accommodate multiple requests to amortize the per buffer processing > costs. > If we added timestamps to the data, the processing time & buffer sizing would be less of an issue - it wouldn't factor into the accuracy of the numbers are are gathering. > Refining these rough edges might likely involve protocol changes. The > LST is not a production service so strict backward compatibility is > not necessary. I think it'd suffice to do a protocol version check at > the time of 'add_node' command and simply refuse to add a test node > whose protocol version is different than that of the console. > > OK. >> BTW - the ability to dump CSV or some other text file with per-node and >> per-group data would also be nice. >> > > That's a good idea, then users could do whatever they'd like to the data. > > Nic From nic at cray.com Tue Sep 29 18:02:25 2009 From: nic at cray.com (Nic Henke) Date: Tue, 29 Sep 2009 13:02:25 -0500 Subject: [Lustre-devel] using LST for performance testing In-Reply-To: <1254245568.5827.5.camel@lap75545.ornl.gov> References: <4ABBD78E.50506@cray.com> <20090928173554.GA4911@sun.com> <4AC23B21.2030207@cray.com> <1254245568.5827.5.camel@lap75545.ornl.gov> Message-ID: <4AC24BB1.8040502@cray.com> David Dillow wrote: > On Tue, 2009-09-29 at 11:51 -0500, Nic Henke wrote: > >> I'm wondering if we couldn't add a new 'batch_stat' command. The idea is >> that the client code will fill in the start/stop times for each test and >> then after the test is done, 'batch_stat' would collect this data. The >> collection would still be passive and a new command should minimize the >> protocol changes. The per-test data would allow us to get accurate perf >> numbers and also provide some data into how parallel the tests were, if >> there are any unfairness issues, etc. >> > > Along these lines, it would be nice if we could specify a run time for > each test rather than an amount of data to be transferred -- it makes it > easier to get aggregate bandwidth numbers, and often shows imbalances > nicely -- the node getting starved is the one that transfers less data. > > It may also make sense to add a 'delay' parameter that causes each test > to wait a specified amount of time from the 'go' signal. This allows the > signal to propagate without running into congestion from the test, > helping to cause all of the clients to start the test closer to > simultaneously. > Interesting - can you elaborate, perhaps in the form of a patch ? :-) I like both ideas, but not signing up to code them just yet. Nic From nic at cray.com Tue Sep 29 18:03:57 2009 From: nic at cray.com (Nic Henke) Date: Tue, 29 Sep 2009 13:03:57 -0500 Subject: [Lustre-devel] using LST for performance testing In-Reply-To: <4AC23B21.2030207@cray.com> References: <4ABBD78E.50506@cray.com> <20090928173554.GA4911@sun.com> <4AC23B21.2030207@cray.com> Message-ID: <4AC24C0D.9030409@cray.com> Nic Henke wrote: > Isaac Huang wrote: > >> On Thu, Sep 24, 2009 at 03:33:18PM -0500, Nic Henke wrote: >> > > I'm wondering if we couldn't add a new 'batch_stat' command. The idea is > So - for a current swag: - I'm thinking of timing the 'test instance' run (sfw_test_instance_t) - this would get timing for each test run end-to-end but be detailed enough to show the different timing with --concurrency > 1 is used. Doing per-RPC timing seemed to heavy and I'm not sure that level of resolution is really needed. - the new command 'test_stat' would collect this data ala the current 'stat' but allow CSV dumping for more fine-grained analysis. I'd probably dump some node & group info as well to make easier to do 'group' stats. Nic From dillowda at ornl.gov Tue Sep 29 17:32:48 2009 From: dillowda at ornl.gov (David Dillow) Date: Tue, 29 Sep 2009 13:32:48 -0400 Subject: [Lustre-devel] using LST for performance testing In-Reply-To: <4AC23B21.2030207@cray.com> References: <4ABBD78E.50506@cray.com> <20090928173554.GA4911@sun.com> <4AC23B21.2030207@cray.com> Message-ID: <1254245568.5827.5.camel@lap75545.ornl.gov> On Tue, 2009-09-29 at 11:51 -0500, Nic Henke wrote: > I'm wondering if we couldn't add a new 'batch_stat' command. The idea is > that the client code will fill in the start/stop times for each test and > then after the test is done, 'batch_stat' would collect this data. The > collection would still be passive and a new command should minimize the > protocol changes. The per-test data would allow us to get accurate perf > numbers and also provide some data into how parallel the tests were, if > there are any unfairness issues, etc. Along these lines, it would be nice if we could specify a run time for each test rather than an amount of data to be transferred -- it makes it easier to get aggregate bandwidth numbers, and often shows imbalances nicely -- the node getting starved is the one that transfers less data. It may also make sense to add a 'delay' parameter that causes each test to wait a specified amount of time from the 'go' signal. This allows the signal to propagate without running into congestion from the test, helping to cause all of the clients to start the test closer to simultaneously. -- Dave Dillow National Center for Computational Science Oak Ridge National Laboratory (865) 241-6602 office From dillowda at ornl.gov Wed Sep 30 13:53:15 2009 From: dillowda at ornl.gov (David Dillow) Date: Wed, 30 Sep 2009 09:53:15 -0400 Subject: [Lustre-devel] using LST for performance testing In-Reply-To: <4AC24BB1.8040502@cray.com> References: <4ABBD78E.50506@cray.com> <20090928173554.GA4911@sun.com> <4AC23B21.2030207@cray.com> <1254245568.5827.5.camel@lap75545.ornl.gov> <4AC24BB1.8040502@cray.com> Message-ID: <1254318795.3114.1.camel@lap75545.ornl.gov> On Tue, 2009-09-29 at 14:02 -0400, Nic Henke wrote: > David Dillow wrote: > > Along these lines, it would be nice if we could specify a run time for > > each test rather than an amount of data to be transferred -- it makes it > > easier to get aggregate bandwidth numbers, and often shows imbalances > > nicely -- the node getting starved is the one that transfers less data. > > > > It may also make sense to add a 'delay' parameter that causes each test > > to wait a specified amount of time from the 'go' signal. This allows the > > signal to propagate without running into congestion from the test, > > helping to cause all of the clients to start the test closer to > > simultaneously. > > > > Interesting - can you elaborate, perhaps in the form of a patch ? :-) I > like both ideas, but not signing up to code them just yet. Maybe, but it is going to be well down the list -- I owe Oleg a large amount of testing before I think of doing anything else. :) -- Dave Dillow National Center for Computational Science Oak Ridge National Laboratory (865) 241-6602 office