<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><br></div><div><br></div><div>On the 4/5 TWG concall we discussed this further, and came up with 5 items that we would</div><div>like to address.</div><div><div><br></div><div>discussion:</div><div> Xyratex started by suggesting that the test suite included with every</div><div> Lustre release needed reworking. This led to a group conversation</div><div> that focused on the following areas:</div><div> </div><div> 1) refactor unused tests</div><div> Test scripts have grown unwieldly and do not form a coherent</div><div> test package. As a result, there are some tests that are</div><div> used regularly and others that ineffective and skipped. We</div><div> should remove the unused tests.</div><div> </div><div> 2) create single test suite</div><div> Each release has its own test suite and assumes that the client</div><div> and server will use the same test version. It is becoming common</div><div> today to have different client/server versions. In this case, it</div><div> is unclear which tests from which release package should be used.</div><div> Instead, we should have a single test suite that accommodates</div><div> different combinations of client and server versions. </div><div> </div><div> 3) extract individual tests from the scripts</div><div> Many of the tests are run as a suite. There needs to be some</div><div> mechanism to allow running components of the suite independently.</div><div> To do this, we will need to make clear the dependencies of the</div><div> subtests so that users know a test can be run only after another</div><div> component of the test has executed. Also mentioned was the need</div><div> for a common header on all test results to facilitate</div><div> post-processing.</div><div> </div><div> 4) client failure shouldn't stop the tests</div><div> The goal is never to have test failures, nonethelees failures</div><div> currently can stall test progress. The suites need to be written</div><div> to allow for client failures. </div><div> </div><div> 5) increase code coverage</div><div> There are new tests that we should consider adding to the Lustre</div><div> suite in order to increase code coverage. Suggestions were xfstest</div><div> and some of the MPI tests. </div></div><div><br></div><div><br></div><div>Obviously this is not as bold as Chris' statement, which, by the way, I am happy to entertain as well. Do others have anything to add to the list above, or thoughts on proceeding with a completely new architecture?</div><div><br></div><div><br></div><div>On Apr 8, 2012, at 5:04 PM, Chris Gearing wrote:</div><blockquote type="cite"><p>Hi Nathan,</p><p>Please excuse the lack of included context but I think it's fair to say that the current test-framework and scripts are at the end of their evolutionary life and that whilst they will be required to fulfil a role in test for the foreseeable future what is required is a vastly more capable and scalable approach to testing Lustre in particular and massively parallel exascale file systems in general.</p><p>Thanks</p><p>Chris</p></blockquote><div><p><br></p><p><br></p></div><div><div>On Apr 3, 2012, at 7:21 AM, Chris Gearing wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div>On 03/04/2012 07:07, Roman Grigoryev wrote:<br><blockquote type="cite">Hi Chris,<br></blockquote><blockquote type="cite">Thank you for answer ( I have cut part of my original message):<br></blockquote><blockquote type="cite"><blockquote type="cite">When we run interop tests the test system runs test scripts belonging to<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">the server version against those belonging to the client version. So we<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">might use 1.8.7 client scripts against 2.2 server scripts. These scripts<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">need to inter-operate in exactly the same way that the Lustre source<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">code itself needs to interoperate.<br></blockquote></blockquote><blockquote type="cite">Yes, it is. But I don't see why we should use old test base for<br></blockquote><blockquote type="cite">interoperability testing? Between 1.8.7 and 2.x tests was fixed and also<br></blockquote><blockquote type="cite">as test framework was changed. For getting same test coverage for old<br></blockquote><blockquote type="cite">features we should backport new fixes in test to old (maybe already<br></blockquote><blockquote type="cite">frozen) code.<br></blockquote><blockquote type="cite">Also, as results, we have different tests sets for compatibility<br></blockquote><blockquote type="cite">testing. For 1.8.7 it will one, for 2.1 - other. Only a part of<br></blockquote><blockquote type="cite">differences shows difference between code base for one feature set.<br></blockquote><blockquote type="cite">(F.e. we see on special 1.8.7 branch failures which already fixed in 2.x<br></blockquote><blockquote type="cite">code.)<br></blockquote><blockquote type="cite"><br></blockquote>We don't have a single script because the tests are at times very <br>tightly coupled to the Lustre version. There were a lot of changes <br>between 1.8.x and 2.x and a lot of corresponding changes to the test <br>scripts. Where the tests are the same and bugs were found in the 2.x <br>test scripts these should have been backported to the 1.8.x test scripts <br>if this was not done then we should do it for inclusion into the 1.8.8 <br>release.<br><br>The notion of making 'master' scripts work with with all versions is <br>obviously possible but it is a very significant task and given that the <br>scripts themselves are written in a language (sic) that does not provide <br>structure a single script strategy is likely to create many more <br>'interoperability issues' than it fixes.<br><br>Also it's worth considering that we have best part of a 1000 discrete <br>changes, whenever a test is re-engineered the test itself must be proven <br>to detect failure as well as success. i.e. If someone produced a version <br>independent test set that passed all versions we would not know that the <br>process was a success, we would need to check that each re-engineered <br>test 'failed' appropriately for each Lustre version, this is a big task <br>that I doubt can be properly achieved in bash.<br><br>So in summary the best solution given what we have today is to back port <br>fixes to the test scripts as we back port fixes to the code. This is an <br>investment in time and requires the same discipline to test as we have <br>for coding. A single set of scripts that caters for all versions appears <br>I believe like an easy solution but actually would require huge <br>investment that would be better spent developing a modern test framework <br>and infrastructure that can support Lustre for the next ten years.<br><br><blockquote type="cite"><br></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Problem 2<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">(to avoid term problems, I call there: sanity = test suite, 130 = test,<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">130c and 130a = test cases)<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite">...<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Answer of this question affect automated test execution and test<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">development, and maybe ask some test-framework changes.<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">I think you highlight a very good point here that we don't really know<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">enough about the test contents, their prerequisites or other<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">dependencies. I would suggest that many attempts have been made over the<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">years to use naming conventions, numeric ordering or other similar<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">mechanisms to track such behaviour.<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">...<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">One reasonable proposal is to add a comment block at the start of each<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">test script and subtest within that script that lists the test name,<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">short and long description that includes what the test is supposed to be<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">doing, what bug (if any) it was originally added for, what part of the<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">code it is intended to cover, prerequisites (filesystem initialization,<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">min/max number of clients, OSTs, MDTs it can test with, etc) in a<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">machine readable format that it not only documents the test today but<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">that can be expanded in the future.<br></blockquote></blockquote><blockquote type="cite">I agree, it is very important to separating meta information and test body.<br></blockquote><blockquote type="cite">Internally in Xyratex, we use external scripts and descriptors which<br></blockquote><blockquote type="cite">somehow add same possibility(per-test timeouts, keywords...).<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><blockquote type="cite">Once we have an agreement on an initial format for this comment block,<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">the development community can work to populate it for each subtest and<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">improve the understanding and usefulness of all existing tests.<br></blockquote></blockquote><blockquote type="cite">I absolutely agree that we need agreement to start any work on test<br></blockquote><blockquote type="cite">improvements. How can we initiate this process? Maybe good first step is<br></blockquote><blockquote type="cite">creating glossary to use and terms and based on these terms fix tests?<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Also, what do you think about a possible simple solutions for decreasing<br></blockquote><blockquote type="cite">dependence problem which is currently pretty painful for us:<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">1) test(test scenario) must have only number name (1,2,3..110...999)<br></blockquote><blockquote type="cite">2) test cases (test step) must have number+char index (1f,2,b...99c)<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Only Test can be executed via ONLY.<br></blockquote><blockquote type="cite">Test cases can be execute only as part of test.<br></blockquote>I don't think there is a problem with this simple solution in that it <br>does no harm as long as you applied any changes to all the branches that <br>are applicable. At the same time I will draft a possible meta data <br>format that includes the extensible metadata within the source in a way <br>that maximizes its value both today and in the future, we can then <br>review, revise and then agree that format on Lustre-Devel, although I'll <br>mail you privately so you can have input before that. It may actually be <br>the case that some work has occurred on this topic previously and if so <br>we can leverage that.<br><br>Thanks<br><br>Chris Gearing<br>Sr. Software Engineer<br>Quality Engineering<br>Whamcloud Inc<br>_______________________________________________<br>Lustre-devel mailing list<br><a href="mailto:Lustre-devel@lists.lustre.org">Lustre-devel@lists.lustre.org</a><br>http://lists.lustre.org/mailman/listinfo/lustre-devel<br></div></blockquote></div><br></body></html>