[Lustre-discuss] lustre-discuss at lists.lustre.org

Chris Gearing chris at whamcloud.com
Wed Dec 22 05:07:45 PST 2010


Hi Aurelien,

That's useful thanks.

Our plan is to create a community database that enables every valuable 
facet of a lustre test to be stored. This means all the results, all the 
logs, stack traces etc etc.  Sometimes we will only store failing logs, 
but for acceptance and release testing we will store passing and 
failing(!) logs. The aim is that when someone is looking into a test 
failure, nothing they require will have been left behind.

A further use of this approach is that we hope to be able to deliver 
landing collateral to reviewers as a standard part of our process. For 
example a patch could have a link to an example of the failing test that 
isolated the bug or missing feature, the code that fixes the failure or 
adds the feature and an example of the passing test showing the bug fix 
or feature addition working. This additional usage is not entirely near 
term, but is a direction that would add great value.

As for yml vs. xml, the reality is that we use the yml as a way of 
transporting the results from the test machine, which could be a 
test-cluster or an individuals laptop running VMs, to the database. If 
xml made this data more widely usable and the requirements of junit do 
not require us to leave information out I can't really see a problem in 
switching to it.

Chris

On 20/12/2010 14:50, DEGREMONT Aurelien wrote:
> Hi Chris,
>
> Here is a rough example of junit report we produce. This was a quick 
> and dirty implementation of Junit that could be improved. Some part of 
> junit report content was limited due to lack of information acceptance 
> small upcalls have.
>
> <?xml version="1.0" encoding="UTF-8" ?>
> <testsuite name="Lustre.acceptance-small">
> <testcase classname="Lustre.sanity" name="Test #0: touch .../ ; rm 
> .../" time="1" />
> <testcase classname="Lustre.sanity" name="Test #0b: chmod 0755 
> /mnt/lustre" time="1" />
> <testcase classname="Lustre.sanity" name="Test #0c: check import proc" 
> time="1" />
> <testcase classname="Lustre.sanity" name="Test #1a: mkdir .../d1; 
> mkdir .../d1/d2" time="1" />
> <testcase classname="Lustre.sanity" name="Test #1b: rmdir .../d1/d2; 
> rmdir .../d1" time="1" />
> <testcase classname="Lustre.sanity" name="Test #2a: mkdir .../d2; 
> touch .../d2/f" time="1" />
> <testcase classname="Lustre.sanity" name="Test #2b: rm -r .../d2; 
> checkstat .../d2/f" time="1" />
> <testcase classname="Lustre.sanity" name="Test #3a: mkdir .../d3" 
> time="1" />
> <testcase classname="Lustre.sanity" name="Test #3b: touch .../d3/f" 
> time="1" />
> <testcase classname="Lustre.sanity" name="Test #3c: rm -r .../d3" 
> time="1" />
> <testcase classname="Lustre.sanity" name="Test #4a: mkdir .../d4" 
> time="1" />
> <testcase classname="Lustre.sanity" name="Test #4b: mkdir .../d4/d2" 
> time="1" />
> ...
> <testcase classname="Lustre.sanity" name="Test #180a: test obdecho on 
> osc" time="80">
> <failure type="FAIL"><![CDATA[rc=1
> test_180a failed with 1]]></failure>
> </testcase>
> ...
> <testcase classname="Lustre.sanity-benchmark" name="Test #dbench: 
> test_dbench" time="0">
> <skipped/>
> </testcase>
> <testcase classname="Lustre.sanity-benchmark" name="Test #dbench: 
> dbench" time="2" />
> ...
> </testsuite>
>
> This is a imple junit report. This could be improved. Partly in 
> improving acceptance-small.sh and test-framework.sh.
> I'm open to switch to any other standard format, supported by Hudson, 
> if this can helps.
>
> What's your needs/plan?
>
> Aurélien
>
> Chris Gearing a écrit :
>> Hi Aurélien,
>>
>> Do you have a specification for the junit test results you produce, 
>> or an example of one of your test results sets. We would be more than
>> willing to pick up and go with something that can be used with a 
>> wider set of tools, with the obvious caveat that it provides 
>> everything needed to completely capture the test results for Lustre 
>> today and in the future.
>>
>> If you have some example results set's that you can forward please 
>> mail them to chris whamcloud.com
>>
>> Thanks
>>
>> Chris
>>
>> I see that PerfPublisher uses xml, although this seems to be the only 
>> specification.
>>
>> On 17/12/2010 20:11, Aurélien wrote:
>>>> Robert Read a écrit :
>>>>> We don't plan to use Hudson to manage our testing results as I 
>>>>> don't think it would scale very well for all the testing we might 
>>>>> do for each build. We're currently building a more custom results 
>>>>> server that's similar (in spirit at least) to the kinds of tools 
>>>>> we had at Oracle.  We'll make it available once it's in 
>>>>> presentable form.
>>>>> Actually, our first step was to replace the acceptance-small.sh 
>>>>> driver script with one that has a more sensible user interface for 
>>>>> running the standard tests.  Since the test-framework.sh on master 
>>>>> has already been changed to produce test results in yaml format, 
>>>>>  the new script collects these with the logs, and is capable of 
>>>>> submitting them to the test results server.   Currently this is 
>>>>> being run manually, though.  Automating the test execution and 
>>>>> connecting all the pieces will be next step.
>>>> Ok. I will be very interested in seeing the final result.
>>>> But I think it is a good idea to stick to standard format and tools 
>>>> as much as possible. This could be a pity if all your new work will 
>>>> be only usable by  your tool.
>>>>
>>>> Junit is quite standard.
>>>> PerfPublisher has its own format due to junit limitations. There is 
>>>> other ones. It could be really good if you do not create a new one.
>>>>
>>>> And indeed, acc-sm is a bit limited and improve it could be really 
>>>> interesting.
>>>>
>>>>
>>>> Aurélien
>>
>> _______________________________________________
>> Lustre-discuss mailing list
>> Lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>




More information about the lustre-discuss mailing list