[Lustre-discuss] coverage measurement for some lustre test suites

[Roman Grigoryev]

Gearing, Chris <chris.gearing at intel.com> wrote at  Mon, 23 Jul 2012  
16:19:27 +0400:

>>> On 2012-07-18, at 6:49 AM, Roman Grigoryev wrote:
>>>> we did some work for collecting code coverage which generate some  
>>>> Lustre
>>>> tests.
>>>>
>>>> Results are available on opensfs site
>>>> http://www.opensfs.org/foswiki/bin/view/Lustre/CodeCoverage ,
>>>> please look and comment it.
>>>
>>> This looks quite interesting.  I guess the next step is to figure out
>>> which functions are receiving no coverage during testing, and write
>>> tests to exercise them.
>>
>> This is the most important one of possible ways for using this info.
>>
>> It is possible to use this info for finding probably death code. It will
>> be  good to check it with static analyzer like Coverity or Polyspace  
>> too.
>>
>> Also, I think, it could be helpful to know which test cover which code
>> (for example for developers, for quick testing new code after fix before
>> uploading for review/wide testing and maybe put changes to these  
>> tests). I
>> have coverage by test and have script for searching test by code line.
>> In future, we could create script for searching tests with maximum
>> codecoverage(and probably we could update SLOW subset) and searching  
>> tests
>> with  minimum coverage or duplicate coverage  and possible refactor them
>>
>> Maybe this can be used for proving orientation of benchmark/stress tests
>> when we will have coverage for them.
>>
>>>
>>> I suspect one type of function which gets very little testing is the
>>> lprocfs functions.  Writing a simple test to read all of the functions
>>> would get coverage, but it would be better to have actual tests that
>>> verify the content of the files is actually correct.
>>
>> Absolutely agree that  just coverage is not enough for prove quality. In
>> coverage theory percent of covered set of all combination of execution
>> paths should be used for proving quality but this 'spherical cow' for  
>> the
>> most of application.
>>
>> I think,  in real life we should use rule like this 'dummy coverage is
>> better than zero coverage but coverage which provide real use case is
>> better dummy coverage'.
>>
>> Ideally, I prefer to have real use cases in tests, dummy coverage in  
>> unit
>> tests and also use advanced static code analyzers for all code.
>>
>> --
>> Thanks,
>>        Roman
>
> I think this is very valuable data and something we should update over  
> time so
> that we can see which way it is trending. Hopefully the effort of a  
> repeat run
> is much less than the original which I sure was quite significant.
>
> On the subject of coverage many people are sceptical of code coverage  
> results,
> and whilst it is true to that that 100% coverage doesn't mean everything  
> is
> tested, 0% does mean nothing is tested. I make this point because what  
> we can
> say is that overtime the % coverages should all be increasing because  
> it's
> difficult to image a case where less coverage is better testing.

I absolutely agree that code coverage is metric which need pretty accurate  
interpretation.

>
> Returning to the specifics of this set, you presumably have the
> report currently on opensfs for any given test, i.e. we could have page  
> that listed
> all ofthe lustre suites/tests that can then be clicked through to the  
> underlying
> report. Is that the case and if would you be able to publish it.

I have only summary html report and set of lcov trace files(one per test)  
and own script for searching with it. Generating lcov report for every  
test is pretty big task and produce a lot of output. Locally I can  
generate report for every test. Maybe it is good idea to just provide  
tools and lcov trace files?


>
> In fact as you say flipping the tree on it's head it should be possible  
> to click
> on any line and find out which tests exercise that line, which would be  
> cool. Of
> course we have to be careful here that data paths are well as code paths  
> need to
> be tested for any change, and data paths are real tricky to map.

For solving this problem lcov report keeps internally sources which were  
used for collecting coverage. Also I wrote few own scripts for merging and  
transforming lcov for simplifying report generation on new place.

-- 
Thanks,
	Roman