[Lustre-devel] [Twg] your opinion about testing improvements (was Lustre-devel Digest, Vol 72, Issue 17)

[Alex Tomas]

while technically this is possible, the framework will have to support
different versions of Lustre
because in some cases they implement functionality different ways with
different side effects
(often used in t-f). which kind of contradicts with the original idea ?

thanks, Alex

On Thu, Apr 5, 2012 at 12:40 PM, Alexander Lezhoev
<Alexander_Lezhoev at xyratex.com> wrote:
> Hi Chris,
>
> I completely agree that the test-framework must be significantly revised. We
> have some plans to improve it and I think, it would be useful to share our
> ideas and visions of this task. Really we considered the separation of tests
> and Lustre code as a part of the framework improvement. So I think, we need
> to share our approaches and develop a conception which would be satisfactory
> for all. What do you think about opening a discussion about new
> test-framework?
>
> --
> Alexander Lezhoev.
> Morpheus test team.
> Xyratex.
>
>
>
>
> On 04/03/2012 06:21 PM, Chris Gearing wrote:
>>
>>
>> We don't have a single script because the tests are at times very tightly
>> coupled to the Lustre version. There were a lot of changes between 1.8.x and
>> 2.x and a lot of corresponding changes to the test scripts. Where the tests
>> are the same and bugs were found in the 2.x test scripts these should have
>> been backported to the 1.8.x test scripts if this was not done then we
>> should do it for inclusion into the 1.8.8 release.
>>
>> The notion of making 'master' scripts work with with all versions is
>> obviously possible but it is a very significant task and given that the
>> scripts themselves are written in a language (sic) that does not provide
>> structure a single script strategy is likely to create many more
>> 'interoperability issues' than it fixes.
>>
>> Also it's worth considering that we have best part of a 1000 discrete
>> changes, whenever a test is re-engineered the test itself must be proven to
>> detect failure as well as success. i.e. If someone produced a version
>> independent test set that passed all versions we would not know that the
>> process was a success, we would need to check that each re-engineered test
>> 'failed' appropriately for each Lustre version, this is a big task that I
>> doubt can be properly achieved in bash.
>>
>> So in summary the best solution given what we have today is to back port
>> fixes to the test scripts as we back port fixes to the code. This is an
>> investment in time and requires the same discipline to test as we have for
>> coding. A single set of scripts that caters for all versions appears I
>> believe like an easy solution but actually would require huge investment
>> that would be better spent developing a modern test framework and
>> infrastructure that can support Lustre for the next ten years.
>>
>
>
> _______________________________________________
> twg mailing list
> twg at lists.opensfs.org
> http://lists.opensfs.org/listinfo.cgi/twg-opensfs.org