[Lustre-discuss] Metadata storage in test script files

[Chris]

On 30/04/2012 19:15, Roman Grigoryev wrote:
> Hi Cris,
> I'm glad to read next emails on this direction.
> Please don't consider this as criticism, I just would like to get more
> clearness: what is target of adding this metadata? Do you have plans to
> use the metadata in other scripts? How? Does this metadata go to to results?
>
> Also please see more my comments inline:
The metadata can be used in a multitude of ways, for example we can 
create dynamic test sets based on
the changes made or target area of testing. What we are doing here is 
creating an understanding of the
tests that we have so that we can improve our processes and testing 
capabilities in the future.

The metadata does not go to the results. The metadata is a database in 
it's own right and should metadata
about a test be required it would be accessed from the source (database) 
itself.

> On 04/30/2012 08:50 PM, Chris wrote:
... snip ...

>> Prerequisites:    Pre-requisite tests that must be run before this test
>> can be run. This is again an array which presumes a test may have
>> multiple pre-requisites, but the data should not contain a chain of
>> prerequisites, i.e. if A requires B and B requires C, the pre-requisites
>> of A is B not B&  C.
> On which step do you want to check chains? And what is logical base for
> this prerequisites exclude case that current tests have hidden
> dependencies?
>   I don't see any difference between one test which have body from tests
> a,b,c and this prerequisites definition.
> Could you please explain more why we need this field?
As I said we can mine this data any-time and anyway that we want, and 
the purpose of this
discussion is the data not how we use it. But as an example something 
that dynamically built
test sets would need to know prerequisites.

The suffix of a,b,c could be used to generate prerequisite information 
but it is firstly inflexible, for example
I bet 'b','c' and 'd' are often dependent on 'a' but not each other, 
secondly and more importantly we want a
standard form for storing metadata because we want to introduce order 
and knowledge into the test
scripts that we have today.

>> TicketIDs:             This is an array of ticket numbers that this test
>> explicitly tests. In theory we should aim for the state where every
>> ticket has a test associated with it, and in future we should be able to
>> carry out a gap analysis.
>>
> I suggest add keywords(Components could be translated as keywords too)
> and test type (stress, benchmark, load, functional, negative, etc) for
> quick filtering. For example, SLOW could transform to keyword.
This seems like a reasonable idea although we need a name that describes 
what it is,
we will need to define that set of possible words as we need to with the 
Components elements.

What should this field be called - we should not reduce the value of 
this data why genericizing it
into 'keywords'.
> Also,  I would like to mention, we have 3 different logical types of data:
> 1) just human-readable descriptions
> 2) filtering and targeting fields (Componens, keywords if you agree with
> my suggestion)
> 3) framework directives(Prerequisites)
>
>> As time goes on we may well expand this compulsory list, but this is I
>> believe a sensible starting place.
>>
>> Being part of the source this data will be subject to the same review
>> process as any other change and so we cannot store dynamic data here,
>> such as pass rates etc.
> What you you think, maybe it is good idea to keep metadata separately?
> This can be useful for simplifying changing data via script for mass
> modification also as adding tickets and pass rate and execution time on
> 'gold' configurations?
It would be easier to store the data separately and we could use Maloo 
but it's very important
that this data becomes part of the Lustre 'source' so that everybody can 
benefit from it. Adding
tickets is not a problem as part of the resolution issue is to ensure 
that at least one test exercises
the problem and proves it has been fixed, the fact that this assurance 
process requires active
interaction by an engineer with the scripts is a positive.

As for pass rate, execution time and gold configurations this 
information is just not 1 dimensional
enough to store in the source.

Chris