[Lustre-discuss] Metadata storage in test script files
chris at whamcloud.com
Tue May 1 09:17:58 PDT 2012
On 30/04/2012 19:15, Roman Grigoryev wrote:
> Hi Cris,
> I'm glad to read next emails on this direction.
> Please don't consider this as criticism, I just would like to get more
> clearness: what is target of adding this metadata? Do you have plans to
> use the metadata in other scripts? How? Does this metadata go to to results?
> Also please see more my comments inline:
The metadata can be used in a multitude of ways, for example we can
create dynamic test sets based on
the changes made or target area of testing. What we are doing here is
creating an understanding of the
tests that we have so that we can improve our processes and testing
capabilities in the future.
The metadata does not go to the results. The metadata is a database in
it's own right and should metadata
about a test be required it would be accessed from the source (database)
> On 04/30/2012 08:50 PM, Chris wrote:
... snip ...
>> Prerequisites: Pre-requisite tests that must be run before this test
>> can be run. This is again an array which presumes a test may have
>> multiple pre-requisites, but the data should not contain a chain of
>> prerequisites, i.e. if A requires B and B requires C, the pre-requisites
>> of A is B not B& C.
> On which step do you want to check chains? And what is logical base for
> this prerequisites exclude case that current tests have hidden
> I don't see any difference between one test which have body from tests
> a,b,c and this prerequisites definition.
> Could you please explain more why we need this field?
As I said we can mine this data any-time and anyway that we want, and
the purpose of this
discussion is the data not how we use it. But as an example something
that dynamically built
test sets would need to know prerequisites.
The suffix of a,b,c could be used to generate prerequisite information
but it is firstly inflexible, for example
I bet 'b','c' and 'd' are often dependent on 'a' but not each other,
secondly and more importantly we want a
standard form for storing metadata because we want to introduce order
and knowledge into the test
scripts that we have today.
>> TicketIDs: This is an array of ticket numbers that this test
>> explicitly tests. In theory we should aim for the state where every
>> ticket has a test associated with it, and in future we should be able to
>> carry out a gap analysis.
> I suggest add keywords(Components could be translated as keywords too)
> and test type (stress, benchmark, load, functional, negative, etc) for
> quick filtering. For example, SLOW could transform to keyword.
This seems like a reasonable idea although we need a name that describes
what it is,
we will need to define that set of possible words as we need to with the
What should this field be called - we should not reduce the value of
this data why genericizing it
> Also, I would like to mention, we have 3 different logical types of data:
> 1) just human-readable descriptions
> 2) filtering and targeting fields (Componens, keywords if you agree with
> my suggestion)
> 3) framework directives(Prerequisites)
>> As time goes on we may well expand this compulsory list, but this is I
>> believe a sensible starting place.
>> Being part of the source this data will be subject to the same review
>> process as any other change and so we cannot store dynamic data here,
>> such as pass rates etc.
> What you you think, maybe it is good idea to keep metadata separately?
> This can be useful for simplifying changing data via script for mass
> modification also as adding tickets and pass rate and execution time on
> 'gold' configurations?
It would be easier to store the data separately and we could use Maloo
but it's very important
that this data becomes part of the Lustre 'source' so that everybody can
benefit from it. Adding
tickets is not a problem as part of the resolution issue is to ensure
that at least one test exercises
the problem and proves it has been fixed, the fact that this assurance
process requires active
interaction by an engineer with the scripts is a positive.
As for pass rate, execution time and gold configurations this
information is just not 1 dimensional
enough to store in the source.
More information about the lustre-discuss