[Lustre-discuss] Metadata storage in test script files

Roman Grigoryev Roman_Grigoryev at xyratex.com
Mon Apr 30 11:15:32 PDT 2012


Hi Cris,
I'm glad to read next emails on this direction.
Please don't consider this as criticism, I just would like to get more
clearness: what is target of adding this metadata? Do you have plans to
use the metadata in other scripts? How? Does this metadata go to to results?

Also please see more my comments inline:

On 04/30/2012 08:50 PM, Chris wrote:
> Hi,
> 
> Further to previous discussions titled "your opinion about testing" I'd 
> like to propose a meta data format for the test script files and would 
> obviously welcome peoples input;
> 
> Effectively each test in the scripts is represented by a function and a 
> call to run_test, so we have
> 
> test_function() {
>      ...code
> }
> 
> run_test function "Description of the function"
> 
> I'd like to propose that above every function a here document is placed 
> that contains yaml v1.2 encoded data (yaml.org) with 2 characters for 
> the indent. The block will start with << TEST_METADATA and be terminated 
> with TEST_METADATA. We might want to place it in a comment block but 
> this is not really required. The block will also be wrapped at 80 
> characters for readability.
> 
> The compulsory elements to the data will be
> Name:                Name of the function, this ensures pairing between 
> function and comments is not just file relative.
> Summary:          Will often be the description after the run_test but 
> not always as the tense will change
> Description:       A full description of the function, the more 
> information here the better.
> Components:      This is the component described in the commit message 
> (http://wiki.whamcloud.com/display/PUB/Commit+Comments) to make this 
> useful we will need to come up a with a defined set of components that 
> will need to be enforced in the commit message. The format of this entry 
> will be a yaml array.
> Prerequisites:    Pre-requisite tests that must be run before this test 
> can be run. This is again an array which presumes a test may have 
> multiple pre-requisites, but the data should not contain a chain of 
> prerequisites, i.e. if A requires B and B requires C, the pre-requisites 
> of A is B not B & C.

On which step do you want to check chains? And what is logical base for
this prerequisites exclude case that current tests have hidden
dependencies?
 I don't see any difference between one test which have body from tests
a,b,c and this prerequisites definition.
Could you please explain more why we need this field?

> TicketIDs:             This is an array of ticket numbers that this test 
> explicitly tests. In theory we should aim for the state where every 
> ticket has a test associated with it, and in future we should be able to 
> carry out a gap analysis.
> 

I suggest add keywords(Components could be translated as keywords too)
and test type (stress, benchmark, load, functional, negative, etc) for
quick filtering. For example, SLOW could transform to keyword.

Also,  I would like to mention, we have 3 different logical types of data:
1) just human-readable descriptions
2) filtering and targeting fields (Componens, keywords if you agree with
my suggestion)
3) framework directives(Prerequisites)

> As time goes on we may well expand this compulsory list, but this is I 
> believe a sensible starting place.
> 
> Being part of the source this data will be subject to the same review 
> process as any other change and so we cannot store dynamic data here, 
> such as pass rates etc.

What you you think, maybe it is good idea to keep metadata separately?
This can be useful for simplifying changing data via script for mass
modification also as adding tickets and pass rate and execution time on
'gold' configurations?

Thanks,
	Roman

> 
> Do people think that additional data fields should be permitted on an 
> adhoc basis or should a control list of permitted data elements be kept. 
> I'm tempted to say that adhoc additional fields should be allowed, 
> although this could lead to name clashes if people are not careful.
> 
> Below is an simple example.
> 
> =======================================================================
> <<TEST_METADATA
> Name:
>    before_upgrade_create_data
> Summary:
>    Copies lustre source into a node specific directory and then creates 
> a tarball using that directory
> Description:
>    This should be called prior to upgrading Lustre and creates a set of 
> data on the Lustre partition
>    which be accessed and checked after the upgrade has taken place. 
> Several methods are using
>    including tar'ing directories so the can later be untar'ed and 
> compared, along with create sha1's
>    of stored data.
> Component:
>    - lnet
>    - recovery
> Prerequisites:
>    - before_upgrade_clear_filesystem
> TicketIDs:
>    - LU-123
>    - LU-432
> TEST_METADATA
> 
> test_before_upgrade_create_data() {
>     ...
> }
> 
> run_test before_upgrade_create_data "Copying lustre source into a 
> directory $IOP_DIR1, creating and then using source to create a tarball"
> =======================================================================
> 
> As I said comments, inputs and thoughts much appreciated
> 
> Thanks
> 
> Chris
> 
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss




More information about the lustre-discuss mailing list