[Lustre-discuss] Metadata storage in test script files

[Chris]

Hi,

Further to previous discussions titled "your opinion about testing" I'd 
like to propose a meta data format for the test script files and would 
obviously welcome peoples input;

Effectively each test in the scripts is represented by a function and a 
call to run_test, so we have

test_function() {
     ...code
}

run_test function "Description of the function"

I'd like to propose that above every function a here document is placed 
that contains yaml v1.2 encoded data (yaml.org) with 2 characters for 
the indent. The block will start with << TEST_METADATA and be terminated 
with TEST_METADATA. We might want to place it in a comment block but 
this is not really required. The block will also be wrapped at 80 
characters for readability.

The compulsory elements to the data will be
Name:                Name of the function, this ensures pairing between 
function and comments is not just file relative.
Summary:          Will often be the description after the run_test but 
not always as the tense will change
Description:       A full description of the function, the more 
information here the better.
Components:      This is the component described in the commit message 
(http://wiki.whamcloud.com/display/PUB/Commit+Comments) to make this 
useful we will need to come up a with a defined set of components that 
will need to be enforced in the commit message. The format of this entry 
will be a yaml array.
Prerequisites:    Pre-requisite tests that must be run before this test 
can be run. This is again an array which presumes a test may have 
multiple pre-requisites, but the data should not contain a chain of 
prerequisites, i.e. if A requires B and B requires C, the pre-requisites 
of A is B not B & C.
TicketIDs:             This is an array of ticket numbers that this test 
explicitly tests. In theory we should aim for the state where every 
ticket has a test associated with it, and in future we should be able to 
carry out a gap analysis.

As time goes on we may well expand this compulsory list, but this is I 
believe a sensible starting place.

Being part of the source this data will be subject to the same review 
process as any other change and so we cannot store dynamic data here, 
such as pass rates etc.

Do people think that additional data fields should be permitted on an 
adhoc basis or should a control list of permitted data elements be kept. 
I'm tempted to say that adhoc additional fields should be allowed, 
although this could lead to name clashes if people are not careful.

Below is an simple example.

=======================================================================
<<TEST_METADATA
Name:
   before_upgrade_create_data
Summary:
   Copies lustre source into a node specific directory and then creates 
a tarball using that directory
Description:
   This should be called prior to upgrading Lustre and creates a set of 
data on the Lustre partition
   which be accessed and checked after the upgrade has taken place. 
Several methods are using
   including tar'ing directories so the can later be untar'ed and 
compared, along with create sha1's
   of stored data.
Component:
   - lnet
   - recovery
Prerequisites:
   - before_upgrade_clear_filesystem
TicketIDs:
   - LU-123
   - LU-432
TEST_METADATA

test_before_upgrade_create_data() {
    ...
}

run_test before_upgrade_create_data "Copying lustre source into a 
directory $IOP_DIR1, creating and then using source to create a tarball"
=======================================================================

As I said comments, inputs and thoughts much appreciated

Thanks

Chris