[Lustre-discuss] Metadata storage in test script files

[Chris Gearing]

Hi Roman,

I think we may have rat-holed here and perhaps it's worth just 
re-stating what I'm trying to achieve here.

We have a need to be able to test in a more directed and targeted 
manner, to be able to focus on a unit of code like lnet or an attribute 
of capability like performance. However since starting work on the 
Lustre test infrastructure it has become clear to me that knowledge 
about the capability, functionality and purpose of individual tests is 
very general and held in the heads of Lustre engineers. Because we are 
talking about targeting tests we require knowledge about the capability, 
functionality and purpose of the tests not the outcome of running the 
tests, or to put it another way what the tests can do not what they have 
done.

One key fact about cataloguing the the capabilities of the tests is that 
for almost every imaginable case the capability of the test only changes 
if the test itself changes and so the rate of change of the data in the 
catalogue is the same and actually much less than the rate of change 
test code itself. The only exception to this could be that a test 
suddenly discovers a new bug which has to have a new ticket attached to 
it, although this should be a very very rare if we manage our 
development process properly.

This requirement leads to the conclusion that we need to catalogue all 
of the tests within the current test-framework and a catalogue equates 
to a database, hence we need a database of the capability, functionality 
and purpose of the individual tests. With this requirement in mind it 
would be easy to create a database using something like mysql that could 
be used by applications like the Lustre test system, but using an 
approach like that would make the database very difficult to share and 
will be even harder to attach the knowledge to the Lustre tree which is 
were it belongs.

So the question I want to solve is how to catalogue the capabilities of 
the individual tests in a database, store that data as part of the 
Lustre source and as a bonus make the data readable and even carefully 
editable by people as well as machines. Now to focus on the last point I 
do not think we should constrain ourselves to something that can be read 
by machine using just bash, we do have access to structure languages and 
should make use of that fact.

The solution to all of this seemed to be to store the catalogue about 
the tests as part of the tests themselves, this provides for human and 
machine accessibility, implicit version control and certainty the what 
ever happens to Lustre source the data goes with it. It is also the case 
that by keeping the catalogue with the subject the maintenance of the 
catalogue is more likely to occur than if the two are separate.

My original use of the term test metadata is intended as a more modern 
term for catalogue or the [test] library.

So to refresh everybody's mind, I'd like to suggest that we place test 
metadata in the source code itself using the following format, where the 
here doc is inserted into the copy about the test function itself.

=======================================================================
<<TEST_METADATA
Name:
   before_upgrade_create_data
Summary:
   Copies lustre source into a node specific directory and then creates 
a tarball using that directory
Description:
   This should be called prior to upgrading Lustre and creates a set of 
data on the Lustre partition
   which be accessed and checked after the upgrade has taken place. 
Several methods are using
   including tar'ing directories so the can later be untar'ed and 
compared, along with create sha1's
   of stored data.
Component:
   - lnet
   - recovery
Prerequisites:
   - before_upgrade_clear_filesystem
TicketIDs:
   - LU-123
   - LU-432
Purposes:
   - upgrade
TEST_METADATA

test_before_upgrade_create_data() {
    ...
}

run_test before_upgrade_create_data "Copying lustre source into a 
directory $IOP_DIR1, creating and then using source to create a tarball"
=======================================================================

Again thoughts and input very much appreciated

Chris