[Lustre-discuss] Metadata storage in test script files

[Nathan Rutman]

On May 4, 2012, at 7:46 AM, Chris Gearing wrote:

> Hi Roman,
> 
> I think we may have rat-holed here and perhaps it's worth just 
> re-stating what I'm trying to achieve here.
> 
> We have a need to be able to test in a more directed and targeted 
> manner, to be able to focus on a unit of code like lnet or an attribute 
> of capability like performance. However since starting work on the 
> Lustre test infrastructure it has become clear to me that knowledge 
> about the capability, functionality and purpose of individual tests is 
> very general and held in the heads of Lustre engineers. Because we are 
> talking about targeting tests we require knowledge about the capability, 
> functionality and purpose of the tests not the outcome of running the 
> tests, or to put it another way what the tests can do not what they have 
> done.
> 
> One key fact about cataloguing the the capabilities of the tests is that 
> for almost every imaginable case the capability of the test only changes 
> if the test itself changes and so the rate of change of the data in the 
> catalogue is the same and actually much less than the rate of change 
> test code itself. The only exception to this could be that a test 
> suddenly discovers a new bug which has to have a new ticket attached to 
> it, although this should be a very very rare if we manage our 
> development process properly.
> 
> This requirement leads to the conclusion that we need to catalogue all 
> of the tests within the current test-framework and a catalogue equates 
> to a database, hence we need a database of the capability, functionality 
> and purpose of the individual tests. With this requirement in mind it 
> would be easy to create a database using something like mysql that could 
> be used by applications like the Lustre test system, but using an 
> approach like that would make the database very difficult to share and 
> will be even harder to attach the knowledge to the Lustre tree which is 
> were it belongs.
> 
> So the question I want to solve is how to catalogue the capabilities of 
> the individual tests in a database, store that data as part of the 
> Lustre source and as a bonus make the data readable and even carefully 
> editable by people as well as machines. Now to focus on the last point I 
> do not think we should constrain ourselves to something that can be read 
> by machine using just bash, we do have access to structure languages and 
> should make use of that fact.
> 
I think we all agree 100% on the above...

> The solution to all of this seemed to be to store the catalogue about 
> the tests as part of the tests themselves
... but not necessarily that conclusion.

> , this provides for human and 
> machine accessibility, implicit version control and certainty the what 
> ever happens to Lustre source the data goes with it. It is also the case 
> that by keeping the catalogue with the subject the maintenance of the 
> catalogue is more likely to occur than if the two are separate.

I agree with all those.  But there are some difficulties with this as well:
1. bash isn't a great language to encapsulate this metadata
2. this further locks us in to current test implementation - there's not much possibility to start writing tests in another language if we're parsing through looking for bash-formatted metadata. Sure, multiple parsers could be written...
3. difficulty changing md of groups of tests en-mass - eg. add "slow" keyword to a set of tests
4. no inheritance of characteristics - each test must explicitly list every piece of md.  This not only blows up the amount of md it also is a source for typos, etc. to cause problems.
5. no automatic modification of characteristics.  In particular, one piece of md I would like to see is "maximum allowed test time" for each test.  Ideally, this could be measured and adjusted automatically based on historical and ongoing run data.  But it would be dangerous to allow automatic modification to the script itself.

To address those problems, I think a database-type approach is exactly right, or perhaps a YAML file with hierarchical inheritance.
To some degree, this is a "evolution vs revolution" question, and I prefer to come down on the revolution-enabling design, despite the problems you list.  Basically, I believe the separated MD model allows for the replacement of test-framework, and this, to my mind, is the majority driver for adding the MD at all.


> 
> My original use of the term test metadata is intended as a more modern 
> term for catalogue or the [test] library.
> 
> So to refresh everybody's mind, I'd like to suggest that we place test 
> metadata in the source code itself using the following format, where the 
> here doc is inserted into the copy about the test function itself.
> 
> =======================================================================
> <<TEST_METADATA
> Name:
>   before_upgrade_create_data
> Summary:
>   Copies lustre source into a node specific directory and then creates 
> a tarball using that directory
> Description:
>   This should be called prior to upgrading Lustre and creates a set of 
> data on the Lustre partition
>   which be accessed and checked after the upgrade has taken place. 
> Several methods are using
>   including tar'ing directories so the can later be untar'ed and 
> compared, along with create sha1's
>   of stored data.
> Component:
>   - lnet
>   - recovery
> Prerequisites:
>   - before_upgrade_clear_filesystem
> TicketIDs:
>   - LU-123
>   - LU-432
> Purposes:
>   - upgrade
> TEST_METADATA
> 
> test_before_upgrade_create_data() {
>    ...
> }
> 
> run_test before_upgrade_create_data "Copying lustre source into a 
> directory $IOP_DIR1, creating and then using source to create a tarball"
> =======================================================================
> 
> Again thoughts and input very much appreciated
> 
> Chris
> 
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss