[Lustre-devel] Language choice for Lustre tests

Nathan Rutman nathan_rutman at xyratex.com
Thu Oct 25 14:36:02 PDT 2012

On Oct 25, 2012, at 2:17 PM, Prakash Surya <surya1 at llnl.gov> wrote:

> On Thu, Oct 25, 2012 at 11:04:55AM -0700, Nathan Rutman wrote:
>>   There are several levels of "test framework" that are involved with
>>   automated testing of Lustre:
>>     * unit tests themselves, written in bash, e.g. sanity 1a
> What do you mean by "unit tests" here? Because as I understand it,
> Lustre has no unit testing at all. It is all functional testing,
> which is related but completely different.
Let's please not get into this.  I just meant the individual tests.

>>     * test-framework.sh & friends, which provides some
>>       support/setup/communal library for unit tests
>>     * automated test systems, which automatically execute the unit tests,
>>       record and report the results.
> I don't quite understand why there is so much discussion regarding a new
> testing framework specifically designed for testing Lustre. Is there
> really nothing already out there we can make use of?
> Brian linked to the Autotest framework, does that not do what we want?

That covers the third meaning, but not the first two.   I'm only concerned about the
first two at the moment. 

> If it doesn't work, why not?
It doesn't contain a Lustre-specific functional library, it doesn't safely limit the tests
that may be run on a specific system, it doesn't handle interoperability testing, etc.

> And how is our home brewed solution going
> to differ? Are there any other alternatives out there? I don't see
> benefit in spending development on a new framework if it's unnecessary.
> There is a lot of discussion regarding a tool (the language) and little
> involving the problem (lack of useful functional and unit tests to
> properly stress the system). Pick something, move on, and start
> improving the codes stability.
That was what I was trying to do.

> And if somebody *really* wants to build a new testing framework, have it
> live out of tree. Create it as a standalone project which interfaces
> with Lustre.
As stated in my presentation, that is part of the plan.

> -- 
> Cheers, Prakash
>>   I'm only speaking about the language for the first of these items at
>>   the moment.
>>   For more context, including the motivation for changing things, see my
>>   LAD'12 presentation (more narrowly focused) on the [1]OpenSFS Wiki.
>>   On Oct 25, 2012, at 10:23 AM, Brian Behlendorf
>>   <[2]behlendorf1 at llnl.gov> wrote:
>>     On Wed, 2012-10-24 at 15:05 -0700, Nathan Rutman wrote:
>>     On Oct 24, 2012, at 1:02 PM, "Gearing, Chris"
>>     <[3]chris.gearing at intel.com> wrote:
>>     Nathan,
>>     I'm not 100% sure what you are proposing here, your LAD presentation
>>     suggested a 'tune-up' of the current test framework rather than a
>>     complete re-write. Which of the two are we discussing?
>>     Both...
>>     The requirements on the framework language are more relaxed, but for
>>     ease of development and developer sanity, I assume that the
>>     framework language should match the test language.  So I'm using the
>>     test language as the requirements driver, and to gage community
>>     preference for that test language.
>>     Before embarking on building yet another new and custom framework
>>     for
>>     Lustre we should evaluate some existing frameworks.  For example,
>>     the
>>     Autotest project was specifically designed to test the Linux kernel.
>>     It's open source, looks active, is flexible, and there is detailed
>>     documentation on how to write tests.  Plus it was designed
>>     specifically
>>     for testing the kernel so there are likely existing file system
>>     tests.
>>      [4]http://autotest.github.com/
>>      "Autotest is a framework for fully automated testing. It is
>>       designed primarily to test the Linux kernel, though it is useful
>>       for many other functions such as qualifying new hardware. It's an
>>       open-source project under the GPL and is used and developed by a
>>       number of organizations, including Google, IBM, Red Hat, and many
>>       others."
>>     Based on the responses so far, it seems that there is a fairly clear
>>     preference for Python as a test language, and so I'll propose that
>>     Python should be used shorter-term to start replacing
>>     test-framework.
>>     If we decide the Autotest framework is a good fit then we'll want to
>>     write the tests in python to be consistent with the framework
>>     language.
>>     However, for a first cut it looks like you could use the existing
>>     bash
>>     tests largely unmodified.
>>     --
>>     Thanks,
>>     Brian
>>     _______________________________________________
>>     Lustre-devel mailing list
>>     [5]Lustre-devel at lists.lustre.org
>>     http://lists.lustre.org/mailman/listinfo/lustre-devel
>> References
>>   1. http://www.opensfs.org/foswiki/bin/view/Lustre/TestingImprovements
>>   2. mailto:behlendorf1 at llnl.gov
>>   3. mailto:chris.gearing at intel.com
>>   4. http://autotest.github.com/
>>   5. mailto:Lustre-devel at lists.lustre.org
>> _______________________________________________
>> Lustre-devel mailing list
>> Lustre-devel at lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-devel

More information about the lustre-devel mailing list