[Lustre-devel] Language choice for Lustre tests

[Prakash Surya]

On Thu, Oct 25, 2012 at 11:04:55AM -0700, Nathan Rutman wrote:
>    There are several levels of "test framework" that are involved with
>    automated testing of Lustre:
> 
>      * unit tests themselves, written in bash, e.g. sanity 1a

What do you mean by "unit tests" here? Because as I understand it,
Lustre has no unit testing at all. It is all functional testing,
which is related but completely different.

>      * test-framework.sh & friends, which provides some
>        support/setup/communal library for unit tests

Again, unit tests? I really wish there _were_ unit tests, but aren't,
and I haven't seen any discussion about creating them.

>      * automated test systems, which automatically execute the unit tests,
>        record and report the results.

I don't quite understand why there is so much discussion regarding a new
testing framework specifically designed for testing Lustre. Is there
really nothing already out there we can make use of?

Brian linked to the Autotest framework, does that not do what we want?
If it doesn't work, why not? And how is our home brewed solution going
to differ? Are there any other alternatives out there? I don't see
benefit in spending development on a new framework if it's unnecessary.

There is a lot of discussion regarding a tool (the language) and little
involving the problem (lack of useful functional and unit tests to
properly stress the system). Pick something, move on, and start
improving the codes stability.

And if somebody *really* wants to build a new testing framework, have it
live out of tree. Create it as a standalone project which interfaces
with Lustre.

-- 
Cheers, Prakash


> 
>    I'm only speaking about the language for the first of these items at
>    the moment.
>    For more context, including the motivation for changing things, see my
>    LAD'12 presentation (more narrowly focused) on the [1]OpenSFS Wiki.
>    On Oct 25, 2012, at 10:23 AM, Brian Behlendorf
>    <[2]behlendorf1 at llnl.gov> wrote:
> 
>      On Wed, 2012-10-24 at 15:05 -0700, Nathan Rutman wrote:
> 
>      On Oct 24, 2012, at 1:02 PM, "Gearing, Chris"
>      <[3]chris.gearing at intel.com> wrote:
> 
>      Nathan,
>      I'm not 100% sure what you are proposing here, your LAD presentation
>      suggested a 'tune-up' of the current test framework rather than a
>      complete re-write. Which of the two are we discussing?
> 
>      Both...
> 
>      The requirements on the framework language are more relaxed, but for
>      ease of development and developer sanity, I assume that the
>      framework language should match the test language.  So I'm using the
>      test language as the requirements driver, and to gage community
>      preference for that test language.
> 
>      Before embarking on building yet another new and custom framework
>      for
>      Lustre we should evaluate some existing frameworks.  For example,
>      the
>      Autotest project was specifically designed to test the Linux kernel.
>      It's open source, looks active, is flexible, and there is detailed
>      documentation on how to write tests.  Plus it was designed
>      specifically
>      for testing the kernel so there are likely existing file system
>      tests.
>       [4]http://autotest.github.com/
>       "Autotest is a framework for fully automated testing. It is
>        designed primarily to test the Linux kernel, though it is useful
>        for many other functions such as qualifying new hardware. It's an
>        open-source project under the GPL and is used and developed by a
>        number of organizations, including Google, IBM, Red Hat, and many
>        others."
> 
>      Based on the responses so far, it seems that there is a fairly clear
>      preference for Python as a test language, and so I'll propose that
>      Python should be used shorter-term to start replacing
>      test-framework.
> 
>      If we decide the Autotest framework is a good fit then we'll want to
>      write the tests in python to be consistent with the framework
>      language.
>      However, for a first cut it looks like you could use the existing
>      bash
>      tests largely unmodified.
>      --
>      Thanks,
>      Brian
>      _______________________________________________
>      Lustre-devel mailing list
>      [5]Lustre-devel at lists.lustre.org
>      http://lists.lustre.org/mailman/listinfo/lustre-devel
> 
> References
> 
>    1. http://www.opensfs.org/foswiki/bin/view/Lustre/TestingImprovements
>    2. mailto:behlendorf1 at llnl.gov
>    3. mailto:chris.gearing at intel.com
>    4. http://autotest.github.com/
>    5. mailto:Lustre-devel at lists.lustre.org

> _______________________________________________
> Lustre-devel mailing list
> Lustre-devel at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-devel