Breaking scripts

I've been thinking about the issues involved with using
IronPython in a product as a scripting language, internally we've
already struggled at Syzmk with my love of refactoring breaking
scripts in our deployed products left right and centre.

I think the problems I need to solve first are:

• Ensuring an understanding of the user experience when
  scripting with the application (read: dogfooding the scripting
  experience)
  
  
• Making sure I identify where helper classes / wrappers etc.
  may be necessary to make the scripting experience more palatable
  to end users.
  
• Ensuring any changes don't clobber the contracts between the
  scripting language and the client when refactoring the underlying
  .Net classes they rely on.
  

Regression tests for expected script usage is what it comes
down... but if you sit down and start trying to write these tests
in NUnit it all gets a bit unwieldy - there's a lot of time
wasted making sure your tabs are right, escaping characters
correctly in strings, and running scripts, evaluating the
results, and then asserting against them - it's frustrating, and
fails to emulate the experience of your users who are writing
scripts purely in Python.

PyUnit?

So my next port of call was to try the unit testing support that
comes with python (PyUnit) - it would definitely do the job - but
then I need to do a couple of things... first I need to start
bundling quite a few standard libraries with the scripting engine
in our product, and then I have to write additional functionality
to support capturing the results of running the PyUnit tests and
presenting them to the automated build process... It's all
sounding like a pain in the ass, and  the automation build
works nicely as it is, I'd rather not have to mess with it...

So I decided to do a little integration work, to see if I could
run a test suite written in python as part of a C# assembly.

NUnit meets (Iron)Python

First off, my solutions user experience is ok - not perfect - but
then I just wanted to make sure I could get it working, before I
put any effort into cleaning it up, and not being an expert in
extending NUnit, or coding python, it was probably doomed from
the start to be a mediocre solution ;o)

The steps to get it working in a test assembly are:

1. Writing a fixture in python, and including it in the assembly
   as an embedded resource.
   
2. Creating a suite which references the embedded python files -
   this is a class derived from the "AbstractPythonSuite".
   
   
3. Creating a Suite builder (I had to do this in the test
   assembly itself, because it wasn't working when I bundled this in
   a support library).
   
4. Run the suite ;o)
   

First off, we write the fixture, the only requirements are:

• The fixture class is derived from "NUnitFixture".
  
• The test methods are prefixed with "test" - as per PyUnit.
  And take only a "self" argument.
  

Here's an example fixture:

import System

from System import Console
class MyPythonFixture(NUnitFixture):
    def testPass(self):

        Console.WriteLine("--testPass--")

        Assert.IsTrue(True)
    def testFail(self):

        Console.WriteLine("--testFail--")

        Assert.IsTrue(False, "this will fail")

As you can see we are just using the existing NUnit Assert
methods... There is one caveat to this that I will cover in
another post (that of testing for expected exceptions).

Next we create a suite - which can wrap one or more python script
files (included as assembly resources) - this is all we need to
bridge the gap.

public class MyPythonSuite : AbstractPythonSuite

{

    public MyPythonSuite()

        : base("MyPythonFixture.py")

    {

    }

}

And last of all, we need to drop in a suite builder - in theory I
should be able to include this in a support assembly - but it
doesn't seem to work, here's the code:

[SuiteBuilder]

public class PythonSuiteExtensionBuilder : ISuiteBuilder

{

    public TestSuite BuildFrom(Type type, int assemblyKey)

    {

        if (CanBuildFrom(type))  return new PythonSuiteExtension(type, assemblyKey);

        return null;

    }
    public bool CanBuildFrom(Type type)

    {

        return (typeof(AbstractPythonSuite).IsAssignableFrom(type) && (type != typeof(AbstractPythonSuite)));

    }

}

At this point when can test the assembly with NUnitGui,
and see our python fixtures appear under the "MyPythonSuite"...
pretty slick huh?

border="0" />

The big downsides of this approach are that you can't run the
tests individually using something like TestDriven.Net,
and Resharper's unit test window doesn't pick them up...
However NUnitGui and the corresponding NUnitConsole
have no problems with them, and running all the tests in your
assembly with TestDriven.Net does pick them up thankfully
- so as long you don't have many slow tests in your assembly it
should still be tolerable (you can run them while still
coding...).

And of course, they will be picked up in your automated builds -
which is great news for me... I hate messing around with
CruiseControl.Net's configuration.

I'll tidy up the implementation a little and post the code
soon.