Multi-Threaded Tests

Description

Like a standard test runner, NCrunch does not have special consideration for multi-threaded tests and will execute them just the same as any other test. Although there should be no functional difference around multi-threading between NCrunch and any other test runner, uncontrolled multi-threaded behaviour can create problems that become more visible when using NCrunch.

Possible Problems

NCrunch will record code coverage based on test boundaries. When the test runner's foreground thread begins execution of a test, NCrunch will immediately begin tracking and recording code coverage data for this test. As soon as the test runner's foreground thread completes execution of the test, NCrunch will stop recording code coverage for the test. In simple synchronous situations this behaviour is very predictable and works very well.

However, NCrunch does not differentiate between threads when recording code coverage data. This means that any background threads launched by a test will continue to have coverage data recorded only until the foreground thread has finished executing the test. Any further execution on backgrounds threads is erroneous execution and can cause problems with coverage data tracking.

Quite often, the 'over-running' of background threads is benign, as NCrunch will discard code coverage data that is collected outside the scope of a test. In situations where multiple tests are being executed as part of a single test run (i.e. several tests inside one execution task in the processing queue), it's possible for the over-run background thread's coverage to be recorded against tests that are executed after the test that launched it. This can create inconsistent coverage data for the tests involved.

Consider the following example:

public class FixtureWithThreadingTests
{
    [Test]
    public void TestThatStartsAThread()
    {
        ThreadPool.QueueUserWorkItem(x =>
            {
                var strings = new StringBuilder();

                for (int i = 0; i < 1000000; i++)
                    strings.Append("a");
            });
    }

    [Test]
    public void ThreadSleepingTest()
    {
        Thread.Sleep(300);
    }
}

When executed repeatedly, the above code can produce 3 possible results:

• Normal expected behaviour - the code queued by the thread pool has code coverage recorded against the first test. This will happen if the thread pool is fast enough to pick up the work and execute it before the first test finishes executing.
• Incorrect code coverage - the code queued by the thread pool has code coverage recorded against the second test (ThreadSleepingTest). This will happen if the background thread is still processing at the time the second test has begun execution.
• NullReferenceException thrown from nCrunch.TestRuntime.TestCoverageEventListener - This is caused by a race condition created in the NCrunch runtime code by the background thread trying to record coverage data while the test runner switches tests. This will only happen in NCrunch releases prior to v1.43.

Solutions

It is vital that test code be designed in a way that makes over-running background threads impossible. This is not only important for NCrunch, but for the integrity of the testing suite as a whole.

Broadly, there are several ways to solve problems caused by over-running background threads in NCrunch:

Isolate The Test

NCrunch's runtime framework includes a special attribute, IsolatedAttribute, which can be used to isolate the test in its own task runner process. When the foreground thread of the test finishes executing, the task runner process is torn down and any background threads are automatically terminated by the O/S. This is a very simple solution but it does have drawbacks. The need to isolate the test within its own process creates additional overhead around the execution of this test, slowing down test cycle times. Also, this approach will only work with NCrunch and not other test runners.

Make The Test Sleep

A simple but crude approach is to slow down the foreground thread of the test to ensure the background thread has time to finish before the test ends. Although this solution technically relies on a race condition, the size of the sleep statement can be made large enough to make any over-run impossible.

Re-engineer The Test

By far the best approach is to design the test in such a way that the foreground thread of the test will wait for any background processing to complete before allowing the test to finish. Consider the following example:

public class FixtureWithThreadingTests
{
    [Test]
    public void TestThatStartsAThread()
    {
        var waitHandle = new ManualResetEvent(false);

        ThreadPool.QueueUserWorkItem(x =>
            {
                var strings = new StringBuilder();

                for (int i = 0; i < 1000000; i++)
                    strings.Append("a");

                waitHandle.Set();
            });

        waitHandle.WaitOne();
    }

    [Test]
    public void ThreadSleepingTest()
    { 
        Thread.Sleep(300);
    }
}

The signalling of the ManualResetEvent in the above code makes it impossible for the foreground thread to progress with the completion of the test without the background thread having first completed its execution.

In most situations, the code responsible for spawning the background thread will reside in production (non-test) code. This can make it more difficult to get a handle on the thread being launched in order to track its lifetime. However, the same concept applies.

When working with production code that launches threads, a useful pattern is to abstract the launching of the background thread behind an injected factory class. This factory class can either maintain a list of the running threads (to be queried by test code), or it can execute the requested code synchronously without using any kind of background processing.