Fast unit tests with databases: Introduction to the fusonic testing framework

In the previous blog post I wrote about our huge performance gains for database tests, from initially 90 minutes run time down to 40 seconds, and about some of our core concepts we apply to our tests.

This second part of the three-part series contains a general introduction to Fusonic.Extensions.UnitTests, without focussing on database tests.

Author: Johannes Hartmann
Date: July 13, 2022
Reading time: 6 Minutes

Libraries and Services

As mentioned in the previous post, we share a few libraries over the projects. We often use the following setup:

Fusonic.Extensions are a set of libraries for various topics that we use in multiple projects. For the context of testing we focus on Fusonic.Extensions.UnitTests.* and Fusonic.Extensions.XUnit
.NET6 for development, as this is the current version. We stick to the latest .NET version for the Fusonic.Extensions, so usually there’ll be an updated version shortly after the .NET releases.
PostgreSQL is our database of choice for most .NET projects. We do have MSSQL and MySQL in some, but our test framework in Fusonic.Extensions only supports PostgreSQL currently. It could be extended to support other databases pretty easily, though.
EntityFramework Core as the OR mapper
XUnit is our testing framework
SimpleInjector for dependency injection, along with Microsofts dependency injection when required.
MediatR for supporting the CQRS pattern.

So those are basically the dependencies you get when you use our full testing framework.

Open Source

Our Fusonic.Extensions, including the unit test extensions are open source and can be found on GitHub. Contributions are welcome!

XUnit extensions

Without going too much into the internals of XUnit, we extended some parts of XUnit to get some parts of our test framework working. The library is Fusonic.Extensions.XUnit.

This includes the addition of a TestContext and a limiter for max. parallel Tests.

In order to add support for those extensions, you need to add an assembly attribute to your test class, eg. in an AssemblyInfo.cs:

[assembly: Fusonic.Extensions.XUnit.Framework.FusonicTestFramework]

This is also required when using Fusonic.Extensions.UnitTests.*, as they rely on some features introduced in our XUnit extensions.

Test context

XUnit v2 does not provide a test context by itself. Our extensions provide a simple TestContext (Fusonic.Extensions.XUnit.TestContext) which provides the following properties:

TestMethod, which is the currently executing test method
TestClass, which is the currently executing test class
Items, a dictionary which allows you to add any objects for your test context
Out for access to the XUnit ITestOutputHelper, which allows you to write messages to the test output

We usually do not use this test context in the tests themselves, but it is required for some framework code.

Max. parallel tests

In XUnit you can limit the max number of parallel threads, but not the number of max parallel tests. That’s a big caveat for database tests: As soon as you await some code, a thread can be marked as free and a new test gets started.

This can result in hundreds of parallel database tests running at once, overloading the database and causing connection and timeout issues.

Starting a new test on await is by design. You can work around that using a semaphore when starting the test, but this falsifies your test run times because the time waiting for the semaphore already is counted to the test run time, which can cause test timeouts in the worst case .

Our framework comes with a built in limiter for max. parallel tests which just won’t allow starting more tests than configured.

When using the FusonicTestFramework, the maximum amount of parallel tests is already limited to the number of CPUs by default. You can modify this setting with an optional framework parameter, or even disable the test limitations by setting it to a negative number:

[assembly: Fusonic.Extensions.XUnit.Framework.FusonicTestFramework(MaxParallelTests = 8)]

BeforeAfterTestInvokeAttribute

Again something you probably won’t need in your tests directly, but is needed for our database tests. You can overwrite this attribute and set it on a method or class. Using this, you can execute any code before or after executing a test method.

XUnit also provides a very similar BeforeAfterTestAttribute. The difference to XUnits attribute is that the xunit version executes the Before-Method after the test class is created and the constructor ran.

Our version runs before the test class is created, allowing us to set up the database providers before the constructor is called.

Configuring a test project

If you don’t need database test support, Fusonic.Extensions.UnitTests already gives you a nice set of features using XUnit, with dependency injection using SimpleInjector, MediatR support and helper methods for running parts of your code within own lifetime scopes, as explained earlier. It also has configuration support, using a testsettings.json, user secrets and environment variables.

You simply add the following base classes, which you can inherit from and get all the features in your tests:

public class TestFixture : UnitTestFixture
{
    protected sealed override void RegisterCoreDependencies(Container container)
    {
        var appSettings = Configuration.Bind<AppSettings>();
        //Your SimpleInjector configuration here
    }
}
 
public class TestBase : UnitTest<TestFixture>
{
    public TestBase(TestFixture fixture) : base(fixture)
    { }
}

If you want some custom DI configuration for a specific test (eg. for a mock), you can do that, too:

public class SomeHandlerTest : UnitTest<SomeHandlerTest.HandlerTestFixture>
{
    // Tests
 
    public class HandlerTestFixture : TestFixture
    {
        protected override void RegisterDependencies(Container container) => container.Register<SomeHandler>();
    }
}

Lifetime scoping

As mentioned in the previous part, we try to avoid side effects by running each section of a test (Arrange / Act / Assert) within an own lifetime scope of our DI-container SimpleInjector.

Having an own lifetime scope for each test-section (Arrange / Act / Assert) usually would result in a huge coding overhead. You’d need to add the following to each section:

await using (var scope = Fixture.BeginLifetimeScope())
{
    newScope.Container.GetInstance<SomeService>().DoSomething();
}

As this isn’t practical, we’ve introduced the following helper methods in our TestBase:

Scoped/ScopedAsync: This is the core method used for scoping. It takes an Action or Func<T> as parameter and executes it within its own lifetime scope. All calls to GetInstance<T> for resolving a service use this own scope. Example:

await ScopedAsync(() => GetInstance<SomeService>().ChangeTitleAsync(entityId, "New title"));

Send/SendAsync: MediatR support for our CQRS-heavy business logic. Calls IMediator.Send() within its own lifetime scope.
Query/QueryAsync: Added with the database unit tests. Passes a DbContext, resolved in its own scope using ScopedAsync in the background.

Let’s quickly revisit the service and example from the last blog post:

public async Task ChangeTitleAsync(int id, string newTitle)
{
     var entity = await dbContext.SomeEntities.FindAsync(id);
     entity.Title = newTitle;
}

It uses the following test:

[Fact]
public async Task ChangeTitle_SetsNewTitle()
{
    // Arrange
    var dbContext = GetInstance<DbContext>();
    var entity = dbContext.Add(new SomeEntity { Title = "Old title "}).Entity;
    await dbContext.SaveChangesAsync();
 
    // Act
    var service = GetInstance<SomeService>();
    await service.ChangeTitleAsync(entity.Id, "New title");
 
    // Assert
    var updatedEntity = await dbContext.FindAsync(entity.Id);
    updatedEntity.Title.Should().Be("New title");
}

Should this test fail? Yes. ChangeTitle() does not save the entity after updating it when it should.

Does this test fail? No. dbContext.FindAsync() retrieves a cached entity. It was added to the cache when arranging the test data, was retrieved using FindAsync() in the tested service method and once more when calling assert.

Using lifetime scoping, with the helper methods from the TestBase the test for it looks like this:

[Fact]
public async Task ChangeTitle_SetsNewTitle()
{
    // Arrange
    var entityId = await QueryAsync(async ctx =>
    {
        var entity = ctx.Add(new SomeEntity { Title = "Old title "}).Entity;
        await dbContext.SaveChangesAsync();
        return entity.Id;            
    });
 
    // Act
    await ScopedAsync(() => GetInstance<SomeService>().ChangeTitleAsync(entityId, "New title"));
 
    // Assert
    var updatedEntity = await QueryAsync(ctx => ctx.SomeEntities.FindAsync(entityId));
    updatedEntity.Title.Should().Be("New title");
}

Should this test fail? Still, yes.

Does this test fail? Yes. A new lifetime scope gets created within the QueryAsync and ScopedAsync methods. None of the arrange, act and assert steps share the same context or the same instances.

Test scoped lifetime

Let's say we want to mock a service for testing. The separated lifetime scopes for Act/Arrange/Assert actually become a problem when registering that mocked service. Example:

public interface ISomeService
{
    int SomeMethod();
}
 
public class SomeHandler
{
    public SomeHandler(ISomeService service) { }
    public int Handle() => service.SomeMethod();
}
 
public class SomeHandlerTests : UnitTest<SomeHandlerTests.Fixture>
{
    [Fact]
    public void Example()
    {
        GetInstance<ISomeService>().SomeMethod().ReturnsForAnyArgs(42);
 
        var result = Scoped(() => GetInstance<SomeHandler>().Handle());
 
        result.Should().Be(42);
    }
 
    public class Fixture : TestFixture
    {
        protected override void RegisterDependencies(Container container)
        {
            container.Register<SomeHandler>();
            container.Register(() => Substitute.For<ISomeService>());
        }
    }
}

Here we mock a service, so it returns 42 if called. The issue? When running the handler that uses the service, the handler does not have the mocked instance that is configured returning 42, but it gets its own, fresh instance. The result will simply be the default, 0.

Now, you could register ISomeService as singleton, but this affects other tests then, as the TestFixture, and thus the Container, is created once per test class. So you’d have to clear the substitute for each test run.

With our library you can simply use this:

container.RegisterTestScoped<ISomeService>();

This scope registers a service basically as a singleton, but only for one specific test case. That way you can configure your substitute in your test as you like, affecting all calls in that test (as expected), without affecting any other test running.

What’s next?

The next and final part of this series contains a concrete setup for the fast unit tests, including an example project. Stay tuned!

Fast unit test_Implementation of our soluation (1)

Dev Diary

Fast unit tests with databases: Implementation of our solution

July 13, 2022 | 7 Min.

Dev Diary

Fast unit tests with databases: A primer

July 13, 2022 | 6 Min.