Assembly filter/selection with commandline support

[nbalu]

Summary

Right now the MsBuild is only capable to run and add the extension for the entry assembly.

I'd like to have a way to specify additional assemblies / packages to be run without the need to rebuild the runner behind the scenes.

This could allow us to create a preconfigured "runner" and the tests could be separated from the concept of patform/host/controller that runs them.

Background and Motivation

We used to split our different application tests into multiple projects assemblies based on which microservice they are testing. Right now we can only make separate executables which cannot simply depend on the MsTest.Framework but they should depend on the whole platform itself.

We've had this capability before using NUnit and my guess this should be a platform level support which different frameworks can leverage. Thus the assembly discovery can be "obsolate" - not that with the current solution we are supporting that. The platform can give a sane default: entry assembly will be used. In case of multiple assemblies the given list is considered as sources/lookup for tests.

The problematic entrypoint is:

testfx/src/Adapter/MSTest.TestAdapter/TestingPlatformAdapter/TestingPlatformBuilderHook.cs

Line 14 in 981c70f

public static void AddExtensions(ITestApplicationBuilder testApplicationBuilder, string[] arguments) => testApplicationBuilder.AddMSTest(() => [Assembly.GetEntryAssembly()!]);

Here I would suggest to use the already available args to parse the list and load the assemblies if they are available. Error in case of any of them is not found.

Proposed Feature

Give a configuration / commandline capability to be able to define which assemblies we would like to run the tests from. This should be a CLI filter IMHO like:

dotnet test --assembly-filter MyTests1.dll MyTests2.dll

Alternative Designs

I don't really have any idea in mind. Any alternative that gives me the flexibility that let's me run tests from multiple sources without having a platform dependency is great.

[nbalu]

What if we say that any implementation Framework (MsTest | XUnit | TUnit | NUnit) could introduce an implicit custom extension method
public static void AddExtensions(ITestApplicationBuilder testApplicationBuilder, string[] arguments) => testApplicationBuilder.AddTestAssemblySource("{frameworkname}", "MyAssembly");

public class TestAssemblySource : ITestAssemblySource
{
    private readonly string frameworkName;
    private readonly string assemblyName;

    public TestAssemblySource(string frameworkName, string assemblyName)
	{
        this.frameworkName = frameworkName;
        this.assemblyName = assemblyName;
    }

	public Assembly? GetTestAssembly(string frameworkName) { 
        if(frameworkName != this.frameworkName)
        {
            return null;
        }

        return Assembly.Load(assemblyName); }
}

public interface ITestAssemblySource
{
    public Assembly GetTestAssembly(string frameworkName);
}

For discovery what if we check the static extension methods given to the assembly filter (dotnet test --assembly-selector MyAssembly). The builder is automaticaly tries to load and reflect back any static method that has two arguments that satisfies the AddTestAssemblySource(ITestApplicationBuilder testApplicationBuilder, string[] arguments). Invoke them.

This will allow us to separate the adapter | test registrations.

Anyway this is just a quick brain dump of mine.

[Evangelink]

Hi @nbalu,

This is part of our new improved experience for dotnet test that we are currently developping.

Note that in the meantime, you could also create a single "runner" app and register multiple assemblies. I'll try to come up with a solution example.

[nbalu]

Thanks @Evangelink!

No need for that. I already do have an example working it just feels hacky as I need to explicitly stating the "dependencies" of the runner. Wanted to raise my concern as a ticket we can follow up.