Skipping parameterized tests based on bug annotation and value of first parameter

[koranke]

I'm coming from TestNG. At a prior company, we had a solution for flagging tests that had open bugs using a custom "Bug" annotation. The bug annotation would store the bug ID and an optional "scenario ID". The simple solution was using this on a standard Test method, but we also had a solution for parameterized tests (in TestNG language, a data provider). We used a TestNG listener and override for the method on starting test. We would check for the Bug annotation. If found, and if the test was also a parameterized test, and if the bug included a value for "scenario ID", then we would check the value of the first parameter for that test, and if it matched the "scenario ID" value, then skip the test. Basically, we wanted to be able to skip any test or test scenario (as represented by a parameterized test) that had an open bug. Can this be done using JUnit 5? I've tried using ExecutionCondition, but that doesn't seem to work with ParameterizedTest methods. I also tried using BeforeTestExecutionCallback, but that doesn't appear to have access to the test parameter values.

Basically, what this should look like, would be something similar to below, where the scenario represented by "T2" would get skipped.

@Bug(id ="12345", scenario = "T2")
@ParameterizedTest
@MethodSource("provideData")
public void testSomething(String scenarioId, int i) {
    // test something here
}

public static Object[][] provideData() {
	return new Object[][] {
			{"T1", 4},
			{"T2", 5},
			{"T3", 6}
	};
}

[marcphilipp]

There are extensions that allow doing so based on the value of arguments or the resulting display name: https://junit-pioneer.org/docs/disable-parameterized-tests/

For your custom logic, you can implement an InvocationInterceptor like this:

import static java.lang.annotation.ElementType.METHOD;
import static java.lang.annotation.RetentionPolicy.RUNTIME;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assumptions.assumeFalse;

import java.lang.annotation.Retention;
import java.lang.annotation.Target;
import java.lang.reflect.Method;
import java.lang.reflect.Parameter;
import java.util.List;
import java.util.Objects;

import org.junit.jupiter.api.extension.ExtendWith;
import org.junit.jupiter.api.extension.ExtensionContext;
import org.junit.jupiter.api.extension.InvocationInterceptor;
import org.junit.jupiter.api.extension.ReflectiveInvocationContext;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.MethodSource;
import org.junit.platform.commons.support.AnnotationSupport;

public class DemoTests {

	@Bug(id = "12345", scenario = "T2")
	@ParameterizedTest
	@MethodSource("provideData")
	public void testSomething(String scenarioId, int id) {
		// test something here
	}

	public static Object[][] provideData() {
		return new Object[][] { { "T1", 4 }, { "T2", 5 }, { "T3", 6 } };
	}

	@Retention(RUNTIME)
	@Target(METHOD)
	@ExtendWith(Bug.Extension.class)
	@interface Bug {
		String id();

		String scenario();

		class Extension implements InvocationInterceptor {

			@Override
			public void interceptTestTemplateMethod(Invocation<Void> invocation,
					ReflectiveInvocationContext<Method> invocationContext, ExtensionContext extensionContext)
					throws Throwable {
				AnnotationSupport.findAnnotation(extensionContext.getElement(), Bug.class).ifPresent(bug -> {
					Parameter[] parameters = invocationContext.getExecutable().getParameters();
					assertEquals(String.class, parameters[0].getType(),
						"First parameter represent scenario ID and must be of type String");
					List<Object> arguments = invocationContext.getArguments();
					assumeFalse(Objects.equals(arguments.get(0), bug.scenario()), () -> "Skipping test for scenario "
							+ bug.scenario() + " that is affected by known bug " + bug.id());
				});
				invocation.proceed();
			}
		}
	}
}

[koranke]

Thanks for the excellent answer, both for the code example and the built-in extensions. Exactly what I was looking for.