AbstractTestNGSpringContextTests is not thread-safe regarding tracked exceptions #35528
Description
TestNG allows running tests in parallel, using the parallel=methods parameter. This causes tests to run in multiple threads using the same class instance.
However, the code in https://github.com/spring-projects/spring-framework/blob/main/spring-test/src/main/java/org/springframework/test/context/testng/AbstractTestNGSpringContextTests.java#L148-L162 is not thread-safe.
As an example, consider the following minimal test case:
@ContextConfiguration(classes = {SampleJavaClassTest.Configuration.class})
public class SampleJavaClassTest extends AbstractTestNGSpringContextTests {
public static class Configuration {
}
@Test(expectedExceptions = RuntimeException.class, expectedExceptionsMessageRegExp = "Message1")
public void message1() {
throw new RuntimeException("Message1");
}
@Test(expectedExceptions = RuntimeException.class, expectedExceptionsMessageRegExp = "Message2")
public void message2() {
throw new RuntimeException("Message2");
}
@Test(expectedExceptions = RuntimeException.class, expectedExceptionsMessageRegExp = "Message3")
public void message3() {
throw new RuntimeException("Message3");
}
@Test(expectedExceptions = RuntimeException.class, expectedExceptionsMessageRegExp = "Message4")
public void message4() {
throw new RuntimeException("Message4");
}
@Test(expectedExceptions = RuntimeException.class, expectedExceptionsMessageRegExp = "Message5")
public void message5() {
throw new RuntimeException("Message5");
}
@Test(expectedExceptions = RuntimeException.class, expectedExceptionsMessageRegExp = "Message6")
public void message6() {
throw new RuntimeException("Message6");
}
@Test(expectedExceptions = RuntimeException.class, expectedExceptionsMessageRegExp = "Message7")
public void message7() {
throw new RuntimeException("Message7");
}
@Test(expectedExceptions = RuntimeException.class, expectedExceptionsMessageRegExp = "Message8")
public void message8() {
throw new RuntimeException("Message8");
}
@Test(expectedExceptions = RuntimeException.class, expectedExceptionsMessageRegExp = "Message9")
public void message9() {
throw new RuntimeException("Message9");
}
@Test(expectedExceptions = RuntimeException.class, expectedExceptionsMessageRegExp = "Message10")
public void message10() {
throw new RuntimeException("Message10");
}
}Running this test with the following TestNG configuration:
<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd" >
<suite name="TestSuite" verbose="2" thread-count="5">
<test name="DemoTest" annotations="JDK" parallel="methods" verbose="2" preserve-order="true" thread-count="5">
<packages>
<package name="com.example.*"/>
</packages>
</test>
</suite>(note thread-count=5 and parallel=methods)
Causes the tests to fail randomly:
org.testng.TestException: The exception was thrown with the wrong message: expected "Message2" but got "Message10"
The stacktrace from such failures also shows the stacktrace originating from the wrong test. The message2 test failure stack trace looks like this:
Caused by: java.lang.RuntimeException: Message10
at com.example.SampleJavaClassTest.message10(SampleJavaClassTest.java:54)
...
In the code above, this.testException is being clobbered by a different thread, causing the issues described.