[SUREFIRE-2239, SUREFIRE-2241, SUREFIRE-2260, SUREFIRE-2274, SUREFIRE-2300] Preserve JUnit Platform test identity across reruns by wilx · Pull Request #3418 · apache/maven-surefire

wilx · 2026-08-08T21:19:20Z

Following this checklist to help us incorporate your
contribution quickly and easily:

Each commit in the pull request should have a meaningful subject line and body.
Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
Run mvn clean install to make sure basic checks pass. A more thorough check will
be performed on your pull request automatically.
You have run the integration tests successfully (mvn -Prun-its clean install).

If your pull request is about ~20 lines of code you don't need to sign an
Individual Contributor License Agreement if you are unsure
please ask on the developers list.

To make clear that you license your contribution under
the Apache License Version 2.0, January 2004
you have to acknowledge this by using the following check-box.

I hereby declare this contribution to be licenced under the Apache License Version 2.0, January 2004
In any other case, please file an Apache Individual Contributor License Agreement.

Summary

Fixes #2686.
Fixes #2670.
Fixes #2626.
Fixes #2656.
Fixes #2602.

JUnit Platform tests were aggregated by their human-readable class and method names. Engines can legitimately report multiple logical tests with the same names, and JUnitParams on the Vintage engine reports actual tests with a ClassSource but no method source. During reruns this collapsed skipped, failing, and successful executions into one apparent flake, could generate an empty XML element name, and could hide real failures.

This change:

uses the JUnit Platform unique ID as the internal test-run identity while retaining readable report names;
propagates that identity into XML and global result aggregation;
restores cached IDs to the output association;
gives ClassSource-backed tests a readable test name;
rediscovers ClassSource-backed failures by class while filtering and reporting only the failed unique-ID branches, allowing runner-backed Vintage tests such as JUnitParams to rerun without parsing engine-specific unique-ID segments;
serializes skipped executions defensively in a flake history;
adds exact JUnitParams and Cucumber regression projects;
recovers the SUREFIRE-2239 fail-first/pass-second TestNG reproducer as an integration test for both Surefire and Failsafe;
adapts the SUREFIRE-2274 Failsafe reproducer into an integration test that verifies rerun flakes reach failsafe-summary.xml and are enforced by verify; and
reports every discovered test below a class aborted by a JUnit 5 @BeforeAll assumption as skipped, while preserving completed test results when a class aborts later in its lifecycle.

An existing integration fixture containing two distinct tests with the same method name now correctly expects both tests in its report.

User impact

Identically named tests and scenarios remain distinct. The JUnitParams reproducer fails both parameterized cases on their first invocation, reruns those two cases successfully, records two flakes in valid XML, and continues to the following test class. The Cucumber reproducer reports three scenarios as one skipped, one failed, and one passed test. TestNG tests are also verified to rerun through the JUnit Platform provider in both Surefire and Failsafe, with one flake recorded by each plugin. Failsafe is additionally protected end to end against losing the flake count between integration-test and verify: two fail-twice/pass-third tests produce <flakes>2</flakes>, and failOnFlakeCount=1 fails the build during verify. A two-test class aborted by a @BeforeAll assumption now reports two tests and two skips, including both XML test cases and the original assumption cause, instead of reporting zero tests and zero skips.

Tests

mvn clean install
nice -n 10 env JAVA_HOME=/opt/jdks/latest-17 /opt/maven/latest/bin/mvn -Prun-its clean install (761 integration tests, 0 failures, 0 errors)
nice -n 10 /opt/maven/latest/bin/mvn clean test -pl surefire-providers/common-java5,surefire-providers/surefire-junit-platform (passed; 85 JUnit Platform provider tests)
focused rerun-related integration tests (83 tests, 0 failures, 0 errors)
exact fail-first/pass-second JUnitParams integration test (7 final tests, 2 flakes, no initializationError)
Java 8 TestNG/JUnitParams/Cucumber related integration-test set (12 tests, 0 failures, 0 errors)
exact fail-first/pass-second TestNG integration test through Surefire and Failsafe (1 flake in each report)
nice -n 10 /opt/maven/latest/bin/mvn clean verify -pl surefire-its -Prun-its -Dit.test=Surefire2274FailsafeFlakeCountIT -Dmaven.build.cache.enabled=false (passed on Java 8; nested Failsafe build failed at verify as expected with 2 flakes)
nice -n 10 /opt/maven/latest/bin/mvn clean test -pl maven-failsafe-plugin -Dtest=RunResultTest -Dmaven.build.cache.enabled=false (7 tests, 0 failures, 0 errors)
nice -n 10 /opt/maven/latest/bin/mvn clean test -pl surefire-providers/surefire-junit-platform -Dmaven.build.cache.enabled=false (87 tests, 0 failures, 0 errors)
nice -n 10 /opt/maven/latest/bin/mvn clean verify -pl surefire-its -Prun-its -Dit.test=Surefire2300BeforeAllAssumptionIT,JUnit5BeforeAllContainerFailureIT,JUnit4PlatformFailingBeforeAllRerunIT -Dmaven.build.cache.enabled=false (3 tests, 0 failures, 0 errors)

wilx · 2026-08-09T15:05:34Z

I have pushed 2d29ea4 to try to avoid some sort of threading issue in possibly TestNG. I have seen the failure in TestMultipleMethodPatternsTestNGIT.shouldMatchMultiplePatternsAsParallel on two separate occasions now:

olamy · 2026-08-11T01:51:21Z

The PR has some conflicts

Use the JUnit Platform unique ID as the internal run identity while retaining human-readable report names. Aggregate XML and global results by that identity so equally named tests and scenarios remain distinct, and keep skipped rerun entries valid in flaky XML output. Add regressions for the JUnitParams and Cucumber reproducers. Fixes apache#2686 Fixes apache#2670

Recover the fail-first/pass-second TestNG reproducer and exercise it through both Surefire and Failsafe using the JUnit Platform provider. Assert that each final report records one flake. Fixes apache#2626

Adapt the linked reproducer to verify that Failsafe persists rerun flakes in its summary and enforces failOnFlakeCount during verify. Fixes apache#2656

Use the JUnit Platform TestPlan to report every discovered test under a class aborted by a BeforeAll assumption as skipped while preserving completed results for later container aborts. Fixes apache#2602

Update the method-pattern fixture to TestNG 7.5.1, the latest version supporting Java 8, to avoid a suspected class-lifecycle race fixed after TestNG 6.14.3. Fixes apache#2686

wilx · 2026-08-11T06:02:01Z

The PR has some conflicts

Rebased.

Copilot

Pull request overview

Preserves JUnit Platform test identity across reruns, improving flake aggregation, runner-backed reruns, and aborted-class reporting.

Changes:

Uses unique IDs to distinguish identically named tests and associate rerun output.
Adds class-based rediscovery and descendant skip reporting.
Adds regression coverage for JUnitParams, Cucumber, TestNG, Failsafe, and @BeforeAll assumptions.

Reviewed changes

Copilot reviewed 34 out of 34 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
`surefire-providers/surefire-junit-platform/src/test/java/org/apache/maven/surefire/junitplatform/RunListenerAdapterTest.java`	Tests identity, filtering, naming, and aborted classes.
`surefire-providers/surefire-junit-platform/src/test/java/org/apache/maven/surefire/junitplatform/JUnitPlatformProviderTest.java`	Tests class-backed rerun discovery.
`surefire-providers/surefire-junit-platform/src/main/java/org/apache/maven/surefire/junitplatform/RunListenerAdapter.java`	Propagates identity and filters rerun events.
`surefire-providers/surefire-junit-platform/src/main/java/org/apache/maven/surefire/junitplatform/JUnitPlatformProvider.java`	Builds filtered rerun discovery requests.
`surefire-providers/common-java5/src/test/java/org/apache/maven/surefire/report/ClassMethodIndexerTest.java`	Tests cached-ID restoration.
`surefire-providers/common-java5/src/main/java/org/apache/maven/surefire/report/ClassMethodIndexer.java`	Restores cached IDs to thread-local state.
`surefire-its/src/test/resources/testng-multiple-method-patterns/pom.xml`	Updates the TestNG fixture version.
`surefire-its/src/test/resources/surefire-2300-before-all-assumption/src/test/java/example/TestAssume.java`	Adds aborted-class fixture tests.
`surefire-its/src/test/resources/surefire-2300-before-all-assumption/pom.xml`	Configures the assumption fixture.
`surefire-its/src/test/resources/surefire-2274-failsafe-flake-count/src/test/java/example/ReproducerIT.java`	Adds fail-twice Failsafe tests.
`surefire-its/src/test/resources/surefire-2274-failsafe-flake-count/pom.xml`	Configures Failsafe flake enforcement.
`surefire-its/src/test/resources/surefire-2260-non-unique-scenarios/src/test/resources/io/cucumber/skeleton/reproducer.feature`	Adds identically named scenarios.
`surefire-its/src/test/resources/surefire-2260-non-unique-scenarios/src/test/java/io/cucumber/skeleton/StepDefinitions.java`	Implements scenario outcomes.
`surefire-its/src/test/resources/surefire-2260-non-unique-scenarios/src/test/java/io/cucumber/skeleton/CucumberReproducerTest.java`	Defines the Cucumber suite.
`surefire-its/src/test/resources/surefire-2260-non-unique-scenarios/pom.xml`	Configures the Cucumber fixture.
`surefire-its/src/test/resources/surefire-2241-junitparams-rerun/src/test/java/org/apache/druid/sql/calcite/CalciteQueryTest.java`	Adds a following-test regression case.
`surefire-its/src/test/resources/surefire-2241-junitparams-rerun/src/test/java/org/apache/druid/sql/calcite/CalciteJoinQueryTest.java`	Reproduces JUnitParams reruns.
`surefire-its/src/test/resources/surefire-2241-junitparams-rerun/pom.xml`	Configures JUnitParams and Vintage.
`surefire-its/src/test/resources/surefire-2239-testng-rerun/src/test/java/example/FlakyTest.java`	Adds a flaky Surefire TestNG test.
`surefire-its/src/test/resources/surefire-2239-testng-rerun/src/test/java/example/FlakyIT.java`	Adds a flaky Failsafe TestNG test.
`surefire-its/src/test/resources/surefire-2239-testng-rerun/pom.xml`	Configures both TestNG plugin paths.
`surefire-its/src/test/java/org/apache/maven/surefire/its/Surefire2300BeforeAllAssumptionIT.java`	Verifies descendant skip reporting.
`surefire-its/src/test/java/org/apache/maven/surefire/its/Surefire2274FailsafeFlakeCountIT.java`	Verifies summary flakes and failure.
`surefire-its/src/test/java/org/apache/maven/surefire/its/Surefire2260NonUniqueScenariosIT.java`	Verifies distinct Cucumber outcomes.
`surefire-its/src/test/java/org/apache/maven/surefire/its/Surefire2241JUnitParamsRerunIT.java`	Verifies JUnitParams reports and flakes.
`surefire-its/src/test/java/org/apache/maven/surefire/its/Surefire2239TestNgRerunIT.java`	Verifies TestNG reruns end to end.
`surefire-its/src/test/java/org/apache/maven/surefire/its/jiras/Surefire1152RerunFailingTestsInSuiteIT.java`	Updates distinct-test expectations.
`maven-surefire-common/src/test/java/org/apache/maven/plugin/surefire/report/StatelessXmlReporterTest.java`	Tests XML identity and skipped reruns.
`maven-surefire-common/src/test/java/org/apache/maven/plugin/surefire/report/DefaultReporterFactoryTest.java`	Tests global identity aggregation.
`maven-surefire-common/src/main/java/org/apache/maven/plugin/surefire/report/TestSetRunListener.java`	Propagates test-run IDs into statistics.
`maven-surefire-common/src/main/java/org/apache/maven/plugin/surefire/report/TestMethodStats.java`	Stores identity in method statistics.
`maven-surefire-common/src/main/java/org/apache/maven/plugin/surefire/report/TestMethodKey.java`	Defines rerun aggregation keys.
`maven-surefire-common/src/main/java/org/apache/maven/plugin/surefire/report/StatelessXmlReporter.java`	Aggregates XML by test identity.
`maven-surefire-common/src/main/java/org/apache/maven/plugin/surefire/report/DefaultReporterFactory.java`	Aggregates global results by identity.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+            if (classSource.isPresent()) {
+                classNames.add(classSource.get().getClassName());
+            } else {
+                builder.selectors(selectUniqueId(identifier.getUniqueId()));
+            }


+                boolean isFailureOrRelatedContainer = failureIds.stream()
+                        .anyMatch(failureId -> candidateId.hasPrefix(failureId) || failureId.hasPrefix(candidateId));


+        return Objects.equals(testClassMethodName, that.testClassMethodName)
+                && Objects.equals(testRunId, that.testRunId);


+        UniqueId candidateId = UniqueId.parse(testIdentifier.getUniqueId());
+        return rerunTestIds.stream()
+                .anyMatch(failureId -> candidateId.hasPrefix(failureId) || failureId.hasPrefix(candidateId));


+<project xmlns="http://maven.apache.org/POM/4.0.0"
+         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+    <modelVersion>4.0.0</modelVersion>


+<project xmlns="http://maven.apache.org/POM/4.0.0"
+         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+    <modelVersion>4.0.0</modelVersion>


+<project xmlns="http://maven.apache.org/POM/4.0.0"
+         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+    <modelVersion>4.0.0</modelVersion>


+<project xmlns="http://maven.apache.org/POM/4.0.0"
+         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+    <modelVersion>4.0.0</modelVersion>


+<project xmlns="http://maven.apache.org/POM/4.0.0"
+         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+    <modelVersion>4.0.0</modelVersion>


wilx marked this pull request as ready for review August 8, 2026 21:20

wilx marked this pull request as draft August 8, 2026 21:32

wilx force-pushed the issue-2686-junit-platform-rerun-identity branch from 171b2c4 to 52fa065 Compare August 8, 2026 22:08

wilx marked this pull request as ready for review August 8, 2026 22:21

This was referenced Aug 8, 2026

[SUREFIRE-2239] Rerun TestNG tests #2626

Closed

[SUREFIRE-2274] failOnFlakeCount does not work for failsafe #2656

Closed

wilx force-pushed the issue-2686-junit-platform-rerun-identity branch from 2642374 to 3f1d74b Compare August 8, 2026 23:14

wilx changed the title ~~[SUREFIRE-2241] Preserve JUnit Platform test identity across reruns~~ [SUREFIRE-2239, SUREFIRE-2241, SUREFIRE-2260, SUREFIRE-2274, SUREFIRE-2300] Preserve JUnit Platform test identity across reruns Aug 8, 2026

olamy added bug Something isn't working conflicts labels Aug 11, 2026

olamy added this to the 3.6.0-M2 milestone Aug 11, 2026

olamy self-assigned this Aug 11, 2026

wilx added 5 commits August 11, 2026 07:55

[SUREFIRE-2239] Add TestNG rerun integration test

780b4d1

Recover the fail-first/pass-second TestNG reproducer and exercise it through both Surefire and Failsafe using the JUnit Platform provider. Assert that each final report records one flake. Fixes apache#2626

[SUREFIRE-2274] Add Failsafe flake-count integration test

8363078

Adapt the linked reproducer to verify that Failsafe persists rerun flakes in its summary and enforces failOnFlakeCount during verify. Fixes apache#2656

[SUREFIRE-2300] Report BeforeAll assumption aborts as skipped

6ceb4b8

Use the JUnit Platform TestPlan to report every discovered test under a class aborted by a BeforeAll assumption as skipped while preserving completed results for later container aborts. Fixes apache#2602

[SUREFIRE-2241] Avoid TestNG parallel fixture race

51d6fab

Update the method-pattern fixture to TestNG 7.5.1, the latest version supporting Java 8, to avoid a suspected class-lifecycle race fixed after TestNG 6.14.3. Fixes apache#2686

wilx force-pushed the issue-2686-junit-platform-rerun-identity branch from 2d29ea4 to 51d6fab Compare August 11, 2026 05:58

olamy requested a balanced review from Copilot August 12, 2026 00:05

olamy removed the conflicts label Aug 12, 2026

Copilot started reviewing on behalf of olamy August 12, 2026 00:05

View session

Copilot AI reviewed Aug 12, 2026

View reviewed changes

olamy merged commit 6bd229c into apache:master Aug 14, 2026
14 checks passed

pan3793 mentioned this pull request Aug 19, 2026

Fix missing UniqueId import in RunListenerAdapter #3436

Merged

1 task done

wilx deleted the issue-2686-junit-platform-rerun-identity branch August 20, 2026 17:09

olamy mentioned this pull request Aug 28, 2026

Fixing issues of new surefire mode using Junit Platform support-and-care/maven-support-and-care#246

Open

Navigation Menu

Repository navigation

[SUREFIRE-2239, SUREFIRE-2241, SUREFIRE-2260, SUREFIRE-2274, SUREFIRE-2300] Preserve JUnit Platform test identity across reruns - #3418

[SUREFIRE-2239, SUREFIRE-2241, SUREFIRE-2260, SUREFIRE-2274, SUREFIRE-2300] Preserve JUnit Platform test identity across reruns#3418
olamy merged 5 commits into
apache:masterfrom
wilx:issue-2686-junit-platform-rerun-identity

wilx commented Aug 8, 2026 •

edited

Loading

Uh oh!

wilx commented Aug 9, 2026

Uh oh!

olamy commented Aug 11, 2026

Uh oh!

wilx commented Aug 11, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		boolean isFailureOrRelatedContainer = failureIds.stream()
		.anyMatch(failureId -> candidateId.hasPrefix(failureId) \|\| failureId.hasPrefix(candidateId));

		return Objects.equals(testClassMethodName, that.testClassMethodName)
		&& Objects.equals(testRunId, that.testRunId);

📋 Server Request & Proxy Logs

⚙️ Pengaturan Proxy Target

Conversation

wilx commented Aug 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

User impact

Tests

Uh oh!

wilx commented Aug 9, 2026

Uh oh!

olamy commented Aug 11, 2026

Uh oh!

wilx commented Aug 11, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

wilx commented Aug 8, 2026 •

edited

Loading