Build projects that are both multi-module and multi-release.

[desruisseaux]

Before this commit, a project could be either multi-module or multi-release, but not both in same time. As a side effect of the debugging work, this commit also fixes the content of the target/javac.args file: before this commit, the source files for all versions were mixed in the same target/javac.args file. After this commit, separated javac.args, javac-17.args, javac-21.args, etc. files are produced.

Directory layout choice

The directory structure produced in the target directory is like below:

target
├─ foo.bar.module1
│  └─ foo/bar/module1/*.class
├─ foo.bar.module2
│  └─ foo/bar/module2/*.class
└─ META-INF
   └─ versions
      └─ 17
          ├─ foo.bar.module1
          │  └─ foo/bar/module1/*.class
          └─ foo.bar.module2
             └─ foo/bar/module2/*.class

This directory structure will not be immediately usable by the Maven JAR plugin, because that plugin would rather expect the META-INF/versions/17 path to be repeated inside each module. But we cannot easily comply with this expectation, because the Java compiler accepts only one -d argument for all modules. We could move the directories after compilation, but incremental compilation of only a few files would require that we move the directories back to the location shown above before compilation, then move these directories again after the compilation. Furthermore, the users would not be able to compile themeselves on the command-line, unless they perform those moves manually (that would be tedious).

I believe that it is easier, more convenient and less risky to leave the directory structure as produced by javac, and instead modify the Maven JAR plugin for reading the files at the locations shown above. An evolution of the Maven JAR plugin will be needed anyway, as the current version will not understand those org.bar.module1 and org.bar.module2 directories anyway, no matter if multi-release or not.

[desruisseaux]

This pull request partially depends on apache/maven#11118. But it still work without that fix if there is at most one version after the base version.

[desruisseaux]

I'm not sure who to ask for a review. Maybe @slawekjaranowski ? Keeping in mind that the decision about "Directory layout choice" is probably not too important for now, because most developers will probably not use it before other plugins have been updated.

[gnodet]

This should also relate to the work needed on m-sources-p to better support source files for multi-release jars.
See apache/maven-source-plugin#234

[gnodet]

This directory structure will not be immediately usable by the Maven JAR plugin, because that plugin would rather expect the META-INF/versions/17 path to be repeated inside each module. But we cannot easily comply with this expectation, because the Java compiler accepts only one -d argument for all modules. We could move the directories after compilation, but incremental compilation of only a few files would require that we move the directories back to the location shown above before compilation, then move these directories again after the compilation. Furthermore, the users would not be able to compile themeselves on the command-line, unless they perform those moves manually (that would be tedious).

Why does the m-jar-plugin comes into play here ? AFAIK, it's completely agnostic and simply jars whatever is in the target/classes directory. Or you mean, this layout is not directly usable by the JVM ? if so, I'd rather comply with what the JVM expects. Can we instruct javac to output the correct files in the final layout:

├─ module-info.class (base version)
├─ foo/bar/module1/
│  └─ *.class
├─ foo/bar/module2/
│  └─ *.class
└─ META-INF/
   ├─ MANIFEST.MF
   └─ versions/
      └─ 17/
         ├─ module-info.class (Java 17+ version)
         ├─ foo/bar/module1/SomeClass.class (only if different from base)
         └─ foo/bar/module2/AnotherClass.class (JDK-17 specific)

One of the reason is that the target/classes can be added to the classpath when building other subprojects, so I think we need to keep it in sync with JVM's expectations.

[desruisseaux]

When target/classes/ is added to the classpath of other subprojects, the directory layout discussed in this issue does not matter, because META-INF/versions/ will be ignored no matter its location. The META-INF/versions/ sub-directories are scanned only if included in a JAR file. Even in that case, they are scanned only if explicitly requested. This is the topic of apache/maven#11153 (by default, even java.util.jar.JarFile ignores these sub-directories).

If we want target/classes/META-INF/versions/ directories to be taken in account during tests and in other sub-projects, it will require modifications in other plugins in any cases, regardless this issue. We will need to put those directories ourselves on the class-path. This is something that I'm planing to contribute: propose improvement in Surefire plugin for JPMS support, and opportunistically improve the multi-release support in same time. Maybe with a Surefire plugin configuration option such as <releaseToTest> which would control which target/classes/META-INF/versions/ directories to add on the class-path or patch-module paths.

Regarding the JAR plugin, there are two scenarios in the context of JPMS. A key point is that module hierarchy is an optional feature of Java. It is possible to do JPMS with package hierarchy, at the cost of being constrained to a single JPMS module per Maven sub-project. When using package hierarchy, the target/classes/ directory content can be used directly by the JAR plugin, including the META-INF/versions/ sub-directories. But when using the module hierarchy, the content of target/classes/ cannot be used directly by the JAR plugin (it can be used directly by the --module-path option of other sub-projects however). The reason is that the JAR format is not yet multi-module. I saw discussion on the OpenJDK mailing list about potentially creating such format, but I don't know if any work started. In the meantime, the JAR plugin would have to create one JAR file per module only in the specific case where module source hierarchy is used. If we accept to put some intelligence in the plugin for that specific case, it could handle the above-described layout directory in same time.

NOTE: the fact that other plugins such has Surefire and JAR have not yet been updated does not block users from using the JPMS support provided by this pull request and #963. It only blocks the use of module source hierarchy. Other JPMS features are available and can be used now if users stick to package hierarchy.

Can we instruct javac to output the correct files in the final layout

For a sub-project using package hierarchy, regardless if JPMS or not, yes and it is already the case. For module hierarchy, not as far as I know. But as said above, module hierarchy is optional even for a JPMS project, and will require update in other plugins anyway, regardless the directory layout discussed above. Even for a classical class-path project ignoring all the JPMS stuff, improvement of the Surefire plugin is needed anyway if we want to test any version other than the base version.

[desruisseaux]

Documentation about this feature has been added in #976, in particular in the sources.md file. Would it help acceptance if I add a commit for printing a message saying that this feature is experimental when multi-module and multi-release are used in same time?

[Copilot]

Pull Request Overview

This PR extends the Maven compiler plugin to support projects that are both multi-module and multi-release, previously only one or the other was supported. It also fixes the separation of javac.args files by Java release version, creating separate files like javac-17.args, javac-21.args, etc.

• Adds support for multi-module, multi-release projects through new directory structure handling
• Implements WorkaroundForPatchModule class to handle compiler compatibility issues with patch module paths
• Refactors debug file generation to create separate javac.args files per Java release

Reviewed Changes

Copilot reviewed 20 out of 20 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
WorkaroundForPatchModule.java	New workaround class for handling unsupported patch module operations in some compilers
ToolExecutor.java	Major refactoring to support multi-module multi-release compilation with new dependency management
AbstractCompilerMojo.java	Updates debug file writing to generate separate files per Java release
SourcesForRelease.java	Adds dependency snapshot tracking and output directory path for debug file generation
Options.java	Fixes command line formatting to handle -J options properly
SourcePathType.java	Adds equals/hashCode implementation for proper map usage
Integration test files	New test case demonstrating multi-module multi-release functionality

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[gnodet]

The reason is that the JAR format is not yet multi-module.

Are we switching to a mode where we'd have a single project, but it would create multiple jars in one run ? Afaik, most plugins expects a single main artefact as the main output, while others side artifacts can be attached to the project, but on the same groupId/artifactId. So generating multiple main artifacts goes really against the way of doing things in maven. I'm not opposed to a change, but this would require a larger consensus imho.

[desruisseaux]

Yes, compiling a project which uses the module source hierarchy would result in many JAR files, one per module. For each JAR, the dependencies in the pom.xml would be the intersection of the dependencies declared in the pom.xml of the Maven project and the dependencies declared directly or indirectly (including transitive dependencies and service providers) in the module-info.java of each Java module. In other words, Maven <dependencies> section would effectively become a <dependenciesManagement>.

Note 1: this is already the case in Maven 3: any dependency declared in Maven <dependencies> will be ignored (except for some particular things such as annotation processor) by the compiler if it is not implied by some statements in module-info, even in single-module projects using the package hierarchy.

Note 2: this idea is not only mine. I saw it independently expressed somewhere on GitHub, but couldn't find it back. For information, some of the ideas related to module-info support are also found in slightly different forms in the Christian Stein's blog, themselves based on Robert Scholte's emails on OpenJDK mailing list (references in the blog). I think that the same idea appears independently because in a modular world, the fact that <dependencies> would effectively behave as <dependenciesManagement> is not in our control (unless we choose to generate --add-modules options).

Note 3: about 10 years ago, the maven-jar-plugin was just zipping whatever exist in target/classes/ with the addition of a MANIFEST.MF file. But now, there are some options such as --hash-modules which require a module-path. So creating a JAR file is already becoming more <dependencies>-dependent than before.

Note 4: while Maven is still quite JAR-oriented, JAR files are not necessarily the main artifacts anymore. The main artifact could be the jlink output, in which case a multi-module project can still produce a single artifact (the executable application), and *.jar files are only intermediate outputs like the *.class files.

[desruisseaux]

I added a commit for enabling multi-release in modular projects only if the -Dmaven.compiler.preview=true option is given to the mvn command. If this option is not provided, the compilation fails with the following error message:

[ERROR] Failed to execute goal (…snip long coordinate…) on project multirelease-with-modules:
Cannot compile org.apache.maven.plugins:multirelease-with-modules:jar:1.0-SNAPSHOT main classes.
[ERROR] The first error is: Multi-release in a modular project is a preview feature.
For enabling this feature, add the -Dmaven.compiler.preview=true option to the mvn command.

If a preview flag is added in Maven core, we could replace the maven.compiler.preview property by that flag in a future version. This approach would bring the bug fixes and improvements of this pull request when the directory layout is classic (i.e., as Maven 3) without committing ourselves to the directory structure described in the "Directory layout choice" section in the summary of this pull request. That layout would be disabled by default.

@gnodet, @cstamas or @slawekjaranowski, would it be an acceptable way to go forward?

[desruisseaux]

I started to dig in the JAR plugin with apache/maven-jar-plugin#481. Below are my remarks about the multiple JARs in one run:

• (Reminder) For the Surefire plugin, it is not an issue. The java command takes all modules together as one entry on the module-path, except META-INF/versions/ which is not supported anyway even with the classical layout (we need to setup class-path / module-path ourselves).
• For the JAR plugin, we already have a code equivalent to the following:

ProducedArtifact artifact = session.createProducedArtifact(
        project.getGroupId(),
        project.getArtifactId(),
        project.getVersion(),
        classifier,
        null,
        getType());

projectManager.attachArtifact(project, artifact, jarFile);

This code would be invoked in a loop, with artifactId derived from the module name by default and the dependencies set to the intersection of <dependencies> and module-info.

[desruisseaux]

Removed the maven.compiler.preview flag since two replies on the question asked on Slack suggest that such flag wouldn't hep.

I have the feeling that we will not have feedback before peoples start using this feature. Are we ready to merge (on a community point of view)?

[gnodet]

Yes, I agree.
Let's merge, the plugins are still in beta anyway, so let's release and get feedback.

[gnodet]

The reason is that the JAR format is not yet multi-module.

Are we switching to a mode where we'd have a single project, but it would create multiple jars in one run ? Afaik, most plugins expects a single main artefact as the main output, while others side artifacts can be attached to the project, but on the same groupId/artifactId. So generating multiple main artifacts goes really against the way of doing things in maven. I'm not opposed to a change, but this would require a larger consensus imho.

@desruisseaux fwiw, apache/maven#1838, we may need to revert that one

[desruisseaux]

Maybe we do not need to change maven#1838. We could keep the current view — each Maven sub-project generates a single main artefact — and instead see each Java module as a dynamic Maven sub-project, except that the sub-project is described by module-info.java instead of pom.xml.

The reason why I'm thinking to this approach is because each Java module will need its own artifactId anyway for deployment on Maven Central (the artifactId could default to the Java module name) with its own pom.xml generated by intersecting the parent pom.xml with module-info. It would be consistent with the new approach in Maven 4 where the pom.xml used for building the project is no longer the same (or not necessarily the same) as the deployed pom.xml.

So in short, I envision that a Maven project using Module Source Hierarchy could be seen as a parent pom.xml with sub-projects defined by the module-info.java files. It may be possible to fit this approach with the "one main artefact per sub-project" expectation.

[desruisseaux]

For being more specific:

• The Maven Compiler plugin has only one main output: the target/classes directory. This output can be used directly by any Java tools having a --module-path argument, such as java and javadoc, but not jar (well… jar also has a --module-path argument, but used only for hashing).
• The JAR plugin could generate dynamically one sub-project per Java module, and each dynamically generated sub-project has its own generated pom.xml and one single main artefact.

The dynamic sub-projects would be generated by the JAR plugin because jar is the Java tools which is not multi-module. Most other Java tools are multi-module. It makes sense that JAR is not multi-module, because it makes smaller files to download.

Even in a non-JPMS world, the view that the JAR plugin should not do much more than just zipping everything in the target/classes/ directory may be obsolete anyway. See for example apache/maven-jar-plugin#484.

[gnodet]

I'm not sure about generating subprojects dynamically. Subprojects are subject to lifecycles, plugin binding, etc... I'm not sure we really want to go that way. I guess some experiments are needed, but the easiest will probably to revert the additional check I mentioned and let jar, sources and javadoc generate multiple artifacts under various groupId/artifactId.

For being more specific:

• The Maven Compiler plugin has only one main output: the target/classes directory. This output can be used directly by any Java tools having a --module-path argument, such as java and javadoc, but not jar (well… jar also has a --module-path argument, but used only for hashing).
• The JAR plugin could generate dynamically one sub-project per Java module, and each dynamically generated sub-project has its own generated pom.xml and one single main artefact.

The dynamic sub-projects would be generated by the JAR plugin because jar is the Java tools which is not multi-module. Most other Java tools are multi-module. It makes sense that JAR is not multi-module, because it makes smaller files to download.

Even in a non-JPMS world, the view that the JAR plugin should not do much more than just zipping everything in the target/classes/ directory may be obsolete anyway. See for example apache/maven-jar-plugin#484.

I'm not really sure we need to create subprojects dynamically. If it's just the jar.

[desruisseaux]

Indeed, if we can allow jar generates multiple artefacts under different artifactId, that would probably be easier.