Access Javac API from a standalone javac jar instead of the default JDK runtime

[testforstephen]

This PR tries to introduce a standalone Javac jar from the repository https://github.com/JaroslavTulach/nb-javac for Javac API access within our project. The benefit is that it allows the language server to support latest language features without requiring user to run the language server with the latest JDK.

This PR is an experiment PR, not full ready yet. It faces some issues with new approach, and I open the PR first for suggestions and feedback.

• The project can compile fine with the embedded nb-javac, but get the module conflict issue during runtime. It reports module conflicts between jdk.compiler module and unnamed module from the embedded nb-javac jar. When running it in Eclipse, it throws exception below.

java.lang.ClassCastException: class com.sun.tools.javac.api.JavacTool cannot be cast to class com.sun.tools.javac.api.JavacTool (com.sun.tools.javac.api.JavacTool is in module jdk.compiler of loader 'app'; com.sun.tools.javac.api.JavacTool is in unnamed module of loader org.eclipse.osgi.internal.loader.EquinoxClassLoader @d014059)
   at org.eclipse.jdt.core.dom.JavacCompilationUnitResolver.parse(JavacCompilationUnitResolver.java:690)
   at org.eclipse.jdt.core.dom.JavacCompilationUnitResolver.toCompilationUnit(JavacCompilationUnitResolver.java:541)
   at org.eclipse.jdt.core.dom.ASTParser.internalCreateASTCached(ASTParser.java:1299)
   at org.eclipse.jdt.core.dom.ASTParser.lambda$1(ASTParser.java:1178)
   at org.eclipse.jdt.internal.core.JavaModelManager.cacheZipFiles(JavaModelManager.java:5692)
   at org.eclipse.jdt.core.dom.ASTParser.internalCreateAST(ASTParser.java:1178)
   at org.eclipse.jdt.core.dom.ASTParser.createAST(ASTParser.java:918)
   at org.eclipse.jdt.internal.core.CompilationUnit.buildStructure(CompilationUnit.java:184)
   at org.eclipse.jdt.internal.core.Openable.generateInfos(Openable.java:246)
   at org.eclipse.jdt.internal.core.JavaElement.openWhenClosed(JavaElement.java:569)
   at org.eclipse.jdt.internal.core.JavaElement.getElementInfo(JavaElement.java:292)
   at org.eclipse.jdt.internal.core.JavaElement.getElementInfo(JavaElement.java:278)
   at org.eclipse.jdt.internal.core.CompilationUnit.getSourceRange(CompilationUnit.java:1087)

• The packaged nb-javac only includes the classes of jdk.compiler, no jdk.javadoc classes. To bypass the compilation error, I temporarily comment out the jdk.javadoc classes reference from JavadocConverter.java.
• License review. nb-javac is under GPL-2.0 license, require legal review.

[mickaelistria]

Does the launch configuration or the java command still set the --add-exports ... flag? If so, please make sure they're removed from most places. The META-INF/p2.inf file could probably be removed too.

[testforstephen]

I removed the --add-exports ... flags from the launch configuration. However, Eclipse automatically adds --add-modules=ALL_SYSTEM to the launch command, which includes the java.compiler module in the runtime by default. This causes the call to ToolProvider.getSystemJavaCompiler() to use the default java.compiler api and return the default JDK compiler instance and its associated jdk.compiler module. Since nb-javac already contains all the classes from both java.compiler and jdk.compiler, we need to find a way to ensure that the nb-javac JAR is prioritized over the JDK module.

eclipse-jdt-core-incubator/org.eclipse.jdt.core.javac/src/org/eclipse/jdt/core/dom/JavacCompilationUnitResolver.java

Line 565 in d8b4f12

var compiler = ToolProvider.getSystemJavaCompiler();

[rgrunber]

Found this on ClearlyDefined with a score that makes it usable, https://clearlydefined.io/definitions/sourcearchive/mavencentral/com.dukescript.nbjavac/nb-javac/jdk-19+33 (not useable, License score is what counts, and it's 25, must be at least 60) . I have filed https://gitlab.eclipse.org/eclipsefdn/emo-team/iplab/-/issues/17140 for the particular version here.

Update: It is Approved ✔️

[testforstephen]

In recent commits, I addressed two issues:

• The first issue is java.lang.ClassCastException: class com.sun.tools.javac.api.JavacTool cannot be cast to class com.sun.tools.javac.api.JavacTool (com.sun.tools.javac.api.JavacTool is in module jdk.compiler of loader 'app'; com.sun.tools.javac.api.JavacTool is in unnamed module of loader org.eclipse.osgi.internal.loader.EquinoxClassLoader @d014059). When launching Eclipse plugins with JDK 9+, most java.* modules (including java.compiler) are loaded as root modules by default, which is not configurable. To workaround the module conflicts, I use com.sun.tools.javac.api.JavacTool.create() to obtain a compiler instance, rather than javax.tools.ToolProvider.getSystemJavaCompiler().

• The second issue is that nbjavac.VMWrapper fails to resolve ct.sym from the OSGi classloader. The nb-javac library includes wrappers to load JDK resources, relying on the classloader mechanism for this purpose. However, its implementation assumes that NetBeans loads the nbjavac JARs as standard jar dependencies on the classpath. In the Eclipse environment, where the javac JAR is embedded within a bundle, the classloader behaves differently, causing the nbjavac implementation to fail in an OSGi bundle context. To address this, I implemented a workaround to help VMWrapper load ct.sym from the OSGi bundle resources.

In the latest commit, the DOM resolver already works with the embedded Javac. But for the build support, it still has some problems because the apt plugin is not loaded well. Here are some potential issues to look further.

• The one unresolved issue is that org.eclipse.jdt.apt.pluggable.core plugin cannot be compiled well because the module namespace conflicts between the java.compiler and unnamed module for the embedded Javac jar. This occurs because the javac plugin is implemented as a fragment host of the org.eclipse.jdt.core plugin, causing any plugin that depends on jdt.core to automatically include the javac plugin in its dependencies. As a result, the transitive unnamed nbjavac module in Javac plugin conflicts with the java.compiler dependency of the apt plugin. To resolve this, we may need to refactor the alternative compiler API support in AbstractImageBuilder into an extension point. This can make the javac plugin into a standalone plugin, and isolate it from conflicts with other plugins.

• Another potential issue relates to Lombok compatibility. Similar to ECJ, Lombok support can encounter compatibility issues when used with Javac. Just as a new ECJ version might disrupt Lombok functionality, a new Javac release can also break compatibility with Lombok. In jdt.ls, we typically embed the latest lombok.jar and utilize it alongside ECJ. However, with Javac, it defaults to using a project's own Lombok dependency, loading it as a standard annotation processor in Javac. I recently observed that lombok 1.18.34 is not compatible with javac 23 because of new Javadoc syntax. As a result, applications still using Lombok 1.18.34 will encounter resolution errors with Javac 23. In future, we might consider using the embedded Lombok version in jdt.ls for the Javac parsing as well.

[testforstephen]

The one unresolved issue is that org.eclipse.jdt.apt.pluggable.core plugin cannot be compiled well because the module namespace conflicts between the java.compiler and unnamed module for the embedded Javac jar. This occurs because the javac plugin is implemented as a fragment host of the org.eclipse.jdt.core plugin, causing any plugin that depends on jdt.core to automatically include the javac plugin in its dependencies. As a result, the transitive unnamed nbjavac module in Javac plugin conflicts with the java.compiler dependency of the apt plugin. To resolve this, we may need to refactor the alternative compiler API support in AbstractImageBuilder into an extension point. This can make the javac plugin into a standalone plugin, and isolate it from conflicts with other plugins.

I attempted to convert the org.eclipse.jdt.core.javac bundle from a fragment to a plugin and used an extension point to register an alternative Javac compiler with AbstractImageBuilder. The image builder successfully locates and utilizes JavacCompiler. However, JavacCompilationUnitResolver fails because it depends on several package-private classes (e.g., org.eclipse.jdt.core.dom.BindingResolver) from the org.eclipse.jdt.core plugin.

java.lang.IllegalAccessError: class org.eclipse.jdt.core.dom.JavacBindingResolver cannot access its superclass 
org.eclipse.jdt.core.dom.BindingResolver (org.eclipse.jdt.core.dom.JavacBindingResolver is in unnamed module of loader 
org.eclipse.osgi.internal.loader.EquinoxClassLoader @5f618123; org.eclipse.jdt.core.dom.BindingResolver is in unnamed module 
of loader org.eclipse.osgi.internal.loader.EquinoxClassLoader @90ba3b1)

I guess this dependency issue is why the javac plugin is set up as a fragment of jdt.core, allowing it to share the same classloader.

[mickaelistria]

See https://ci.eclipse.org/ls/job/jdt-core-incubator/job/PR-924/3/console , this breaks current compilation since some Java 21 features are already adopted in the code. Would 21 be fine?

[testforstephen]

yes, it breaks Java 21 usage (e.g. some utility of String and pattern matching). We need to rewrite them with Java 17 compatible way. The purpose here is to make the minimum JDK requirement same as other JDT plugins.

[mickaelistria]

IMO, at this stage, we should minimize the changes in the code and stick to Java 21 to facilitate progress. When we've adopted nb-javac, then we'll be more easily able to move code back to Java 17. Moreover, m2e already requires Java 21, so JDT-LS already requires Java 21, so trying to go below that won't produce any benefit for JDT-LS nor vscode-java.

[testforstephen]

I see redhat.java extension still requires a minimum Java 17 because it stays at m2e@2.6.0. Good to know we're going to upgrade to Java 21.