eclipse / transformer

Eclipse Transformer provides tools and runtime components that transform Java binaries, such as individual class files and complete JARs and WARs, mapping changes to Java packages, type names, and related resource names.

Eclipse Transformer

The Eclipse Transformer project is part of the Eclipse Technology top-level project.

Eclipse Transformer provides tools and runtime components that transform Java binaries, such as individual class files and complete JARs and WARs, mapping changes to Java packages, type names, and related resource names.

While the initial impetus for the project was the Jakarta EE package renaming issue, the scope of the project is broader to support other renaming scenarios. For example, shading.

We operate under the Eclipse Code of Conduct to promote fairness, openness, and inclusion.

Build Status

Latest release

Project licence

All Eclipse Transformer project materials are made available under the Eclipse Public License 2.0 or the Apache 2.0 license. You can choose which license you wish to follow. Please see our LICENSE file for more details.

Summary

The Eclipse Transformer is a generic framework for transforming resources (files and archive) based on updates specified by property files.

The Eclipse Transformer is provided with two specific built-in capabilities:

• To update the target resources according to user specified rules data, which includes Jakarta package renaming rules. The target resource is an JavaEE version of an implementation jar, or is a JavaEE version of an OSGi implementation jar, or is a JavaEE version of an application archive.

• To shade a target resource according to user specified rules, including, in particular, package renaming rules. The target resource is an implementation jar.

In addition to the specific built-in capabilities, other transformations may be performed by changing the rules data, or by adding new types of update action.

The Eclipse Transformer is provided with Command Line and with Java APIs:

• Transformer classes may be used directly within java code.

• Transformation function is accessible through two Command Line APIs. A generic command line API, which requires user specified update data, may be used. A Jakarta command line API, which has built-in Jakarta rules data, may also be used.

Scope

The Eclipse Transformer operates on resources, including but not limited to the following:

Simple resources:

• Java class files
• OSGi feature manifest files
• Properties files
• Service loader configuration files
• Text files (of several types: java source, XML, TLD, HTML, and JSP)

Container resources:

• Directories
• Java archives (JAR, WAR, RAR, and EAR files)
• ZIP archives

APIs

The Eclipse Transformer provides three core APIs: A java entry point and two command line entry points:

• Class org.eclipse.transformer.Transformer provides for direct invocation from java.

• Class org.eclipse.transformer.cli.TransformerCLI provides a standard main command line entry point.

• Class org.eclipse.transformer.cli.JakartaTransformerCLI provides an addition main command line entry point. JakartaTransformerCLI uses JakartaTransform which is packaged with default update properties files and supplies these as default property files for the several update properties files.

Use Cases

Primary use cases are as follows:

• Simple resource case: A target simple file is specified to be transformed. The command line interface is used to create a transformed copy of the file. Particular transformations are selected for the file based on the type of the file. Particular updates made by the transformation are based on property files which are specified to the transformer.

• Archive case: A target archive is specified to be transformed. The command line interface is used to create a transformed copy of the archive. All contents of the target archive, including nested archives, are transformed. Particular transformations are selected for the archive based on the type of the file. Particular updates made by the transformation are based on property files which are specified to the transformer.

• Directory case: A target directory is specified to be transformed. The command line interface is used to create a transformed copy of all the resources within the directory, and within all subdirectories of the specified directory. Transformation of each file within the target directory is according the simple resource case and the archive case.

Possible secondary use cases are as follows:

• Classloading case: A java class action is created and initialized, and is wired into a custom class loader. When loading resources, in particular, when loading a class resource, the java class action is used to transform the contents of the resource before making them available to the class loader.

Usage

The transformer CLI can be invoked by using java -jar org.eclipse.transformer.cli-<version>.jar from now on referred as TransformerCLI. This file can be found in the org.eclipse.transformer.cli/target folder when you build this project by yourself, or you can download and extract the org.eclipse.transformer.cli-<version>-distribution.jar into a directory of your choice and start it from there.

The primary use of the Eclipse Transformer is to create an updated copy an input file or directory. Generally:

TransformerCLI input [ output ] [ options ... ]

To get more information on available options use:

TransformerCLI --help

For simplicity, the transformer does not perform any "in-place" updates. Replacement of an input file with an output file must be performed as an operation external to the transformer.

The input file is expected to be either a specific type of file (for example, a java class file), or a container type file; usually, a zip, java, or JavaEE archive, or a directory.

In addition to the output file, the transformer produces a change report. The change report tells if any updates were made to the target file, and provides details on what changes were made. What details are provided depends on the type of file which is transformer.

When the input file is a container type file, the transformer is applied to all the elements of the container.

The transformer recursively processes nested archives. For example, Web Application Archives (WAR files) located within Enterprise Application Archives (EAR files) are processed.

Updates

A core function of the Eclipse Transformer is to locate java package references within resources and to update these references using package rename data.

Additional functions are to update OSGi bundle metadata, for example, to change the bundle metadata, or to change package versions for renamed package references.

For class shading, usual target resources are java class files or a java archives.

For Jakarta related updates, usual target resources are implementation jar files and JavaEE application archives.

Rules Data Files

Update data is specified using property files. A set of default property files are included in the packaging of the Jakarta command line interface. User specified property files must be provided when using the generic command line interface. Property files are specified using command line options:

Command Line Option	Description	Jakarta Example
-ts --selection	Target selections	jakarta-selection.properties
-tr --renames	Package renames	jakarta-renames.properties
-tv --versions	Package version assignments	jakarta-versions.properties
-tb --bundles	Bundle identity data	jakarta-bundles.properties
-td --direct	Java class direct updates	jakarta-direct.properties

All property options may be specified zero, one, or multiple times.

When the same property option is specified multiple times, the property file data for that option is read first to last, where later data overwrites earlier data. Overwriting occurs on individual property values.

An additional core property file is used as a master table for text updates:

Command Line Option	Description	Jakarta Example
-tf --text	Text master data	jakarta-text-master.properties

The master text properties file maps file selection patterns (for example, "*.xml", or "*.html") to specific text properties files.

A special "immediate" option exists that allows the specification of rules data directly on the command line:

Command Line Option	Description	Jakarta Example
-ti --immediate	Immediate data	None

The immediate option has the format:

-ti ( tr | tv | tb | td | tf ) key value

The first parameter is the tag for one of the usual property options, specified without the '-' prefix.

The "key" and "value" values are composed to create a property definition, the equivalent of specifying "key=value" in a properties file. The property value is taken as new property data according to the tag which was used.

Immediate options are processed after property file options are processed. Individual property definitions obtained as immediate data are overlaid onto and replace individual property definitions obtained from property files.

Immediate options may be used with or without specifying a corresponding properties file.

Artifacts

The Eclipse Transformer artifacts are released to Maven Central.

Snapshot artifacts are released to the Sonatype OSS repository: https://oss.sonatype.org/content/repositories/snapshots/.

To use the snapshot artifacts in your Maven build, you can configure the Sonatype OSS snapshot repository.

<repositories>
    <repository>
        <id>ossrh</id>
        <url>https://oss.sonatype.org/content/repositories/snapshots/</url>
        <layout>default</layout>
        <releases>
            <enabled>false</enabled>
        </releases>
    </repository>
</repositories>

To use the snapshot transformer maven plugin in your Maven build, you can configure the Sonatype OSS snapshot repository.

<pluginRepositories>
    <pluginRepository>
        <id>ossrh</id>
        <url>https://oss.sonatype.org/content/repositories/snapshots/</url>
        <layout>default</layout>
        <releases>
            <enabled>false</enabled>
        </releases>
    </pluginRepository>
</pluginRepositories>

Contributing

If you would like to contribute to Transformer, check out the contributing guide for more information.

Additional documentation

See the following files for more detailed information:

• INFRA.md Project infrastructure and build notes
• DEV.md Developer overview
• RULES.md Transformer actions and rules details

transformer's Issues

Fill in missing unit tests

From @tbitonti

There are a handful of unit tests, not even close to what are needed.

Use this issue for deliveries of missing unit tests.

Incorrect transformation of annotation value.

We have some edge cases on the TomEE transformation where annotations are not translated correctly. Here's an example:

https://github.com/dblevins/tomee-analysis/blob/master/apache-tomee-microprofile-8.0.3-SNAPSHOT.zip/apache-tomee-microprofile-8.0.3-SNAPSHOT/lib/javaee-api-8.0-4.jar/jakarta/validation/constraints/AssertFalse-asmified.java#L69

This class has been renamed correctly, but an annotation has been missed. I'll add a test for this case and fix.

Add package renames for outstanding package renames.

From @tbitonti

Add to the default transformation rules all remaining package renames for jakarta.

Initially, many of these will be commented out, pending the availability of the corresponding implementation.

Aquire open-liberty rules into the jakarta defaults (20-May-2020)

From @tbitonti

Refresh of the default jakarta rules with the latest updates from open-liberty.

Annotation String parameter value not transformed

For example:
@MessageDriven(name = "TestEnvEntryMD", activationConfig = {
@ActivationConfigProperty(propertyName = "destinationType", propertyValue = "javax.jms.Queue"),
@ActivationConfigProperty(propertyName = "destination", propertyValue = "java:jboss/queue/testEnvEntry") })
public class TestEnvEntryMDBean implements MessageListener {
}

"javax.jms.Queue" is of type String and https://github.com/eclipse/transformer/blob/master/org.eclipse.transformer/src/main/java/org/eclipse/transformer/action/impl/ClassActionImpl.java#L983
doesn't handle String type. Direct String replacement should occur.

Service Loader files can become corrupted

The org.eclipse.transformer.action.impl.ServiceLoaderConfigActionImpl action fails to check the inputLength in the apply method. When transforming a zip/jar file, this can result in corruption in the transformed file, such as additional null characters, or other data from the zip file.

We've observed this "in the wild" while transforming the Mojarra jar file while running the transformer on Apache TomEE.

Update version to 0.1.1

From @tbitonti

Update version for the changes to XML/Text updates, and for acquiring the most recent rules data from open-liberty

Consider renaming `master` branch to `main` for inclusivity

In comport with being a more inclusive world, we should consider renaming the "main" branch from master to main.

Preserve manifest formatting: Attribute ordering and comments

From @volosied

When I run the transfer tool on a feature manifest, the attributes are reorganized. Rules and Manifests are attached.

rules.zip

mf.zip

I don't believe this should affect the transformation's ability to run, but I think it should output everything in the same order for consistency. And it's easier to run diff on it, too.

I upgraded this to a bug. The manifest and feature updates should keep these files in their original format / order as much as possible. That means preserving attribute order. That also means preserving comments. White space generally cannot be preserved because of manifest format requirements.

Update bndlib dependency to latest 5.0.0 release

We should stay current on the latest bndlib version.

Centralize XML factory object creation for proper configuration

We need to properly configure XML factory objects to make sure they are
not vulnerable to certain security attacks such as XML External Entity
(XXE) attacks.

See https://github.com/bndtools/bnd/blob/master/aQute.libg/src/aQute/lib/xml/XML.java for an example of this. We can pull this package into the biz.aQute.bnd.transform dependency that we use from Bnd.

More options for package name matching.

From @tbitonti

The rules for matching package names needs to be reviewed.

The question is to require exact whole package matches, or to allow prefix matching.

For example, currently, "javax.servlet" matches "javax.servlet" but not "prefixjavax.servlet",
"prefix.javax.servlet", "javax.servletsuffix", or "javax.servlet.suffix".

This might be widened to enable matching against "prefix.javax.servlet" and "javax.servlet.suffix".

However, the case of "javax.transaction.xa" is a problem. "javax.transaction" was renamed within Jakarta EE 9, but "javax.transaction.xa" was not renamed. Any prefix and suffix matching which is added must provide settings which disallow the transaction case.

That is, if "javax.transaction" is allowed to match "javax.transaction.suffix", it must be disallowed at the same time from matching "javax.transaction.xa".

Support for pluggability of various class file transformers

From @scottmarlow

Add ability to plug different transformer implementations in, such that a no dependency transformer can be used, as well as an ASM based transformer, and potentially others. These transformers will be based on github.com/wildfly-extras/batavia which are working prototypes. The goal being that the BND transformer can be enabled, as well as the other third-party transformers.

Consolidate action resource command line options to a single pattern

From @tbitonti

The several action command line options are too busy, and should be consolidated.

Using the XML selection pattern as a guide, the several current options can be consolidated to just two:

-rulesAll
-rulesOne

-rulesAll would provide a master selection file; entries would be triples consisting of an action name, a selection expression, and a resource reference
-rulesOne would provide one selection triple, the same as is provided by individual lines in the master selection file

(The tags '-rulesAll' and '-rulesOne' are placeholders; other better tags can probably be found.)

Outer class references not transformed

@jgallimore has this one fixed and will be sending a PR, but FYI to others filing bugs that when translating an inner class, the reference to the outerclass is not transformed.

Here's an example (as noted previously we're using ASM to create text versions of the transformed classes so we can manually audit the results):

• https://github.com/dblevins/tomee-analysis/blob/ff44d86b2670b4355d51e2ff85735da7273328ca/apache-tomee-microprofile-8.0.3-SNAPSHOT.zip/apache-tomee-microprofile-8.0.3-SNAPSHOT/lib/jsp-api.jar/jakarta/servlet/jsp/el/ImplicitObjectELResolver$ScopeManager$9-asmified.java#L27

Post transform reference to getRequestScope of jakarta/servlet/jsp/el/ImplicitObjectELResolver$ScopeManager is left as javax/servlet/jsp/el/ImplicitObjectELResolver$ScopeManager

Create transforms for batch XML (and other XMLs using updated Jakarta EE 9 XSDs?)

For Jakarta Batch, I was wondering about rule(s) that would transform XML from the Java/Jakarta EE 8 XSD to the Jakarta EE 9 format.

A typical transform of a batch job XML file would require:

1. namespace substitution JCP->Jakarta
2. Update of the @version attribute on the top-level <job> element, 1.0=>2.0.

E.g., from job_batchlet_1step.xml in the TCK:

-<job id="job_batchlet_1step" xmlns="http://xmlns.jcp.org/xml/ns/javaee" version="1.0">
+<job id="job_batchlet_1step" xmlns="https://jakarta.ee/xml/ns/jakartaee" version="2.0">

(Note also if you look at the latest: https://jakarta.ee/xml/ns/jakartaee/#9 you won't see the version bump to 2.0 I mentioned, since it's still "in the pipeline".. but it seemed worth opening the issue now anyway).

One challenge here is that Batch doesn't have a well-known filename or pattern that could go in jakarta-text-master.properties like web-fragment.xml. Not sure a good way to address that.

This type of transform would probably be useful for other EE XML files with similar XSD updates. So some others would have the same NS transform but with different "version" transforms than the batch job XML example.

In addition Batch has a second XML schema, "batch.xml" which is simpler, and only requires a NS transform, not a version bump.

I think this could probably be done with "glob" matching: <job*version="1.0"*> => <job*version="2.0"*> though perhaps something more structured like regex or XPath would be better (e.g. since in XML you can use single or double quotes).

Transformer Maven Plugin

@jgallimore has written a Maven Plugin that allows you to transform the artifacts produced and re-attach them to the Maven module with a new classifier of your choosing.

Here's what the plugin configuration looks like:

      <plugin>
        <groupId>org.tomitribe.transformer</groupId>
        <artifactId>transformer-maven-plugin</artifactId>
        <version>0.0.8</version>
        <configuration>
          <classifier>jakartaee9</classifier>
        </configuration>
        <executions>
          <execution>
            <goals>
              <goal>run</goal>
            </goals>
            <phase>package</phase>
          </execution>
        </executions>
      </plugin>

This is currently living in a fork that needs to be rebased to the new repo location so we can submit a PR. The groupId org.tomitribe.transformer is temporary just so we can ensure we'll be able to have a release of it far before the TomEE release before the Jakarta EE 9 milestone release.

If others really want this badly now "asap" we can try to prioritize a PR for it. We're currently heads down.

Move to a maven build?

The current build is a gradle build inherited from my original prototype.

I wonder if it might be better, in the long run, to migrate to a maven build.

Does anyone have any opinions? :-)

OSGi Declarative Services Component XML JavaEE services are not being transformed.

When marking a component for JakartaEE transformation, it appears JavaEE services in the component XML files are not being transformed.

See the com.ibm.ws.cache.CacheServiceImpl.xml file in the com.ibm.ws.dynacache Open Liberty component as an example.

Create brew formula for transformer cli

Once we get a release in maven central, we should create a brew formula for the transformer cli so that it can be installed as a command on macOS.

Extend XML file selection pattern to other rules applications.

From @tbitonti

The XML action uses a more sophisticated selection mechanism:
Instead of a single set of updates, as is done by other actions, the XML action uses a single master selection file which maps selection expressions to update property files:
( selection pattern -> ( initial value -> final value ) )
This pattern should be applied to other actions.

SwitchMap and SWITCH_TABLE references should be updated

This is an optional enhancement that doesn't materially affect runtime execution, but still could be nice.

When you refer to an enum in a switch/case the JVM actually compiles that into a int-based switch/case indirectly using the ordinals of the enum values as the ints. The ordinal references are not hardcoded into the compiled class so that if the enum was changed in the future old switch cases referring to still-valid enum names would still work.

To achieve this the VM generates an inner-class for you and in that inner-class is a field that holds a int-to-int mapping which is populated at runtime (effectively, old ordinals to new ordinals). The name of this field will look like this:

• $SwitchMap$javax$faces$component$visit$VisitResult

In this case, the enum being referenced is javax.faces.component.visit.VisitResult. Here's an example of how this looks via the ASMifier:

• https://github.com/dblevins/tomee-analysis/blob/ff44d86b2670b4355d51e2ff85735da7273328ca/apache-tomee-microprofile-8.0.3-SNAPSHOT.zip/apache-tomee-microprofile-8.0.3-SNAPSHOT/lib/myfaces-api-2.3.6.jar/jakarta/faces/component/UIData$2-asmified.java#L32

Though this field has what appears to be a reference to a renamed class the code will still run fine as in truth the generated field name could be anything.

That said, any introspection tools, including decompilers may break. These tools are aware of the naming format and use it to imply the source code.

It's not critical, but we should eventually fix this as time permits.

Updates to README and other top level documentation

This is a refresh of README.md and other top level documentation.

The initial PR is a draft, and will likely be adjusted before being merged.

String Transformation Fails in Certain Scenarios

The following code snippet is not changed.
Line 350: this.writer.print("(javax.servlet.jsp.tagext.JspFragment ");
GenerateTagFileVisitor.class.zip

I've tried the following transformation rules

Renames:
javax.servlet.jsp.tagext=jakarta.servlet.jsp.tagext

Direct
javax.servlet.jsp.tagext=jakarta.servlet.jsp.tagext

In neither case, does not transform that particular line. But the rule is correct because I get a transformation in other parts of the code:
Line 216:
this.writer.print("public class " + this.tagFileFiles.getClassName() + " extends jakarta.servlet.jsp.tagext.SimpleTagSupport");
Line 329:
this.writer.print("jakarta.servlet.jsp.tagext.JspFragment ");

A similar situation occurs with this line too:

writer.print("(javax.servlet.jsp.tagext.SimpleTag) this ));");

BaseTagGenerator.class.zip

Refactor state information used by the transformer and by actions

The current action framework is stateless, which is desirable for reusing the same action objects in the action hierarchy, but is a problem for managing state when invoking actions.

This update is to introduce a single "TransformerState" object which will encapsulate the current transformer state and which will be provided to actions as a way of associating state more cleanly.

TransformerState.java.txt

Tolerate duplication on container actions

From @tbitonti

Would be helpful to have the tool tolerate duplicate entries when processing archive entries.

Currently, an exception is thrown when a duplicate is encountered.

Perhaps better: The tool should ignore second (and subsequent) duplicates when processing archives.

This will mean adding a set of visited entries to container actions, which will need to be pushed/popped when change data is pushed/popped.

Add JSP Transformation step

Need a rule to transform JSP files, which contain package references. For example, javax.el should be replaced with jakarta.el in the following JSP text:

<%@ page import="com.sun.ts.tests.jsp.common.util.JspTestUtil,
             javax.el.MethodExpression,
             javax.el.MethodInfo,
             javax.el.ELContext"%>

Extend transformer public API to expose the change information

When calling into the Transformer, knowing the set of applied changes that occurred is of interest. A Changes getLastActiveChanges() method could be added.

Duplicate file property in jakarta-xml-master.properties causes java.io.FileNotFoundException

I saw a few issues relating to XML transformations, but I didn't did see anything specific as this situation:

My rules within jakarta-xml-master.properties:

com.ibm.ws.wsock.initializer.xml=jakarta-xml-servletContainerInitializer.properties
com.ibm.ws.webcontainer.xml=jakarta-xml-servletContainerInitializer.properties

Here jakarta-xml-servletContainerInitializer.properties is used twice. The file does exist, and the error is not seen when the second rule is comment out and only the first is used.

Tested in Open Liberty jakartaee-transformer version 0.0.9.

Here's the output:

[main] INFO Transformer - External [ Substitions matching [ web-fragment.xml ] ] URL [ file:/Users/siedlecki/libertyGit/open-liberty/dev/wlp-jakartaee-transform/rules/jakarta-xml-dd.properties ]
[main] INFO Transformer - Adjusted substition reference from [ jakarta-xml-servletContainerInitializer.properties  ] to [ /Users/siedlecki/libertyGit/open-liberty/dev/wlp-jakartaee-transform/rules/jakarta-xml-servletContainerInitializer.properties  ]
[main] INFO Transformer - External [ Substitions matching [ com.ibm.ws.webcontainer.xml ] ] URL [ file:/Users/siedlecki/libertyGit/open-liberty/dev/wlp-jakartaee-transform/rules/jakarta-xml-servletContainerInitializer.properties%20 ]
[main] ERROR Transformer - Exception loading rules:
java.io.FileNotFoundException: /Users/siedlecki/libertyGit/open-liberty/dev/wlp-jakartaee-transform/rules/jakarta-xml-servletContainerInitializer.properties  (No such file or directory)
        at java.io.FileInputStream.open(FileInputStream.java:195)
        at java.io.FileInputStream.<init>(FileInputStream.java:138)
        at java.io.FileInputStream.<init>(FileInputStream.java:93)
        at sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90)
        at sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:188)
        at java.net.URL.openStream(URL.java:1057)
        at org.eclipse.transformer.util.FileUtils.loadProperties(FileUtils.java:214)
        at org.eclipse.transformer.Transformer.loadExternalProperties(Transformer.java:708)
        at org.eclipse.transformer.Transformer$TransformOptions.setRules(Transformer.java:934)
        at org.eclipse.transformer.Transformer.run(Transformer.java:1425)
        at com.ibm.ws.jakarta.transformer.JakartaTransformer.main(JakartaTransformer.java:36)

Need transform rules for Jakarta Authentication packages

From @scottmarlow

We hit this in the migration of the Jakarta EE Platform TCK issue 244, in that we had changed "import javax.security.auth.;" to "import jakarta.security.auth.;", however, we really needed the JDK classes (not the Jakarta EE 9).

Perhaps the transformer should generate an extra import jakarta.security.auth.* classes, so that both jakarta.security.auth.* + javax.security.auth.* classes are imported.

Similar for importing javax.transaction., which could be translated to include both the JDK javax.transaction. classes, as well as jakarta.transaction.*.

Support updating bean validation constraint mappings.

From @nmittles

Bean Validation can have XML configuration files that reference javax.validation.constraints.* constraint classes. These XML files are references from the main bean validation deployment descriptor named validation.xml.

For examples, more details, and info on testing see the Open Liberty issue:
OpenLiberty/open-liberty#12431

Separate Jakarta specific rules and code into its own artifact

Currently we have the Jakarata rules and code intermixed in the general purpose artifacts.

We should create a new artifact to hold Jakarta specific items.

The main transformer artifact would define a ServiceLoader service name (e.g. DefaultRules) which the jakarta artifact will implement.

Then by placing the jakarta artifact on the class path (of the cli or maven plugin), the transformer implementation can discover any default rules via ServiceLoader. This "plugin" model allows different rulesets to be made default by inclusion of the ServiceLoader implementation on the classpath.

Setup up actions build to publish artifacts to a nexus repo

Our actions build needs to publish snapshot and release builds.

More cleanly define core java API; refactor core command line API and core java API into distinct components

This issue is to create a more well defined core java API and to cleanly separate the core command line API from the core java API.

Current project structure has a tightly coupled and thick core command line interface which provides a front-end to the (theoretical) core java API. While the java API can be invoked as a public API, the API is too heavy to be a proper API.

Having a more well defined core java API, the core command line API should be placed in a separate component. However, common utilities, for example, for reading rules data from properties files, should be retained as a part of the core java API for use by the command line and other APIs, for example, for use by maven or gradle plugins.

Transformer RC, Alpha, or Beta ready for Jakarta EE 9 Milestone in June?

The Jakarta EE 9 Milestone "release" is planned for June 2020:

• June 12, Development Complete
• June 19, Testing Complete
• June 23, "announcement"

It would be great if we could include a Transformer deliverable in our blogs, presentations, and "announcement" literature. Some type of pre-built download (milestone, RC, Alpha, whatever) available via Maven would be excellent. Also, having some type of simple instructions or README to explain how to use the Transformer with applications that might want to try out the Jakarta EE 9 Milestone deliverable would be helpful.

Thanks!

How to handle pruned technologies?

From @scottmarlow

What is the requirement around handling application bytecode that contains a pruned technology (e.g. javax.xml.rpc or javax.management.j2ee)?

If transformers ignores pruned classes being referenced, the application will likely not work on an EE 9 implementation (likely failing to deploy with a ClassNotFoundException.)

If transformers detect pruned classes being referenced and produce error or failures, then at least users will know that they need to remove the pruned classes from their application first.

In terms of application bytecode modification strategies, if anyone has one, please add a comment to this issue. For example, replacing javax.xml.rpc references to a different class that is added to the application that throws a RuntimeException, if the code that depends on javax.xml.rpc is reached. I'm not sure if this is possible or would help enough, but am just mentioning as a possible (not really thought out) idea.

Incorrect sub-package transformation

From @jfdenise

The package rename rule contains javax.transaction=jakarta.transaction

Constant Pool extract:
#1 UTF8 javax/transaction/xa/XAResource
#2 Class #1

This constant shouldn't be transformed. Although it is. The culprit call stack:
https://github.com/eclipse/transformer/blob/master/transformer/src/main/java/org/eclipse/transformer/action/impl/SignatureRuleImpl.java#L530

https://github.com/eclipse/transformer//blob/master/transformer/src/main/java/org/eclipse/transformer/action/impl/SignatureRuleImpl.java#L282

https://github.com/eclipse/transformer//blob/master/transformer/src/main/java/org/eclipse/transformer/action/impl/SignatureRuleImpl.java#L347

I have attached a verbose output

output.txt

Reconsider using `java.nio.ByteBuffer` instead of `byte[]` or `InputBuffer`

ByteBuffer is a well known type providing a view over a byte array.

Output rules for Export/Import package

From @tjwatson

With OSGi there is a concern about package versions when an exported package uses one of the renamed packages. For example, consider the following export from a Bundle X:

Export-Package: some.api; uses:="javax.servlet"; version="1.0"

Here javax.servlet is renamed to jakarata.servlet and the package some.api has types from javax.servlet in its API signature.

When transforming bundle X the transformer must transform the Export-Package for some.api to:

Export-Package: some.api; uses:="jakarta.servlet"; version="2.0"

Here the uses directive needed to be updated, but also the major version of the package should be updated because some.api had a breaking change with the rename of javax.servlet->jakarta.servlet.

We need to ensure this version transformation occurs when the transformer detects that a used packages (as specified by the uses:= directive). In addition we also need a way to output a future rule that this package some.api had a major version change due to a transformation. This information then can be used when transforming a later bundle that imports the some.api package:

Import-Package: some.api; version="[1.0, 2.0)"

If a bundle being transformed has the above import the transformed import should use a range that included the major version bump included with the transformed export:

Import-Package: some.api; version="[2.0, 3.0)"

The assumption is that the importing bundles will be transformed after the exporting bundles and the output of the transformer on the exporter can be used to feed the new rule for range transformation to the later transformation of the importing bundles.

Break CLI out into its own project

We need to break the cli driver out into its own artifact so that the tranforming code is a separate artifact.

This will be important as we move to add maven and gradle drivers for transformer.

Supply build version externally, and automatically generate the build date

Redesign the processing of build properties to have the version be externally specified, and to have the build date generated by the build scripts.

Handle ZipSlip vulnerability when processing ZipEntry names in a zip file

See https://github.com/bndtools/bnd/blob/ea6b9b1d3e097a443d707a6623d8542044bf4c94/aQute.libg/src/aQute/lib/zip/ZipUtil.java#L29-L76 for a method to clean a zip entry name which fails upon a malformed zip entry name.

Regular expression text substitution for javadoc transformation

A case of using regular expressions to transform javadoc.

This will likely be folded into one of the other regular expression cases; I'm posting it as a distinct issue for now.

---

Regular expressions for javadoc transformation:

---

Using transaction.2.0_1.1-javadoc.zip as an example, two types of references are seen.

The first is a dotted qualified package name, for example:

overview-tree.html:

javax.transaction.TransactionManager

javax.transaction.xa.XAResource

The second is a slash qualified package name used within a hyperlink, for example:

com.ibm.tx.jta.TransactionInflowManager

---

Javadoc text files which may contain references are:

*.html
*.js
*.css
package-list

---

The requirements for the transformation of the transaction javadoc, are, most generally:

"javax.transaction" is to be replaced with "jakarta.transaction", except that "javax.transaction.xa" is not to be modified. This includes alternate forms, for example, hyperlinks, which use slash ('/') delimiters instead of dots ('.').

More specifically, the match must be aligned to the beginning of a qualified package reference, and must not be a part of a longer package name.

For example:

Candidate Text Match?

javax.transaction Y
javax.transaction.TransactionManager Y
javax.transaction.transactional Y
javax.transaction.xa N

ajavax.transaction N
a.javax.transaction N
javax.transactiona N
javax.transaction.a Y

---

Based on these examples, the following regular expressions are proposed:

Regular Expression Replacement Text

(?<![_a-zA-Z0-9])javax.transaction(?!([_a-zA-Z0-9]|.xa)) jakarta.transaction
(?<![_a-zA-Z0-9])javax/transaction(?!([_a-zA-Z0-9]|/xa)) jakarta/transaction

This uses the regular expression syntax of the java "Pattern" class, see:

https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html#quote-java.lang.String-

Of particular note are look-ahead and look-behind special constructs:

Negative look-ahead: (?!X)
Negative look-behind: (?<!X)

And uses these specifications for java identifiers:

The only allowed characters for java identifiers are:
all alphanumeric characters ([A-Z],[a-z],[0-9])
dollar sign ('$')
underscore ('_')

Java regular expression for identifiers:
[_a-zA-Z][_a-zA-Z0-9]*

Java first identifier character regular expression:
[_a-zA-Z]

Java second and following identifier character regular expression:
[_a-zA-Z0-9]

Properties files are not transformed

From @jfdenise

When trying to transform servlet API ( https://github.com/jboss/jboss-servlet-api_spec
) to jakarta 9, the properties files, such as: https://github.com/jboss/jboss-servlet-api_spec/blob/master/src/main/java/javax/servlet/LocalStrings.properties are not transformed. This makes runtime resolution of these resources to fail.

Reconsider using `Optional` instead of using `null`

Returning an empty Optional is no more expensive than return null.

Using Optional can avoid numerous null checks at call sites (which can be forgotten as new code is written).

Modern JVMs are very efficient at managing short-lived objects, so using an Optional to wrap a result is not excessively costly. Of course, actual performance data to prove using null is better here can change the story, but until we have such concrete performance data, we are better off using better programming practices and not using null for control flow.

Command line rules data additions

From @tbitonti

Open-liberty, when specifying bundle identity updates, would prefer to specify the identity information at the project level and through command line arguments, instead of through a global rules file. For example:

JavaEE-Bundle-Symbolic-Name: ...
JakartaEE-Bundle-Symbolic-Name: ...
JakartaEE-Bundle-Name: ...
JakartaEE-Bundle-Version: ...
JakartaEE-Bundle-Description: ...

These would be read and processed by BND tooling in the open-liberty build and would be provided to the transformer using a new command line option.

Possibly:

--rule-data ruleName ruleDataLine

"ruleDataLine" would add to the data from the rules properties file matching the specified "ruleName".

module-info transformation

There are a small number of uses of module-info.class in some of the implementations in the Jakarta ecosystem. These references do not appear to be completely updated.

For example, the jaxb-api 2.3.2 jar has this module-info.java:

• https://github.com/eclipse-ee4j/jaxb-api/blob/2.3.2/jaxb-api/src/main/java/module-info.java

module java.xml.bind {
    requires transitive jakarta.activation;
    requires transitive java.xml;
    requires java.logging;
    requires java.desktop;

    exports javax.xml.bind;
    exports javax.xml.bind.annotation;
    exports javax.xml.bind.annotation.adapters;
    exports javax.xml.bind.attachment;
    exports javax.xml.bind.helpers;
    exports javax.xml.bind.util;

    uses javax.xml.bind.JAXBContextFactory;
}

Post transformation all those class and package references are still untransformed (output from ASMifier).

• https://github.com/dblevins/tomee-analysis/blob/ff44d86b2670b4355d51e2ff85735da7273328ca/apache-tomee-microprofile-8.0.3-SNAPSHOT.zip/apache-tomee-microprofile-8.0.3-SNAPSHOT/lib/jakarta.xml.bind-api-2.3.2.jar/module-info-asmified.java#L26

Publish artifacts to Maven Central?

From @jgallimore

One of the things I was attempted to achieve in this PR: #102 was get artifacts pushed to Maven. This certainly seems to work for pushing to a local repository on my development machine.

It could be useful to push some SNAPSHOT builds to Maven so that folks can start using this without needing to build from source (which would prove useful for some work I'm doing with the Maven plugin my PR adds).

Direct string replacement per class

Today direct String replacement is common to all transformed classes. We have a need to only transformer String constant for certain classes.
A master index similar to master index would contain the class to mapping properties file.

It would look like:
org/jboss/as/jaxrs/JaxrsAnnotations$Constants.class=jaxrs-annotations.properties
org/apache/jasper/compiler/Generator.class=jsp-compiler.properties
org/jboss/as/weld/CdiAnnotations$Constants.class=cdi-annotations.properties

Add TLD transformation actions

From @jfdenise

TLD can contain Validation classes that would need to be transformed.