gradle / api-evolution-demo

Demo of API evolution tricks

Gradle API evolution demo

This is a playground repo to test out ideas around allowing code compiled against an older version of Gradle to run with a newer version of Gradle that have breaking changes.

Structure

There are the following subprojects:

• :old-api defines an old version of the Server type written in Java,
• :old-client contains client code compiled against the old version of Server; there are clients written in Java, Kotlin and static and dynamic Groovy,
• :old-app represents the old version of the application, with a test to run :old-client against :old-api.
• :new-api defines the new version of the Server type, also written in Java,
• :new-client is to demonstrate how the code manually written against the new API would look like,
• :new-app represents the new version of the application, with a test that tries to run each of the clients in :old-client against the :new-api. It uses :upgrader to upgrade the old classes, by defining the actual upgrade steps to execute against the :old-api.
• :upgrader is the actual abstract upgrade logic.

Try with:

$ ./gradlew check

Property upgrade bytecode

Java/Kotlin/static Groovy

diff --git a/javaold b/javanew
index 02f1620..0c43e4c 100644
--- a/javaold
+++ b/javanew
@@ -11,15 +11,16 @@ Label label1 = new Label();
 methodVisitor.visitLabel(label1);
 methodVisitor.visitLineNumber(6, label1);
 methodVisitor.visitVarInsn(ALOAD, 0);
+methodVisitor.visitMethodInsn(INVOKEVIRTUAL, "org/gradle/demo/api/evolution/Server", "getName", "()Lorg/gradle/demo/api/evolution/Property;", false);
 methodVisitor.visitLdcInsn("lajos");
-methodVisitor.visitMethodInsn(INVOKEVIRTUAL, "org/gradle/demo/api/evolution/Server", "setName", "(Ljava/lang/String;)V", false);
+methodVisitor.visitMethodInsn(INVOKEVIRTUAL, "org/gradle/demo/api/evolution/Property", "set", "(Ljava/lang/Object;)V", false);
 Label label2 = new Label();
 methodVisitor.visitLabel(label2);
 methodVisitor.visitLineNumber(7, label2);
 methodVisitor.visitFieldInsn(GETSTATIC, "java/lang/System", "out", "Ljava/io/PrintStream;");
 methodVisitor.visitVarInsn(ALOAD, 0);
-methodVisitor.visitMethodInsn(INVOKEVIRTUAL, "org/gradle/demo/api/evolution/Server", "getName", "()Ljava/lang/String;", false);
-methodVisitor.visitMethodInsn(INVOKEVIRTUAL, "java/io/PrintStream", "println", "(Ljava/lang/String;)V", false);
+methodVisitor.visitMethodInsn(INVOKEVIRTUAL, "org/gradle/demo/api/evolution/Server", "getName", "()Lorg/gradle/demo/api/evolution/Property;", false);
+methodVisitor.visitMethodInsn(INVOKEVIRTUAL, "java/io/PrintStream", "println", "(Ljava/lang/Object;)V", false);
 Label label3 = new Label();
 methodVisitor.visitLabel(label3);
 methodVisitor.visitLineNumber(8, label3);

Dynamic Groovy

• Full filtered diff: https://gist.github.com/lptr/cd7c9e149e2db5d2c92df2cb6d709728
• Full unfiltered diff: https://gist.github.com/lptr/a7a8557be684978a553ae3d56ae73006

Old code

    private static void doSet(Object server) {
        server.setTestProperty("lajos")
    }
    private static Object doGet(Object server) {
        return server.getTestProperty()
    }

New code

    private static void doSet(Object server) {
        server.getTestProperty().set("lajos")
    }
    private static Object doGet(Object server) {
        return server.getTestProperty().get()
    }

Get

   private static java.lang.Object doGet(java.lang.Object);
     descriptor: (Ljava/lang/Object;)Ljava/lang/Object;
     flags: (0x000a) ACC_PRIVATE, ACC_STATIC
     Code:
-      stack=2, locals=2, args_size=1
+      stack=3, locals=2, args_size=1
         : nop
         : invokestatic  #_                 // Method $getCallSiteArray:()[Lorg/codehaus/groovy/runtime/callsite/CallSite;
         : astore_1
         : aload_1
-        : ldc           #_                 // int 5
+        : ldc           #_                 // int 6
+        : aaload
+        : aload_1
+        : ldc           #_                 // int 7
         : aaload
         : aload_0
         : invokeinterface #_,  2           // InterfaceMethod org/codehaus/groovy/runtime/callsite/CallSite.call:(Ljava/lang/Object;)Ljava/lang/Object;
+        : invokeinterface #_,  2           // InterfaceMethod org/codehaus/groovy/runtime/callsite/CallSite.call:(Ljava/lang/Object;)Ljava/lang/Object;
         : areturn
       LineNumberTable:
         line 14: 5
       LocalVariableTable:
         Start  Length  Slot  Name   Signature
-            0      16     0 server   Ljava/lang/Object;
+            0      25     0 server   Ljava/lang/Object;

Set

   private static void doSet(java.lang.Object);
     descriptor: (Ljava/lang/Object;)V
@@ -246,7 +280,11 @@
         : aload_1
         : ldc           #_                 // int 4
         : aaload
+        : aload_1
+        : ldc           #_                 // int 5
+        : aaload
         : aload_0
+        : invokeinterface #_,  2           // InterfaceMethod org/codehaus/groovy/runtime/callsite/CallSite.call:(Ljava/lang/Object;)Ljava/lang/Object;
         : ldc           #_                 // String lajos
         : invokeinterface #_,  3           // InterfaceMethod org/codehaus/groovy/runtime/callsite/CallSite.call:(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;
         : pop
@@ -255,27 +293,31 @@
         line 11: 5
       LocalVariableTable:
         Start  Length  Slot  Name   Signature
-            0      18     0 server   Ljava/lang/Object;
+            0      27     0 server   Ljava/lang/Object;

Our approach

Upgrading statically compiled bytecode from old API to new is relatively easy, at least when we need to replace single method calls. We can do this in two, fairly similar ways:

1. remove the original bytecode calling the old API, and generate new bytecode that calls the new API, so the resulting code looks exactly as if the original code was rewritten and recompiled against the new API,
2. remove the original bytecode, and replace it with a call to some compatibility class in Gradle that will call through to the new API.

For dynamic Groovy is harder, but not impossible. The problem is that in the bytecode we have no type information that would allow us to figure out which INVOKEDYNAMIC instruction corresponds to what actual API call.

During runtime, before executing the code of every dynamic method, Groovy generates an array of CallSites. The dynamic calls go through these call sites, and when they do, the necessary type information is available. The call sites are created by a generated static method called $getCallSiteArray().

We are borrowing ideas from Gradle's InstrumentingTransformer that solve our problem in two steps:

1. via bytecode transformation of the client code we decorate the call to $getCallSiteArray() at the beginning of every dynamic method, and process the generated CallSite objects via a static method. We basically wrap the code like this:

   /* ... */ = Instrumented.processCallSites($getCallSiteArray());

2. in the Instrumented.processCallSites() method we wrap each CallSite with a wrapper that can detect calls to old APIs, and instead execute calls to the new methods.

In the case of the doSet() method, Instrumented would be aware that Server.setTestProperty() needs to be substituted with getTestProperty().set(). So the CallSite wrapper would do something like this:

        @Override
        public Object call(Object receiver, Object arg) throws Throwable {
            if (receiver instanceof Server && getName().equals("setTestProperty()")) {
                return ((Server) receiver).getTestProperty().set((String) arg);
            } else {
                return super.call(receiver, arg);
            }
        }

api-evolution-demo's Issues

Method replacement: handle replacing super calls ('INVOKESPECIAL')

Consider upgrading this:

class Client extends Server {
    @Override
    String getTestProperty() {
        return super.getTestProperty() + "-overridden"
    }

    @Override
    void setTestProperty(String value) {
        super.setTestProperty("overridden-" + value)
    }
}

When replacing a call like super.setProperty(), the JVM bytecode instruction used is INVOKESPECIAL instead of INVOKEVIRTUAL. Let's handle this use case and add some tests.

Replace type

Like move types from one package to another for example.

Making thrown exceptions more general is another example.

Refactor testing the bytecode upgrades

Currently testing works by executing *Client.main(). This was good for testing the basic functionality, but now we would need more fine-grained test logic. Figure out a way to have some nice syntax over upgraded classes.

Method replacement: support replacing `INVOKEDYNAMIC` calls

Groovy 4 is going indy only: https://groovy-lang.org/releasenotes/groovy-4.0.html#Groovy4.0-indy-only

Method replacement: add typed method replacers

Currently when replacing a method there is very little type information exposed at registration. We could make it harder to write invalid code by adding replacement builders that work with type information about arguments and the return type.

Execute only necessary upgrades based on client version

Bind upgrades to a client version and execute only the ones that are required for a particular version of the plugin. Make sure we don't incur overhead when an upgrade is not necessary.

Method replacement: handle method references

When passing method references to replaced methods, let's make sure those get converted to their upgraded counterparts.

Replace annotations

This is yet another feature we'd like to demonstrate.

Fix `double` parameter handling for dynamic Groovy

Dynamic Groovy converts double parameters to BigDecimal internally. This causes the following exception to be thrown when this code is uncommented:

api-evolution-demo/old-client/src/main/groovy/org/gradle/demo/api/evolution/DynamicGroovyClient.groovy

Lines 10 to 12 in 2337013

	// TODO Fix this, apparently Groovy sends double values over as BigDecimal
	// doSetDouble(server)
	// println doGetDouble(server)

java.lang.ClassCastException: java.math.BigDecimal cannot be cast to java.lang.Double
	at org.gradle.demo.api.evolution.ApiUpgradeManager.lambda$addSetterReplacement$2(ApiUpgradeManager.java:110)
	at org.gradle.demo.api.evolution.MethodReplacement$1.call(MethodReplacement.java:125)
	at org.gradle.demo.api.evolution.DynamicGroovyClient.doSetDouble(DynamicGroovyClient.groovy:40)
	at org.gradle.demo.api.evolution.DynamicGroovyClient.main(DynamicGroovyClient.groovy:11)
	at org.gradle.demo.api.evolution.PropertyUpgradeTest.dynamic Groovy client works(PropertyUpgradeTest.groovy:37)

Method replacement: handle `INVOKEINTERFACE`

This is probably very similar to #5.

Allow decorating `System.getProperties()`

This is a special replacement already implemented by gradle/gradle in InstrumentingTransformer. We need to have this functionality working if we want to use this tool in Gradle itself.