Track JVM version more precisely about gradle

[lptr]

Another option is to track "vendor + major + minor" and allow the user to switch it off. They can re-add any parts of the version number as an input to the task if they want to track that.

[bmuskalla]

We now take the following into account as inputs:
If no toolchain is configured:

• Java language version (major version)

If toolchains are configured:

• Java language version (major version)
• Vendor, if specified as toolchain requirement (starting with 6.8)
• JVM implementation (e.g. J9), if specified as toolchain requirement (starting with 6.8)

As an example, using the following setup

java {
    toolchain {
        languageVersion = JavaLanguageVersion.of(11)
        vendor = JvmVendorSpec.ADOPTOPENJDK
    }
}

Compile and test tasks (and any custom tasks using toolchains) take 11 and the vendor into account. Wit this, we're closing this issue for now until there is a good reason to take more nuanced version information into account that is not possible to workaround as described above.

[wolfs]

We should at least track the vendor.

[lptr]

Just think of IBM JDK 6!

[lptr]

This is important for Java compilation, and also for running tests.

[wolfs]

We might be doing something by tracking the classpath.

[big-guy]

need to confirm that rt.jar is hashed

[lptr]

We have everything we might need in rt.jar:

Manifest-Version: 1.0
Implementation-Vendor: Oracle Corporation
Implementation-Title: Java Runtime Environment
Implementation-Version: 1.8.0_101
Specification-Vendor: Oracle Corporation
Created-By: 1.7.0_07 (Oracle Corporation)
Specification-Title: Java Platform API Specification
Specification-Version: 1.8

[wolfs]

Note that we use should use the same strategy for the test task: https://github.com/gradle/build-cache/issues/145.

[wolfs]

We also need to keep in mind what to do for forking Java compilation. Currently, we just pretend that we do not fork - i.e. we use the Java version of the Gradle JVM as input. This is just plain wrong.
As a workaround in the Gradle build we ignore the executable in the fork options and override the version of the tool chain.
We also need to consider if we want to have fork options as an input at all - given that we should have the same compilation result no matter if we are forking or not.

[lptr]

☝️ We now have proof that even different Oracle Java update versions (7u45 vs 7u80) produce different outputs. This is a strong reason to track the full version number along with the vendor.

[lptr]

We think it's manageable for 3.3 to live with just tracking <major>.<minor> (e.g. 1.8). We can track more in 4.0 if we decide that's what we want.

[wolfs]

Currently, we track the Java version of the JVM Gradle runs with (JavaVersion.current()). We need to track the Java version of the forked JVM (forkOptions.executable).

[wolfs]

We are now using the JvmVersionDetector to obtain the version of the JVM used for compilation and testing. This currently only includes the version itself. In order to obtain the vendor, we would need to use features currently only present in JavaInstallationProbe.

[lptr]

@vydra it seems you needed this at some point. Can you explain the use case please?

[vydra]

@lptr I just wanted to make sure that we use exactly the same version for cache key. Many customers re-write bytecodes and different javac versions may produce different results.

[big-guy]

We don't. We use the major version only, so we treat any minor/patch version of the compiler the same. We also treat all vendor (IBM JDK, Oracle JDK and OpenJDK, etc) compilers as the same.

[lptr]

To be clear: we are planning to change that, but probably not in 4.0.

[lptr]

@wolfs you said that you are not completely convinced about tracking the full version number (https://github.com/gradle/task-output-cache/issues/645#issuecomment-302373098). Could you explain your reasoning?

[wolfs]

We have various aspects here: We track the version number of Java for compilation and for runtime. We also want to support pulling results for developers and having "perfect" results on CI.

For compilation we saw that sometimes differences in the produced bytecode for different minor version numbers. I am pretty confident that the bytecode is still the same with respect to CompileClasspath snapshotting. So I guess only the major version matters.
For test execution, runtime classpath snapshotting still would detect differences between the bytecode. But once again I believe that the differences should treated as the same - we could improve our runtime classpath snapshotting for this case.

Treating the minor number always as an input would it make very hard for developers to pull from the cache. We saw that even in bigger companies different developers on the same team were using different Java versions. We could force them have to use the same Java version but I am not sure that this is what we want to do here.

I could even imagine that developers could somewhat say that they do not care about the minor version number when pulling from the cache while CI would only pull results with the same minor number for the cache. Our current cache interface would not support this currently, but I wouldn't discard this as an option because of this.

Having said all this it is somewhat difficult to come up with a default strategy for tracking the Java version number which is the same in all the above situations. There are different compromises which we need to weigh against each other.

[lptr]

Thanks, this all makes sense.

I guess there's the option to allow configuring whether you want the update version tracked. That would apply to CI and developers alike, which I'm not sure is a bad idea.

In general I'm against treating CI builds different to local builds. If you take the long view, with distributed builds there will only be builds happening somewhere. Provisioning the Java version used is also not impossible. There are lots of options, we probably should support multiple of these.

FTR, one scenario where update version matters a lot is if there's an important fix in the JDK. Since we don't track the update version, upgrading the JDK on CI would not force anyone to recompile, as they could still pull the previously compiled class files from cache.

[vydra]

I think that we need to take a stand and have the default mode track the full Java version exactly while at the same time encouraging customers to use Gradle to provision the JDK. In general, we should sell folks on the idea that Gradle is in charge of provisioning the tooling. We can offer a 'loose' mode for backward compatibility, but more importantly, we can inform the developer that they are not getting the benefit of the cache because they are out of compliance with CI setup. I think the real breakthrough will come when we assume full responsibility for the context of the integration test execution. For example, we can control the docker container that is running the DB, or any other required service and since the docker image used is immutable and we can make it part of the cache key. To make a profit, most of this work needs to be on the GE side, I presume.

…

On Thu, May 18, 2017 at 6:16 AM, Lóránt Pintér ***@***.***> wrote: Thanks, this all makes sense. I guess there's the option to make tracking the update version tracked via configuration, too. That would apply to CI and developers alike, which I'm not sure is a bad idea. In general I'm against treating CI builds different to local builds. If you take the long view, with distributed builds there will only be builds happening *somewhere.* Provisioning the Java version used is also not impossible. There are lots of options, we probably should support multiple of these. FTR, one scenario where update version matters a lot is if there's an important fix in the JDK. Since we don't track the update version, upgrading the JDK on CI would not force anyone to recompile, as they could still pull the previously compiled class files from cache. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <https://github.com/gradle/task-output-cache/issues/211#issuecomment-302400402>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEy5PsbPr80H9oXqOF-_j1IAlU55A9Kks5r7ES6gaJpZM4KdRxt> .

-- [image: [email protected]] David Vydra Senior Solutions Architect Gradle Inc. P. [phone] W. gradle.com <https://www.gradle.com/> Gradle Summit is now open Click HERE to register <https://summit.gradle.com/>

[lptr]

In general I agree with @vydra's long-term vision. Though we won't ever be able to fully require everyone to use a provisioned container or something. One of the major advantages of Gradle is its graceful degradation: instead of prescribing how things should work, we give you a list of requirements that if you can fulfill, then we give you excellent service. If you can't fulfill some of them, we still give you pretty good service, though you might need to sacrifice some features or performance.

For the short term Gradle offers squat right now wrt provisioning, so we can't rely on that anyway. We need to figure out how to make this work. We can require external provisioning for example. There's lots of options. The most important would be to let users know if they are using the wrong Java version for whatever reason, whether or not we'd be tracking the full version number.

[wolfs]

As soon as we track the vendor we can remove the custom inputs added here: 06c5da7

[lptr]

Here's a comparison between some files compiled with Oracle Java compiler 7u45 vs 7u80:

https://github.com/lptr/javac-diff-7u45-vs-7u80/compare/u80

Several of the files end up being different byte-by-byte, mostly because of their constant tables are organized somewhat differently. I didn't add a javap file for each difference.

[lptr]

Talked to @adammurdoch about this. He thinks we should offer a way to configure Java version normalization. I would go with these options:

• vendor + major + minor (default)
• vendor + major
• major
• ignore completely

[DPUkyle]

Thanks for sharing; @big-guy and I discussed this in some detail in a private forum. I'll be watching to see how this evolves!

[pioterj]

For the record, the workaround for tracking the vendor is described at https://guides.gradle.org/using-build-cache/#java_version_tracking.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. Given the limited bandwidth of the team, it will be automatically closed if no further activity occurs. If you're interested in how we try to keep the backlog in a healthy state, please read our blog post on how we refine our backlog. If you feel this is something you could contribute, please have a look at our Contributor Guide. Thank you for your contribution.