spring-projects / spring-data-release

Command-line application to ship Spring Data releases

General Notes

• Make sure to initially set-up your environment.

• Use the command help to get a list of all commands in the release tools.

• After fixing a problem use workspace cleanup to clean-up any mess left behind by the previous step.

The release process

The Pre-Release tasks and Post-Release tasks phases of making a release (opening tickets/closing tickets) is still done manually.

The Build and publish the release phase is what’s covered by the CI server.

🍃 Pre-Release tasks

1. Ensure all work on CVEs potentially contained in the release is done (incl. backports etc.).

2. Upgrade dependencies in Spring Data Build parent pom (mind minor/major version rules).

3. Review open tickets for release (CLI: $ tracker open-tickets $trainIteration).

4. Announce release preparations to mailing list (https://groups.google.com/forum/#!forum/spring-data-dev)

5. Create release tag, verify the release can be built, and publish tags (CLI: $ prepare-it $trainIteration)

6. 🚥 Continue with Build and distribute the release

🌿 Build and distribute the release

Action	Command
Open a terminal pointing to the release tools directory
Checkout the release branch	$ git checkout release
Set the HEAD to main to point the content to what is on main	$ git reset --hard main
Edit ci/release.properties and assign the version number of the release train you wish to release to release.version.
Commit the change to this repository	$ git add ci/release.properties && git commit
🚨 To trigger 🛣 the release git push the release branch	$ git push -f origin release
Go to https://jenkins.spring.io/view/SpringData/job/spring-data-release/job/release, Navigate to the release branch. A job should have started. Click on the active job, and then click on Open Blue Ocean.	Monitor the release process.

• For a Maven central release, if the smoke test has passed, the repository will already have been released, so there is nothing more to do.

• For an Artifactory release, if the smoke test has passed, the release will already have been staged and promoted, so there is nothing more to do.

• 🚥 Continue with Post-release tasks

🍀 Post-Release tasks

Action	Command
Finally, push the commit that changed release.properties to main	$ git push origin main (where you’ve checked out spring-data-release)
Close release tickets.	$ tracker close $trainIteration
Create new release versions and tickets for upcoming version	$ tracker setup-next $trainIteration.next
Update versions in the Projects Service. $targets is given as comma separated lists of code names, without spaces. E.g. Moore,Neumann	$ projects update $releasetrains
Create list of docs for release announcements	$ announcement $trainIteration
Announce release (Blog, Twitter) and notify downstream dependency projects as needed.	N.A.

Congratulations 🥳 You completed the release ❤️.

Appendix

One Time Setup

Infrastructure requirements

• Ensure you have the credentials for buildmaster accounts on https://repo.spring.io.

Both are available in the Spring/Pivotal Last Pass repository.

Prepare local configuration and credentials

Add an application-local.properties to the project root and add the following properties:

• git.username - Your GitHub username.

• git.password - Your GitHub Password (or API key with scopes: public_repo, read:org, repo:status, repo_deployment, user when using 2FA).

• git.author - Your full name (used for preparing commits).

• git.email - Your email (used for preparing commits).

• maven.mavenHome - Pointing to the location of your Maven installation.

• deployment.username - Your Artifactory user.

• deployment.api-key - The Artifactory API key to use for artifact promotion.

• deployment.password - The encrypted Artifactory password..

• gpg.keyname - The GPG key name.

• gpg.passphrase - The password of your GPG key.

• gpg.executable - Path to your GPG executable, typically /usr/local/MacGPG2/bin/gpg2 or /usr/local/bin/gpg.

• project-service.key - Project Service authentication token.Must be a valid GitHub token.Can be the same as git.password when using a GitHub token as password.

After that, run the verify local command ($ verify local) to verify your local settings (authentication, correct Maven, Java, and GPG setup).

See application-local.template for details.

Detailed commands performed by spring-data-release-cli

Action	Command
All release tickets are present	$ tracker releasetickets $trainIteration
Self-assign release tickets	$ tracker prepare $trainIteration
Prepare the release
	$ release prepare $trainIteration
	$ release conclude $trainIteration
Build the release
Build the artifacts from tag and push them to the appropriate maven repository. Also runs smoke tests, does Sonatype "release" if applicable, and does Artifactory "promote" if applicable.	$ release build $trainIteration
Distribute documentation and static resources from tag	$ release distribute $trainIteration
Push the created commits to GitHub	$ github push $trainIteration
Push new maintenance branches if the release version was a GA release (X.Y.0 version)	$ git push $trainIteration.next
Post-release tasks
Close JIRA tickets and GitHub release tickets.	$ tracker close $trainIteration
Create new release versions and tickets for upcoming version	$ tracker setup-next $trainIteration.next
Update versions in Projects Service. $targets is given as comma separated lists of code names, without spaces. E.g. Moore,Neumann	$ projects update $releasetrains
Create list of docs for release announcements	$ announcement $trainIteration

Utilities

Java and Maven Versions used in the Container

Java and Maven versions are installed via SDKman during the Dockerfile build. See ci/java-tools.properties for further details.

GitHub Labels

ProjectLabelConfiguration contains a per-project configuration which labels should be present in a project. To apply that configuration (create or update), use:

$ github update labels $project

Dependency Upgrade

ProjectDependencies contains a per-project configuration of dependencies.

Workflow:

• Check for dependency upgrades $ dependency check $trainIteration

Reports upgradable dependencies for Build and Modules and creates dependency-upgrade-build.properties file. Edit dependency-upgrade-build.properties to specify the dependency version to upgrade. Removing a line will omit that dependency upgrade.

• Apply dependency upgrade with $ dependency upgrade $trainIteration. Applies dependency upgrades currently only to Spring Data Build.

• Report store-specific dependencies to Spring Boot’s current upgrade ticket (sample) $ dependency report $trainIteration

CI Properties Distribution

To distribute ci/pipeline.properties across all modules use:

$ infra distribute ci-properties $trainIteration

Broken Link Report

Resolve external links in reference documentation and print their status.

$ docs check-links $trainIteration

Flag	Description
--local	read the documentation from disk (target module workspace directory)
--project	only check links of a specific project (eg. redis)
--report	only report errors of the given categories (ERROR,REDIRECT,OK). Default is ALL

Examples - Check links of release train/module

$ docs check-links Turing GA --report ERROR

$ docs check-links Turing SR1 --local true --project redis

Resolve external links of any web page (remote/local) and print their status.

$ check-links $url

Flag	Description
--report	only report errors of the given categories (ERROR,REDIRECT,OK). Default is ALL

Example - Check links on any url

$ check-links file:///usr/git/spring-data-mongodb/target/site/reference/html/index.html --report ERROR,REDIRECT

spring-data-release's Issues

Break out Artifactory promotion into a separate task

Right now, staging and promotion are done all at the same time for an Artifactory release.

If we could capture all the necessary metadata and leave the user with either A) instructions or B) commands to execute (or some other CI mechanism), it would be nice to better align Artifactory staging with Maven Central staging.

Maven Central staging is very easy to drop the release if something went wrong and simply try again, while Artifactory is arduous to do the same. That's because Artifactory doesn't afford us a single "build" record.

Upgrade U release train to 2023.0

Adopt Artifactory client to changed authentication scheme

Investigate documentation build for snapshots

We should investigate how we can build snapshot docs for all modules with daily CI Job. Documentation requires all modules to be checked out so it might be useful to do this on our main branches.

Consider moving documentation build into the `release build` phase

Documentation can be re-deployed so we can move that step into a phase where we haven't pushed github tags yet

Capture procedure to handle container updates

When updating the tools to a new version of Maven or making tweaks to the Dockerfile, we should have a nicely defined procedure on how to update the respective container so it doesn't have to be done during release day.

Revise release check list

We should re-introduce the checklist that reflects the entire process including ticket self-assignments and mailing list updates

Review each @CliCommand (annotation) to ensure the command specifies “help” (attribute)

As suggested by @jxblum:

• Include a brief summary of the commands actions and expected outcome.
• Optionally, include a link referring to additional (external) documentation used to supplement the help information with additional details, such as how to respond in a failure case.

“Contextual-based help” will potentially go a long way in adequately documenting the build/release process in order to help guide SD veterans and newcomers, alike, navigate a build and release process across environments.

Make the tool JDK 17 compatible.

Fail immediately when project metadata cannot be updated.

Instead of retrying multiple times before eventually failing we aim to reduce pressure on the target system by failing right away.

Switch to GPGv1 for key management

JGit doesn't fully handle GPGv2 KBX files. Switching to GPGv1 for managing keyrings on CI servers makes things work properly.

Use GPG2 binary for Maven

GPG1 doesn't work as it isn't compatible with --pinentry-mode.

Add 2023.1 (Vaughan) release train

Use Spring Data BOM's GitHub Release as Release Notes Link aggregator

We should put links to the individual module's release notes into the release notes of the BOM

Introduce "Post ALL modules released" section

Some of the commands in the release process should only be done when ALL release trains for a given release cycle are completed.

Ensure command shell exits when cmdfile is finished

Previously, we didn't consider ExitShellRequest

Pushing ci/release.properties to release branch requires "-f"

Since every time we are forcibly updating the release branch, the command in the instructions should include "-f".

Allow task separation through `release.properties`

We should introduce means of controlling what targets should be run by the automated release tooling, ideally through release.properties as this is a place that needs to be edited for a given release.

We have three scenarios:

1. Run the full release (full)
2. Prepare the release and push tags (tag)
3. (Re-)run binary and documentation builds including staging. This step requires release tags to be present (build)

The full target is composed from tag and build targets. That separation allows for retries.

Promote the Maven Central staging repository

Now that we have smoke tests in place, we can fully promote the staging repo to Maven Central.

We should also refactor the place where we open our staging repo and run smoke tests to pull these into a higher level for easier maintenance.

Update documentation

prepare-it takes care of self-assignment of tickets, so we no longer need to handle this as individual step.

Confine building the container only when the Dockerfile changes

Per #1 (review), alter Jenkins to ONLY re-build the container when the Dockerfile changes.

Upgrade JDKs in build image

• spring-projects/spring-data-build#1898
• spring-projects/spring-data-build#1899
• spring-projects/spring-data-build#1900

Upgrade to Maven 3.9.1

Add `2022.1` (U) release train

Add "git" to the release tool's Docker container

We run into scenarios where having git on the command line to diagnose circumstances from within Jenkins would be quite handy.

Introduce assertion that a given release tag is really a tag

When performing a git co-tag operation, it would be nice to assert that this is REALLY a tag before moving forward.

@mp911de said: "in the git operations, also, you need to translate the train iteration into individual versions and then tag names".

I frankly don't understand this. I tried to look through the metadata, and couldn't see what exactly a train-level tag is nor what to check it against.

Update working directories to reside in the Jenkins workspace directory (within the release tools checkout dir)

Avoid out of disk issues and allow access to log files

Enable authentication for repo.spring.io downloads

libs-milestone and libs-snapshot require authentication now.

Remove Spring Data Geode from `2022.0` release train

Update Slack message to reflect current release process

No need for the Slack message to say "Release the build (if needed)". The tools now release it either way.

Asciidoctor build fails on linux

[ERROR] Failed to execute goal org.asciidoctor:asciidoctor-maven-plugin:2.1.0:process-asciidoc (html) on project spring-data-neo4j: Execution html of goal org.asciidoctor:asciidoctor-maven-plugin:2.1.0:process-asciidoc failed: An API incompatibility was encountered while executing org.asciidoctor:asciidoctor-maven-plugin:2.1.0:process-asciidoc: java.lang.UnsatisfiedLinkError: /home/jenkins/.sdkman/candidates/java/8.0.352-tem/jre/lib/amd64/libfontmanager.so: libfreetype.so.6: cannot open shared object file: No such file or directory

Not sure why this is the only build that fails. We can resort to local docs builds for the time being.

Rotate keys used on CI server

Provide Spring Builds GPG public keys

See spring-projects/spring-data-commons@e92eee6

Currently, we use the key id 9A2C7A98E457C53D

Migrate toward VMware's Docker Hub caching service.

`tracker close` and `tracker setup-next` should handle milestones of the Spring Data Release repository

We now keep track of release dates using this repository. As part of the release tooling, we should automate the creation and close of release milestones.

Try to resolve SDKMAN candidates directory from environment

In some cases the SDKMAN candidates directory cannot be found in its default location which is currently hard coded in JavaRuntimes.
However, SDKMAN typically sets 2 environment variables (SDKMAN_DIR & SDKMAN_CANDIDATES_DIR) that can hint us to the actual location. So we should check for the presence of those variables before falling back to the default location.

Update procedures to ONLY put changes on ci/release.properties on the release branch itself

Since this is merely a signal used during a release, there is no need to actually capture this on the main branch. The tag is the real unit of history that we capture for a given release.

Simplify JDK and Maven installation and version upgrading

Right now, Maven version is not controlled and every time we upgrade Java versions, we forget one or the other place to upgrade resulting in additional cycles to fix the issues.

It would be good to have a definition file such as java-tools.properties in the ci and Docker environment that is a single place to be changed so that both, the Docker image and the release tooling can consider the required versions.

Additionally, the Release tooling should verify the installed versions on startup.

Add support for projects service at api.spring.io

Refine `build` and `release build`/`build-distribute` commands for Git operations

Between "release prepare " and "release conclude ", it would be nice to do a "build " command. This is the version that WILL BE tagged, right before the tagging step happens. Doing a "mvn clean install" against this version would be nice as a way to fail faster if something is wrong.

However, the following issues preclude this from happening:

• "build " actually does a git.prepare operation before doing the build. We'd need a command option (@CliOption(key="in-place") Boolean inPlace) that lets us skip over this and simply do the build job on all the modules.
• Even with this option plugged in, for some reason, Jenkins would get stuck on SD MongoDB. I didn't have time to investigate why this build would stall out for over an hour.

Either we need to uncover what is going wrong so we can bring in this extra check, or we need an alternative way to do "mvn verify" on every module in the BuildCommands.

Remove link formatting for release contributors

GitHub renders profile images for each contributor without link formatting on the release page.

Run maven with `-B` to run the process in batch mode

Without -B, we get a lot of download progress updates that clutter the log

Double `/` in secring paths

-Dgpg.secretKeyring=/home/jenkins/.gnupg//secring.gpg

Upgrade to Java 1.8.0.372

Introduce `verify-local` command to verify the local build setup required to release Spring Data

Revise repository declarations handling

During our releases, we switch repositories/repository elements to libs-snapshot, libs-milestone, and libs-release. With the ongoing restrictions to repo.spring.io, we are required to update our scheme to:

Snapshots:

• repo.spring.io/snapshot
• repo.spring.io/milestone (to be able to build against other milestones)

Milestones:

• repo.spring.io/milestone

Release:

• (remove the spring repository entry and fall back to Maven Central)

Add smoke tests

We should have a smoke tests application that we can point to a repository and Spring Data version to run light smoke tests (e.g. spinning up MappingContext's) to verify that our built JARs are working properly.