wcm-io-devops / jenkins-pipeline-library

wcm.io Jenkins Pipeline Library for CI/CD

Jenkins Pipeline Library

Since Jenkins Pipeline has reached a certain state of production scripted pipelines are the way to go.

But: Not everything known from the UI is available in Pipeline and configuring and writing scripts is not so easy for the normal developer.

The target of this library is to take out some complexity (and yes adding some too) of the pipeline creation and to bring back some known functionality (for example GIT_BRANCH and SCM_URL environment variables, mail notification on still unstable etc.)

Want to see an example? Have look at Usage examples

Table of contents

• Key concepts
• Requirements
• Steps
• Utilities
• Credential and managed file auto lookup
• Support for command line execution
• Setup your environment to use the pipeline library
• Building/Testing
  • Building with maven
• Changes / Version History

Key concepts

The pipeline library was developed with a focus to ease Java and Maven build processes within companies which have a more or less similiar project structure e.g.

• Maven/Java
• local Artifact Server (like Sonatype Nexus or Artifactory)
• GIT

The assumption is that in these environments

• Jenkins has a dedicated user account to checkout code (or one per project)
• the artifact server caches public artifacts and acts as a internal artifact server

❓ So why configure maven repositories and scm credentials in every pipeline?

So the key concepts of the pipeline enable you to

• Auto provide credentials (no worries, only Jenkins credential ids, not the credential itself) (see Credentials)
• Auto provide maven settings (see ManagedFiles)
• configure each job the same way (see ConfigStructure)
• log and see the things you are interested in (see Logging)

to builds.

Running this pipeline library will result in more structured and easier to maintain pipeline scripts.

Configured properly this library enables you to checkout scm with these lines of code:

import static io.wcm.devops.jenkins.pipeline.utils.ConfigConstants.*
checkoutScm( (SCM) : [
        (SCM_URL) : "[email protected]/group/project.git",
    ]
)

Or running maven with local and global maven settings with these lines of code:

import static io.wcm.devops.jenkins.pipeline.utils.ConfigConstants.*
execMaven( 
    (SCM) : [
        (SCM_URL) : "[email protected]/group/project.git",
    ],
    (MAVEN): [
        (MAVEN_GOALS) : [ "clean", "install" ]
    ]
)

❓ Looking for an example on how a pipeline script looks like when using Pipeline? Have a look at Usage examples

💡 Have a look at the setup tutorial to start using Pipeline Library.

Requirements

Have a look at requirements to get the library running.

Steps

The pipeline library comes with the following steps:

• ansible
  • ansible.checkoutRoles
  • ansible.execPlaybook
  • ansible.getGalaxyRoleInfo
  • ansible.installRoles
• checkoutScm
• conditionalStage
• credentials
  • lookupHttpCredential
  • lookupScmCredential
  • lookupSshCredential
• execMaven
• execMavenRelease
• execNpm
• genericConfig
  • load
• getScmUrl
• gitTools
  • mirrorRepository
  • mirrorRepositoryToRemote
  • mirrorRepositoryToWorkspace
  • getFetchOrigin
  • getPushOrigin
  • getParentBranch
• im (instant messaging)
  • im.mattermost()
  • im.teams()
• jenkinsRestApi
  • jenkinsRestApi.findJobsByNameRegex
• managedScripts
  • managedScripts.execJenkinsShellScript
  • managedScripts.execPipelineShellScript
• maven
  • purgeSnapshots
• notify
  • notify.mail
  • notify.mattermost
  • notify.mqtt
  • notify.teams
• setBuildName
• setScmUrl
• setupTools
• sshAgentWrapper
• transferScp
• wrappers
  • wrappers.color

Utilities

• Build Parameter Factory
• Integration Testing
• Logging
  • Logger
  • LogLevel
• Configuration map merging
  • MapUtils
  • MapMergeMode

Credential and managed file auto lookup

• Credentials
  • Credential
  • CredentialParser
  • CredentialConstants
• Generic Configuration
• ManagedFiles
  • ManagedFile
  • ManagedFileParser
  • ManagedFileConstants
• PatternMatching
  • PatternMatchable
  • PatternMatcher

Support for command line execution

• CommandBuilder
• MavenCommandBuilder

Setup your environment to use the pipeline library

Please refer to SetupTutorial for detailed instruction on how to setup the library in your environment.

Building/Testing

The library uses two approaches for testing.

The class parts are tested by unit testing using JUnit/Surefire. All unit tests have the naming format *Test.groovy and are located below test/io.

The step parts are tested by using Jenkins Pipeline Unit with jUnit/Failsafe. All integration tests have the naming format *IT.groovy and are located below test/vars.

Building with maven

mvn clean install

Add license file headers

mvn license:update-file-header

Changes / Version History

Please have a look at the Releases

jenkins-pipeline-library's Issues

java.lang.IllegalAccessError: class io.wcm.devops.jenkins.pipeline.managedfiles.ManagedFile tried to access private field io.wcm.devops.jenkins.pipeline.model.PatternMatchable.id

When running the jenkins-pipeline-library with the newest Jenkins (2.387.1) execMaven throws an error:

java.lang.IllegalAccessError: class io.wcm.devops.jenkins.pipeline.managedfiles.ManagedFile tried to access private field io.wcm.devops.jenkins.pipeline.model.PatternMatchable.id (io.wcm.devops.jenkins.pipeline.managedfiles.ManagedFile and io.wcm.devops.jenkins.pipeline.model.PatternMatchable are in unnamed module of loader org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$CleanGroovyClassLoader @f16bbba)

Looking at the respective classes Line 38 in src/io/wcm/devops/jenkins/pipeline/managedfiles/ManagedFile.groovy one can see that this.id = id is applied, after the super call to PatternMatchable. Looking inside that class under src/io/wcm/devops/jenkins/pipeline/model/PatternMatchable.groovy one can also see that inside the constructor the id is already set. The call inside the ManageFile class is therefore redunant.

Removing it from there will still lead to the id being set and therefore not break anything.

Scripts not permitted to use new

When setting up the library on a blank jenkins with the latest plugin versions, it's not possible to build as the script uses "illegal" methods which are rejected by the sandbox.
A quick win solution is to install the Permissive Script Security Plugin (i know what I'm doing...), but this will remove the complete security layer provided by the sandbox.

org.jenkinsci.plugins.scriptsecurity.sandbox.RejectedAccessException: Scripts not permitted to use new io.wcm.devops.jenkins.pipeline.utils.logging.Logger java.lang.Object
	at org.jenkinsci.plugins.scriptsecurity.sandbox.whitelists.StaticWhitelist.rejectNew(StaticWhitelist.java:184)
	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onNewInstance(SandboxInterceptor.java:170)
	at org.kohsuke.groovy.sandbox.impl.Checker$3.call(Checker.java:197)
	at org.kohsuke.groovy.sandbox.impl.Checker.checkedConstructor(Checker.java:202)
	at org.kohsuke.groovy.sandbox.impl.Checker$checkedConstructor$3.callStatic(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallStatic(CallSiteArray.java:56)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callStatic(AbstractCallSite.java:194)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callStatic(AbstractCallSite.java:214)
	at io.wcm.devops.jenkins.pipeline.utils.maps.MapUtils.<clinit>(MapUtils.groovy:35)

enum in library accessible from the jenkins pipeline

is it possible to define enum in library and use its value as a function parameter?
where enum should be located in src or vars?

GitRepository.groovy sample

it's possible use the jenkins-pipeline-library/src/io/wcm/devops/jenkins/pipeline/scm/GitRepository.groovy

to fetch a branch into workspace ?

Can you post an example ?

Implement mock for findFiles

Would like to see a findFIlesCallback implemented in the LibraryIntegrationTestBase to allow returning of all files that match a pattern provided.
Example searches:

• findFiles glob: "*.gemspec"
• findFiles glob: "lib/*/version.rb"
• findFiles glob: "src/*.groovy"
• findFiles glob: "my-example.rb"

helper.registerAllowedMethod(FIND_FILES, [LinkedHashMap.class], findFilesCallback)
...
def findFilesCallback = {
    Map config ->
    // TODO:
  }

Was reviewing the locateTestResource for a possible existing workaround but doesn't account for the fuzzy searches only exact file names. Initially tried

File file = this.dslMock.locateTestResource(config["glob"].toString())

But is when discovered the fuzzy search did not work. I do like being able to drop all resources in the /test/resources folder for any of the searches so would like the findFiles to follow that pattern.