epam / edp-tekton

Tekton library for EDP pipelines. EDP Interceptor that enriches payload from VCS with EDP entities

EDP Tekton

• EDP Tekton
  • EDP Interceptor
  • Tekton Pipelines

The edp-tekton repository consolidates elements for Tekton integration with EDP EPAM Delivery Platform (EDP). and disposes of two main components:

• EDP Interceptor. Follows Tekton Interceptor paradigm and enriches payload from different Version Control Systems (VCS) like Gerrit, GitHub or GitLab with EDP specific data.
• Tekton Pipelines. Consists of Tekton Tasks, Pipelines, Triggers and implements EDP CI Pipelines logic. Some of the tasks are forks from origin source, the others are EDP specific.

EDP Interceptor

EDP Interceptor is used as a component that provides EDP data for Tekton Pipelines. The code is based on Upstream implementation.

EDP Interceptor extracts information from VCS payload, like repository_name. The repository_name has 1-2-1 mapping with EDP Codebase (kind: Codebase; apiVersion:v2.edp.epam.com/v1). Interceptor populates Tekton Pipelines with Codebase SPEC data, see the diagram below:

    ┌────────────┐              ┌─────────────────┐       ┌─────────────┐
    │            │              │ EDP Interceptor │       │   Tekton    │
    │  VCS(Git)  ├──────────────►                 ├───────►             │
    │            │              │                 │       │  Pipelines  │
    └──────┬─────┘              └────────┬────────┘       └─────────────┘
           │                             │
    ┌──────┴─────┐                       │ extract
    │    Repo    │                       │
    │            │                       │
    │            │      ┌────────────────▼───────────────┐
    └────────────┘      │ apiVersion: v2.edp.epam.com/v1 │
                        │ kind: Codebase                 │
                        │                                │
                        │ spec:                          │
                        └────────────────────────────────┘

The data, retrieved from the Codebase SPEC, is used in Tekton Pipelines logic. The docker images for EDP Interceptor are available on the DockerHub. The helm-chart for interceptor deployment is in the same repository by the charts/interceptor directory.

Tekton Pipelines

Tekton Pipelines supports three VCS: Gerrit, GitHub, GitLab. To check the VCS Import strategy, please refer to the EDP Documentation.

EDP Tekton Pipelines are implemented and packaged using the helm-chart approach. The helm-chart contains:

• Tasks - basic building block for Tekton. Some of the tasks are forks from Upstream Tekton Catalog.
• Pipelines, which consist of Tasks and implement logic for the CI flow. EDP follows the below approach for pipelines definition:
  • Each type of VCS has its own Pipelines, e.g. for Gerrit, GitHub, GitLab;
  • EDP has two types of Pipelines: CodeReview - triggers on Review, Build - triggers on Merged Event.
• Triggers, TriggerBindings, TriggerTemplates - defines the logic for specific VCS Events (Gerrit, GitHub, GitLab) and Pipelines.
• Resources - Kubernetes resources, that are used from Pipelines, e.g. ServiceAccount with IRSA Enablement, ConfigMaps for Maven/Gradle Pipelines, Tekton cache, CodeNarc, CTLint, and PVC to share resources between Tasks.
• Tekton Pipeline pruner - created as a cron job, it is designed to clear outdated pipelines.

edp-tekton's Issues

Upgrade Go to 1.20

Create cdxgen Task

Create cdxgen Task and add it to the EDP4EDP Pipelines (in SCA step)

Here is a code that works:

apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  annotations:
    tekton.dev/displayName: Dep-Analysis
    tekton.dev/platforms: linux/amd64
  name: cdxgen
spec:
  params:
    - default: ''
      description: Dependency Track URL which is used for pushing SBOM
      name: DEPTRACK_URL
      type: string
  steps:
    - env:
        - name: API_TOKEN
          valueFrom:
            secretKeyRef:
              key: token
              name: ci-dependency-track
        - name: DEPTRACK_URL
          value: $(params.DEPTRACK_URL)
      image: >-
        ghcr.io/cyclonedx/cdxgen:v9.6.0@sha256:ea01324872d2c21b024264a2224d761ab63851b9cc4722903b5e74be56ca6fa6
      name: cdxgen
      resources: {}
      script: >
        #!/usr/bin/env sh
        set -e
        set +x
        /opt/cdxgen/bin/cdxgen.js --api-key=$API_TOKEN --server-url=$DEPTRACK_URL --project-name=CODEBASENAME
      workingDir: $(workspaces.source.path)
  workspaces:
    - name: source

Remove NuGet token from output log

Acceptance Criteria:

• Log Tekton task does not contain NuGet token in the output

Failed build pipeline for helm application on helm-template step for default versioning

Support POM packaging in build pipelines for Maven libraries

Is your feature request related to a problem? Please describe.

Tekton pipelines for Maven library build don't support projects with <packaging>pom</packaging>.
Absense of support of the feature causes errors, please check attached logs for details.

Describe the solution you'd like

Either treat projects with <packaging>pom</packaging> in pom.xml as multimodule projects, or add a check for target/ and src/test/ folders before preparing them to copy in prepare-files step of the sonar-prepare-files task.

Describe alternatives you've considered

As a workaround it helps to force switching to multimodule build by adding <deployable.module>pom</deployable.module> to <properties/>. But this can be confusing.

Additional context

More information on POM packaging:
https://maven.apache.org/pom.html#aggregation-or-multi-module
https://maven.apache.org/pom.html#inheritance

Logs:
lb-parent-master-review-p6m9v-sonar-prepare-files-pod__step-prepare-files__log.txt

Pipelines can't build multi-module Maven projects

Describe the bug

Build and Code Review pipelines with Maven fail for multi-module projects.
Build pipeline fails on building docker images.
Code Review pipeline fails on sonar / mvn-goals task.

Steps To Reproduce

What did you do?

1. Create a sample multi-module Java 8 / Maven project in EDP v2.12.1 Admin Console with the Create strategy. See screenshots section for details.
2. Clone the new repository and transfer it to EDP v3.2.1.
3. Make a change and start the Code Review pipeline.
4. Observe the failure on the Tekton sonar / mvn-goals task.
5. Ignore the failed Code Review and proceed to merge the change to start the Build pipeline.
6. Observe the failure on the Tekton kaniko-build / build-and-push task.

What did you expect to see?

A successfully executed Code Review and Build pipelines.

What did you see instead? Under which circumstances?

Failure on the sonar / mvn-goals task of the Code Review pipeline.
Failure on the kaniko-build / build-and-push task of the Build pipeline.

Versions

EDP: v3.2.1
Git Provider: Gerrit
Kubernetes: AWS EKS v1.24.11

Tekton:

• Pipelines: v0.45.0
• Triggers: v0.23.0
• Trigger Interceptors: v0.23.0

Screenshots

Create multi-module application in EDP v2.12.1:

Failure on the Tekton sonar task of the Code Review pipeline:

vvvvvvvvvvvvvvvvvv

Failure on the Tekton kaniko task of the Build pipeline:

Additional context

Application code review pipeline fails on dockerbuild-verify due to absence of a build step/task

Describe the bug

Some Code Review pipelines do not contain a separate Build step, for example:

• gerrit-maven-java11-app-review
• gerrit-maven-java8-app-review
• gerrit-npm-react-app-review

Which causes a failure on the dockerbuild-verify step due to the absence of the artifacts to be copied or added with a corresponding instruction in Dockerfile.

Steps To Reproduce

What did you do?

1. Create a sample JavaScript / NPM project in EDP Headlamp with the Create strategy.
2. Clone the Gerrit repository of the new project and change directory into the cloned folder.
3. In the package.json file within scripts section set build:clean command to a more appropriate:
   npm install && npm cache clean --force && npm ci.
4. Save the file, stage, commit, and start a code review for the change.

What did you expect to see?

A successfully executed Code Review pipeline.

What did you see instead? Under which circumstances?

Failure on the dockerbuild-verify step.

Versions

EDP: v3.2.1
Git Provider: Gerrit
Kubernetes: AWS EKS v1.24.11

Screenshots

Additional context

Add possible to push release artifact to dockerhub instead of ecr

Migrate edp-common to Tekton

Acceptance Criteria:

• Tekton custom pipelines contain review and build platform pipelines;

Autotest failed for maven framework at GitLab VCS

STR:

Create maven-autotest codebase using Clone or Import strategy
Run review pipeline
ER: Pipeline should be completed successfully

AR: Pipeline failed

Migrate release custom pipelines to Tekton CI

Update external component logic

Acceptance Criteria:

• mount necessary secrets with integration to tekton tasks instead of hardcoode or get values from EDPComponents;

Enable resources (limit/request) definition

Acceptance Criteria:

• edp-tekton repository contain resources variable in values.yaml for each of deployed containers

Migrate edp-autotests pipelines to Tekton

Acceptance Criteria:

• Tekton custom pipelines contain review and build autotests pipelines;

Tekton Task update-build-number-gradle: Regular expression bug in updating version key

The Tekton Task update-build-number-gradle currently has a bug in its regular expression used within the sed command. The sed command should update the version key in the build.gradle file with the new BRANCH_VERSION. However, the existing regular expression implementation has an unintended impact, as it updates all the version keys present in the file.

Failed build pipeline for application helm default versioning type

STR:

Create helm application codebase with default versioning type
Run build pipeline
ER: Build pipeline should be passed

AR: Build pipeline failed with an error

invalid result reference in pipeline task "helm-push": "BUILD_VERSION" is not a named result returned by pipeline task "get-version"

Same issue for Helm Charts library.

add e2e test for sonar operator in tekton pipeline

Add e2e test for new sonar operator in tekton pipelines

Release 0.8.0

Acceptance Criteria:

• Release edp-tekton 0.8.0 with latest fixes, features;

Add review and build pipelines for autotest type for java17

Add review and build pipelines for autotest type for java17.

Migrate Kaniko parameter definition from edp-component to edp-config configmap

Wrong order of Push and Kaniko steps in Maven app build pipeline cause unnecessary jar in docker image

Describe the bug

Tekton pipeline for Maven application build has Push (mvn deploy) step before Kaniko step.
This causes creation of unnecessary artifacts (for example *-sources.jar) and copying them into docker image.
Jenkins build pipelines in EDP v2.12 had Kaniko step before Push.

As a workaround a .dockerignore file can be placed into the project's root directory or deployable module directory. The latter can be confusing.

To Reproduce

What did you do?

1. Create an Maven application project with sources plugin set up like this:

  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-source-plugin</artifactId>
        <version>${maven-source-plugin.version}</version>
        <executions>
          <execution>
            <id>attach-sources</id>
            <phase>verify</phase>
            <goals>
              <goal>jar</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>

2. Add the application to EDP and start an application build pipeline.
3. Pull the image from image registry, run it and check what's inside.

What did you expect to see?

Only the deployable *.jar file(s).

What did you see instead? Under which circumstances?

The *-sources.jar file(s) along with the deployable *.jar file(s).

Kubernetes cluster type:

AWS EKS v1.24

$ operator-sdk version

Not applicable.

$ go version

Not applicable.

$ kubectl version

Client Version: v1.24.3
Kustomize Version: v4.5.4
Server Version: v1.24.13-eks-0a21954

Migrate platform pipelines to Tekton

Acceptance Criteria:

• Tekton custom pipelines contain review and build platform pipelines;

Migrate content from ct-config configmap to values.yaml

As EDP User, I want to have opportunity to redefine default ct configuration during EDP deploy.

Acceptance Criteria:

• the user can specify сt configs through the values.yaml file when deploying the platform;
• by default values.yaml contains default configs from the EDP team

chore(deps): Bump golang.org/x/net from 0.9.0 to 0.17.0

Update dependencies

Release 0.6.0

Acceptance Criteria:

• edp-tekton 0.6.0 released;

Disable read-only mode for tekton dashboard

Acceptance Criteria:

• by default Tekton dashboard deployed with read/write permissions https://github.com/epam/edp-tekton/blob/master/charts/pipelines-library/values.yaml#L215

Sonar 9.9 LTS deprecates java8 for sonar-scan

Acceptance Criteria:

• sonar push work for sonar java8.

Add ability to use dockerhub like container registry in EDP.

Add ability to use dockerhub in CI/CD process

"Push" task pushes incorrect application version into Nexus.

"Push" task pushes incorrect application version into the Nexus.
You can reproduce this bug in Java applications.

Failed sonar-properties-create step in the sonar stage in build and review pipelines

Failed build pipeline for dotnet 6.0

Remove edpName usage

Migrate from edpName usage to ReleaseName as default variable in helm chart.
https://github.com/epam/edp-install/blob/master/deploy-templates/values.yaml#L10

AC:

• align remove edpName from values.yaml
• align helm Release Name usege;
• align documentation;

Release 0.7.0

Acceptance Criteria:

• edp-tekton 0.7.0 released;

Upgrade vcluster to version v0.15.7

We need to update our helm custom image with new vcluster version

Upgrade pull request template

Ensure no pre-commit or any other type for git hooks are present

It appears that there may be an issue with the git-push step in the build pipeline, specifically in regards to the git-cli task. This is potentially due to the presence of pre-commit hooks in the code. To mitigate this, it is necessary to confirm that no hooks are present before executing any git-cli commands.

Enable kaniko custom certs support

Acceptance Criteria:

• in case if registry use custom certs, kaniko can use the same certificate or flag to skip verification;
• case valid for okd and eks clusters;

Make helm-docs as a mandatory step in CI/CD

Create helm-docs template to run from it in order not to be dependent on the helm-docs version.

This line should be absent in the template:
{{ template "helm-docs.versionFooter" . }}

Examples of templates can be found here: https://github.com/norwoodj/helm-docs/blob/master/example-charts/custom-value-notation-type/README.md.gotmpl

Acceptance criteria:

• helm-docs is a mandatory step for CI pipelines
• helm-docs templates are added to repos which are used for "create" strategy
• helm-docs are added as a part of Application onboarding process (clone, import strategy)

We need to ensure tekton chain controller has access to registry

Provide feature to signed image while pushing to harbor.

Align Tekton pipelines diff

Acceptance Criteria:

• tekton master branch contain valid yaml spec without argocd diff (gerrit/github/gitlab);

Add java maven support for autotests codebase

Add java-maven support for autotests codebase in QG, which currently support only gradle project. Maven failed on the init stage.

Acceptance Criteria:

• EDP user can onboard and use java maven as autotests;

Use default gitlab server gitlab.com

Acceptance Criteria:

• host for gitlab server - gitlab.com

Review pipeline fails on nested Maven multimodule projects

Describe the bug

Review pipeline fails for nested Maven multimodule projects.
The same project successfully passes review pipeline in Jenkins.

To Reproduce

What did you do?

1. In EDP Headlamp navigate to Components and click + to open Create a new component form.
2. Click Application and Clone project.
3. Use https://github.com/bwgjoseph/maven-nested-multi-module-project.git as the Repository URL.
4. Set corresponding values to Git repo relative path and Component name fields.
5. Click Java, then Java 17, and then Maven.
6. Wait for the application to be processed by EDP
7. Clone the repository, add a minimal Dockerfile, commit it and start a review process.
8. When the pipeline fails for the first time (see "Failure before" error log) add a <deployable.module> tag with the value project to the parent pom.xml and initiate the review process once again.

What did you expect to see?

A successful completion of a review pipeline.

What did you see instead? Under which circumstances?

At first, the review pipeline fails on the sonar-prepare-files Task. Check the attached "Failure before" log file.
Then it fails on the sonar Task. Check the attached "Failure after" log files.

Kubernetes cluster type:

AWS EKS v1.24

$ operator-sdk version

Not applicable.

$ go version

Not applicable.

$ kubectl version

Client Version: v1.24.3
Kustomize Version: v4.5.4
Server Version: v1.24.13-eks-0a21954

Screenshots

Not applicable.

Additional context

Failure before <deployable.module> specified:
maven-nestef5a827ec027260d0dd80746c973645ecf91d4dec7b2067c9-pod__step-prepare-files__log.txt

Failure after:
maven-nested-multi-module-project-main-review-w9vg2-sonar-pod__step-mvn-goals__log.txt
maven-neste0955a42c3f5caa8a220e270ec94953d0f3b7849e6a1706f9-pod__step-prepare-files__log.txt

Minimal Dockerfile: Dockerfile.txt

Align sonar-operator pipelines

Align sonar-operator pipelines to keycloak-operator

• Acceptance Criteria:
  sonar-operator runs the same tasks as the keycloak-operator pipeline

Align VCS secret name pattern

Acceptance Criteria:

• secret name - ci-gitlab/github;
• gerrit - without modification;

Use github as default gitserver

Acceptance Criteria:

• edp-tekton use github as a default gitserver;

Update tekton-dashboard to LTS

In current release EDP use 0.35.0 version of Tekton Dashboard.
Available new Tekton Dashboard Releases - https://github.com/tektoncd/dashboard/releases

Acceptance Criteria:

• EDP use latest stable version of Tekton Dashboard

Release Tekton

GitServer has wrong spec property

GitServer has wrong spec properties

we need to set SkipWebhookSSLVerification instead of skipWebhookTlsVerification

https://github.com/epam/edp-tekton/blob/master/charts/pipelines-library/templates/resources/gitservers/gitlab.yaml#L15

https://github.com/epam/edp-codebase-operator/blob/master/docs/api.md#gitserverspec

Automate image bump in Chart.yaml for release process

https://github.com/epam/edp-headlamp/blob/master/deploy-templates/Chart.yaml#L31

Acceptance Criteria:

in release pipeline automate in Chart.yaml in annotations section image version bump;