jenkins-x-plugins / jx-build-controller Goto Github PK

a controller which watches PipelineRuns and updates PipelineActivity resources and stores container logs in bucket storage

Dockerfile 0.28% Makefile 10.07% Go 67.47% Smarty 4.55% Shell 16.30% Mustache 1.34%

hacktoberfest

jx-build-controller's People

Contributors

Stargazers

Watchers

Forkers

rawlingsj yelhouti cameronbraid chrismellard vbehar hervelemeur ankitm123 keskad rajatgupta24 jordangoasdoue rubinus xkfen joker00777 spring-financial-group hazem-internship

jx-build-controller's Issues

Release 0.0.18 - fatal: could not read Username for 'https://github.com': No such device or address'

fatal: could not read Username for 'https://github.com': No such device or address'

After upgrading from 0.0.17 to 0.0.18 I'm getting the above error

store the pipelinerun in the long-term storage

related to jenkins-x/jx-pipelines-visualizer#54 : when we try to render the pipeline and logs for a pipelinerun that has been garbage-collected... well, we can't, because we have no information (except for the pipelinerun name and its namespace).

we could ensure links always work by storing the pipelineruns in the long-term storage, alongside the logs - similar to #11 for the pipelineactivities.

the main difference is that we can't use the standard layout of jenkins-x/logs/owner/repository/branch/build.log, because the pipelines-visualizer doesn't know the owner/repo/branch/build - only the pr name and ns.
so maybe we could store the pipelineruns under jenkins-x/logs/namespace/name.yaml? or jenkins-x/pipelineruns/namespace/name.yaml?

@jstrachan what do you think?

support for other git providers

There is an issue with: https://github.com/jenkins-x-plugins/jx-build-controller
GIT_SECRET_SERVER is used internally to know which server to clone jx-requirements from but the env variable is never set and there is no way to set it in the helm chart.
it should be set to in my case to gitlab.com I guess we could add this to the deployment in the helm chart but I think it's cleaner to fix requirements.go in jx-helpers to load and write info correctly to $XDG_CONFIG_HOME/git/credentials
@jstrachan what do you think.

JX Build Controller Fails to start- could not read Username No such device or address jx-build-controller

Using and - EKS/Bitbucket combo , the controller fails to start. I verified that the secret-git token is mounted properly in the file system yet this exception occurs :

fatal: could not read Username for 'https://XXXXXXXXX.com': No such device or address
jx-build-controller
FATAL: failed to get cluster requirements: failed to clone cluster git repo https://XXXXXXXXX.com/scm/project/jx-config.git: failed to clone repositoryhttps://XXXXXXXXX.com/scm/project/jx-config.git to directory: /tmp/jx-git-782291392: failed to run 'git clone https://XXXXXXXXX.com/scm/project/jx-config.git /tmp/jx-git-782291392' command in directory '/tmp', output: 'Cloning into '/tmp/jx-git-782291392'...

Logs not uploaded

Problem

I have recently installed jx3 in AWS EKS using and existing cluster and the TF provider in your instructions. When a build runs the logs are not accessible or stored in s3. I go into dashboard and attempt to access the raw logs and get this error:

failed to read key jenkins-x/logs/REDACTED/jenkinsx-node-quickstart/PR-3/2.log in bucket s3://logs-REDACTED-20220420182851392800000001: blob (key "jenkins-x/logs/sREDACTED/jenkinsx-node-quickstart/PR-3/2.log") (code=NotFound): NoSuchKey: The specified key does not exist.
	status code: 404, request id: REDACTED, host id: REDACTED/TgW8ggfHHH0/vzJFuXwh8fgk=

The build is successful but no logs are stored.

Debugging Done

Here are some steps I have used to debug to rule out permission issues.

All three buckets exist
The RBAC role that builder-controller is using is pointing to the correct IAM role
The service account that the builder-controller is using is bound to the correct RBAC role and permissions
The AWS IAM role give builder-controller full admin perms on all of s3

Versions

ghcr.io/jenkins-x/jx-build-controller:0.3.15
EKS v1.21.9-eks-0d102a7

jx-reqirements

apiVersion: core.jenkins-x.io/v4beta1
kind: Requirements
spec:
  autoUpdate:
    enabled: false
    schedule: ""
  cluster:
    chartRepository: http://jenkins-x-chartmuseum.jx.svc.cluster.local:8080
    clusterName: REDACTED
    devEnvApprovers:
    - todo
    environmentGitOwner: REDACTED
    gitKind: gitlab
    gitName: gl
    gitServer: https://gitlab.com
    project: "REDACTED"
    provider: eks
    region: us-west-2
    registry: REDACTED.dkr.ecr.us-east-2.amazonaws.com
  environments:
  - key: dev
    owner: REDACTED
    repository: REDACTED
  - key: staging
  - key: production
  ingress:
    domain: REDACTED.REDACTED.com
    externalDNS: true
    kind: ingress
    namespaceSubDomain: -jx.
    tls:
      email: REDACTED
      enabled: true
      production: true
  pipelineUser:
    username: REDACTED
  repository: nexus
  secretStorage: secretsManager
  storage:
  - name: logs
    url: s3://logs-REDACTED-dev-eks-20220420182851392800000001
  - name: reports
    url: s3://reports-REDACTED-dev-eks-20220420182851770800000006
  - name: repository
    url: s3://repository-REDACTED-dev-eks-20220420182851398900000002
  terraform: true
  vault: {}
  webhook: lighthouse

Questions

How do I debug more?
What process is actually writing the logs? Is there another pod that is missing the IAM token? I know we have moved from ClusterRoles to Roles recently

TLDR;

The s3 buckets exist, the build runs, no logs are saved, and we cannot see any logs in the dashboard.

Issues updating pipeline activities

Our pipeline activities are in a pending state:

jx          my-org-document-recogniser-master-2         https://github.com/my-org/document-recogniser.git         Succeeded
jx          my-org-document-recogniser-master-3         https://github.com/my-org/document-recogniser.git         Succeeded
jx          my-org-document-recogniser-master-4         https://github.com/my-org/document-recogniser.git         Succeeded
jx          my-org-document-recogniser-master-5         https://github.com/my-org/document-recogniser.git         Succeeded
jx          my-org-document-recogniser-master-6         https://github.com/my-org/document-recogniser.git         Failed
jx          my-org-document-recogniser-master-7                                                                                         
jx          my-org-document-recogniser-pr-123-1         https://github.com/my-org/document-recogniser.git         Succeeded
jx          my-org-document-recogniser-pr-123-19        https://github.com/my-org/document-recogniser.git         Pending
jx          my-org-document-recogniser-pr-123-2         https://github.com/my-org/document-recogniser.git         Failed
jx          my-org-document-recogniser-pr-123-20        https://github.com/my-org/document-recogniser.git         Pending
jx          my-org-document-recogniser-pr-123-21        https://github.com/my-org/document-recogniser.git         Pending
jx          my-org-document-recogniser-pr-123-4         https://github.com/my-org/document-recogniser.git         Succeeded
jx          my-org-document-recogniser-pr-123-5         https://github.com/my-org/document-recogniser.git         Failed
jx          my-org-document-recogniser-pr-123-6         https://github.com/my-org/document-recogniser.git         Pending
jx          my-org-document-recogniser-pr-123-9         https://github.com/my-org/document-recogniser.git         Running
jx          my-org-document-recogniser-pr-125-1         https://github.com/my-org/document-recogniser.git         Failed
jx          my-org-document-recogniser-pr-125-2         https://github.com/my-org/document-recogniser.git         Failed
jx          my-org-document-recogniser-pr-126-1         https://github.com/my-org/document-recogniser.git         Pending
jx          my-org-document-recogniser-pr-127-1         https://github.com/my-org/document-recogniser.git         Pending

I spoke to @vbehar and he told me to raise an issue here.

he also pointed me to the jx build controller and I found this log here:

{"level":"warning","msg":"failed to process PipelineRun pr-ldrnf in namespace jx: failed to update PipelineActivity after 5 attempts: failed to update PipelineActivity my-org-document-recogniser-pr-123-20 in namespace jx: PipelineActivity.jenkins.io \"my-org-document-recogniser-pr-123-20\" is invalid: spec.steps.stage.steps.startedTimestamp: Invalid value: \"null\": spec.steps.stage.steps.startedTimestamp in body must be of type string: \"null\"","time":"2021-06-18T09:12:21Z"

Here's our PR pipeline

apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
  creationTimestamp: null
  name: pullrequest
spec:
  pipelineSpec:
    tasks:
    - name: from-build-pack
      resources: {}
      taskSpec:
        metadata: {}
        stepTemplate:
          image: uses:jenkins-x/jx3-pipeline-catalog/tasks/python/pullrequest.yaml@versionStream
          name: ""
          resources:
            requests:
              cpu: 400m
              memory: 1Gi
          workingDir: /workspace/source
        steps:
        - image: uses:jenkins-x/jx3-pipeline-catalog/tasks/git-clone/git-clone-pr.yaml@versionStream
          name: ""
          resources: {}
        - name: jx-variables
          resources: {}
        - name: check-registry
          resources: {}
        - name: build-container-build
          resources: {}
        - image: ghcr.io/jenkins-x-plugins/jx-preview:0.0.143
          name: promote-jx-preview
          resources: {}
          script: |
            #!/usr/bin/env sh 
            source .jx/variables.sh
            jx secret copy --create-namespace --selector "secret.jenkins-x.io/replica-source=true" -n preview-secrets --to jx-$REPO_OWNER-$APP_NAME-pr-$PULL_NUMBER
            jx preview create
  podTemplate: {}
  serviceAccountName: tekton-bot
  timeout: 1h0m0s
status: {}

Not sure what's happening here. I'm going to look into it and I'll update the issue if I get anywhere

logs not available for pipelines with multiple Tasks

it looks like the logging code isn't working correctly for PipelineRuns with multiple tasks

Logging sensitive information

If you have a .git/config that has a token in it for authentication, that token is logged.

{"level":"info","msg":"about to run: git clone https://chrislovecnm:[email protected]/REDACTED/jx3-eks-asm.git /tmp/jx-git-3272795265 in dir /tmp","time":"2022-04-28T16:39:19Z"}

We should not log passwords.