clemlesne / azure-pipelines-agent Goto Github PK

Since the code refacto 60c077e the variable is broken. Variable need need to have either template or Pod name. Actually, the car is fixed.

Functionally, it is required to clean the disk after each pipeline execution. The security breach is enormous if data or context is shared between pipelines.

emptyDir are reused across Pod restart, but not across Pods.

emptyDir without wipe at Pod restart (what does ephemeral storage) creates a security risk.

Using the preStop hook to clean the directory, like so:

bash config.sh remove --auth PAT --token ${AZP_TOKEN}; rm -rf ${AZP_WORK}

Complete example in Helm:

lifecycle:
  preStop:
    exec:
      {{- if .Values.image.isWindows }}
      command: [powershell, -Command, ".\\config.cmd remove --auth PAT --token $Env:AZP_TOKEN; Remove-Item -Recurse -Force $Env:AZP_WORK"]
      {{- else }}
      command: [bash, -c, "bash config.sh remove --auth PAT --token ${AZP_TOKEN}; rm -rf ${AZP_WORK}"]
      {{- end }}

This is the mitigation I propose.

Originally posted by @clemlesne in #10 (comment)

Hi, trying to install node14 and other tools got this

Using version latest chart: 5.2.0 from: https://artifacthub.io/packages/helm/azure-pipelines-agent/azure-pipelines-agent

I had a problem compiling a large image on linux (Image Jammy 5.2.1).

I put the cache and tmpdir with 40G and the following error occurs

ephemeral-storage. Threshold quantity: ,available: Ki. Container azp-agent was using ││ Ki, request is 0, has larger consumption of ephemeral-storage

When I put ephemeral-storage in the Request, the PVC is in Peding status and the POD does not run...

resources:
requests:
cpu: 1000m
memory: 2Gi
ephemeral-storage: 20Gi

I'm analyzing if there is any wrong configuration in the volumes

Is it possible to create an option where I can choose whether I want an aspnet6 or aspnet7 docker image?

FROM mcr.microsoft.com/dotnet/aspnet:7.0-jammy as base

Or in another scenario Net Framework 4.8 with windows image...

Ref:

• microsoft/azure-pipelines-agent#4255
• https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/service-principal-agent-registration?view=azure-devops

Is this chart compatible with OpenShift with SCC restrictions? It seems to not like user 0.

$ helm install clem-ado clemlesne-azure-pipelines-agent/azure-pipelines-agent -n sandbox -f values.yaml
Error: INSTALLATION FAILED: 1 error occurred:
        * pods "clem-ado-azure-pipelines-agent-1" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider "pipelines-scc": Forbidden: not usable by user or serviceaccount, spec.volumes[0]: Invalid value: "ephemeral": ephemeral volumes are not allowed to be used, spec.volumes[1]: Invalid value: "ephemeral": ephemeral volumes are not allowed to be used, spec.containers[0].securityContext.runAsUser: Invalid value: 0: must be in the ranges: [1000920000, 1000929999], provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "noobaa": Forbidden: not usable by user or serviceaccount, provider "noobaa-endpoint": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "elasticsearch-scc": Forbidden: not usable by user or serviceaccount, provider "log-collector-scc": Forbidden: not usable by user or serviceaccount, provider "machine-api-termination-handler": Forbidden: not usable by user or serviceaccount, provider "nginx-ingress-permissions": Forbidden: not usable by user or serviceaccount, provider "k10-prometheus-server": Forbidden: not usable by user or serviceaccount, provider "k10-grafana": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "rook-ceph": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "rook-ceph-csi": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount]

Hello, thanks for making the awesome chart.

I'm having throubles when applying the chart, the error is:

Error: UPGRADE FAILED: failed to create resource: Pod "azdo-agents-cluster-non-production-azure-pipelines-agent-3" is invalid: [spec.volumes[0]: Required value: must specify a volume type, spec.volumes[1]: Required value: must specify a volume type, spec.containers[0].volumeMounts[0].name: Not found: "azp-work", spec.containers[0].volumeMounts[1].name: Not found: "local-tmp"]

Helm command:

helm upgrade
azdo-agents-cluster-non-production stable/azure-pipelines-agent --version 4.16.0
-f values.yml
--kube-context=my-context
--namespace=azdo-agents
--install
--wait
--set serviceName=azdo-agents-cluster-non-production
--output table

Chart Version: 4.16.0

My values.yaml

pipelines:
  cacheSize: 10Gi
  cacheType: gp2

  tmpdirSize: 1Gi
  tmpdirType: gp2
    
  # capabilities: []
  url: https://dev.azure.com/....
  pat: ....
  pool: default-non-production  

kubectl get storageclass

NAME            PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION 
gp2 (default)   kubernetes.io/aws-ebs   Delete          WaitForFirstConsumer   false                  

Hi @clemlesne , I've added kubectl cli using your image as base, so my dockerfile looks like this:

FROM ghcr.io/clemlesne/azure-pipelines-agent:jammy-main

RUN curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" \
    && sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

the error thru the pipeline agent:

command: kubectl config set-credentials "name" --token="token"

error: mkdir /app-root/.kube: permission denied
error: mkdir /app-root/.kube: permission denied
error: mkdir /app-root/.kube: permission denied

If I build and do a normal docker run -it image:1 bash and try to configure the kubectl, It works as expected, but thru the vstsagent running no.

Off-topic: is it possible to add some kind of container runner (not only builder)? I've tried to install podman, docker, ctr (containerd), but I can't pass the error about the is docker sock running? or running as pid 1, systemd not found or "/" is not a shared dir

Covered:

• Test against multiple Kubernetes versions
• Local Kubernetes distribution
• Ensure the Helm chart start
• Ensure the health check is confirmed
• Test with and without internet

Not covered:

• Test in a cloud provider
• End-to-end tests with pipeline run, analyzing result
• Performance test

Integration ideas:

• Use k3s for tuning locally the Kubernetes cluster in GitHub Actions

Hi,

Thanks for the great work setting this repo! :)

I've noticed that when setting the ScaleJob you hard coded the minReplicaCount number to 0. While I understand the rationale of setting it to 0 I also wanted it to be possible to set it to an arbitrary value.

I've also wanted to be able to not pass secrets as plain text on the chart, it would be helpful to be able to create the secret apart from the main helm chart (e. g.: using sealed secrets and passing on the encrypted manifest)

I've created a PR that adds these features :)

Since I was unable to create replicas with Helm, I am trying to implement Azure agent through Docker image (ghcr.io/clemlesne/azure-pipelines-agent:ubi8-main).
I am failing to create replicas this way, since AZP_AGENT_NAME is required and each replica would then take same from ENV.

Would you be able to make it optional, so AZP_AGENT_NAME takes initialy Pod name so each replica would be agent for itself ?
Also, if this is bad idea beacuse of optimization, then you can ignore this request.
Thanks.

The Helm Chart Presently as the option to add additional pod annotations, however additional pod labels would be useful to enable workload id to access other Azure Resources.

Also the ability to add extra sidecar containers could be useful

See code at

If possible, it would be interesting in the Windows image to have BuildTools:

https://learn.microsoft.com/en-us/visualstudio/install/build-tools-container?view=vs-2022

The official task from Azure to publish test results fails. This happens because the task checks if dotnet is installed by running the command dotnet --version. This arg is not available in the dotnetcore package installed in the image of the runner.

/usr/bin/dotnet --version
The command could not be loaded, possibly because:
  * You intended to execute a .NET application:
      The application '--version' does not exist.
  * You intended to execute a .NET SDK command:
      No .NET SDKs were found.

Using version 5, in the linux environment I have problem.

chart requires kubeVersion: >=1.19 which is incompatible with Kubernetes v1.26.2-eks

Locally I made the correction by removing this requirement.

I was alternating in versions 15.2 and 5.0.

Version 5 (using the jammy image, I haven't tested other distributions) gives an error in :

rootlesskit buildkitd --oci-worker-no-process-sandbox --addr $BUILDKIT_HOST &

and consequently :

buildctl build ...

To resolve, add the value of the variable:

export BUILDKIT_HOST=unix:///run/app-root/0/buildkit/buildkitd.sock

With this change version 5 works

Issue:
Building Azure Agent with helm on openshift (https://artifacthub.io/packages/helm/azure-pipelines-agent/azure-pipelines-agent) causes issue on pod. It requires licence to be approved upon configuration, which is not possible since pod is being unmanaged.

Adding following configuration into template should resolve this issue:
--unattended - agent setup will not prompt for information, and all settings must be provided on the command line

Thank you

EKS 1.26
Image: Jammy

I did a quick test and got the following error (folder ***) :

time="2023-05-11T12:27:57Z" level=warning msg="/proc/sys/ *** /max_ *** _namespaces needs to be set to non-zero."
error: failed to list workers: Unavailable: connection error: desc = "transport: error while dialing: dial unix /run/ *** /0/buildkit/buildkitd.sock: connect: no such file or directory"

Hi!

There is the possibility to include customization for flags:

Keda (https://keda.sh/docs/2.10/concepts/scaling-jobs/#scaledjob-spec)

• successfulJobsHistoryLimit
• failedJobsHistoryLimit

Job (into ScaledJob) https://kubernetes.io/docs/concepts/workloads/controllers/job/#ttl-mechanism-for-finished-jobs

• ttlSecondsAfterFinished

The intention is to have greater flexibility on the completion of jobs/scaleJobs and greater control related to the volumes created in cloud.

Tks!