samcontesse / gitlab-merge-request-resource Goto Github PK

HI @samcontesse , how can i pull you a merge request ?
i've made few changements to gitlab-merge-request-resource in my fork => here

1. add Drafts params into ressource, WIP is no more used in gitlab since few release.

2. add possibility to use private fork on merge request ,to do this i've made:

   • do not clone fork project , but recreate patch file from API in this case
   • when private fork is used, , API don't provide the possibility to update BuildState because it's relative to SourceCommit, in this case only, i create a note to the merge request with the

• Build URL
• Buid Name
• Build state
• gitlab Issue related

NB: i've started GO saturday, be gentle if you start reviewing my code ^^

Very often, esp when updating an existing MR, we see this error in the put:

selected worker: prd-concourse-worker-5f4f9f5ff7-6js4c
error running command: POST https://gitlab/api/v4/projects/208/statuses/cc7e5d138745dd28793b64bcabb28da395a38220: 400 {message: Cannot transition status via :run from :running (Reason(s): Status cannot transition via "run")}

Obviously some kind of race condition. Re-running the same pipeline with same params works. Any idea what we might be doing wrong?

I mean, we do have multiple jobs running simultaneously, i.e.:

How would you propose to address this other than making the jobs sequential?

Problem

The resource doesn't work for namespaced projects.

For a namespaced project, the following error message will be displayed:

resource script '/opt/resource/check []' failed: exit status 1

stderr:
error retrieving opened merge requests: GET https://git.example.com/api/v4/projects/path/to/my/project/merge_requests: 404 {message: 404 Project Not Found}

Solution

See gitlab documentation.

If using namespaced API calls, make sure that the NAMESPACE/PROJECT_NAME is URL-encoded.

For example, / is represented by %2F

Commenting on a MR with [trigger ci] must emit a version with the timestamp of the comment but it doesn't.

We are trying to use the resource gitlab-merge-request in the concourse pipeline, but seeing the error below

resource script '/opt/resource/check []' failed: exit status 2

stderr:
panic: runtime error: index out of range

goroutine 1 [running]:
github.com/samcontesse/gitlab-merge-request-resource.(*Source).GetProjectPath(0xc000058240, 0xc000090a00, 0x7)
/go/src/github.com/samcontesse/gitlab-merge-request-resource/models.go:43 +0x88
main.main()
/go/src/github.com/samcontesse/gitlab-merge-request-resource/check/cmd/main.go:29 +0x288

I would appreciate a lot if you have any suggestion where to grab merge request source branch name from in order to use in the task.

Cannot put a resource that have not successfully passed a get before. It fails with the following error :
error changing directory to /tmp/build/put/source: chdir /tmp/build/put/source: no such file or directory

A better message must be provided to the user.

Trigger pipelines only if changes have occurred in specific folders on the MR.

To support the above use case, request GitlabMR to expose a param to specify 'paths' and 'ignore_paths' similar to something GIT resource offers today.

For pre-commit pipelines we would be needing this.

Currently if I have two separate jobs in concourse, one for unit tests and another for integration tests, there is no way of getting Gitlab to link to both of these jobs. It only ever links to the latest one that sends a status to it.

  - name: mr-integ-tests
    max_in_flight: 10
    plan:
      # task gets the source code from the repository
      - in_parallel:
          - get: source-code
            resource: merge-request
            passed: [ "mr-unit-tests" ]
            trigger: true
            on_success:
              put: merge-request
              params:
                repository: source-code
                status: running

Could we add stage ? So then the pipelines are built in gitlab based on the data we send it from the merge request pipeline.

Example:

Could be something like:

  - name: build
    max_in_flight: 10
    plan:
      # task gets the source code from the repository
      - in_parallel:
          - get: source-code
            resource: merge-request
            trigger: true
            on_success:
              put: merge-request
              params:
                repository: source-code
                status: running
                stage: build

  - name: test1
    max_in_flight: 10
    plan:
      # task gets the source code from the repository
      - in_parallel:
          - get: source-code
            resource: merge-request
            trigger: true
            on_success:
              put: merge-request
              params:
                repository: source-code
                status: running
                stage: test1

or something like that. as at the moment, we have many concurrent pipelines but there is no way to link them to a MR in Gitlab for each of the jobs that are run as part of that pipeline.

Hello,

We're trying to use this resource but it looks like it does not support a MR when source is a Fork.

Does anybody have the code to do this stashed away somewhere ?

Feature request

Currently, it is possible to set the merge-request resource to filter to only watch for MR's that are targetting a specific branch. However, for the use-case of updating only a single MR from a pipeline, this does not suffice, since sometimes it may be desireable to only track a specific MR.

A great example of this is with instanced/grouped pipelines:

If I create a bunch of pipelines using the git-branches resource, which are all based on a template, and the only difference between them is which branch they are tracking, it would be desireable to set the MR resource to only track the source branch for the purposes of updating the MR's status. This is because it is not desireable to then go and update a bunch of other MR's when the pipeline itself is only tracking a single branch. This can be jerry-rigged by using a task to look at the source_branch in the .git/merge-request.json of a get step and then only acting if the source branch matches, but it would be a great QoL improvement to have the resource itself do this.

Can a filter be added on source_branch in addition to target_branch?

Please let me know how to use check flag ,. I want to create new merge request everytime for merge request

I'm just trying to set up a PoC project to see how this resource works, and I've set up the resource as suggested in the readme. I'm getting the access token from a k8s secret.

The resource itself seems to work fine. It checks successfully and triggers the job. However, when the job actually starts (atm it's just a bare-bones task which prints hello world, nothing fancy), it fails with fatal: unable to access 'https://gitlab-ci-token:((redacted))@gitlab.company.com/path/to/repo.git/': The requested URL returned error: 403

the mr resource yaml is:

resources:
- name: merge-request-src
  icon: git
  type: merge-request
  source:
    uri: https://gitlab.company.com/path/to/repo.git
    private_token: ((gitlab.access_token))

2020/07/01 09:12:30 Set commit status failed: 403, response

target := request.Source.GetTargetURL()

I'm guessing the url points to the fork for some reason; works when I have permission to the fork.

The OCI image is currently using concourse/buildroot:git as it's base which hasn't been updated in over five years (and I have not found the repository of these images to make a PR).
This provides git version 2.12.3 which is, as of today, 121 version behind the current v2.36.1 (c.f.: https://github.com/git/git/tags / https://git-scm.com/downloads).

While far from being critical, it's always nice to not lag too much behind the mainstream product (and pulling your image from the Hub is quite convenient).
But this raise a few question on how would you like to address this topic (if you would).
Would you prefer to use an existing image as a base? I found:

• https://hub.docker.com/r/bitnami/git larger image (because it's Debian based) but the image tag is the git version, it help to be explicit
• https://hub.docker.com/r/rancher/gitjob smaller image but the image version does not reflect the git version
  Or shall we use Alpine as a base and select the version?

Should the tag published on Docker Hub also reflect the git version from now on?

Cheers.

When setting the status as running the put step returns

Set commit status failed: 400, response

Any idea why this is? The api access token have all the scopes.

pipeline:

resource_types:
- name: merge-request
  type: docker-image
  source:
    repository: ((mr-resource-uri))

resources:
  - name: enrolment-application-mr
    type: merge-request
    source:
      uri: ((enrolment-app-uri))
      branch: chore/ci
      private_token: ((enrolment-react-access-token))
      target_branch: dev
      skip_work_in_progress: true

jobs:
  - name: run-tests
    public: true
    plan:
      - get: enrolment-application-mr
        trigger: true
      - put: enrolment-application-mr
        params:
          repository: enrolment-application-mr
          status: running
      - task: run-unit-tests
        config:
          platform: linux
          inputs:
            - name: enrolment-application-mr
          image_resource:
            type: docker-image
            source:
              repository: ((devops-utils-image-uri))
          run:
            path: bash
            args:
            - -exc
            - |
              export SKIP_PREFLIGHT_CHECK=true
              pwd
              ls -la
              cd enrolment-application-mr
              npm i
              npx lerna bootstrap --hoist
              npm run test
        on_failure:
          put: enrolment-application-mr
          params:
            repository: enrolment-application-mr
            status: failed
            labels: ['unit-test', 'stage']
            comment:
              text: UNIT TESTS FAILED! Save the world, write tests!
        on_success:
          put: enrolment-application-mr
          params:
            repository: enrolment-application-mr
            status: success
            labels: ['unit-test', 'stage']
            comment:
              file: out/commt.txt
              text: |
                Add new comment.
                $FILE_CONTENT

If I've just blinked and missed the file or comments someplace, forgive me, but can you add an open source license? If you prefer a suggestion, an MIT or ISC license is always pretty flexible!

But I need to know before I fork this and give you a pull request...

As per Gitlab documentation, work_in_progress is deprecated so skip_work_in_progress is not working as expected, either add a new parameter skip_draft to support draft status or add the draft in the resource validation

Resource validation:

GitLab client draft:
https://github.com/xanzy/go-gitlab/blob/83608f75f2b1e695af24482791b78fe3e938d7d3/merge_requests.go#LL57C2-L57C7

If the source branch of a merge request cannot be merged to the target branch, the resource return with exit code 1 but do not update the commit status to failed.

I'm trying to follow the example code with an internally hosted gitlab instance.

Here is my pipeline definition (sanitized)

resource_types:
- name: merge-request
  type: docker-image
  source:
    repository: samcontesse/gitlab-merge-request-resource

resources:
- name: merge-request
  type: merge-request
  source:
    uri: https://gitlab.myco.int/terraform/proof-of-concepts/terraform-pipelines.git
    private_token: <private_token>

jobs:
- name: sample-merge-request
  plan:
  - get: merge-request
    trigger: true
  - put: merge-request
    params:
      repository: merge-request
      status: running
  - task: unit-test
    file: merge-request/ci/tasks/unit-test.yml
  on_failure:
    put: merge-request
    params:
      repository: merge-request
      status: failed
  on_success:
    put: merge-request
    params:
      repository: merge-request
      status: success
      labels: ['unit-test', 'stage']
      comment:
        file: out/commt.txt
        text: |
          Add new comment.
          $FILE_CONTENT

I'm seeing this error message:

error running command: Get "https://gitlab.myco.int/api/v4/projects/terraform%2Fproof-of-concepts%2Fterraform-pipelines/merge_requests?labels=&order_by=updated_at&sort=asc&source_branch=&state=opened&target_branch=": dial tcp: lookup gitlab.myco.int on 8.8.8.8:53: no such host

It seems like it is trying to resolve my internal url with Google's dns. Any ideas?

This is my resource configuration

resource_types:

• name: merge-request
  type: docker-image
  source:
  repository: samcontesse/gitlab-merge-request-resource
  tag: latest
  resources:
• name: merge-request
  type: merge-request
  source:
  uri: ((using an https repository url))
  private_token: ((personal access token of the user))
  jobs:
• name: build
  public: true
  plan:
  • get: ((another resource))
  • get: merge-request
    trigger: true
  • put: merge-request
    params:
    repository: merge-request
    status: running
  • task: build
    config:
    platform: linux
    image_resource:
    type: registry-image
    source: ((source for the image))
    inputs:
    - name: ((another resource))
    run: ((script))
    on_failure:
    put: merge-request
    params:
    repository: merge-request
    status: failed
    on_success:
    put: merge-request
    params:
    repository: merge-request
    status: success
    labels: ['compilation', 'stage']

While concourse is checking the resource merge-request it is giving the below error

run check step: run check step: check: resource script '/opt/resource/check []' failed: exit status 2

stderr:
panic: runtime error: index out of range [3] with length 0

goroutine 1 [running]:
github.com/samcontesse/gitlab-merge-request-resource.(*Source).GetProjectPath(0xc0000ba000, 0xc0000e0000, 0x2e)
/go/src/github.com/samcontesse/gitlab-merge-request-resource/models.go:45 +0xa0
main.main()
/go/src/github.com/samcontesse/gitlab-merge-request-resource/check/cmd/main.go:34 +0x37c

How should I solve this error and proceed ?

There is a parameter to skip resource to check ssl certificate, because we use a self sign-in certificate in our gitlab installation.
We build an image using your repository or if we import self sign-in din S.O. container during build go can understand a certificate and skip ssl validation doesn't matter.
Thanks for help

Above are four commits, yet hijacking into Concourse shows that the resource only has the latest from the original repo, not the forked one that has the commits coming in. Perhaps I implemented incorrectly?

Below is the hijacked container, running git log

If there's any other details I can provide to help, will do. Thanks for the help 👍!

When I click the following button

I am taking to the following URL:
https://concourse.url/teams/dev-team/pipelines/test-service/jobs/builds/put.

However, I have both unit test and integration test jobs that both ran. But for some reason the MR still seems to say the pipeline is pending. It's worth noting, we have quite a lot of problems when the amount of MR's open increase past 1. Now we have a dependency bot that opens MR's for us, there are more than 5 MR's open for my current repository and Gitlab MR's have a real tough time tracking the progress of them. Concourse itself runs the jobs with no issues, but how the Gitlab MR tracks the progress of this, it takes about 1 hour to resolve 5 MR's, because we constantly have to re-run them separately so there's only 1 running at one time. Which as you can imagine isn't ideal.

Since release v0.1.2, I get the following error during the check phase against GitLab 14.9.3 and Concourse 7.7.1 (workers version 2.4):

version is missing from previous step

Even more surprising, this error message only appear when running in a Concourse worker, executing the program locally gives me the exact same output with check version 0.1.1 and 0.1.3:

./dist/assets_linux_amd64/check <<EOF
{
  "source": {
    "uri": "https://gitlab.example.net/group/project.git",
    "sort": "asc",
    "private_token": "QlgyS8gVV0qUEKpbs1Hg"
  },
  "version": { "id": "36" }
}
EOF
[{"id":"36","updated_at":"2022-04-27T13:25:37+02:00"}]

fly check-resource did not help.

I did not found any obvious reason for this error to appear, but downgrading to v0.1.1 did fix.

If we have a build that has a MR notify Gitlab it's failed on it's pipeline and we sent a label to the Gitlab MR that it has failed. We get the failed label show up fine.

But when we re-run the pipeline with a fix, which then has the pipeline succeed, the labels that indicate that the MR has passed build are added to the MR, but the failed label still persists on the MR. Is there a way of overriding current specific labels? We don't want to completely delete them all because of the fact we have other labels on the MR indicating the type of MR it is. But a feature where we could reset specific labels when a PR re-runs would be great.

When using instanced pipelines, puting a merge-request resource and setting the status on gitlab works fine, however the URL on gitlab will be incorrect.

For example: The resource works fine and sets things in gitlab nicely so that the UI works:

However, when you click the link in the above image, it does not include the instance_vars in the URL, so you get a 404 in Concourse

The URL that gets set and gitlab takes you to is in the format:
teams/<team>/pipelines/<pipeline>/jobs/<job>/builds/<build>

However for instanced pipelines the format should be:
teams/<team>/pipelines/<pipeline>?vars.<instance_vars.key>=%22<instance_vars.value>instances-D%22

Hi 😃 ,

Since today I am having issue with pipelines triggered by merge requests.

Updating the pipeline description and enforcing tag v0.1.3 when declaring the gitlab merge request resource solves the issue.

Our Concourse pipelines run since two years and there was no configuration update so I assume that it could come from today's release.

hope this can be of any help.

Best,

Hi,
I'm running into an issue where pipelines are created but never seem to run. I haven't been able to reproduce this issue consistently. So far I've been able to cancel them via a curl script using the gitlab api in the hopes it will pass the resource check but that is only sometimes effective. Any ideas on what's causing this?

We have integration tests that launches an embedded PostgreSQL instance using the OpenTable provided https://github.com/opentable/otj-pg-embedded maven dependency

These are actual binaries

2019-11-07 05:06:42,767 INFO (main) [com.opentable.db.postgres.embedded.EmbeddedPostgres]  - Detected a Linux x86_64 system
2019-11-07 05:06:42,901 INFO (main) [com.opentable.db.postgres.embedded.EmbeddedPostgres]  - Extracting Postgres...
2019-11-07 05:06:44,585 INFO (main) [com.opentable.db.postgres.embedded.EmbeddedPostgres]  - Postgres binaries at ./embedded-pg/PG-785b618641f5eefce5b5079f2c9458dd

However running the binary produces a no such file or directory

bash: ./initdb: No such file or directory

There is no strace or other debugging tools on the container from this image but I suppose the image is likely lacking several dependencies for this to work.

Any suggestions ?

Can you update your go.mod file? Your docker image doesn't build.

Running the test suite fails on the main branch. The test fails at the In Suite with a nil pointer error:

Running Suite: In Suite - /Users/jedge/git/simspace/gitlab-merge-request-resource/pkg/in
========================================================================================
Random Seed: 1670432625

Will run 1 of 1 specs
------------------------------
•! [PANICKED] [0.003 seconds]
In
/Users/jedge/git/simspace/gitlab-merge-request-resource/pkg/in/command_test.go:20
  Run
  /Users/jedge/git/simspace/gitlab-merge-request-resource/pkg/in/command_test.go:49
    When it has a minimal valid configuration
    /Users/jedge/git/simspace/gitlab-merge-request-resource/pkg/in/command_test.go:87
      [It] Should clone repository
      /Users/jedge/git/simspace/gitlab-merge-request-resource/pkg/in/command_test.go:89

  Test Panicked
  In [It] at: /usr/local/Cellar/go/1.19.1/libexec/src/runtime/panic.go:260

  runtime error: invalid memory address or nil pointer dereference

  Full Stack Trace
    github.com/samcontesse/gitlab-merge-request-resource/pkg/in.(*Command).Run(0xc000202450, {_, _}, {{{0xc00020c120, 0x2c}, {0x14559e8, 0x1}, 0x0, 0x0, 0x0, ...}, ...})
    	/Users/jedge/git/simspace/gitlab-merge-request-resource/pkg/in/command.go:38 +0xe7
    github.com/samcontesse/gitlab-merge-request-resource/pkg/in_test.glob..func1.3.2.1()
    	/Users/jedge/git/simspace/gitlab-merge-request-resource/pkg/in/command_test.go:101 +0x138
------------------------------


Summarizing 1 Failure:
  [PANICKED!] In Run When it has a minimal valid configuration [It] Should clone repository
  /usr/local/Cellar/go/1.19.1/libexec/src/runtime/panic.go:260

Ran 1 of 1 Specs in 0.003 seconds
FAIL! -- 0 Passed | 1 Failed | 0 Pending | 0 Skipped
--- FAIL: TestIn (0.00s)
FAIL
FAIL	github.com/samcontesse/gitlab-merge-request-resource/pkg/in	0.292s
ok  	github.com/samcontesse/gitlab-merge-request-resource/pkg/out	(cached)
FAIL

Using the debugger, I walked through the steps and saw that it appears to be hitting a nil pointer trying to access user.Email here:

> github.com/samcontesse/gitlab-merge-request-resource/pkg/in.(*Command).Run() ./pkg/in/command.go:38 (PC: 0x147956d)
    33:		if err != nil {
    34:			return Response{}, err
    35:		}
    36:
    37:		user, _, err := command.client.Users.CurrentUser()
=>  38:		err = command.runner.Run("config", "--global", "user.email", user.Email)
    39:		if err != nil {
    40:			return Response{}, err
    41:		}
    42:
    43:		err = command.runner.Run("config", "--global", "user.name", user.Name)

I think this is probably related to this commit, adding a feature to set user name and email from the current user: #100

It looks like there needs to be data configured into the user field before testing in order for the CurrentUser method to work correctly.

During the get step, the file .git/merge-request-source-branch should contain the contents of the source_branch key in the .git/merge-request.json object, but it is currently just an empty file.

Hello Samuel,

I realized that when the repository is without MR the component is waiting for new MR to trigger job.
After the MR is created the component releases the pipeline
But when the MR is accepted and the repository has no MR open and a job is started the component update status on MR that has already been accepted.

I'm using

  - name: merge-request
    type: docker-image
    source:
      repository: samcontesse/gitlab-merge-request-resource
      # tag: feature_skip-ssl
      tag: bugfix_failed-merge-commit-status

But both tags occurs the same problem.
If you need more information let me know

Thanks for help

Gustavo

I want to use instance vars combined with pipeline groups. For example, a merge check pipeline that has multiple different merge checks running in parallel, grouped for organization, and instance vars for each merge check for separate branches. I want these merge checks to add their own individual commit status updates to the merge request, without needing to define multiple merge-request resources pointing to the same repository and overwriting BUILD_PIPELINE_NAME using the pipeline_name parameter. This gets very messy when you have 3+ merge checks. It would be much cleaner if there was an optional parameter on the put step that overwrote BUILD_PIPELINE_NAME.

- put: my-repository
  params:
    repository: my-repository
    name: my-commit-status-name
    status: running

Our repositories are usually private and require ssh keys.

I was only able to make the get: merge-request build step work after I made my repository public and used the https:// URL of my repository.

Can we please have support for git: urls that must use ssh keys? Sth like:

resources:
  - name: merge-request
    type: merge-request
    source:
      uri: [email protected]:foo/bar/myproject.git
      private_key: ((user.private_key))
      private_token: ((user.gitlab_personal_accesss_token))
      insecure: true

For the future:

Replace external call to ssh-agent with https://pkg.go.dev/golang.org/x/crypto/ssh/agent

In case the repository has references to Git Submodules and depends on it for steps of a Pipeline, it fails as this plugin doesn't load the submodules and doesn't expose a way to load it.

Refer to #21, which contains such feature addition but is not merged yet.