2.12.x throws could not read Username for 'https://gitlab.com' from the UI about argo-cd

[argo1984]

Hi together,

i have the same issue with 2.12.X and it seems that the login to the repository works (it is green) but the Projects and Applications are not able to access the credentials.

[mozemke]

hi there,

we're facing the same problems and I read through #19587 IMO we should not be affected with our setup but are still getting the listed error.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: project-1-appset
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - git:
        repoURL: https://gitlab.xxxxx/project-1/deployment.git
        revision: dev
        directories:
        - path: '*/overlays/*'
        - path: '*/overlays/prod*'
          exclude: true
        - path: '*/overlays/xx*'
          exclude: true
        values:
          appName: '{{index .path.segments 0}}-{{.path.basenameNormalized}}'
  template:
    metadata:
      name: '{{.values.appName}}'
      labels:
        env: '{{.path.basenameNormalized}}'
    spec:
      project: < project-1 >
      source:
        repoURL: https://gitlab.xxxxx/project-1/deployment.git
        targetRevision: dev
        path: '{{.path.path}}'
      destination:
        server: https://kubernetes.default.svc
        namespace: 'xx-{{.values.appName}}'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true
        managedNamespaceMetadata:
          labels:
            app.kubernetes.io/instance: '{{.values.appName}}'
          annotations: 
            field.cattle.io/projectId: xx
            argocd.argoproj.io/tracking-id: xx
---
# add creds for appset in admin project
apiVersion: v1
kind: Secret
metadata:
  name: xxx-admin-repo-creds
  labels:
    argocd.argoproj.io/secret-type: repository
stringData:
  type: git
  url: https://gitlab.xxxxx/project-1/deployment.git
  username: xxxx
  password: xxxx
  project: < admin-project >
---
# add creds for project-1 apps
apiVersion: v1
kind: Secret
metadata:
  name: xxx-xxx-repo-creds
  labels:
    argocd.argoproj.io/secret-type: repository
stringData:
  type: git
  url: https://gitlab.xxxxx/project-1/deployment.git
  username: xxxx
  password: xxxx
  project: < project-1 >

the appset of project-1 is deployed to the admin-project, that's why we are having the same secret two times, but with different project scope, so that the admin project is able to generate the apps. If now the new way to match repo creds to apps is using project + url - I don't see why our setup is failing :(

The apps themselves have no connection problems.

error generating params from git: error getting directories from repo: error retrieving Git Directories: rpc error: code = Internal desc = unable to checkout git repo https://gitlab.xxxxx/project-1/deployment.git with revision e32c9a160962137771b976eff8c82c0c28bad41a: failed to initialize repository resources: rpc error: code = Internal desc = Failed to fetch default: `git fetch origin --tags --force --prune` failed exit status 128: fatal: could not read Username for 'https://gitlab.xxxx': terminal prompts disabled

We are facing this problem with all our Appsets, above is just one example.

[blakepettersson]

I suspect it's the same thing actually :( An applicationset doesn't have a project; it's admin-scoped. I believe the resolution is exactly the same as for #19587 - having a repo-cred that doesn't have a project set.

[mozemke]

yep that did indeed fix it. I now understand what you mean :D

We actually deploy our appsets, with one bootstrapping app, that one has set a project. But since AppSets do not have the key, it all makes sense now! :)

The downside now is, that the repo creds can now be used by all projects.

thank you! :)

[blakepettersson]

The downside now is, that the repo creds can now be used by all projects.

Yes, that is true; that was however the same behavior in < 2.12. A workaround for that would be to disallow that source to be used in your AppProjects:

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: < project-1 >
spec:
  sourceRepos:
    - '!https://gitlab.xxxxx/project-1/deployment.git'

[mozemke]

that was however the same behavior in < 2.12.

I actually never knew that before I read through the issue/pr to enhance the situation :D
We'll implement your workaround! :)

thanks! :)

[ChristianCiach]

@blakepettersson Yes, downgrading immediately fixes all apps, and upgrading immediately triggers the issue again. Weird..

[ChristianCiach]

It is not. It is used for my bootstrap application (which belongs to project default) to set up the other AppProjects and Applications.

Unsurprisingly, deleting this other secret breaks the bootstrap app, but doesn't fix all the other apps.

[ChristianCiach]

Weird... The error message disappears from the UI as soon as I sync the application, even if there are no changes. I need to do this once for each application, even though they are all synced from the same git repository. But the error message is gone for good after syncing each application once. (We don't use auto-sync in this cluster)

[blakepettersson]

@c-buisson and @argo1984 can you give the specs of the applications which are failing, and the (redacted) specs of the repo credentials that are expected to work with these applications? I suspect shenanigans from #18388.

[crenshaw-dev]

+1 to Blake's question, but to anyone who's experiencing this issue. :-)

[ChristianCiach]

I am confused. I am seeing this error after upgrading from v2.11.7 to v.2.12.2, but after reading this and the linked issue, I don't think I should be affected, because my Secret doesn't have a project set:

apiVersion: v1
kind: Secret
metadata:
  labels:
    argocd.argoproj.io/secret-type: repository
  name: repocred-gitoops
  namespace: argocd
type: Opaque
data:
  password: base64(somePassword)
  url: base64(https://internal-url/some-group/gitoops.git)
  username: base64(gitlab+deploy-token-62)

---
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  annotations:
    argocd.argoproj.io/tracking-id: bootstrap:argoproj.io/AppProject:argocd/egs-external
  finalizers:
  - resources-finalizer.argocd.argoproj.io
  labels:
    argocd.argoproj.io/instance: bootstrap
  name: egs-external
  namespace: argocd
spec:
  clusterResourceWhitelist:
  - group: '*'
    kind: '*'
  description: EGS External Cluster
  destinations:
  - name: egs-external
    namespace: '*'
  sourceRepos:
  - '*'

---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  finalizers:
  - resources-finalizer.argocd.argoproj.io
  annotations:
    argocd.argoproj.io/tracking-id: bootstrap:argoproj.io/Application:argocd/netshoot
  labels:
    argocd.argoproj.io/instance: bootstrap
  name: netshoot
  namespace: argocd
spec:
  destination:
    name: egs-external
    namespace: default
  project: egs-external
  source:
    path: netshoot
    repoURL: https://internal-url/some-group/gitoops.git
    targetRevision: HEAD

I've tried to debug this for hours, but I am none the wiser.

[blakepettersson]

@ChristianCiach from what I can see you should indeed be unaffected - does rolling back to 2.11.x solve your issue?

[blakepettersson]

@ChristianCiach are you sure that there are no other repo-creds making use of the same repo url?

[ChristianCiach]

@blakepettersson This is a small test cluster. There is a single other repo-cred (not technically, they are both argocd.argoproj.io/secret-type: repository, not repocreds), but it is referencing another project on the same gitlab server.

[blakepettersson]

That other argocd.argoproj.io/secret-type: repository secret - is it referring to the same repo url?

[blakepettersson]

@ChristianCiach are you around on the CNCF slack?

[ChristianCiach]

@blakepettersson Not yet, but I could change that :) I was just about to leave for vacation for the next three weeks, but this is too interesting to let go. Give me a few minutes to figure out how to join the slack channel.

Btw, the crazy thing is that I can sync the applications just fine! Argo CD picks up the changes from the repo without issues. It's just the UI that shows this error message all the time...

[ChristianCiach]

This may or may not have something to do with the fact that we moved the gitlab project from one group to another group (on the same server) last week. But we did this way before upgrading Argo CD to v2.12, including updating all Applications and Secrets to the new repo url. But maybe the old repo url was still cached somewhere and only triggered this issue with v2.12...?

Anyway, I seem to be the only one affected by this, so this probably has something to do with dirty caches and the recently moved git repo, so we probably shouldn't think too hard about this. I probably should have started with this information, so I'm sorry about that!