Panic with ServerSideDiff=true (but without IncludeMutationWebhook=true) about argo-cd

[sherif-fanous]

I've run into the exact same situation today

{
    "Version": "v2.10.8+37b1cf5",
    "BuildDate": "2024-04-26T13:20:04Z",
    "GitCommit": "37b1cf5306f9c245f188c4c0566c23a0f80cdc65",
    "GitTreeState": "clean",
    "GoVersion": "go1.21.3",
    "Compiler": "gc",
    "Platform": "linux/amd64",
    "KustomizeVersion": "v5.2.1 2023-10-19T20:13:51Z",
    "HelmVersion": "v3.14.3+gf03cc04",
    "KubectlVersion": "v0.26.11",
    "JsonnetVersion": "v0.20.0"
}

Set http://argocd.argoproj.io/compare-options: ServerSideDiff=true annotation on an app and this results in the following

Recovered from panic: runtime error: invalid memory address or nil pointer dereference
goroutine 206 [running]:
runtime/debug.Stack()
	/usr/local/go/src/runtime/debug/stack.go:24 +0x5e
github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).processAppRefreshQueueItem.func1()n	/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:1470 +0x54
panic({0x3658d60?, 0x7189d60?})
	/usr/local/go/src/runtime/panic.go:920 +0x270
k8s.io/apimachinery/pkg/util/managedfields.(*GvkParser).Type(...)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/managedfields/gvkparser.go:43
githu
b.com/argoproj/gitops-engine/pkg/diff.removeWebhookMutation(0xc01c6acd70, 0xc01b7ae850, 0x0, {0x50b4020?, 0x3d7cb48?})
	/go/pkg/mod/github.com/argoproj/[email protected]/pkg/diff/diff.go:210 +0xfd
github.com/argoproj/gitops-engine/pkg/diff.serverSideDiff(0xc01b7ae7f0, 0xc01b7ae85
0, {0xc006fc18a8, 0x8, 0x8})
	/go/pkg/mod/github.com/argoproj/[email protected]/pkg/diff/diff.go:176 +0x2bd
github.com/argoproj/gitops-engine/pkg/diff.ServerSideDiff(0xc01b7ae850?, 0xc003a718a8?, {0xc003a718a8?, 0x8?, 0xc0075e0d70?})
	/go/pkg/mod/github.com/argoproj/gitops-engi
[email protected]/pkg/diff/diff.go:138 +0x2a
github.com/argoproj/gitops-engine/pkg/diff.Diff(0xc01b7ae178, 0xc01b7ae180, {0xc003a718a8, 0x8, 0x8})
	/go/pkg/mod/github.com/argoproj/[email protected]/pkg/diff/diff.go:88 +0x1d6
github.com/argoproj/gitops-engin
e/pkg/diff.DiffArray({0xc00dbf8600, 0x29, 0x7fb710832a68?}, {0xc00dbf8400, 0x29?, 0xc01a3ce4e0?}, {0xc006fc18a8, 0x8, 0x8})
	/go/pkg/mod/github.com/argoproj/[email protected]/pkg/diff/diff.go:814 +0x132
github.com/argoproj/argo-cd/v2/util/argo/diff.StateDiffs({0xc011890580?, 0xc0
22c63460?, 0x1e?}, {0xc00dbf8200?, 0xc00c588380?, 0xe?}, {0x50c90c0, 0xc01b343680?})
	/go/src/github.com/argoproj/argo-cd/util/argo/diff/diff.go:318 +0x74b
github.com/argoproj/argo-cd/v2/controller.(*appStateManager).CompareAppState(0xc00075e8c0, 0xc00ca36800, 0xc012bf2240, {0xc00c2ca500, 0x5, 0x8}, {0xc019ec400
0?, 0x5, 0x8}, 0x1, ...)
	/go/src/github.com/argoproj/argo-cd/controller/state.go:651 +0x3e1f
github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).processAppRefreshQueueItem(0xc0007da700)
	/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:1581 +0x1223
github.com/argoproj/argo-cd/v2
/controller.(*ApplicationController).Run.func3()
	/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:819 +0x25
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:157 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x0?,
 {0x5071dc0, 0xc000dc6240}, 0x1, 0xc000082d20)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:158 +0xaf
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0?, 0x3b9aca00, 0x0, 0x0?, 0x0?)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:135 +0x7f
k8s.io/apimachinery/pkg/util/wait.
Until(0x0?, 0x0?, 0x0?)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:92 +0x1e
created by github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).Run in goroutine 99
	/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:818 +0x7ce

Add IncludeMutationWebhook=true to the annotation and the app successfully syncs

[sherif-fanous]

Set http://argocd.argoproj.io/compare-options: ServerSideDiff=true annotation on an app and this results in the following

@sherif-fanous not sure if this is just a typo in your message but the correct annotation is: argocd.argoproj.io/compare-options: ServerSideDiff=true and not http://argocd.argoproj.io/compare-options: ServerSideDiff=true

Please confirm.

Sorry that was a typo. I used the correct annotation

[sherif-fanous]

@leoluz I'm using Talos Linux 1.7 running Kubernetes 1.30.0. This is a home lab cluster.

[jackmaninov]

Ok, small world...

[leoluz]

@blakepettersson Can you please take a look at argoproj/gitops-engine#574
It addresses the panic issue. However the reason why the ParseableType can not resolve the GVK is still unknown.

[leoluz]

@jackmaninov I just tested the kube-prometheus-stack helm chart applying the following application in my local kubernetes cluster:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: monitoring
  namespace: argocd
  annotations:
    argocd.argoproj.io/compare-options: ServerSideDiff=true
spec:
  project: default
  source:
    chart: kube-prometheus-stack
    repoURL: https://prometheus-community.github.io/helm-charts
    targetRevision: 51.2.0
    helm:
      valuesObject:
        kubeStateMetrics:
          enabled: false
        nodeExporter:
          enabled: false
        grafana:
          enabled: false
        alertmanager:
          enabled: false
        serviceMonitor:
          enabled: true
          prometheusSpec:
            serviceMonitorSelectorNilUsesHelmValues: false
  destination:
    namespace: monitoring
    server: https://kubernetes.default.svc
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - ServerSideApply=true
      - CreateNamespace=true

I wasn't able to reproduce the error. The application is synced as expected and no errors are returned in the UI or in the logs. I tested with Argo CD master build. Will run another test with 2.10.8 and confirm.

Can you please validate with the application above? It would be great if you could provide a minimal Application yaml that I can apply locally to try to reproduce.

[leoluz]

Set http://argocd.argoproj.io/compare-options: ServerSideDiff=true annotation on an app and this results in the following

@sherif-fanous not sure if this is just a typo in your message but the correct annotation is:
argocd.argoproj.io/compare-options: ServerSideDiff=true
and not
http://argocd.argoproj.io/compare-options: ServerSideDiff=true

Please confirm.

[jackmaninov]

Can you please validate with the application above? It would be great if you could provide a minimal Application yaml that I can apply locally to try to reproduce.

Yes, that application causes the panic as well. The only modification I made was in argocd's namespace (not the destination namespace), which is instead something like namespace: prod-argocd (which I don't believe should matter, but you never know).

Recovered from panic: runtime error: invalid memory address or nil pointer dereference
goroutine 264 [running]:
runtime/debug.Stack()
	/usr/local/go/src/runtime/debug/stack.go:24 +0x5e
github.com/argoproj/argo-cd/v2/controller.(
*ApplicationController).processAppRefreshQueueItem.func1()
	/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:1470 +0x54
p
anic({0x3658d60?, 0x7189d60?})
	/usr/local/go/src/runtime/panic.go:920 +0x270
k8s.io/apimachinery/pkg/util/managedfields.(*GvkParser).Ty
pe(...)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/managedfields/gvkparser.go:43
github.com/argoproj/gitops-engine/pkg/diff.remo
veWebhookMutation(0xc001d7b090, 0xc005680638, 0x0, {0x50b4020?, 0x3d7cb48?})
	/go/pkg/mod/github.com/argoproj/[email protected]
0122213038-792124280fcc/pkg/diff/diff.go:210 +0xfd
github.com/argoproj/gitops-engine/pkg/diff.serverSideDiff(0xc005680550, 0xc005680638, {
0xc0014d38a8, 0x8, 0x8})
	/go/pkg/mod/github.com/argoproj/[email protected]/pkg/diff/diff.go:176 +0x2bd\
ngithub.com/argoproj/gitops-engine/pkg/diff.ServerSideDiff(0xc005680638?, 0xc0014d38a8?, {0xc0014d38a8?, 0x8?, 0xc00e508eb0?})
	/go/pkg/m
od/github.com/argoproj/[email protected]/pkg/diff/diff.go:138 +0x2a
github.com/argoproj/gitops-engine/pkg
/diff.Diff(0xc00d405ec0, 0xc004f320e8, {0xc0014d38a8, 0x8, 0x8})
	/go/pkg/mod/github.com/argoproj/[email protected]
92124280fcc/pkg/diff/diff.go:88 +0x1d6
github.com/argoproj/gitops-engine/pkg/diff.DiffArray({0xc000600c00, 0x41, 0x7ffa57cadf18?}, {0xc000
600800, 0x41?, 0xc00902cea0?}, {0xc0014d38a8, 0x8, 0x8})
	/go/pkg/mod/github.com/argoproj/[email protected]
fcc/pkg/diff/diff.go:814 +0x132
github.com/argoproj/argo-cd/v2/util/argo/diff.StateDiffs({0xc00875b8c0?, 0xc01005a840?, 0x1e?}, {0xc019003
800?, 0xc00c862b60?, 0xf?}, {0x50c90c0, 0xc01134e9c0?})
	/go/src/github.com/argoproj/argo-cd/util/argo/diff/diff.go:318 +0x74b
github.co
m/argoproj/argo-cd/v2/controller.(*appStateManager).CompareAppState(0xc00062a8c0, 0xc00cfc3000, 0xc00a4b0000, {0xc0078076c0, 0x1, 0x1}, {0x
c00c862af0?, 0x1, 0x1}, 0x0, ...)
	/go/src/github.com/argoproj/argo-cd/controller/state.go:651 +0x3e1f
github.com/argoproj/argo-cd/v2/co
ntroller.(*ApplicationController).processAppRefreshQueueItem(0xc000bac380)
	/go/src/github.com/argoproj/argo-cd/controller/appcontroller.
go:1581 +0x1223
github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).Run.func3()
	/go/src/github.com/argoproj/argo-cd/cont
roller/appcontroller.go:819 +0x25
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
	/go/pkg/mod/k8s.io/[email protected]/
pkg/util/wait/wait.go:157 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x0?, {0x5071dc0, 0xc000ab4a80}, 0x1, 0xc000e4e660)
	/go/
pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:158 +0xaf
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0?, 0x3b9aca00, 0x0, 
0x0?, 0x0?)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:135 +0x7f
k8s.io/apimachinery/pkg/util/wait.Until(0x0?, 0x0?
, 0x0?)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:92 +0x1e
created by github.com/argoproj/argo-cd/v2/controller.(*
ApplicationController).Run in goroutine 127
	/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:818 +0x7ce

[leoluz]

@jackmaninov where are you testing this? Is this running in a local Kubernetes cluster? Which Kubernetes version are you running?

[jackmaninov]

@jackmaninov where are you testing this? Is this running in a local Kubernetes cluster? Which Kubernetes version are you running?

This is in my (remote) development environment, running under Talos Linux v1.7.0, k8s version v1.30.0.

Might be a bit too bleeding edge, but since @sherif-fanous is seeing this as well maybe it's not so k8s version specific.

[jackmaninov]

After a bit more digging, I find that there's a failure to call a webhook repeatedly while the application is in the bad state (paniced and stuck refreshing), so the panic could be in reaction to that:

From my kube-apiserver log, this repeats rapidly:

E0503 14:48:24.681458 1 dispatcher.go:213] failed calling webhook "prometheusrulemutate.monitoring.coreos.com": failed to call webhook: Post "https://monitoring-kube-prome-operator.monitoring.svc:443/admission-prometheusrules/validate?timeout=10s": tls: failed to verify certificate: x509: certificate signed by unknown authority

I'll try to track down what's causing the bad certificate and see if it improves things.

[leoluz]

@sherif-fanous can you please confirm in which environment you are running your tests and what is the Kubernetes server version?

[jackmaninov]

And, yes, I'm (now) aware argo-cd is only tested up to k8s v1.28 as per: https://argo-cd.readthedocs.io/en/stable/operator-manual/tested-kubernetes-versions/

[leoluz]

I was having a TLS error (invalid cipher suite specified) during deployment and I had to apply this patch (#17569) on top of Argo CD 2.10.9 to make it work.

I tested the patched version against Kubernetes 1.30 and wasn't able to reproduce the issue. I applied the prometheus-stack helm and it is working as expected.

@jackmaninov @sherif-fanous Can you please try to deploy Argo CD built from master branch and let me know if it addresses the issue?

[blakepettersson]

I got a similar panic with 2.10.7. I downgraded to 2.9.12, and did another sync and the issue went away. Maddeningly enough, after upgrading back to 2.10.7, deleting the application and trying again, I could not reproduce this issue.

github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).processAppRefreshQueueItem.func1()\n\t/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:1466 +0x50\npanic({0x2f89740?, 0x6a4bdc0?})\n
\t/usr/local/go/src/runtime/panic.go:920 +0x26c\ngithub.com/argoproj/gitops-engine/pkg/diff.structuredMergeDiff(0x4000bb54c0)\n\t/go/pkg/mod/github.com/argoproj/[email protected]/pk
g/diff/diff.go:321 +0x9c\ngithub.com/argoproj/gitops-engine/pkg/diff.StructuredMergeDiff(0x40015afe90?, 0x4000bb5858?, 0x8?, {0x36ad003?, 0x49e6390?})\n\t/go/pkg/mod/github.com/argoproj/[email protected]
122213038-792124280fcc/pkg/diff/diff.go:301 +0x4c\ngithub.com/argoproj/gitops-engine/pkg/diff.Diff(0x40015af670, 0x40015af678, {0x4000bb5858, 0x8, 0x8})\n\t/go/pkg/mod/github.com/argoproj/[email protected]
40122213038-792124280fcc/pkg/diff/diff.go:110 +0x234\ngithub.com/argoproj/gitops-engine/pkg/diff.DiffArray({0x400827dc00, 0x7e, 0x4000bb57d8?}, {0x400827d800, 0x7e?, 0xa?}, {0x4000bb5858, 0x8, 0x8})\n\t/go/pkg/mod/
github.com/argoproj/[email protected]/pkg/diff/diff.go:814 +0x11c\ngithub.com/argoproj/argo-cd/v2/util/argo/diff.StateDiffs({0x400b90e800?, 0x40011992e0?, 0x36ffd83?}, {0x400827c800
?, 0x4000f76b60?, 0x2?}, {0x49fa270, 0x4001ef7ad0?})\n\t/go/src/github.com/argoproj/argo-cd/util/argo/diff/diff.go:310 +0x5a8\ngithub.com/argoproj/argo-cd/v2/controller.(*appStateManager).CompareAppState(0x4000f777
a0, 0x400170ec00, 0x4007e40240, {0x4007b7e5c0, 0x2, 0x2}, {0x4000f76b60?, 0x2, 0x2}, 0x0?, ...)\n\t/go/src/github.com/argoproj/argo-cd/controller/state.go:649 +0x2e24\ngithub.com/argoproj/argo-cd/v2/controller.(*Ap
plicationController).processAppRefreshQueueItem(0x4000a74700)\n\t/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:1577 +0xe44\ngithub.com/argoproj/argo-cd/v2/controller.(*ApplicationController).Run.f
unc3()\n\t/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:815 +0x2c\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x0?)\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:157
+0x40\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x0?, {0x49a3f00, 0x400162c900}, 0x1, 0x40009a5980)\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:158 +0x90\nk8s.io/apimachinery/pkg/util/wa
it.JitterUntil(0x0?, 0x3b9aca00, 0x0, 0x0?, 0x0?)\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:135 +0x80\nk8s.io/apimachinery/pkg/util/wait.Until(0x0?, 0x0?, 0x0?)\n\t/go/pkg/mod/k8s.io/apimach
[email protected]/pkg/util/wait/wait.go:92 +0x28\ncreated by github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).Run in goroutine 114\n\t/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:8
14 +0x5f0\n","time":"2024-05-08T11:44:38Z"}

The appset used:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: kube-prometheus-stack
  namespace: argocd
spec:
  goTemplate: true
  generators:
    - list:
        elements:
          - name: blake
            chartRevision: 55.1.0
            valuesRevision: main
  template:
    metadata:
      name: '{{.name}}-kube-prometheus-stack'
    spec:
      project: default
      sources:
        - chart: kube-prometheus-stack
          repoURL: https://prometheus-community.github.io/helm-charts
          targetRevision: '{{.chartRevision}}'
          helm:
            releaseName: kube-prometheus-stack
            valueFiles:
              - $values/values.yaml
        - repoURL: https://github.com/blakepettersson/argocd-kube-prometheus-stack
          targetRevision: '{{.valuesRevision}}'
          ref: values
      destination:
        namespace: monitoring
        name: '{{.name}}'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true
          - ServerSideApply=true

kubectl (I know I need to bump my k3s, it's way too old 😄)

kubectl version
Client Version: v1.29.4
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.23.5+k3s1
WARNING: version difference between client (1.29) and server (1.23) exceeds the supported minor version skew of +/-1

Working Argo CD version:

argocd version
argocd-server: v2.9.12+487dce5.dirty
  BuildDate: 2024-04-19T03:54:04Z
  GitCommit: 487dce58bfa228a4f8bc22c626d1920f1617c066
  GitTreeState: dirty
  GoVersion: go1.21.3
  Compiler: gc
  Platform: linux/amd64
  Kustomize Version: v5.2.1 2023-10-19T20:13:51Z
  Helm Version: v3.13.2+g2a2fb3b
  Kubectl Version: v0.24.17

Non-working Argo CD version:

argocd-server: v2.10.7+96c0c41.dirty
  BuildDate: 2024-04-19T03:53:43Z
  GitCommit: 96c0c41b8948b0738567bd5c748a3de0cec5d19e
  GitTreeState: dirty
  GoVersion: go1.21.3
  Compiler: gc
  Platform: linux/amd64
  Kustomize Version: v5.2.1 2023-10-19T20:13:51Z
  Helm Version: v3.14.3+gf03cc04
  Kubectl Version: v0.26.11
  Jsonnet Version: v0.20.0

[sherif-fanous]

@jackmaninov @sherif-fanous Can you please try to deploy Argo CD built from master branch and let me know if it addresses the issue?

How do we do that? Set image tag to latest?

[jackmaninov]

@jackmaninov @sherif-fanous Can you please try to deploy Argo CD built from master branch and let me know if it addresses the issue?

I managed this today and running master did not resolve the issue.

I did however try deploying the test application in a freshly set-up local Talos cluster, and the issue did not present itself using argo-cd v2.11.0, so it is something more complicated going on with my setup. I suspect the failing admission webhook error I noted above, which I have still not tracked down.

EDIT: @sherif-fanous: deploying master involves pulling the git repo, running docker build, uploading the image to a repo somewhere, setting global.image.registry and global.image.tag in the helm values (in my case I installed with helm), then patching in an imagePullSecret into each of argo's ServiceAccounts.

[sherif-fanous]

@leoluz Can this issue please be re-opened?

I built master and still getting the panic

Recovered from panic: runtime error: invalid memory address or nil pointer dereference
goroutine 236 [running]:
runtime/debug.Stack()
        /usr/local/go/src/runtime/debug/stack.go:24 +0x5e
github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).processAppRefreshQueueItem.func1()
        /go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:1485 +0x54
panic({0x37ee8c0?, 0x76c5910?})
        /usr/local/go/src/runtime/panic.go:770 +0x132
k8s.io/apimachinery/pkg/util/managedfields.(*GvkParser).Type(...)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/managedfields/gvkparser.go:43
github.com/argoproj/gitops-engine/pkg/diff.removeWebhookMutation(0xc00c5b6530, 0xc00c5b6250, 0x0, {0x552c700?, 0x3f1dff0?})
        /go/pkg/mod/github.com/argoproj/[email protected]/pkg/diff/diff.go:210 +0xfd
github.com/argoproj/gitops-engine/pkg/diff.serverSideDiff(0xc00c5b61e0, 0xc00c5b6250, {0xc00dc5f898, 0x8, 0x8})
        /go/pkg/mod/github.com/argoproj/[email protected]/pkg/diff/diff.go:176 +0x2bd
github.com/argoproj/gitops-engine/pkg/diff.ServerSideDiff(0xc00c5b6250?, 0xc00dc5f898?, {0xc00dc5f898?, 0x8?, 0xc00de232c0?})
        /go/pkg/mod/github.com/argoproj/[email protected]/pkg/diff/diff.go:138 +0x2a
github.com/argoproj/gitops-engine/pkg/diff.Diff(0xc00cece000, 0xc00cece008, {0xc00dc5f898, 0x8, 0x8})
        /go/pkg/mod/github.com/argoproj/[email protected]/pkg/diff/diff.go:88 +0x1cf
github.com/argoproj/gitops-engine/pkg/diff.DiffArray({0xc00bdcc800, 0x33, 0x7?}, {0xc00bdcc600, 0x33?, 0xc00a629f80?}, {0xc00dc5f898, 0x8, 0x8})
        /go/pkg/mod/github.com/argoproj/[email protected]/pkg/diff/diff.go:821 +0x132
github.com/argoproj/argo-cd/v2/util/argo/diff.StateDiffs({0xc004ab71e0?, 0x3c5ae00?, 0x1e?}, {0xc00f386800?, 0xc00c6fe7e0?, 0x7?}, {0x5542f20, 0xc00932f680})
        /go/src/github.com/argoproj/argo-cd/util/argo/diff/diff.go:318 +0x74e
github.com/argoproj/argo-cd/v2/controller.(*appStateManager).CompareAppState(0xc0005f8e00, 0xc00737ec08, 0xc00a205448, {0xc00dc1db00, 0x4, 0x4}, {0xc00dc54e00, 0x4, 0x4}, 0x0, ...)
        /go/src/github.com/argoproj/argo-cd/controller/state.go:684 +0x3d3f
github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).processAppRefreshQueueItem(0xc000789880)
        /go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:1595 +0x1183
github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).Run.func3()
        /go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:833 +0x25
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000de4df0, {0x54e88e0, 0xc00119f1d0}, 0x1, 0xc000dd8de0)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000de4df0, 0x3b9aca00, 0x0, 0x1, 0xc000dd8de0)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:204 +0x7f
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:161
created by github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).Run in goroutine 125
        /go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:832 +0x865

I've not used Go in a long time but looking at the trace my understanding is that the panic is resulting from /go/pkg/mod/k8s.io/[email protected]/pkg/util/managedfields/gvkparser.go:43 which is the first statement in the following method.

func (p *GvkParser) Type(gvk schema.GroupVersionKind) *typed.ParseableType {
	typeName, ok := p.gvks[gvk]
	if !ok {
		return nil
	}
	t := p.parser.Type(typeName)
	return &t
}

This would indicate to me that either p or p.gvks are nil