unauthorized: access to the requested resource is not authorized about deploy

[tpouyer]

@achuzhoy this is a hack but it will get you past the issue:

oc patch serviceaccount default -p '{"imagePullSecrets": [{"name": "multiclusterhub-operator-pull-secret"}]}'

[achuzhoy]

Was able to pass this step.

[tpouyer]

You were in the qe team on our quay org open-cluster-management. That team did not have read permissions for the multicloudhub-operator-bundle image nor the multicloudhub-operator-index image... I've updated the permissions for those repos.

You can test this from your command line by seeing if you can successfully pull these two images using:

docker login quay.io
docker pull quay.io/open-cluster-management/multicloudhub-operator-index:1.0.0-SNAPSHOT-2020-03-10-15-49-00
docker pull quay.io/open-cluster-management/multicloudhub-operator-bundle@sha256:ba919b34aa7c7c7135f4474791defb6240ff1af2491b876c25ba960af81c5267

If you can pull those images then you should be good to go now

[achuzhoy]

I'm able to pull the images upon login to quay.io.

Regenerated the key according to the guide to assure there's no error. Still fail on the same issue.

Normal Scheduled default-scheduler Successfully assigned open-cluster-management/3ed3478e5608f39e9a7f144fae67618bf4ed284b1561ed1bc28407ce858fqst to worker-1
Normal Pulled 23s kubelet, worker-1 Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1f1038e9a52b30deb7106c5ea6ed44f91493c0bdeffe32aebfd5eacc906550cb" already present on machine
Normal Created 22s kubelet, worker-1 Created container util
Normal Started 22s kubelet, worker-1 Started container util
Normal Pulling 22s kubelet, worker-1 Pulling image "quay.io/open-cluster-management/multicloudhub-operator-bundle@sha256:5b5e92445e9754fff43753d54813aca5c192b8f0e50dff84dc86f7beb1e58830"
Warning Failed 21s kubelet, worker-1 Failed to pull image "quay.io/open-cluster-management/multicloudhub-operator-bundle@sha256:5b5e92445e9754fff43753d54813aca5c192b8f0e50dff84dc86f7beb1e58830": rpc error: code = Unknown desc = Error reading manifest sha256:5b5e92445e9754fff43753d54813aca5c192b8f0e50dff84dc86f7beb1e58830 in quay.io/open-cluster-management/multicloudhub-operator-bundle: unauthorized: access to the requested resource is not authorized
Warning Failed 21s kubelet, worker-1 Error: ErrImagePull
Normal BackOff 21s kubelet, worker-1 Back-off pulling image "quay.io/open-cluster-management/multicloudhub-operator-bundle@sha256:5b5e92445e9754fff43753d54813aca5c192b8f0e50dff84dc86f7beb1e58830"
Warning Failed 21s kubelet, worker-1 Error: ImagePullBackOff

[berenss]

I also got this on a 4.4 build. I thought I was just doing dumb PM stuff. Probably still am.

[berenss]

multicloudhub-operator/kustomization.yaml using tag:

  newTag: 1.0.0-SNAPSHOT-2020-03-10-19-58-31

Failed to pull image "quay.io/open-cluster-management/multicloudhub-operator-bundle@sha256:5b5e92445e9754fff43753d54813aca5c192b8f0e50dff84dc86f7beb1e58830": rpc error: code = Unknown desc = Error reading manifest sha256:5b5e92445e9754fff43753d54813aca5c192b8f0e50dff84dc86f7beb1e58830 in quay.io/open-cluster-management/multicloudhub-operator-bundle: unauthorized: access to the requested resource is not authorized

[tpouyer]

So you're credentials are working... I can see where your user achuzhoy was able to pull the multicloudhub-operator-bundle via it's sha recently in the quay logs.

So there must be something wrong with the quay-secret.yaml file... When you followed the instructions to generate a pull secret from quay did you save the file as quay-secret.yaml in the multicloudhub-operator directory?

Did you update the metadata.name value in the quay-secret.yaml file to use the name quay-secret:

apiVersion: v1
kind: Secret
metadata:
  name: quay-secret
...

Is the mulitcloudhub-operator-registry deployment in a READY state? Can you share the output of:

oc get deployment open-cluster-management-registry -n open-cluster-management -o yaml

and what about open-cluster-management catalogsource? can you share the output of:

oc get catalogsource open-cluster-management -n open-cluster-management -o yaml

And can you share the content of your kustomization.yaml file in the mulitcloudhub-operator dir?

[tpouyer]

multicloudhub-operator/kustomization.yaml using tag:

newTag: 1.0.0-SNAPSHOT-2020-03-10-19-58-31

Failed to pull image "quay.io/open-cluster-management/multicloudhub-operator-bundle@sha256:5b5e92445e9754fff43753d54813aca5c192b8f0e50dff84dc86f7beb1e58830": rpc error: code = Unknown desc = Error reading manifest sha256:5b5e92445e9754fff43753d54813aca5c192b8f0e50dff84dc86f7beb1e58830 in quay.io/open-cluster-management/multicloudhub-operator-bundle: unauthorized: access to the requested resource is not authorized

@berenss I added you to the dev team in open-cluster-management org on quay... please try again.

[achuzhoy]

[kni@provisionhost-0 ~]$ oc get deployment open-cluster-management-registry -n open-cluster-management -o yaml
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "1"
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"app":"open-cluster-management-registry"},"name":"open-cluster-management-registry","namespace":"open-cluster-management"},"spec":{"selector":{"matchLabels":{"app":"open-cluster-management-registry"}},"template":{"metadata":{"labels":{"app":"open-cluster-management-registry"}},"spec":{"containers":[{"image":"quay.io/open-cluster-management/multicloudhub-operator-index:1.0.0-SNAPSHOT-2020-03-10-19-58-31","name":"multicloudhub-operator-index","ports":[{"containerPort":50051}]}],"imagePullSecrets":[{"name":"quay-secret"}]}}}}
creationTimestamp: "2020-03-11T03:29:45Z"
generation: 1
labels:
app: open-cluster-management-registry
name: open-cluster-management-registry
namespace: open-cluster-management
resourceVersion: "2138493"
selfLink: /apis/apps/v1/namespaces/open-cluster-management/deployments/open-cluster-management-registry
uid: 900eb533-89f8-43ee-9bfa-3f17d77e93b7
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app: open-cluster-management-registry
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
app: open-cluster-management-registry
spec:
containers:
- image: quay.io/open-cluster-management/multicloudhub-operator-index:1.0.0-SNAPSHOT-2020-03-10-19-58-31
imagePullPolicy: IfNotPresent
name: multicloudhub-operator-index
ports:
- containerPort: 50051
protocol: TCP
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
imagePullSecrets:
- name: quay-secret
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
status:
availableReplicas: 1
conditions:

• lastTransitionTime: "2020-03-11T03:29:56Z"
  lastUpdateTime: "2020-03-11T03:29:56Z"
  message: Deployment has minimum availability.
  reason: MinimumReplicasAvailable
  status: "True"
  type: Available
• lastTransitionTime: "2020-03-11T03:29:45Z"
  lastUpdateTime: "2020-03-11T03:29:56Z"
  message: ReplicaSet "open-cluster-management-registry-dbc98b957" has successfully
  progressed.
  reason: NewReplicaSetAvailable
  status: "True"
  type: Progressing
  observedGeneration: 1
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1
  [kni@provisionhost-0 ~]$

[achuzhoy]

[kni@provisionhost-0 ~]$ oc get catalogsource open-cluster-management -n open-cluster-management -o yaml
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"operators.coreos.com/v1alpha1","kind":"CatalogSource","metadata":{"annotations":{},"name":"open-cluster-management","namespace":"open-cluster-management"},"spec":{"address":"open-cluster-management-registry.open-cluster-management.svc:50051","sourceType":"grpc"}}
creationTimestamp: "2020-03-11T03:29:45Z"
generation: 1
name: open-cluster-management
namespace: open-cluster-management
resourceVersion: "2412749"
selfLink: /apis/operators.coreos.com/v1alpha1/namespaces/open-cluster-management/catalogsources/open-cluster-management
uid: 3015fc78-5243-4a63-8f61-b9ce6c82a6da
spec:
address: open-cluster-management-registry.open-cluster-management.svc:50051
sourceType: grpc
status:
connectionState:
address: open-cluster-management-registry.open-cluster-management.svc:50051
lastConnect: "2020-03-11T14:15:24Z"
lastObservedState: READY
registryService:
createdAt: "2020-03-11T03:29:45Z"
protocol: grpc
[kni@provisionhost-0 ~]$

[achuzhoy]

[kni@provisionhost-0 multicloudhub-operator]$ cat kustomization.yaml 
# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

generatorOptions:
  disableNameSuffixHash: true

# namespace to deploy all Resources to
namespace: open-cluster-management

images:
  - name: multicloudhub-operator-index
    newName: quay.io/open-cluster-management/multicloudhub-operator-index
    newTag: 1.0.0-SNAPSHOT-2020-03-10-19-58-31

# list of Resource Config to be Applied
resources:
  - quay-secret.yaml
  - deployment.yaml
  - service.yaml
  - catalog-source.yaml
  - operator-group.yaml
  - subscription.yaml

[tpouyer]

@achuzhoy All this looks good... you were able to pull the multicloudhub-operator-index image so you should have no issues pulling the multicloudhub-operator-bundle image.

I made some changes to this repo last night to work with some changes that multicloudhub-operator made this morning wrt new secrets... please pull the latest code and be sure to read the updated README.md

[achuzhoy]

Reproduced the issue.
Note that I'm trying it on 4.4 (deviates from the README.md)

Warning Failed 22s kubelet, worker-1 Error: ImagePullBackOff
Normal Pulling 10s (x2 over 23s) kubelet, worker-1 Pulling image "quay.io/open-cluster-management/multicloudhub-operator-bundle@sha256:0edb06bb5c8e9c49a21bbb678709c524f549cbd23ec637987ac08feac8a9f5be"
Warning Failed 10s (x2 over 22s) kubelet, worker-1 Failed to pull image "quay.io/open-cluster-management/multicloudhub-operator-bundle@sha256:0edb06bb5c8e9c49a21bbb678709c524f549cbd23ec637987ac08feac8a9f5be": rpc error: code = Unknown desc = Error reading manifest sha256:0edb06bb5c8e9c49a21bbb678709c524f549cbd23ec637987ac08feac8a9f5be in quay.io/open-cluster-management/multicloudhub-operator-bundle: unauthorized: access to the requested resource is not authorized

[schmidtd]

I did find that some repos in quay didn't have read permissions set for some accounts - QE and Robots among them. I've set all existing repos now to have read permissions for everyone, so that might have an effect here.

[achuzhoy]

oc patch serviceaccount default -p '{"imagePullSecrets": [{"name": "multiclusterhub-operator-pull-secret"}]}'

@tpouyer tried that on a new setup (and thus new clone) - same error persists.

[bjacot]

I too am having the same issue. I can pull from open-cluster-management/multicloudhub-operator-index but dont seem to have permissions to pull from open-cluster-management/multicloudhub-operator-bundle

[root@sealusa11 multicloudhub-operator]# oc get pods
NAME READY STATUS RESTARTS AGE
0b302b2b0acd2947160616eec1ae06326a854a7e4a0e1d3925f0e40d1d2nff9 0/1 Init:ImagePullBackOff 0 3m54s
open-cluster-management-registry-c7dbc8f47-ctlkt 1/1 Running 0 4m1s

[root@sealusa11 multicloudhub-operator]# oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.4.0-0.ci-2020-03-11-095511 True False 22h Cluster version is 4.4.0-0.ci-2020-03-11-095511

[tpouyer]

I have just pushed an update to the repo... I'm now patching the default service-account with the pull-secret that gets created as part of the prereqs here: https://github.com/open-cluster-management/deploy/blob/master/prereqs/serviceaccount.yaml#L7

This should resolve the issue some people are having... I have not been able to isolate why some people are having auth issues pulling the bundle image and some are not... permissions are the same on the bundle repo as the index repo in quay... patching the default service account with the pull-secret seems to be the only way to get around the problem...

Ultimately these repos will all be opensourced at some point and the need for pull secrets to pull the index and bundle images will no longer be necessary.

Please pull the latest code down and try it out, be sure to rerun kubectl apply -k . in the prereqs dir to patch the service account even if you are reusing your OCP cluster and already have applied the prereqs before.

[achuzhoy]

Now failing with #6

The error is for another image, but the resolution may be the same in the end.
Still stuck.