I suspect this has to do with the permissions granted to the CRD, but I don't know how to fix or debug this.
Here is the debugs.
[2020-06-27 04:34:49,685] kopf.reactor.activit [INFO ] Initial authentication has been initiated.
[2020-06-27 04:34:49,686] kopf.activities.auth [INFO ] Activity 'login_via_pykube' succeeded.
[2020-06-27 04:34:49,689] kopf.activities.auth [INFO ] Activity 'login_via_client' succeeded.
[2020-06-27 04:34:49,689] kopf.reactor.activit [INFO ] Initial authentication has finished.
[2020-06-27 04:34:49,698] kopf.engines.peering [WARNING ] Default peering object not found, falling back to the standalone mode.
[2020-06-27 04:34:49,814] kopf.objects [INFO ] [None/default] Handler 'namespace_watcher' succeeded.
[2020-06-27 04:34:49,814] kopf.objects [INFO ] [None/default] All handlers succeeded for creation.
[2020-06-27 04:34:49,817] kopf.objects [INFO ] [None/kube-system] Handler 'namespace_watcher' succeeded.
[2020-06-27 04:34:49,818] kopf.objects [INFO ] [None/kube-system] All handlers succeeded for creation.
[2020-06-27 04:34:49,820] kopf.objects [INFO ] [None/kube-public] Handler 'namespace_watcher' succeeded.
[2020-06-27 04:34:49,821] kopf.objects [INFO ] [None/kube-public] All handlers succeeded for creation.
[2020-06-27 04:34:49,823] kopf.objects [INFO ] [None/kube-node-lease] Handler 'namespace_watcher' succeeded.
[2020-06-27 04:34:49,823] kopf.objects [INFO ] [None/kube-node-lease] All handlers succeeded for creation.
[2020-06-27 04:34:49,825] kopf.objects [INFO ] [None/dev-services] Handler 'namespace_watcher' succeeded.
[2020-06-27 04:34:49,825] kopf.objects [INFO ] [None/dev-services] All handlers succeeded for creation.
[2020-06-27 04:34:49,827] kopf.objects [INFO ] [None/home-services] Handler 'namespace_watcher' succeeded.
[2020-06-27 04:34:49,827] kopf.objects [INFO ] [None/home-services] All handlers succeeded for creation.
[2020-06-27 04:34:49,833] kopf.objects [INFO ] [None/clustersecret] Handler 'namespace_watcher' succeeded.
[2020-06-27 04:34:49,833] kopf.objects [INFO ] [None/clustersecret] All handlers succeeded for creation.
[2020-06-27 04:34:49,851] kopf.reactor.queuein [ERROR ] functools.partial(<function process_resource_event at 0x7fb3091fab00>, lifecycle=<function asap at 0x7fb309419dd0>, registry=<kopf.toolkits.legacy_registries.SmartGlobalRegistry object at 0x7fb307e84a50>, settings=OperatorSettings(logging=LoggingSettings(), posting=PostingSettings(enabled=True, level=20), watching=WatchingSettings(server_timeout=None, cl
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/kopf/reactor/queueing.py", line 187, in worker
await processor(raw_event=raw_event, replenished=replenished)
File "/usr/local/lib/python3.7/site-packages/kopf/reactor/processing.py", line 204, in process_resource_event
replenished=replenished,
File "/usr/local/lib/python3.7/site-packages/kopf/reactor/processing.py", line 227, in apply_reaction_outcomes
await patching.patch_obj(resource=resource, patch=patch, body=body)
File "/usr/local/lib/python3.7/site-packages/kopf/clients/auth.py", line 45, in wrapper
return await fn(*args, **kwargs, context=context)
File "/usr/local/lib/python3.7/site-packages/kopf/clients/patching.py", line 59, in patch_obj
raise_for_status=True,
File "/usr/local/lib/python3.7/site-packages/aiohttp/client.py", line 588, in _request
resp.raise_for_status()
File "/usr/local/lib/python3.7/site-packages/aiohttp/client_reqrep.py", line 946, in raise_for_status
headers=self.headers)
aiohttp.client_exceptions.ClientResponseError: 403, message='Forbidden', url=URL('https://10.43.0.1:443/api/v1/namespaces/dev-services')
[2020-06-27 04:34:49,853] kopf.clients.events [WARNING ] Failed to post an event. Ignoring and continuing. Status: 403. Message: Forbidden. Event: type='Normal', reason='Logging', message="Handler 'namespace_watcher' succeeded.".
[2020-06-27 04:34:49,855] kopf.reactor.queuein [ERROR ] functools.partial(<function process_resource_event at 0x7fb3091fab00>, lifecycle=<function asap at 0x7fb309419dd0>, registry=<kopf.toolkits.legacy_registries.SmartGlobalRegistry object at 0x7fb307e84a50>, settings=OperatorSettings(logging=LoggingSettings(), posting=PostingSettings(enabled=True, level=20), watching=WatchingSettings(server_timeout=None, cl
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/kopf/reactor/queueing.py", line 187, in worker
await processor(raw_event=raw_event, replenished=replenished)
File "/usr/local/lib/python3.7/site-packages/kopf/reactor/processing.py", line 204, in process_resource_event
replenished=replenished,
File "/usr/local/lib/python3.7/site-packages/kopf/reactor/processing.py", line 227, in apply_reaction_outcomes
await patching.patch_obj(resource=resource, patch=patch, body=body)
File "/usr/local/lib/python3.7/site-packages/kopf/clients/auth.py", line 45, in wrapper
return await fn(*args, **kwargs, context=context)
File "/usr/local/lib/python3.7/site-packages/kopf/clients/patching.py", line 59, in patch_obj
raise_for_status=True,
File "/usr/local/lib/python3.7/site-packages/aiohttp/client.py", line 588, in _request
resp.raise_for_status()
File "/usr/local/lib/python3.7/site-packages/aiohttp/client_reqrep.py", line 946, in raise_for_status
headers=self.headers)
aiohttp.client_exceptions.ClientResponseError: 403, message='Forbidden', url=URL('https://10.43.0.1:443/api/v1/namespaces/default')
[2020-06-27 04:34:49,855] kopf.reactor.queuein [ERROR ] functools.partial(<function process_resource_event at 0x7fb3091fab00>, lifecycle=<function asap at 0x7fb309419dd0>, registry=<kopf.toolkits.legacy_registries.SmartGlobalRegistry object at 0x7fb307e84a50>, settings=OperatorSettings(logging=LoggingSettings(), posting=PostingSettings(enabled=True, level=20), watching=WatchingSettings(server_timeout=None, cl
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/kopf/reactor/queueing.py", line 187, in worker
await processor(raw_event=raw_event, replenished=replenished)
File "/usr/local/lib/python3.7/site-packages/kopf/reactor/processing.py", line 204, in process_resource_event
replenished=replenished,
File "/usr/local/lib/python3.7/site-packages/kopf/reactor/processing.py", line 227, in apply_reaction_outcomes
await patching.patch_obj(resource=resource, patch=patch, body=body)
File "/usr/local/lib/python3.7/site-packages/kopf/clients/auth.py", line 45, in wrapper
return await fn(*args, **kwargs, context=context)
File "/usr/local/lib/python3.7/site-packages/kopf/clients/patching.py", line 59, in patch_obj
raise_for_status=True,
File "/usr/local/lib/python3.7/site-packages/aiohttp/client.py", line 588, in _request
resp.raise_for_status()
File "/usr/local/lib/python3.7/site-packages/aiohttp/client_reqrep.py", line 946, in raise_for_status
headers=self.headers)
aiohttp.client_exceptions.ClientResponseError: 403, message='Forbidden', url=URL('https://10.43.0.1:443/api/v1/namespaces/kube-system')
[2020-06-27 04:34:49,856] kopf.reactor.queuein [ERROR ] functools.partial(<function process_resource_event at 0x7fb3091fab00>, lifecycle=<function asap at 0x7fb309419dd0>, registry=<kopf.toolkits.legacy_registries.SmartGlobalRegistry object at 0x7fb307e84a50>, settings=OperatorSettings(logging=LoggingSettings(), posting=PostingSettings(enabled=True, level=20), watching=WatchingSettings(server_timeout=None, cl
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/kopf/reactor/queueing.py", line 187, in worker
await processor(raw_event=raw_event, replenished=replenished)
File "/usr/local/lib/python3.7/site-packages/kopf/reactor/processing.py", line 204, in process_resource_event
replenished=replenished,
File "/usr/local/lib/python3.7/site-packages/kopf/reactor/processing.py", line 227, in apply_reaction_outcomes
await patching.patch_obj(resource=resource, patch=patch, body=body)
File "/usr/local/lib/python3.7/site-packages/kopf/clients/auth.py", line 45, in wrapper
return await fn(*args, **kwargs, context=context)
File "/usr/local/lib/python3.7/site-packages/kopf/clients/patching.py", line 59, in patch_obj
raise_for_status=True,
File "/usr/local/lib/python3.7/site-packages/aiohttp/client.py", line 588, in _request
resp.raise_for_status()
File "/usr/local/lib/python3.7/site-packages/aiohttp/client_reqrep.py", line 946, in raise_for_status
headers=self.headers)
aiohttp.client_exceptions.ClientResponseError: 403, message='Forbidden', url=URL('https://10.43.0.1:443/api/v1/namespaces/kube-public')
[2020-06-27 04:34:49,856] kopf.reactor.queuein [ERROR ] functools.partial(<function process_resource_event at 0x7fb3091fab00>, lifecycle=<function asap at 0x7fb309419dd0>, registry=<kopf.toolkits.legacy_registries.SmartGlobalRegistry object at 0x7fb307e84a50>, settings=OperatorSettings(logging=LoggingSettings(), posting=PostingSettings(enabled=True, level=20), watching=WatchingSettings(server_timeout=None, cl
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/kopf/reactor/queueing.py", line 187, in worker
await processor(raw_event=raw_event, replenished=replenished)
File "/usr/local/lib/python3.7/site-packages/kopf/reactor/processing.py", line 204, in process_resource_event
replenished=replenished,
File "/usr/local/lib/python3.7/site-packages/kopf/reactor/processing.py", line 227, in apply_reaction_outcomes
await patching.patch_obj(resource=resource, patch=patch, body=body)
File "/usr/local/lib/python3.7/site-packages/kopf/clients/auth.py", line 45, in wrapper
return await fn(*args, **kwargs, context=context)
File "/usr/local/lib/python3.7/site-packages/kopf/clients/patching.py", line 59, in patch_obj
raise_for_status=True,
File "/usr/local/lib/python3.7/site-packages/aiohttp/client.py", line 588, in _request
resp.raise_for_status()
File "/usr/local/lib/python3.7/site-packages/aiohttp/client_reqrep.py", line 946, in raise_for_status
headers=self.headers)
aiohttp.client_exceptions.ClientResponseError: 403, message='Forbidden', url=URL('https://10.43.0.1:443/api/v1/namespaces/kube-node-lease')
[2020-06-27 04:34:49,857] kopf.reactor.queuein [ERROR ] functools.partial(<function process_resource_event at 0x7fb3091fab00>, lifecycle=<function asap at 0x7fb309419dd0>, registry=<kopf.toolkits.legacy_registries.SmartGlobalRegistry object at 0x7fb307e84a50>, settings=OperatorSettings(logging=LoggingSettings(), posting=PostingSettings(enabled=True, level=20), watching=WatchingSettings(server_timeout=None, cl
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/kopf/reactor/queueing.py", line 187, in worker
await processor(raw_event=raw_event, replenished=replenished)
File "/usr/local/lib/python3.7/site-packages/kopf/reactor/processing.py", line 204, in process_resource_event
replenished=replenished,
File "/usr/local/lib/python3.7/site-packages/kopf/reactor/processing.py", line 227, in apply_reaction_outcomes
await patching.patch_obj(resource=resource, patch=patch, body=body)
File "/usr/local/lib/python3.7/site-packages/kopf/clients/auth.py", line 45, in wrapper
return await fn(*args, **kwargs, context=context)
File "/usr/local/lib/python3.7/site-packages/kopf/clients/patching.py", line 59, in patch_obj
raise_for_status=True,
File "/usr/local/lib/python3.7/site-packages/aiohttp/client.py", line 588, in _request
resp.raise_for_status()
File "/usr/local/lib/python3.7/site-packages/aiohttp/client_reqrep.py", line 946, in raise_for_status
headers=self.headers)
aiohttp.client_exceptions.ClientResponseError: 403, message='Forbidden', url=URL('https://10.43.0.1:443/api/v1/namespaces/home-services')
[2020-06-27 04:34:49,858] kopf.reactor.queuein [ERROR ] functools.partial(<function process_resource_event at 0x7fb3091fab00>, lifecycle=<function asap at 0x7fb309419dd0>, registry=<kopf.toolkits.legacy_registries.SmartGlobalRegistry object at 0x7fb307e84a50>, settings=OperatorSettings(logging=LoggingSettings(), posting=PostingSettings(enabled=True, level=20), watching=WatchingSettings(server_timeout=None, cl
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/kopf/reactor/queueing.py", line 187, in worker
await processor(raw_event=raw_event, replenished=replenished)
File "/usr/local/lib/python3.7/site-packages/kopf/reactor/processing.py", line 204, in process_resource_event
replenished=replenished,
File "/usr/local/lib/python3.7/site-packages/kopf/reactor/processing.py", line 227, in apply_reaction_outcomes
await patching.patch_obj(resource=resource, patch=patch, body=body)
File "/usr/local/lib/python3.7/site-packages/kopf/clients/auth.py", line 45, in wrapper
return await fn(*args, **kwargs, context=context)
File "/usr/local/lib/python3.7/site-packages/kopf/clients/patching.py", line 59, in patch_obj
raise_for_status=True,
File "/usr/local/lib/python3.7/site-packages/aiohttp/client.py", line 588, in _request
resp.raise_for_status()
File "/usr/local/lib/python3.7/site-packages/aiohttp/client_reqrep.py", line 946, in raise_for_status
headers=self.headers)
aiohttp.client_exceptions.ClientResponseError: 403, message='Forbidden', url=URL('https://10.43.0.1:443/api/v1/namespaces/clustersecret')
[2020-06-27 04:34:49,860] kopf.clients.events [WARNING ] Failed to post an event. Ignoring and continuing. Status: 403. Message: Forbidden. Event: type='Normal', reason='Logging', message='All handlers succeeded for creation.'.
[2020-06-27 04:34:49,861] kopf.clients.events [WARNING ] Failed to post an event. Ignoring and continuing. Status: 403. Message: Forbidden. Event: type='Normal', reason='Logging', message="Handler 'namespace_watcher' succeeded.".
[2020-06-27 04:34:49,863] kopf.clients.events [WARNING ] Failed to post an event. Ignoring and continuing. Status: 403. Message: Forbidden. Event: type='Normal', reason='Logging', message='All handlers succeeded for creation.'.
[2020-06-27 04:34:49,864] kopf.clients.events [WARNING ] Failed to post an event. Ignoring and continuing. Status: 403. Message: Forbidden. Event: type='Normal', reason='Logging', message="Handler 'namespace_watcher' succeeded.".
[2020-06-27 04:34:49,866] kopf.clients.events [WARNING ] Failed to post an event. Ignoring and continuing. Status: 403. Message: Forbidden. Event: type='Normal', reason='Logging', message='All handlers succeeded for creation.'.
[2020-06-27 04:34:49,867] kopf.clients.events [WARNING ] Failed to post an event. Ignoring and continuing. Status: 403. Message: Forbidden. Event: type='Normal', reason='Logging', message="Handler 'namespace_watcher' succeeded.".
[2020-06-27 04:34:49,868] kopf.clients.events [WARNING ] Failed to post an event. Ignoring and continuing. Status: 403. Message: Forbidden. Event: type='Normal', reason='Logging', message='All handlers succeeded for creation.'.
[2020-06-27 04:34:49,870] kopf.clients.events [WARNING ] Failed to post an event. Ignoring and continuing. Status: 403. Message: Forbidden. Event: type='Normal', reason='Logging', message="Handler 'namespace_watcher' succeeded.".
[2020-06-27 04:34:49,871] kopf.clients.events [WARNING ] Failed to post an event. Ignoring and continuing. Status: 403. Message: Forbidden. Event: type='Normal', reason='Logging', message='All handlers succeeded for creation.'.
[2020-06-27 04:34:49,873] kopf.clients.events [WARNING ] Failed to post an event. Ignoring and continuing. Status: 403. Message: Forbidden. Event: type='Normal', reason='Logging', message="Handler 'namespace_watcher' succeeded.".
[2020-06-27 04:34:49,874] kopf.clients.events [WARNING ] Failed to post an event. Ignoring and continuing. Status: 403. Message: Forbidden. Event: type='Normal', reason='Logging', message='All handlers succeeded for creation.'.
[2020-06-27 04:34:49,875] kopf.clients.events [WARNING ] Failed to post an event. Ignoring and continuing. Status: 403. Message: Forbidden. Event: type='Normal', reason='Logging', message="Handler 'namespace_watcher' succeeded.".
[2020-06-27 04:34:49,877] kopf.clients.events [WARNING ] Failed to post an event. Ignoring and continuing. Status: 403. Message: Forbidden. Event: type='Normal', reason='Logging', message='All handlers succeeded for creation.'.
[2020-06-27 04:34:49,878] kopf.clients.events [WARNING ] Failed to post an event. Ignoring and continuing. Status: 403. Message: Forbidden. Event: type='Normal', reason='Logging', message="Handler 'namespace_watcher' succeeded.".
[2020-06-27 04:34:49,880] kopf.clients.events [WARNING ] Failed to post an event. Ignoring and continuing. Status: 403. Message: Forbidden. Event: type='Normal', reason='Logging', message='All handlers succeeded for creation.'.
[2020-06-27 04:34:49,881] kopf.clients.events [WARNING ] Failed to post an event. Ignoring and continuing. Status: 403. Message: Forbidden. Event: type='Normal', reason='Logging', message="Handler 'namespace_watcher' succeeded.".
[2020-06-27 04:34:49,882] kopf.clients.events [WARNING ] Failed to post an event. Ignoring and continuing. Status: 403. Message: Forbidden. Event: type='Normal', reason='Logging', message='All handlers succeeded for creation.'.
[2020-06-27 04:44:26,527] kopf.reactor.queuein [ERROR ] functools.partial(<function process_resource_event at 0x7fb3091fab00>, lifecycle=<function asap at 0x7fb309419dd0>, registry=<kopf.toolkits.legacy_registries.SmartGlobalRegistry object at 0x7fb307e84a50>, settings=OperatorSettings(logging=LoggingSettings(), posting=PostingSettings(enabled=True, level=20), watching=WatchingSettings(server_timeout=None, cl
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/kopf/reactor/queueing.py", line 187, in worker
await processor(raw_event=raw_event, replenished=replenished)
File "/usr/local/lib/python3.7/site-packages/kopf/reactor/processing.py", line 204, in process_resource_event
replenished=replenished,
File "/usr/local/lib/python3.7/site-packages/kopf/reactor/processing.py", line 227, in apply_reaction_outcomes
await patching.patch_obj(resource=resource, patch=patch, body=body)
File "/usr/local/lib/python3.7/site-packages/kopf/clients/auth.py", line 45, in wrapper
return await fn(*args, **kwargs, context=context)
File "/usr/local/lib/python3.7/site-packages/kopf/clients/patching.py", line 59, in patch_obj
raise_for_status=True,
File "/usr/local/lib/python3.7/site-packages/aiohttp/client.py", line 588, in _request
resp.raise_for_status()
File "/usr/local/lib/python3.7/site-packages/aiohttp/client_reqrep.py", line 946, in raise_for_status
headers=self.headers)
aiohttp.client_exceptions.ClientResponseError: 403, message='Forbidden', url=URL('https://10.43.0.1:443/apis/clustersecret.io/v1/namespaces/clustersecret/clustersecrets/test-cluster-secret')