ValueError: Protocol message AutoprovisioningNodePoolDefaults has no "management" field. about python-container

[askmeegs]

Hi Rafael, sorry for the delay here. I was able to reproduce this bug --

Using the same code + setup you have, except for composer version you used is no longer available, so I'm on composer-1.17.7-airflow-2.0.2.

Then when I edited my DAG to print out the attributes of a blank AutoprovisioningNodePoolDefaults object, management doesn't appear, along with others that should, given the documentation here. Here's what appears in Airflow logs when I run the dir() command on one of these objects:

[2022-01-05 20:10:03,017] {gke.py:37} INFO - ['ByteSize', 'Clear', 'ClearExtension', 'ClearField', 'CopyFrom', 'DESCRIPTOR', 'DiscardUnknownFields', 'Extensions', 'FindInitializationErrors', 'FromString', 'HasExtension', 'HasField', 'IsInitialized', 'ListFields', 'MergeFrom', 'MergeFromString', 'ParseFromString', 'RegisterExtension', 'SerializePartialToString', 'SerializeToString', 'SetInParent', 'UnknownFields', 'WhichOneof', '_CheckCalledFromGeneratedFile', '_SetListener', '__class__', '__deepcopy__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getstate__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__setstate__', '__sizeof__', '__slots__', '__str__', '__subclasshook__', '__unicode__', '_extensions_by_name', '_extensions_by_number', 'oauth_scopes', 'service_account']

I'm going to reach out to the Cloud Composer team to determine exactly what version of the Python GKE client library is running under the hood, and will update this issue after. thanks!

[askmeegs]

Hello! Following up on this, the Apache Airflow team responded to that issue ^ that they are deliberately limiting google-cloud-container versions to < 2.0 due to some breaking changes that were introduced. (docs here)

apache/airflow#10317

Given this, and the hypothesis that this bug is caused by the out of date version of google-cloud-container missing certain object fields like management closing this issue as infeasible - until at which date airflow (and in turn, Cloud Composer) upgrades to beyond 2.0.0.