google-cloud-compute: ImagesClient does not terminate, and hangs instead about google-cloud-java

[Martiul]

I've also dug a bit into the library code and it might have something to do with gax?

[suztomo]

Probably not the core of the problem, but the code example is not using try-with-resources: try (ImagesClient imagesClient = ImagesClient.create(ImagesSettings.newBuilder().build())) { .

[meltsufin]

I'm able to reproduce this. The regression was introduced in google-cloud-compute:1.27.0 in May, corresponding to libraries-bom:26.15.0.

@lqiu96 I confirmed that this is the PR that caused the regression: googleapis/sdk-platform-java#1603

[lqiu96]

Hi @Martiul, thank you for reporting this. I followed the steps that you listed above and was able to reproduce this locally. With or without the client initialized in try-with resources, the client isn't able to fully close properly and the program isn't able to terminate. Thanks @meltsufin for pointing to my previous PR, I reverted gax-httpjson back to the version before the PR (v0.111.0) and was able to confirm that the program was able to terminate properly.

My repro pom.xml that reverts a previously working version

    <dependency>
      <groupId>com.google.cloud</groupId>
      <artifactId>google-cloud-compute</artifactId>
      <version>1.43.0</version>
    </dependency>
    <dependency>
      <groupId>com.google.api</groupId>
      <artifactId>gax-httpjson</artifactId>
      <version>0.111.0</version>
    </dependency>

Output:

Exception in thread "main" java.lang.NoSuchMethodError: 'com.google.api.gax.httpjson.HttpJsonCallSettings$Builder com.google.api.gax.httpjson.HttpJsonCallSettings$Builder.setParamsExtractor(com.google.api.gax.rpc.RequestParamsExtractor)'
	at com.google.cloud.compute.v1.stub.HttpJsonGlobalOperationsStub.<init>(HttpJsonGlobalOperationsStub.java:368)
	at com.google.cloud.compute.v1.stub.HttpJsonGlobalOperationsStub.create(HttpJsonGlobalOperationsStub.java:337)
	at com.google.cloud.compute.v1.stub.HttpJsonImagesStub.<init>(HttpJsonImagesStub.java:629)
	at com.google.cloud.compute.v1.stub.HttpJsonImagesStub.<init>(HttpJsonImagesStub.java:614)
	at com.google.cloud.compute.v1.stub.HttpJsonImagesStub.create(HttpJsonImagesStub.java:594)
	at com.google.cloud.compute.v1.stub.ImagesStubSettings.createStub(ImagesStubSettings.java:277)
	at com.google.cloud.compute.v1.ImagesClient.<init>(ImagesClient.java:343)
	at com.google.cloud.compute.v1.ImagesClient.create(ImagesClient.java:326)
	at com.google.cloud.compute.v1.ImagesClient.create(ImagesClient.java:318)
	at org.example.Main.main(Main.java:21)

Process finished with exit code 1 <--- I see the exit code for termination

It's a different error regarding dynamic routing headers, but I believe it shows that the program was able to terminate properly.

I checked the RPC's settings and saw that it was configured with a default timeout value of 10 minutes (as defined in the RetrySettings here and then configured here). I believe this value determines when the program would actually terminate and the spring shell timeout is probably just a coincidence that matches.

My previous PR introduced the concept of terminating the call if the call's duration has exceed the total timeout configured for the RPC. This is done via a "deadline schedule executor" which just schedules a timeout task once the configured timeout elapsed. The deadline scheduler's timeout tasks is scheduled for current_time() + rpc_timeout(), which for this RPC is 10m from when the RPC was invoked.

I believe the issue is with the deadlineScheduler's shutdown() call. The javadocs for shutdown() suggests if the task has been added to the scheduler, the task will continue until it's completed. I think the intended behavior is that once we've received a result (either error or response) prior to the timeout, the timeout task should be cancelled and the client should return the response back to the user.

I ran a test locally which cancels the timeout task once a response has been received and was able to see the client terminate properly:

...
      }
    ],
    "status": "PERMISSION_DENIED",
    "details": [
      {
        "@type": "type.googleapis.com/google.rpc.Help",
        "links": [
          {
            "description": "Google developers console API activation",
            "url": "https://console.developers.google.com/apis/api/compute.googleapis.com/overview?project=foo"
          }
        ]
      },
      {
        "@type": "type.googleapis.com/google.rpc.ErrorInfo",
        "reason": "SERVICE_DISABLED",
        "domain": "googleapis.com",
        "metadatas": {
          "service": "compute.googleapis.com",
          "consumer": "projects/foo"
        }
      }
    ]
  }
}

	at com.google.api.client.http.HttpResponseException$Builder.build(HttpResponseException.java:293)
	at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1118)
	at com.google.api.gax.httpjson.HttpRequestRunnable.run(HttpRequestRunnable.java:115)
	... 6 more

Process finished with exit code 1 <--- Expected to see this to show that the JVM has exited

I believe the output above should be the intended result. I think there should be a simple fix for this, but I'll do some more investigating to confirm.

[Martiul]

Probably not the core of the problem, but the code example is not using try-with-resources: try (ImagesClient imagesClient = ImagesClient.create(ImagesSettings.newBuilder().build())) { .

yes. I split the example into try-catch-finally so it's easier to debug imagesClient.isTerminated()

[Martiul]

thanks @lqiu96 for the deepdive on the scheduler, i was also under the impression that there was some unfinished job somewhere. Looking forward to the fix

[lqiu96]

I added a draft POC of a possible fix: googleapis/sdk-platform-java#2360. I'm taking a look into grpc-java (httpjson is mostly a mirror/ inspired from there) and other possible implementations.

CC: @blakeli0

[blakeli0]

@Martiul @meltsufin @suztomo, I added priority:p2 to this bug, let me know if you feel differently.