Cloud Run + Cloud SQL hanging issue about functions-framework-dart

[brianquinlan]

I've assigned the Dart issue (dart-lang/sdk#47899) to myself but I probably won't have time to take a look at this until next week.

[AlexKenbo]

There are more details here: dart-lang/sdk#47899

But my current hypothesis is that the gVisor containerization used in the gen1 Cloud Run execution environment implements a syscall differently than Dart expects.

You can work around that by using the gen2 execution environment, see selecting an execution environment.

I can no longer reproduce the issue if I change this repo: https://github.com/davidmartos96/cloud-run-dart-pg-issue

with:

diff --git a/Makefile b/Makefile
		--add-cloudsql-instances=$(CLOUD_SQL_CONNECTION_NAME) \
		--set-env-vars=DB_USER=$(DB_USER),DB_PASS=$(DB_PASS),DB_NAME=$(DB_NAME),CLOUD_SQL_CONNECTION_NAME=$(CLOUD_SQL_CONNECTION_NAME) \
+		--execution-environment=gen2

My next step is to try to reproduce this using gVisor directly to see if this is a bug in Dart or gVisor.

Thank you it helped only this (gen2)

[kevmoo]

CC @mit-mit – I'm guessing this is not specific to this framework, and likely something deeper in pkg:shelf or dart:io

Will dig in here!

[brianquinlan]

There are more details here: dart-lang/sdk#47899

But my current hypothesis is that the gVisor containerization used in the gen1 Cloud Run execution environment implements a syscall differently than Dart expects.

You can work around that by using the gen2 execution environment, see selecting an execution environment.

I can no longer reproduce the issue if I change this repo:
https://github.com/davidmartos96/cloud-run-dart-pg-issue

with:

diff --git a/Makefile b/Makefile
		--add-cloudsql-instances=$(CLOUD_SQL_CONNECTION_NAME) \
		--set-env-vars=DB_USER=$(DB_USER),DB_PASS=$(DB_PASS),DB_NAME=$(DB_NAME),CLOUD_SQL_CONNECTION_NAME=$(CLOUD_SQL_CONNECTION_NAME) \
+		--execution-environment=gen2

My next step is to try to reproduce this using gVisor directly to see if this is a bug in Dart or gVisor.

[kevmoo]

Could you get a repro of this working with docker-compose? (With Dart + postgres?)

Would make debugging a LOT easier!

[davidmartos96]

Could you get a repro of this working with docker-compose? (With Dart + postgres?)

Would make debugging a LOT easier!

You mean working with a regular Postgres database on Docker with Unix sockets? I can try to do that.

[kevmoo]

Yeah – and if you included a docker compose setup yaml in the repo, it makes reproducing on our side a LOT easier!

[davidmartos96]

@kevmoo I've updated the sample with a docker compose ready to run the app with a local Postgres. As you'll see, the socket connection works correctly locally.

[kevmoo]

I'd be surprised if unix socket works on run at all. You're saying it DOES work?

Have you tried TCP? Does that work?

[kevmoo]

Are you using Cloud SQL?

Have you looked at https://cloud.google.com/sql/docs/postgres/connect-run#connecting_to ?

[davidmartos96]

@kevmoo Yes, Unix sockets work correctly, but only locally 😅
TCP works just fine both locally and on Cloud Run. The problem is that in the Cloud Run environment the IP is unknown, so we cannot add an authorized network for it in Cloud SQL. We would need to add 0.0.0.0/0 to use TCP from Cloud Run.

Yes, we looked at the provided docs, and that's how we got to connect it. The issue is just in the second connection. Our Cloud SQL uses public IP, so the docs state to use the Unix socket connection mounted in the /cloudsql path.

[davidmartos96]

We've also tried deploying the node.js demo, which connects to the same socket path, and it works. It seems to be something related to Dart.

[davidmartos96]

Hey @kevmoo, any updates on this? Like some other issue I could track?

[marcusmonteirodesouza]

I think I might be experiencing the same problem. When I deploy this app to Cloud Run (via terraform) and then make a HTTP request to it, in the majority of times the request hangs on indefinitely until it returns 504, and sometimes it returns OK. I even opened an issue for that because it is puzzling.

I'm using the postgres_pool package to create the connection.

[brianquinlan]

Is there currently an owner for this issue? If not, I'll assign it (and the related dart-lang/sdk#47899) to myself.

[kevmoo]

@brianquinlan Please! Happy to help here. I don't have the tech chops to help figure out what's happening.

Other folks here, I have a question: have you ever tried to containerize your app locally and then connect to Cloud SQL?

Trying to figure out if this is a Docker/container thing or a Cloud Run thing...

[davidmartos96]

@kevmoo When I encountered with this issue I tried connecting via Unix socket through the Cloud SQL Proxy (https://cloud.google.com/sql/docs/postgres/sql-proxy) and the socket connection through Dart was working fine, but I wasn't able to containerize the proxy.

From the tests I did my guess is that it could be something in Dart io or Cloud Run, and not a Postgres library issue.

[marcusmonteirodesouza]

@kevmoo I created a branch to test this https://github.com/marcusmonteirodesouza/dart-shelf-realworld-example-app/tree/test/cloudsql-proxy-containerized

I made some requests to my local endpoint and they all returned OK. I checked and the data was saved in my CloudSQL database. So a local containerized app connected to CloudSQL Proxy seems to work.

[brianquinlan]

I'm pretty sure that this is a gVisor issue.

I've filed a bug with the gVisor team: google/gvisor#7331

@kevmoo The workaround is to use the gen2 execution environment, see selecting an execution environment.

Do we want to update our documentation (e.g. https://github.com/GoogleCloudPlatform/functions-framework-dart/blob/main/docs/quickstarts/03-quickstart-cloudrun.md) to suggest using the gen2 execution environment?

[kevmoo]

@brianquinlan – not a bad idea! Let's leave this issue open to track that...

[brianquinlan]

@kevmoo I can change the Cloud Run quickstart. Do you know of any other documents that need to be updated?

[brianquinlan]

@kevmoo Also, are you an admin for this repo? If so, could you give me permission to modify issues?

[brianquinlan]

The docs for Cloud Run now say that docker is used as the execution environment:
https://cloud.google.com/run/docs/about-execution-environments

So is this still an issue?

[kevmoo]

I don't think so. We found the root cause.

[brianquinlan]

If anyone hits this issue again, please reopen this bug.