BuildKit socket timeout on 3.12.0 about convox

[beastawakens]

@beastawakens can you increase the node disk and check if it helps?

I've increased this and am monitoring. So far it seems to have helped...

[nightfury1204]

@beastawakens thanks for reporting, I already updated the time and the version in this release: https://github.com/convox/convox/releases/tag/3.12.1
it should be resolved, let me know if you still see this

[beastawakens]

@nightfury1204 Amazing, thank you so much! Will try that out!
Now just very much looking forward to buildkit 0.12 landing so we can support ECR image caching for vastly faster builds!
moby/buildkit#3724

[beastawakens]

@nightfury1204 Just updated, and now I'm getting:

$ convox build /tmp/074381573 --app partner1-1295 --description '' --id --manifest convox.yml --development
Packaging source... OK
Uploading source... OK
Starting build... OK
could not connect to unix:///run/buildkit/buildkitd.sock after 30 trials
========== log ==========

So the issue is still persisting, just with more retries now 😞

[nightfury1204]

@beastawakens can you share your rack params?

[beastawakens]

@nightfury1204 Sure:

high_availability   false
key_pair_name       dev-us-keypair-xxxxxxx
node_capacity_type  spot
node_type           t3a.large,r5.large,t3.large
proxy_protocol      true
region              us-west-2
release             3.12.1

[beastawakens]

We're also seeing builds hanging after starting, getting most of the way through the docker layers, and then just sitting there doing nothing for 4 hours before eventually being killed off.
Is it because HA is false? Get semi-frequent 502 gateway timeouts 🤔

We have 6 instances running on this rack currently...

[nightfury1204]

@beastawakens can you increase the node disk and check if it helps?