Alternative remote caching hosts about turbo

[weyert]

I have written my own remote cache server for turbo yesterday for Google Cloud Storage. I will see if it can be open-sourced

[m19c]

Yeah, that's a really good idea.
I am also happy to help with the implementation.

@thib92 we can also work together if you wan't to.

[weyert]

I have talked with my work and I am allowed to open-source my proof of concept server which is compatible with the remote cache of Turborepo and supports Google Cloud Storage, and Amazon S3. Azure Storage is also possible but would need someone to with a such blob storage to test it out.

[weyert]

Please find it here: https://github.com/Tapico/tapico-turborepo-remote-cache :)

[weyert]

If this support will be added to turborepo you also need to think about companies that are for all kind of compliance reasons can not just directly connect to a cloud storage provider or might need to access via a VPN and/or IAP. For example, at work I am unable to directly access cloud storage and write to it from my computer system.

One of the reasons that I am worked on my above mentioned project to ease this problem. It's probably a good start for people that are concerned about not wanting to store build artefacts on Vercel. The project comes with binaries for amd64, windows, and arm and has docker images pre-build that can be used. I hope to find some free time to make an example how it can be ran via Cloud Run on Google Cloud.

If anyone want support for Azure Storage, please reach out to me and we can work on it together. I used a cloud abstraction library for Go that supports it already: https://github.com/graymeta/stow which supports the following:

• Microsoft Azure Blob Storage
• Openstack Swift (with auth v2)
• Oracle Storage Cloud Service
• SFTP

besides that ones already leveraged by the proxy server solution:

• Local (folders are containers, files are items)
• Amazon S3
• Google Cloud Storage

[thib92]

Note that it's not all or nothing:

Implementing a few providers does not mean that using a plugin-based or a proxy-based approach is not possible.

Turborepo could implement the most needed providers, and leave the more exotic ones to a plugin-based or proxy-based approach. This allows for most users to only depend on Turborepo, and a few to need an additional moving part.

S3 would probably be the best candidate for an in-tree provider, for two reasons:

1. AWS is the most used Cloud provider
2. The S3 API is a standard that has been re-used by multiple late-comers (MinIO, BackBlaze B2, CloudFlare R2, etc.)

The rest can be implemented by plugins or proxies.

This means that there will be some triaging involved here, as repeated requests for additional in-tree providers will likely come. A fixed list of supported providers with a clear statement that no additional providers will be added to the main repo could reduce this a bit.

I'd recommend a plugin-based approach for the less-used providers, as it is quite simple to install (just have a binary with the right name in your path and you're good to go).

[thib92]

I'm also happy to help with the implementation once the correct solution has been established with the core team. I don't have a lot of experience with Go but that will make for some nice practice!

[m19c]

One thing the project manager should consider:
Adding different storage providers will also increase the size of the binary.

[shaulgo]

It would be great if there will be an option for a Node custom storage provider. This could be file/files that expose the correct api.

[djamaile]

@weyert when will you open-source it?