Using google cloud services about sorry-cypress

[francescov1]

Sorry for the delay! Here you go, let me know if you want any additional details:

• My setup runs on GCP Cloud Run, 1 cloud run service for each of the sorry-cypress services.
• Cypress framework runs in Github actions - ill explain how I set this up as well since there was a few gotchas
• Mongo hosted in Atlas, no IP whitelist since it would require a NAT on the GCP side, credential-based authentication
• The only thing I haven't been able to get to work is image and video uploads (haven't really spent any time debugging this though, I think its likely a straight forward fix)
• Used "Unauthenticated access" option for the cloud run services, ideally we can tighten up security in the future based on my previous comments.

1. I based my setup off the docker-compose.full.yml file (although didn't actually use it), ignoring the Mongo piece as this was done using Atlas. Dont stress about getting the service URL environment variables (DASHBOARD_URL, GRAPHQL_SCHEMA_URL) right from the start, once each service has generated their domain names its easy to go back and update them all.
2. Build each of the 3 images and push to GCR: gcloud builds submit --tag gcr.io/<project-id>/sorry-cypress-<api | dashboard | director>:latest
3. Create API services. Very straight forward with Cloud run. All you need here is select the container image gcr.io/<project-id>/sorry-cypress-api, set the MONGODB_URI and MONGODB_DATABASE variables, and set port to 4000.
4. Create dashboard service. Grab the generated domain to the API service and set it as the GRAPHQL_SCHEMA_URL env var. Run on port 8080. I also added the CI_URL here (optional), for Github actions this looked like Github Actions,https://github.com/<user>/<repo>/actions/runs/{build_id}. I found I had to double the default memory allocated to the container (256 MB -> 512 MB) for this service, it would just crash with the default 256 MB.
5. Create director service. Grab the generated dashboard service domain and set as DASHBOARD_URL. EXECUTION_DRIVER and SCREENSHOTS_DRIVER can stay the same as in docker-compose.full.yml. Set AWS/S3 credentials. I also added the ALLOWED_KEYS variable (optional). Run on port 1234

Everything should be working now!

Quick explanation of GitHub actions:

• I didn't find a way to use the offical Cypress Action, I tried overwriting the Cypress api_url but couldn't get it to work. I think there's likely a way but I don't have a great understanding of how a Github Action gets injected into the build environment.
• There is no great way to get a CI_BUILD_ID that changes if your parallel builds fail and you re-run them from Github (found lots of frustrated people online about this lol), which can be a problem for this setup. I ended up digging into the Cypress Action code and finding a script they use for this, copied the snippet into a similar script for my repo, pasted below getCiBuildId.js (note that I just commented out the lines Cypress had in there that didn't apply to this use case). Then, I added this build step prior to running cypress test (note the GITHUB_TOKEN variable, which is required to be passed in):

- name: Set CI_BUILD_ID
   run: |
     CI_BUILD_ID=$(node scripts/getCiBuildId.js)
     echo "CI_BUILD_ID=$CI_BUILD_ID" >> $GITHUB_ENV
   env: 
     GITHUB_TOKEN: ${{ github.token }}

This sets CI_BUILD_ID as an env variable for future build steps.

• In the next build step I run my cypress tests. Before doing this (ie yarn run test:integration) I added the command to replace the Cypress api url: sed -i -e 's|api_url:.*$|api_url: "<director-url>"|g' /home/runner/.cache/Cypress/*/Cypress/resources/app/packages/server/config/app.yml. I think that command should be pretty universal (since a wildcard is used for the version) but not sure if theres other possible folder structures for github actions. I found the "one-liner" in the docs didn't work for me here.

In the actual test command, just need to make sure to provide --ci-build-id $CI_BUILD_ID

And that's it! I set this all up a couple weeks ago, so if anyone tries this and runs into issues I'm happy to colour in any missing details. Hope that helps 😀

getCiBuildId.js

// NOTE: taken from https://github.com/cypress-io/github-action/blob/master/index.js
// github actions does not provide a way to get a unique build id (unique between reruns of a job)
const { Octokit } = require("@octokit/core");

const getCiBuildId = async () => {
    const {
        GITHUB_WORKFLOW,
        GITHUB_SHA,
        GITHUB_TOKEN,
        GITHUB_RUN_ID,
        GITHUB_REPOSITORY
    } = process.env;

    const [owner, repo] = GITHUB_REPOSITORY.split("/");
    // let branch;
    let parallelId = `${GITHUB_WORKFLOW} - ${GITHUB_SHA}`;

    if (GITHUB_TOKEN) {
        // console.log(`Determining build id by asking GitHub about run ${GITHUB_RUN_ID}`);

        const client = new Octokit({
            auth: GITHUB_TOKEN
        });

        // const resp = await client.request(
        //     "GET /repos/:owner/:repo/actions/runs/:run_id",
        //     {
        //         owner,
        //         repo,
        //         // eslint-disable-next-line radix
        //         run_id: parseInt(GITHUB_RUN_ID)
        //     }
        // );

        // if (resp && resp.data && resp.data.head_branch) {
        //     branch = resp.data.head_branch;
        //     console.log(`found the branch name ${branch}`);
        // }

        // This will return the complete list of jobs for a run with their steps,
        // this should always return data when there are jobs on the workflow.
        // Every time the workflow is re-run the jobs length should stay the same
        // (because the same amount of jobs were ran) but the id of them should change
        // letting us, select the first id as unique id
        // https://docs.github.com/en/rest/reference/actions#list-jobs-for-a-workflow-run
        const runsList = await client.request(
            "GET /repos/:owner/:repo/actions/runs/:run_id/jobs",
            {
                owner,
                repo,
                // eslint-disable-next-line radix
                run_id: parseInt(GITHUB_RUN_ID)
            }
        );

        if (
            runsList &&
            runsList.data &&
            runsList.data.jobs &&
            runsList.data.jobs.length
        ) {
            const jobId = runsList.data.jobs[0].id;
            // console.log(`fetched run list with jobId ${jobId}`);
            parallelId = `${GITHUB_RUN_ID}-${jobId}`;
        } else {
            // console.log("could not get run list data");
        }
    }

    // console.log(`determined branch ${branch} and parallel id ${parallelId}`);
    // return { branch, parallelId };
    console.log(parallelId);
};

getCiBuildId();

[agoldis]

@francescov1 thanks a lot for this guide. Recently I've got a chance to work more on documentation and used your instructions to compile this guide https://agoldis.gitbook.io/sorry-cypress/cloud-setup/google-cloud

It also includes instruction for enabling screenshots and videos upload. Would appreciate any feedback.

I will also use your post to document for enabling Github Actions. Closing that issue due to https://agoldis.gitbook.io/sorry-cypress/cloud-setup/google-cloud

EDIT: here's the correct link https://docs.sorry-cypress.dev/cloud-setup/google-cloud

[agoldis]

@coltadorsey I hope to have more time in future to figure how to run it easily on google cloud.

[homostellaris]

Hey @agoldis 👋

I was indeed just using the director service, this was for a POC to demonstrate parallelisation of Cypress tests to my employer so we're not using it right now but will be implementing it for real in the near future 🙂

Yep the current solution works fine on App Engine so please do create a guide for it and let me know if I can help. Unfortunately I've just not had time to write the terraform script as I intended 😅

I never took the experiment with Cloud Run any further after I got blocked by Mongo Atlas' requirement for whitelisting client IPs (Cloud Run instances have dynamic IPs). I could have just hosted a MongoDB instance elsewhere but I ran out of time.

[francescov1]

Tried setting this up on a regular GCE VM, everything was pretty smooth and the director seems to be working, but uploads to S3 fail and the dashboard gives me "Error: Failed to fetch" after hanging with "Loading..." for a while (I made sure to change "localhost" to the VMs internal IP in the docker-compose file, still no luck). Any ideas?

Ended up getting it working, there were a couple minor networking issues on my part.

[homostellaris]

Thanks for the info @agoldis 🙂 I'll keep this issue updated with my progress. I've added some notes below to potentially save others going down the same dead-end I did.

---

My initial attempt using Cloud Run was unsuccessful because MongoDB Atlas requires that client IPs are whitelisted and static IPs are currently not possible with the managed version of Cloud Run. Apparently support for this is coming in future.

The simplest option would just be to deploy to Google App Engine (PAAS) which has a free tier but I too am still curious about the stateless setup. This is still possible if we deploy our own instance of MongoDB for the Cloud Run instance of directory to talk to. This is probably best deployed as a container to a compute engine instance running the container-optimized OS.

[agoldis]

@francescov1 that's awesome! I'll update the documentation site with those instructions

[homostellaris]

I'm currently looking into this and planning to use Cloud Run (serverless container service) and MongoDB Atlas' free sandbox tier (can be partially managed through Google Cloud).

I've had a brief look at the code and can't see any reason this shouldn't be able to run statelessly when the MongoDB driver is used, is that correct @agoldis ?

P.S. Many thanks for the wonderful work on this repo and the brilliant name 😂

[agoldis]

@moderatemisbehaviour Hi Daniel!

I guess you're talking about the "director" service with MongoDB drivers for persistency.

You can start the service on-demand in serverless environment and point all the requests to temporary service. It might be a bit challenging to discover the service address for configuring clients.

I never actually tried to run it in stateless setup, I am very curious about that case, though!

In theory you could also have multiple "director" service instances - the implementation is concurrency-aware - i.e. multiple instances of the same service should be able to run concurrently without messing with the data.

Tell me if that help and if you need any help with the setup.

P.S. 😃

[homostellaris]

Very easy to deploy the director to GAE standard environment, I have it working on my fork.

I'd like to write a terraform script that will set this up on Google Cloud along with the dashboard, API, and database. Would you be open to a PR at some point @agoldis ?

[agoldis]

@moderatemisbehaviour terraform would be awesome! I don't have much experience with it, but AFAIK that will allow deploying to different cloud providers, am I right?

[homostellaris]

Yes indeed! Although I think there is still a layer of vendor-specific configuration required.

[agoldis]

Hey @moderatemisbehaviour, are you still using this project? I have seen the changes on your fork, it seems like you're only using the director service.

• Did you have any success with Cloud Run?
• Is the current solution working with App Engine?

I am thinking about creating a tutorial, guide based on your progress.

[francescov1]

Tried setting this up on a regular GCE VM, everything was pretty smooth and the director seems to be working, but uploads to S3 fail and the dashboard gives me "Error: Failed to fetch" after hanging with "Loading..." for a while (I made sure to change "localhost" to the VMs internal IP in the docker-compose file, still no luck). Any ideas?

[francescov1]

Was able to also get this running nicely on Cloud Run. Essentially each of the three services gets their own Cloud Run Service, and a mongo instance can be spun up in Mongo Atlas with an IP whitelist of "0.0.0.0/0". Not sure if there's a trivial way to IP whitelist the cloud run services from Mongo Atlas (as per @moderatemisbehaviour's issue) but you can still implement one of Atlas' other forms of auth.

Something I would like to explore is secure communication between services (ie CORS & possibly oauth for dashboard->api, IAM for CI->director). Has anyone looked into this?

By the way, amazing job on this, it's a life saver and provides everything we need 🙌. Love seeing & using projects like this.

[agoldis]

Great to see your success! I am hoping to eventually write a guide / tutorial at https://sorry-cypress.dev and also enable gc storage in addition to S3. BTW, if you feel like writing a few paragraphs / sharing your setup - it might be a good start for documentation.

Something I would like to explore is secure communication between services (ie CORS & possibly oauth for dashboard->api, IAM for CI->director). Has anyone looked into this?

There's definitely a demand for better security and privacy, I have seen some people using reverse proxies, but no standard way / best practice has been formed so far

[francescov1]

Sure thing! I'll be happy to write that up in the next few days 😀

Something I would like to explore is secure communication between services (ie CORS & possibly oauth for dashboard->api, IAM for CI->director). Has anyone looked into this?

There's definitely a demand for better security and privacy, I have seen some people using reverse proxies, but no standard way / best practice has been formed so far

Ahh okay. Would love to help with this or at least give some suggestions. Not super familiar with this codebase but I'll poke around a bit!

[lslavkov]

@agoldis hey I can't access to your gitbook for cloud setup.

[agoldis]

@lslavkov it's working https://docs.sorry-cypress.dev/cloud-setup/google-cloud just verified... what are you seeing?

[dzzk]

Thank you, guys. You doing great feature implementation for GCP. Did anyone do something to get into GCP Cloud Logging? If not, I would be glad to participate also on this part contribution.

As I see we can have some hooks also to store logging to the GCP CL instead of Web Panel over MongoDB.

[agoldis]

Thank you, guys. You doing great feature implementation for GCP. Did anyone do something to get into GCP Cloud Logging? If not, I would be glad to participate also on this part contribution.

As I see we can have some hooks also to store logging to the GCP CL instead of Web Panel over MongoDB.

Hi @dzzk thanks for suggesting your help. I am not very familiar with GCP internals and only implemented the bare minimum for the setup. Please go ahead and propose the changes to the setup that you consider useful