relaycorp / cloud-gateway Goto Github PK
View Code? Open in Web Editor NEWInfrastructure as Code and configuration for all Awala-Internet Gateways run by Relaycorp
License: MIT License
Infrastructure as Code and configuration for all Awala-Internet Gateways run by Relaycorp
License: MIT License
We're currently running it on demand, which means Cloud Run will kill the service when it goes unused for a few minutes.
We're doing this to save costs whilst we're beta testing Awala/Letro, but we should undo this when we go live because it can take ~15 seconds for the service to start, which would be problematic with couriers.
See also #64.
And it shouldn't take more than a second to complete successfully.
This started happening today. All backing services look healthy.
What we see on the Android Gateway logs:
2020-11-25 11:20:51.616 6001-6026/tech.relaycorp.gateway I/DeliverParcelsToGateway: Could not deliver parcels due to server error
tech.relaycorp.relaynet.bindings.pdc.ServerConnectionException: Failed to connect to https://poweb-test.relaycorp.tech:443/v1/parcels
Caused by: io.ktor.network.sockets.SocketTimeoutException: Socket timeout has been expired [url=https://poweb-test.relaycorp.tech:443/v1/parcels, socket_timeout=unknown] ms
Caused by: java.net.SocketTimeoutException: timeout
What we see on the LB logs:
I don't think that's possible but I'm checking with GCP: https://console.cloud.google.com/support/cases/detail/26077434?project=relaycorp-cloud-gateway
Node/pod affinity is overly complex, error-prone and brittle. Especially on the gateway, given that it has 5 components.
Pod Topology Spread Constraints solves this issue, but it requires Kubernetes 1.19, which isn't yet supported by GKE.
A HA MongoDB cluster is too complicated and expensive for what it's used for today, which is mostly to store public keys and certificates.
Use GCP Firestore instead, which is fully managed by GCP and doesn't require any capacity planning/monitoring.
TODO
FirestorePublicKeyStore
.FirestoreCertificateStore
.FirestorePublicKeyStore
and FirestoreCertificateStore
in the gateway.See also:
Blocked by nats-io/k8s#174
The community builder for helmfile isn't great:
kubectl
from hooks because gcloud
isn't where the kube context says it is.helm diff
doesn't actually work.I had to create https://github.com/relaycorp/cloud-gateway/blob/d18b95cd2590fea7bedfa4744a12eeb81e51a7e2/charts/scripts/helmfile.sh and patch the builder to work around most of the issues above.
Output: https://github.com/relaycorp/cloud-gateway
*-chart
repos into respective app repos.helmfile diff
on PRs.Both the CogRPC server and the certificate rotator job are failing because PKI.js is trying to export a public key from a private key using the private key's parameters (instead of actually export()
ing the private key to SPKI).
GCPKeystoreError: Private key cannot be exported (requested format: jwk)
at .GcpKmsRsaPssProvider.onExportKey ( /opt/gw/node_modules/@relaycorp/awala-keystore-cloud/src/lib/gcp/GcpKmsRsaPssProvider.ts:36 )
at .GcpKmsRsaPssProvider.exportKey ( /opt/gw/node_modules/webcrypto-core/build/webcrypto-core.js:220 )
at .SubtleCrypto.exportKey ( /opt/gw/node_modules/webcrypto-core/build/webcrypto-core.js:1465 )
at .CryptoEngine.exportKey ( /opt/gw/node_modules/pkijs/build/index.js:5555 )
at .derSerializePublicKey ( /opt/gw/node_modules/@relaycorp/relaynet-core/src/lib/crypto/keys/serialisation.ts:17 )
at .getRSAPublicKeyFromPrivate ( /opt/gw/node_modules/@relaycorp/relaynet-core/src/lib/crypto/keys/generation.ts:55 )
at .InternetGatewayManager.get ( /opt/gw/node_modules/@relaycorp/relaynet-core/src/lib/nodes/managers/NodeManager.ts:46 )
at process.processTicksAndRejections ( node:internal/process/task_queues:95 )
at .InternetGatewayManager.getCurrent ( /opt/gw/src/node/InternetGatewayManager.ts:37 )
at .<anonymous> ( /opt/gw/src/queueWorkers/crcIncoming.ts:48 )
https://console.cloud.google.com/errors/detail/CMyRtJ_Opc7mHg?project=gw-frankfurt-4065
And whilst I'm at it, configure the bucket to have objects expire after 48 hours (for the alpha testing).
Depends on relaycorp/awala-gateway-internet#200
For example, we should:
We must prove that our cloud infrastructure matches the Docker images, Kubernetes resources and Terraform resources in our open source repositories. Simply asking people to trust us is not an option: They must have the certainty that we're not spying on them, selling their data, running a mass surveillance programme for the Five Eyes or censoring people.
Another (equally important) reason to do this is to protect the Relaycorp SRE team from powerful adversaries who might secretly try to force us (collectively or individually) to give away certain metadata or censor certain users/services. If every change or external access to the infrastructure is independently verified, this threat should be avoided. (Unfortunately, an attacker unaware of this measure may still target us, but we can mitigate this by advertising very prominently the fact that our cloud infrastructure is independently verified.)
We basically have to prove two things: That our cloud infrastructure is exactly what people can find on GitHub, and that we don't have any backdoors.
Indeed, those two properties make Relaynet apps immune to a wide range of privacy threats you'd tend to find in Internet apps. However, we could theoretically still infer the following:
https://relaynet.twitter.com
. This doesn't apply to decentralised services, or centralised services where the provider chooses to run their server-side app behind a Relaynet-Internet Gateway.Additionally, we need to log the IP addresses from end users and couriers so that we ensure our systems are being used fairly and to triage production issues.
Finally, it's likely we'll eventually have to block certain centralised services to comply with UK/US legislation, so in this case we'd have to prove that we're only blocking the services listed publicly. (This doesn't apply to decentralised services, which we could never block)
In an ideal world, our cloud providers (Terraform Cloud, Kubernetes, GCP, Mongo Atlas and Cloudflare) would use a tool like Google Trillian to log provisioning, deprovisioning and access events. This would allow us to broadcast logs so that anyone anywhere can verify the integrity of our cloud infrastructure.
We'd essentially be moving the provenance issue up in the chain, and it'd be up to cloud providers to honour their contractual obligations with Relaycorp and comply with applicable legislation. They'd have a lot to lose if they don't.
But this option isn't really an option in the foreseeable future.
They'd basically get read-only access to the configuration of our cloud resources (but no access to the data inside), as well as their (de)provisioning and access logs. With this level of access, they could operate a system 24/7 to monitor our cloud resources and make sure they match the public Docker images, Kubernetes resources and Terraform resources.
I don't think a software tool like this exists yet, so we'll have to build it and make it open source. This tool has to be trivial to deploy, run and upgrade.
This tool should effectively make sure that provisioning and deprovisioning events match changes to cloud resources on GitHub. Additionally, the tool could also consume access logs so this independent party can be alerted to any direct access to the DB (for example) -- If we need to access the DB, we should justify that access to them (e.g., investigating a security vulnerability).
This would make offsite backups tricky, because we'd need a secret key to decrypt the backups if we need to restore them. One way to address this is by splitting the key, and having their part of the key available on demand in the tool. But this would introduce two additional challenges:
We'd leverage the tool described in Option B, but we'd deploy it ourselves to a separate GCP project whose audit logs are publicly available.
Publishing audit logs is a bit risky, since they might (occasionally) contain sensitive information or PII about Relaycorp staff, which is why we're not making audit logs publicly available in the GCP projects hosting the services.
This option has no dependencies on third parties so it seems like the most likely approach to begin with.
There are many more things we have to do to gain people's trust, including non-technical measures such as transparency when dealing with law enforcement (I think Signal is an example to follow in this regard).
Blocked by nats-io/k8s#169
For security (e.g., mTLS) and observability reasons.
We'll probably just use istio as GKE supports it out of the box.
Note: Knative comes with service mesh built-in (see relaycorp/awala-gateway-internet#761).
Because of nats-io/k8s#167
It also means that the password is included in the GCB logs, so we must reset the password when this is fixed.
Blocked by hashicorp/vault-helm#424
NATS Streaming is a delight to use in development, but less so to deploy and operate (especially when using Kubernetes + GitOps):
initdb
script) and the overall insecurity of the chart (e.g., storing sensitive values in the clear instead of using secrets).This is what the systems architecture looks like today:
This is what it'll look like afterwards:
Kafka is too expensive at +$1.2k/month for the simplest possible HA cluster. Kafka also feels like a massive overkill for what it'd be used for.
Instead of a password managed by Terraform.
Here's how to do this with a sidecar: https://cloud.google.com/sql/docs/postgres/connect-kubernetes-engine
Which is consistent with their documentation: https://cloud.google.com/kubernetes-engine/docs/concepts/ingress-xlb#support_for_websocket
The only "solution" I could find involves setting a global limit on the LB backend, which we can't use as that'd affect the regular HTTP endpoints in the PoWeb service. I'm checking with GCP if there's an alternative to this, but I doubt it: https://console.cloud.google.com/support/cases/detail/26133079?project=relaycorp-cloud-gateway.
I think the long-term solution should be to move the parcel collection endpoint to a separate service (relaycorp/awala-gateway-internet#324). But for now, we'll have to get private gateways to reconnect.
failed to create resource: clusterroles.rbac.authorization.k8s.io "vault-agent-injector-clusterrole" is forbidden: user "<redacted>@cloudbuild.gserviceaccount.com" (groups=["system:authenticated"]) is attempting to grant RBAC permissions not currently held:
{APIGroups:["admissionregistration.k8s.io"], Resources:["mutatingwebhookconfigurations"], Verbs:["get" "list" "watch" "patch"]}
I've just raised a P2 case with GCP: https://console.cloud.google.com/support/cases/detail/26015995?orgonly=true&project=relaycorp-cloud-gateway
As a workaround, I'm going to grant GCB the GKE cluster admin role.
Once #86 is done, a whole HA MongoDB cluster would only be used to store 2-3 entries in a key/value table... Which is a bit of an overkill.
Use a Kubernetes ConfigMap
for this key/value configuration. It'd be good to implement the core of it as a separate library so it can also be used in relaycorp/awala-pong#26
TODO:
keyev-firestore
, replacing MongoDB.See also:
For example, the Frankfurt environment should live in its own dedicated GCP project.
This is needed because Datastore/Firestore ties the whole project to a region ๐คฆ๐พโโ๏ธ : https://cloud.google.com/datastore/docs/locations#selecting_a_location
That would explain relaycorp/relaynet-gateway-android#724
Vault is a complex system to maintain, especially in production, as it uses many Kubernetes resources, a GCS bucket and a KMS keyring.
Use GCP KMS in production, whilst keeping Vault in development.
This is what we'd need to do:
The Relaynet spec allows messages to be kept for up to 6 months, but I'm configuring GCS in #65 to delete messages after a small number of days whilst we're in an alpha phase.
Keep the messages for a few months.
This annotation:
Will cause the following provisioning issue:
Error during sync: error running load balancer syncing routine: loadbalancer <whatever> does not exist: invalid ingress frontend configuration, please check your usage of the 'kubernetes.io/ingress.allow-http' annotation.
GCP support issue: https://console.cloud.google.com/support/cases/detail/26077051?project=relaycorp-cloud-gateway
See also:
I've just emptied the bucket by mistake, and it would've been kind of nice to have a backup to restore. But that isn't the case, so I had to start afresh, which means that Android Gateway users must now reset their app's data.
Vault's GCS bucket to be backed up every 8-24 hours (TBD). And a way to restore the backup when needed.
I didn't do it as part of the provisioning because deprovisioning is a risky endeavour and has to be considered properly.
I'm inclined to solve this in two steps:
prevent_destroy
and resource-specific equivalents.terraform destroy
(see next step).That way, deprovisioning an environment will involve the following steps:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.