Syft decouples private data from model training, using Federated Learning, Differential Privacy, and Encrypted Computation (like Multi-Party Computation (MPC) and Homomorphic Encryption (HE)) within the main Deep Learning frameworks like PyTorch and TensorFlow. Join the movement on Slack.
Most software libraries let you compute over the information you own and see inside of machines you control. However, this means that you cannot compute on information without first obtaining (at least partial) ownership of that information. It also means that you cannot compute using machines without first obtaining control over those machines. This is very limiting to human collaboration and systematically drives the centralization of data, because you cannot work with a bunch of data without first putting it all in one (central) place.
The Syft ecosystem seeks to change this system, allowing you to write software which can compute over information you do not own on machines you do not have (total) control over. This not only includes servers in the cloud, but also personal desktops, laptops, mobile phones, websites, and edge devices. Wherever your data wants to live in your ownership, the Syft ecosystem exists to help keep it there while allowing it to be used privately for computation.
The current stable release is 0.6.0
which is available on:
For many use cases you can simply use:
$ pip install syft
If you are doing the Private AI Series or you are an external party developing against Syft and Grid please use the syft_0.6.0
branch.
This is the dev
branch and to accommodate our need to experiment with new ideas and implementations we will be moving a few things around during the early stages of 0.7.0 beta
. Currently the core syft
library and code will remain fairly stable, while we do some much needed quality improvements and refactors to the grid
codebase and its tooling for deployment and orchestration of nodes.
During the process of development we will be moving examples from the /packages/syft/examples
folder down to the /notebooks
folder and ensuring they are working and tested with the latest dev
code.
This repo contains multiple sub-projects which work together.
OpenMined/PySyft
βββ README.md <-- You are here π
βββ notebooks <-- Notebook Examples and Tutorials
βββ packages
βββ grid <-- Grid - A network aware, persistent & containerized node running Syft
βββ syft <-- Syft - A package for doing remote data science on private data
To read more about what Syft is please consult the current 0.6.0
Documentation.
To read more about what Grid is please consult the old PyGrid README until we finish writing the new one.
- docker
- tox
- python 3.8+
You will need docker
and docker-compose
to do development on the monorepo
tooling.
Run the FastAPI Dev environment using:
$ cd packages/grid
$ source .env && docker compose up
$ cd packages/grid
$ docker compose build
You will need tox
to run some of our build and test tools.
$ pip install tox
$ tox -l
You should see the following:
syft.jupyter
syft.lint
syft.test.fast
syft.test.libs
syft.test.security
These commands can be run like so:
$ tox -e syft.lint
We are providing a simple way to deploy all of our stack inside a single VM so that no matter where you want to run everything you can do so easily by thinking in terms of a single machine either bare metal or VM and have it provisioned and auto updated.
To develop against this locally you will want the following:
- vagrant
- virtualbox
- ansible
- hagrid <-- in packages/hagrid
You can install HAGrid with pip:
$ pip install hagrid
$ brew install vagrant virtualbox ansible
Hagrid the Grid deployment tool:
$ cd packages/hagrid
$ pip install -e .
Vagrant allows us to create and manage VMs locally for development. During the startup
process of creating the VM the ansible provisioning scripts will be applied automatically
to the VM. If you change the Vagrantfile which describes how the VM is defined you will
need to either vagrant reload
or destroy and re-create it.
Making changes to the VM state should be done through the ansible
scripts so that
the state of the box is idempotent and re-running the ansible provisioning scripts
should always result in the same working grid node state.
To allow rapid development we mount the PySyft source repo into the VM at the path:
/home/om/PySyft
which is where it would be if it was cloned down on a real remote VM.
The configuration is done via a Vagrantfile
which is written in ruby.
The VM will be accessible on the IP 10.0.1.2
which is defined in the Vagrantfile
.
The Landrush plugin for vagrant gives us an automatic dns service so we can access our local VM as though it were a real live domain on the internet.
$ vagrant plugin install landrush
With this enabled you can access the box on:
http://node.openmined.grid
NOTE: You may need your sudo password to enable the landrush DNS entry on startup.
$ cd packages/grid
$ vagrant up --provision
You want to do this any time you are testing out your ansible
changes.
$ cd packages/grid
$ vagrant provision
If you want to do a quick deploy where you skip the system provisioning you can run:
$ ANSIBLE_ARGS='--extra-vars "deploy_only=true"' vagrant provision
$ cd packages/grid
$ vagrant ssh
You can enable TLS in HAGrid by passing in the --tls param:
$ hagrid launch domain to docker:8081+ --tls
This will go looking for a certificate and private key here:
packages/grid/traefik/certs/key.pem
packages/grid/traefik/certs/cert.pem
These files and their settings are defined in:
packages/grid/traefik/dynamic-configurations/certs.yaml
During development you will need to generate ones which match localhost and then enable --test mode so that these invalid self-signed certificates are accepted by various code and libraries.
First, get the tool mkcert
.
$ brew install mkcert
cd PySyft
export GRID=$(pwd)/packages/grid && export CAROOT=$GRID/tls && export CERTS=$GRID/traefik/certs
mkcert -cert-file="$CERTS/cert.pem" -key-file="$CERTS/key.pem" '*.openmined.grid' docker-host localhost 127.0.0.1 ::1
This will have created the certificate and private key as well as placed the root signing certificate (think fake SSL authority who can charge for SSL certs) in the following dir:
packages/grid/tls/rootCA-key.pem
packages/grid/tls/rootCA.pem
To ensure that tailscale will accept these certs we mount the file into the tailscale container like so:
version: "3.8"
services:
tailscale:
volumes:
- ./tls/rootCA.pem:/usr/local/share/ca-certificates/rootCA.pem
The startup script runs update-ca-certificates
so that the tailscale container is now
aware of this fake authority and will accept the fake cert you have created for it.
If you wish to visit web pages with localhost:8081
or network1.openmined.grid
and
have the TLS certificate warning disappear you need to install the certificate with:
$ mkcert -install
Alternatively as we do in the integration tests you can pass it as an environment variable to programming languages like Python like so:
REQUESTS_CA_BUNDLE=packages/grid/tls/rootCA.pem pytest tests/integration ...
When you do this, python will also accept these certificates in libraries like requests.
Alternatively you can also tell requests to ignore invalid certificates with the verify
kwarg like so:
import requests
request.get(url, verify=False)
To make this more convenient we have added an ENV called IGNORE_TLS_ERRORS
which we
set using the --test
param in hagrid
like so:
$ hagrid launch test_network_1 network to docker:9081 --tail=false --tls --test
You can check for this with sy.util.verify_tls()
like so:
import requests
import syft as sy
request.get(url, verify=sy.util.verify_tls())
It is important not to run IGNORE_TLS_ERRORS=true
in production.
Normally web traffic is served over port 80
for http
and port 443
for https
.
Naturally during development we need to use multiple ports for multiple stacks and bind
them to localhost
and use things like docker-host
to resolve this global address
space from within any isolated containers. Currently what we do is if the port
you
supply to hagrid
is 80
then we assume you are running in production and want port
443
for https
. If you use any other port we will automatically find a port from
444
onwards and this port will be included in the http
-> https
301 redirect in
traefik proxy. Because of this most browsers and network libraries will simply follow
this redirect so you can continue to use the normal http
ports for everything.
The Grid API inside Syft detects if a url and port combination provided gets redirected
to https
and will change the base_url
which should mean that login credentials are
sent over https
not http
where possible.
Due to the fact that TLS Certificates are only valid for domains and not IPs and the
VPN is currently configured to use IPs only, we do not redirect or serve https
over
the VPN. The traffic being sent by wireguard
over the VPN is already encrypted so
there should be no need for TLS
. The way this works is, when Grid is using TLS
we
forward external port 80
traffic to port 81
and then use this for the 301 redirect
to the TLS
port 443+
. If traffic arrives on port 80
inside the cluster it does not
get redirected which allows the VPN IPs to respond via http
.
To install certs via hagrid
simply supply the two cert files as arguments:
$ hagrid launch domain to azure --tls --upload_tls_key=/path/to/certs/key.pem --upload_tls_cert=/path/to/certs/cert.pem
Alternatively if your machine is already setup without TLS you can simply reprovision with:
$ hagrid launch node_name domain to 123.x.x.x --tls --upload_tls_key=/path/to/certs/key.pem --upload_tls_cert=/path/to/certs/cert.pem
We also have an example Quickstart Template for use in your own automation pipelines.
HAGrid supports creating VMs and provisioning with PyGrid directly using:
$ hagrid launch to azure
Follow the prompts and enter resource groups, size, location, auth credentials etc.
HAGrid supports creating VMs and provisioning with PyGrid directly using:
$ hagrid launch to gcp
Follow the prompts and enter project id, machine type, zone, auth credentials etc. Note Since we use the gcloud cli tool for provisioning we rely on the gcloud cli tool generated SSH keys so if you use a custom key path other than the default (~/.ssh/google_compute_engine) please specify the correct path.
Create a VM on your cloud provider with Ubuntu 20.04 with at least:
- 2x CPU
- 4gb RAM
- 40gb HDD
Generate or supply a private key and note down the username.
Run the following:
$ hagrid launch node --type=domain --host=104.42.26.195 --username=ubuntu --key_path=~/.ssh/key.pem
If you want to later skip the setup process of installing packages and docker engine etc you can pass in --mode=deploy which will skip those steps.
If you wish to use a different fork of PySyft you can pass in --repo=The-PET-Lab-at-the-UN-PPTTT/PySyft --branch=ungp_pet_lab
$ sudo su - om
We are using Packer to build cloud images in a very similar fashion to the dev Vagrant box.
To build images you will need the following:
- packer
- vagrant
- virtualbox
- ansible
$ brew install packer vagrant virtualbox ansible
Go to the following directory:
cd packages/grid/packer
Run:
./build_vagrant.sh
What this does is first build the base image, by downloading a Ubuntu .iso and automating an install to a virtual machine. After the base image is created, the same ansible provisioning scripts that we use in HAGrid and the Vagrant Dev environment above are run against the image and finally a few shell scripts are executed to update some Ubuntu packages and clean out a lot of unused stuff to squeeze the image size down.
To verify it worked you can start the Vagrant file like this:
cd packages/grid/packer
vagrant up
This system will start and automatically have the stack running and available on the local ip http://10.0.1.3/ you can also SSH into this box using the credentials in the Vagrantfile.
To create the azure cloud image you need to have the az
cli tool and make sure you are authenticated.
Install the CLI tool:
$ pip install az
Authenticate your CLI tool:
$ az login
You will need to use a resource group and create a storage account within that resource group.
Create a resource group called: openmined-images
$ az group create -n openmined-images -l westus
Create an app to use within the packer file:
$ az ad sp create-for-rbac --name openmined-images > azure_vars.json
This will create a file called azure_vars.json
which will look something like this:
{
"appId": "21b92977-8ad0-467c-ae3a-47c864418126",
"displayName": "openmined-images",
"password": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"tenant": "e3f9defa-1378-49b3-aed7-3dcacb468c41"
}
You need to know your subscription_id
:
$ SUBSCRIPTION_ID=$(az account show --query id | tr -d '"')
You can now build the image:
$ ./build_azure.sh ${SUBSCRIPTION_ID}
Create a Shared image gallery within Azure.
We provide an option to deploy the stack using kubernetes.
To test and run this locally we use k3d
and devspace
.
- docker
- k3d
- devspace
- kubectl
- k9s
We will be using docker however you do not need to enable kubernetes
in your docker desktop app.
If its enabled, disable it and click Apply & Restart
.
Instead we will use k3d
which will create and manage all the k8s resources we require as a normal container in docker engine.
$ brew install k3d
k3d lets you create a test registry and cluster like so
$ export NODE_NAME=test-network-1
$ export NODE_PORT=9081
$ k3d cluster create $NODE_NAME -p "$NODE_PORT:80@loadbalancer" --registry-use k3d-registry.localhost
$ k3d cluster start $NODE_NAME
If you ever need to reset things you can just kill the docker containers and volumes:
$ docker rm $(docker ps -aq) --force
$ docker volume prune --force || true
Once k3s is running you should see the container in docker.
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
9b765411a2f2 rancher/k3s:v1.22.7-k3s1 "/bin/k3d-entrypointβ¦" 20 minutes ago Up 20 minutes k3d-test-network-1-server-0
878bc1bd630c registry:2 "/entrypoint.sh /etcβ¦" 20 minutes ago Up 20 minutes 0.0.0.0:12345->5000/tcp k3d-registry.localhost
kubectl is the CLI tool for kubernetes. If you have ran k3s it should have configured your kubectl to have a context for the local k3s cluster by default.
You should be able to run something like:
$ kubectl get all
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 45h
Think of a namespace as a grouping of resources and permissions which lets you easily create and destroy everything related to a single keyword.
$ kubectl get namespaces
NAME STATUS AGE
default Active 45h
kube-node-lease Active 45h
kube-public Active 45h
kube-system Active 45h
kubernetes-dashboard Active 45h
All k8s have a default namespace and the other ones here are from kubernetes.
The most popular way to deploy applications to k8s is with a tool called Helm. What helm aims to do is to provide another layer of abstraction over kubernetes yaml configuration with hierarchical variables, templates and a package definition which can be hosted over HTTP allowing custom applications to depend on other prefabricated helm charts or to provide consumable packages of your code as a helm chart itself.
To make development and deployment of our kubernetes code easier we use a tool called devspace
which aims to be like a hot reloading dev optimised version of docker compose
but for kubernetes. https://devspace.sh/
Additionally devspace
allows us to deploy using helm by auto-generating the values and charts from the devspace.yaml which means the single source of truth can be created which includes both production helm charts and kubernetes yaml configuration as well as local dev overrides.
$ brew install devspace
Now run the dev
command with devspace
:
To run a network with headscale VPN:
$ cd packages/grid
$ devspace dev -b -p network
To run a domain without the headscale VPN:
$ cd packages/grid
$ devspace dev -b -p domain
To deploy to your local k3s do the following:
$ export NODE_NAME=test-network-1
$ export NODE_PORT=9081
$ devspace --no-warn --kube-context "k3d-$NODE_NAME" --namespace $NODE_NAME \
--var DOMAIN_NAME=$NODE_NAME \
--var NETWORK_CHECK_INTERVAL=5 \
--var CONTAINER_REGISTRY=k3d-registry.localhost:12345/ \
deploy -b -p network
The important thing to note is the custom container registry. This will tag the images with that local registry and k3s will know how to pull them from its custom registry.
You may need to add this to your /etc/hosts
file if you are not on linux:
127.0.0.1 k3d-registry.localhost
There is a tool like ctop which lets you easily see the kubernetes cluster called k9s.
Install it like so:
$ brew install derailed/k9s/k9s
When you run it you can select the local context k3d-test-network-1 and see the namespace you deployed.
$ devspace purge
The database and the VPN containers have persistent volumes. You can check them with:
$ kubectl get persistentvolumeclaim
And then delete PostgreSQL with something like:
$ kubectl delete persistentvolumeclaim app-db-data-db-0
This will show all the unique images and their tags currently deployed which is useful when debugging which version is actually running in the cluster.
$ kubectl get pods --all-namespaces -o jsonpath="{.items[*].spec.containers[*].image}" | tr -s '[[:space:]]' '\n' | sort | uniq -c
Get all the deployments
$ kubectl get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
backend 1/1 1 1 18m
backend-stream 1/1 1 1 18m
backend-worker 1/1 1 1 18m
frontend 1/1 1 1 18m
queue 1/1 1 1 19m
Restart the backend-worker
$ kubectl rollout restart deployment backend-worker
Configure kubectl context with GKE:
$ gcloud container clusters get-credentials --region us-central1-c staging-cluster-1
Check that you have the correct context
$ kubectx
Configure your Google Container Registry (GCR):
$ gcloud auth configure-docker
Check your settings with print
$ devspace print -p domain --var=CONTAINER_REGISTRY=gcr.io/reflected-space-315806/
You should see that you are creating a domain and that the container registry variable changes the image name to:
images:
backend:
image: gcr.io/reflected-space-315806/openmined/grid-backend
This will tell devspace
to publish to the GCR for your active GCP project.
Create the openmined namespace
$ kubectl create namespace openmined
Tell devspace to use the openmined namespace
$ devspace use namespace openmined
Deploy to GKE:
$ devspace deploy -p domain --var=CONTAINER_REGISTRY=gcr.io/reflected-space-315806/
Access a container directly:
$ devspace enter
Attach to container stdout:
$ devspace attach
Use port forwarding to access an internal service:
$ kubectl port-forward deployment/tailscale :4000
Create a cluster and optional registry.
I named my registry: omazuretest
Log into the registry with:
$ az acr login -n omazuretest.azurecr.io
Check you have permission by tagging and uploading nginx:
$ docker pull nginx:latest
$ docker tag nginx omazuretest.azurecr.io/nginx
$ docker push omazuretest.azurecr.io/nginx
devspace --no-warn --kube-context "azure-test" --namespace azure-network-1 --var DOMAIN_NAME=azure-network-1 --var CONTAINER_REGISTRY=omazuretest.azurecr.io/ build -b
devspace --no-warn --kube-context "azure-test" --namespace azure-network-1 --var DOMAIN_NAME=azure-network-1 --var CONTAINER_REGISTRY=omazuretest.azurecr.io/ deploy -p network
You can just output plain kubernetes manifests like so:
For domains:
$ devspace render -p domain --skip-build --silent > k8s/rendered/domain.yaml
For networks:
$ devspace render -p network --skip-build --silent > k8s/rendered/network.yaml
To publish hagrid to pypi create a virtualenv in the /packages/hagrid
directory.
$ cd packages/hagrid
Use a tool like pipenv
or manually create and source like so:
$ python3 -m venv ./venv
$ source .venv/bin/activate
Install some tools:
$ pip install --upgrade bandit safety setuptools wheel twine tox
Bump the version inside /packages/hagrid/hagrid/__init__.py
Build a wheel:
$ ./build_wheel.sh
Check with twine:
$ twine check `find -L ./ -name "*.whl"`
Upload with twine:
$ twine upload dist/hagrid-x.x.x-py3-none-any.whl
$ cd packages/hagrid
Build and tag the images:
$ ./build_docker.sh
Publish to docker hub:
$ docker push openmined/hagrid:latest
$ docker push openmined/hagrid:x.x.x
Also, join the rapidly growing community of 12,000+ on Slack. The Slack community is very friendly and great about quickly answering questions about the use and development of PySyft!
This software is in beta. Use at your own risk.
For support in using this library, please join the #support Slack channel. Click here to join our Slack community!
We are very grateful for contributions to Syft and Grid from the following organizations!
![]() |
![]() |
|
![]() |
![]() |
![]() |