Hi,
Great idea and I'm liking the ease of implementation. The company I'm working with at the moment has deployed Openshift 3 rather than pure kubernetes, so I was hoping to have native support for Openshift 3 in Fission.
Openshift is mostly a superset of Kubernetes so it's not not too difficult to get working, I managed to get Fission up and running today with broad strokes, but was hoping to have it done at the application level so there are fewer security concessions.
Openshift has stricter security rules than Kubernetes with a few specifically affecting Fission. That being said the security rules are pretty good, so might be something worth adding to Fission on Kubernetes as well. The main things are:
- Containers run as a random numeric user (so either need a volume or world writable directory to make file system changes)
- Containers run with fewer capabilities
- There is a concept of projects in Openshift. They are directly mapped to namespaces, but you need to explicitly allow permissions for API calls between projects
Summary
In short the following needs doing to allow Fission to work on Openshift 3:
- The fission etcd deployment can't create it's config store directory due to permission. The fix for Openshift would be to specify ETCD_DATA_DIR to either use a volume or create the etcd config dir in something like /tmp, which is world writable
- The fission controller can't create it's /filestore directory. In Openshift it needs to create the /filestore directory somewhere that is world writable in the container or on a volume. The controller also seem to fail to function correctly unless the filestore directory is created but doesn't panic/error on that failure, so I think it's worth changing the behaviour to panic/exit on /filestore creation failure so that it's easier to debug and more clearly shows as an error (pod restarts)
- The pool manager doesn't have permission to call cross namespace in Openshift 3, so calls to the fission-function namespace are rejected. The best fix in Openshift would be to create a custom role for the pool manager in the deployment template (with only the API calls to the other project the pool manager needs to make) and assigned it to the correct service account in the fission-function project. An easier but less security fix is to assign the cluster-administrator role in the fission-function project to the default fission service account (system:serviceaccount:fission:default). Due to those missing rules this means that at least the following API calls fail from the pool manager in the current deployment:
- list extensions.deployments in the fission-function namespace
- create extensions.deployments in the fission-function namespace
- It might just be the file permissions above, but until I granted the capability to run as root for the controller (the same set of capabilities as Docker assigns by default) I would get the exception right at the bottom of this ticket. The only way I could fix the issue was to grant those capabilities (as I couldn't change the directory without changing source code like for etcd) so I'm not sure if there's more that needs doing.
Developing on Openshift
Openshift is super simple to get up and running, the client tool sets up a docker component based cluster to work with. The CLI tool is also just a superset of Kubectl so there's not much new to learn other than the additional concepts, to get going:
- Get the latest client (cli) tool version from https://github.com/openshift/origin/releases/tag/v1.4.1
- Run
oc cluster up --version 'v1.4.0'
. This will pull down the Openshift Origin containers and run them, making your cluster
- NOTE: If using docker toolbox you might need to switch docker-machine context context to the newly created docker machine if it fails half way through and run the same command again, e.g.
eval $(docker-machine env --shell bash openshift)
oc cluster up --version 'v1.4.0'
oc adm policy add-cluster-role-to-user cluster-admin system --as=system:admin
oc login -u system:admin
- Use the oc CLI tool (like kubectl) and/or login to the console with the same credentials on port 8443 on the host that is running the docker containers (so maybe your vbox host
docker-machine ip openshift
)
Getting Fission Deployed on Openshift
Below are the steps I took to get Fission up and running on Openshift 3, including a script with the hacks necessary to bypass the security features.
First of all you need to patch http://fission.io/fission.yaml with the following to change the etcd configstore dir:
142a143,144
> - name: ETCD_DATA_DIR
> value: /tmp/default.etcd
Then running the following script will do the job (in a shell that is logged in with oc
):
#!/bin/bash
oc new-project fission # The existing fission templates already create the project correctly by using the namespace attribute, but I still do it in order to make the changes below before creation of the resources
oc adm policy add-scc-to-user anyuid -z default # Hack 1
oc new-project fission-function
oc policy add-role-to-user cluster-admin system:serviceaccount:fission:default # Hack 2
oc create -f fission.yaml # Hack 3
oc create -f fission-nodeport.yaml
- Hack 1: This fixes item number 2 from the summary section, but in a broad way which gives the container more capabilities than it needs
- Hack 2: This fixes item number 3 from the summary section, but in a way that is overly permissive for the pool manager.
- Hack 3: Deploying the etcd pod so it creates the configstore in /tmp, not necessarily a hack.
With the above hacks I was able to successfully deploy the hello world function, so it's definitely possible to get it working!
Exception
The exception referenced in the last point of the summary section
2017/02/06 00:45:07 http: panic serving 172.17.0.1:59594: runtime error: invalid memory address or nil pointer dereference
goroutine 4036 [running]:
net/http.(*conn).serve.func1(0x19f62e00)
/usr/local/go/src/net/http/server.go:1491 +0xa2
panic(0x8fe7940, 0x19c1e008)
/usr/local/go/src/runtime/panic.go:458 +0x40b
github.com/fission/fission/controller.(*ResourceStore).writeFile(0x19e2bea0, 0x1a018a80, 0x5, 0x1a073800, 0x53, 0x60, 0x0, 0x0, 0x0, 0x0, ...)
/Users/soam/gopath/src/github.com/fission/fission/controller/resourceStore.go:156 +0x18d
github.com/fission/fission/controller.(*FunctionStore).Create(0x19e2bea0, 0x19f3c9c0, 0x0, 0x0, 0x0, 0x0)
/Users/soam/gopath/src/github.com/fission/fission/controller/functionStore.go:31 +0xec
github.com/fission/fission/controller.(*API).FunctionApiCreate(0x19e2bea0, 0xf76a6988, 0x19f13fc0, 0x1a032380)
/Users/soam/gopath/src/github.com/fission/fission/controller/functionApi.go:67 +0x232
github.com/fission/fission/controller.(*API).FunctionApiCreate-fm(0xf76a6988, 0x19f13fc0, 0x1a032380)
/Users/soam/gopath/src/github.com/fission/fission/controller/api.go:73 +0x38
net/http.HandlerFunc.ServeHTTP(0x19fdf738, 0xf76a6988, 0x19f13fc0, 0x1a032380)
/usr/local/go/src/net/http/server.go:1726 +0x34
github.com/fission/fission/vendor/github.com/gorilla/mux.(*Router).ServeHTTP(0x19e4c270, 0xf76a6988, 0x19f13fc0, 0x1a032380)
/Users/soam/gopath/src/github.com/fission/fission/vendor/github.com/gorilla/mux/mux.go:103 +0x210
github.com/fission/fission/vendor/github.com/gorilla/handlers.loggingHandler.ServeHTTP(0x9887b70, 0x19c1e0d8, 0x9885430, 0x19e4c270, 0x988dd40, 0x19d041b0, 0x1a032380)
/Users/soam/gopath/src/github.com/fission/fission/vendor/github.com/gorilla/handlers/handlers.go:69 +0xf0
github.com/fission/fission/vendor/github.com/gorilla/handlers.(*loggingHandler).ServeHTTP(0x1a031240, 0x988dd40, 0x19d041b0, 0x1a032380)
<autogenerated>:8 +0x9b
net/http.serverHandler.ServeHTTP(0x19f88410, 0x988dd40, 0x19d041b0, 0x1a032380)
/usr/local/go/src/net/http/server.go:2202 +0x136
net/http.(*conn).serve(0x19f62e00, 0x988e940, 0x19f13e40)
/usr/local/go/src/net/http/server.go:1579 +0xe86
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:2293 +0x489