Restarting faasd leaves containers about faasd

[alexellis]

@carlosedp please could you take a look?

[martindekov]

I can explore this further so I will assign myself

[martindekov]

Derek assign: me

[carlosedp]

The containers shouldn't stay but the IP allocation is kept on the filesystem.
The CNI plugin uses the "host-local" IPAM that by default stores the allocated IPs in /var/lib/cni/networks/$NETWORK_NAME.

I think it would be a good thing to move this directory to /var/run/cni for example so it would be cleaned between reboots.

What do you think @alexellis?

[carlosedp]

This does not happen here, as can be seen below, first I curl directly to the gateway IP address (.10) and list the containerd tasks. Gateway has the PID 5736.

Then I restart faasd and the gateway container has other PID (6664). Also it has a new IP address (.15).

Curl to it on a deployed function returns as well. Also there are no duplicate containers.

❯ curl -d Test 10.62.0.10:8080/function/figlet1
 _____         _
|_   _|__  ___| |_
  | |/ _ \/ __| __|
  | |  __/\__ \ |_
  |_|\___||___/\__|

❯ sudo ctr  task ls
TASK                 PID     STATUS
prometheus           5612    RUNNING
gateway              5736    RUNNING
queue-worker         5852    RUNNING
basic-auth-plugin    5393    RUNNING
nats                 5493    RUNNING
❯ sc-restart faasd.service
❯ sudo ctr  task ls
TASK                 PID     STATUS
basic-auth-plugin    6311    RUNNING
nats                 6411    RUNNING
prometheus           6535    RUNNING
gateway              6664    RUNNING
queue-worker         6778    RUNNING
❯ sudo cat  /var/lib/cni/results/openfaas-cni-bridge-gateway-6664-eth1
{"cniVersion":"0.4.0","interfaces":[{"name":"openfaas0","mac":"72:db:c2:d4:d3:69"},{"name":"veth6470e54c","mac":"2e:24:51:57:34:c1"},{"name":"eth1","mac":"8e:92:84:46:5c:8a","sandbox":"/proc/6664/ns/net"}],"ips":[{"version":"4","interface":2,"address":"10.62.0.15/16","gateway":"10.62.0.1"}],"routes":[{"dst":"0.0.0.0/0"}],"dns":{}}%                                                           ❯ curl -d Test 10.62.0.15:8080/function/figlet1
 _____         _
|_   _|__  ___| |_
  | |/ _ \/ __| __|
  | |  __/\__ \ |_
  |_|\___||___/\__|

❯ sudo ctr container ls
CONTAINER            IMAGE                                           RUNTIME
basic-auth-plugin    docker.io/openfaas/basic-auth-plugin:0.18.10    io.containerd.runc.v2
gateway              docker.io/openfaas/gateway:0.18.8               io.containerd.runc.v2
nats                 docker.io/library/nats-streaming:0.11.2         io.containerd.runc.v2
prometheus           docker.io/prom/prometheus:v2.14.0               io.containerd.runc.v2
queue-worker         docker.io/openfaas/queue-worker:0.9.0           io.containerd.runc.v2

[carlosedp]

Also I can't deploy the same figlet function twice:

❯ faas ls
Function                      	Invocations    	Replicas
figlet1                       	0              	1
❯ faas store deploy figlet --name=figlet1 --env write_timeout=1s
WARNING! Communication is not secure, please consider using HTTPS. Letsencrypt.org offers free SSL/TLS certificates.
Function figlet1 already exists, attempting rolling-update.

Deployed. 200 OK.
URL: http://127.0.0.1:8080/function/figlet1

❯ faas ls
Function                      	Invocations    	Replicas
figlet1                       	0              	1

I believe a function with or without labels should be unique by it's name.

[martindekov]

Thanks for looking at this @carlosedp I have also tried to replicate and it seems like the problem did not persist. I only deployed the same function with two different set of labels this time though and restarted the faasd and faasd-provider services which is a subset of the things I did originally.

I will close this I think what I have noticed was related with me recompiling the binary or some very corner case.