GithubHelp home page GithubHelp logo

Comments (4)

alexellis avatar alexellis commented on July 20, 2024

So my primary concern here is if we are changing the code that queries netlink to something less robust, like a cache from the filesystem. What about surviving reboots and restarts? How does this compare? Why do we think it's better or worse?

I'd like to see more of a proposal for this, especially before working on a PR.

See "I have a great idea" in the contributing guide. https://github.com/openfaas/faas/blob/master/CONTRIBUTING.md#i-have-a-great-idea

from faasd.

alexellis avatar alexellis commented on July 20, 2024

/add label: design/review

from faasd.

carlosedp avatar carlosedp commented on July 20, 2024

Reboot is not an issue here since by using the CNI bridge plugin and the host-local IPAM, in the event of a reboot when faasd and faasd-provider processes start, they will assign new IP addresses to all containers as seen here:

Before reboot:

❯ faas-cli ls
Function                      	Invocations    	Replicas
figlet1                       	0              	1
❯ curl -d Test http://127.0.0.1:8080/function/figlet1
 _____         _
|_   _|__  ___| |_
  | |/ _ \/ __| __|
  | |  __/\__ \ |_
  |_|\___||___/\__|

❯ ll /var/lib/cni/results
ls: cannot access '/var/lib/cni/results': Permission denied
❯ sudo ll /var/lib/cni/results
total 40K
-rw------- 1 root root 229 Feb  3 16:57 cni-loopback-basic-auth-plugin-15374-lo
-rw------- 1 root root 332 Feb  3 16:57 openfaas-cni-bridge-basic-auth-plugin-15374-eth1
-rw------- 1 root root 229 Feb  3 16:57 cni-loopback-nats-15472-lo
-rw------- 1 root root 332 Feb  3 16:57 openfaas-cni-bridge-nats-15472-eth1
-rw------- 1 root root 229 Feb  3 16:57 cni-loopback-prometheus-15589-lo
-rw------- 1 root root 332 Feb  3 16:57 openfaas-cni-bridge-prometheus-15589-eth1
-rw------- 1 root root 229 Feb  3 16:57 cni-loopback-gateway-15713-lo
-rw------- 1 root root 332 Feb  3 16:57 openfaas-cni-bridge-gateway-15713-eth1
-rw------- 1 root root 229 Feb  3 16:57 cni-loopback-queue-worker-15822-lo
-rw------- 1 root root 332 Feb  3 16:57 openfaas-cni-bridge-queue-worker-15822-eth1
❯ sudo ll /var/lib/cni/networks/openfaas-cni-bridge/
total 24K
-rwxr-x--- 1 root root  0 Feb  3 16:57 lock*
-rw-r--r-- 1 root root 29 Feb  3 16:57 10.62.0.2
-rw-r--r-- 1 root root 16 Feb  3 16:57 10.62.0.3
-rw-r--r-- 1 root root 22 Feb  3 16:57 10.62.0.4
-rw-r--r-- 1 root root 19 Feb  3 16:57 10.62.0.5
-rw-r--r-- 1 root root  9 Feb  3 16:57 last_reserved_ip.0
-rw-r--r-- 1 root root 24 Feb  3 16:57 10.62.0.6

After reboot:

❯ jo faasd
-- Logs begin at Tue 2020-02-04 07:43:25 EST. --
Feb 04 07:43:25 debian10 systemd[1]: Started faasd.
-- Subject: A start job for unit faasd.service has finished successfully
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- A start job for unit faasd.service has finished successfully.
--
-- The job identifier is 101.
Feb 04 07:43:27 debian10 faasd[419]: 2020/02/04 07:43:27 File exists: "/var/lib/faasd/secrets/basic-auth-password"
Feb 04 07:43:27 debian10 faasd[419]: 2020/02/04 07:43:27 File exists: "/var/lib/faasd/secrets/basic-auth-user"
Feb 04 07:43:28 debian10 faasd[419]: 2020/02/04 07:43:28 Writing network config...
Feb 04 07:43:28 debian10 faasd[419]: 2020/02/04 07:43:28 Supervisor created in: 918.662518ms
Feb 04 07:43:28 debian10 faasd[419]: Preparing: basic-auth-plugin with image: docker.io/openfaas/basic-auth-plugin:0.18.10
Feb 04 07:43:28 debian10 faasd[419]: Prepare done for: docker.io/openfaas/basic-auth-plugin:0.18.10, 7171180 bytes
Feb 04 07:43:28 debian10 faasd[419]: Preparing: nats with image: docker.io/library/nats-streaming:0.11.2
Feb 04 07:43:28 debian10 faasd[419]: Prepare done for: docker.io/library/nats-streaming:0.11.2, 4647125 bytes
Feb 04 07:43:28 debian10 faasd[419]: Preparing: prometheus with image: docker.io/prom/prometheus:v2.14.0
Feb 04 07:43:28 debian10 faasd[419]: Prepare done for: docker.io/prom/prometheus:v2.14.0, 53527559 bytes
Feb 04 07:43:28 debian10 faasd[419]: Preparing: gateway with image: docker.io/openfaas/gateway:0.18.8
Feb 04 07:43:28 debian10 faasd[419]: Prepare done for: docker.io/openfaas/gateway:0.18.8, 11117867 bytes
Feb 04 07:43:28 debian10 faasd[419]: Preparing: queue-worker with image: docker.io/openfaas/queue-worker:0.9.0
Feb 04 07:43:28 debian10 faasd[419]: Prepare done for: docker.io/openfaas/queue-worker:0.9.0, 4293659 bytes
Feb 04 07:43:28 debian10 faasd[419]: Reconciling: basic-auth-plugin
Feb 04 07:43:28 debian10 faasd[419]: 2020/02/04 07:43:28 Created container basic-auth-plugin
Feb 04 07:43:28 debian10 faasd[419]: 2020/02/04 07:43:28 basic-auth-plugin has IP: 10.62.0.7
Feb 04 07:43:28 debian10 faasd[419]: 2020/02/04 07:43:28 Task: basic-auth-plugin        Container: basic-auth-plugin
Feb 04 07:43:28 debian10 faasd[419]: Reconciling: nats
Feb 04 07:43:28 debian10 faasd[419]: 2020/02/04 12:43:28 Listening on: 8080
Feb 04 07:43:28 debian10 faasd[419]: 2020/02/04 07:43:28 Created container nats
Feb 04 07:43:29 debian10 faasd[419]: 2020/02/04 07:43:29 nats has IP: 10.62.0.8
Feb 04 07:43:29 debian10 faasd[419]: 2020/02/04 07:43:29 Task: nats        Container: nats
Feb 04 07:43:29 debian10 faasd[419]: Reconciling: prometheus
Feb 04 07:43:29 debian10 faasd[419]: [1] 2020/02/04 12:43:29.141095 [INF] STREAM: Starting nats-streaming-server[faas-cluster] version 0.11.2
Feb 04 07:43:29 debian10 faasd[419]: [1] 2020/02/04 12:43:29.141129 [INF] STREAM: ServerID: nztAQ3YCMbpeEtrLfFx8kg
Feb 04 07:43:29 debian10 faasd[419]: [1] 2020/02/04 12:43:29.141131 [INF] STREAM: Go version: go1.11.1
Feb 04 07:43:29 debian10 faasd[419]: [1] 2020/02/04 12:43:29.141615 [INF] Starting nats-server version 1.3.0
Feb 04 07:43:29 debian10 faasd[419]: [1] 2020/02/04 12:43:29.141626 [INF] Git commit [not set]
Feb 04 07:43:29 debian10 faasd[419]: [1] 2020/02/04 12:43:29.141769 [INF] Starting http monitor on 0.0.0.0:8222
Feb 04 07:43:29 debian10 faasd[419]: [1] 2020/02/04 12:43:29.141806 [INF] Listening for client connections on 0.0.0.0:4222
Feb 04 07:43:29 debian10 faasd[419]: [1] 2020/02/04 12:43:29.141809 [INF] Server is ready
Feb 04 07:43:29 debian10 faasd[419]: 2020/02/04 07:43:29 Created container prometheus
Feb 04 07:43:29 debian10 faasd[419]: [1] 2020/02/04 12:43:29.171611 [INF] STREAM: Recovering the state...
Feb 04 07:43:29 debian10 faasd[419]: [1] 2020/02/04 12:43:29.171956 [INF] STREAM: No recovered state
Feb 04 07:43:29 debian10 faasd[419]: [1] 2020/02/04 12:43:29.422974 [INF] STREAM: Message store is MEMORY
Feb 04 07:43:29 debian10 faasd[419]: [1] 2020/02/04 12:43:29.423006 [INF] STREAM: ---------- Store Limits ----------
Feb 04 07:43:29 debian10 faasd[419]: [1] 2020/02/04 12:43:29.423009 [INF] STREAM: Channels:                  100 *
Feb 04 07:43:29 debian10 faasd[419]: [1] 2020/02/04 12:43:29.423011 [INF] STREAM: --------- Channels Limits --------
Feb 04 07:43:29 debian10 faasd[419]: [1] 2020/02/04 12:43:29.423012 [INF] STREAM:   Subscriptions:          1000 *
Feb 04 07:43:29 debian10 faasd[419]: [1] 2020/02/04 12:43:29.423014 [INF] STREAM:   Messages     :       1000000 *
Feb 04 07:43:29 debian10 faasd[419]: [1] 2020/02/04 12:43:29.423015 [INF] STREAM:   Bytes        :     976.56 MB *
Feb 04 07:43:29 debian10 faasd[419]: [1] 2020/02/04 12:43:29.423017 [INF] STREAM:   Age          :     unlimited *
Feb 04 07:43:29 debian10 faasd[419]: [1] 2020/02/04 12:43:29.423018 [INF] STREAM:   Inactivity   :     unlimited *
Feb 04 07:43:29 debian10 faasd[419]: [1] 2020/02/04 12:43:29.423045 [INF] STREAM: ----------------------------------
Feb 04 07:43:29 debian10 faasd[419]: 2020/02/04 07:43:29 prometheus has IP: 10.62.0.9
Feb 04 07:43:29 debian10 faasd[419]: 2020/02/04 07:43:29 Task: prometheus        Container: prometheus
Feb 04 07:43:29 debian10 faasd[419]: Reconciling: gateway
Feb 04 07:43:29 debian10 faasd[419]: 2020/02/04 07:43:29 Created container gateway
Feb 04 07:43:29 debian10 faasd[419]: level=info ts=2020-02-04T12:43:29.642Z caller=main.go:296 msg="no time or size retention was set so using the default time retention" duration=15d
Feb 04 07:43:29 debian10 faasd[419]: level=info ts=2020-02-04T12:43:29.642Z caller=main.go:332 msg="Starting Prometheus" version="(version=2.14.0, branch=HEAD, revision=edeb7a44cbf745f1d8be4ea6f215e79e651bfe19)"
Feb 04 07:43:29 debian10 faasd[419]: level=info ts=2020-02-04T12:43:29.642Z caller=main.go:333 build_context="(go=go1.13.4, user=root@df2327081015, date=20191111-14:27:12)"
Feb 04 07:43:29 debian10 faasd[419]: level=info ts=2020-02-04T12:43:29.642Z caller=main.go:334 host_details="(Linux 4.19.0-6-amd64 #1 SMP Debian 4.19.67-2+deb10u2 (2019-11-11) x86_64 debian10 (none))"
Feb 04 07:43:29 debian10 faasd[419]: level=info ts=2020-02-04T12:43:29.642Z caller=main.go:335 fd_limits="(soft=1024, hard=1024)"
Feb 04 07:43:29 debian10 faasd[419]: level=info ts=2020-02-04T12:43:29.643Z caller=main.go:336 vm_limits="(soft=unlimited, hard=unlimited)"
Feb 04 07:43:29 debian10 faasd[419]: level=info ts=2020-02-04T12:43:29.646Z caller=main.go:657 msg="Starting TSDB ..."
Feb 04 07:43:29 debian10 faasd[419]: level=info ts=2020-02-04T12:43:29.647Z caller=web.go:496 component=web msg="Start listening for connections" address=0.0.0.0:9090
Feb 04 07:43:29 debian10 faasd[419]: level=info ts=2020-02-04T12:43:29.654Z caller=head.go:535 component=tsdb msg="replaying WAL, this may take awhile"
Feb 04 07:43:29 debian10 faasd[419]: level=info ts=2020-02-04T12:43:29.655Z caller=head.go:583 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0
Feb 04 07:43:29 debian10 faasd[419]: level=info ts=2020-02-04T12:43:29.656Z caller=main.go:672 fs_type=794c7630
Feb 04 07:43:29 debian10 faasd[419]: level=info ts=2020-02-04T12:43:29.656Z caller=main.go:673 msg="TSDB started"
Feb 04 07:43:29 debian10 faasd[419]: level=info ts=2020-02-04T12:43:29.656Z caller=main.go:743 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
Feb 04 07:43:29 debian10 faasd[419]: level=info ts=2020-02-04T12:43:29.667Z caller=main.go:771 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml
Feb 04 07:43:29 debian10 faasd[419]: level=info ts=2020-02-04T12:43:29.667Z caller=main.go:626 msg="Server is ready to receive web requests."
Feb 04 07:43:29 debian10 faasd[419]: 2020/02/04 07:43:29 gateway has IP: 10.62.0.10
Feb 04 07:43:29 debian10 faasd[419]: 2020/02/04 07:43:29 Task: gateway        Container: gateway
Feb 04 07:43:29 debian10 faasd[419]: Reconciling: queue-worker
Feb 04 07:43:29 debian10 faasd[419]: 2020/02/04 12:43:29 HTTP Read Timeout: 1m0s
Feb 04 07:43:29 debian10 faasd[419]: 2020/02/04 12:43:29 HTTP Write Timeout: 1m0s
Feb 04 07:43:29 debian10 faasd[419]: 2020/02/04 12:43:29 Binding to external function provider: http://faas-containerd:8081/
Feb 04 07:43:29 debian10 faasd[419]: 2020/02/04 12:43:29 Async enabled: Using NATS Streaming.
Feb 04 07:43:29 debian10 faasd[419]: 2020/02/04 12:43:29 Opening connection to nats://nats:4222
Feb 04 07:43:29 debian10 faasd[419]: 2020/02/04 12:43:29 Connect: nats://nats:4222
Feb 04 07:43:29 debian10 faasd[419]: 2020/02/04 07:43:29 Created container queue-worker
Feb 04 07:43:30 debian10 faasd[419]: 2020/02/04 07:43:30 queue-worker has IP: 10.62.0.11
Feb 04 07:43:30 debian10 faasd[419]: 2020/02/04 07:43:30 Task: queue-worker        Container: queue-worker
Feb 04 07:43:30 debian10 faasd[419]: 2020/02/04 07:43:30 Supervisor init done in: 2.541132157s
Feb 04 07:43:30 debian10 faasd[419]: 2020/02/04 07:43:30 faasd: waiting for SIGTERM or SIGINT
Feb 04 07:43:30 debian10 faasd[419]: Loading basic authentication credentials
Feb 04 07:43:30 debian10 faasd[419]: Connect: nats://nats:4222
Feb 04 07:43:30 debian10 faasd[419]: Subscribing to: faas-request at nats://nats:4222
Feb 04 07:43:30 debian10 faasd[419]: Wait for  5m5s
Feb 04 07:43:30 debian10 faasd[419]: [1] 2020/02/04 12:43:30.618374 [INF] STREAM: Channel "faas-request" has been created
Feb 04 07:43:30 debian10 faasd[419]: Listening on [faas-request], clientID=[faas-worker-debian10], qgroup=[faas] durable=[]
Feb 04 07:43:33 debian10 faasd[419]: 2020/02/04 07:43:33 [up] Sending 10.62.0.10 to proxy
Feb 04 07:43:33 debian10 faasd[419]: 2020/02/04 07:43:33 Starting faasd proxy on 8080
Feb 04 07:43:33 debian10 faasd[419]: Gateway: 10.62.0.10:8080
Feb 04 07:43:33 debian10 faasd[419]: 2020/02/04 07:43:33 [proxy] Wait for done
Feb 04 07:43:33 debian10 faasd[419]: 2020/02/04 07:43:33 [proxy] Begin listen on 8080
Feb 04 07:44:29 debian10 faasd[419]: [faasd] proxy: http://10.62.0.10:8080/system/functions
Feb 04 07:44:29 debian10 faasd[419]: 2020/02/04 12:44:29 Validated request 200.
Feb 04 07:44:29 debian10 faasd[419]: 2020/02/04 12:44:29 Forwarded [GET] to /system/functions - [200] - 0.010728 seconds
Feb 04 07:44:48 debian10 faasd[419]: [faasd] proxy: http://10.62.0.10:8080/function/figlet1
Feb 04 07:44:48 debian10 faasd[419]: 2020/02/04 12:44:48 GetReplicas [figlet1.] took: 0.009235s
Feb 04 07:44:48 debian10 faasd[419]: 2020/02/04 12:44:48 GetReplicas [figlet1.] took: 0.009253s
Feb 04 07:44:48 debian10 faasd[419]: 2020/02/04 12:44:48 GetReplicas [figlet1.] took: 0.009021s
Feb 04 07:44:48 debian10 faasd[419]: 2020/02/04 12:44:48 GetReplicas [figlet1.] took: 0.009042s
Feb 04 07:44:48 debian10 faasd[419]: 2020/02/04 12:44:48 [Scale 0] function=figlet1 0 => 1 requested
Feb 04 07:44:48 debian10 faasd[419]: 2020/02/04 12:44:48 SetReplicas [figlet1.] took: 0.340124s
Feb 04 07:44:49 debian10 faasd[419]: 2020/02/04 12:44:49 GetReplicas [figlet1.] took: 0.010957s
Feb 04 07:44:49 debian10 faasd[419]: 2020/02/04 12:44:49 GetReplicas [figlet1.] took: 0.010975s
Feb 04 07:44:49 debian10 faasd[419]: 2020/02/04 12:44:49 [Scale] function=figlet1 0 => 1 successful - 0.370325 seconds
Feb 04 07:44:49 debian10 faasd[419]: 2020/02/04 12:44:49 Forwarded [POST] to /function/figlet1 - [500] - 0.003176 seconds


^C%                                                                                                                                                                                                 ❯ sudo ls -l /var/lib/cni/results/
total 80K
-rw------- 1 root root 227 Feb  4 07:43 cni-loopback-basic-auth-plugin-674-lo
-rw------- 1 root root 228 Feb  4 07:44 cni-loopback-figlet1-1625-lo
-rw------- 1 root root 228 Feb  4 07:43 cni-loopback-gateway-1139-lo
-rw------- 1 root root 229 Feb  3 16:57 cni-loopback-gateway-15713-lo
-rw------- 1 root root 229 Feb  3 16:57 cni-loopback-nats-15472-lo
-rw------- 1 root root 227 Feb  4 07:43 cni-loopback-nats-865-lo
-rw------- 1 root root 228 Feb  4 07:43 cni-loopback-prometheus-1030-lo
-rw------- 1 root root 229 Feb  3 16:57 cni-loopback-prometheus-15589-lo
-rw------- 1 root root 228 Feb  4 07:43 cni-loopback-queue-worker-1246-lo
-rw------- 1 root root 229 Feb  3 16:57 cni-loopback-queue-worker-15822-lo
-rw------- 1 root root 330 Feb  4 07:43 openfaas-cni-bridge-basic-auth-plugin-674-eth1
-rw------- 1 root root 332 Feb  4 07:44 openfaas-cni-bridge-figlet1-1625-eth1
-rw------- 1 root root 332 Feb  4 07:43 openfaas-cni-bridge-gateway-1139-eth1
-rw------- 1 root root 332 Feb  3 16:57 openfaas-cni-bridge-gateway-15713-eth1
-rw------- 1 root root 332 Feb  3 16:57 openfaas-cni-bridge-nats-15472-eth1
-rw------- 1 root root 330 Feb  4 07:43 openfaas-cni-bridge-nats-865-eth1
-rw------- 1 root root 331 Feb  4 07:43 openfaas-cni-bridge-prometheus-1030-eth1
-rw------- 1 root root 332 Feb  3 16:57 openfaas-cni-bridge-prometheus-15589-eth1
-rw------- 1 root root 332 Feb  4 07:43 openfaas-cni-bridge-queue-worker-1246-eth1
-rw------- 1 root root 332 Feb  3 16:57 openfaas-cni-bridge-queue-worker-15822-eth1
❯ sudo ll /var/lib/cni/networks/openfaas-cni-bridge/
total 44K
-rwxr-x--- 1 root root  0 Feb  3 16:57 lock*
-rw-r--r-- 1 root root 16 Feb  3 16:57 10.62.0.3
-rw-r--r-- 1 root root 22 Feb  3 16:57 10.62.0.4
-rw-r--r-- 1 root root 19 Feb  3 16:57 10.62.0.5
-rw-r--r-- 1 root root 24 Feb  3 16:57 10.62.0.6
-rw-r--r-- 1 root root 27 Feb  4 07:43 10.62.0.7
-rw-r--r-- 1 root root 14 Feb  4 07:43 10.62.0.8
-rw-r--r-- 1 root root 21 Feb  4 07:43 10.62.0.9
-rw-r--r-- 1 root root 18 Feb  4 07:43 10.62.0.10
-rw-r--r-- 1 root root 23 Feb  4 07:43 10.62.0.11
-rw-r--r-- 1 root root 10 Feb  4 07:44 last_reserved_ip.0
-rw-r--r-- 1 root root 18 Feb  4 07:44 10.62.0.12

❯ faas-cli ls
Function                      	Invocations    	Replicas
figlet1                       	1              	1
❯ curl -d Test http://127.0.0.1:8080/function/figlet1
 _____         _
|_   _|__  ___| |_
  | |/ _ \/ __| __|
  | |  __/\__ \ |_
  |_|\___||___/\__|

All containers got new IPs. This is already used on K3C where the IPs from created containers are read from the CNI results.

from faasd.

carlosedp avatar carlosedp commented on July 20, 2024

As spoken before, here a list of pros/cons on this implementation:

Current netlink/netns imlementation:

Pros

  • Fetches the current IP directly from Linux interface configuration
  • Has been used on other well tested libraries like Weave Net

Cons

  • Depends on two external libraries
  • Depends on two internal modules to provide access to the libraries
  • More code == more bugs and more maintainability. Lines: +43 −18,800

CNI Results implementation (this PR)

Pros

  • Uses pure artifacts from CNI results and no external libraries or complex namespace/netlink manipulation.
  • Less code == less bugs and less maintainability. Lines: +43 −18,800
  • Is currently used on new applications based on pure CNI like rancher/k3c
  • Based on CNI spec. File format not supposed to change.

Cons

  • Result files could be changed by user (only root) resulting in wrong IP
  • Is not as tested as other implementation
  • Depends on filesystem result files instead of live fetch from Kernel config

from faasd.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.