Comments (4)
So my primary concern here is if we are changing the code that queries netlink to something less robust, like a cache from the filesystem. What about surviving reboots and restarts? How does this compare? Why do we think it's better or worse?
I'd like to see more of a proposal for this, especially before working on a PR.
See "I have a great idea" in the contributing guide. https://github.com/openfaas/faas/blob/master/CONTRIBUTING.md#i-have-a-great-idea
from faasd.
/add label: design/review
from faasd.
Reboot is not an issue here since by using the CNI bridge plugin and the host-local IPAM, in the event of a reboot when faasd and faasd-provider processes start, they will assign new IP addresses to all containers as seen here:
Before reboot:
❯ faas-cli ls
Function Invocations Replicas
figlet1 0 1
❯ curl -d Test http://127.0.0.1:8080/function/figlet1
_____ _
|_ _|__ ___| |_
| |/ _ \/ __| __|
| | __/\__ \ |_
|_|\___||___/\__|
❯ ll /var/lib/cni/results
ls: cannot access '/var/lib/cni/results': Permission denied
❯ sudo ll /var/lib/cni/results
total 40K
-rw------- 1 root root 229 Feb 3 16:57 cni-loopback-basic-auth-plugin-15374-lo
-rw------- 1 root root 332 Feb 3 16:57 openfaas-cni-bridge-basic-auth-plugin-15374-eth1
-rw------- 1 root root 229 Feb 3 16:57 cni-loopback-nats-15472-lo
-rw------- 1 root root 332 Feb 3 16:57 openfaas-cni-bridge-nats-15472-eth1
-rw------- 1 root root 229 Feb 3 16:57 cni-loopback-prometheus-15589-lo
-rw------- 1 root root 332 Feb 3 16:57 openfaas-cni-bridge-prometheus-15589-eth1
-rw------- 1 root root 229 Feb 3 16:57 cni-loopback-gateway-15713-lo
-rw------- 1 root root 332 Feb 3 16:57 openfaas-cni-bridge-gateway-15713-eth1
-rw------- 1 root root 229 Feb 3 16:57 cni-loopback-queue-worker-15822-lo
-rw------- 1 root root 332 Feb 3 16:57 openfaas-cni-bridge-queue-worker-15822-eth1
❯ sudo ll /var/lib/cni/networks/openfaas-cni-bridge/
total 24K
-rwxr-x--- 1 root root 0 Feb 3 16:57 lock*
-rw-r--r-- 1 root root 29 Feb 3 16:57 10.62.0.2
-rw-r--r-- 1 root root 16 Feb 3 16:57 10.62.0.3
-rw-r--r-- 1 root root 22 Feb 3 16:57 10.62.0.4
-rw-r--r-- 1 root root 19 Feb 3 16:57 10.62.0.5
-rw-r--r-- 1 root root 9 Feb 3 16:57 last_reserved_ip.0
-rw-r--r-- 1 root root 24 Feb 3 16:57 10.62.0.6
After reboot:
❯ jo faasd
-- Logs begin at Tue 2020-02-04 07:43:25 EST. --
Feb 04 07:43:25 debian10 systemd[1]: Started faasd.
-- Subject: A start job for unit faasd.service has finished successfully
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- A start job for unit faasd.service has finished successfully.
--
-- The job identifier is 101.
Feb 04 07:43:27 debian10 faasd[419]: 2020/02/04 07:43:27 File exists: "/var/lib/faasd/secrets/basic-auth-password"
Feb 04 07:43:27 debian10 faasd[419]: 2020/02/04 07:43:27 File exists: "/var/lib/faasd/secrets/basic-auth-user"
Feb 04 07:43:28 debian10 faasd[419]: 2020/02/04 07:43:28 Writing network config...
Feb 04 07:43:28 debian10 faasd[419]: 2020/02/04 07:43:28 Supervisor created in: 918.662518ms
Feb 04 07:43:28 debian10 faasd[419]: Preparing: basic-auth-plugin with image: docker.io/openfaas/basic-auth-plugin:0.18.10
Feb 04 07:43:28 debian10 faasd[419]: Prepare done for: docker.io/openfaas/basic-auth-plugin:0.18.10, 7171180 bytes
Feb 04 07:43:28 debian10 faasd[419]: Preparing: nats with image: docker.io/library/nats-streaming:0.11.2
Feb 04 07:43:28 debian10 faasd[419]: Prepare done for: docker.io/library/nats-streaming:0.11.2, 4647125 bytes
Feb 04 07:43:28 debian10 faasd[419]: Preparing: prometheus with image: docker.io/prom/prometheus:v2.14.0
Feb 04 07:43:28 debian10 faasd[419]: Prepare done for: docker.io/prom/prometheus:v2.14.0, 53527559 bytes
Feb 04 07:43:28 debian10 faasd[419]: Preparing: gateway with image: docker.io/openfaas/gateway:0.18.8
Feb 04 07:43:28 debian10 faasd[419]: Prepare done for: docker.io/openfaas/gateway:0.18.8, 11117867 bytes
Feb 04 07:43:28 debian10 faasd[419]: Preparing: queue-worker with image: docker.io/openfaas/queue-worker:0.9.0
Feb 04 07:43:28 debian10 faasd[419]: Prepare done for: docker.io/openfaas/queue-worker:0.9.0, 4293659 bytes
Feb 04 07:43:28 debian10 faasd[419]: Reconciling: basic-auth-plugin
Feb 04 07:43:28 debian10 faasd[419]: 2020/02/04 07:43:28 Created container basic-auth-plugin
Feb 04 07:43:28 debian10 faasd[419]: 2020/02/04 07:43:28 basic-auth-plugin has IP: 10.62.0.7
Feb 04 07:43:28 debian10 faasd[419]: 2020/02/04 07:43:28 Task: basic-auth-plugin Container: basic-auth-plugin
Feb 04 07:43:28 debian10 faasd[419]: Reconciling: nats
Feb 04 07:43:28 debian10 faasd[419]: 2020/02/04 12:43:28 Listening on: 8080
Feb 04 07:43:28 debian10 faasd[419]: 2020/02/04 07:43:28 Created container nats
Feb 04 07:43:29 debian10 faasd[419]: 2020/02/04 07:43:29 nats has IP: 10.62.0.8
Feb 04 07:43:29 debian10 faasd[419]: 2020/02/04 07:43:29 Task: nats Container: nats
Feb 04 07:43:29 debian10 faasd[419]: Reconciling: prometheus
Feb 04 07:43:29 debian10 faasd[419]: [1] 2020/02/04 12:43:29.141095 [INF] STREAM: Starting nats-streaming-server[faas-cluster] version 0.11.2
Feb 04 07:43:29 debian10 faasd[419]: [1] 2020/02/04 12:43:29.141129 [INF] STREAM: ServerID: nztAQ3YCMbpeEtrLfFx8kg
Feb 04 07:43:29 debian10 faasd[419]: [1] 2020/02/04 12:43:29.141131 [INF] STREAM: Go version: go1.11.1
Feb 04 07:43:29 debian10 faasd[419]: [1] 2020/02/04 12:43:29.141615 [INF] Starting nats-server version 1.3.0
Feb 04 07:43:29 debian10 faasd[419]: [1] 2020/02/04 12:43:29.141626 [INF] Git commit [not set]
Feb 04 07:43:29 debian10 faasd[419]: [1] 2020/02/04 12:43:29.141769 [INF] Starting http monitor on 0.0.0.0:8222
Feb 04 07:43:29 debian10 faasd[419]: [1] 2020/02/04 12:43:29.141806 [INF] Listening for client connections on 0.0.0.0:4222
Feb 04 07:43:29 debian10 faasd[419]: [1] 2020/02/04 12:43:29.141809 [INF] Server is ready
Feb 04 07:43:29 debian10 faasd[419]: 2020/02/04 07:43:29 Created container prometheus
Feb 04 07:43:29 debian10 faasd[419]: [1] 2020/02/04 12:43:29.171611 [INF] STREAM: Recovering the state...
Feb 04 07:43:29 debian10 faasd[419]: [1] 2020/02/04 12:43:29.171956 [INF] STREAM: No recovered state
Feb 04 07:43:29 debian10 faasd[419]: [1] 2020/02/04 12:43:29.422974 [INF] STREAM: Message store is MEMORY
Feb 04 07:43:29 debian10 faasd[419]: [1] 2020/02/04 12:43:29.423006 [INF] STREAM: ---------- Store Limits ----------
Feb 04 07:43:29 debian10 faasd[419]: [1] 2020/02/04 12:43:29.423009 [INF] STREAM: Channels: 100 *
Feb 04 07:43:29 debian10 faasd[419]: [1] 2020/02/04 12:43:29.423011 [INF] STREAM: --------- Channels Limits --------
Feb 04 07:43:29 debian10 faasd[419]: [1] 2020/02/04 12:43:29.423012 [INF] STREAM: Subscriptions: 1000 *
Feb 04 07:43:29 debian10 faasd[419]: [1] 2020/02/04 12:43:29.423014 [INF] STREAM: Messages : 1000000 *
Feb 04 07:43:29 debian10 faasd[419]: [1] 2020/02/04 12:43:29.423015 [INF] STREAM: Bytes : 976.56 MB *
Feb 04 07:43:29 debian10 faasd[419]: [1] 2020/02/04 12:43:29.423017 [INF] STREAM: Age : unlimited *
Feb 04 07:43:29 debian10 faasd[419]: [1] 2020/02/04 12:43:29.423018 [INF] STREAM: Inactivity : unlimited *
Feb 04 07:43:29 debian10 faasd[419]: [1] 2020/02/04 12:43:29.423045 [INF] STREAM: ----------------------------------
Feb 04 07:43:29 debian10 faasd[419]: 2020/02/04 07:43:29 prometheus has IP: 10.62.0.9
Feb 04 07:43:29 debian10 faasd[419]: 2020/02/04 07:43:29 Task: prometheus Container: prometheus
Feb 04 07:43:29 debian10 faasd[419]: Reconciling: gateway
Feb 04 07:43:29 debian10 faasd[419]: 2020/02/04 07:43:29 Created container gateway
Feb 04 07:43:29 debian10 faasd[419]: level=info ts=2020-02-04T12:43:29.642Z caller=main.go:296 msg="no time or size retention was set so using the default time retention" duration=15d
Feb 04 07:43:29 debian10 faasd[419]: level=info ts=2020-02-04T12:43:29.642Z caller=main.go:332 msg="Starting Prometheus" version="(version=2.14.0, branch=HEAD, revision=edeb7a44cbf745f1d8be4ea6f215e79e651bfe19)"
Feb 04 07:43:29 debian10 faasd[419]: level=info ts=2020-02-04T12:43:29.642Z caller=main.go:333 build_context="(go=go1.13.4, user=root@df2327081015, date=20191111-14:27:12)"
Feb 04 07:43:29 debian10 faasd[419]: level=info ts=2020-02-04T12:43:29.642Z caller=main.go:334 host_details="(Linux 4.19.0-6-amd64 #1 SMP Debian 4.19.67-2+deb10u2 (2019-11-11) x86_64 debian10 (none))"
Feb 04 07:43:29 debian10 faasd[419]: level=info ts=2020-02-04T12:43:29.642Z caller=main.go:335 fd_limits="(soft=1024, hard=1024)"
Feb 04 07:43:29 debian10 faasd[419]: level=info ts=2020-02-04T12:43:29.643Z caller=main.go:336 vm_limits="(soft=unlimited, hard=unlimited)"
Feb 04 07:43:29 debian10 faasd[419]: level=info ts=2020-02-04T12:43:29.646Z caller=main.go:657 msg="Starting TSDB ..."
Feb 04 07:43:29 debian10 faasd[419]: level=info ts=2020-02-04T12:43:29.647Z caller=web.go:496 component=web msg="Start listening for connections" address=0.0.0.0:9090
Feb 04 07:43:29 debian10 faasd[419]: level=info ts=2020-02-04T12:43:29.654Z caller=head.go:535 component=tsdb msg="replaying WAL, this may take awhile"
Feb 04 07:43:29 debian10 faasd[419]: level=info ts=2020-02-04T12:43:29.655Z caller=head.go:583 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0
Feb 04 07:43:29 debian10 faasd[419]: level=info ts=2020-02-04T12:43:29.656Z caller=main.go:672 fs_type=794c7630
Feb 04 07:43:29 debian10 faasd[419]: level=info ts=2020-02-04T12:43:29.656Z caller=main.go:673 msg="TSDB started"
Feb 04 07:43:29 debian10 faasd[419]: level=info ts=2020-02-04T12:43:29.656Z caller=main.go:743 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
Feb 04 07:43:29 debian10 faasd[419]: level=info ts=2020-02-04T12:43:29.667Z caller=main.go:771 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml
Feb 04 07:43:29 debian10 faasd[419]: level=info ts=2020-02-04T12:43:29.667Z caller=main.go:626 msg="Server is ready to receive web requests."
Feb 04 07:43:29 debian10 faasd[419]: 2020/02/04 07:43:29 gateway has IP: 10.62.0.10
Feb 04 07:43:29 debian10 faasd[419]: 2020/02/04 07:43:29 Task: gateway Container: gateway
Feb 04 07:43:29 debian10 faasd[419]: Reconciling: queue-worker
Feb 04 07:43:29 debian10 faasd[419]: 2020/02/04 12:43:29 HTTP Read Timeout: 1m0s
Feb 04 07:43:29 debian10 faasd[419]: 2020/02/04 12:43:29 HTTP Write Timeout: 1m0s
Feb 04 07:43:29 debian10 faasd[419]: 2020/02/04 12:43:29 Binding to external function provider: http://faas-containerd:8081/
Feb 04 07:43:29 debian10 faasd[419]: 2020/02/04 12:43:29 Async enabled: Using NATS Streaming.
Feb 04 07:43:29 debian10 faasd[419]: 2020/02/04 12:43:29 Opening connection to nats://nats:4222
Feb 04 07:43:29 debian10 faasd[419]: 2020/02/04 12:43:29 Connect: nats://nats:4222
Feb 04 07:43:29 debian10 faasd[419]: 2020/02/04 07:43:29 Created container queue-worker
Feb 04 07:43:30 debian10 faasd[419]: 2020/02/04 07:43:30 queue-worker has IP: 10.62.0.11
Feb 04 07:43:30 debian10 faasd[419]: 2020/02/04 07:43:30 Task: queue-worker Container: queue-worker
Feb 04 07:43:30 debian10 faasd[419]: 2020/02/04 07:43:30 Supervisor init done in: 2.541132157s
Feb 04 07:43:30 debian10 faasd[419]: 2020/02/04 07:43:30 faasd: waiting for SIGTERM or SIGINT
Feb 04 07:43:30 debian10 faasd[419]: Loading basic authentication credentials
Feb 04 07:43:30 debian10 faasd[419]: Connect: nats://nats:4222
Feb 04 07:43:30 debian10 faasd[419]: Subscribing to: faas-request at nats://nats:4222
Feb 04 07:43:30 debian10 faasd[419]: Wait for 5m5s
Feb 04 07:43:30 debian10 faasd[419]: [1] 2020/02/04 12:43:30.618374 [INF] STREAM: Channel "faas-request" has been created
Feb 04 07:43:30 debian10 faasd[419]: Listening on [faas-request], clientID=[faas-worker-debian10], qgroup=[faas] durable=[]
Feb 04 07:43:33 debian10 faasd[419]: 2020/02/04 07:43:33 [up] Sending 10.62.0.10 to proxy
Feb 04 07:43:33 debian10 faasd[419]: 2020/02/04 07:43:33 Starting faasd proxy on 8080
Feb 04 07:43:33 debian10 faasd[419]: Gateway: 10.62.0.10:8080
Feb 04 07:43:33 debian10 faasd[419]: 2020/02/04 07:43:33 [proxy] Wait for done
Feb 04 07:43:33 debian10 faasd[419]: 2020/02/04 07:43:33 [proxy] Begin listen on 8080
Feb 04 07:44:29 debian10 faasd[419]: [faasd] proxy: http://10.62.0.10:8080/system/functions
Feb 04 07:44:29 debian10 faasd[419]: 2020/02/04 12:44:29 Validated request 200.
Feb 04 07:44:29 debian10 faasd[419]: 2020/02/04 12:44:29 Forwarded [GET] to /system/functions - [200] - 0.010728 seconds
Feb 04 07:44:48 debian10 faasd[419]: [faasd] proxy: http://10.62.0.10:8080/function/figlet1
Feb 04 07:44:48 debian10 faasd[419]: 2020/02/04 12:44:48 GetReplicas [figlet1.] took: 0.009235s
Feb 04 07:44:48 debian10 faasd[419]: 2020/02/04 12:44:48 GetReplicas [figlet1.] took: 0.009253s
Feb 04 07:44:48 debian10 faasd[419]: 2020/02/04 12:44:48 GetReplicas [figlet1.] took: 0.009021s
Feb 04 07:44:48 debian10 faasd[419]: 2020/02/04 12:44:48 GetReplicas [figlet1.] took: 0.009042s
Feb 04 07:44:48 debian10 faasd[419]: 2020/02/04 12:44:48 [Scale 0] function=figlet1 0 => 1 requested
Feb 04 07:44:48 debian10 faasd[419]: 2020/02/04 12:44:48 SetReplicas [figlet1.] took: 0.340124s
Feb 04 07:44:49 debian10 faasd[419]: 2020/02/04 12:44:49 GetReplicas [figlet1.] took: 0.010957s
Feb 04 07:44:49 debian10 faasd[419]: 2020/02/04 12:44:49 GetReplicas [figlet1.] took: 0.010975s
Feb 04 07:44:49 debian10 faasd[419]: 2020/02/04 12:44:49 [Scale] function=figlet1 0 => 1 successful - 0.370325 seconds
Feb 04 07:44:49 debian10 faasd[419]: 2020/02/04 12:44:49 Forwarded [POST] to /function/figlet1 - [500] - 0.003176 seconds
^C% ❯ sudo ls -l /var/lib/cni/results/
total 80K
-rw------- 1 root root 227 Feb 4 07:43 cni-loopback-basic-auth-plugin-674-lo
-rw------- 1 root root 228 Feb 4 07:44 cni-loopback-figlet1-1625-lo
-rw------- 1 root root 228 Feb 4 07:43 cni-loopback-gateway-1139-lo
-rw------- 1 root root 229 Feb 3 16:57 cni-loopback-gateway-15713-lo
-rw------- 1 root root 229 Feb 3 16:57 cni-loopback-nats-15472-lo
-rw------- 1 root root 227 Feb 4 07:43 cni-loopback-nats-865-lo
-rw------- 1 root root 228 Feb 4 07:43 cni-loopback-prometheus-1030-lo
-rw------- 1 root root 229 Feb 3 16:57 cni-loopback-prometheus-15589-lo
-rw------- 1 root root 228 Feb 4 07:43 cni-loopback-queue-worker-1246-lo
-rw------- 1 root root 229 Feb 3 16:57 cni-loopback-queue-worker-15822-lo
-rw------- 1 root root 330 Feb 4 07:43 openfaas-cni-bridge-basic-auth-plugin-674-eth1
-rw------- 1 root root 332 Feb 4 07:44 openfaas-cni-bridge-figlet1-1625-eth1
-rw------- 1 root root 332 Feb 4 07:43 openfaas-cni-bridge-gateway-1139-eth1
-rw------- 1 root root 332 Feb 3 16:57 openfaas-cni-bridge-gateway-15713-eth1
-rw------- 1 root root 332 Feb 3 16:57 openfaas-cni-bridge-nats-15472-eth1
-rw------- 1 root root 330 Feb 4 07:43 openfaas-cni-bridge-nats-865-eth1
-rw------- 1 root root 331 Feb 4 07:43 openfaas-cni-bridge-prometheus-1030-eth1
-rw------- 1 root root 332 Feb 3 16:57 openfaas-cni-bridge-prometheus-15589-eth1
-rw------- 1 root root 332 Feb 4 07:43 openfaas-cni-bridge-queue-worker-1246-eth1
-rw------- 1 root root 332 Feb 3 16:57 openfaas-cni-bridge-queue-worker-15822-eth1
❯ sudo ll /var/lib/cni/networks/openfaas-cni-bridge/
total 44K
-rwxr-x--- 1 root root 0 Feb 3 16:57 lock*
-rw-r--r-- 1 root root 16 Feb 3 16:57 10.62.0.3
-rw-r--r-- 1 root root 22 Feb 3 16:57 10.62.0.4
-rw-r--r-- 1 root root 19 Feb 3 16:57 10.62.0.5
-rw-r--r-- 1 root root 24 Feb 3 16:57 10.62.0.6
-rw-r--r-- 1 root root 27 Feb 4 07:43 10.62.0.7
-rw-r--r-- 1 root root 14 Feb 4 07:43 10.62.0.8
-rw-r--r-- 1 root root 21 Feb 4 07:43 10.62.0.9
-rw-r--r-- 1 root root 18 Feb 4 07:43 10.62.0.10
-rw-r--r-- 1 root root 23 Feb 4 07:43 10.62.0.11
-rw-r--r-- 1 root root 10 Feb 4 07:44 last_reserved_ip.0
-rw-r--r-- 1 root root 18 Feb 4 07:44 10.62.0.12
❯ faas-cli ls
Function Invocations Replicas
figlet1 1 1
❯ curl -d Test http://127.0.0.1:8080/function/figlet1
_____ _
|_ _|__ ___| |_
| |/ _ \/ __| __|
| | __/\__ \ |_
|_|\___||___/\__|
All containers got new IPs. This is already used on K3C where the IPs from created containers are read from the CNI results.
from faasd.
As spoken before, here a list of pros/cons on this implementation:
Current netlink/netns imlementation:
Pros
- Fetches the current IP directly from Linux interface configuration
- Has been used on other well tested libraries like Weave Net
Cons
- Depends on two external libraries
- Depends on two internal modules to provide access to the libraries
- More code == more bugs and more maintainability. Lines: +43 −18,800
CNI Results implementation (this PR)
Pros
- Uses pure artifacts from CNI results and no external libraries or complex namespace/netlink manipulation.
- Less code == less bugs and less maintainability. Lines: +43 −18,800
- Is currently used on new applications based on pure CNI like rancher/k3c
- Based on CNI spec. File format not supposed to change.
Cons
- Result files could be changed by user (only root) resulting in wrong IP
- Is not as tested as other implementation
- Depends on filesystem result files instead of live fetch from Kernel config
from faasd.
Related Issues (20)
- Update to containerd 1.7.0 - last release with 32-bit Arm support
- Support request logging in HOT 2
- Issues with rust HOT 1
- Access host's network from a docker-compose.yaml's service HOT 5
- Support request for Docker installed with faasd (invalid) HOT 5
- Can docker-compose be used with faasd? HOT 8
- Support question on MQTT HOT 4
- Executing install.sh fails because of wrong version HOT 1
- Grafana installation - volume permissions HOT 2
- Multi-node support HOT 3
- Support request for monitoring with cAdvisor HOT 6
- NATS JetStream support HOT 2
- disable printing function invocation duration
- Terrraform install fails on macOS Sonoma HOT 2
- faasd and faasd provider in different ports HOT 1
- faasd namespace support openfaas=true instead of openfaas=1
- Is Scale to Zero included in the Community Edition of OpenFaaS? HOT 8
- support faasd get HOT 1
- Postgresql database connection problem HOT 16
- MemoryLimit needs changing to: MemoryMax
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from faasd.