GithubHelp home page GithubHelp logo

scaleway-k8s-node-coffee's Introduction

scaleway-k8s-node-coffee โ˜•

Kubernetes toolkit controller project for Scaleway k8s nodes, that does a lot of different things based on changes in a Kubernetes cluster (especially Kapsule).

Getting started ๐Ÿš€

Configuration

Below environment variables have to be defined in the controller container to configure it. Leaving a feature-related variable empty will disable it

Variable Description Example
KUBECONFIG optional. KUBECONFIG file path to the cluster you want to run the controller against ~/.kube/config/my-kubeconfig.yaml
SCW_ACCESS_KEY required. Your scaleway project access key (docs) SCWxxxxxxxxxxxxxxxxx
SCW_SECRET_KEY required. Your scaleway project secret key (docs) 11111111-1111-1111-2111-111111111111
SCW_DEFAULT_REGION Your Scaleway DBaaS default region (docs, guides) fr-par
SCW_DEFAULT_ZONE Your Scaleway DBaaS default zone (docs, guides) fr-par-1
RESERVED_IPS_POOL List of already existing reserved IP, comma-separated 51.15.15.15,51.15.15.32
REVERSE_IP_DOMAIN Your desired domain name example.com
DATABASE_IDS List of DBaaS IDs (with optional regional IDs), comma-separated 11111111-1111-1111-2111-111111111111,nl-ams/11111111-1111-1111-2111-111111111112
REDIS_IDS List of Redis IDs (with optional zonal IDs), comma-separated 11111111-1111-1111-2111-111111111111,nl-ams-1/11111111-1111-1111-2111-111111111112
SECURITY_GROUP_IDS List of security group IDs (with optional zonal IDs), comma-separated xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
NUMBER_RETRIES optional. Retries on error amount (default: 30) 15

Local tests

You can test it against a remote cluster by providing the corresponding KUBECONFIG environment variable to the container, like the following :

docker run sh4d1/scaleway-k8s-node-coffee \
  # Remote cluster to run against
  --env KUBECONFIG="~/.kube/config/my-kubeconfig.yaml" \
  ...

Below snippet will build and run the controller (all features enabled) locally against the given cluster (kubeconfig)

# Run the tests and build the image (if working on the project)
make docker-build

# Build the image
docker build -t sh4d1/scaleway-k8s-node-coffee .

# Run it with the required environment variables
docker run sh4d1/scaleway-k8s-node-coffee \
  # Remote cluster to run against
  --env KUBECONFIG="~/.kube/config/my-kubeconfig.yaml" \
  # Authentication/global configuration
  --env SCW_ACCESS_KEY="SCWxxxxxxxxxxxxxxxxx" \
  --env SCW_SECRET_KEY="11111111-1111-1111-2111-111111111111" \
  --env RETRIES_NUMBER="15" \
  # Reserved IP
  --env RESERVED_IPS_POOL="51.15.15.15,51.15.15.32" \
  # Reverse IP
  --env REVERSE_IP_DOMAIN="example.com" \
  # Database ACL
  --env SCW_DEFAULT_REGION="fr-par" \
  --env DATABASE_IDS="11111111-1111-1111-2111-111111111111,nl-ams/11111111-1111-1111-2111-111111111112" \
  # Redis ACL
  --env REDIS_IDS="11111111-1111-1111-2111-111111111111,nl-ams-1/11111111-1111-1111-2111-111111111112" \
  --env SCW_DEFAULT_ZONE="fr-par-1" \
  # Security groups
  --env SECURITY_GROUP_IDS=my-value \

Deploy

Below snippet will deploy the controller on the current cluster context you're authenticated against (kubectl config current-context)

โš ๏ธ Please note that you'll have to edit ./deploy/{configmap,secret}.yaml to define your credentials and custimze your needs!

# Using make (requires to clone the repository)
make deploy

# Using kubectl
kubectl create -f https://raw.githubusercontent.com/Sh4d1/scaleway-k8s-node-coffee/main/deploy/deploy.yaml
kubectl create -f https://raw.githubusercontent.com/Sh4d1/scaleway-k8s-node-coffee/main/deploy/secret.yaml --edit --namespace scaleway-k8s-node-coffee
kubectl create -f https://raw.githubusercontent.com/Sh4d1/scaleway-k8s-node-coffee/main/deploy/configmap.yaml --edit --namespace scaleway-k8s-node-coffee

Helm

A helm implementation of this controller is in progress, feel free to contribute (#7)

Features โœจ

Reserved IP

This feature allows a set of predefined reserved IP to be used as the nodes IP. Once a new node appears, it will try to assign a free reserved IP out of the given list to the node.

Variable(s) ๐Ÿ“

  • RESERVED_IPS_POOL
    • list of already existing reserved IP, comma-separated
    • e.g. 51.15.15.15,51.15.15.32

Notes

  • โ„น๏ธ A label reserved-ip: true will be added to the nodes with a reserved IP.

Reverse IP

This feature allows you to set the reverse IP of the reserved IP to a custom one. It will only work if a reserved IP is already set on the node (to use with the Reserved IP feature).

Variable(s) ๐Ÿ“

  • REVERSE_IP_DOMAIN
    • desired domain name
    • e.g. example.com will update the reserved IP 51.16.17.18 with the reverse 18-17-16-51.example.com

Notes

  • โ„น๏ธ If your domain is hosted on Scaleway, the record such as 18-17-16-51.example.com will be added (and removed if not needed anymore).

Database ACLs

This feature allows to update the ACL rules of several DB to allow of all the cluster nodes (adding new ones, and removing old ones).

Variable(s) ๐Ÿ“

  • DATABASE_IDS
    • list of DBaaS IDs (with optional regional IDs), comma-separated
    • e.g. 11111111-1111-1111-2111-111111111111,nl-ams/11111111-1111-1111-2111-111111111112
  • SCW_DEFAULT_REGION
    • Default DBaaS resources region
    • e.g. fr-par

Notes

  • โ„น๏ธ Will update the ACL of the database with ID 11111111-1111-1111-2111-111111111111 in the region specified by the environment variable SCW_DEFAULT_REGION and the database 11111111-1111-1111-2111-111111111112 in the nl-ams region.

  • โ„น๏ธ If your database is in a different project than the cluster nodes, please set the environment variable NODES_IP_SOURCE to kubernetes.

  • โ„น๏ธ If your DBaaS already have ACL rules allowing your k8s nodes' IPs, and not named following their IDs, you'll have to delete them or rename them with the corresponding nodes' IDs

Redis ACLs

This feature allows to update the ACL rules of several Redis instances to allow of all the cluster nodes (adding new ones, and removing old ones).

Variable(s) ๐Ÿ“

  • REDIS_IDS
    • list of Redis IDs (with optional zonal IDs), comma-separated
    • e.g. 11111111-1111-1111-2111-111111111111,nl-ams-1/11111111-1111-1111-2111-111111111112
  • SCW_DEFAULT_ZONE
    • Default Redis resources zone
    • e.g. fr-par-1

Notes

  • โ„น๏ธ Will update the ACL of the redis instance with ID 11111111-1111-1111-2111-111111111111 in the zone specified by the environment variable SCW_DEFAULT_ZONE and the instance 11111111-1111-1111-2111-111111111112 in the nl-ams-1 zone.

  • โ„น๏ธ If your redis instance is in a different project than the cluster nodes, please set the environment variable NODES_IP_SOURCE to kubernetes.

Security Group

This feature allows you to update multiple security groups with:

  • The Public and Private IPs of all nodes of the cluster
  • The Node Ports of the NodePort and LoadBalancer services

Variable(s) ๐Ÿ“

  • SECURITY_GROUP_IDS
    • list of security group IDs (with optional zonal IDs), comma-separated
    • e.g. 11111111-1111-1111-2111-111111111111,nl-ams-1/11111111-1111-1111-2111-111111111112

Notes

  • โ„น๏ธ However due to several lack of features, the deletion of the rules if best effort for the nodes, and non existent for the services.

Contribution

Feel free to submit any issue, feature request or pull request ๐Ÿ˜„!

Artwork edited, initially from scaleway-cli

scaleway-k8s-node-coffee's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

scaleway-k8s-node-coffee's Issues

Add a label to the node when a reserved ips is assigned

Hello and thanks for this project :)

This a feature request

We have a scenario were we need to approve grant all nodes public IP on a external service.
To be sure our application run on approved nodes, we would like to use a labels/annotations to be used with nodeSelector

Thanks

Could not get instance which is in different project space

When having the database in a different project space as the Kubernetes cluster, the mechanics for adding the Node IPs to the database ACL doesnt work.
Logs:

I0314 18:57:25.560047 1 main.go:28] Collecting coffee beans
W0314 18:57:25.560186 1 client_config.go:614] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0314 18:57:25.563618 1 main.go:50] Starting the coffee machine
I0314 18:57:26.159438 1 database_acls.go:29] whitelisting IP on node scw-k8s-inheaden-dev-preview-e-pool-dev-XXX on database XXX
E0314 18:57:28.660378 1 database_acls.go:79] could not get instance scw-k8s-inheaden-dev-preview-e-pool-dev-XXX: got 0 servers instead of 1
I0314 18:57:28.660427 1 database_acls.go:29] whitelisting IP on node scw-k8s-inheaden-dev-preview-e-pool-dev-XXX on database XXX
E0314 18:57:30.243994 1 database_acls.go:79] could not get instance scw-k8s-inheaden-dev-preview-e-pool-dev-XXX: got 0 servers instead of 1
...

Recommendation:
Get the node IP from the Kubernetes API (describe nodes) instead of the SCW API for instances.

Reconciler error

Hello,

Our nodes are changed everyday (long story) but sometimes, VPC connection is not working and rebooting vpc-node pod results in:

I1020 17:25:00.639714       1 listener.go:44] controller-runtime/metrics "msg"="metrics server is starting to listen"  "addr"=":8080"
I1020 17:25:01.137435       1 node.go:108] setup "msg"="starting manager"  
I1020 17:25:01.139272       1 internal.go:406] controller-runtime/manager "msg"="starting metrics server"  "path"="/metrics"
I1020 17:25:01.140499       1 controller.go:142] controller "msg"="Starting EventSource" "controller"="networkinterface" "reconcilerGroup"="vpc.scaleway.com" "reconcilerKind"="NetworkInterface" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"id":"","nodeName":""},"status":{"linkName":"","macAddress":""}}}
I1020 17:25:01.242497       1 controller.go:142] controller "msg"="Starting EventSource" "controller"="networkinterface" "reconcilerGroup"="vpc.scaleway.com" "reconcilerKind"="NetworkInterface" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"id":""},"status":{}}}
I1020 17:25:01.344076       1 controller.go:149] controller "msg"="Starting Controller" "controller"="networkinterface" "reconcilerGroup"="vpc.scaleway.com" "reconcilerKind"="NetworkInterface" 
I1020 17:25:01.344206       1 controller.go:176] controller "msg"="Starting workers" "controller"="networkinterface" "reconcilerGroup"="vpc.scaleway.com" "reconcilerKind"="NetworkInterface" "worker count"=1
E1020 17:25:01.756659       1 networkinterface_controller.go:144] controllers/NetworkInterface "msg"="unable to get link" "error"="link with address 02:00:00:00:72:23: NIC not found" "networkinterface"={"Namespace":"","Name":"example-scw-sxq7k"} 
E1020 17:25:01.756829       1 controller.go:246] controller "msg"="Reconciler error" "error"="link with address 02:00:00:00:72:23: NIC not found" "controller"="networkinterface" "name"="example-scw-sxq7k" "namespace"="" "reconcilerGroup"="vpc.scaleway.com" "reconcilerKind"="NetworkInterface" 
E1020 17:25:03.206108       1 networkinterface_controller.go:144] controllers/NetworkInterface "msg"="unable to get link" "error"="link with address 02:00:00:00:72:23: NIC not found" "networkinterface"={"Namespace":"","Name":"example-scw-sxq7k"} 
E1020 17:25:03.206251       1 controller.go:246] controller "msg"="Reconciler error" "error"="link with address 02:00:00:00:72:23: NIC not found" "controller"="networkinterface" "name"="example-scw-sxq7k" "namespace"="" "reconcilerGroup"="vpc.scaleway.com" "reconcilerKind"="NetworkInterface" 
E1020 17:25:04.662746       1 networkinterface_controller.go:144] controllers/NetworkInterface "msg"="unable to get link" "error"="link with address 02:00:00:00:72:23: NIC not found" "networkinterface"={"Namespace":"","Name":"example-scw-sxq7k"} 
E1020 17:25:04.662920       1 controller.go:246] controller "msg"="Reconciler error" "error"="link with address 02:00:00:00:72:23: NIC not found" "controller"="networkinterface" "name"="example-scw-sxq7k" "namespace"="" "reconcilerGroup"="vpc.scaleway.com" "reconcilerKind"="NetworkInterface" 
E1020 17:25:06.086721       1 networkinterface_controller.go:144] controllers/NetworkInterface "msg"="unable to get link" "error"="link with address 02:00:00:00:72:23: NIC not found" "networkinterface"={"Namespace":"","Name":"example-scw-sxq7k"} 
E1020 17:25:06.086872       1 controller.go:246] controller "msg"="Reconciler error" "error"="link with address 02:00:00:00:72:23: NIC not found" "controller"="networkinterface" "name"="example-scw-sxq7k" "namespace"="" "reconcilerGroup"="vpc.scaleway.com" "reconcilerKind"="NetworkInterface" 
E1020 17:25:07.542605       1 networkinterface_controller.go:144] controllers/NetworkInterface "msg"="unable to get link" "error"="link with address 02:00:00:00:72:23: NIC not found" "networkinterface"={"Namespace":"","Name":"example-scw-sxq7k"} 
E1020 17:25:07.542748       1 controller.go:246] controller "msg"="Reconciler error" "error"="link with address 02:00:00:00:72:23: NIC not found" "controller"="networkinterface" "name"="example-scw-sxq7k" "namespace"="" "reconcilerGroup"="vpc.scaleway.com" "reconcilerKind"="NetworkInterface" 
E1020 17:25:08.978448       1 networkinterface_controller.go:144] controllers/NetworkInterface "msg"="unable to get link" "error"="link with address 02:00:00:00:72:23: NIC not found" "networkinterface"={"Namespace":"","Name":"example-scw-sxq7k"} 
E1020 17:25:08.978572       1 controller.go:246] controller "msg"="Reconciler error" "error"="link with address 02:00:00:00:72:23: NIC not found" "controller"="networkinterface" "name"="example-scw-sxq7k" "namespace"="" "reconcilerGroup"="vpc.scaleway.com" "reconcilerKind"="NetworkInterface" 
E1020 17:25:10.621461       1 networkinterface_controller.go:144] controllers/NetworkInterface "msg"="unable to get link" "error"="link with address 02:00:00:00:72:23: NIC not found" "networkinterface"={"Namespace":"","Name":"example-scw-sxq7k"} 
E1020 17:25:10.622092       1 controller.go:246] controller "msg"="Reconciler error" "error"="link with address 02:00:00:00:72:23: NIC not found" "controller"="networkinterface" "name"="example-scw-sxq7k" "namespace"="" "reconcilerGroup"="vpc.scaleway.com" "reconcilerKind"="NetworkInterface" 
E1020 17:25:12.095565       1 networkinterface_controller.go:144] controllers/NetworkInterface "msg"="unable to get link" "error"="link with address 02:00:00:00:72:23: NIC not found" "networkinterface"={"Namespace":"","Name":"example-scw-sxq7k"} 
E1020 17:25:12.095689       1 controller.go:246] controller "msg"="Reconciler error" "error"="link with address 02:00:00:00:72:23: NIC not found" "controller"="networkinterface" "name"="example-scw-sxq7k" "namespace"="" "reconcilerGroup"="vpc.scaleway.com" "reconcilerKind"="NetworkInterface" 
E1020 17:25:13.571530       1 networkinterface_controller.go:144] controllers/NetworkInterface "msg"="unable to get link" "error"="link with address 02:00:00:00:72:23: NIC not found" "networkinterface"={"Namespace":"","Name":"example-scw-sxq7k"} 
E1020 17:25:13.642116       1 controller.go:246] controller "msg"="Reconciler error" "error"="link with address 02:00:00:00:72:23: NIC not found" "controller"="networkinterface" "name"="example-scw-sxq7k" "namespace"="" "reconcilerGroup"="vpc.scaleway.com" "reconcilerKind"="NetworkInterface" 
E1020 17:25:15.368710       1 networkinterface_controller.go:144] controllers/NetworkInterface "msg"="unable to get link" "error"="link with address 02:00:00:00:72:23: NIC not found" "networkinterface"={"Namespace":"","Name":"example-scw-sxq7k"} 
E1020 17:25:15.368933       1 controller.go:246] controller "msg"="Reconciler error" "error"="link with address 02:00:00:00:72:23: NIC not found" "controller"="networkinterface" "name"="example-scw-sxq7k" "namespace"="" "reconcilerGroup"="vpc.scaleway.com" "reconcilerKind"="NetworkInterface" 
E1020 17:25:18.359452       1 networkinterface_controller.go:144] controllers/NetworkInterface "msg"="unable to get link" "error"="link with address 02:00:00:00:72:23: NIC not found" "networkinterface"={"Namespace":"","Name":"example-scw-sxq7k"} 
E1020 17:25:18.359611       1 controller.go:246] controller "msg"="Reconciler error" "error"="link with address 02:00:00:00:72:23: NIC not found" "controller"="networkinterface" "name"="example-scw-sxq7k" "namespace"="" "reconcilerGroup"="vpc.scaleway.com" "reconcilerKind"="NetworkInterface" 
E1020 17:25:23.932903       1 networkinterface_controller.go:144] controllers/NetworkInterface "msg"="unable to get link" "error"="link with address 02:00:00:00:72:23: NIC not found" "networkinterface"={"Namespace":"","Name":"example-scw-sxq7k"} 
E1020 17:25:23.933131       1 controller.go:246] controller "msg"="Reconciler error" "error"="link with address 02:00:00:00:72:23: NIC not found" "controller"="networkinterface" "name"="example-scw-sxq7k" "namespace"="" "reconcilerGroup"="vpc.scaleway.com" "reconcilerKind"="NetworkInterface" 
E1020 17:25:34.586873       1 networkinterface_controller.go:144] controllers/NetworkInterface "msg"="unable to get link" "error"="link with address 02:00:00:00:72:23: NIC not found" "networkinterface"={"Namespace":"","Name":"example-scw-sxq7k"} 
E1020 17:25:34.587262       1 controller.go:246] controller "msg"="Reconciler error" "error"="link with address 02:00:00:00:72:23: NIC not found" "controller"="networkinterface" "name"="example-scw-sxq7k" "namespace"="" "reconcilerGroup"="vpc.scaleway.com" "reconcilerKind"="NetworkInterface" 
E1020 17:25:55.501756       1 networkinterface_controller.go:144] controllers/NetworkInterface "msg"="unable to get link" "error"="link with address 02:00:00:00:72:23: NIC not found" "networkinterface"={"Namespace":"","Name":"example-scw-sxq7k"} 
E1020 17:25:55.502024       1 controller.go:246] controller "msg"="Reconciler error" "error"="link with address 02:00:00:00:72:23: NIC not found" "controller"="networkinterface" "name"="example-scw-sxq7k" "namespace"="" "reconcilerGroup"="vpc.scaleway.com" "reconcilerKind"="NetworkInterface"

It's second time happening in 3 months. Replacing the node fixes the issue.

If a node as already a reserved ip, the label should be present on node anyway

After an upgrade of k8s-node-coffee, i expected to see the label: reserved-ip: true

The controller correctly see the reserved ip is assigned

I0409 09:52:59.422153 1 reserved_ip.go:18] adding a reserved IP on node scw-staging-staging-0cc9ecaafded4f819ffb839882
W0409 09:52:59.661995 1 reserved_ip.go:44] node scw-staging-staging-0cc9ecaafded4f819ffb839882 already have a public IP

But the label is not present :)

I can handle it manually would be great to fix ^^

where do i find the settings required

Hi,,

this is a bit of a noobie question, for the following 2 files we need to supply our own settings, but I would like to confirm where we grab them from

Secrets

where is the Zone and Region Id's, you can see I have guessed these values:

image

ConfigMap

according to the URL, when I'm looking at the security group, I have an fr-par-1/guid, is this correct?

image

any help would be appreciated

IP reverse broken?

Looks like the IP reverse feature is broken:

could not update reverse on IP W.X.Y.Z for node scw-*****: scaleway-sdk-go: invalid argument(s): reverse does not respect constraint, Your reverse must resolve. Ensure the command 'dig +short Z.Y.X.W.*.fr' matches your IP address 'W.X.Y.Z'. If it does, please contact our support.

Not sure since when, maybe the Scaleway API now enforce a reverse DNS record to be created?

[Reserved IP] WARNING no available reserved IPs for node

Hello,
After deployed the controller to a Kubernetes Kapsule cluster at Scaleway
kubectl create -f https://raw.githubusercontent.com/Sh4d1/scaleway-k8s-node-coffee/main/deploy.yaml

set the secret

apiVersion: v1
stringData:
  SCW_ACCESS_KEY: XXXXXXXXXXX
  SCW_SECRET_KEY: XXXXXXXXXXX
  SCW_DEFAULT_ZONE: nl-ams-1
  SCW_DEFAULT_REGION: nl-ams
kind: Secret
metadata:
  name: scaleway-k8s-node-coffee
  namespace: scaleway-k8s-node-coffee
type: Opaque

set the configuration

apiVersion: v1
kind: ConfigMap
metadata:
  name: scaleway-k8s-node-coffee
  namespace: scaleway-k8s-node-coffee
data:
  REVERSE_IP_DOMAIN: ""
  DATABASE_IDS: ""
  REDIS_IDS: ""
  RESERVED_IPS_POOL: "51.15.15.32"
  SECURITY_GROUP_IDS: ""
  RETRIES_NUMBER: "30"

Deployed a generic deployment that pointed to a node pool (with autoscaling on)
I got the below warning after the first node came up
I0915 13:46:52.944681 1 reserved_ip.go:18] adding a reserved IP on node scw-t2-k8s-dev-pool-test-reservedip-c7c12842e8
W0915 13:46:53.241201 1 reserved_ip.go:60] no available reserved IPs for node scw-t2-k8s-dev-pool-test-reservedip-c7c12842e8

Checking the related code

if err != nil {
		klog.Errorf("could not get a free IP for node %s: %v", nodeName, err)
		return err
	}

if ip == nil {
		klog.Warningf("no available reserved IPs for node %s", nodeName)
		return nil
	}

It seems that ip and err are nil because of

ip, err := c.getFreeIP()

We know the result of the function getFreeIP() yet: nil nil

func (c *NodeController) getFreeIP() (*instance.IP, error) {
	instanceAPI := instance.NewAPI(c.scwClient)

	ipsList, err := instanceAPI.ListIPs(&instance.ListIPsRequest{}, scw.WithAllPages())
	if err != nil {
		return nil, err
	}

	for _, ip := range ipsList.IPs {
		if ip.Server == nil && stringInSlice(ip.Address.String(), c.reservedIPs) {
			return ip, nil
		}
	}
	return nil, nil
}

Therefore the following statement is always false
ip.Server == nil && stringInSlice(ip.Address.String(), c.reservedIPs)
Could you please help me on addressing the issue?
I really appreciate any help you can provide.

Best regards
Michele

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.