haproxytech / client-native Goto Github PK
View Code? Open in Web Editor NEWGo client for HAProxy configuration and runtime API
License: Apache License 2.0
Go client for HAProxy configuration and runtime API
License: Apache License 2.0
It seems that the global part lacks the maxconnrate option. Do you have any idea to solve this problem?
When attempting to update unique-id-format
within a frontend section a panic is encountered.
To duplicate:
$ curl -XPUT -sH 'Content-Type: application/json' -u dataplaneapi:mypassword 'localhost:5555/v2/services/haproxy/configuration/frontends/fe_main?version=1' -d '{"name":"fe_main", "unique_id_format":"%{+X}o_%ci:%cp_%fi:%fp_%Ts_%rt:%pid" }'|python -m json.tool
{
"code": 500,
"message": "reflect: call of reflect.Value.Elem on string Value: /home/go/pkg/mod/github.com/haproxytech/client-native/[email protected]/configuration/configuration.go:1473 setFieldValue"
}
Stacktrace:
time="2020-05-03T21:36:15-04:00" level=error msg="Panic reflect: call of reflect.Value.Elem on string Value" stack_trace="
/home/go/pkg/mod/github.com/haproxytech/client-native/[email protected]/configuration/configuration.go:1473 setFieldValue
/home/go/pkg/mod/github.com/haproxytech/client-native/[email protected]/configuration/configuration.go:828 CreateEditSection
/home/go/pkg/mod/github.com/haproxytech/client-native/[email protected]/configuration/configuration.go:1674 (*Client).editSection
/home/go/pkg/mod/github.com/haproxytech/client-native/[email protected]/configuration/frontend.go:100 (*Client).EditFrontend
/home/src/dataplane/dataplaneapi/handlers/frontend.go:194 (*ReplaceFrontendHandlerImpl).Handle
/home/src/dataplane/dataplaneapi/operations/frontend/replace_frontend.go:84 (*ReplaceFrontend).ServeHTTP
/home/go/pkg/mod/github.com/go-openapi/[email protected]/middleware/operation.go:28 NewOperationExecutor.func1
/snap/go/5646/src/net/http/server.go:2012 HandlerFunc.ServeHTTP
/home/go/pkg/mod/github.com/go-openapi/[email protected]/middleware/router.go:77 NewRouter.func1
/snap/go/5646/src/net/http/server.go:2012 HandlerFunc.ServeHTTP
/home/go/pkg/mod/github.com/go-openapi/[email protected]/middleware/redoc.go:72 Redoc.func1
/snap/go/5646/src/net/http/server.go:2012 HandlerFunc.ServeHTTP
/home/go/pkg/mod/github.com/go-openapi/[email protected]/middleware/spec.go:46 Spec.func1
/snap/go/5646/src/net/http/server.go:2012 HandlerFunc.ServeHTTP
/home/src/dataplane/dataplaneapi/adapters/adapters.go:152 RecoverMiddleware.func1.1
/snap/go/5646/src/net/http/server.go:2012 HandlerFunc.ServeHTTP
/home/go/pkg/mod/github.com/rs/[email protected]/cors.go:219 (*Cors).Handler.func1
/snap/go/5646/src/net/http/server.go:2012 HandlerFunc.ServeHTTP
/home/src/dataplane/dataplaneapi/adapters/adapters.go:166 LoggingMiddleware.func1.1
/snap/go/5646/src/net/http/server.go:2012 HandlerFunc.ServeHTTP
/snap/go/5646/src/net/http/server.go:2807 serverHandler.ServeHTTP
/snap/go/5646/src/net/http/server.go:1895 (*conn).serve
/snap/go/5646/src/runtime/asm_amd64.s:1373 goexit" time="2020-05-03T21:36:15-04:00" level=info msg="completed handling request" length=197B status=500 took=4.697327ms
EditServer has a linear time complexity because all backend servers are parsed for each call.
It would be useful to have a constant execution time especially this tend to be called frequently to scale up/down backend servers in Ingress Controller
runtime initialization: here
which ultimately probes socket to check if its alive here: here
However, is that really essential? Ultimately, every operation on socket is done via this function which is a full sequence of Dial/Write/Close
. That is great, because this way, if HAproxy restarts or crashes while the client-native runtime object is still alive in another process, it can gracefully recover from it as we do a full Dial/Write/Close
unless the socket path changed etc, that's not in scope for this discussion.
Now, with that thought in mind, why is checking aliveness
a hard requirement under runtime.Init
? At best, we should make it optional with default as check enabled
.
Thoughts?
Related with #1. We need this for implementing header based routing in the HAProxy Ingress controller.
I am running the below code in Minishift v3.11 but unable to initialise the configuration client and getting the below error
./haproxy-main.go:42:17: undefined: configuration
./haproxy-main.go:43:16: undefined: configuration
./haproxy-main.go:54:25: undefined: configuration
./haproxy-main.go:57:11: undefined: api
./haproxy-main.go:61:20: undefined: runtime_api
./haproxy-main.go:73:16: too many arguments to return
have (nil)
want ()
./haproxy-main.go:87:15: undefined: h
./haproxy-main.go:87:50: undefined: t
package main
import (
"log"
"fmt"
corev1client "k8s.io/client-go/kubernetes/typed/core/v1"
"k8s.io/client-go/tools/clientcmd"
"github.com/haproxytech/client-native"
)
func main() {
// Instantiate loader for kubeconfig file.
kubeconfig := clientcmd.NewNonInteractiveDeferredLoadingClientConfig(
clientcmd.NewDefaultClientConfigLoadingRules(),
&clientcmd.ConfigOverrides{},
)
// Determine the Namespace referenced by the current context in the
// kubeconfig file.
namespace, _, err := kubeconfig.Namespace()
if err != nil {
log.Fatal(err)
}
// Get a rest.Config from the kubeconfig file. This will be passed into all
// the client objects we create.
restconfig, err := kubeconfig.ClientConfig()
if err != nil {
log.Fatal(err)
}
// Create a Kubernetes core/v1 client.
coreclient, err := corev1client.NewForConfig(restconfig)
if err != nil {
log.Fatal(err)
}
// Haproxy client
confClient := &configuration.Client{}
confParams := configuration.ClientParams{
ConfigurationFile: "/etc/haproxy/haproxy.cfg",
Haproxy: "/usr/sbin/haproxy",
UseValidation: true,
PersistentTransactions: true,
TransactionDir: "/tmp/haproxy",
}
err = confClient.Init(confParams)
if err != nil {
fmt.Println("Error setting up configuration client, using default one")
confClient, err = configuration.DefaultClient()
if err != nil {
fmt.Println("Error setting up default configuration client, exiting...")
api.ServerShutdown()
}
}
runtimeClient := &runtime_api.Client{}
globalConf, err := confClient.GetGlobalConfiguration("")
if err == nil {
socketList := make([]string, 0, 1)
runtimeAPIs := globalConf.Data.RuntimeApis
if len(runtimeAPIs) != 0 {
for _, r := range runtimeAPIs {
socketList = append(socketList, *r.Address)
}
if err := runtimeClient.Init(socketList, "", 0); err != nil {
fmt.Println("Error setting up runtime client, not using one")
return nil
}
} else {
fmt.Println("Runtime API not configured, not using it")
runtimeClient = nil
}
} else {
fmt.Println("Cannot read runtime API configuration, not using it")
runtimeClient = nil
}
client := &client_native.HAProxyClient{}
client.Init(confClient, runtimeClient)
bcks, err := h.Client.Configuration.GetBackends(t)
if err != nil {
fmt.Println(err.Error())
}
backendsJSON, err := bcks.MarshallBinary()
if err != nil {
fmt.Println(err.Error())
}
fmt.Println(string(backendsJSON))
}
It seems there's no way to write the log-format
directive currently.
frontend my-frontend
log-format "..."
If you give me some help I can send a PR for this, I'm not sure how to work with the code generation.
When attempting to update http_reuse
within the defaults section a panic is encountered.
To duplicate:
$ curl -XPUT -sH 'Content-Type: application/json' -u dataplaneapi:mypassword 'localhost:5555/v2/services/haproxy/configuration/defaults?version=1' -d '{"http_reuse":"safe"}'
{"code":500,"message":"reflect: call of reflect.Value.Elem on string Value: /home/src/dataplane/client-native/configuration/configuration.go:1450 setFieldValue"}```
Stack trace:
time="2020-05-03T20:00:17-04:00" level=error msg="Panic reflect: call of reflect.Value.Elem on string Value" stack_trace="
/home/src/dataplane/client-native/configuration/configuration.go:1450 setFieldValue
/home/src/dataplane/client-native/configuration/configuration.go:828 CreateEditSection
/home/src/dataplane/client-native/configuration/configuration.go:1674 (*Client).editSection
/home/src/dataplane/client-native/configuration/defaults.go:53 (*Client).PushDefaultsConfiguration
/home/src/dataplane/dataplaneapi/handlers/defaults.go:75 (*ReplaceDefaultsHandlerImpl).Handle
/home/src/dataplane/dataplaneapi/operations/defaults/replace_defaults.go:84 (*ReplaceDefaults).ServeHTTP
/home/go/pkg/mod/github.com/go-openapi/[email protected]/middleware/operation.go:28 NewOperationExecutor.func1
/snap/go/5646/src/net/http/server.go:2012 HandlerFunc.ServeHTTP
/home/go/pkg/mod/github.com/go-openapi/[email protected]/middleware/router.go:77 NewRouter.func1
/snap/go/5646/src/net/http/server.go:2012 HandlerFunc.ServeHTTP
/home/go/pkg/mod/github.com/go-openapi/[email protected]/middleware/redoc.go:72 Redoc.func1
/snap/go/5646/src/net/http/server.go:2012 HandlerFunc.ServeHTTP
/home/go/pkg/mod/github.com/go-openapi/[email protected]/middleware/spec.go:46 Spec.func1
/snap/go/5646/src/net/http/server.go:2012 HandlerFunc.ServeHTTP
/home/src/dataplane/dataplaneapi/adapters/adapters.go:152 RecoverMiddleware.func1.1
/snap/go/5646/src/net/http/server.go:2012 HandlerFunc.ServeHTTP
/home/go/pkg/mod/github.com/rs/[email protected]/cors.go:219 (*Cors).Handler.func1
/snap/go/5646/src/net/http/server.go:2012 HandlerFunc.ServeHTTP
/home/src/dataplane/dataplaneapi/adapters/adapters.go:166 LoggingMiddleware.func1.1
/snap/go/5646/src/net/http/server.go:2012 HandlerFunc.ServeHTTP
/snap/go/5646/src/net/http/server.go:2807 serverHandler.ServeHTTP
/snap/go/5646/src/net/http/server.go:1895 (*conn).serve
/snap/go/5646/src/runtime/asm_amd64.s:1373 goexit"
When attempting to generate a golang client based on the openapiv3 specification exposed by the dataplane api, I got the following error:
error generating code: error generating client with responses: error generating client-with-responses.tmpl: template: client-
with-responses.tmpl:50:14: executing "client-with-responses.tmpl" at <getResponseTypeDefinitions .>: error calling
getResponseTypeDefinitions: Unable to determine Go type for GetOneStorageGeneralFile.application/octet-stream: error
resolving primitive type: unhandled Schema type: file
This was using oapi-codegen like so:
/home/alex/go/bin/oapi-codegen --package=main -generate=types,client -o ./client.gen.go /home/alex/client/openapi.json
I took the json and tried processing it with https://editor.swagger.io/ and it complained about the same problem.
In general_files.yaml, there are two spots where we define the scheme type as file
:
I was able to workaround the problem by setting the type to string
, but this would certainly be problematic whenever I get around to using these Storage APIs
If I understand correctly, clientnative.HAProxyClient
is the interface that is used to manage HAProxy instance in this library.
I can see that, an operation like AddMapEntry
is available via both *SingleRuntime.AddMapEntry
and clientnative.HAProxyClient
, link, hence a map entry can be added via clientnative.HAProxyClient
.
How are we supposed to perform an operation like SetCertEntry
from clientnative.HAProxyClient
? link
Another observation is that runtimes
is a private field, hence we can't directly access a SingleRuntime
from clientnative.HAProxyClient
. Is there a way that I overlooked? The end goal is to update certificates by using this library.
N/A
Using the code below:
client2 := v2.SingleRuntime{}
err = client2.Init("/var/run/hapee-lb.sock", 1, 1)
if err != nil {
log.Println(err)
}
env, err := client2.ExecuteRaw("show env")
if err != nil {
log.Println(err)
}
log.Println(env)
I keep getting
2021/03/30 16:32:32 Unknown command. Please enter one of the following commands only :
help : this message
prompt : toggle interactive mode with prompt
quit : disconnect
After a talk with @bedis we concluded that support for this option is needed for my use case.
What I'm trying to achieve is to be able to have header based routing in the HAProxy Ingress controller and for that I would have a config similar to the next one:
resolvers dns
hold nx 3s
hold obsolete 0s
hold other 3s
hold valid 10s
parse-resolv-conf
timeout retry 10s
frontend frontend-host-traffic
bind :8140 ssl crt /cert ca-file /ca verify required no-sslv3 v4v6
default_backend backend-host-traffic
http-request set-var(sess.namespace) hdr(X-Namespace)
http-request set-var(sess.service) hdr(X-Service)
http-request do-resolve(txn.service,dns,ipv4) str(),concat(,sess.service,.),concat(,sess.namespace,.svc.cluster.local)
http-request set-header X-Client-Verify-Real %[ssl_c_verify]
http-request set-header X-Client-Verify NONE if !{ hdr_val(X-Client-Verify-Real) eq 0 }
http-request set-header X-Client-Verify SUCCESS if { hdr_val(X-Client-Verify-Real) eq 0 }
http-request set-header X-Client-DN CN=%{+Q}[ssl_c_s_dn(cn)]
http-request deny if { method PUT POST } { hdr_val(Content-length) ge 62914560 }
backend backend-host-traffic
http-request set-dst var(txn.service)
http-request set-dst-port int(8140)
server service 0.0.0.0:0
Most of the options are supported but this specific one is not. Let me know if I can be of any help!
Please improve usage examples. Current Readme is hard to understand.
A good usage examples would be small application snippets with all imports and results are visible. I.e. see
https://github.com/nats-io/go-nats-examples/tree/main/api-examples
https://github.com/gnet-io/gnet-examples
When enabling stats in a frontend it appears that the keyword "enabled" is being used and not "enable" which causes the following error within HAProxy:
[ALERT] 123/220133 (22396) : parsing [haproxy.cfg:82]: unknown stats parameter 'enabled', expects 'admin', 'uri', 'realm', 'auth', 'scope', 'enable', 'hide-version', 'show-node', 'show-desc' or 'show-legends'.
To duplicate:
curl -sH 'Content-Type: application/json' -u dataplaneapi:mypassword 'localhost:5555/v2/services/haproxy/configuration/frontends?version=1' -d '{"name":"stats", "stats_options": {"stats_enable":true, "stats_uri_prefix":"/", "stats_refresh_delay":10000} }'
HAProxy documentation tells that deny status is 403 by default and not required:
http-request deny [deny_status <status>] [ { if | unless } <condition> ]
http-request deny [ { status | deny_status } <code>] [content-type <type>]
[ { default-errorfiles | errorfile <file> | errorfiles <name> |
file <file> | lf-file <file> | string <str> | lf-string <fmt> } ]
[ hdr <name> <fmt> ]*
[ { if | unless } <condition> ]
This stops the evaluation of the rules and immediately rejects the request. By default an HTTP 403 error is returned.
But parser actually requires deny status: at
it sets an error, and at it ignores resulting rule. Thus an inconsistent behaviour is observed: configuration client can create a rule which it cannot parse.client-native fails under certain conditions when handling multiple transactions by a consumer that uses the client in go routines (like dataplaneapi).
Example Stacktrace:
Jul 12 04:05:39 localhost env[42327]: fatal error: concurrent map writes
Jul 12 04:05:39 localhost env[42327]: goroutine 832915 [running]:
Jul 12 04:05:39 localhost env[42327]: runtime.throw(0x1bff306, 0x15)
Jul 12 04:05:39 localhost env[42327]: runtime/panic.go:1116 +0x72 fp=0xc002653090 sp=0xc002653060 pc=0x4360f2
Jul 12 04:05:39 localhost env[42327]: runtime.mapassign_faststr(0x19e1b40, 0xc00108f260, 0xc00082d170, 0x24, 0x0)
Jul 12 04:05:39 localhost env[42327]: runtime/map_faststr.go:291 +0x3d8 fp=0xc0026530f8 sp=0xc002653090 pc=0x414398
Jul 12 04:05:39 localhost env[42327]: github.com/haproxytech/client-native/v2/configuration.(*Client).AddParser(0xc000fc4300, 0xc00082d170, 0x24, 0x0, 0x0)
Jul 12 04:05:39 localhost env[42327]: github.com/haproxytech/client-native/[email protected]/configuration/configuration.go:219 +0x30e fp=0xc002653188 sp=0xc0026530f8 pc=0xa523
Jul 12 04:05:39 localhost env[42327]: github.com/haproxytech/client-native/v2/configuration.(*Transaction).startTransaction(0xc000fc4300, 0x2, 0xc00235f800, 0xc001ff4b90, 0xc0026532b8, 0xb8ac6a)
Jul 12 04:05:39 localhost env[42327]: github.com/haproxytech/client-native/[email protected]/configuration/transaction.go:115 +0x30d fp=0xc002653270 sp=0xc002653188 pc=0xaaf36d
Jul 12 04:05:39 localhost env[42327]: github.com/haproxytech/client-native/v2/configuration.(*Transaction).StartTransaction(...)
Jul 12 04:05:39 localhost env[42327]: github.com/haproxytech/client-native/[email protected]/configuration/transaction.go:86
Jul 12 04:05:39 localhost env[42327]: github.com/haproxytech/dataplaneapi/handlers.(*StartTransactionHandlerImpl).Handle(0xc000128430, 0xc0023b8900, 0x2, 0x1961be0, 0xc0021e0590, 0x0, 0x0)
Jul 12 04:05:39 localhost env[42327]: github.com/haproxytech/dataplaneapi/handlers/transaction.go:66 +0x46 fp=0xc0026532c8 sp=0xc002653270 pc=0x180dcc6
Jul 12 04:05:39 localhost env[42327]: github.com/haproxytech/dataplaneapi/handlers.RateLimitedStartTransactionHandlerImpl.Handle(0x1f4dd00, 0xc001126010, 0x1f475a0, 0xc000128430, 0xc0023b8900, 0x2, 0x1961be
Jul 12 04:05:39 localhost env[42327]: github.com/haproxytech/dataplaneapi/handlers/transaction.go:165 +0x12c fp=0xc002653320 sp=0xc0026532c8 pc=0x180ee8c
Jul 12 04:05:39 localhost env[42327]: github.com/haproxytech/dataplaneapi/handlers.(*RateLimitedStartTransactionHandlerImpl).Handle(0xc001114080, 0xc0023b8900, 0x2, 0x1961be0, 0xc0021e0590, 0x0, 0x0)
Jul 12 04:05:39 localhost env[42327]: <autogenerated>:1 +0x8a fp=0xc002653380 sp=0xc002653320 pc=0x181120a
Jul 12 04:05:39 localhost env[42327]: github.com/haproxytech/dataplaneapi/operations/transactions.(*StartTransaction).ServeHTTP(0xc000fa43c0, 0x1f87bc0, 0xc00235f7e0, 0xc0023b8900)
Jul 12 04:05:39 localhost env[42327]: github.com/haproxytech/dataplaneapi/operations/transactions/start_transaction.go:86 +0x290 fp=0xc002653408 sp=0xc002653380 pc=0x17db430
Jul 12 04:05:39 localhost env[42327]: github.com/go-openapi/runtime/middleware.NewOperationExecutor.func1(0x1f87bc0, 0xc00235f7e0, 0xc0023b8800)
Jul 12 04:05:39 localhost env[42327]: github.com/go-openapi/[email protected]/middleware/operation.go:28 +0x75 fp=0xc002653440 sp=0xc002653408 pc=0xb2ae35
Jul 12 04:05:39 localhost env[42327]: net/http.HandlerFunc.ServeHTTP(0xc000fe0e80, 0x1f87bc0, 0xc00235f7e0, 0xc0023b8800)
Jul 12 04:05:39 localhost env[42327]: net/http/server.go:2042 +0x44 fp=0xc002653468 sp=0xc002653440 pc=0x7371c4
Jul 12 04:05:39 localhost env[42327]: github.com/go-openapi/runtime/middleware.NewRouter.func1(0x1f87bc0, 0xc00235f7e0, 0xc000b03f00)
Jul 12 04:05:39 localhost env[42327]: github.com/go-openapi/[email protected]/middleware/router.go:77 +0x359 fp=0xc002653538 sp=0xc002653468 pc=0xb2b619
Jul 12 04:05:39 localhost env[42327]: net/http.HandlerFunc.ServeHTTP(0xc000f46c40, 0x1f87bc0, 0xc00235f7e0, 0xc000b03f00)
Jul 12 04:05:39 localhost env[42327]: net/http/server.go:2042 +0x44 fp=0xc002653560 sp=0xc002653538 pc=0x7371c4
Jul 12 04:05:39 localhost env[42327]: github.com/go-openapi/runtime/middleware.Redoc.func1(0x1f87bc0, 0xc00235f7e0, 0xc000b03f00)
Jul 12 04:05:39 localhost env[42327]: github.com/go-openapi/[email protected]/middleware/redoc.go:72 +0x28e fp=0xc002653608 sp=0xc002653560 pc=0xb2b0ee
Jul 12 04:05:39 localhost env[42327]: net/http.HandlerFunc.ServeHTTP(0xc0011fc880, 0x1f87bc0, 0xc00235f7e0, 0xc000b03f00)
Jul 12 04:05:39 localhost env[42327]: net/http/server.go:2042 +0x44 fp=0xc002653630 sp=0xc002653608 pc=0x7371c4
Jul 12 04:05:39 localhost env[42327]: github.com/go-openapi/runtime/middleware.Spec.func1(0x1f87bc0, 0xc00235f7e0, 0xc000b03f00)
Jul 12 04:05:39 localhost env[42327]: github.com/go-openapi/[email protected]/middleware/spec.go:46 +0x190 fp=0xc0026536b8 sp=0xc002653630 pc=0xb2b7d0
Jul 12 04:05:39 localhost env[42327]: net/http.HandlerFunc.ServeHTTP(0xc0011fc8c0, 0x1f87bc0, 0xc00235f7e0, 0xc000b03f00)
Jul 12 04:05:39 localhost env[42327]: net/http/server.go:2042 +0x44 fp=0xc0026536e0 sp=0xc0026536b8 pc=0x7371c4
Jul 12 04:05:39 localhost env[42327]: github.com/haproxytech/dataplaneapi/adapters.RecoverMiddleware.func1.1(0x1f87bc0, 0xc00235f7e0, 0xc000b03f00)
Jul 12 04:05:39 localhost env[42327]: github.com/haproxytech/dataplaneapi/adapters/adapters.go:152 +0x9d fp=0xc002653750 sp=0xc0026536e0 pc=0xbd257d
Jul 12 04:05:39 localhost env[42327]: net/http.HandlerFunc.ServeHTTP(0xc000340480, 0x1f87bc0, 0xc00235f7e0, 0xc000b03f00)
Jul 12 04:05:39 localhost env[42327]: net/http/server.go:2042 +0x44 fp=0xc002653778 sp=0xc002653750 pc=0x7371c4
Jul 12 04:05:39 localhost env[42327]: github.com/rs/cors.(*Cors).Handler.func1(0x1f87bc0, 0xc00235f7e0, 0xc000b03f00)
Jul 12 04:05:39 localhost env[42327]: github.com/rs/[email protected]/cors.go:219 +0x1b9 fp=0xc0026537d0 sp=0xc002653778 pc=0xbce179
Jul 12 04:05:39 localhost env[42327]: net/http.HandlerFunc.ServeHTTP(0xc000f470e0, 0x1f87bc0, 0xc00235f7e0, 0xc000b03f00)
Jul 12 04:05:39 localhost env[42327]: net/http/server.go:2042 +0x44 fp=0xc0026537f8 sp=0xc0026537d0 pc=0x7371c4
Jul 12 04:05:39 localhost env[42327]: github.com/haproxytech/dataplaneapi/adapters.UniqueIDMiddleware.func1.1(0x1f87bc0, 0xc00235f7e0, 0xc000b03f00)
Jul 12 04:05:39 localhost env[42327]: github.com/haproxytech/dataplaneapi/adapters/adapters.go:172 +0x1fb fp=0xc002653a18 sp=0xc0026537f8 pc=0xbd28bb
Jul 12 04:05:39 localhost env[42327]: net/http.HandlerFunc.ServeHTTP(0xc000f47100, 0x1f87bc0, 0xc00235f7e0, 0xc000b03f00)
Jul 12 04:05:39 localhost env[42327]: net/http/server.go:2042 +0x44 fp=0xc002653a40 sp=0xc002653a18 pc=0x7371c4
Jul 12 04:05:39 localhost env[42327]: github.com/haproxytech/dataplaneapi/adapters.LoggingMiddleware.func1.1(0x1f87c00, 0xc0018ce560, 0xc000b03f00)
Jul 12 04:05:39 localhost env[42327]: github.com/haproxytech/dataplaneapi/adapters/adapters.go:191 +0x155 fp=0xc002653ae0 sp=0xc002653a40 pc=0xbd2f15
Jul 12 04:05:39 localhost env[42327]: net/http.HandlerFunc.ServeHTTP(0xc000f47120, 0x1f87c00, 0xc0018ce560, 0xc000b03f00)
Jul 12 04:05:39 localhost env[42327]: net/http/server.go:2042 +0x44 fp=0xc002653b08 sp=0xc002653ae0 pc=0x7371c4
Jul 12 04:05:39 localhost env[42327]: github.com/lestrrat-go/apache-logformat.(*ApacheLog).Wrap.func1(0x1f87fc0, 0xc0021267e0, 0xc000b03f00)
Jul 12 04:05:39 localhost env[42327]: github.com/lestrrat-go/[email protected]/logformat.go:65 +0x19f fp=0xc002653bc8 sp=0xc002653b08 pc=0xb8949f
Jul 12 04:05:39 localhost env[42327]: net/http.HandlerFunc.ServeHTTP(0xc0003404e0, 0x1f87fc0, 0xc0021267e0, 0xc000b03f00)
Jul 12 04:05:39 localhost env[42327]: net/http/server.go:2042 +0x44 fp=0xc002653bf0 sp=0xc002653bc8 pc=0x7371c4
Jul 12 04:05:39 localhost env[42327]: net/http.serverHandler.ServeHTTP(0xc002127500, 0x1f87fc0, 0xc0021267e0, 0xc000b03f00)
This happens due to the fact that access to configuration.Client.parsers is not synchronized.
I looked pretty hard to see if this is supported and I'm pretty sure it isn't.
global
setenv PROXYNAME lb2
frontend www
bind :80
# Uses the overridden value, 'lb2'
http-request add-header Via "%[req.ver] %[env(PROXYNAME)]"
Hello,
I've found a bug that can cause a panic. I was able to trigger this by having Dataplane API sync map files while another external process was restarting HAProxy. If I was correct in my reading of the code the cause here is that client-native
is reading from a socket and the connection is lost when HAProxy restarts. Because the data is not validated as complete it is then returned and failed to be parsed correctly by other functions.
client-native
to read from a socket.panic: runtime error: index out of range [2] with length 2
goroutine 503 [running]:
github.com/haproxytech/client-native/v2/runtime.parseMapEntry(0xc005147871, 0x10, 0xc01, 0xc001a99f80)
/home/mjuraga/projects/go/pkg/mod/github.com/haproxytech/client-native/[email protected]/runtime/maps.go:171 +0x226
github.com/haproxytech/client-native/v2/runtime.ParseMapEntries(0xc005112001, 0x35880, 0x1, 0xc005112001, 0x35880, 0x0)
/home/mjuraga/projects/go/pkg/mod/github.com/haproxytech/client-native/[email protected]/runtime/maps.go:148 +0x10b
github.com/haproxytech/client-native/v2/runtime.(*SingleRuntime).ShowMapEntries(0xc000c4fe30, 0xc0010f8388, 0x3, 0xc0010f8388, 0x3, 0x0, 0x0, 0x47c762)
/home/mjuraga/projects/go/pkg/mod/github.com/haproxytech/client-native/[email protected]/runtime/maps.go:133 +0x1ba
github.com/haproxytech/client-native/v2/runtime.(*Client).ShowMapEntries(0xc002fc3d10, 0xc0010f8388, 0x3, 0x1, 0x1, 0xc0010f8388, 0x3, 0x24)
/home/mjuraga/projects/go/pkg/mod/github.com/haproxytech/client-native/[email protected]/runtime/runtime_client.go:493 +0x38d
github.com/haproxytech/dataplaneapi.syncMapFilesToRuntimeEntries(0xc002135920, 0xc001e96fa0)
/home/mjuraga/projects/haproxytech/release/dataplaneapi/configure_data_plane.go:806 +0x189
created by github.com/haproxytech/dataplaneapi.syncMaps
/home/mjuraga/projects/haproxytech/release/dataplaneapi/configure_data_plane.go:788 +0xbe
storage.Storage
struct creates SSL certificate files on disk with 0o644
permission.
Can this be made configurable by user as needed?
The specification appears to have a pair of transcription errors, in which L4OK (ell four oh kay) is replaced with L40K (ell four zero kay). This causes validation failures in generated clients, as the server code returns L4OK (ell four oh kay) in stats responses.
Instances:
client-native/specification/models/stats.yaml
Line 184 in 0e352d7
client-native/specification/models/stats.yaml
Line 291 in 0e352d7
When 2 backends have servers being generated with random names, it's very likely that they will have similar server names. For example:
backend aws-us-east-1-development-api-80
server SRV_Bd56t 172.31.31.151:80 check weight 128
server SRV_i2SMt 127.0.0.1:80 disabled weight 128
server SRV_YvSgD 127.0.0.1:80 disabled weight 128
backend aws-us-east-1-production-www-80
server SRV_Bd56t 172.31.31.174:80 check weight 128
server SRV_i2SMt 127.0.0.1:80 disabled weight 128
server SRV_5xAV0 127.0.0.1:80 disabled weight 128
I believe this is due to the rand.Seed() not being set here:
client-native/misc/stringutil.go
Lines 201 to 208 in 72f0246
According to documentation [1] and the dataplane spec [2] General Purpose Counter 0 field name is gpc0
. There is an extra _
in the expected field name in [3] that breaks parsing stick table entries.
Here is an example:
Exec output: 0x7f80783a1750: key=<redacted> use=0 exp=86395752 gpt0=0 gpc0=484 gpc0_rate(30000)=5 http_err_rate(300000)=59
Parsed entry: &{BytesInCnt:<nil> BytesInRate:<nil> BytesOutCnt:<nil> BytesOutRate:<nil> ConnCnt:<nil> ConnCur:<nil> ConnRate:<nil> Exp:0xc000482d88 Gpc0:0xc000482d90 Gpc0Rate:<nil> Gpc1:<nil> Gpc1Rate:<nil> HTTPErrCnt:<nil> HTTPErrRate:0xc000482d98 HTTPReqCnt:<nil> HTTPReqRate:<nil> ID:0x7f80783a1750 Key:<redacted> ServerID:<nil> SessCnt:<nil> SessRate:<nil> Use:false}
The same applies to gpc0_rate
, gpc1
, and gpc1_rate
.
The General Purpose Tag 0 gpt0
is not covered in the parser.
[1] https://cbonte.github.io/haproxy-dconv/2.0/configuration.html#4.2-stick-table%20type
[2] https://github.com/haproxytech/dataplaneapi-specification/blob/5d0b7ffcb38214efeb6c7ba03c310223083c7138/models/runtime.yaml#L272
[3]
client-native/runtime/stick_tables.go
Lines 133 to 134 in 51acd2a
for runtime single client, the timeout duration in executeRaw select should not be the same with unix dial timeout duration
think about the fake code below
package main
import (
"context"
"fmt"
"time"
)
const (
taskTimeout = 5 * time.Second
)
// TaskResponse ...
type TaskResponse struct {
result string
err error
}
// Task has command to execute on runtime api, and response channel for result
type Task struct {
command string
response chan TaskResponse
}
// SingleRuntime handles one runtime API
type SingleRuntime struct {
jobs chan Task
}
func (s *SingleRuntime) Init() error {
s.jobs = make(chan Task)
go s.handleIncommingJobs(context.Background())
return nil
}
func (s *SingleRuntime) handleIncommingJobs(ctx context.Context) {
for {
select {
case job, ok := <-s.jobs:
if !ok {
return
}
// just sleep to simulate dial timeout
time.Sleep(taskTimeout)
job.response <- TaskResponse{result: "result"}
case <-ctx.Done():
return
}
}
}
func (s *SingleRuntime) executeRaw(command string, retry int) (string, error) {
response := make(chan TaskResponse)
task := Task{
command: command,
response: response,
}
s.jobs <- task
select {
case rsp := <-response:
if rsp.err != nil && retry > 0 {
retry--
return s.executeRaw(command, retry)
}
return rsp.result, rsp.err
case <-time.After(taskTimeout):
return "", fmt.Errorf("timeout reached")
}
}
func main() {
client := &SingleRuntime{}
if err := client.Init(); err != nil {
fmt.Printf("init single runtime failed, err %s", err.Error())
}
result1, err := client.executeRaw("test command1", 1)
if err != nil {
fmt.Printf("executeRaw command1 failed, err %s\n", err.Error())
} else {
fmt.Printf("result is %s", result1)
}
result2, err := client.executeRaw("test command2", 1)
if err != nil {
fmt.Printf("executeRaw command2 failed, err %s\n", err.Error())
} else {
fmt.Printf("result is %s", result2)
}
}
go run main.go
get deadlock
`
executeRaw command1 failed, err timeout reached
fatal error: all goroutines are asleep - deadlock!
goroutine 1 [chan send]:
main.(*SingleRuntime).executeRaw(0xc00000e028, 0x4d0689, 0xd, 0x1, 0xc000072f58, 0x1, 0x1, 0x30)
/data/remote-vscode/tmp-bcs/test/test/main.go:58 +0x8f
main.main()
/data/remote-vscode/tmp-bcs/test/test/main.go:83 +0x173
goroutine 6 [chan send]:
main.(*SingleRuntime).handleIncommingJobs(0xc00000e028, 0x4ec7e0, 0xc0000160a0)
/data/remote-vscode/tmp-bcs/test/test/main.go:45 +0x90
created by main.(*SingleRuntime).Init
/data/remote-vscode/tmp-bcs/test/test/main.go:32 +0x8c
exit status 2
`
How can I interact with the runtime api over tcp?
I've created a fork https://github.com/theunknownport/client-native where I just override the net.DialTimeout Function
- if api, err = net.DialTimeout("unix", s.socketPath, taskTimeout); err != nil {
+ if api, err = net.DialTimeout("tcp", s.socketPath, taskTimeout); err != nil {
My haproxy config
global
stats socket [email protected]:9999 level admin
stats socket /var/run/haproxy-runtime-api.sock mode 666 level admin
stats timeout 2m
defaults
timeout connect 5s
timeout client 1m
timeout server 1m
listen stats
bind *:8080
mode http
stats enable
stats uri /stats
C:\Users\finnb\Downloads\socat-1.7.2.1>socat stdio tcp4-connect:127.0.0.1:9999
0 [main] socat 186416 find_fast_cwd: WARNING: Couldn't compute FAST_CWD pointer. Please report this problem to
the public mailing list [email protected]
show version
2.4.17-9f97155
My code to interact with the api
func ConnectHaProxy(ctx context.Context) (configuration.Configuration, error) {
confClient, _ := configuration.New(context.Background(),
cfg_opt.ConfigurationFile("/usr/local/etc/haproxy/haproxy.cfg"),
cfg_opt.UsePersistentTransactions,
cfg_opt.MasterWorker,
cfg_opt.UseMd5Hash,
)
// runtime
// import runtime_options "github.com/haproxytech/client-native/v4/runtime/options"
// or if not using master-worker
socketList := map[int]string{
1: "127.0.0.1:9999",
}
sockets := runtime_options.Sockets(socketList)
runtimeClient, _ := runtime_api.New(ctx, sockets)
// end runtime
// import "github.com/haproxytech/client-native/v4/options"
opt := []options.Option{
options.Configuration(confClient),
options.Runtime(runtimeClient),
}
// combine all together and create a client
client, err := client_native.New(ctx, opt...)
if err != nil {
log.Fatalf("Error initializing configuration client: %v", err)
}
c, err := client.Configuration()
if err != nil {
log.Fatalf("Error initializing configuration client: %v", err)
}
t, err := c.StartTransaction(1)
if err != nil {
log.Fatalf("Error initializing configuration client: %v", err)
}
c.IncrementVersion()
_, b, err := c.GetBackends(t.ID)
if err != nil {
log.Fatalf("Error initializing configuration client: %v", err)
}
fmt.Print(b)
// fetch configuration handler
return client.Configuration()
}
But when I run the code I always get option is not available
at client.Configuration()
.
Sorry if this issue seems annoying but I'm not so experience at coding in go yet, just started a month ago. I have my use case of interacting with the haproxy api via an external go program, maybe someone can help me with that.
Thanks in advance.
In this location, we use err.Error()
after a potential situation of err == nil
.
Line 177 in bee8caf
response, err := s.ExecuteWithResponse(cmd)
if err != nil {
return "", fmt.Errorf("%s %w", err.Error(), native_errors.ErrGeneral)
}
parts := strings.Split(response, ":")
if len(parts) < 3 {
return "", fmt.Errorf("%s %w", err.Error(), native_errors.ErrGeneral)
}
If there's no error in s.ExecuteWithResponse(cmd)
, err is nil
and then later if len(parts) < 3
, then there's a crash scenario because we're calling function .Error()
on a nil err
.
I'm observing that in my tests.
Environment:
HAproxy version: 2.8.0-01b97d-15
, this behavior is confirmed in 2.9-dev9-f1f8e2-108
(master HEAD, as of today) as well.
clientnative: v4.1.3
This is an older library, but I see a similar scenario possibility in HEAD, as shared above.
client-native socket is configured to work with MasterSocket
.
After some debugging, the response of "prepare acl" command is missing "first part, note the missing : in the beginning".
If I use runtimeoptions.Socket
instead of runtime.MasterSocket
, the response
is different, which doesn't trigger this condition.
It seems that set severity-output;
is ignored in MasterSocket
scenario.
➤ echo 'set severity-output number; prepare acl #18' | socat stdio /export/content/lid/apps/traffic-gateway/i001/haproxy.sock
[6]: New version created: 29
➤ echo '@1 set severity-output number;@1 prepare acl #18' | socat stdio /export/content/lid/apps/traffic-gateway/i001/master.sock
New version created: 30
Stacktrace when using runtimeoptions.MasterSocket
:-
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0xddce2f]
goroutine 21 [running]:
testing.tRunner.func1.2({0xf14300, 0x157afa0})
/home/arastogi/.gradle/language/golang/1.19/go/src/testing/testing.go:1396 +0x372
testing.tRunner.func1()
/home/arastogi/.gradle/language/golang/1.19/go/src/testing/testing.go:1399 +0x5f0
panic({0xf14300, 0x157afa0})
/home/arastogi/.gradle/language/golang/1.19/go/src/runtime/panic.go:890 +0x262
github.com/haproxytech/client-native/v4/runtime.(*SingleRuntime).PrepareACL(0x0?, {0xffcff0, 0x5d})
/basedir/src/github.com/haproxytech/client-native/v4/runtime/acls.go:177 +0x1cf
github.com/haproxytech/client-native/v4/runtime.(*client).AddACLAtomic(0xc0003fe8c0, {0xffcff0, 0x5d}, {0xc000400790, 0x2, 0x0?})
/basedir/src/github.com/haproxytech/client-native/v4/runtime/runtime_client.go:1125 +0x493
There's quite a lot of places where data.Index
is blindly dereferenced without checking if it's nil first.
configuration/acl.go
138: if err := p.Insert(section, parentName, "acl", SerializeACL(*data), int(*data.Index)); err != nil {
139: return c.HandleError(strconv.FormatInt(*data.Index, 10), parentType, parentName, t, transactionID == "", err)
configuration/log_target.go
160: if err := p.Insert(section, parentName, "log", SerializeLogTarget(*data), int(*data.Index)); err != nil {
161: return c.HandleError(strconv.FormatInt(*data.Index, 10), parentType, parentName, t, transactionID == "", err)
configuration/tcp_request_rule.go
152: if err := p.Insert(section, parentName, "tcp-request", s, int(*data.Index)); err != nil {
153: return c.HandleError(strconv.FormatInt(*data.Index, 10), parentType, parentName, t, transactionID == "", err)
configuration/tcp_check.go
152: if err := p.Insert(section, parentName, "tcp-check", s, int(*data.Index)); err != nil {
153: return c.HandleError(strconv.FormatInt(*data.Index, 10), parentType, parentName, t, transactionID == "", err)
...
Example of segfaulting code:
check := hamodels.TCPCheck{
Action: "connect",
}
config.CreateTCPCheck("backend", backend.Name, &check, t, 0)
It would be nice to add some validation that Index
is not nil.
https://github.com/haproxytech/client-native/blob/v2.0.2/configuration/configuration.go#L1485-L1502
in setFieldValue
field.Elem().Value()
is called to get the value of the frontends UniqueIdHeader
field. This field is type string, so calling Elem()
on it results in a panic.
panic: reflect: call of reflect.Value.Elem on string Value [recovered]
panic: reflect: call of reflect.Value.Elem on string Value
goroutine 9 [running]:
testing.tRunner.func1.1(0x184a180, 0xc000465220)
/usr/local/Cellar/go/1.14.2_1/libexec/src/testing/testing.go:940 +0x4dc
testing.tRunner.func1(0xc000165200)
/usr/local/Cellar/go/1.14.2_1/libexec/src/testing/testing.go:943 +0x317
panic(0x184a180, 0xc000465220)
/usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/panic.go:975 +0x41b
reflect.Value.Elem(0x1826260, 0xc000d5c138, 0x198, 0x0, 0x0, 0x0)
/usr/local/Cellar/go/1.14.2_1/libexec/src/reflect/value.go:820 +0x3b5
github.com/haproxytech/client-native/v2/configuration.setFieldValue(0x18ed2bb, 0x8, 0xc00015d241, 0x9, 0x1809847, 0xe, 0x1826260, 0xc000d5c138, 0x198, 0xc000ff2450, ...)
/Users/z003tt1/go/pkg/mod/github.com/haproxytech/client-native/[email protected]/configuration/configuration.go:1496 +0xdfde
github.com/haproxytech/client-native/v2/configuration.CreateEditSection(0x18d3580, 0xc000d5c000, 0x18ed2bb, 0x8, 0xc00015d241, 0x9, 0xc000ff2450, 0x0, 0x0)
/Users/z003tt1/go/pkg/mod/github.com/haproxytech/client-native/[email protected]/configuration/configuration.go:830 +0x2d7
github.com/haproxytech/client-native/v2/configuration.(*Client).createSection(0xc00012ca80, 0x18ed2bb, 0x8, 0xc00015d241, 0x9, 0x18d3580, 0xc000d5c000, 0xc0010372f0, 0x24, 0x0, ...)
/Users/z003tt1/go/pkg/mod/github.com/haproxytech/client-native/[email protected]/configuration/configuration.go:1705 +0x564
github.com/haproxytech/client-native/v2/configuration.(*Client).CreateFrontend(0xc00012ca80, 0xc000d5c000, 0xc0010372f0, 0x24, 0x0, 0x0, 0x0)
/Users/z003tt1/go/pkg/mod/github.com/haproxytech/client-native/[email protected]/configuration/frontend.go:117 +0x1a5
For the k8s ingress controller, I would need to support the proxy protocol, used when HAProxy is used as an external load-balancer.
This is basically supporting the "accept-proxy" keyword on the bind line.
The commit ea81ccc (which intended to add support for options http-no-delay
and http_proxy
) introduced three new model.Backend
attributes that clash with HTTPConnectionMode
.
These new attributes override whatever value HTTPConnectionMode
has previously set, even when they are undefined. Due to the order in which they were inserted into the Backend, sometimes the HTTPConnectionMode
is overridden and sometimes it overrides the others.
For example:
var b []models.Backend
// via attributes created by commit ea81ccc
b = append(b, models.Backend{Name: "1_http-keep-alive", HTTPKeepAlive: "enabled"})
b = append(b, models.Backend{Name: "2_http-server-close", HTTPServerClose: "enabled"})
b = append(b, models.Backend{Name: "3_httpclose", Httpclose: "enabled"})
// via HTTPConnectionMode
b = append(b, models.Backend{Name: "4_conmode_http-keep-alive", HTTPConnectionMode: "http-keep-alive"})
b = append(b, models.Backend{Name: "5_conmode_http-server-close", HTTPConnectionMode: "http-server-close"})
b = append(b, models.Backend{Name: "6_conmode_httpclose", HTTPConnectionMode: "httpclose"})
for i, backend := range b {
config.CreateBackend(&backend, "", int64(i+1))
}
The resulting config will be:
Backends defined using new attributes created by commit ea81ccc
backend 1_http-keep-alive
# This section should contain `option http-keep-alive`,
# but is blank because `HTTPConnectionMode` (which is undefined) overwrote
# the attribute `HTTPKeepAlive`
backend 2_http-server-close
# This section should contain `option http-server-close`,
# but is blank because `HTTPConnectionMode` (which is undefined) overwrote
# the attribute `HTTPServerClose`
backend 3_httpclose
# This section is ok. It wasn't overwritten because the attribute `Httpclose` is
# evaluated after `HTTPConnectionMode` on model.Backend
option httpclose
Backends defined using HTTPConnectionMode
attribute
backend 4_conmode_http-keep-alive
# This section is ok. It wasn't overwritten because `HTTPKeepAlive` is
# evaluated before `HTTPConnectionMode`
option http-keep-alive
backend 5_conmode_http-server-close
# This section is ok. It wasn't overwritten because `HTTPServerClose` is
# evaluated before `HTTPConnectionMode`
option http-server-close
backend 6_conmode_httpclose
# This section should contain `option httpclose`,
# but is blank because the `Httpclose `(which is undefined) overwrote
# the attribute `HTTPConnectionMode`.
Since these new attributes are mutually exclusive, perhaps it might be better to remove them and use only HTTPConnectionMode
.
client-native
treats address specification without square brackets (for example ::1:8888) as one IPv6 address without port:
client-native/configuration/server.go
Lines 196 to 205 in 62f8396
/* search for : or ] whatever comes first */
for (chr = end-1; chr > str2; chr--) {
if (*chr == ']' || *chr == ':')
break;
}
if (*chr == ':') {
/* Found a colon before a closing-bracket, must be a port separator.
* This guarantee backward compatibility.
*/
if (!(opts & PA_O_PORT_OK)) {
memprintf(err, "port specification not permitted here in '%s'", str);
goto out;
}
*chr++ = '\0';
port1 = chr;
}
So client can't extract port from notation like ::1:8888
but HAProxy can work with it.
Please, add support for these commands in runtime client:
new ssl cert
set ssl cert
commit ssl cert
abort ssl cert
show ssl cert
del ssl cert
add ssl crt-list
del ssl crt-list
show ssl crt-list
Hi,
I'd like to bring your attention to a minor problem with Site objects.
The SiteService
and SiteFarm
objects don't have a From
field which prevents users from specifying a named defaults section to be used when configuring Sites.
The
client-native/configuration/site.go
Lines 530 to 539 in fd6b9e5
and
client-native/configuration/site.go
Lines 541 to 548 in fd6b9e5
methods don't set the
client-native/models/frontend.go
Line 139 in fd6b9e5
client-native/models/backend.go
Line 143 in fd6b9e5
[WARNING] (1) : config : parsing [/etc/haproxy/haproxy.cfg:48] : defaults section 'defaults' (declared at /etc/haproxy/haproxy.cfg:22) is explicitly referenced by another proxy and implicitly used here. To avoid any ambiguity don't mix both usage. Add a last defaults section not explicitly used or always use explicit references.
This is not ideal because if we try to update a configuration file with an unnamed defaults
section, the config parser automatically renames it as unnamed_defaults_1
and updates all existing sections with from unnamed_defaults_1
. When we subsequently update the configuration file via dataplaneapi we get the above warning if we don't set the From
attribute for all frontend and backend sections we are trying to create, including the ones created through a Site object
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.