russellluo / kun Goto Github PK
View Code? Open in Web Editor NEWA communication toolkit for Go services.
Home Page: https://pkg.go.dev/github.com/RussellLuo/kun
License: MIT License
A communication toolkit for Go services.
Home Page: https://pkg.go.dev/github.com/RussellLuo/kun
License: MIT License
Original syntax (after #18):
//kok:param <argName> [<parameter> [; <parameter2> [; ...]]]
in=<in> name=<name> required=<required> type=<type> descr=<descr>
New syntax:
//kok:param <argName> [<parameter> [, <parameter2> [, ...]]]
in=<in> name=<name> required=<required> type=<type> descr=<descr>
//kok:param
directive, or by appending the binding to the end of the last //kok:param
directive in a semicolon-separated list.After the above syntax improvements, here are the equivalent annotations:
type Service interface {
//kok:op POST /users
//kok:param name
//kok:param age in=query
//kok:param ip in=header name=X-Forwarded-For, in=request name=RemoteAddr
CreateUser(ctx context.Context, name string, age int, ip net.IP) (err error)
}
// HTTP request:
// $ http POST /users?name=tracey&age=1
type Service interface {
//kok:op POST /users
//kok:param name; age in=query; ip in=header name=X-Forwarded-For, in=request name=RemoteAddr
CreateUser(ctx context.Context, name string, age int, ip net.IP) (err error)
}
// HTTP request:
// $ http POST /users?name=tracey&age=1
type Service interface {
//kok:op POST /users
//kok:param name; \
// age in=query; \
// ip in=header name=X-Forwarded-For, in=request name=RemoteAddr
CreateUser(ctx context.Context, name string, age int, ip net.IP) (err error)
}
// HTTP request:
// $ http POST /users?name=tracey&age=1
type User interface {
// @kok(op): POST /users
// @kok(param): operatorID < in:header,name:Authorization,required:true
Create(ctx context.Context, operatorID int) (err error)
// @kok(op): GET /users/{id}
// @kok(param): operatorID < in:header,name:Authorization,required:true
Read(ctx context.Context, id, operatorID int) (err error)
// @kok(op): PUT /users/{id}
// @kok(param): operatorID < in:header,name:Authorization,required:true
Update(ctx context.Context, id, operatorID int) (err error)
// @kok(op): DELETE /users/{id}
// @kok(param): operatorID < in:header,name:Authorization,required:true
Delete(ctx context.Context, id, operatorID int) (err error)
}
// @kok(alias): auth=`in:header,name:Authorization,required:true`
type User interface {
// @kok(op): POST /users
// @kok(param): operatorID < $auth
Create(ctx context.Context, operatorID int) (err error)
// @kok(op): GET /users/{id}
// @kok(param): operatorID < $auth
Read(ctx context.Context, id, operatorID int) (err error)
// @kok(op): PUT /users/{id}
// @kok(param): operatorID < $auth
Update(ctx context.Context, id, operatorID int) (err error)
// @kok(op): DELETE /users/{id}
// @kok(param): operatorID < $auth
Delete(ctx context.Context, id, operatorID int) (err error)
}
Or
// @kok(alias): opID=`operatorID < in:header,name:Authorization,required:true`
type User interface {
// @kok(op): POST /users
// @kok(param): $opID
Create(ctx context.Context, operatorID int) (err error)
// @kok(op): GET /users/{id}
// @kok(param): $opID
Read(ctx context.Context, id, operatorID int) (err error)
// @kok(op): PUT /users/{id}
// @kok(param): $opID
Update(ctx context.Context, id, operatorID int) (err error)
// @kok(op): DELETE /users/{id}
// @kok(param): $opID
Delete(ctx context.Context, id, operatorID int) (err error)
}
In some cultures, the pronunciation of kok
may lead to misunderstanding. (See comments here and here.)
The new name under consideration is kin
, which means "Kin Is Not just a kit of Go kit".
Other related names:
kingen
//kin:op
//kin:param
//kin:body
//kin:success
//kin:oas
//kin:alias
//kin:grpc
kin:"..."
Hi @RussellLuo thanks creating kun, great toolkit! i'm running into a (minor) issue with the generated returnErr when generating an http client.
As an example, suppose I have this service with a method that contains an interface (i.e. Animal)
package service
//go:generate kungen -force ./service.go Service
type Service interface {
//kun:op GET /animal/{id}
Find(id string) (Animal, error)
}
type Animal interface {
IsAnimal()
}
After generating the code http_client.go will contain the following:
req, err := http.NewRequest("GET", u.String(), nil)
if err != nil {
return Animal{}, err
}
returning Animal{}
results in an InvalidLiteral error.
If you're open to contributions I created a PR: #29
Feel free to give feedback, or let me know if you'd like to see tests for contributions:)
Problems in the current implementation:
Not sure but it looks like openapi v2 is what is generated ?
The generated code just says swagger v2 with indicating if it’s OpenAPI v3.1., v3.0 or v2
Since the underlying interface parser is borrowed from the old version of moq, kokgen
has the same legacy issues as moq:
In order to address these issues, we must improve the interface parser by leveraging the new version of moq (thanks to matryer/moq#141).
Since the core packages for interface parsing in moq are internal, we have to copy the code from moq instead.
Define the HTTP operation
Directive: //kok:op
Arguments: <method> <pattern>
Examples:
type Service interface {
//kok:op DELETE /users/{id}
DeleteUser(ctx context.Context, id int) (err error)
}
// HTTP request:
// $ http DELETE /users/101
Define the HTTP request parameters
//kok:param
<argName> [<parameter> [; <parameter2> [; ...]]]
in=<in> name=<name> required=<required> type=<type> descr=<descr>
Bind request parameters to simple arguments:
type Service interface {
//kok:op PUT /users/{id}
//kok:param name in=header name=X-User-Name
UpdateUser(ctx context.Context, id int, name string) (err error)
}
// HTTP request:
// $ http PUT /users/101 X-User-Name:tracey
Bind multiple request parameters to a struct according to tags:
type User struct {
ID int `kok:"in=path"` // name defaults to snake case `id`
Name string `kok:"in=query"` // name defaults to snake case `name`
Age int `kok:"in=header name=X-User-Age"`
}
type Service interface {
//kok:op PUT /users/{id}
//kok:param user
UpdateUser(ctx context.Context, user User) (err error)
}
// HTTP request:
// $ http PUT /users/101?name=tracey X-User-Age:1
Bind multiple query parameters to a struct with no tags:
type User struct {
Name string // equivalent to `kok:"in=query name=name"`
Age int // equivalent to `kok:"in=query name=age"`
Hobbies []string // equivalent to `kok:"in=query name=hobbies"`
}
type Service interface {
//kok:op POST /users
//kok:param user
CreateUser(ctx context.Context, user User) (err error)
}
// HTTP request:
// $ http POST /users?name=tracey&age=1&hobbies=music&hobbies=sport
Argument aggregation:
type Service interface {
//kok:op POST /logs
//kok:param ip in=header name=X-Forwarded-For; in=request name=RemoteAddr
Log(ctx context.Context, ip net.IP) (err error)
}
// The equivalent annotations =>
// (using backslash `\` for line continuation)
type Service interface {
//kok:op POST /logs
//kok:param ip in=header name=X-Forwarded-For; \
// in=request name=RemoteAddr
Log(ctx context.Context, ip net.IP) (err error)
}
// HTTP request:
// $ http POST /logs
Define the HTTP request body
//kok:body
<field>
or <manipulation> [; <manipulation2> [; ...]]
<argName> name=<name> type=<type> descr=<descr>
Omitted:
type Service interface {
//kok:op POST /users
CreateUser(ctx context.Context, name string, age int) (err error)
}
// HTTP request:
// $ http POST /users name=tracey age=1
Specified as a normal argument:
type User struct {
Name string `json:"name"`
Age int `json:"age"`
}
type Service interface {
//kok:op POST /users
//kok:body user
CreateUser(ctx context.Context, user User) (err error)
}
// HTTP request:
// $ http POST /users name=tracey age=1
Specified as -
:
type User struct {
Name string
Age int
Hobbies []string `kok:"name=hobby"`
}
type Service interface {
//kok:op POST /users
//kok:body -
CreateUser(ctx context.Context, user User) (err error)
}
// HTTP request:
// $ http POST /users?name=tracey&age=1&hobby=music&hobby=sport
Define the success HTTP response
Directive: //kok:success
Arguments: statusCode=<statusCode> body=<body> manip=`<manipulation> [; <manipulation2> [; ...]]`
<argName> name=<name> type=<type> descr=<descr>
Examples:
type User struct {
Name string `json:"name"`
Age int `json:"age"`
}
type Service interface {
//kok:op POST /users
//kok:success statusCode=201 body=user
CreateUser(ctx context.Context) (user User, err error)
}
Define the OAS metadata
Directive: //kok:oas
Arguments: <property>=<value>
Examples:
// This is the API documentation of User.
//kok:oas docsPath=/api-docs
//kok:oas title=User API
//kok:oas version=1.0.0
//kok:oas basePath=/v1
//kok:oas tags=user
type Service interface {
//kok:op POST /users
CreateUser(ctx context.Context, name string, age int) (err error)
}
Define the annotation alias
Directive: //kok:alias
Arguments: <name>=`<value>`
Examples:
type Service interface {
//kok:op POST /users
//kok:param operatorID in=header name=Authorization required=true
CreateUser(ctx context.Context, operatorID int) (err error)
//kok:op DELETE /users/{id}
//kok:param operatorID in=header name=Authorization required=true
DeleteUser(ctx context.Context, id, operatorID int) (err error)
}
// The equivalent annotations =>
//kok:alias opID=`operatorID in=header name=Authorization required=true`
type Service interface {
//kok:op POST /users
//kok:param $opID
CreateUser(ctx context.Context, operatorID int) (err error)
//kok:op DELETE /users/{id}
//kok:param $opID
DeleteUser(ctx context.Context, id, operatorID int) (err error)
}
//kok:grpc
request=<request> response=<response>
Omitted:
type Service interface {
//kok:grpc
CreateUser(ctx context.Context, name string, age int) (err error)
}
// gRPC request:
// $ grpcurl -d '{"name": "tracey", "age": 1}' ... pb.Service/CreateUser
Specified:
type User struct {
Name string `json:"name"`
Age int `json:"age"`
}
type Service interface {
//kok:grpc request=user
CreateUser(ctx context.Context, user User) (err error)
}
// gRPC request:
// $ grpcurl -d '{"name": "tracey", "age": 1}' ... pb.Service/CreateUser
Take helloworld as an example, this is the original construction code:
If we want to add request validation for the SayHello
operation, we can change the above code as follows:
validators := []httpoption.NamedValidator{
httpoption.Op("SayHello", httpoption.FuncValidator(func(value interface{}) error {
req := value.(*helloworld.SayHelloRequest)
if len(req.Name) > 10 {
return werror.Wrapf(gcode.ErrInvalidArgument, "name: length exceeds 10")
}
reVarName := regexp.MustCompile(`^\w+$`)
if !reVarName.MatchString(req.Name) {
return werror.Wrapf(gcode.ErrInvalidArgument, "name: invalid name format")
}
return nil
})),
}
r := helloworld.NewHTTPRouter(svc, httpcodec.NewDefaultCodecs(nil), httpoption.RequestValidators(validators...))
validators := []httpoption.NamedValidator{
httpoption.Op("SayHello", helloworld.ValidateSayHelloRequest(func(req *helloworld.SayHelloRequest) v.Schema {
return v.Schema{
v.F("name", &req.Name): v.All(
v.Len(0, 10).Msg("length exceeds 10"),
v.Match(regexp.MustCompile(`^\w+$`)).Msg("invalid name format"),
),
}
})),
}
r := helloworld.NewHTTPRouter(svc, httpcodec.NewDefaultCodecs(nil), httpoption.RequestValidators(validators...))
NOTE:
ValidateSayHelloRequest()
function is designed to be generated by kokgen
Add support for the following annotations:
type User struct {
ID int `kok:"path.id"`
Name string `kok:"query.name"`
Age int `kok:"header.X-User-Age"`
}
type Service interface {
// @kok(op): PUT /users/{id}
// @kok(param): user
UpdateUser(ctx context.Context, user User) (err error)
}
// HTTP request:
// $ http PUT /users/101?name=tracey X-User-Age:1
type User struct {
Name string
Age int
Hobbies []string
}
type Service interface {
// @kok(op): POST /users
// @kok(param): user
CreateUser(ctx context.Context, user User) (err error)
}
// HTTP request:
// $ http POST /users?Name=tracey&Age=1&Hobbies=music&Hobbies=sport
Sometimes, an HTTP request parameter is only used in the Transport/Endpoint layer, and no further argument binding (in the Service layer) is needed.
Taking the following GetUser
operation as an example:
type Service interface {
// @kok(op): GET /users/{id}
// @kok(success): body:user
GetUser(ctx context.Context, id int) (user User, err error)
}
In order to protect the operation, we may implement service-to-service authentication via the Authorization
header. Meanwhile, there is no corresponding argument in GetUser
to be mapped to.
So how to represent this Authorization
header in kok's HTTP annotation?
Add support for the blank identifier __
(double underscore, similar to Go's blank identifier) in @kok(param)
:
type Service interface {
// @kok(op): GET /users/{id}
// @kok(param): __ < in:header,name:Authorization,required:true
// @kok(success): body:user
GetUser(ctx context.Context, id int) (user User, err error)
}
And here is the possible codec.go
for the Authorization
header:
import (
"fmt"
"github.com/RussellLuo/kok/pkg/codec/httpcodec"
)
const (
authPrivateKey = "AUTH_PRIVATE_KEY"
)
// AuthCodec is used to encode and decode the `Authorization` header. It can be reused wherever needed.
type AuthCodec struct{}
func (c AuthCodec) Decode(in []string, out interface{}) error {
// NOTE: never use out, which is nil here.
if len(in) == 0 || in[0] != authPrivateKey {
return fmt.Errorf("authentication failed")
}
return nil
}
func (c AuthCodec) Encode(in interface{}) (out []string) {
// NOTE: never use in, which is nil here.
return []string{authPrivateKey}
}
func NewCodecs() *httpcodec.DefaultCodecs {
// Use AuthCodec to encode and decode the argument named "__", if exists,
// for the operation named "GetUser".
return httpcodec.NewDefaultCodecs(nil,
httpcodec.Op("GetUser", httpcodec.NewPatcher(httpcodec.JSON{}).Param("__", AuthCodec{})))
}
In complicated cases, where multiple blank identifiers are involved and need to be differentiated from one another (typically for applying different encoders and decoders), we can use meaningful names, each with a double underscore prefix __
(e.g. __auth
).
In summary, any name with a double underscore prefix __
is a valid blank identifier:
__
__auth
__other
kun
has already supported request validation since #10. However, there are some disadvantages:
kun
must support.Validation grammars from the current leading Go frameworks:
Other interesting references:
Take the helloworld service as an example, the validation schemas become (see previous schema definition):
// Service is used for saying hello.
type Service interface {
// SayHello says hello to the given name.
//
// @schema:
// name: len(0, 10).msg("bad length") && match(`^\w+$`)
//
//kun:op POST /messages
SayHello(ctx context.Context, name string) (message string, err error)
}
and this is the corresponding generated service middleware:
import (
v "github.com/RussellLuo/validating/v3"
)
func ValidatingMiddleware() func(next Service) Service {
return func(next Service) Service {
return &validatingMiddleware{
next: next,
}
}
}
type validatingMiddleware struct {
next Service
}
func (mw *validatingMiddleware) SayHello(ctx context.Context, name string) (string, error) {
schema := v.Schema{
v.F("name", name): v.All(
v.LenString(0, 10).Msg("bad length"),
v.Match(regexp.MustCompile(`^\w+$`)),
),
}
if err := v.Validate(schema); err != nil {
return "", werror.Wrap(gcode.ErrInvalidArgument, err)
}
return mw.next.SayHello(ctx, name)
}
Take the usersvc an example, the schema definition will be:
type User struct {
Name string
Age int
IP net.IP `kun:"in=header name=X-Forwarded-For, in=request name=RemoteAddr"`
}
func (u User) Schema() v.Schema {
return v.Schema{
v.F("name", u.Name): v.All(
v.LenString(0, 10),
v.Match(regexp.MustCompile(`^\w+$`)),
),
v.F("age", u.Age): v.Range(0, 100),
v.F("ip", user.IP): vext.IP(),
}
}
type Service interface {
// CreateUser creates a user with the given attributes.
//
//kun:op POST /users
//kun:param user
//kun:success body=result
CreateUser(ctx context.Context, user User) (result User, err error)
}
and this is the corresponding generated service middleware:
import (
v "github.com/RussellLuo/validating/v3"
)
func ValidatingMiddleware() func(next Service) Service {
return func(next Service) Service {
return &validatingMiddleware{
next: next,
}
}
}
type validatingMiddleware struct {
next Service
}
func (mw *validatingMiddleware) CreateUser(ctx context.Context, user User) (result User, err error) {
schema := v.Schema{
v.F("user", user): user.Schema(),
}
if err := v.Validate(schema); err != nil {
return User{}, werror.Wrap(gcode.ErrInvalidArgument, err)
}
return mw.next.CreateUser(ctx, user)
}
Sometimes, a Go method needs to correspond to more than one URI (or HTTP method). For example:
type Service interface {
//kun:op GET /messages/{messageID}
//kun:op GET /users/{userID}/messages/{messageID}
GetMessage(ctx context.Context, userID string, messageID string) (text string, err error)
}
This will enable the following two alternative HTTP to Go-method mappings:
HTTP | Go Method |
---|---|
GET /messages/123456 | GetMessage(ctx, "", "123456") |
GET /users/me/messages/123456 | GetMessage(ctx, "me", "123456") |
This feature is much like (and is partially inspired by) gRPC's additional_bindings
option (also see Multiple URI bindings).
i really like the approach this framework takes of using annotations.
i would like to try to add async communication generators .
i use NATS quite a bit . It’s a message bus and broker .
https://github.com/nats-io/nats.go
It has HTTP / web socket and tcp transports built in.
So for iot people use the tcp layer .
But for web they use the http and web sockets .
Nats is globally clustered , self load balancing and has build in security. It also has a clustered kv that replaced etc and consul.
So it remove the need for a lot of other layers typically needed for global internet scale systems engineering .
It’s 100% golang and runs on anything .
It can do RPC and PUB SUB also.
You can use protobufs, json , anything you want serialisation . It does not care .
I don’t know how familiar you are with NATS . But just let me know if this is something if interest.
i currently use it in iot, web, desktop and servers .
—-
there is a standard approach to async communications that is emerging also called “async api”.
It’s like openapi but for async.
it also has generators but like openapi takes the approach of you first writing your api description in json or yaml and then code generating from that .
it’s quite possible however to generate an async api document using kok !!!! This I think would be pretty cool because then others can generate the client in whatever language they want .
https://github.com/orgs/asyncapi/repositories?q=&type=&language=go&sort=
Take the IP decoding
mentioned in README.md as an example:
type Service interface {
// @kok(op): POST /logs
// @kok(param): ip < in:header,name:X-Forwarded-For
// @kok(param): ip < in:request,name:RemoteAddr
Log(ctx context.Context, ip net.IP) (err error)
}
// The equivalent annotations.
type Service interface {
// @kok(op): POST /logs
// @kok(param): ip < in:header,name:X-Forwarded-For
// @kok(param): < in:request,name:RemoteAddr
Log(ctx context.Context, ip net.IP) (err error)
}
// You must customize the decoding of `ip` later (conventionally in another file named `codec.go`).
// See examples in the `Encoding and decoding` section.
// HTTP request:
// $ http POST /logs
The existing solution is to implement a new codec:
// codec.go
import (
"fmt"
"net"
"strings"
"github.com/RussellLuo/kok/pkg/codec/httpcodec"
)
type Codec struct {
httpcodec.JSON
}
func (c Codec) DecodeRequestParams(name string, values map[string][]string, out interface{}) error {
switch name {
case "ip":
// We are decoding the "ip" argument.
remote := values["request.RemoteAddr"][0]
if fwdFor := values["header.X-Forwarded-For"][0]; fwdFor != "" {
remote = strings.TrimSpace(strings.Split(fwdFor, ",")[0])
}
ipStr, _, err := net.SplitHostPort(remote)
if err != nil {
ipStr = remote // OK; probably didn't have a port
}
ip := net.ParseIP(ipStr)
if ip == nil {
return fmt.Errorf("invalid client IP address: %s", ipStr)
}
outIP := out.(*net.IP)
*outIP = ip
return nil
default:
// Use the JSON codec for other arguments.
return c.JSON.DecodeRequestParams(name, values, out)
}
}
func NewCodecs() *httpcodec.DefaultCodecs {
return httpcodec.NewDefaultCodecs(Codec{})
}
While the above solution is feasible, the custom encoding and decoding behavior here is so common that we should provide:
Add support for patching existing codecs, which is shown as below:
// codec.go
import (
"fmt"
"net"
"strings"
"github.com/RussellLuo/kok/pkg/codec/httpcodec"
)
// IPCodec is used to encode and decode an IP. It can be reused wherever needed.
type IPCodec struct{}
func (c IPCodec) Decode(in map[string][]string, out interface{}) error {
remote := in["request.RemoteAddr"][0]
if fwdFor := in["header.X-Forwarded-For"][0]; fwdFor != "" {
remote = strings.TrimSpace(strings.Split(fwdFor, ",")[0])
}
ipStr, _, err := net.SplitHostPort(remote)
if err != nil {
ipStr = remote // OK; probably didn't have a port
}
ip := net.ParseIP(ipStr)
if ip == nil {
return fmt.Errorf("invalid client IP address: %s", ipStr)
}
outIP := out.(*net.IP)
*outIP = ip
return nil
}
func (c IPCodec) Encode(in interface{}) (out map[string][]string) {
return nil
}
func NewCodecs() *httpcodec.DefaultCodecs {
// Use IPCodec to encode and decode the argument named "ip", if exists,
// for the operation named "Log".
return httpcodec.NewDefaultCodecs(nil,
httpcodec.Op("Log", httpcodec.NewPatcher(httpcodec.JSON{}).Params("ip", IPCodec{})))
}
// Another way to create the codecs.
func NewCodecs2() *httpcodec.DefaultCodecs {
// Use IPCodec to encode and decode the argument named "ip", if exists,
// for all the operations.
return httpcodec.NewDefaultCodecs(nil).
PatchAll(func(c httpcodec.Codec) *httpcodec.Patcher {
return httpcodec.NewPatcher(c).Params("ip", IPCodec{})
})
}
Wrap an error and inherit the error message from another one:
// Lengthy
werror.Wrap(googlecode.ErrInvalidArgument).SetError(err)
// Simplified
werror.Wrap(gcode.ErrInvalidArgument, err)
Wrap an error and set a custom error message:
// Lengthy
werror.Wrap(googlecode.ErrInvalidArgument).SetErrorf("bad request")
// Simplified
werror.Wrapf(gcode.ErrInvalidArgument, "bad request")
.proto
file from the service definition in Go.proto
file to the gRPC definitionA declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.