russellluo / kun

A communication toolkit for Go services.

kun

kun is a communication toolkit for Go services. It concentrates on handling the communications between Go services, to free developers to focus on the business logic.

Ultimately, kun may support the following communication types:

• In-process function call
• RPC (e.g., HTTP and gRPC)
• Asynchronous messaging
• Cron Jobs

中文博客：Go 服务通信工具 Kun。

The Zen of kun

• Focus on the most valuable things

  Service communication is important, but is less important than the service itself (i.e., your business value).

  Furthermore, it should be effortless to change the communication types.

• Write code in Go whenever possible

  Prefer Go to other DSLs (e.g., OpenAPI, Protocol Buffers or AsyncAPI) for service definitions.

• Maintainability counts

  Embrace the spirit of the Clean Architecture for non-trivial applications.

Features

1. Code Generation Tool

   • HTTP
     • HTTP Server
     • HTTP Client
     • OAS2 Document
   • gRPC
     • Protocol Buffers
     • gRPC Server
     • gRPC Client
   • Event
     • Event Subscriber
     • Event Publisher
   • Cron
     • Cron Jobs

2. Useful Packages

   • appx: Application framework for HTTP and CRON applications (a wrapper of appx).
   • prometheus: Prometheus metrics utilities.
   • trace: A thin wrapper of x/net/trace for Go kit.
   • werror: Classified business errors.

How it works

HTTP Generation

gRPC Generation

Installation

$ go install github.com/RussellLuo/kun/cmd/kungen@latest

Usage

$ kungen -h
kungen [flags] source-file interface-name
  -flat
    	whether to use flat layout (default true)
  -fmt
    	whether to make code formatted (default true)
  -force
    	whether to remove previously generated files before generating new ones
  -out string
    	output directory (default ".")
  -snake
    	whether to use snake-case for default names (default true)
  -trace
    	whether to enable tracing

Quick Start

HTTP

NOTE: The following code is located in helloworld.

1. Define the interface

   type Service interface {
       SayHello(ctx context.Context, name string) (message string, err error)
   }

2. Implement the service

   type Greeter struct{}
   
   func (g *Greeter) SayHello(ctx context.Context, name string) (string, error) {
       return "Hello " + name, nil
   }

3. Add HTTP annotations

   type Service interface {
       //kun:op POST /messages
       SayHello(ctx context.Context, name string) (message string, err error)
   }

4. Generate the HTTP code

   $ cd examples/helloworld
   $ kungen ./service.go Service

5. Consume the service

   Run the HTTP server:

   $ go run cmd/main.go
   2020/09/15 18:06:22 transport=HTTP addr=:8080

   Consume by HTTPie:

   $ http POST :8080/messages name=Tracey
   HTTP/1.1 200 OK
   Content-Length: 27
   Content-Type: application/json; charset=utf-8
   Date: Tue, 15 Sep 2020 10:06:34 GMT
   
   {
       "message": "Hello Tracey"
   }

6. See the OAS documentation

   (Click to expand)

   $ http GET :8080/api
   HTTP/1.1 200 OK
   Content-Length: 848
   Content-Type: text/plain; charset=utf-8
   Date: Tue, 15 Sep 2020 10:08:24 GMT
   
   swagger: "2.0"
   info:
     title: "No Title"
     version: "0.0.0"
     description: "Service is used for saying hello."
     license:
       name: "MIT"
   host: "example.com"
   basePath: "/"
   schemes:
     - "https"
   consumes:
     - "application/json"
   produces:
     - "application/json"
   
   paths:
     /messages:
       post:
         description: "SayHello says hello to the given name."
         operationId: "SayHello"
         parameters:
           - name: body
             in: body
             schema:
               $ref: "#/definitions/SayHelloRequestBody"
   
         produces:
           - application/json; charset=utf-8
         responses:
           200:
             description: ""
             schema:
               $ref: "#/definitions/SayHelloResponse"
   
   
   definitions:
     SayHelloRequestBody:
       type: object
       properties:
         name:
           type: string
     SayHelloResponse:
       type: object
       properties:
         message:
           type: string

gRPC

NOTE: The following code is located in helloworldgrpc.

1. Define the interface

   type Service interface {
       SayHello(ctx context.Context, name string) (message string, err error)
   }

2. Implement the service

   type Greeter struct{}
   
   func (g *Greeter) SayHello(ctx context.Context, name string) (string, error) {
       return "Hello " + name, nil
   }

3. Add gRPC annotations

   type Service interface {
       //kun:grpc
       SayHello(ctx context.Context, name string) (message string, err error)
   }

4. Generate the gRPC code

   $ cd examples/helloworldgrpc
   $ kungen ./service.go Service

5. Consume the service

   Run the gRPC server:

   $ go run cmd/main.go
   2020/09/15 18:06:22 transport=HTTP addr=:8080

   Consume by grpcurl:

   $ grpcurl -plaintext -d '{"name": "Tracey"}' :8080 pb.Service/SayHello
   {
     "message": "Hello Tracey"
   }

See more examples here.

HTTP

Annotations

Define the HTTP request operation

Directive //kun:op

Syntax

//kun:op <method> <pattern>

If a Go method needs to correspond to more than one URI (or HTTP method), you can specify multiple //kun:op directives, which will produce multiple HTTP request operations.

Note that there are only three possible differences among these HTTP request operations:

• HTTP method
• URI
• Path parameters (defined in URI)

Arguments

• method: The request method.
• pattern: The request URI.
  • NOTE: All variables in pattern will automatically be bound to their corresponding method arguments (match by names in lower camel case), as path parameters, if these variables have not yet been specified explicitly by //kun:param.

Examples

• Single operation:

  type Service interface {
      //kun:op DELETE /users/{id}
      DeleteUser(ctx context.Context, id int) (err error)
  }
  
  // HTTP request:
  // $ http DELETE /users/101

• Multiple operations:

  type Service interface {
      //kun:op GET /messages/{messageID}
      //kun:op GET /users/{userID}/messages/{messageID}
      GetMessage(ctx context.Context, userID string, messageID string) (text string, err error)
  }
  
  // See a runnable example in examples/messaging.
  
  // HTTP request:
  // $ http GET /messages/123456
  // $ http GET /users/me/messages/123456

Define the HTTP request parameters

Directive //kun:param

Syntax

//kun:param <argName> [<parameter> [, <parameter2> [, ...]]]

If multiple method arguments are involved, you may need to apply multiple bindings. This can be done by adding a new //kun:param directive, or by appending the binding to the end of the last //kun:param directive in a semicolon-separated list.

Arguments

• argName: The name of the method argument.
  • Argument aggregation: By specifying multiple <parameter>s in a comma-separated list, multiple request parameters (each one is of basic type or repeated basic type) can be aggregated into one method argument (of any type).
  • Blank identifier: By specifying the argName with a double underscore prefix __, the corresponding request parameter(s) will not be mapped to any method argument. See here for more details.
• parameter: The definition of a single request parameter, to which the method argument will be mapped.
  • Syntax: in=<in> name=<name> required=<required> type=<type> descr=<descr>
  • Options:
    • in:
      • path: The request parameter is a path parameter.
        • Optional: All variables in pattern will automatically be bound to their corresponding method arguments (match by names in lower camel case), as path parameters.
      • query: The request parameter is a query parameter.
        • To receive values from a multi-valued query parameter, the method argument can be defined as a slice of basic type.
      • header: The request parameter is a header parameter.
        • To receive values from a multi-valued header parameter, the method argument can be defined as a slice of basic type.
      • cookie: The request parameter is a cookie parameter.
        • Not supported yet.
      • request: The request parameter is a property of Go's http.Request.
        • This is a special case, and only one property RemoteAddr is available now.
        • Note that parameters located in request have no relationship with OAS.
    • name: The name of the request parameter.
      • Optional: Defaults to argName (snake-case, or lower-camel-case if -snake=false) if not specified.
    • required: Determines whether this parameter is mandatory.
      • Optional: Defaults to false, if not specified.
      • If the parameter location is path, this property will be set to true internally, whether it's specified or not.
    • type: The OAS type of the request parameter.
      • Optional: Defaults to the type of the method argument, if not specified.
    • descr: The OAS description of the request parameter.
      • Optional: Defaults to "", if not specified.

Examples

• Bind request parameters to simple arguments:

  type Service interface {
      //kun:op PUT /users/{id}
      //kun:param name in=header name=X-User-Name
      UpdateUser(ctx context.Context, id int, name string) (err error)
  }
  
  // HTTP request:
  // $ http PUT /users/101 X-User-Name:tracey

• Bind multiple request parameters to a struct according to tags:

  type User struct {
      ID   int    `kun:"in=path"`  // name defaults to snake case `id`
      Name string `kun:"in=query"` // name defaults to snake case `name`
      Age  int    `kun:"in=header name=X-User-Age"`
  }
  
  type Service interface {
      //kun:op PUT /users/{id}
      //kun:param user
      UpdateUser(ctx context.Context, user User) (err error)
  }
  
  // HTTP request:
  // $ http PUT /users/101?name=tracey X-User-Age:1

• Bind multiple query parameters to a struct with no tags:

  type User struct {
      Name    string   // equivalent to `kun:"in=query name=name"`
      Age     int      // equivalent to `kun:"in=query name=age"`
      Hobbies []string // equivalent to `kun:"in=query name=hobbies"`
  }
  
  type Service interface {
      //kun:op POST /users
      //kun:param user
      CreateUser(ctx context.Context, user User) (err error)
  }
  
  // HTTP request:
  // $ http POST /users?name=tracey&age=1&hobbies=music&hobbies=sport

• Argument aggregation:

  type Service interface {
      //kun:op POST /logs
      //kun:param ip in=header name=X-Forwarded-For, in=request name=RemoteAddr
      Log(ctx context.Context, ip net.IP) (err error)
  }
  
  // The equivalent annotations =>
  // (using backslash-continued annotations)
  type Service interface {
      //kun:op POST /logs
      //kun:param ip in=header name=X-Forwarded-For, \
      //             in=request name=RemoteAddr
      Log(ctx context.Context, ip net.IP) (err error)
  }
  
  // You must customize the decoding of `ip` later (conventionally in another file named `codec.go`).
  // See a runnable example in examples/usersvc.
  
  // HTTP request:
  // $ http POST /logs

• Multiple bindings in a single //kun:param:

  type Service interface {
      //kun:op POST /users
      //kun:param name; age; ip in=header name=X-Forwarded-For, in=request name=RemoteAddr
      CreateUser(ctx context.Context, name string, age int, ip net.IP) (err error)
  }
  
  // The equivalent annotations =>
  // (using backslash-continued annotations)
  
  type Service interface {
      //kun:op POST /users
      //kun:param name; \
      //          age; \
      //          ip in=header name=X-Forwarded-For, in=request name=RemoteAddr
      CreateUser(ctx context.Context, name string, age int, ip net.IP) (err error)
  }
  
  // HTTP request:
  // $ http POST /users?name=tracey&age=1

Define the HTTP request body

Directive //kun:body

Syntax

//kun:body <field>

or

//kun:body <manipulation> [; <manipulation2> [; ...]]

Arguments

• field: The name of the method argument whose value is mapped to the HTTP request body.
  • Optional: When omitted, a struct containing all the arguments (except context.Context), which are not located in path/query/header, will automatically be mapped to the HTTP request body.
  • The special name - can be used, to define that there is no HTTP request body. As a result, every argument, which is not located in path/query/header, will automatically be mapped to one or more query parameters.
• manipulation:
  • Syntax: <argName> name=<name> type=<type> descr=<descr> required=<required>
  • Options:
    • argName: The name of the method argument to be manipulated.
    • name: The name of the request parameter.
      • Optional: Defaults to argName (snake-case, or lower-camel-case if -snake=false) if not specified.
    • type: The OAS type of the request parameter.
      • Optional: Defaults to the type of the method argument, if not specified.
    • descr: The OAS description of the request parameter.
      • Optional: Defaults to "", if not specified.
    • required: Determines whether this parameter is mandatory.
      • Optional: Defaults to false, if not specified.

Examples

• Omitted:

  type Service interface {
      //kun:op POST /users
      CreateUser(ctx context.Context, name string, age int) (err error)
  }
  
  // HTTP request:
  // $ http POST /users name=tracey age=1

• Specified as a normal argument:

  type User struct {
      Name string `json:"name"`
      Age  int    `json:"age"`
  }
  
  type Service interface {
      //kun:op POST /users
      //kun:body user
      CreateUser(ctx context.Context, user User) (err error)
  }
  
  // HTTP request:
  // $ http POST /users name=tracey age=1

• Specified as -:

  type User struct {
      Name    string
      Age     int
      Hobbies []string `kun:"name=hobby"`
  }
  
  type Service interface {
      //kun:op POST /users
      //kun:body -
      CreateUser(ctx context.Context, user User) (err error)
  }
  
  // HTTP request:
  // $ http POST /users?name=tracey&age=1&hobby=music&hobby=sport

• Manipulation:

  type Service interface {
      //kun:op POST /users
      //kun:body age name=user_age type=string descr='The user age'
      CreateUser(ctx context.Context, name string, age int) (err error)
  }
  
  // HTTP request:
  // $ http POST /users name=tracey user_age=1

Define the success HTTP response

Directive //kun:success

Syntax

//kun:success statusCode=<statusCode> body=<body> manip=`<manipulation> [; <manipulation2> [; ...]]`

Arguments

• statusCode: The status code of the success HTTP response.
  • Optional: Defaults to 200, if not specified.
• body: The name of the response field whose value is mapped to the HTTP response body.
  • Optional: When omitted, a struct containing all the results (except error) will automatically be mapped to the HTTP response body.
• manipulation:
  • Syntax: <argName> name=<name> type=<type> descr=<descr>
  • Not supported yet.

Examples

type User struct {
    Name string `json:"name"`
    Age  int    `json:"age"`
}

type Service interface {
    //kun:op POST /users
    //kun:success statusCode=201 body=user
    CreateUser(ctx context.Context) (user User, err error)
}

Define the OAS metadata

Directive //kun:oas

Syntax

//kun:oas <property>=<value>

Arguments

• property: The property to set. Supported properties:
  • docsPath: The URL path to the OAS documentation itself.
    • Optional: Defaults to "/api" if not specified.
  • title: The title field of Info Object, see Basic Structure.
    • Optional: Defaults to "No Title" if not specified.
  • version: The version field of Info Object, see Basic Structure.
    • Optional: Defaults to "0.0.0" if not specified.
  • description: The description field of Info Object, see Basic Structure.
    • Unavailable: Automatically extracted from the Go documentation of the interface definition.
  • basePath: The basePath property, see API Host and Base URL.
  • tags: A list of tags (comma-separated), see Grouping Operations With Tags.
• value: The value of the property.

Examples

// This is the API documentation of User.
//kun:oas docsPath=/api-docs
//kun:oas title=User-API
//kun:oas version=1.0.0
//kun:oas basePath=/v1
//kun:oas tags=user
type Service interface {
    //kun:op POST /users
    CreateUser(ctx context.Context, name string, age int) (err error)
}

Define the annotation alias

Directive //kun:alias

Syntax

//kun:alias <name>=`<value>`

Arguments

• name: The name of the alias.
• value: The string value that the alias represents.

Examples

type Service interface {
    //kun:op POST /users
    //kun:param operatorID in=header name=Authorization required=true
    CreateUser(ctx context.Context, operatorID int) (err error)

    //kun:op DELETE /users/{id}
    //kun:param operatorID in=header name=Authorization required=true
    DeleteUser(ctx context.Context, id, operatorID int) (err error)
}

// The equivalent annotations =>

//kun:alias opID=`operatorID in=header name=Authorization required=true`
type Service interface {
    //kun:op POST /users
    //kun:param $opID
    CreateUser(ctx context.Context, operatorID int) (err error)

    //kun:op DELETE /users/{id}
    //kun:param $opID
    DeleteUser(ctx context.Context, id, operatorID int) (err error)
}

Encoding and decoding

See the HTTP Codec interface.

Also see here for examples.

OAS Schema

See the OAS Schema interface.

gRPC

Annotations

Directive //kun:grpc

Syntax

//kun:grpc request=<request> response=<response>

Arguments

• request: The name of the method argument, whose value will be mapped to the gRPC request.
  • Optional: When omitted, a struct containing all the arguments (except context.Context) will automatically be mapped to the gRPC request.
• response: The name of the method result, whose value will be mapped to the gRPC response.
  • Optional: When omitted, a struct containing all the results (except error) will automatically be mapped to the gRPC response.

Examples

• Omitted:

  type Service interface {
      //kun:grpc
      CreateUser(ctx context.Context, name string, age int) (err error)
  }
  
  // gRPC request:
  // $ grpcurl -d '{"name": "tracey", "age": 1}' ... pb.Service/CreateUser

• Specified:

  type User struct {
      Name string `json:"name"`
      Age  int    `json:"age"`
  }
  
  type Service interface {
      //kun:grpc request=user
      CreateUser(ctx context.Context, user User) (err error)
  }
  
  // gRPC request:
  // $ grpcurl -d '{"name": "tracey", "age": 1}' ... pb.Service/CreateUser

Event

Annotations

Directive //kun:event

Syntax

//kun:event type=<type> data=<data>

Arguments

• type: The type of the event.
  • Optional: Defaults to the name of the corresponding method (snake-case, or lower-camel-case if -snake=false) if not specified.
• data: The name of the method argument whose value is mapped to the event data.
  • Optional: When omitted, a struct containing all the arguments (except context.Context) will automatically be mapped to the event data.

Examples

• Omitted:

  type Service interface {
      //kun:event
      EventCreated(ctx context.Context, id int) (err error)
  }
  
  // event: {"type": "event_created", "data": `{"id": 1}`}

• Specified:

  type Data struct {
      ID int `json:"id"`
  }
  
  type Service interface {
      //kun:event type=created data=data
      EventCreated(ctx context.Context, data Data) (err error)
  }
  
  // event: {"type": "created", "data": `{"id": 1}`}

Cron

Annotations

Directive //kun:cron

Syntax

//kun:cron name=<name> expr=<expr>

Arguments

• name: The job name.
  • Optional: Defaults to the name of the corresponding method (snake-case, or lower-camel-case if -snake=false) if not specified.
• expr: The cron expression.
  • Required: Three formats are supported.

Examples

• Name omitted:

  type Service interface {
      //kun:cron expr='@every 5s'
      SendEmail(ctx context.Context) error
  }
  
  // job: {"name": "send_email", "expr": "@every 5s"}

• Name specified:

  type Service interface {
      //kun:cron name=send expr='@every 5s'
      SendEmail(ctx context.Context) error
  }
  
  // job: {"name": "send", "expr": "@every 5s"}

Documentation

Checkout the Godoc.

License

MIT

kun's Issues

Add support for the business level input validation

Problem

kun has already supported request validation since #10. However, there are some disadvantages:

• The request validation is done at the transport layer, while the request fields to validate are actually designed at the service layer
• Input validation of pure services (i.e. without the transport layer) are not supported, which is a critical problem since "In-process function call" is the communication type kun must support.
• Validation schemas are not visible from the documentation (i.e. Godoc for pure services, OAS for HTTP services, Protobuf for gRPC services, etc)

Validation Grammar

Validation grammars from the current leading Go frameworks:

• Gin: uses go-playground/validator/v10 (see example)
• go-zero: has very limited built-in support (see the "tag modifier" part) and recommends using third party validator
• gRPC-Gateway: uses mwitkow/go-proto-validators (see doc)
• Kratos: uses bufbuild/protoc-gen-validate (see doc)
• Micro: has no support?
• Go kit: uses a custom middleware (see discussion)

Other interesting references:

• CUE
• CEL-Go

Proposed Solution

• Define validation schemas in the comments of interface methods.
• Generate the validation code, in the form of a service middleware (i.e. at the service layer), per the schema definitions.

Example

helloworld

Take the helloworld service as an example, the validation schemas become (see previous schema definition):

// Service is used for saying hello.
type Service interface {
	// SayHello says hello to the given name.
	//
	// @schema:
	//   name: len(0, 10).msg("bad length") && match(`^\w+$`)
	//
	//kun:op POST /messages
	SayHello(ctx context.Context, name string) (message string, err error)
}

and this is the corresponding generated service middleware:

import (
	v "github.com/RussellLuo/validating/v3"
)

func ValidatingMiddleware() func(next Service) Service {
	return func(next Service) Service {
		return &validatingMiddleware{
			next: next,
		}
	}
}

type validatingMiddleware struct {
	next Service
}

func (mw *validatingMiddleware) SayHello(ctx context.Context, name string) (string, error) {
	schema := v.Schema{
		v.F("name", name): v.All(
			v.LenString(0, 10).Msg("bad length"),
			v.Match(regexp.MustCompile(`^\w+$`)),
		),
	}
	if err := v.Validate(schema); err != nil {
		return "", werror.Wrap(gcode.ErrInvalidArgument, err)
	}

	return mw.next.SayHello(ctx, name)
}

usersvc

Take the usersvc an example, the schema definition will be:

type User struct {
	Name string
	Age  int
	IP   net.IP `kun:"in=header name=X-Forwarded-For, in=request name=RemoteAddr"`
}

func (u User) Schema() v.Schema {
	return v.Schema{
		v.F("name", u.Name): v.All(
			v.LenString(0, 10),
			v.Match(regexp.MustCompile(`^\w+$`)),
		),
		v.F("age", u.Age): v.Range(0, 100),
		v.F("ip", user.IP): vext.IP(),
	}
}

type Service interface {
	// CreateUser creates a user with the given attributes.
	//
	//kun:op POST /users
	//kun:param user
	//kun:success body=result
	CreateUser(ctx context.Context, user User) (result User, err error)
}

and this is the corresponding generated service middleware:

import (
	v "github.com/RussellLuo/validating/v3"
)

func ValidatingMiddleware() func(next Service) Service {
	return func(next Service) Service {
		return &validatingMiddleware{
			next: next,
		}
	}
}

type validatingMiddleware struct {
	next Service
}

func (mw *validatingMiddleware) CreateUser(ctx context.Context, user User) (result User, err error) {
	schema := v.Schema{
		v.F("user", user): user.Schema(),
	}
	if err := v.Validate(schema); err != nil {
		return User{}, werror.Wrap(gcode.ErrInvalidArgument, err)
	}

	return mw.next.CreateUser(ctx, user)
}

Refactor the HTTP Codec

Problems in the current implementation:

1. ParamCodec.Deoce: Comma-separated query or header strings will be truncated.
   • test-case: TODO
   • solution: TODO
2. Codec.DecodeRequestBody: the method signature can not satisfy the requirements for file uploading.
   • test-case: TODO
   • solution: TODO

Add support for using blank identifiers in `@kok(param)`

Problem

Sometimes, an HTTP request parameter is only used in the Transport/Endpoint layer, and no further argument binding (in the Service layer) is needed.

Taking the following GetUser operation as an example:

type Service interface {
	// @kok(op): GET /users/{id}
	// @kok(success): body:user
	GetUser(ctx context.Context, id int) (user User, err error)
}

In order to protect the operation, we may implement service-to-service authentication via the Authorization header. Meanwhile, there is no corresponding argument in GetUser to be mapped to.

So how to represent this Authorization header in kok's HTTP annotation?

Proposed Solution

Add support for the blank identifier __ (double underscore, similar to Go's blank identifier) in @kok(param):

type Service interface {
	// @kok(op): GET /users/{id}
	// @kok(param): __ < in:header,name:Authorization,required:true
	// @kok(success): body:user
	GetUser(ctx context.Context, id int) (user User, err error)
}

And here is the possible codec.go for the Authorization header:

import (
	"fmt"

	"github.com/RussellLuo/kok/pkg/codec/httpcodec"
)

const (
	authPrivateKey = "AUTH_PRIVATE_KEY"
)

// AuthCodec is used to encode and decode the `Authorization` header. It can be reused wherever needed.
type AuthCodec struct{}

func (c AuthCodec) Decode(in []string, out interface{}) error {
	// NOTE: never use out, which is nil here.

	if len(in) == 0 || in[0] != authPrivateKey {
		return fmt.Errorf("authentication failed")
	}
	return nil
}

func (c AuthCodec) Encode(in interface{}) (out []string) {
	// NOTE: never use in, which is nil here.

	return []string{authPrivateKey}
}

func NewCodecs() *httpcodec.DefaultCodecs {
	// Use AuthCodec to encode and decode the argument named "__", if exists,
	// for the operation named "GetUser".
	return httpcodec.NewDefaultCodecs(nil,
		httpcodec.Op("GetUser", httpcodec.NewPatcher(httpcodec.JSON{}).Param("__", AuthCodec{})))
}

Multiple blank identifiers

In complicated cases, where multiple blank identifiers are involved and need to be differentiated from one another (typically for applying different encoders and decoders), we can use meaningful names, each with a double underscore prefix __ (e.g. __auth).

In summary, any name with a double underscore prefix __ is a valid blank identifier:

• __
• __auth
• __other

OpenAPI v3.1., v3.0

Not sure but it looks like openapi v2 is what is generated ?

The generated code just says swagger v2 with indicating if it’s OpenAPI v3.1., v3.0 or v2

Add support for `@kok(alias)`

Lengthy

type User interface {
    // @kok(op): POST /users
    // @kok(param): operatorID < in:header,name:Authorization,required:true
    Create(ctx context.Context, operatorID int) (err error)

    // @kok(op): GET /users/{id}
    // @kok(param): operatorID < in:header,name:Authorization,required:true
    Read(ctx context.Context, id, operatorID int) (err error)

    // @kok(op): PUT /users/{id}
    // @kok(param): operatorID < in:header,name:Authorization,required:true
    Update(ctx context.Context, id, operatorID int) (err error)

    // @kok(op): DELETE /users/{id}
    // @kok(param): operatorID < in:header,name:Authorization,required:true
    Delete(ctx context.Context, id, operatorID int) (err error)
}

Simplified

// @kok(alias): auth=`in:header,name:Authorization,required:true`
type User interface {
    // @kok(op): POST /users
    // @kok(param): operatorID < $auth
    Create(ctx context.Context, operatorID int) (err error)

    // @kok(op): GET /users/{id}
    // @kok(param): operatorID < $auth
    Read(ctx context.Context, id, operatorID int) (err error)

    // @kok(op): PUT /users/{id}
    // @kok(param): operatorID < $auth
    Update(ctx context.Context, id, operatorID int) (err error)

    // @kok(op): DELETE /users/{id}
    // @kok(param): operatorID < $auth
    Delete(ctx context.Context, id, operatorID int) (err error)
}

Or

// @kok(alias): opID=`operatorID < in:header,name:Authorization,required:true`
type User interface {
    // @kok(op): POST /users
    // @kok(param): $opID
    Create(ctx context.Context, operatorID int) (err error)

    // @kok(op): GET /users/{id}
    // @kok(param): $opID
    Read(ctx context.Context, id, operatorID int) (err error)

    // @kok(op): PUT /users/{id}
    // @kok(param): $opID
    Update(ctx context.Context, id, operatorID int) (err error)

    // @kok(op): DELETE /users/{id}
    // @kok(param): $opID
    Delete(ctx context.Context, id, operatorID int) (err error)
}

Invalid literal error when generating code for an Interface

Hi @RussellLuo thanks creating kun, great toolkit! i'm running into a (minor) issue with the generated returnErr when generating an http client.

As an example, suppose I have this service with a method that contains an interface (i.e. Animal)

package service 

//go:generate kungen -force ./service.go Service
type Service interface {
	//kun:op GET /animal/{id}
	Find(id string) (Animal, error)
}

type Animal interface {
	IsAnimal()
}

After generating the code http_client.go will contain the following:

req, err := http.NewRequest("GET", u.String(), nil)
if err != nil {
	return Animal{}, err
}

returning Animal{} results in an InvalidLiteral error.

If you're open to contributions I created a PR: #29
Feel free to give feedback, or let me know if you'd like to see tests for contributions:)

Add support for patching existing codecs

Motivation

Take the IP decoding mentioned in README.md as an example:

type Service interface {
        // @kok(op): POST /logs
        // @kok(param): ip < in:header,name:X-Forwarded-For
        // @kok(param): ip < in:request,name:RemoteAddr
        Log(ctx context.Context, ip net.IP) (err error)
}

// The equivalent annotations.
type Service interface {
        // @kok(op): POST /logs
        // @kok(param): ip < in:header,name:X-Forwarded-For
        // @kok(param):    < in:request,name:RemoteAddr
        Log(ctx context.Context, ip net.IP) (err error)
}

// You must customize the decoding of `ip` later (conventionally in another file named `codec.go`).
// See examples in the `Encoding and decoding` section.

// HTTP request:
// $ http POST /logs

The existing solution is to implement a new codec:

// codec.go

import (
	"fmt"
	"net"
	"strings"

	"github.com/RussellLuo/kok/pkg/codec/httpcodec"
)

type Codec struct {
	httpcodec.JSON
}

func (c Codec) DecodeRequestParams(name string, values map[string][]string, out interface{}) error {
	switch name {
	case "ip":
		// We are decoding the "ip" argument.

		remote := values["request.RemoteAddr"][0]
		if fwdFor := values["header.X-Forwarded-For"][0]; fwdFor != "" {
			remote = strings.TrimSpace(strings.Split(fwdFor, ",")[0])
		}

		ipStr, _, err := net.SplitHostPort(remote)
		if err != nil {
			ipStr = remote // OK; probably didn't have a port
		}

		ip := net.ParseIP(ipStr)
		if ip == nil {
			return fmt.Errorf("invalid client IP address: %s", ipStr)
		}

		outIP := out.(*net.IP)
		*outIP = ip
		return nil

	default:
		// Use the JSON codec for other arguments.
		return c.JSON.DecodeRequestParams(name, values, out)
	}
}

func NewCodecs() *httpcodec.DefaultCodecs {
	return httpcodec.NewDefaultCodecs(Codec{})
}

While the above solution is feasible, the custom encoding and decoding behavior here is so common that we should provide:

• an easier way to customize codecs for request parameters
• and better code reusability for custom codecs

Proposed Solution

Add support for patching existing codecs, which is shown as below:

// codec.go

import (
	"fmt"
	"net"
	"strings"

	"github.com/RussellLuo/kok/pkg/codec/httpcodec"
)

// IPCodec is used to encode and decode an IP. It can be reused wherever needed.
type IPCodec struct{}

func (c IPCodec) Decode(in map[string][]string, out interface{}) error {
	remote := in["request.RemoteAddr"][0]
	if fwdFor := in["header.X-Forwarded-For"][0]; fwdFor != "" {
		remote = strings.TrimSpace(strings.Split(fwdFor, ",")[0])
	}

	ipStr, _, err := net.SplitHostPort(remote)
	if err != nil {
		ipStr = remote // OK; probably didn't have a port
	}

	ip := net.ParseIP(ipStr)
	if ip == nil {
		return fmt.Errorf("invalid client IP address: %s", ipStr)
	}

	outIP := out.(*net.IP)
	*outIP = ip
	return nil
}

func (c IPCodec) Encode(in interface{}) (out map[string][]string) {
	return nil
}

func NewCodecs() *httpcodec.DefaultCodecs {
	// Use IPCodec to encode and decode the argument named "ip", if exists,
	// for the operation named "Log".
	return httpcodec.NewDefaultCodecs(nil,
		httpcodec.Op("Log", httpcodec.NewPatcher(httpcodec.JSON{}).Params("ip", IPCodec{})))
}

// Another way to create the codecs.
func NewCodecs2() *httpcodec.DefaultCodecs {
	// Use IPCodec to encode and decode the argument named "ip", if exists,
	// for all the operations.
	return httpcodec.NewDefaultCodecs(nil).
		PatchAll(func(c httpcodec.Codec) *httpcodec.Patcher {
			return httpcodec.NewPatcher(c).Params("ip", IPCodec{})
		})
}

Async communication

i really like the approach this framework takes of using annotations.

i would like to try to add async communication generators .

i use NATS quite a bit . It’s a message bus and broker .

https://github.com/nats-io/nats.go

It has HTTP / web socket and tcp transports built in.

So for iot people use the tcp layer .
But for web they use the http and web sockets .

Nats is globally clustered , self load balancing and has build in security. It also has a clustered kv that replaced etc and consul.
So it remove the need for a lot of other layers typically needed for global internet scale systems engineering .

It’s 100% golang and runs on anything .

It can do RPC and PUB SUB also.

You can use protobufs, json , anything you want serialisation . It does not care .

I don’t know how familiar you are with NATS . But just let me know if this is something if interest.

i currently use it in iot, web, desktop and servers .

—-

there is a standard approach to async communications that is emerging also called “async api”.

https://www.asyncapi.com/

It’s like openapi but for async.

it also has generators but like openapi takes the approach of you first writing your api description in json or yaml and then code generating from that .

it’s quite possible however to generate an async api document using kok !!!! This I think would be pretty cool because then others can generate the client in whatever language they want .

https://github.com/orgs/asyncapi/repositories?q=&type=&language=go&sort=

Simplify the annotation syntax for kok(param)

Add support for the following annotations:

1. Bind multiple parameters from different locations to a struct according to tags

type User struct {
	ID   int    `kok:"path.id"`
	Name string `kok:"query.name"`
	Age  int    `kok:"header.X-User-Age"`
}

type Service interface {
	// @kok(op): PUT /users/{id}
	// @kok(param): user
	UpdateUser(ctx context.Context, user User) (err error)
}

// HTTP request:
// $ http PUT /users/101?name=tracey X-User-Age:1

2. Bind multiple query parameters to a struct with no tags

type User struct {
	Name    string
	Age     int
	Hobbies []string
}

type Service interface {
	// @kok(op): POST /users
	// @kok(param): user
	CreateUser(ctx context.Context, user User) (err error)
}

// HTTP request:
// $ http POST /users?Name=tracey&Age=1&Hobbies=music&Hobbies=sport

Add support for the gRPC transport

• Generate the .proto file from the service definition in Go
• Compile the .proto file to the gRPC definition
• Adapt the gRPC definition to Go kit

Improve the interface parser

Since the underlying interface parser is borrowed from the old version of moq, kokgen has the same legacy issues as moq:

• matryer/moq#68
• matryer/moq#94

In order to address these issues, we must improve the interface parser by leveraging the new version of moq (thanks to matryer/moq#141).

Since the core packages for interface parsing in moq are internal, we have to copy the code from moq instead.

Rename this project

Motivation

In some cultures, the pronunciation of kok may lead to misunderstanding. (See comments here and here.)

New name

The new name under consideration is kin, which means "Kin Is Not just a kit of Go kit".

Other related names:

• Module path
  • github.com/RussellLuo/kin
• Code generation tool
  • kingen
• Annotation directives
  • //kin:op
  • //kin:param
  • //kin:body
  • //kin:success
  • //kin:oas
  • //kin:alias
  • //kin:grpc
• Struct tags
  • kin:"..."

Tasks

• Rename the repo
• Update the code
• Update the docs

Add support for request validation

Take helloworld as an example, this is the original construction code:

https://github.com/RussellLuo/kok/blob/1a22aefc486c8250d788a4d36012af5b391425ed/examples/helloworld/cmd/main.go#L22

If we want to add request validation for the SayHello operation, we can change the above code as follows:

1. Validation rules written in Go

validators := []httpoption.NamedValidator{
    httpoption.Op("SayHello", httpoption.FuncValidator(func(value interface{}) error {
        req := value.(*helloworld.SayHelloRequest)
        if len(req.Name) > 10 {
            return werror.Wrapf(gcode.ErrInvalidArgument, "name: length exceeds 10")
        }

        reVarName := regexp.MustCompile(`^\w+$`)
        if !reVarName.MatchString(req.Name) {
            return werror.Wrapf(gcode.ErrInvalidArgument, "name: invalid name format")
        }

        return nil
    })),
}
r := helloworld.NewHTTPRouter(svc, httpcodec.NewDefaultCodecs(nil), httpoption.RequestValidators(validators...))

2. Validation rules written in DSL of validating

validators := []httpoption.NamedValidator{
    httpoption.Op("SayHello", helloworld.ValidateSayHelloRequest(func(req *helloworld.SayHelloRequest) v.Schema {
        return v.Schema{
            v.F("name", &req.Name): v.All(
                v.Len(0, 10).Msg("length exceeds 10"),
                v.Match(regexp.MustCompile(`^\w+$`)).Msg("invalid name format"),
            ),
        }
    })),
}
r := helloworld.NewHTTPRouter(svc, httpcodec.NewDefaultCodecs(nil), httpoption.RequestValidators(validators...))

NOTE:

• The ValidateSayHelloRequest() function is designed to be generated by kokgen
• The validation of each validator will be triggered in its corresponding DecodeRequestFunc (go-kit/kit#908 (comment))

annotation: Add support for multiple HTTP operations

Sometimes, a Go method needs to correspond to more than one URI (or HTTP method). For example:

type Service interface {
	//kun:op GET /messages/{messageID}
	//kun:op GET /users/{userID}/messages/{messageID}
	GetMessage(ctx context.Context, userID string, messageID string) (text string, err error)
}

This will enable the following two alternative HTTP to Go-method mappings:

HTTP	Go Method
GET /messages/123456	GetMessage(ctx, "", "123456")
GET /users/me/messages/123456	GetMessage(ctx, "me", "123456")

This feature is much like (and is partially inspired by) gRPC's additional_bindings option (also see Multiple URI bindings).

Redesign the annotation syntax

Motivation

1. Follow the Go conventions for comment directives (and this).
2. Provide a unified annotation syntax for both comments and struct tags.
3. Make it possible to apply Argument aggregation in a single-line annotation.
4. Improve the comment syntax for manipulating HTTP request/response body fields.

Proposed annotation syntax

HTTP

1. Define the HTTP operation

   • Directive: //kok:op

   • Arguments: <method> <pattern>

     • ...

   • Examples:

     type Service interface {
         //kok:op DELETE /users/{id}
         DeleteUser(ctx context.Context, id int) (err error)
     }
     
     // HTTP request:
     // $ http DELETE /users/101

2. Define the HTTP request parameters

   • Directive: //kok:param
   • Arguments: <argName> [<parameter> [; <parameter2> [; ...]]]
     • parameter: in=<in> name=<name> required=<required> type=<type> descr=<descr>
       • ...
   • Examples:

     • Bind request parameters to simple arguments:

       type Service interface {
           //kok:op PUT /users/{id}
           //kok:param name in=header name=X-User-Name
           UpdateUser(ctx context.Context, id int, name string) (err error)
       }
       
       // HTTP request:
       // $ http PUT /users/101 X-User-Name:tracey

     • Bind multiple request parameters to a struct according to tags:

       type User struct {
           ID   int    `kok:"in=path"`  // name defaults to snake case `id`
           Name string `kok:"in=query"` // name defaults to snake case `name`
           Age  int    `kok:"in=header name=X-User-Age"`
       }
       
       type Service interface {
           //kok:op PUT /users/{id}
           //kok:param user
           UpdateUser(ctx context.Context, user User) (err error)
       }
       
       // HTTP request:
       // $ http PUT /users/101?name=tracey X-User-Age:1

     • Bind multiple query parameters to a struct with no tags:

       type User struct {
           Name    string   // equivalent to `kok:"in=query name=name"`
           Age     int      // equivalent to `kok:"in=query name=age"`
           Hobbies []string // equivalent to `kok:"in=query name=hobbies"`
       }
       
       type Service interface {
           //kok:op POST /users
           //kok:param user
           CreateUser(ctx context.Context, user User) (err error)
       }
       
       // HTTP request:
       // $ http POST /users?name=tracey&age=1&hobbies=music&hobbies=sport

     • Argument aggregation:

       type Service interface {
           //kok:op POST /logs
           //kok:param ip in=header name=X-Forwarded-For; in=request name=RemoteAddr
           Log(ctx context.Context, ip net.IP) (err error)
       }
       
       // The equivalent annotations =>
       // (using backslash `\` for line continuation)
       
       type Service interface {
           //kok:op POST /logs
           //kok:param ip in=header name=X-Forwarded-For; \
           //             in=request name=RemoteAddr
           Log(ctx context.Context, ip net.IP) (err error)
       }
       
       // HTTP request:
       // $ http POST /logs

3. Define the HTTP request body

   • Directive: //kok:body
   • Arguments: <field> or <manipulation> [; <manipulation2> [; ...]]
     • manipulation: <argName> name=<name> type=<type> descr=<descr>
       • ...
   • Examples:

     • Omitted:

       type Service interface {
           //kok:op POST /users
           CreateUser(ctx context.Context, name string, age int) (err error)
       }
       
       // HTTP request:
       // $ http POST /users name=tracey age=1

     • Specified as a normal argument:

       type User struct {
           Name string `json:"name"`
           Age  int    `json:"age"`
       }
       
       type Service interface {
           //kok:op POST /users
           //kok:body user
           CreateUser(ctx context.Context, user User) (err error)
       }
       
       // HTTP request:
       // $ http POST /users name=tracey age=1

     • Specified as -:

       type User struct {
           Name    string
           Age     int
           Hobbies []string `kok:"name=hobby"`
       }
       
       type Service interface {
           //kok:op POST /users
           //kok:body -
           CreateUser(ctx context.Context, user User) (err error)
       }
       
       // HTTP request:
       // $ http POST /users?name=tracey&age=1&hobby=music&hobby=sport

4. Define the success HTTP response

   • Directive: //kok:success

   • Arguments: statusCode=<statusCode> body=<body> manip=`<manipulation> [; <manipulation2> [; ...]]`

     • manipulation: <argName> name=<name> type=<type> descr=<descr>
       • ...

   • Examples:

     type User struct {
         Name string `json:"name"`
         Age  int    `json:"age"`
     }
     
     type Service interface {
         //kok:op POST /users
         //kok:success statusCode=201 body=user
         CreateUser(ctx context.Context) (user User, err error)
     }

5. Define the OAS metadata

   • Directive: //kok:oas

   • Arguments: <property>=<value>

   • Examples:

     // This is the API documentation of User.
     //kok:oas docsPath=/api-docs
     //kok:oas title=User API
     //kok:oas version=1.0.0
     //kok:oas basePath=/v1
     //kok:oas tags=user
     type Service interface {
         //kok:op POST /users
         CreateUser(ctx context.Context, name string, age int) (err error)
     }

6. Define the annotation alias

   • Directive: //kok:alias

   • Arguments: <name>=`<value>`

   • Examples:

     type Service interface {
         //kok:op POST /users
         //kok:param operatorID in=header name=Authorization required=true
         CreateUser(ctx context.Context, operatorID int) (err error)
     
         //kok:op DELETE /users/{id}
         //kok:param operatorID in=header name=Authorization required=true
         DeleteUser(ctx context.Context, id, operatorID int) (err error)
     }
     
     // The equivalent annotations =>
     
     //kok:alias opID=`operatorID in=header name=Authorization required=true`
     type Service interface {
         //kok:op POST /users
         //kok:param $opID
         CreateUser(ctx context.Context, operatorID int) (err error)
     
         //kok:op DELETE /users/{id}
         //kok:param $opID
         DeleteUser(ctx context.Context, id, operatorID int) (err error)
     }

gRPC

• Directive: //kok:grpc
• Arguments: request=<request> response=<response>
• Examples:

  • Omitted:

    type Service interface {
        //kok:grpc
        CreateUser(ctx context.Context, name string, age int) (err error)
    }
    
    // gRPC request:
    // $ grpcurl -d '{"name": "tracey", "age": 1}' ... pb.Service/CreateUser

  • Specified:

    type User struct {
        Name string `json:"name"`
        Age  int    `json:"age"`
    }
    
    type Service interface {
        //kok:grpc request=user
        CreateUser(ctx context.Context, user User) (err error)
    }
    
    // gRPC request:
    // $ grpcurl -d '{"name": "tracey", "age": 1}' ... pb.Service/CreateUser

Minor syntax improvements for `//kok:param`

Syntax improvements

Original syntax (after #18):

• //kok:param <argName> [<parameter> [; <parameter2> [; ...]]]
  • parameter: in=<in> name=<name> required=<required> type=<type> descr=<descr>

New syntax:

• //kok:param <argName> [<parameter> [, <parameter2> [, ...]]]
  • parameter: in=<in> name=<name> required=<required> type=<type> descr=<descr>
• NOTE: If multiple method arguments are involved, you may need to apply multiple bindings. This can be done by adding a new //kok:param directive, or by appending the binding to the end of the last //kok:param directive in a semicolon-separated list.

Example

After the above syntax improvements, here are the equivalent annotations:

One binding per //kok:param

type Service interface {
    //kok:op POST /users
    //kok:param name
    //kok:param age in=query
    //kok:param ip in=header name=X-Forwarded-For, in=request name=RemoteAddr
    CreateUser(ctx context.Context, name string, age int, ip net.IP) (err error)
}

// HTTP request:
// $ http POST /users?name=tracey&age=1

Multiple bindings in a single //kok:param

type Service interface {
    //kok:op POST /users
    //kok:param name; age in=query; ip in=header name=X-Forwarded-For, in=request name=RemoteAddr
    CreateUser(ctx context.Context, name string, age int, ip net.IP) (err error)
}

// HTTP request:
// $ http POST /users?name=tracey&age=1

Multiple bindings in a single //kok:param (but using backslash for better readability)

type Service interface {
    //kok:op POST /users
    //kok:param name; \
    //          age in=query; \
    //          ip in=header name=X-Forwarded-For, in=request name=RemoteAddr
    CreateUser(ctx context.Context, name string, age int, ip net.IP) (err error)
}

// HTTP request:
// $ http POST /users?name=tracey&age=1

Simplify werror

Wrap an error and inherit the error message from another one:

// Lengthy
werror.Wrap(googlecode.ErrInvalidArgument).SetError(err)

// Simplified
werror.Wrap(gcode.ErrInvalidArgument, err)

Wrap an error and set a custom error message:

// Lengthy
werror.Wrap(googlecode.ErrInvalidArgument).SetErrorf("bad request")

// Simplified
werror.Wrapf(gcode.ErrInvalidArgument, "bad request")