cloudwego / thriftgo Goto Github PK

An implementation of thrift compiler in go language.

License: Apache License 2.0

Go 79.06% Thrift 17.29% Makefile 0.41% Shell 1.38% Lua 1.86%

thriftgo's Introduction

thriftgo

Thriftgo is an implementation of thrift compiler in go language. It has a command line interface similar to the apache/thrift compiler and is enhanced with a plugin mechanism which makes it more powerful.

Installation

Note: before executing the following commands, make sure your GOPATH environment is properly set.

Using go install:

GO111MODULE=on go install github.com/cloudwego/thriftgo

Or build from source:

git clone https://github.com/cloudwego/thriftgo.git
cd thriftgo
export GO111MODULE=on
go mod tidy
go build
go install

Usage

The thriftgo command line tool accepts IDL files and compiles them to the target language. Each backend has a plentiful set of options to customize the generated code.

By the moment, thriftgo can generates golang code only. More backends will be added in the future.

To compile an thrift IDL to golang files with the default setting, you can just run:

thriftgo -g go the-idl-file.thrift

Run thriftgo -h to see all available options for each backend and their meanings.

Plugin

If the code generated by thriftgo does not satisfy your needs and the options provideds do not meet your requirements. You may also write plugins to generate code beside thriftgo while taking the advantage of thriftgo's IDL parser. Check the documentation of the plugin package for more details.

thriftgo's People

Contributors

Stargazers

Watchers

thriftgo's Issues

cannot use comment with field in one line

Describe the bug

As for now, thriftgo only support head comment for field struct service, but we want to use trailing comment for field. because the heading comment for field will make the line of file grow 2x.

To Reproduce

namespace go platform
struct PingReq {
    1: string message // filed comment1
}

struct PongResp {
    // field comment2
    1: string message
}

service Platform {
    PongResp Ping(PingReq req)(api.get='/ping')
}

use hertz to genCode

hz new -idl=idl.thrift

Expected behavior

PingRep.Message should have a trailing comment.

Screenshots

If applicable, add screenshots to help explain your problem.

Kitex version:

Please provide the version of Kitex you are using.

Environment:

The output of go env.

Additional context

Add any other context about the problem here.

Missing context.Context in the generated code

Describe the bug

生成的 go 文件中许多方法缺少 context.Context 作为第一个参数

To Reproduce

test.thrift

namespace go test

enum EntryType {
    Normal = 0,
    Config = 1,
}

struct Entry {
    1: optional EntryType type,
    2: optional i64 term,
    3: optional i64 index,
    4: optional binary command,
}

thriftgo -g go test.thrift

Screenshots

Environment:

thriftgo 0.3.2
go 1.21.0

[help] 有办法根据一个thrift文件动态解析出其中的一个struct 并实例化成一个go对象么

Generate option `thrift_import_path` leaving unused default Thrift library

Hello, generated code has default thrift library as thrift0, leaving it unused.

Generate comand: thriftgo -o . --gen go:gen_deep_equal=true,frugal_tag=true,thrift_import_path=github.com/apache/thrift/lib/go/thrift/x model.thrift

Thanks.

Generated code can't pass staticcheck

Description

thriftgo generated code can't pass staticcheck, so we must write an extra config file staticcheck.conf to ignore some checks, but staticcheck will ignore all golang code files under the same directory tree which the config file placed

Reproduction

assume we have a thrift file named 'foo.thrift' and below is the content

namespace golang foo

struct Foo {
    1: required string name,
    2: required i64 age,
}

execute shell commands below

go mod init code.byted.org/demo
thriftgo -g go -o ./ foo.thrift
go mod tidy
go mod edit -replace  github.com/apache/thrift=github.com/apache/[email protected]
go mod tidy

staticcheck ./...

and will get the output

foo/foo.go:113:67: error strings should not be capitalized (ST1005)

Recommandation

add //lint:ignore directive to generator/golang/templates/struct.go::StructLikeRead

RequiredFieldNotSetError:
	//lint:ignore ST1005 {some reasons}
	return thrift.NewTProtocolExceptionWithType(thrift.INVALID_DATA, fmt.Errorf("Required field %s is not set", fieldIDToName_ModelSync[fieldId]))

use lowercase word instead

return thrift.NewTProtocolExceptionWithType(thrift.INVALID_DATA, fmt.Errorf("required field %s is not set", fieldIDToName_ModelSync[fieldId]))

Reference

https://staticcheck.io/docs/configuration/#line-based-linter-directives

Please add release for tag 0.2.9

Homebrew/homebrew-core#127198

can you show thrift & thriftgo version dependency in readme.md ?

Describe the bug
thriftgo 0.2.4
kitex v0.4.3
go version go1.18.8 darwin/amd64
requires github.com/apache/thrift v0.17.0

To Reproduce

Steps to reproduce the behavior:

kitex -type thrift -module xx xx.thrift
go build

A clear and concise description of what you expected to happen.

Screenshots
github.com/cloudwego/[email protected]/pkg/utils/thrift.go:114:20: not enough arguments in call to msg.Read

Environment:

The output of go env.

Additional context

go 1.18

require (
	github.com/apache/thrift v0.17.0
	github.com/cloudwego/hertz v0.4.1
	github.com/cloudwego/kitex v0.4.3
	github.com/jessevdk/go-flags v1.5.0
	gopkg.in/yaml.v3 v3.0.1
)

require (
	github.com/bytedance/go-tagexpr/v2 v2.9.2 // indirect
	github.com/bytedance/gopkg v0.0.0-20220531084716-665b4f21126f // indirect
	github.com/bytedance/sonic v1.5.0 // indirect
	github.com/chenzhuoyu/base64x v0.0.0-20211019084208-fb5309c8db06 // indirect
	github.com/chenzhuoyu/iasm v0.0.0-20220818063314-28c361dae733 // indirect
	github.com/choleraehyq/pid v0.0.15 // indirect
	github.com/cloudwego/fastpb v0.0.2 // indirect
	github.com/cloudwego/frugal v0.1.3 // indirect
	github.com/cloudwego/netpoll v0.3.1 // indirect
	github.com/cloudwego/thriftgo v0.2.1 // indirect
	github.com/fsnotify/fsnotify v1.5.4 // indirect
	github.com/golang/protobuf v1.5.2 // indirect
	github.com/google/pprof v0.0.0-20220608213341-c488b8fa1db3 // indirect
	github.com/henrylee2cn/ameda v1.4.10 // indirect
	github.com/henrylee2cn/goutil v0.0.0-20210127050712-89660552f6f8 // indirect
	github.com/jhump/protoreflect v1.8.2 // indirect
	github.com/json-iterator/go v1.1.12 // indirect
	github.com/klauspost/cpuid/v2 v2.1.0 // indirect
	github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421 // indirect
	github.com/modern-go/reflect2 v1.0.2 // indirect
	github.com/nyaruka/phonenumbers v1.0.55 // indirect
	github.com/oleiade/lane v1.0.1 // indirect
	github.com/tidwall/gjson v1.13.0 // indirect
	github.com/tidwall/match v1.1.1 // indirect
	github.com/tidwall/pretty v1.2.0 // indirect
	github.com/twitchyliquid64/golang-asm v0.15.1 // indirect
	golang.org/x/arch v0.0.0-20220722155209-00200b7164a7 // indirect
	golang.org/x/net v0.0.0-20210614182718-04defd469f4e // indirect
	golang.org/x/sync v0.0.0-20210220032951-036812b2e83c // indirect
	golang.org/x/sys v0.0.0-20220817070843-5a390386f1f2 // indirect
	golang.org/x/text v0.3.6 // indirect
	google.golang.org/genproto v0.0.0-20210513213006-bf773b8c8384 // indirect
	google.golang.org/protobuf v1.28.0 // indirect
)

thriftgo需要配合kitex使用吗？

parser: int const type support is not friendly

例子:

// 某业务，定义常量的时候会根据位数进行区分，比如下列例子是前4位和后四位分别有不同含义
const i16   SUB_BIZ_TYPE_INSURANCE_CLAIM    = 00040001  
const i16   SUB_BIZ_TYPE_TEST_SCENE_A       = 00049999
const i16   SUB_BIZ_TYPE_TEST_SCENE_B       = 00049998

目前解析报错:

parse demo.thrift err: parseConstValue failed at '00049999': strconv.ParseInt: parsing "00049999": invalid syntax

期望: 支持这种类型
sdk版本: v0.1.3

Feature Request: Configurable Plugin Execution Time Limits

Is your feature request related to a problem? Please describe.

I'm trying to generate codes with cloudwego/kitex which is a plugin of thriftgo (I think you must know it well), but my IDL file is huge and it takes a long time to be processed by the kitex plugin.

As far as I know the plugin execution time is limited to 1 minute. So when the sizes of my IDL files are large enough, thriftgo won't make it but throw an error says execute plugin 'kitex' failed: signal: killed.

Describe the solution you'd like

A CLI argument that overrides the time limit. A possible solution could be found below:

thriftgo -p foo=/path/to/plugin --plugin-time-limit=1000 # plugins have to reponse in 1000s
thriftgo -p foo=/path/to/plugin --plugin-time-limit=10m  # plugins have to reponse in 10 minutes
thriftgo -p foo=/path/to/plugin --plugin-time-limit=1h   # plugins have to reponse in 1 hour
thriftgo -p foo=/path/to/plugin --plugin-time-limit=0    # explicitly remove the time limitation

When the arguments were resolved, the global variable that contains the time limit should be modified, which initially was

// MaxExecutionTime is a timeout for executing external plugins.
var MaxExecutionTime = time.Minute

Describe alternatives you've considered

Currently I haven't come up with any ideas but to split my IDL files.

Additional context

None.

import "github.com/apache/thrift/lib/go/thrift" 不能够resolve的问题

在终端执行 thriftgo -g go 之后，生成了相应的go文件，但是"github.com/apache/thrift/lib/go/thrift"报错，can‘t resolve ，不知道什么原因。在kitex的测试example中不会出现这个问题。

package api

import (
	"context"
	"fmt"
	"github.com/apache/thrift/lib/go/thrift"
)

补充一下:
用thrift -r --gen go tutorial.thrift命令生成的go文件如下：

func (p *Request) Read(ctx context.Context, iprot thrift.TProtocol) error {
  if _, err := iprot.ReadStructBegin(ctx); err != nil {
    return thrift.PrependError(fmt.Sprintf("%T read error: ", p), err)
  }
...

用thriftgo -g go tutorial.thrift 命令生成的go文件如下：

func (p *Request) Read(iprot thrift.TProtocol) (err error) {

	var fieldTypeId thrift.TType
	var fieldId int16

	if _, err = iprot.ReadStructBegin(); err != nil {
		goto ReadStructBeginError
	}
...

后者在ReadStructBegin()缺少参数，编译器报错，很多函数函数调用都有这个问题，不知道如何解决。

Missing exception in the generated code

Hello, I tried to generate a Go code from simple Thrift file and I noticed that there are some difference in the code generated by Apache Thrift and ThriftGo. The code generated by ThriftGo is missing some lines that handle exception. It happen to void function that throw exception.

This is the thrift file:

exception SomeException {
	1: string message
}

service SomeService
{
	void some_function() throws(1:SomeException o1)
}

Then using Apache Thrift (v0.13.0) the generated Go code is:

func (p *SomeServiceClient) SomeFunction(ctx context.Context) (err error) {
	var _args0 SomeServiceSomeFunctionArgs
	var _result1 SomeServiceSomeFunctionResult
	if err = p.Client_().Call(ctx, "some_function", &_args0, &_result1); err != nil {
		return
	}
	switch {
	case _result1.O1 != nil:
		return _result1.O1
	}

	return nil
}

But, when using ThriftGo (v0.3.1) the generated code is:

func (p *SomeServiceClient) SomeFunction(ctx context.Context) (err error) {
	var _args SomeServiceSomeFunctionArgs
	var _result SomeServiceSomeFunctionResult
	if err = p.Client_().Call(ctx, "some_function", &_args, &_result); err != nil {
		return
	}
	return nil
}

As you can see, the switch block that handle the exception is missing.

I'm not familiar with the ThriftGo, but I guess it is because in this template https://github.com/cloudwego/thriftgo/blob/main/generator/golang/templates/client.go#L96-L103 the throw exception will be generated if the function is not a void type.

Thank you

Reporting a vulnerability

Hello!

I hope you are doing well!

We are a security research team. Our tool automatically detected a vulnerability in this repository. We want to disclose it responsibly. GitHub has a feature called Private vulnerability reporting, which enables security research to privately disclose a vulnerability. Unfortunately, it is not enabled for this repository.

Can you enable it, so that we can report it?

Thanks in advance!

PS: you can read about how to enable private vulnerability reporting here: https://docs.github.com/en/code-security/security-advisories/repository-security-advisories/configuring-private-vulnerability-reporting-for-a-repository

Add document for plugins?

Can we add document for plugins?

compile error in generated go code with use_type_alias=false and `typedef`-ed struct type

Describe the bug

As stated in the title.

To Reproduce

With thrift file test.thrift:

namespace go test

struct Req {
    1: required i64 id,
}

typedef Req Req2

And thriftgo command

$ thriftgo --gen go:use_type_alias=false --out test test.thrift

Will produce uncompilable code like:

type Req2 Req

func NewReq2() *Req2 {
	return NewReq() // this line fails
}

Expected behavior

The generated code should compiles well.

Environment:


GO111MODULE="on"
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOOS="darwin"
GOROOT="/usr/local/Cellar/go/1.19.5/libexec"
GOSUMDB="sum.golang.google.cn"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.19.5/libexec/pkg/tool/darwin_amd64"
GOVCS=""
GOVERSION="go1.19.5"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/5h/hw5wnt8d0q91n8yl5h5p1sdw0000gp/T/go-build20884326=/tmp/go-build -gno-record-gcc-switches -fno-common"

Processor method `exception` field is not optional.

Describe the bug

Service method exception field is not optional, so that will always be written to the wire. This resulting a misbehaviour for clients thinking that there's an error/exception.
I am not using Kitex, but this should be reproducible with it.
I was looking for Kitex example but it seem there's no case for it (referring to bizdemo/easy_note/idl/note.thrift and search "throws").

To Reproduce

Steps to reproduce the behavior:

Default generate config, with unknown field enabled (thriftgo).
Apache Thrift generate code for NodeJS (js:"node,ts,es6").

Expected behavior

Works well with method exceptions. make exception field optional.

Screenshots

Server (thriftgo)

type CoreServiceGetClassroomsResult struct {
	Success        *structs.GetClassRoomsResponse `thrift:"success,0,optional" frugal:"0,optional,structs.GetClassRoomsResponse" db:"success" json:"success,omitempty"`
	E              *exceptionsc.CommonException   `thrift:"e,1" frugal:"1,default,exceptionsc.CommonException" db:"e" json:"e"`
	_unknownFields unknown.Fields
}

Client (thrift/NodeJS)

Kitex version:

Direct use of thriftgo.

Environment:

go env

set GO111MODULE=
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\Nugraha\AppData\Local\go-build
set GOENV=C:\Users\Nugraha\AppData\Roaming\go\env
set GOEXE=.exe
set GOEXPERIMENT=
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMODCACHE=C:\Users\Nugraha\go\pkg\mod
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=C:\Users\Nugraha\go
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=C:\Program Files\Go
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=C:\Program Files\Go\pkg\tool\windows_amd64
set GOVCS=
set GOVERSION=go1.19
set GCCGO=gccgo
set GOAMD64=v1
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=NUL
set GOWORK=
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=C:\Users\Nugraha\AppData\Local\Temp\go-build439957652=/tmp/go-build -gno-record-gcc-switches

Additional context

tbd.

Can thriftgo generates snakecase go tag for struct?

Proposal: Add a new flag to generate another type String()

Is your feature request related to a problem? Please describe.

It is inconvenience when I print logs with String().
%#v it is not readable enough.

Describe the solution you'd like

I suppose we could add a new flag to indicate another String() function.
Such as
-generate_json_string() to indicate thriftgo generate the string() function with "json.MarshalString()"

v0.2.5/v0.2.6 will cause format problem in Windows

Describe the bug

v0.2.5/v0.2.6 这俩版本在Windows下应该有问题，from hertz user:

安装了Hz和thriftgo之后，试着编译cloudwego教程中的hello.thrift，可是他生成了代码之后，却提示格式化错误。
D:\dox\hertz_demo>hz new -idl idl/hello.thrift -module hzdemo
[WARN] Failed to format D:\dox\hertz_demo\biz\model\hello\example\hello.go: 4:15: illegal character U+005C '' (and 3 more errors)

降级到v0.2.4问题解决。

To Reproduce
cc @FGYFFFF

Additional context
cloudwego/hertz#588

version field did not get updated for the latest release 0.2.10

👋 it looks like the version field did not get changed for the latest release.

relates to Homebrew/homebrew-core#128688

How to place the generated code file in the same level directory of IDL file

如何将生成的代码文件放置在和 IDL 同级目录下：

|-- exampledir
   |-- example.thrift
   |-- example.go

通过 thriftgo -g go -o ./ example.thrift 命令仍然会保留一个以 namespace 命名的子文件夹：

|-- exampledir
   |-- example.thrift
   |-- examplenamespace
     |-- example.go

license should update your self name

at : https://github.com/cloudwego/thriftgo/blob/main/LICENSE#L189

   Copyright [yyyy] [name of copyright owner]

feat: peek/modify decoded unknown fields

Is your feature request related to a problem? Please describe.

We've got CarryingUnknownFields to see if there's unknown fields, but what we could do with it besides reflecting them back to wire data? Currently, _unknownFields is not exported, and no methods expose it. So technically, there's no way to touch them without going to unsafe.

Describe the solution you'd like, and alternatives you've considered

Expose _unknownFields field, or provide a method to expose it.
In addition, since field ids can be unordered, implement new method on type unknown.Fields to find the field id and field type, or matcher that calls back whether there's matched field id and field type.

Additional context

No additional context.

FR: filed comment in idl to code generated

enum EEE{
    One = 1, // one
   Two, // two
}

struct One{
    1: EEE which // which
}

to generate

type EEE int64
const (
    EEE_One EEE = 1 // one
    EEE_Two EEE = 2 // two
)

type One struct{
    Which EEE // which
}

parser: apache thrift reserved keywords are not supported

Apache thrift has defined a set of reserved keywords that users can not use. But thriftgo does not yet support.

[GLCC 赛题] 基于 Thriftgo 实现一个 IDL 裁切工具

题目简介：
1、题目描述

Thriftgo 是 Go 语言实现的 Thrift IDL 解析和代码生成器，支持完善的 Thrift IDL 语法和语义检查以及代码生成。在实际场景中，线上 IDL 仓库存常常会有 IDL 过大，未使用的字段过多，导致生成大量无效代码，所以期望基于 Thriftgo 的语法解析能力提供一个 IDL 裁切工具来解决这个问题。

2、编码任务

在 Thriftgo 上进行开发，在 Thriftgo 语法解析部分进行扩展开发，实现相应的接口，提供一个 IDL 裁切工具，能够根据入口 IDL 所用到的结构体进行处理，找出所有直接或间接引用到的结构，进行精简，并重新输出为 Thrift 文件。

3、技能要求和编程语言

熟悉 golang 编程
熟悉 Thrift IDL 语法
了解语法解析等相关知识

4、预期完成结果

提供一个 IDL 裁切工具，能够
- 输入：一个或者多个主 IDL 入口，以及一份 IDL 文件目录路径
- 输出：精简后的一组 IDL 文件，namespace 和路径等不变，里面的结构体或内容被裁切，去除掉所有无用部分

cloudwego / thriftgo Goto Github PK

thriftgo's Introduction

thriftgo

Installation

Usage

Plugin

thriftgo's People

Contributors

Stargazers

Watchers

Forkers

thriftgo's Issues

Description

Reproduction

Recommandation

Reference

Recommend Projects

Recommend Topics

Recommend Org

Jobs