ezbuy / ezorm Goto Github PK

现有强限制: 主键命名必须是 <表名> Id 的形式
是否考虑支持可自定义主键名?

lastInsertId, err := result.LastInsertId()
if err != nil {
	return result, err
}

In mssql driver cases , LastInsertId will occasionally return the wrong auto-increase id.

The mssql driver already reported it in its document.

这个支持事物操作吗？

branch: develop
generated code sample:

func (o *_OrderItemMgr) Query(query interface{}, limit, offset int, sortFields []string) (*mgo.Session, *mgo.Query) {
	session, col := OrderItemMgr.GetCol()
	q := col.Find(query)
	if limit > 0 {
		q.Limit(limit)
	}
	if offset > 0 {
		q.Skip(offset)
	}

	_OrderItemSort(q, sortFields)
	return session, q
}

reason:
ORM should not have logic, such as limit > 0, offset > 0, cause if these two params equal to 0, it is OK for mongo shell execution. And we should put logic check in Service Layer. In this example, if limit equals to 0, does it return all the records?

Background

ezorm's mongo driver assigns all sessions when user invokes MustNewMgoSessions to create sessions according to the MaxSessions. However , after we created the session. the socket is already bind to the allocated sessions . All socket will be reset to the unreachable server when the internal Ping GoRoutine reported that one sever is down at that time. After this migration, all sockets will connect to the “that time reachable” server and not come back to the "that time unreachable" server even if we fix the server issue , unless we restart our service to trigger the re-assignment.

And when the "that time reachable" server down because of the huge QPS. The sockets will migrate to the "that time unreachable" server, our two server will be both down at last.

We lost our LB feature which is implemented in our new mgo driver !

Proposal

We should do some sth to let the migrated sockets back to its previous server when its previous server back to available state from the unreachable state.

We can deploy a timer into the setup function ,such as MustNewMgoSessions. And refresh session-socket mapping more frequently ,for now, we only refresh the sockets when the Ping is timeout or even unavailable .

What's more

With this enhancement, we can now gracefully enlarge our mongo cluster without restarting our service (X

The generated orm source codes already have the function that can return a list of qualified objects. But in some scenarios, especially an given object has many properties, that we only interest with one property. So it's better if ezorm can provide a facility that can return a list of the objects' property.

ezorm.v2 should only generate operation functions with context.Context .

Background

For now , in our generator , we should do lots of type assert for different template engine, such as this:

		for _, obj := range databases {
			switch obj := obj.(type) {
			case *parser.Obj:
				for _, t := range obj.GetConfigTemplates() {
					fileAbsPath := output + "/gen_" + t + ".go"
					executeTpl(fileAbsPath, t, obj)
				}
			case *mysqlr.MetaObject:
				if err := mysqlr.GenerateConfTemplate(output, genGoPackageName); err != nil {
					return err
				}
			}
		}

or this:

		for db, objs := range dbObjs {
			switch db {
			default:
				continue
			case "mysqlr":
				for _, obj := range objs {
					if err := mysqlr.GenerateScriptTemplate(output, genGoPackageName, obj.(*mysqlr.MetaObject)); err != nil {
						return err
					}
				}
			case "mysql":
				path := fmt.Sprintf("%s/create_%s.sql", output, db)
				genType := db + "_script"
				executeTpl(path, genType, objs)

			}

which can't be maintainable if we need to support more template engine .

Proposal

It will be better if we have a register mechanism to allow template engine to register their own template code , parser , as well as the generator .

For internal generator

Our internal generator (AKA mysql/mysqlr/mongo) will be integrated as the lib dependency . Thus , they will all have their own Register methods to tell ezorm the template path , generator handler as well as the metadata(YAML) parsing logic . Like:

// internal/plug/mysql_reg.go
type MySQLRegister struct{}

func (m *MySQLRegister) MustRegister(path string , generator GenerateHandler) {}

// cmd/gen.go

MustRegister(&MySQLRegister{})

For 3rd generator

Outside template generators will be considered to integrate with the binary , like protoc-gen-x , the 3rd should be named as the ezorm-gen-x to allow ezorm to use the binary , and ezorm will offer the new subcommand to support this mechanism other than ezorm code.

Goals

• common: raw query generator
• builtin plugin: mysql
• builtin plugin: mysqlr
• builtin plugin: mongo
• plugin: support 3rd bin-like generator

Since we migrate from mgo to mongo-go-driver , we lost our monitor policy for the mongo connection pool .

We need work out the new way to full-fill it .

Problem

mongo orm's method Count(query interface{}) int return an ambiguous value cause return 0 even if fail to connect the databases.

Proposal

orm should return errors returned by the database driver and hand it over to the developer.

Background

• Since the redis-orm mysql template engine is widely-used among us , we should migrate it into ezorm v2.
• The current ezorm v2 mysql template lacks the support of database transition which is the very important feature in our daily development .

About new template engine

names: mysqlr (which should declare in the yaml file and replace of the origin mysql)

Behavior will be the same as redis-orm's mysql template usage .

Except:

• Removing all methods do not support context
• Removing deprecated methods
• Removing some useless plugins (e.g.: protobuf to model , model to protobuf)
• Fixes all lint errors

OSX 10.11.3
go version go1.6 darwin/amd64

PropertyValue:
  db: mongo
  fields:
    - Pid: int
      flags: [index]
    - PropertyTranslation: Translation
    - Vid: int
      flags: [index]
    - ValueTranslation: Translation
  uniques: [[Pid, Vid]]

类似上面的定义 生成了以下代码:

func (o *_PropertyValueMgr) FindOneByPidVid(Pid int32, Vid int32) (result *PropertyValue, err error) {
    query := db.M{
        "PidVid": PidVid,
    }
    session, q := PropertyValueMgr.Query(query, 1, 0, nil)
    defer session.Close()
    err = q.One(&result)
    return
}

func (o *_PropertyValueMgr) MustFindOneByPidVid(Pid int32, Vid int32) (result *PropertyValue) {
    result, _ = o.FindOneByPidVid(PidVid)
    if result == nil {
        result = PropertyValueMgr.NewPropertyValue()
        result.PidVid = PidVid
        result.Save()
    }
    return
}

编译时产生如下错误:

src/some/models/gen_PropertyValue_mongo_orm.go:181: undefined: PidVid
src/some/models/gen_PropertyValue_mongo_orm.go:190: undefined: PidVid
src/some/models/gen_PropertyValue_mongo_orm.go:193: result.PidVid undefined (type *PropertyValue has no field or method PidVid)
src/some/models/gen_PropertyValue_mongo_orm.go:193: undefined: PidVid

Hi guys,

Hope you are all well !

I was wondering if would be to generate an admin ui interface from the yaml file generated.

I saw a couple of admin interface boilerplates like:
https://github.com/marmelab/admin-on-rest
https://github.com/marmelab/ng-admin

Also, I found some example binding a go-rest api but not generating dynamically the forms through a schema export for ng-admin:
https://github.com/Onefootball/entity-rest-api
https://github.com/deluan/ngago

Ultimately, I would like to browse the content of any of my gorm models and bootstrap a web admin to have a unique way to edit or test my content.

Do you have any idea about how to make it happen ? Does ezorm provide a gin-gonic based api providing models definitions ?

Thanks in advance.

Cheers,
Richard

• Since go-bindata is deprecated , and can be fully replaced with the Go embed package .
• Go embed package is introduced in Go 1.16 , and v2 should require Go version 1.18+

ezorm v2 should only focus :

• MySQL (5.7)
• Mongo

for now .

背景

在一些特定的业务场景(审计日志)下，往往会需要在 model 层 (不管是 M¹ -> M 还是 P² ->(C³) -> M ) 记录一些 changelog(auditlog) 变化

Proposal

The Generator (diff模板生成)

在 (YAML)model struct 上生成 diff 方法，diff 方法接受另一个 M (提供 option , 可以带入一个 P -> C -> M 的转换过程)。

// 方法摘要
func (m *Model) Diff(m1 *Model) (bool,Differ,error)

The Differ

Differ 用来描述一次具体的DIFF结果，DIFF 结果可以按照不同业务场景组装:

• Extras: 提供额外信息展示
• OnKey: 用来描述oplog是否需要在 Key 的层级聚合

type Differ struct{
   // not export field , export behavior as method
   before *M
  // not export field , export behavior as method
  after *M
}

Now we use "DataSourceDSN" to init mysql connection. We need to add a new manner: use struct's fields to init mysql.

The struct is:

type MysqlFieldConfig struct {
	Host            string
	Port            int
	UserName        string
	Password        string
	Database        string
	PoolSize        int
	ConnMaxLifeTime time.Duration
}

We will convert it to standard mysql DSN to connect mysql database.

In addition, directly connect to the database when init instead of trying to connect during use.

Per #144 , we plan to drop all no context function , but mgo.v2 itself do not support context in most of Query methods .

We need to replace the underlying mongo driver to the other version supports context .

Prefer migrates to its official repository

Ref

https://dev.mysql.com/doc/refman/8.0/en/built-in-function-reference.html

Lists

• COUNT
• UPPPER
• LENGTH
• CAST
• UCASE
• LOWER
• LCASE

现状

对于复杂sql，例如涉及JOIN操作，我们一般需要在程序代码中书写冗长的sql语句和rows.Scan语句。这导致了代码可读性不高，并且不同开发的实现千奇百怪，并没有一个统一的规范。

这主要的原因是ezorm v1并不支持复杂sql，它只支持单表操作，而一旦涉及到多表，就会出现类似如下的代码：

func queryUsers(db *sql.DB, id int64) error {
  sql := `
  SELECT u.id, u.name, u.phone, ud.text
  FROM user u
  JOIN user_detail ud ON ud.user_id=u.id
  WHERE u.id=?
  `
  rows, err := db.Query(sql, id)
  if err != nil { return err }
  defer rows.Close()

  for rows.Next() {
    var u User
    err = rows.Scan(&u.Id, &u.Name, &u.Phone, &u.Text)
    if err != nil {
      return err
    }
    ....
  }
}

甚至对于一些动态的sql语句，我们还需要用到一系列字符串处理函数去做拼接。

提案：自动生成sql调用代码

这个提案的核心**是，让用户在单独的文件中输入复杂sql语句，通过生成器来解析sql语句，生成语句的调用函数，类似于Java中的Mybatis。

在mybatis中，一般把sql语句定义在xml文件中，而在ezorm中使用的一直是更加现代化的yaml，所以我认为可以在yaml中给出sql语句、方法的入参出参，然后由生成器解析自动生成对该sql的调用。

目前我初步给出的格式是：

results: 
  User:
    - Id: int64
      map: [u.id]
    - Name: string
      map: [u.name]
    - Phone: string
      map: [u.phone]
    - Text: nullstring
      map: [ud.text]

methods:
  FindUserById: 
    in: uid int64
    ret: User
    sql: |
      SELECT u.id, u.name, u.phone, ud.text
      FROM user u
      JOIN user_detail ud ON u.id=ud.user_id
      WHERE u.id=${uid}

  ListUsers:
    in: offset, limit int32
    ret: '[]*User'
    sql: |
      SELECT u.id, u.name, u.phone, ud.text
      FROM user u
      JOIN user_detail ud ON u.id=ud.user_id
      LIMIT ${offset}, ${limit}

这样我们只需要关心结构体的定义和sql语句的编写，不需要再关心rows.Scan等问题了。

生出来的代码可以通过以下语句调用：

// 查询id为123的用户
user, err := model.FindUserById(123)
// 列出前100个用户
users, err := model.ListUsers(0, 100)

自动生成result结构体

如果生成器能够连接数据库，那么就能通过解析sql语句来自动生成其返回的数据结构体：

methods:
  FindUserById: 
    in: uid int64

    # 自动生成User结构体，这里需要生成器连接数据库
    ret: User
    gen-ret: true

    sql: |
      SELECT u.id, u.name, u.phone, ud.text
      FROM user u
      JOIN user_detail ud ON u.id=ud.user_id
      WHERE u.id=${uid}

  ListUsers:
    in: offset, limit int32

    # 这里不再需要gen-ret了，因为User已经通过FindUserById生成好了
    ret: '[]*User'

    sql: |
      SELECT u.id, u.name, u.phone, ud.text
      FROM user u
      JOIN user_detail ud ON u.id=ud.user_id
      LIMIT ${offset}, ${limit}

这样我们就不需要再去手动定义results了。

sql复用

如果某一段sql出现频率过高，可以使用如下方式复用：

sqls:
  UserFields: u.id, u.name, u.phone, ud.text
  JoinDetail: JOIN user_detail ud ON u.id=ud.user_id

methods:
  FindUserById: 
    in: uid int64
    ret: User
    sql: |
      SELECT @{UserFields}
      FROM user u
      @{JoinDetail}
      WHERE u.id=${uid}

  ListUsers:
    in: offset, limit int32
    ret: '[]*User'
    sql: |
      SELECT @{UserFields}
      FROM user u
      @{JoinDetail}
      LIMIT ${offset}, ${limit}

动态sql

如果用户希望sql语句根据传入的参数有所变化，可以使用动态sql：

methods:
  BatchInserts:
    in: 'users []*User'
    ret: affected
    sql: |
      INSERT INTO `user`(name, phone, email)
      VALUES
      %{for user in users join ','}
       (${user.Name}, ${user.Phone}, ${user.Email})
      %{end}

以上只是初步方案。另外，我个人针对这些特性有一个简陋的实现：go-orm-generator。

I heard that ezorm v2 introduced Generics, but I can not find any related example?

In tpl/mongo_orm/mong_orm.gogo does not apply the table attributes.

By using the "table" attribute in model.yaml, ezorm should generate {{.Table}} instead of {{.Package}} for RegisterEzOrmObjXXX funtions, just like GetCol().

It causes multiple namespaces are generated in a single scheme.

For the sql-method generator, make some optimizations:

• For nullable fields, use sql.NullXxx to scan and assign values.
• When defining yaml, cancel the definition of method level, and level 0 defaults to method definition.