mijia / modelq

ModelQ is a code generator for creating Golang codes/models to access RDBMS database/tables

ModelQ

ModelQ is a code generator for creating Golang codes/models to access RDBMS database/tables (only MySQL/PostgresQL supported for now).

Updates

1. Add the template support for generated codes, e.g. examples/custom.tmpl, you can define you own code segments for each part and don't need to define all of them, the generated code is listed in examples/custom/*.go

Simple Idea

Read the schema from MySQL/PostgresQL database (the whole database or only some tables), and Bang! The go models are there. I embrace the "SQL First" for modeling the business, then use the ModelQ to generate corresponding models for accessing. ModelQ is concerning about two aspects:

1. Easy CRUD interface and query builder but without the golang reflection involved.
2. Facilitate the Go compiler for the correctness (I think this is very important.)

A simple example could be found under ./examples, it is about a blog model contains Users and Articles. So the database can be set up using the examples/blog.mysql.sql or examples/blog.pq.sql, then run

$ modelq -db="root@/blog" -pkg=mysql -tables=user,article -driver=mysql -schema=blog

or

$ modelq -db="dbname=blog sslmode=disable" -pkg=postgres -tables=user,article -driver=postgres -schema=public

Then the models for User and Article would be generated in the directory of "./examples".

CLI Usage

-db="": Target database source string: e.g. root@tcp(127.0.0.1:3306)/test?charset=utf-8
-dont-touch-timestamp=false: Should touch the datetime fields with default value or on update
-driver="mysql": Current supported drivers include mysql, postgres
-p=4: Parallell running for code generator
-pkg="": Go source code package for generated models
-schema="": Schema for postgresql, database name for mysql
-tables="": You may specify which tables the models need to be created, e.g. "user,article,blog"
-template="": Passing the template to generate code, or use the default one

You can embed this CLI command in go generate tools

API

Please check the ./examples/model_test.go to take a glance. Some basic queries are supported, and to take advantage of the compiler, have to use many type guarded funcs defined for the model, e.g.

objs := models.UserObjs
users, err := objs.Select("Id", "Name", "Age").
                   Where(objs.FilterAge(">=", 15).Or(objs.FilterAge("IN", 8, 9, 10))).
                   OrderBy("-Age", "Name").Limit(1, 20).List(db)

The Age of User model is a int, so go compiler will complain if a string is sent in like objs.FilterAge(">", "15"). ModelQ will generate all the filters for each field/column of each model then the type requirements would be in the func signatures.

To support different drivers, modelq have to use gmq.Open and gmq.Beginx for gmq.Db and gmq.Tx objects, like

db, err := gmq.Open("postgres", "dbname=blog sslmode=disable")
tx, err := db.Beginx()
gmq.WithinTx(db, func(tx *gmq.Tx) error {...})

Can't do so far

This is only a early rough implementation, missing a lot of things so far.

• distinct, sum, average and etc. Definitely will get those.
• Joins and Unions. Those seems very likely to the count/distinct/sum and etc. Complicated data structure may be needed.
• The generated models rely on the modelq/gmq package, I am not sure if this would be OK, or could this be changable and plugable, no idea so far.
• No relations for complicated modeling (maybe will never consider this)
• Only MySQL, PostgresQL supported
• Seems github.com/lib/pq has problems to support time.Time scan

But I just want to release it early and get the feedbacks early. So ideas and pull requests would be really welcomed and appreciated!

License

MIT

modelq's Issues

Postgres date fields become strings

If I have a date field - the gmq model created translates this into a string field containing (for example)

2007-09-10T00:00:00Z

I see no obvious reason why this could not be created as a golang datetime?

Any chance of this being the standard behaviour - or maybe optional as a switch?

Make it easier to generate sqlx code

To quote the README:

The generated models rely on the modelq/gmq package, I am not sure if this would be OK, or could this be changable and plugable, no idea so far.

I think different users have different preferences. What I think woul be nice, is if it was easier to support generating sqlx code (with a custom template is OK). sqlx uses a "db" tag rather than a "json" tag by default.

With that in mind, it would be great if the default struct template stated something like:

var modelStruct string = `type {{.Name}} struct {
    {{range .Fields}}{{.Name}} {{.Type}} `json:"{{.FieldMeta}}"`{{if .Comment}} // {{.Comment}}{{end}}
    {{end}}
}

instead of

var modelStruct string = `type {{.Name}} struct {
    {{range .Fields}}{{.Name}} {{.Type}} {{.JsonMeta}}{{if .Comment}} // {{.Comment}}{{end}}
    {{end}}
}

That way it would be easier/less hacky to make a "db" tag for sqlx in place of your default JSON tag;-)

Joins

It often is such that you can create struts in go that are a merge on one of more tables. These are sometimes called datasets, because they are a set of tables.

Then to fill the dataset, you can call the crud for each table manually doing the joins.

I did this before in c# and it worked OK.
I think we had a dB with almost 1,000 tables, and so this simple technique removed lots of boring coding.

It also properly separates for middler tier / client types from you database types.

The only real work is to write data transformation code that maps your database types to and from you datset types.

For many use cases of just managing simple things in the dB you don't need to use dataset types still. For example a user table. Then just the crud is fine.

The dataset types can be generated using protocol buffers or flat buffers. I prefer protocol buffers because it also generates the service calls also.

Then you have code generated a hell of alot of the boring code.

Just an idea !!

Problem with Multi-Column Primary Key in MySQL

In case when table has multi-column primary key (also if table hase only one unique key and it's multi-column mysql shows it as primary) only one of fields of this index is marked as primary and this breaks generated functions like updating record in database.

Columns should have type safety

I believe it should be a great feature if we can have struct tags generated for sql column names. Ex:

objs.Select("Id", "Age").Where(objs.FilterAge(">=", 10)).
OrderBy("Age", "+Id").Page(9, 5)

Should be:

objs.Select(User.Id, User.Age).Where(objs.FilterAge(">=", 10)).
OrderBy(User.Age, User.Id).Page(9, 5)

Or

objs.Select( _.Id, _.Age).Where(objs.FilterAge(">=", 10)).
OrderBy( _.Age, _.Id).Page(9, 5)**

This is very important when you're designing big projects and you're refactoring the database models. The compiler we'll help to identify the errors after code generation from the refactored database.

Consideration of Slick code base for some of the complex join tasks

Hello Mijia:

This code generator looks amazing at it's early stage, so please keep working and i hope it could be something powerful like the Scala Slick Code generator and Framework. I'm in my baby steps for GOLANG, so i'm learning a lot from your clever code. I would like to know if you can release the code with an MIT license and add it to the project, i'll keep watching :).

Rayner

dont-touch-timestamp=true is treated as CodeGen.TouchTimestamp=true

problematic line in question: https://github.com/mijia/modelq/blob/master/main.go#L28

Looks awesome, any plans to support PostgreSQL?

Firstly, modelq looks fantastic, exactly the right approach to reducing the leg work of working with an RDBMS in Go, thanks a lot. Were you inspired by this by any chance?

We're working with PostgreSQL. Any interest in supporting it? Any idea how much work would be involved in adding it if we were to try it ourselves?

modelq does not remove deprecated files

when tables are dropped modelq doesn't clean up the generated golang files. are there any plans to address this?

Support of Postgres < 8.2

Hi guys,

Thanks for the interesting project. Unfortunately, we have PostgreSQL version < 8.2, so we can't use position_in_unique_constraint.
modelq crashes with next error message:

2015/04/23 12:16:15 [Postgres Driver] Start to load tables schema from database, name, tables=tables
2015/04/23 12:16:16 Cannot load table schemas from database.
2015/04/23 12:16:16 pq: column keycolumnusage.position_in_unique_constraint does not exist

I found, that these are generated in https://github.com/mijia/modelq/blob/master/drivers/postgres/key_column_usage.go#L24 What do you think is the best way to adapt this to my case?

Regards,
Sergii

'json:' tags for fields should support camelCase as a config

Our API supports camelCase as a standard, so instead of json:first_name we support json:firstName. So it would be nice to use this framework, but it doesn't currently support our needs.

Prepared statements should close

Prepare statements are leaking whenever an UPDATE is made. The statements are getting closed for queries (q _Query) query(...)but not for any updates made (q _Query) exec(...) see modelq/gmq/package.go. The prepares get deallocated whenever the connection disappears, but in the case that the connection persists and many updates are being done via transactions, we will eventually hit the maximum number of prepared statements (which by default is 13682) and receive error 1461. This shouldn't happen. A workaround would be to increase the limit, but the preferred way would be to close the statement when it's finished.

https://dev.mysql.com/doc/refman/5.7/en/server-status-variables.html#statvar_Prepared_stmt_count

Support your own queries (for custom things like full text queries)

Well... We could do that easily if we had access to functionality like the "toType" function, the one that takes (columns []gmq.Column, rb []sql.RawBytes). It just does the hard work for you.

There has to be some way to do outlandish queries onto a type, having modelq trust you that you're returning the right columns to it. Even if it's just a blaobjs.Query("query here") that returns a _BlaQuery so you can start outputting results. It doesn't have to support any more Filters or anything.

Plans for aggregate functions?

Do you have ideas on how you want to do aggregate functions like MAX()? You mention them in the readme, but I don't see anything hinting what your ideas were on this. I'd be happy to take a whack at it regardless.

And ideas on how to do it with #5 in mind would be good.

Generalize template generation

Hi, first of all, nice job. I had the exact same idea for this :) Its great to be able to scaffold repositories for data manipulation, since its quite repetative task and prone to errors.
Just a tought - would it be possible to do something like adding a switch to CLI to use custom templates? I think you can use {{define XXX}} to declare fragments, so parsing one file with all the fragments needed and using them instead of deafults for code generation shouldn't be a overkill.
It would add a great flexibility to the lib. I can help of course ;)

Code generated from table with no primary key does not compile

Code generated from a table with no primary key does not compile:

db/schema/subnet.go:10: imported and not used: "fmt"
db/schema/subnet.go:31: cannot use 0 (type int) as type Subnet in return argument

The first error is that the fmt package is only used in a section of the template included if the table has a primary key.
The second error is that 0 is being returned as a dummy value instead of nil. Solution here is probably to not generate the Get method at all if it isn't going to work with no primary key.

time fields with default value CURRENT_TIMESTAMP(X) in sql are still written to with dont-touch-timestamp=True flag

codegen.go only checks for
strings.ToUpper(f.DefaultValue) == "CURRENT_TIMESTAMP" or
strings.ToUpper(f.DefaultValue) == "NOW()"
when deciding whether a time column has some default value. See https://github.com/mijia/modelq/blob/master/main.go#L28

However, MySql also allows default values like CURRENT_TIMESTAMP(X) and NOW(X) where X is a number. See http://dev.mysql.com/doc/refman/5.7/en/timestamp-initialization.html

I think code generator should check for matches against ^NOW(?:\\(\\d\\))?$" and "^CURRENT_TIMESTAMP(?:\\(\\d\\))?$"

is this project still active?

Queries should not silently fail when iterating over the result set

On line 131 in gmq/query.go , error checks aren't being done after iterating over the rows' result set. As a result, this can lead to silent failures if rows.Next() returns halfway through due to an intermittent error. The caller should be expected to distinguish between reaching the end of the set or an error occurring when preparing that next item. See https://golang.org/pkg/database/sql/#Rows.Next