fraenky8 / tables-to-go

convert your database tables to structs easily

Tables-to-Go

convert your database tables to structs easily

A small and convenient tool supporting development against a changing database schema.

Tables change, run the tool, get your structs!

Requirements

• Go 1.20+

Install

This project provides a make file but can also simply be installed with the go-install command:

go install github.com/fraenky8/tables-to-go@master

To enable SQLite3 support, clone the repo manually and run the make file:

make sqlite3

See this PR why it's disabled by default.

Getting Started

tables-to-go -v -of ../path/to/my/models

This gets all tables of a local running PostgreSQL database. Therefore, it uses the database postgres, schema public and user postgres with no password. Flag -v is verbose mode, -of is the output file path where the go files containing the structs will get created (default: current working directory).

Features

• convert your tables to structs
• table with name a_foo_bar will become file AFooBar.go with struct AFooBar
• properly formatted files with imports
• automatically typed struct fields, either with sql.Null* or primitive pointer types
• struct fields with db-tags for ready to use in database code
• partial support for Masterminds/structable
  • only primary key & auto increment columns supported
  • struct fields with stbl tags
  • ability to generate structs only for Masterminds/structable:
    • without db-tags
    • with or without structable.Recorder
• currently supported:
  • PostgreSQL (9.5 tested)
  • MySQL (5.5+, 8 tested)
  • SQLite (3 tested)
• currently, the following basic data types are supported:
  • numeric: integer, serial, double, real, float
  • character: varying, text, char, varchar, binary, varbinary, blob
  • date/time: timestamp, date, datetime, year, time with time zone, timestamp with time zone, time without time zone, timestamp without time zone
  • others: boolean

Examples

Assuming you have the following table definition (PostgreSQL):

CREATE TABLE some_user_info  (
  id SERIAL NOT NULL PRIMARY KEY,
  first_name VARCHAR(20),
  last_name  VARCHAR(20) NOT NULL,
  height DECIMAL
);

Run the following command (default local PostgreSQL instance):

tables-to-go

The following file SomeUserInfo.go with default package dto (data transfer object) will be created:

package dto

import (
	"database/sql"
)

type SomeUserInfo struct {
	ID        int             `db:"id"`
	FirstName sql.NullString  `db:"first_name"`
	LastName  string          `db:"last_name"`
	Height    sql.NullFloat64 `db:"height"`
}

The column id got automatically converted to upper-case to follow the idiomatic go guidelines. See here for more details. Words which gets converted can be found here.
This behaviour can be disabled by providing the command-line flag -no-initialism.

Running on remote database server (eg. Mysql@Docker)

tables-to-go -v -t mysql -h 192.168.99.100 -d testdb -u root -p mysecretpassword

PostgreSQL example with different default schema but default database postgres:

tables-to-go -v -t pg -h 192.168.99.100 -s test -u postgres -p mysecretpassword

Note: since database type pg is default, following command will be equivalent:

tables-to-go -v -h 192.168.99.100 -s test -u postgres -p mysecretpassword

You can also specify the package or prefix and suffix.

tables-to-go -v -t mysql -h 192.168.99.100 -d testdb -u root -p mysecretpassword -pn models -pre model_ -suf _model

With same table given above, following file with Name ModelSomeUserInfoModel.go will be created:

package models

import (
	"database/sql"
)

type ModelSomeUserInfoModel struct {
	ID        int             `db:"id"`
	FirstName sql.NullString  `db:"first_name"`
	LastName  string          `db:"last_name"`
	Height    sql.NullFloat64 `db:"height"`
}

Where Are The JSON-Tags?

This is a common question asked by contributors and bug reporters.

Fetching data from a database and representation of this data in the end (JSON, HTML template, cli, ...) are two different concerns and should be decoupled. Therefore, this tool will not generate json tags for the structs.

There are tools like gomodifytags which enables you to generate json tags for existing structs. The call for this tool applied to the example above looks like the following:

gomodifytags -file SomeUserInfo.go -w -all -add-tags json

This adds the json tags directly to the file:

type SomeUserInfo struct {
	ID        int             `db:"id" json:"id"`
	FirstName sql.NullString  `db:"first_name" json:"first_name"`
	LastName  string          `db:"last_name" json:"last_name"`
	Height    sql.NullFloat64 `db:"height" json:"height"`
}

Command-line Flags

Print usage with -? or -help

Usage of tables-to-go:
  -?	shows help and usage
  -d string
    	database name (default "postgres")
  -f	force; skip tables that encounter errors
  -fn-format string
    	format of the filename: camelCase (c, default) or snake_case (s) (default c)
  -format string
    	format of struct fields (columns): camelCase (c) or original (o) (default c)
  -h string
    	host of database (default "127.0.0.1")
  -help
    	shows help and usage
  -no-initialism
    	disable the conversion to upper-case words in column names
  -null string
    	representation of NULL columns: sql.Null* (sql) or primitive pointers (native|primitive) (default sql)
  -of string
    	output file path (default "current working directory")
  -p string
    	password of user
  -pn string
    	package name (default "dto")
  -port string
    	port of database host, if not specified, it will be the default ports for the supported databases
  -pre string
    	prefix for file- and struct names
  -s string
    	schema name (default "public")
  -socket string
    	The socket file to use for connection. Takes precedence over host:port.
  -structable-recorder
    	generate a structable.Recorder field
  -suf string
    	suffix for file- and struct names
  -t string
    	type of database to use, currently supported: [pg mysql sqlite3] (default pg)
  -tags-no-db
    	do not create db-tags
  -tags-structable
    	generate struct with tags for use in Masterminds/structable (https://github.com/Masterminds/structable)
  -tags-structable-only
    	generate struct with tags ONLY for use in Masterminds/structable (https://github.com/Masterminds/structable)
  -u string
    	user to connect to the database (default "postgres")
  -v	verbose output
  -vv
    	more verbose output

Contributing

If you find any issues or missing a feature, feel free to contribute or make suggestions! You can fork the repository and use a feature branch too. Feel free to send me a pull request. The PRs have to come with appropriate unit tests, documentation of the added functionality and updated README with optional examples.

To start developing clone via git or use go's get command to fetch this project.

This project uses go modules so make sure when adding new dependencies to update the go.mod file and the vendor directory:

go mod tidy
go mod vendor

Licensing

The code in this project is licensed under MIT license.

tables-to-go's Issues

add unit tests

Add unit tests!

creates duplicate struct-fields

Given this table

create table foo_has_bar
(
  foo_id integer not null
    constraint foo_has_bar_foo_id _fk
    references foo (id),
  bar_id integer not null
    constraint foo_has_bar_bar_id_fk
    references bar (id),
  constraint foo_has_bar_foo_id_bar_id_pk
  primary key (foo_id, bar_id)
);

the tool produces the following struct

type FooHasBar struct {
	FooId int `db:"FooId"`
	FooId int `db:"FooId"`
	BarId  int `db:"BarId"`
	BarId  int `db:"BarId"`
}

therefore go reports errors of duplicate fields.

MySQL 8 has UPPERCASE columns in information_schema

Hi,

Mysql 8 has UPPERCASE columns in information_schema.tables and information_schema.columns therefore it will trigger errors:
could not get tables: missing destination name TABLE_NAME in *[]*database.Table
TABLE_NAME should be lowercase disregarding that in the query is lowercase select table_name ...

However the fix is pretty easy by adding select table_name as 'table_name' from... in the queries in mysql.go like this:
SELECT table_name as 'table_name' FROM information_schema.tables WHERE table_type = 'BASE TABLE' AND table_schema = ? ORDER BY table_name
and
SELECT ordinal_position as 'ordinal_position', column_name as 'column_name', data_type as 'data_type', column_default as 'column_default', is_nullable as 'is_nullable', character_maximum_length as 'character_maximum_length', numeric_precision as 'numeric_precision', column_key as 'column_key', extra as 'extra' FROM information_schema.columns WHERE table_name = ? AND table_schema = ? ORDER BY ordinal_position

Thank you

Required versus optional value mappings

• https://s.natalian.org/2019-03-31/nullstring.mp4

If a schema is say just age INT I think you set the type sql.NullInt64. But this generates gnarly JSON mappings like

    "Age": {
        "Int64": 0,
        "Valid": true
    }

My workaround is to then edit the generated type to for the optional value:

Age  int     `db:"age" json:"age,omitempty"`

However I've noticed that go sets 0 (ZERO) in my JSON to DB mapping with this "workaround".

INSERT INTO `users` (`id`, `name`, `age`) VALUES
(1,	'Arnie',	0);

If it was the original sql.NullInt64 the value in DB would be:

(2,	'Arnie 2',	NULL);

Which is more correct in terms of the DBMS, but then how does one handle the JSON ugliness?!

I also have a type issue when it comes to required values (name): https://s.natalian.org/2019-03-31/required.mp4

database/sql import not being set on models

Generating models for a medium sized database and half the table models end up not getting decorated with database/sql. I can't see any reason in particular, most tables are simple. Some have a single import while others also import "time" or "github.com/go-sql-driver/mysql".

Using latest version installable by go get github.com/fraenky8/tables-to-go as of this morning, 9/21/2021.

Version: latest
Datbase: MariaDB - 10.2.9-MariaDB-log

Examples:

---

package database

import (
"time"
)

type ActionAuditLog struct {
Type string db:"Type"
User string db:"User"
Details sql.NullString db:"Details"
Time time.Time db:"Time"
}

---

package database

import (
"github.com/go-sql-driver/mysql"
)

type MonitorSnapshot struct {
SystemID string db:"SystemId"
Name string db:"Name"
Time mysql.NullTime db:"Time"
Data sql.NullString db:"Data"
}

---

package database

type SeriesData struct {
SeriesID int db:"SeriesId"
StartTime int db:"StartTime"
EndTime int db:"EndTime"
Value sql.NullString db:"Value"
Weight sql.NullString db:"Weight"
}

Where is the SQLite makefile?

Sorry if this has been answered before but I'm a little confused, I ran the go get command and it installed the binary into my $PATH, but I can't find the source or anything, so idk where to run the Makefile for SQLite3 from.

Thanks.

can you support sqlite?

i already use tabbles-to-go with myql, so thanks ,
, now i want use it with sqlite ,can you support it?

https://github.com/mattn/go-sqlite3

MySQL 5.5: run error could not prepare the get-column-statement: Error 1054: Unknown column 'datetime_precision' in 'field list'%

In MySQL 5.5, tables-to-go run failed because of datetime_precision is not exists.

running for "mysql"...
> number of tables: 513
run error could not prepare the get-column-statement: Error 1054: Unknown column 'datetime_precision' in 'field list'%

datetime_precision option is available from MySQL 5.6.4. https://dev.mysql.com/doc/refman/5.6/en/fractional-seconds.html

In tables-to-go, datetime_precision is not used yet (or, any plan to use it?). For supporting MySQL under 5.6, it can be fixed by removing from PrepareGetColumnsOfTableStmt.

// PrepareGetColumnsOfTableStmt prepares the statement for retrieving the columns of a specific table for a given database
func (mysql *mysql) PrepareGetColumnsOfTableStmt() (err error) {

	mysql.getColumnsOfTableStmt, err = mysql.db.Preparex(`
		SELECT
		  ordinal_position,
		  column_name,
		  data_type,
		  column_default,
		  is_nullable,
		  character_maximum_length,
		  numeric_precision,
		  datetime_precision,
		  column_key,
		  extra
		FROM information_schema.columns
		WHERE table_name = ?
		AND table_schema = ?
		ORDER BY ordinal_position
	`)

	return err
}

generate json tag [feature request]

add option to generate json tag

Error if column names start with digits

If the column names of a table start with digits, the tool fails to create the struct.

> processing table "MyTable"
        > number of columns: 4
                > id
                > name
                > 1person
                > 2person

run error: could not create struct file for table MyTable: could not format file /home/ronnie/project/models/MyTable.go: 12:1: expected '}', found 1% 

I'm not sure what the best thing to do here would be. I know it's bad practice to have SQL column names start with numbers but it's technically allowed. So we may have to deal with it if we don't have authority to change the actual database schema.

But Golang obviously doesn't allow fieldnames to start with numbers, so what would be a sensible thing to do?

• Prefix them with a letter? x1person
• Swap the digits to the end? person1

This may open up a whole can of worms because MySQL table and column names can be full unicode but Go only allows letters, underscore, and digits, and each identifier has to start with a letter or underscore.

Then again, it is really absurd to use non-letter characters in table and column names, so it would probably be fair enough to just not deal with them.

Generate structs of database views

This is a kind feature request :)

Is there any way of generate structs also of database views?

Thank you.

support unix socket connection

I always use unix socket and disable network in my local machine. it'd be great if this tool support to connect to mysql trough unix socket 🙂

How to generate struct with gureu/null?

Is there a plan to support the use of gureu/null?

mysql type of database is still generating with pg types

https://github.com/unee-t/tables-to-go/blob/bug1/tests/Persons.go#L14 is such an example

Please see:

• https://github.com/unee-t/tables-to-go/blob/bug1/tests/Makefile
• https://github.com/unee-t/tables-to-go/blob/bug1/tests/docker-compose.yml

This is how I gen the output.

go:build comment without // +build comment

I run go get,and it report this error:
internal/cli/tables-to-go-cli.go:12:2: //go:build comment without // +build comment

Create structs from string instead of live database connection

First off thanks for the great repo, I'm sure this would be of use to many people.

I wonder if it's possible to get the structs from a SQL definition instead of connecting to the database? I'm thinking it would accept the create table SQL either through a file, stdin or a flag. This would allow a nice web UI like https://mholt.github.io/json-to-go/

Id should be all caps

Noticed it generates:

Id   int           `db:"id"`

It should be

ID   int           `db:"id"`

For idiomatic Go IIUC

Force NULL for all columns?

Hi, possibly a weird feature request, but would it be reasonable to add a -force-null flag to force all fields to appear NULLable?

It seems to be the norm where I work to use sql.Null* for everything, including for columns that are NOT NULL. There are some arguments I could give for and against this practice, but in general it seems like it'd be a roadblock (at least at my shop) to adoption of an otherwise perfect developer tool.

I'm guessing it'd be simple enough implementation-wise that I could write a PR for this, but I wanted to test the waters first

give an option to choose beteween sql.Null* and pointers for NULL columns

Implement one more command line flag to determine which data type to use for NULL columns:

• sql.Null* (current implementation)
• pointer to primitves

db struct tags not mapping to column names

For example the gen result https://github.com/unee-t/tables-to-go/blob/bug1/tests/Persons.go#L10 is:

IdPerson             int            `db:"IdPerson"`

I expected it to be

IdPerson                int            `db:"id_person"`

Better still would (probably) be:

ID                int            `db:"id_person"`

I'm aware of -format o but I don't want my struct names to have underscores in them! 😢

Please see:

• https://github.com/unee-t/tables-to-go/blob/bug1/tests/Makefile
• https://github.com/unee-t/tables-to-go/blob/bug1/tests/docker-compose.yml

This is how I gen the output.

Add an option to continue through errors

Currently the tool stops on the first table that it finds an error on.

It would be nice if there was a flag to continue so we can get structs for all tables that we can, and then we can just examine and manually construct the tables that wouldn't work.

tags getting title'ized instead of preserving the original column name

given table:

CREATE TABLE test (
  id integer PRIMARY KEY AUTO_INCREMENT,
  name varchar(100) NOT NULL
)

expected output:

type Test struct {
	Id   int         `db:"id"`
	Name string      `db:"name"`
}

actual output:

type Test struct {
	Id   int         `db:"Id"`
	Name string      `db:"Name"`
}

The db-tags Id and Name start with upper letter instead of original column name.

How to connect to Postgres/CockroachDB without SSL mode?!

Hi

I'm trying to connect to CockroachDB which uses the same connection DSN string as Postgres, but cannot find any arg parameter to disable the SSL mode! I'm receiving this error message:

pq: SSL is not enabled on the server

NB: I checked the source code, file "pkg/database/postgresql.go", function DSN(), line: 47. The sslmode already set to "disable" which means it should connect without SSL! Is this a bug?!

Any hints please?

Not null enum columns return pointers

Hey!

In postgres if I have a column as:

+--------------------------+--------------------------+-----------+
| Column                   | Type                     | Modifiers |
|--------------------------+--------------------------+-----------|
| rule_severity            | severity                 |  not null |

where severity is a type CREATE TYPE severity AS ENUM ('low', 'medium', 'high', 'critical');

tables-to-go will generate: RuleSeverity *string

I was expecting RuleSeverity string since we have a not null column.

I will see if I can have a PR for this (if this is simple!).

Specify table(s) and generated filename

Hi, thanks for great repo.

I wonder if it is possible if we can specify which table(s) that we'll be convert into struct.
maybe using flag table ?
and also option to define the generated filename, such as model.go

example 1: tables-to-go -d booking_app -table invoice -of /internal/domain/invoice -o model
--> will create file /internal/domain/invoice/model.go

example 2: tables-to-go -d booking_app -table order -of /internal/domain/order -o model
--> will create file /internal/domain/order/model.go

I think this would allow us to be more selective on generating the struct and saving it into each domain folder with specified filename.