GithubHelp home page GithubHelp logo

jmoiron / sqlx Goto Github PK

View Code? Open in Web Editor NEW
15.5K 197.0 1.1K 866 KB

general purpose extensions to golang's database/sql

Home Page: http://jmoiron.github.io/sqlx/

License: MIT License

Go 99.74% Makefile 0.26%

sqlx's Introduction

sqlx

CircleCI Coverage Status Godoc license

sqlx is a library which provides a set of extensions on go's standard database/sql library. The sqlx versions of sql.DB, sql.TX, sql.Stmt, et al. all leave the underlying interfaces untouched, so that their interfaces are a superset on the standard ones. This makes it relatively painless to integrate existing codebases using database/sql with sqlx.

Major additional concepts are:

  • Marshal rows into structs (with embedded struct support), maps, and slices
  • Named parameter support including prepared statements
  • Get and Select to go quickly from query to struct/slice

In addition to the godoc API documentation, there is also some user documentation that explains how to use database/sql along with sqlx.

Recent Changes

1.3.0:

  • sqlx.DB.Connx(context.Context) *sqlx.Conn
  • sqlx.BindDriver(driverName, bindType)
  • support for []map[string]interface{} to do "batch" insertions
  • allocation & perf improvements for sqlx.In

DB.Connx returns an sqlx.Conn, which is an sql.Conn-alike consistent with sqlx's wrapping of other types.

BindDriver allows users to control the bindvars that sqlx will use for drivers, and add new drivers at runtime. This results in a very slight performance hit when resolving the driver into a bind type (~40ns per call), but it allows users to specify what bindtype their driver uses even when sqlx has not been updated to know about it by default.

Backwards Compatibility

Compatibility with the most recent two versions of Go is a requirement for any new changes. Compatibility beyond that is not guaranteed.

Versioning is done with Go modules. Breaking changes (eg. removing deprecated API) will get major version number bumps.

install

go get github.com/jmoiron/sqlx

issues

Row headers can be ambiguous (SELECT 1 AS a, 2 AS a), and the result of Columns() does not fully qualify column names in queries like:

SELECT a.id, a.name, b.id, b.name FROM foos AS a JOIN foos AS b ON a.parent = b.id;

making a struct or map destination ambiguous. Use AS in your queries to give columns distinct names, rows.Scan to scan them manually, or SliceScan to get a slice of results.

usage

Below is an example which shows some common use cases for sqlx. Check sqlx_test.go for more usage.

package main

import (
    "database/sql"
    "fmt"
    "log"
    
    _ "github.com/lib/pq"
    "github.com/jmoiron/sqlx"
)

var schema = `
CREATE TABLE person (
    first_name text,
    last_name text,
    email text
);

CREATE TABLE place (
    country text,
    city text NULL,
    telcode integer
)`

type Person struct {
    FirstName string `db:"first_name"`
    LastName  string `db:"last_name"`
    Email     string
}

type Place struct {
    Country string
    City    sql.NullString
    TelCode int
}

func main() {
    // this Pings the database trying to connect
    // use sqlx.Open() for sql.Open() semantics
    db, err := sqlx.Connect("postgres", "user=foo dbname=bar sslmode=disable")
    if err != nil {
        log.Fatalln(err)
    }

    // exec the schema or fail; multi-statement Exec behavior varies between
    // database drivers;  pq will exec them all, sqlite3 won't, ymmv
    db.MustExec(schema)
    
    tx := db.MustBegin()
    tx.MustExec("INSERT INTO person (first_name, last_name, email) VALUES ($1, $2, $3)", "Jason", "Moiron", "[email protected]")
    tx.MustExec("INSERT INTO person (first_name, last_name, email) VALUES ($1, $2, $3)", "John", "Doe", "[email protected]")
    tx.MustExec("INSERT INTO place (country, city, telcode) VALUES ($1, $2, $3)", "United States", "New York", "1")
    tx.MustExec("INSERT INTO place (country, telcode) VALUES ($1, $2)", "Hong Kong", "852")
    tx.MustExec("INSERT INTO place (country, telcode) VALUES ($1, $2)", "Singapore", "65")
    // Named queries can use structs, so if you have an existing struct (i.e. person := &Person{}) that you have populated, you can pass it in as &person
    tx.NamedExec("INSERT INTO person (first_name, last_name, email) VALUES (:first_name, :last_name, :email)", &Person{"Jane", "Citizen", "[email protected]"})
    tx.Commit()

    // Query the database, storing results in a []Person (wrapped in []interface{})
    people := []Person{}
    db.Select(&people, "SELECT * FROM person ORDER BY first_name ASC")
    jason, john := people[0], people[1]

    fmt.Printf("%#v\n%#v", jason, john)
    // Person{FirstName:"Jason", LastName:"Moiron", Email:"[email protected]"}
    // Person{FirstName:"John", LastName:"Doe", Email:"[email protected]"}

    // You can also get a single result, a la QueryRow
    jason = Person{}
    err = db.Get(&jason, "SELECT * FROM person WHERE first_name=$1", "Jason")
    fmt.Printf("%#v\n", jason)
    // Person{FirstName:"Jason", LastName:"Moiron", Email:"[email protected]"}

    // if you have null fields and use SELECT *, you must use sql.Null* in your struct
    places := []Place{}
    err = db.Select(&places, "SELECT * FROM place ORDER BY telcode ASC")
    if err != nil {
        fmt.Println(err)
        return
    }
    usa, singsing, honkers := places[0], places[1], places[2]
    
    fmt.Printf("%#v\n%#v\n%#v\n", usa, singsing, honkers)
    // Place{Country:"United States", City:sql.NullString{String:"New York", Valid:true}, TelCode:1}
    // Place{Country:"Singapore", City:sql.NullString{String:"", Valid:false}, TelCode:65}
    // Place{Country:"Hong Kong", City:sql.NullString{String:"", Valid:false}, TelCode:852}

    // Loop through rows using only one struct
    place := Place{}
    rows, err := db.Queryx("SELECT * FROM place")
    for rows.Next() {
        err := rows.StructScan(&place)
        if err != nil {
            log.Fatalln(err)
        } 
        fmt.Printf("%#v\n", place)
    }
    // Place{Country:"United States", City:sql.NullString{String:"New York", Valid:true}, TelCode:1}
    // Place{Country:"Hong Kong", City:sql.NullString{String:"", Valid:false}, TelCode:852}
    // Place{Country:"Singapore", City:sql.NullString{String:"", Valid:false}, TelCode:65}

    // Named queries, using `:name` as the bindvar.  Automatic bindvar support
    // which takes into account the dbtype based on the driverName on sqlx.Open/Connect
    _, err = db.NamedExec(`INSERT INTO person (first_name,last_name,email) VALUES (:first,:last,:email)`, 
        map[string]interface{}{
            "first": "Bin",
            "last": "Smuth",
            "email": "[email protected]",
    })

    // Selects Mr. Smith from the database
    rows, err = db.NamedQuery(`SELECT * FROM person WHERE first_name=:fn`, map[string]interface{}{"fn": "Bin"})

    // Named queries can also use structs.  Their bind names follow the same rules
    // as the name -> db mapping, so struct fields are lowercased and the `db` tag
    // is taken into consideration.
    rows, err = db.NamedQuery(`SELECT * FROM person WHERE first_name=:first_name`, jason)
    
    
    // batch insert
    
    // batch insert with structs
    personStructs := []Person{
        {FirstName: "Ardie", LastName: "Savea", Email: "[email protected]"},
        {FirstName: "Sonny Bill", LastName: "Williams", Email: "[email protected]"},
        {FirstName: "Ngani", LastName: "Laumape", Email: "[email protected]"},
    }

    _, err = db.NamedExec(`INSERT INTO person (first_name, last_name, email)
        VALUES (:first_name, :last_name, :email)`, personStructs)

    // batch insert with maps
    personMaps := []map[string]interface{}{
        {"first_name": "Ardie", "last_name": "Savea", "email": "[email protected]"},
        {"first_name": "Sonny Bill", "last_name": "Williams", "email": "[email protected]"},
        {"first_name": "Ngani", "last_name": "Laumape", "email": "[email protected]"},
    }

    _, err = db.NamedExec(`INSERT INTO person (first_name, last_name, email)
        VALUES (:first_name, :last_name, :email)`, personMaps)
}

sqlx's People

Contributors

a13xb avatar asheikm avatar c4milo avatar cmars avatar cmillward avatar dlsniper avatar dominikh avatar flimzy avatar hmgle avatar j-zeitler avatar jeetjitsu avatar jmoiron avatar kisielk avatar liaoliaopro avatar loafoe avatar mantenie avatar markuswustenberg avatar npiganeau avatar nussjustin avatar ojrac avatar quangtung97 avatar rayfenwindspear avatar shanemhansen avatar suzuki-shunsuke avatar taik avatar thedevsaddam avatar tockn avatar troyk avatar tt avatar wyattjoh avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

sqlx's Issues

Combine features of NamedQuery and Get

What's nice about the Get method is that you can auto-populate a struct.

What's nice about the NamedQuery method is that you can use a struct to build named queries.

It would be great to have, e.g., "GetNamedQuery", which lets me populate a struct by querying, using a struct to populate my query.

StructScan Unmarshalling Behaviour: no fields in dest. interface

Is there a way to change the behaviour (default or optional) of StructScan to match that of Go's json.Unmarshal where any fields in the destination struct that don't exist in the source are implicitly skipped?

Example: I have a tsv field that contains tsvector data in a postgres DB for full text search. I can avoid having a matching field in the destination struct if I SELECT <list of columns not including tsv however it's pretty verbose once you have a few columns and it'd be easier to just SELECT * FROM <table> and drop the columns not in the dest struct.

Embedded struct support not backward compat with time.Time

I use the MySql driver from go-sql-driver with the parseTime option. This lets me define fields in structs as time.Time and the driver gets the data loaded properly. Example struct:

type Session struct {
    Id     string
    UserId uint64
    Expire time.Time
}

I have been using sqlx for a few months with no trouble. I recently updated to the latest version of sqlx that has the embedded struct support and now this type of setup no longer works. I suspect it is because time.Time implements scanner and causes sqlx to think it is an embedded struct (which of course it is). In the end I get an error like "Could not find name expire in interface"

Trying to figure out how to properly model this and get it working in conjunction with my mysql driver. Any help would be appreciated.

Get() not closing sql.Rows

While debugging a time.Time mapping issue #25 I noticed that my rollback was not working, mysql was giving me a Commands out of sync. Did you run multiple statements at once? error.
Adding defer r.rows.Close() to the sqlx.Get() function solved that issue. Not sure if there are other places not closing rows, but that one was causing my problem.

sqlx Get cannot fetsh array

hi, sqlx.Get fails on returning array of objects complaining that it sholud be a struct.
call is as follows:
ur.Get(&users, ur.Rebind("select * from "user" where username=?"), name);
and error message is:
destination must be struct

am i missing something? i need to do parametrized query. am i doing it wrong?

StructScan doesn't work with embedded pointer fields.

sqlx supports embedded structs, but pointers to structs are not followed. This is a bug, because it's contrary to reasonable expectations, ie:

type Foo struct {
    Name string
}

// can set 'Name' on this struct
type Place struct {
    Foo
}

// cannot set 'Name' on this struct
type Person struct {
    *Foo
}

Currently, when you scan into a nil pointer, database/sql allocates the space for you. This is how []byte and string work as well; a new one will always be allocated. I'm going to copy this behavior, even though it's not well known, because I don't want StructScan to behave differently and because I think it could end up creating incredibly difficult to figure out bugs.

The setValues function can allocate these using reflect.

Note that all this per-row allocation is going to cost a little something, so if speed is a priority I recommend against pointer embeds.

Query Logging

@jmoiron I'm thinking of writing a query logging component for sqlx, but first I want to see how it jives with you.

It would be useful having it as an optional setting (off by default) because:

  • Not everyone has access to a remote database and its log files.
  • For short running CLI utilities (eg: migration scripts), sysadmins love to see a dump of everything a program is doing.
  • It can streamline development of web services by throwing accidental N+1 queries right in your face.
  • If there is also an optional bool setting, a production system can be asked to start dumping queries to the log on a server or per-request level without restarting the service.

Thoughts?

NamedQuery causes panic for embedded struct

I spoke to @jmoiron briefly about this on Friday, and have put together a simple example to show problem.

When using NamedQuery with an embedded struct, Go panics:

go run main.go
classic OK - id = 16
panic: reflect: Field index out of range

goroutine 1 [running]:
reflect.Value.Field(0x1cef60, 0xc2000796c0, 0x192, 0x1, 0x1, ...)
    /usr/local/share/go/src/pkg/reflect/value.go:785 +0x164
github.com/jmoiron/sqlx.BindStruct(0x2, 0x25f770, 0x56, 0x1cef60, 0xc2000796c0, ...)
    <snip>/src/github.com/jmoiron/sqlx/bind.go:84 +0x4d9
github.com/jmoiron/sqlx.(*Tx).BindStruct(0xc200095700, 0x25f770, 0x56, 0x1cef60, 0xc2000796c0, ...)
    <snip>/src/github.com/jmoiron/sqlx/sqlx.go:298 +0x71
github.com/jmoiron/sqlx.NamedQuery(0xc2000b5300, 0xc200095700, 0x25f770, 0x56, 0x1cef60, ...)
    <snip>/src/github.com/jmoiron/sqlx/sqlx.go:1021 +0x61
github.com/jmoiron/sqlx.(*Tx).NamedQuery(0xc200095700, 0x25f770, 0x56, 0x1cef60, 0xc2000796c0, ...)
    <snip>/src/github.com/jmoiron/sqlx/sqlx.go:303 +0x67
main.InsertNamed(0x0, 0x16, 0xc2000b3150, 0x1)
    <snip>/src/scratch/sqlx_bug/main.go:49 +0xfc
main.main()
    <snip>/src/scratch/sqlx_bug/main.go:78 +0x16d

goroutine 2 [syscall]:
exit status 2

Reproduce Issue

I can reproduce the issue by creating a PostgreSQL table according to the following schema:

CREATE TABLE "public"."table" (
    "id" SERIAL PRIMARY KEY,
    "field" int4
);

And then using the following Go program:

package main

import (
    "fmt"
    "github.com/jmoiron/sqlx"
    _ "github.com/lib/pq"
)

type T struct {
    Id    int
    Field int `db:"field"`
}

type Container struct {
    T
}

func InsertClassic(c Container, db *sqlx.DB) (id int) {
    stmt := `INSERT into "table" ("field")
             VALUES ($1)
             RETURNING "id"`

    tx, err := db.Begin()
    if err != nil {
        panic(err)
    }

    if err = tx.QueryRow(stmt, c.Field).Scan(&id); err != nil {
        panic(err)
    }

    if err = tx.Commit(); err != nil {
        panic(err)
    }
    return
}

func InsertNamed(c Container, db *sqlx.DB) (id int) {
    stmt := `INSERT into "table" ("field")
             VALUES (:field)
             RETURNING "id"`

    tx, err := db.Beginx()
    if err != nil {
        panic(err)
    }

    var rows *sqlx.Rows
    if rows, err = tx.NamedQuery(stmt, c); err != nil {
        panic(err)
    }
    defer rows.Close()

    for rows.Next() {
        if err = rows.Scan(&id); err != nil {
            panic(err)
        }
    }

    if err = rows.Err(); err != nil {
        panic(err)
    }

    if err = tx.Commit(); err != nil {
        panic(err)
    }
    return
}

func main() {
    db, err := sqlx.Connect("postgres", "dbname=postgres sslmode=disable port=5432")
    if err != nil {
        panic(err)
    }

    c := Container{T{Field: 22}}
    fmt.Printf("classic OK - id = %v\n", InsertClassic(c, db))
    fmt.Println("named OK - id = %v\n", InsertNamed(c, db))
}

The database/sql approach of using PG parameters works fine, but using sqlx's NamedQuery results in a panic.

I believe that the issue is stemming from the fact sqlx.getFieldMap maps the fields in the embedded struct T to indexes as follows:

map[string]int{
    "id": 0, 
    "field": 1,
}

, while the application of that mapping, via reflect.ValueOf(arg), where arg is the Container struct c, results in the first index of the map (which should map to c.T.Id) actually mapping to c.T. From there it's straightforward to see that there is no field available at the index 1 in the struct.

So, it appears that the implementation in sqlx.BindStruct does not make the right calls to the reflect package to get fields in the embedded field T, but instead gets the field(s) in the Container struct.

Checking if a result/result set is nil

Is there a simple way to do this? It'd be nice to be able to delineate between driver errors (no relation, column doesn't exist, etc) and a "no rows in result set" error without having to do string comparison.

e.g. A DB error will pass an error that will throw a HTTP 500 to the user and log it; a record that doesn't will just through a HTTP 404.

Something like the below:

        l := Listing{}
        l.Id = "1c9ea98d8ca5DDDD2631" // This doesn't exist
        row := db.QueryRowx("SELECT * FROM listings WHERE id = $1", l.Id)
        if row.IsNil() {
            return false, nil
        }
        err = row.StructScan(&l)
        if err != nil {
            return false, err
        }

Name mapping strategy

Would be nice to allow a pluggable strategy, or at least options among a few common types, to control the mapping of struct field name to db field name. Sqlx currently tries lowercase, I use camel case and so changed this line in getFieldMap from

name = strings.ToLower(f.Name)

to

name = strings.ToLower(f.Name[:1]) + f.Name[1:]

An option is preferable to having to declare db attributes on all the struct fields

Could not find name `col_name` in interface

For anyone getting the error message in the title.

Try changing the struct key names before continuing your search. It may help you save a lot of time.

Good - struct
type Item struct {
    Id      string `db:"_id"`
    Name    string `db:"item_name"`
    BrandId string `db:"brand_id"`
}
BAD!! - struct
type Item struct {
    _id       string
    item_name string
    brand_id  string
}

Logging interface

Do you think it could be useful to use log.Logger and allow the ability to set a custom logger instead of using log.Println? Execv and friends are nice, but it would be great if those logs could be sent somewhere other than STDOUT.

Error in documentation (http://jmoiron.github.io/sqlx/) Alternate Scan Types

When i try the second alternate scan type i get an error. I created a test program directly from your documentation. Am i doing something wrong?

package main

import (
    "github.com/jmoiron/sqlx"
)

func main () {
    rows, err := db.Queryx("SELECT * FROM place")
    for rows.Next() {
        results := map[string]interface{}
        err = rows.MapScan(results)
    }
}

I get the following error:
./test.go:10: syntax error: unexpected semicolon or newline, expecting {

Error while doing a MapScan or SliceScan time.Time with PostgreSQL

Not really sure if it's a sqlx issue. When i use a MapScan or a SliceScan in combination with a Postgresql timestamp field it will result in a scan error, or did i miss something?

rows, err := db.NamedQuery(query, where)
[...] -> err
for rows.Next() {
        results := make(map[string]interface{})
        err = rows.MapScan(results)
        [...] -> err

        fmt.Println(fmt.Sprintf("%v", results)))
}

Error:

sql: Scan error on column index 3: unsupported driver -> Scan pair: time.Time -> *string

Database Postgresql column:

\d users
  Column  |            Type             |                     Modifiers
----------+-----------------------------+----------------------------------------------------
 id       | integer                     | not null default nextval('users_id_seq'::regclass)
 added    | timestamp without time zone | not null default now()

sqlx.DB makes a copy of sql.DB instead of using a pointer

I am getting data races with sqlx because it takes a copy of sql.DB and therefore the mutexes are copies as well. This manifests itself when using sqlx.NewDb multiple times on the same sql.DB (this happens because of different libraries using sqlx but their interfaces pass around sql.DB).

changing

type DB struct {
        sql.DB
        driverName string
}

func NewDb(db *sql.DB, driverName string) *DB {
        return &DB{*db, driverName}
}

to

type DB struct {
    *sql.DB
    driverName string
}

func NewDb(db *sql.DB, driverName string) *DB {
    return &DB{db, driverName}
}

fixes this issue.

Also have to change func Open(driverName, dataSourceName string) (*DB, error)

See https://groups.google.com/forum/#!topic/golang-nuts/C4Cmjv6owCs

Numerical Identifiers

Right now it looks like binding/scanning names with numerical values doesn't work. I'm just guessing ... but there's a lot of unicode.IsLetter() usage in bind.go could that be the issue?

For example this does not work...

type Address struct {
   Address0 string `db:"address_0"`
   Address1 string `db:"address_1"`
}

q := "select * from addresses where address_0 = :address_0"
sqlx.BindStruct(q, &Address{})

It attempts to find an arg called address_ without the 0

Feature request: Select into map[?]struct{} where ? is type of first field

Sqlx is a nice time saver. Thanks.

It is quite handy to have sql results indexed by key. I was wanting to do a select into a map[key]struct{} instead of an []struct{}. I was thinking you could keep the existing Select interface but if the dest is a map and the type matched the type of the first field name treat it as the primary key and insert data into the map indexed by it.

I started looking at the code and it looks like StructScan does all the work. I was thinking if BaseSliceType return err, then do a test on a similar function BaseMapType then instead of doing appends in the loop, insert into the map instead.

I might have a hack on it on the weekend. But does this seem reasonable?

Strange behavior using NamedExec with a non-pointer, mysql, and sql.Null*

So here is the test code that is failing for me:

package main

import (
    "database/sql"
    "fmt"
    _ "github.com/go-sql-driver/mysql"
    "github.com/jmoiron/sqlx"
    // _ "github.com/mattn/go-sqlite3"
    "log"
)

var schema = `
CREATE TABLE person (
    first_name text NULL,
    last_name text NULL,
    email text NULL
);
`

type Person struct {
    FirstName sql.NullString `db:"first_name"`
    LastName  sql.NullString `db:"last_name"`
    Email     sql.NullString
}

func main() {
    //db, err := sqlx.Connect("sqlite3", "/tmp/sqlxbugtest.db")
    db, err := sqlx.Connect("mysql", fmt.Sprintf("%s:%s@/%s?parseTime=true", "sqlx", "sqlx", "sqlxtest"))
    if err != nil {
        log.Fatal(err)
    }

    db.Execl("drop table person")
    db.Execf(schema)

    p := Person{
        FirstName: sql.NullString{"ben", true},
        LastName:  sql.NullString{"doe", true},
        Email:     sql.NullString{"[email protected]", true},
    }
    _, err = db.NamedExec("INSERT INTO person (first_name, last_name, email) VALUES (:first_name, :last_name, :email)", p)
    if err != nil {
        log.Fatal(err)
    }

    p2 := &Person{}
    rows, err := db.NamedQuery("SELECT * FROM person WHERE first_name=:first_name", p)
    if err != nil {
        log.Fatal(err)
    }
    for rows.Next() {
        err = rows.StructScan(p2)
        if err != nil {
            log.Fatal(err)
        }
        if p2.FirstName.String != "ben" {
            log.Fatal("Expected first name of `ben`, got " + p2.FirstName.String)
        }
        if p2.LastName.String != "doe" {
            log.Fatal("Expected first name of `doe`, got " + p2.LastName.String)
        }
    }
}

Here is the output:

2014/01/03 22:42:18 Expected first name of `doe`, got 1
exit status 1

This is what the mysql database looks like:

first_name, last_name, email
'ben', '1', NULL

So basically, I can get this to work by doing 1 of any of the following things:

  • Change p := Person {... to p := &Person {...
  • Change all sql.NullString to string
  • Use sqlite3 instead of mysql

I'm not entirely sure why this is failing, but I found this in my production code.

Can not build with -race

go build -race main.go returns

src/github.com/jmoiron/sqlx/sqlx.go:350: cannot take the address of runtime.assertE2T(runtime.typΒ·2 = &type."".Stmt, runtime.ifaceΒ·3 = stmt).Stmt

on projects using sqlx.

Golang Sqlx not throwing error

I am using sqlx in golang, which is very helpful, but it does not seem to throw errors when I use struct scan and the types of the struct dont match the sql types. For example here I set up a database to have a name (string) and age(int):

+-------+--------------+------+-----+---------+-------+
| Field | Type         | Null | Key | Default | Extra |
+-------+--------------+------+-----+---------+-------+
| name  | varchar(255) | NO   |     | NULL    |       |
| age   | int(11)      | NO   |     | NULL    |       |
+-------+--------------+------+-----+---------+-------+
+------+-----+
| name | age |
+------+-----+
| bob  |  10 |
+------+-----+

I then use sqlx to read out the values into a struct, but the struct has the wrong types.

package main

import (
  "log"
  "github.com/jmoiron/sqlx"
  _ "github.com/go-sql-driver/mysql"
)

// in database name is a string and age is an int

type Person struct{
  Name int
  Age string
}

func main() {
  sqlSession, err := sqlx.Open("mysql", "root:@(localhost:3306)/dashboard?parseTime=true")
  if err != nil {
    log.Panic(err)
  }
  err = sqlSession.Ping()
  if err != nil {
    log.Panic(err)
  }
  query := "SELECT * FROM test"
  rows, errSql := sqlSession.Queryx(query)
  if errSql != nil {
    log.Panic(errSql)
  }
  for rows.Next() {
    var p Person
    errScan := rows.StructScan(&p)
    if errScan != nil {
      log.Panic(errScan)
    }
    log.Println("Person:", p)
  }
}

So instead of giving me an error, it has zeroed out values. Person: {0 }
I feel this is a bug of some kind.

P.S. sqlx is very useful, thank you for writing it

Panic on something innocent looking

Hi @jmoiron I was around in #go-nuts if you want to talk 1:1. I'm doing something innocuous enough looking, but it's causing a panic in sqlx at 41035eb.

I have a struct:

type User struct {
    Uuid               string    `json:"uuid"`
    Email              string    `json:"email"`
    Name               string    `json:"name"`
    Password           string    `json:"password"`
    CreatedAt          time.Time `json:"createdAt"          db:"created_at"`
    TotpSecret         string    `json:"totpSecret"         db:"totp_secret"`
    PasswordResetToken string    `json:"passwordResetToken" db:"password_reset_token"`
}

And I'm querying to get the user:

func (store DbUserStore) FindByUuid(uuid string) (*domain.User, error) {

    var user *domain.User

    err := store.tx.Get(&user, `SELECT uuid, email, name, created_at, totp_secret, password_reset_token FROM users WHERE uuid = $1::uuid`, uuid)

    if err == sql.ErrNoRows {
        return nil, new(NotFoundError)
    }

    if err != nil {
        return nil, err
    }

    return user, nil
}

The backtrace looks like this:

-- FAIL: Test_UserStore_CreatingANewUserSuccessfully (0.01 seconds)
panic: reflect: call of reflect.Value.NumField on ptr Value [recovered]
    panic: reflect: call of reflect.Value.NumField on ptr Value

goroutine 26 [running]:
runtime.panic(0x657860, 0xc2100bb220)
    /usr/local/go/src/pkg/runtime/panic.c:266 +0xb6
testing.funcΒ·005()
    /usr/local/go/src/pkg/testing/testing.go:385 +0xe8
runtime.panic(0x657860, 0xc2100bb220)
    /usr/local/go/src/pkg/runtime/panic.c:248 +0x106
reflect.flag.mustBe(0x166, 0x19)
    /usr/local/go/src/pkg/reflect/value.go:241 +0x8c
reflect.Value.NumField(0x5ee4e0, 0xc2100b6248, 0x166, 0xc2100aef50)
    /usr/local/go/src/pkg/reflect/value.go:1213 +0x30
github.com/jmoiron/sqlx.getValues(0x5ee4e0, 0xc2100b6248, 0x166, 0x5ee4e0, 0xc2100b6248, ...)
    /myproj/api/src/github.com/jmoiron/sqlx/sqlx.go:849 +0x23d
github.com/jmoiron/sqlx.setValues(0xc2100b75a0, 0x6, 0x6, 0x5ee4e0, 0xc2100b6248, ...)
    /myproj/api/src/github.com/jmoiron/sqlx/sqlx.go:908 +0x40
github.com/jmoiron/sqlx.(*Row).StructScan(0xc2100bb1c0, 0x5e9f60, 0xc2100b6248, 0x0, 0x0)
    /myproj/api/src/github.com/jmoiron/sqlx/sqlx.go:960 +0x3e1
github.com/jmoiron/sqlx.Get(0x2ac9db3dbc50, 0xc2100a7e00, 0x5e9f60, 0xc2100b6248, 0x747bb0, ...)
    /myproj/api/src/github.com/jmoiron/sqlx/sqlx.go:629 +0x89
github.com/jmoiron/sqlx.(*Tx).Get(0xc2100a7e00, 0x5e9f60, 0xc2100b6248, 0x747bb0, 0x68, ...)
    /myproj/api/src/github.com/jmoiron/sqlx/sqlx.go:351 +0x97
myproj/stores.DbUserStore.FindByUuid(0xc2100a7e00, 0xc2100ba9a0, 0xc2100b7510, 0x24, 0x24, ...)
    /myproj/api/src/myproj/stores/user_store.go:96 +0x117
myproj/stores_test.Test_UserStore_CreatingANewUserSuccessfully(0xc2100b9240)
    /myproj/api/src/myproj/stores/user_store_test.go:30 +0x25b
testing.tRunner(0xc2100b9240, 0x98aa98)
    /usr/local/go/src/pkg/testing/testing.go:391 +0x8b
created by testing.RunTests
    /usr/local/go/src/pkg/testing/testing.go:471 +0x8b2

goroutine 1 [chan receive]:
testing.RunTests(0x73dc40, 0x98a8a0, 0x21, 0x21, 0x424001)
    /usr/local/go/src/pkg/testing/testing.go:472 +0x8d5
testing.Main(0x73dc40, 0x98a8a0, 0x21, 0x21, 0x98d020, ...)
    /usr/local/go/src/pkg/testing/testing.go:403 +0x84
main.main()
    myproj/stores/_test/_testmain.go:111 +0x9c

goroutine 4 [chan receive]:
database/sql.(*DB).connectionOpener(0xc210076780)
    /usr/local/go/src/pkg/database/sql/sql.go:574 +0x3e
created by database/sql.Open
    /usr/local/go/src/pkg/database/sql/sql.go:436 +0x24d

goroutine 5 [syscall]:
runtime.goexit()
    /usr/local/go/src/pkg/runtime/proc.c:1394

I can't see why this is a problem, or what might possibly be going wrong (I've not used the reflect package myself)

Some background:

$ go version
go version go1.2 linux/amd64
$ uname -a
Linux harrow-appliance 3.12-0.bpo.1-amd64 #1 SMP Debian 3.12.6-2~bpo70+1 (2014-01-07) x86_64 GNU/Linux
$ cat /etc/debian_version
7.3

And, for whatever it's worth:

development=# SELECT version();
                                           version
----------------------------------------------------------------------------------------------
 PostgreSQL 9.3.2 on x86_64-unknown-linux-gnu, compiled by gcc (Debian 4.7.2-5) 4.7.2, 64-bit
(1 row)

I'm genuinely not sure what's going wrong here, but I don't think it's my fault!

How to deal with type alias?

I have this setup:

type Key string
type Model struct {
    // Id Key `db:"id"`
}

type User struct {
    // *Model
    Id Key `db:"id"`
    FirstName string `db:"first_name"`
    LastName string `db:"last_name"`
}

So this fails with the type Key not being supported. Also breaks with a strange error when I move the Id type to the Model struct and use embedding instead.

This works perfectly if I remove the type alias and just use string directly, of course.

So if I would like code above struct to works with sqlx (i.e. pass as argument to NamedExec), what do I need to do?
Is this even a supported use case?

runtime errors on OS X

I built a fairly simple example inserting a row into an SQLite3 table, which works sometimes, but crashes about 50% of the time (non-deterministically as far as I can tell):

Here it crashed creating a table in the database:

2013/07/12 17:08:23 runtime error: invalid memory address or nil pointer dereference:
/project/src/ldc/db/db.go:52 (0x4048553)
  func.002: log.Printf("%s:\n%s", r, debug.Stack())
/usr/local/Cellar/go/1.1.1/src/pkg/runtime/panic.c:229 (0x4016331)
  panic: reflectΒ·call(d->fn, (byte*)d->args, d->siz);
/usr/local/Cellar/go/1.1.1/src/pkg/runtime/panic.c:487 (0x4016ad3)
  panicstring: runtimeΒ·panic(err);
/usr/local/Cellar/go/1.1.1/src/pkg/runtime/os_darwin.c:462 (0x4015a28)
  sigpanic: runtimeΒ·panicstring("invalid memory address or nil pointer dereference");
/usr/local/Cellar/go/1.1.1/src/pkg/sync/atomic/asm_amd64.s:14 (0x412509d)
  CompareAndSwapUint32: LOCK
/usr/local/Cellar/go/1.1.1/src/pkg/sync/mutex.go:43 (0x40aaa65)
  (*Mutex).Lock: if atomic.CompareAndSwapInt32(&m.state, 0, mutexLocked) {
/usr/local/Cellar/go/1.1.1/src/pkg/database/sql/sql.go:470 (0x41291b5)
  (*DB).conn: db.mu.Lock()
/usr/local/Cellar/go/1.1.1/src/pkg/database/sql/sql.go:659 (0x412a0a8)
  (*DB).exec: dc, err := db.conn()
/usr/local/Cellar/go/1.1.1/src/pkg/database/sql/sql.go:650 (0x4129f88)
  (*DB).Exec: res, err = db.exec(query, args)
/project/src/ldc/db/db.go:39 (0x4047d1c)
  db: _, err := db.Exec(sql)
[...]

And here it crashed inserting a row:

/usr/local/Cellar/go/1.1.1/src/pkg/runtime/os_darwin.c:462 (0x4015bb8)
  sigpanic: runtimeΒ·panicstring("invalid memory address or nil pointer dereference");
/project/src/github.com/jmoiron/sqlx/sqlx.go:137 (0x40c40fc)
  com/jmoiron/sqlx.(*DB).BindMap: return BindMap(BindType(db.driverName), query, argmap)
/project/src/github.com/jmoiron/sqlx/sqlx.go:845 (0x40c9741)
  com/jmoiron/sqlx.NamedExecMap: q, args, err := e.BindMap(query, argmap)
/project/src/github.com/jmoiron/sqlx/sqlx.go:152 (0x40c438d)
  com/jmoiron/sqlx.(*DB).NamedExecMap: return NamedExecMap(db, query, argmap)
[...]

This happens on go version go1.1.1 darwin/amd64.

go race detector finds race condition

I have 3 examples. Two examples don't trigger a race warning from the go tool, while one does. The case where a race condition is detected is when I call QueryRow instead of QueryRowx in the SQL examples.

diff sqlx-no-race.go sqlx-race.go

46c46
<       row := db.QueryRow("SELECT COUNT(*) FROM Test")

---
>       row := db.QueryRowx("SELECT COUNT(*) FROM Test")

This last example does the same as the others, except it does not use SQLx. sql-no-race.go

When running sqlx-race.go i get the following warning.

==================
WARNING: DATA RACE
Write by goroutine 5:
  container/list.(*List).insertValue()
      /usr/local/go/src/pkg/container/list/list.go:105 +0x151
  container/list.(*List).PushFront()
      /usr/local/go/src/pkg/container/list/list.go:133 +0x63
  database/sql.(*DB).putConnDBLocked()
      /usr/local/go/src/pkg/database/sql/sql.go:795 +0x32f
  database/sql.(*DB).putConn()
      /usr/local/go/src/pkg/database/sql/sql.go:766 +0x29f
  database/sql.(*driverConn).releaseConn()
      /usr/local/go/src/pkg/database/sql/sql.go:240 +0x5e
  database/sql.*driverConn.(database/sql.releaseConn)Β·fm()
      /usr/local/go/src/pkg/database/sql/sql.go:913 +0x47
  database/sql.(*Rows).Close()
      /usr/local/go/src/pkg/database/sql/sql.go:1587 +0x1ba
  database/sql.(*Row).Scan()
      /usr/local/go/src/pkg/database/sql/sql.go:1636 +0x7d
  main.funcΒ·001()
      /home/simon/src/github.com/simonz05/db-tests/race-sqlx/main.go:48 +0x132

Previous read by goroutine 6:
  database/sql.(*DB).conn()
      /usr/local/go/src/pkg/database/sql/sql.go:646 +0x1c1
  database/sql.(*DB).query()
      /usr/local/go/src/pkg/database/sql/sql.go:908 +0x39
  database/sql.(*DB).Query()
      /usr/local/go/src/pkg/database/sql/sql.go:899 +0xa4
  database/sql.(*DB).QueryRow()
      /usr/local/go/src/pkg/database/sql/sql.go:977 +0x77
  main.funcΒ·001()
      /home/simon/src/github.com/simonz05/db-tests/race-sqlx/main.go:46 +0x7c

Goroutine 5 (running) created at:
  main.main()
      /home/simon/src/github.com/simonz05/db-tests/race-sqlx/main.go:61 +0x6f0

Goroutine 6 (running) created at:
  main.main()
      /home/simon/src/github.com/simonz05/db-tests/race-sqlx/main.go:62 +0x70b
==================
==================
WARNING: DATA RACE
Write by goroutine 6:
  container/list.(*List).insertValue()
      /usr/local/go/src/pkg/container/list/list.go:105 +0x119
  container/list.(*List).PushFront()
      /usr/local/go/src/pkg/container/list/list.go:133 +0x63
  database/sql.(*DB).putConnDBLocked()
      /usr/local/go/src/pkg/database/sql/sql.go:795 +0x32f
  database/sql.(*DB).putConn()
      /usr/local/go/src/pkg/database/sql/sql.go:766 +0x29f
  database/sql.(*driverConn).releaseConn()
      /usr/local/go/src/pkg/database/sql/sql.go:240 +0x5e
  database/sql.*driverConn.(database/sql.releaseConn)Β·fm()
      /usr/local/go/src/pkg/database/sql/sql.go:913 +0x47
  database/sql.(*Rows).Close()
      /usr/local/go/src/pkg/database/sql/sql.go:1587 +0x1ba
  database/sql.(*Row).Scan()
      /usr/local/go/src/pkg/database/sql/sql.go:1636 +0x7d
  main.funcΒ·001()
      /home/simon/src/github.com/simonz05/db-tests/race-sqlx/main.go:48 +0x132

Previous write by goroutine 5:
  container/list.(*List).insertValue()
      /usr/local/go/src/pkg/container/list/list.go:105 +0x5a
  container/list.(*List).PushFront()
      /usr/local/go/src/pkg/container/list/list.go:133 +0x63
  database/sql.(*DB).putConnDBLocked()
      /usr/local/go/src/pkg/database/sql/sql.go:795 +0x32f
  database/sql.(*DB).putConn()
      /usr/local/go/src/pkg/database/sql/sql.go:766 +0x29f
  database/sql.(*driverConn).releaseConn()
      /usr/local/go/src/pkg/database/sql/sql.go:240 +0x5e
  database/sql.*driverConn.(database/sql.releaseConn)Β·fm()
      /usr/local/go/src/pkg/database/sql/sql.go:913 +0x47
  database/sql.(*Rows).Close()
      /usr/local/go/src/pkg/database/sql/sql.go:1587 +0x1ba
  database/sql.(*Row).Scan()
      /usr/local/go/src/pkg/database/sql/sql.go:1636 +0x7d
  main.funcΒ·001()
      /home/simon/src/github.com/simonz05/db-tests/race-sqlx/main.go:48 +0x132

Goroutine 6 (running) created at:
  main.main()
      /home/simon/src/github.com/simonz05/db-tests/race-sqlx/main.go:62 +0x70b

Goroutine 5 (finished) created at:
  main.main()
      /home/simon/src/github.com/simonz05/db-tests/race-sqlx/main.go:61 +0x6f0
==================
1
1
Found 2 data race(s)

Date mapping

It seems there's a problem when mapping date data type, is there an go type to use in the struct field for date data type?

Custom types in structs.

An example

type ExpireTimestamp int64

type User struct {
  Name string
  BackpackTimestamp  ExpireTimestamp
}

When I save a user, it saves the int64 correctly, but when I go to look it up:

db.Get(&userObj, "SELECT * FROM users WHERE id = $1", id)

I get an error that I panic on:

panic: reflect.Set: value of type int64 is not assignable to type user.BackpackTimestamp

I'm new to both Go and especially the database/sql implementations, so I'm not sure what would be the best way to solve this problem. Any help or examples?

Scan NULL values to nil pointers

I was wondering if it would be possible to scan sql rows into struct containing pointers and use nil pointers when encountering NULL values ?

I had the idea while struggling to do clean and neat code using the Null* types of the database/sql. I really find this types ugly and unpracticables, and the lack of alternative surprised me.

I did some proof-of-concept code (you can see it here https://gist.github.com/elwinar/79d206f12b6a97e6af43) using your new reflectx package (copy-pasted in the same package for simplicity's sake).

Do you see major issues with the approach ? (other than the code being ugly at the moment :P )

Ugly error when trying to select into a single struct

So, I misunderstood the API of Select, and didn't realize it was always expecting a slice... and so I passed it a pointer to a struct, figuring it would realize it was a struct and just do one row.... I got back a really ugly panic:

reflect.(_rtype).Elem(...)
/home/nate/go/src/pkg/reflect/type.go:584 +0x118
github.com/jmoiron/sqlx.StructScan(...)
/home/nate/code/go/src/github.com/jmoiron/sqlx/sqlx.go:761 +0x1b4
github.com/jmoiron/sqlx.Select(...)
/home/nate/code/go/src/github.com/jmoiron/sqlx/sqlx.go:508 +0xbe
github.com/jmoiron/sqlx.(_DB).Select(...)
/home/nate/code/go/src/github.com/jmoiron/sqlx/sqlx.go:160 +0x71

There seem to be a couple ways to handle this with better results:

1.) Allow this to work, just check to see if the thing passed in is a struct, and if so, just scan the first returned row
2.) Don't allow it to work, but handle the error with a more friendly error message

for #2, it might also be beneficial to rename the parameter as something like dest_slice ... to make it a little more obvious that it requires a slice.

StructScan infinite loop

I had a case earlier which made me pull my hair for a full day and half :

Let's say we have the following structs :

type Entity struct {
    ID int64 `db:"id"`
    Service *Service
}

type Service struct {
    Entity *Entity
}

func NewEntity(id int64) *Entity {
    e := &Entity{}
    e.Service = &Service{}
    e.Service.Entity = e
    return e
}

When StructScaning, the function tries to establish a field map by exploring the tree of embedded types of the struct. I my case, the exploration never end because of the pointers.

This issue bring the following question to my mind : what is the point of exploring non-embedded fields ? It seem rather non-semantic to me…
And independantly, why making a field map and not simply look for the fields as needed when scanning ?

Ability to prepare named query

It'd be nice to have a PrepareNamedQuery() function, along the lines of:

func (db *DB) PrepareNamedQuery(query string, arg interface{}) (*Stmt, error) {
    return PrepareNamedQuery(db, query, arg)
}

// I guess this means that Ext would need to include a Preparer?  That's probably ok,
// since both DB and Tx support Prepare().
func PrepareNamedQuery(e Ext, query string, arg interface{}) (*Stmt, error) {
    q, args, err := e.BindStruct(query, arg)
    if err != nil {
        return nil, err
    }
        return e.Preparex(q, args...)
}

"index out of range" error with conflicting tags in embedded struct

Scanning to a struct that embeds another struct, with conflicting field names in the embedding and embedded structs, yields an unintuitive error:

package main

import (
    "fmt"
    "io/ioutil"
    "os"
    "path"

    "github.com/jmoiron/sqlx"
    _ "github.com/mattn/go-sqlite3"
)

type Bar struct {
    SomeOtherInt string `db:"someint"`
}

type Foo struct {
    Bar
    SomeInt int `db:"someint"`
}

func main() {
    tmpDir, err := ioutil.TempDir(os.TempDir(), "sqlx-test")
    if err != nil {
        panic(err)
    }

    if err := os.MkdirAll(tmpDir, os.FileMode(0755)); err != nil {
        panic(err)
    }

    dbPath := path.Join(tmpDir, "sqlx-test.db")
    db, err := sqlx.Open("sqlite3", dbPath)
    if err != nil {
        panic(err)
    }

    if _, err := db.Exec("CREATE TABLE foo (someint INTEGER);"); err != nil {
        panic(err)
    }

    if _, err := db.Exec("INSERT INTO foo (someint) VALUES (42);"); err != nil {
        panic(err)
    }

    var foos []*Foo
    if err := db.Select(&foos, "SELECT someint FROM foo"); err != nil {
        panic(err)
    }

    fmt.Println("Got foos", len(foos))

    if err := db.Close(); err != nil {
        panic(err)
    }

    if err := os.RemoveAll(tmpDir); err != nil {
        panic(err)
    }
}

gives:

$ ./sqlx 
panic: runtime error: index out of range

goroutine 1 [running]:
runtime.panic(0x41a5980, 0x42e85f7)
    /usr/local/go/src/pkg/runtime/panic.c:266 +0xb6
github.com/jmoiron/sqlx.fieldMap.allValues(0xc21001e4e0, 0x41a6d40, 0xc210048140, 0x196, 0x41a6d40, ...)
    /Users/jobi/go/src/github.com/jmoiron/sqlx/reflect.go:262 +0x76b
github.com/jmoiron/sqlx.fieldMap.getValues(0xc21001e4e0, 0x41a6d40, 0xc210048140, 0x196, 0xc210000088, ...)
    /Users/jobi/go/src/github.com/jmoiron/sqlx/reflect.go:172 +0x5e
github.com/jmoiron/sqlx.StructScan(0x43c02f0, 0xc210056000, 0x4170e60, 0xc210048100, 0x0, ...)
    /Users/jobi/go/src/github.com/jmoiron/sqlx/sqlx.go:1001 +0x6b5
github.com/jmoiron/sqlx.Select(0x43c0258, 0xc210048060, 0x4170e60, 0xc210048100, 0x41d1450, ...)
    /Users/jobi/go/src/github.com/jmoiron/sqlx/sqlx.go:711 +0x116
github.com/jmoiron/sqlx.(*DB).Select(0xc210048060, 0x4170e60, 0xc210048100, 0x41d1450, 0x17, ...)
    /Users/jobi/go/src/github.com/jmoiron/sqlx/sqlx.go:230 +0x97
main.main()
    /Users/jobi/tests/sqlx/test.go:47 +0x333

goroutine 3 [chan receive]:
database/sql.(*DB).connectionOpener(0xc21004b000)
    /usr/local/go/src/pkg/database/sql/sql.go:574 +0x3e
created by database/sql.Open
    /usr/local/go/src/pkg/database/sql/sql.go:436 +0x24d

goroutine 4 [syscall]:
runtime.goexit()
    /usr/local/go/src/pkg/runtime/proc.c:1394

The error is unclear, it took me a while to figure out what was causing this.

Question on choice of MySQL driver

I notice that in this project you use github.com/go-sql-driver/mysql, whereas in modl you use ziutek's driver. I have found the former to be cleaner, and more idiomatic and go-like -- subjectively, of course. I'm curious to learn more about your choices on drivers.

separate unit & integration tests

sqlx is mostly integration tests. Having written libraries dealing with databases in the past, I am a firm believer that if you aren't testing against an actual database then you aren't testing anything.

However, it does have some bits which are unit testable, and it does have these unit tests. It'd be nice to make these easily runnable outside the integration tests, so that database drivers do not have to be available to run sanity checks on parts of the code.

Stmt close unexpectedly after first query.

DB Schema:

     CREATE TABLE IF NOT EXISTS Product (                                
         ProductID         INT(11)      NOT NULL AUTO_INCREMENT,
         CONSTRAINT Pk_Product PRIMARY KEY ( ProductID )
     ) engine=InnoDB;

Implementation without sqlx. Verify that it should not panic.

    package main

    import (
        "fmt"
        "database/sql"

        _ "github.com/go-sql-driver/mysql"
    )

    func main() {
        dsn := "testing:testing@tcp(localhost:3306)/testing?charset=utf8&parseTime=True"
        db, _ := sql.Open("mysql", dsn)
        stmt, err := db.Prepare("SELECT ProductID FROM Product WHERE ProductID = ?")

        if err != nil {
            panic(err)
        }

        var productID int 
        row := stmt.QueryRow(1)
        err = row.Scan(&productID)

        if err != nil {
            fmt.Println(err) // sql: no rows in result set
        }

        err = row.Scan(&productID)
        if err != nil {
            fmt.Println(err) // sql: no rows in result set
        }
    }

sqlx equivalent panics at second call using the stmt.

    package main

    import (
        "fmt"
        "database/sql"

        _ "github.com/go-sql-driver/mysql"
        "github.com/jmoiron/sqlx"
    )

    type Product struct {
        ProductID int
    }

    func main() {
        dsn := "testing:testing@tcp(localhost:3306)/testing?charset=utf8&parseTime=True"
        db, _ := sql.Open("mysql", dsn)
        sqlxDb := sqlx.NewDb(db, "mysql")

        stmt, err := sqlxDb.Preparex("SELECT ProductID FROM Product WHERE ProductID = ?")

        if err != nil {
            panic(err)
        }

        p := &Product{}
        err = stmt.Get(p, 1) // calls stmt.close
        fmt.Println(err) // sql: no rows in result set
        stmt.Get(p, 1) // Panic
    }

nil pointer due to a closed connection:

panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x41 pc=0x44c602]

goroutine 1 [running]:
runtime.panic(0x564c60, 0x73d908)
/usr/local/go/src/pkg/runtime/panic.c:266 +0xb6
github.com/go-sql-driver/mysql.(*mysqlStmt).writeExecutePacket(0xc21001e780, 0xc21000a5c0, 0x1, 0x1, 0x484ffa, ...)
/home/simon/src/github.com/go-sql-driver/mysql/packets.go:717 +0x152
github.com/go-sql-driver/mysql.(*mysqlStmt).Query(0xc21001e780, 0xc21000a5c0, 0x1, 0x1, 0xc21000a5c0, ...)
/home/simon/src/github.com/go-sql-driver/mysql/statement.go:76 +0x4a
database/sql.rowsiFromStatement(0x7f56706b9610, 0xc2100381e0, 0x7f56706b9570, 0xc21001e780, 0xc21000a5a0, ...)
/usr/local/go/src/pkg/database/sql/sql.go:1406 +0x295
database/sql.(*Stmt).Query(0xc210052300, 0xc21000a5a0, 0x1, 0x1, 0x0, ...)
/usr/local/go/src/pkg/database/sql/sql.go:1367 +0x1f6
github.com/jmoiron/sqlx.(*qStmt).QueryRowx(0xc210052300, 0x58d7e0, 0x0, 0xc21000a5a0, 0x1, ...)
/home/simon/src/github.com/jmoiron/sqlx/sqlx.go:414 +0x58
github.com/jmoiron/sqlx.Get(0x7f56706b95d8, 0xc210052300, 0x519ee0, 0xc2100000c8, 0x58d7e0, ...)
/home/simon/src/github.com/jmoiron/sqlx/sqlx.go:607 +0x67
github.com/jmoiron/sqlx.(*Stmt).Get(0xc210052200, 0x519ee0, 0xc2100000c8, 0xc21000a5a0, 0x1, ...)
/home/simon/src/github.com/jmoiron/sqlx/sqlx.go:442 +0xcb
main.main()
/home/simon/src/github.com/simonz05/test/main.go:29 +0x317

goroutine 3 [chan receive]:
database/sql.(*DB).connectionOpener(0xc210052080)
/usr/local/go/src/pkg/database/sql/sql.go:571 +0x3e
created by database/sql.Open
/usr/local/go/src/pkg/database/sql/sql.go:433 +0x24d

goroutine 4 [syscall]:
runtime.goexit()
/usr/local/go/src/pkg/runtime/proc.c:1396

Reverse callstack from call to close():

/usr/local/go/src/pkg/database/sql/sql.go 721
/usr/local/go/src/pkg/database/sql/sql.go 1463
/usr/local/go/src/pkg/database/sql/sql.go 394
/usr/local/go/src/pkg/database/sql/sql.go 373
/usr/local/go/src/pkg/database/sql/sql.go 1383
/usr/local/go/src/pkg/database/sql/sql.go 1584
/usr/local/go/src/pkg/database/sql/sql.go 1508
/home/simon/src/github.com/jmoiron/sqlx/sqlx.go 101
/home/simon/src/github.com/jmoiron/sqlx/sqlx.go 866
/home/simon/src/github.com/jmoiron/sqlx/sqlx.go 608
/home/simon/src/github.com/jmoiron/sqlx/sqlx.go 442
/home/simon/src/github.com/simonz05/test/main.go 27

StructScan fails with embedded structs

I've been having difficulty using StructScan with a struct that contains other (embedded or non-embedded) structs. I'm not sure if I am doing something wrong but I get the following panic:

panic: reflect: Field index out of range [recovered]
    panic: reflect: Field index out of range

Below is a test case based on TestEmbeddedStructs that reproduces the panic.

    // test embedded structs with StructScan
    rows, err := db.Queryx(
    `SELECT person.*, place.* FROM
         person natural join place`)
    if err != nil {
        t.Errorf("Poop.")
    }

    pp := PersonPlace{}
    rows.StructScan(&pp)

    if len(pp.Person.FirstName) == 0 {
        t.Errorf("Expected non zero lengthed first name.")
    }
    if len(pp.Place.Country) == 0 {
        t.Errorf("Expected non zero lengthed country.")
    }

Lower columns for name matching

Because the struct field name is lowered, if your db uses PascalCasing like your struct, it won't match.

It would be nice if sqlx lowered both sides, such that dbs using pascal casing won't need redundant int FooBar 'db:"FooBar"' definitions.

Edit: I found that you can override this behaviour by doing sqlx.NameMapper = func(s string) string { return s }

Still, lowering just one side strikes me as a really poor default, when otherwise the natural mapping from Go to SQL (PascalCasing) would just work.

no tables created after running CreateTablesIfNotExists

Given the following struct:

type Product struct {
    Id        int64     `json:"id"`
    Name      string    `json:"name"`
    ImageUrl  string    `json:"image_url" db:"image_url"`
    CreatedAt time.Time `json:"created_at" db:"created_at"`
}

and the following piece of code:

db, err := sql.Open("postgres", "user=go dbname=test host=localhost")
if err != nil {
    fmt.Printf("sql.Open error: %v\n", err)
    return nil
}

dbmap := &gorp.DbMap{Db: db, Dialect: gorp.PostgresDialect{}}
dbmap.AddTableWithName(Product{}, "products").SetKeys(true, "Id")
dbmap.TraceOn("[gorp]", log.New(os.Stdout, "", log.Lmicroseconds))
dbmap.CreateTablesIfNotExists()

I see the following when my app starts, but no table is created. What's wrong?

11:28:27.170174 [gorp] create table if not exists "products" ("id" bigserial not null primary key , "name" varchar(255), "image_url" varchar(255), "created_at" timestamp with time zone) ; [[]]

panic: prepared statement inside transaction

panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x442e9c]

goroutine 1 [running]:
runtime.panic(0x564fe0, 0x73f928)
    /usr/local/go/src/pkg/runtime/panic.c:266 +0xb6
database/sql.(*Stmt).Query(0x0, 0x7fcb02fa1ef8, 0x1, 0x1, 0x0, ...)
    /usr/local/go/src/pkg/database/sql/sql.go:1358 +0x3fc
github.com/jmoiron/sqlx.(*qStmt).QueryRowx(0xc2100000e8, 0x58de80, 0x0, 0x7fcb02fa1ef8, 0x1, ...)
    /home/simon/src/github.com/jmoiron/sqlx/sqlx.go:415 +0x50
github.com/jmoiron/sqlx.(*Stmt).QueryRowx(0xc2100000e0, 0x7fcb02fa1ef8, 0x1, 0x1, 0x0)
    /home/simon/src/github.com/jmoiron/sqlx/sqlx.go:476 +0x82
main.main()
    /home/simon/src/github.com/simonz05/test/main.go:40 +0x253

goroutine 3 [chan receive]:
database/sql.(*DB).connectionOpener(0xc210051080)
    /usr/local/go/src/pkg/database/sql/sql.go:571 +0x3e
created by database/sql.Open
    /usr/local/go/src/pkg/database/sql/sql.go:433 +0x24d

goroutine 4 [syscall]:
runtime.goexit()
    /usr/local/go/src/pkg/runtime/proc.c:1396

The first one here is my test-program without using sqlx:

    package main

    import (
        "database/sql"
        "fmt"

        _ "github.com/go-sql-driver/mysql"
    )

    type Product struct {
        ProductID int `db:"ProductID"`
    }

    func main() {
        dsn := "testing:testing@tcp(localhost:3306)/testing?charset=utf8&parseTime=True"
        db, _ := sql.Open("mysql", dsn)

        stmt, err := db.Prepare("SELECT ProductID FROM Product WHERE ProductID = ?")

        if err != nil {
            panic(err)
        }

        p := &Product{}
        tx, err := db.Begin()

        if err != nil {
            panic(err)
        }

        txStmt := tx.Stmt(stmt)
        row := txStmt.QueryRow(200)

        if err := row.Scan(&p.ProductID); err != nil {
            panic(err)
        }

        tx.Commit()
        fmt.Println(p)
    }

The following is the test using sqlx, creating a panic.

    package main

    import (
        "database/sql"
        "fmt"

        _ "github.com/go-sql-driver/mysql"
        "github.com/jmoiron/sqlx"
    )

    type Product struct {
        ProductID int `db:"ProductID"`
    }

    func main() {
        dsn := "testing:testing@tcp(localhost:3306)/testing?charset=utf8&parseTime=True"
        db, _ := sql.Open("mysql", dsn)
        sqlxDb := sqlx.NewDb(db, "mysql")

        stmt, err := sqlxDb.Preparex("SELECT ProductID FROM Product WHERE ProductID = ?")

        if err != nil {
            panic(err)
        }

        p := &Product{}

        tx, err := sqlxDb.Beginx()
        if err != nil {
            panic(err)
        }

        txStmt := tx.Stmtx(stmt)
        row := txStmt.QueryRowx(200) // panic
        err = row.StructScan(p)
        tx.Commit()
        fmt.Println(p)
    }

Question: what are the mapping rules?

I could not find them stated in an easy to read form, in this example

type Person struct {
    FirstName string `db:"first_name"`
    LastName  string `db:"last_name"`
    Email     string
}

if I changed it to

type Person struct {
    FirstName string
    LastName  string 
    Email     string
}

and my columns were FirstName, LastName and Email (pascal case) would the auto mapping not work?

Performance Query (postgres) - Large Drop in req/s

I've made some changes to this to reflect the current state after some more testing

This isn't strictly related to sqlx, but I'm hoping someone with more experience than I might have some tips on how to improve throughput:

  • Routes without any DB ops: 33k req/s (inc. template rendering with multiple fragments)
  • Routes with a single SELECT on six columns by the id (PK): 5.9k req/s query plan
  • The index page with a SELECT * FROM table WHERE expiry_date > current_date ORDER BY expiry_date DESC LIMIT 15 where there is a secondary index on expiry_date (timestamp with timezone) tops out at about 1.5k req/s query plan
  • This is using wrk with 128 open connections and 64 threads.
  • The Go code behind these can be found here: https://gist.github.com/elithrar/c525cfb7c3b51418305c

Update:

I've done some more testing on this, and it seems the biggest bottleneck is just rendering the slice of 15 structs (the Listing struct has 21 fields). I mocked up a function that just returns a slice of Listings and renders it in the template (instead of fetching them from the DB) and the performance gap is a little smaller, but still exists (2300 req/s vs. 1500 req/s with real queries). Returning an empty slice nets 27k req/s. Using make(&models.Listings, 0, 15) to create a slice of the appropriate capacity does not yield any improvement.

Profiling didn't seem to indicate that this was the problem, but there's obviously more overhead in rendering this slice using html/template and a {{ with }} and {{ range . }} clauses than I'd thought. I'd still like to see if this can be remedied (if at all) though, and am open to suggestions for doing so.

Unexport internal functions

It will break the API, but really it's quite unlikely anyone is using them. It will make the documentation more clear.

eg: BaseSliceType, BaseStructType

Embedded structs?

Any thoughts on supporting embedded structs? I generally use a struct per table and then embed them in a parent struct for when I do joins across tables (which is pretty often).

Just wondering if you had tried it and ran into a problem, or just hadn't implemented it because of a lack of need on your part?

My Reflect knowledge is pretty limited, but I guess there has to be a reason to learn, and implementing embedded struct support seems as good a reason as any.

Thoughts?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.