data-dog / go-txdb

Immutable transaction isolated sql driver for golang

Single transaction based sql.Driver for GO

Package txdb is a single transaction based database sql driver. When the connection is opened, it starts a transaction and all operations performed on this sql.DB will be within that transaction. If concurrent actions are performed, the lock is acquired and connection is always released the statements and rows are not holding the connection.

Why is it useful. A very basic use case would be if you want to make functional tests you can prepare a test database and within each test you do not have to reload a database. All tests are isolated within transaction and though, performs fast. And you do not have to interface your sql.DB reference in your code, txdb is like a standard sql.Driver.

This driver supports any sql.Driver connection to be opened. You can register txdb for different sql drivers and have it under different driver names. Under the hood whenever a txdb driver is opened, it attempts to open a real connection and starts transaction. When close is called, it rollbacks transaction leaving your prepared test database in the same state as before.

Given, you have a mysql database called txdb_test and a table users with a username column.

package main

import (
    "database/sql"
    "log"

    "github.com/DATA-DOG/go-txdb"
    _ "github.com/go-sql-driver/mysql"
)

func init() {
    // we register an sql driver named "txdb"
    txdb.Register("txdb", "mysql", "root@/txdb_test")
}

func main() {
    // dsn serves as an unique identifier for connection pool
    db, err := sql.Open("txdb", "identifier")
    if err != nil {
        log.Fatal(err)
    }
    defer db.Close()

    if _, err := db.Exec(`INSERT INTO users(username) VALUES("gopher")`); err != nil {
        log.Fatal(err)
    }
}

You can also use sql.OpenDB (added in Go 1.10) rather than registering a txdb driver instance, if you prefer:

package main

import (
    "database/sql"
    "log"

    "github.com/DATA-DOG/go-txdb"
    _ "github.com/go-sql-driver/mysql"
)

func main() {
    db := sql.OpenDB(txdb.New("mysql", "root@/txdb_test"))
    defer db.Close()

    if _, err := db.Exec(`INSERT INTO users(username) VALUES("gopher")`); err != nil {
        log.Fatal(err)
    }
}

Every time you will run this application, it will remain in the same state as before.

Testing

Usage is mainly intended for testing purposes. Tests require database access, support using postgres and mysql databases. The easiest way to do this is by using testcontainers, which is enabled by setting the respective database DSN values to AUTO. Example:

MYSQL_DSN=AUTO PSQL_DSN=AUTO go test ./...

If you wish to use a running local database instance, you can also provide the DSN directly, and it will be used:

MYSQL_DSN=root:pass@/ PSQL_DSN=postgres://postgres:pass@localhost/ go test ./...

To run tests only against MySQL or PostgreSQL, you may provide only the respective DSN values; any unset DSN is skipped for tests.

Documentation

See godoc for general API details. See .travis.yml for supported go versions.

Contributions

Feel free to open a pull request. Note, if you wish to contribute an extension to public (exported methods or types) - please open an issue before to discuss whether these changes can be accepted. All backward incompatible changes are and will be treated cautiously.

The public API is locked since it is an sql.Driver and will not change.

License

txdb is licensed under the three clause BSD license

go-txdb's Issues

Looking for maintainers

Hi, I do not have much spare time for this library and haven't used go over few years. Open up a conversation if you are using this library and willing to maintain it for some time. I'll give the permission to manage repository.

Suggestion: txdb.Close(name string) to close txdb.txDriver.db

I’ve run into a small problem with txdb because it automatically opens a new connection here:

go-txdb/db.go

Lines 122 to 128 in 6ca798a

	if d.db == nil {
	db, err := sql.Open(d.drv, d.dsn)
	if err != nil {
	return nil, err
	}
	d.db = db
	}

However, there is no way to call txdb.txDriver.db.Close(), which would be nice because I am spawning temporary test databases that are dropped after the tests are run. Unfortunately, because PostgreSQL doesn’t let you destroy an active database, having something hold the sql.DB connection makes the following SQL fail:

db.Exec(`DROP DATABASE test_database`)
pq: database "test_database" is being accessed by other users

My suggestion is to create a new function called txdb.Reset:

var drivers struct {
	sync.Mutex
	drvs map[string]*txDriver
}

func Reset(name string) error {
	drivers.Lock()
	defer drivers.Unlock()

	d, ok := drivers.drv[name]
	if !ok {
		return ErrNotRegistered
	}

	if d.db == nil {
		return nil
	}

	var err error
	for _, c := range d.conns {
		if cerr := c.Close(); cerr != nil && err == nil {
			err = cerr
		}
	}
	if err != nil {
		return err
	}

	if err := d.db.Close(); err != nil {
		return err
	}

	d.db = nil
	return nil
}

This would also require changes to txdb.Register:

func Register(name, drv, dsn string, options ...func(*conn) error) {
	drv := &txDriver{
		dsn:     dsn,
		drv:     drv,
		conns:   make(map[string]*conn),
		options: options,
	}
	sql.Register(name, drv)

	drivers.Lock()
	defer drivers.Unlock()
	drivers[name] = drv
}

Sorry I have not tested or documented this code.

Fedora 25 - go test: FAIL: TestShouldRunWithinTransaction

cat /etc/redhat-release
#Fedora release 25 (Twenty Five)                                                                                                                                                                            
dnf -y install "compiler(go-compiler)"

go version
#go version go1.7.5 linux/amd64

dnf -y install "golang(github.com/go-sql-driver/mysql)"

rm -rf /tmp/go
mkdir -p /tmp/go/src
export GOPATH=/tmp/go:/usr/share/gocode
cd /tmp/go/src
mkdir -p github.com/DATA-DOG
cd github.com/DATA-DOG
git clone https://github.com/DATA-DOG/go-txdb
cd go-txdb
git rev-parse HEAD
#ca6ebaaaa8271297fab0f1170eadf5c2c80e7629                                                                                                                                                                   
go test
#--- FAIL: TestShouldRunWithinTransaction (0.00s)                                                                                                                                                            
#db_test.go:28: failed to insert an user: dial tcp 127.0.0.1:3306: getsockopt: connection refused                                                                                                            
#--- FAIL: TestShouldHandlePrepare (0.00s)                                                                                                                                                                   
#panic: runtime error: invalid memory address or nil pointer dereference [recovered]                                                                                                                         
#panic: runtime error: invalid memory address or nil pointer dereference                                                                                                                                     
#[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x4ddc67]         

does this support orm?

Hey, this library is working well with https://github.com/jmoiron/sqlx, but it seems not working with orm, such as https://github.com/gobuffalo/pop and https://github.com/jinzhu/gorm. Is there any workaround that developer can implement to use this library with these orms? Thanks!

retry on failure

Hello!

We are using txdb along with cockroachdb for integration testing (using pgx driver) and it's working fine.

The thing is that sometimes, CockroachDB can throw some errors that need to be retried, and for that purpose there is this driver:

https://github.com/cockroachdb/cockroach-go

The question is... any idea or anyone built a logic around retrial with txdb? It doesn't have to be with cockroachdb or anything, I just would need some guidance if someone has any idea on what should I do if the engine errs:

ROLLBACK, BEGIN again (with txdb_1 SAVEPOINT or something) and retry?

Any ideas would be welcome, thank you! 🙏

CC/ @l3pp4rd @Yiling-J @flimzy

Prepared statement creation doesn't return error

This just may be something that won't work here but I thought I'd check and see.

As an example, this simple main.go

package main

import (
    "database/sql"
    "log"

    _ "github.com/mattn/go-sqlite3"
)

func main() {
    _, err := sql.Open("sqlite3", "test.db")
    if err != nil {
        log.Fatal(err)
    }
}

and this main_test.go

package main

import (
    "database/sql"
    "testing"

    "github.com/DATA-DOG/go-txdb"
    _ "github.com/mattn/go-sqlite3"
)

func init() {
    txdb.Register("txdb", "sqlite3", "test.db")
}

func TestPreparedStatement(t *testing.T) {
    db, err := sql.Open("txdb", "id")
    if err != nil {
        t.Fatal(err)
    }
    _, err = db.Prepare("THIS SHOULD FAIL...")
    if err != nil {
        t.Fatal(err)
    }
}

I was expecting/hoping/wishing that TestPreparedStatement would fail but I'm open to the possibility that either a) it's not possible or b) I'm holding it wrong. 😃

pq: syntax error at or near "," while using txdb driver. works with plain sqlx

this is sqlx.NamedExec call which has problem while running with txdb

sql := "INSERT INTO jobs(id, url_hash, created_at, job) VALUES (:id, :url_hash, :created_at, :job);"
_, err := s.DB.NamedExecContext(ctx, sql, jobs)

err is pq: syntax error at or near ","

i have tried master branch as well as go mod

Issue when bumping to v0.1.9 from v0.1.8

go mod tidy
go: github.com/DATA-DOG/[email protected] requires
        github.com/testcontainers/[email protected] requires
        github.com/Microsoft/[email protected] requires
        github.com/open-policy-agent/[email protected] requires
        oras.land/[email protected] requires
        github.com/distribution/distribution/[email protected] requires
        github.com/mitchellh/[email protected]: invalid version: exit status 128:
        remote: Repository not found.
        fatal: repository 'https://github.com/mitchellh/osext/' not found

Context cancellation rolls transaction back (MySQL)

It looks like I have found some unexpected behavior with this library when it comes to how context and transactions interact.

Description

When using a context that has a cancel function with db.QueryContext and similar functions, sometimes, when that cancel function is called, the transaction is rolled back, returning sql: transaction has already been committed or rolled back on subsequent calls.

Reproduction

This code reproduces the issue about 20% of times that it runs. After the aux function countUsers returns and dispatches the cancel function, the next call will return the above transaction error.

func main() {
	txdb.Register("mysqltx", "mysql", "root@/txdb_test")
	db, err := sql.Open("mysqltx", fmt.Sprintf("connection_%d", time.Now().UnixNano()))
	if err != nil {
		log.Fatalf("could not open DB tx: %s", err.Error())
	}
	n, err := countUsers(db)
	if err != nil {
		log.Fatalf("error doing query: %s", err.Error())
	}
	log.Printf("Found %d users", n)
	if err := db.Close(); err != nil {
                // error here
		log.Fatalf("could not close DB tx: %s", err.Error())
	}
}

func countUsers(db *sql.DB) (int, error) {
	ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
	defer cancel()
	var n int
	if err := db.QueryRowContext(ctx, "SELECT COUNT(*) FROM user").Scan(&n); err != nil {
		return 0, err
	}

	return n, nil
}

Notes

• Go version: go version go1.13.5 linux/amd64
• MySQL version mysql:5.7.12 Docker image
• Not able to reproduce this in my example, but on my project that I've found this issue, I'm also seeing series of these errors:

[mysql] 2020/01/12 20:17:11 packets.go:412: busy buffer
[mysql] 2020/01/12 20:17:11 packets.go:433: busy buffer
[mysql] 2020/01/12 20:17:11 packets.go:393: busy buffer
[mysql] 2020/01/12 20:17:11 packets.go:412: busy buffer
[mysql] 2020/01/12 20:17:11 packets.go:433: busy buffer
[mysql] 2020/01/12 20:17:11 packets.go:393: busy buffer

Let me know if I'm doing anything improperly or if you need any more information.

Thanks!

savepoint "tx_17" does not exist

I get this error running my test suite.

It can occur in different tests but it is always tx_17.

Is there some hardcoded limit for 16 savepoints somewhere?

Occasional sql: transaction has already been committed or rolled back errors

We are seeing transaction has already been committed or rolled back errors if we try to use the database "too quickly" after initializing it, with a cancel context.

Code looks like this:

func init() {
	dbURL := os.Getenv("DATABASE_URL")
	if dbURL == "" {
		log.Fatal("You must provide a DATABASE_URL environment variable")
	}

	txdb.Register("txdb", "postgres", dbURL)
}


func DBWithDefaultContext(db *gorm.DB, fc func(db *gorm.DB) error) error {
	ctx, cancel := context.WithTimeout(context.Background(), 10 * time.Second)
	defer cancel()
	return fc(db.WithContext(ctx))
}

func Test_example(t *testing.T) {

db, err := sql.Open("txdb", uuid.NewV4().String())
require.NoError(t, err)
t.Cleanup(func() { assert.NoError(t, db.Close()) })
gormDB, err := gorm.Open(postgres.New(postgres.Config{
	Conn: sqlDB,
	DSN:  uuid.NewV4().String(),
}), &gorm.Config{})
require.NoError(t, err)

err = DBWithDefaultContext(gormDB, func(db *gorm.DB) error {
	return db.Order("created_at ASC, address ASC").Where("deleted_at IS NULL").Find(&keys).Error
})

err := db.Exec(`SELECT 1`).Error // ERROR: sql: transaction has already been committed or rolled back
}

Note that this error about "transaction already committed or rolled back" has nothing to do with postgres (actually the transaction never gets closed). The postgres logs look like this:

2021-07-15 10:05:52.719 BST [63393] LOG:  statement: ;
2021-07-15 10:05:52.720 BST [63393] LOG:  statement: BEGIN READ WRITE
2021-07-15 10:05:52.720 BST [63393] LOG:  statement: SELECT * FROM "keys" WHERE deleted_at IS NULL AND "keys"."deleted_at" IS NULL ORDER BY created_at ASC, address ASC
// ENDS HERE

It must be a timing issue because it can be resolved by inserting a sleep or gosched here:

db, err := sql.Open("txdb", uuid.NewV4().String())
require.NoError(t, err)
t.Cleanup(func() { assert.NoError(t, db.Close()) })
runtime.Gosched() // FIXES THE TEST
// or
time.Sleep(1 * time.Millisecond) // FIXES THE TEST
// ... rest of the test code

Note that I was running this test with a parallelism flag of 1 - single threaded mode.

UPDATE:

It seems that calling runtime.Gosched() only sometimes fixes it - not always. Calling db.Exec('SELECT 1') in place of that seems to be a more reliable fix.

New release

Hi, @l3pp4rd

Is master stable or it needs anything before releasing a new version?

ALTER TABLE table_name AUTO_INCREMENT does not rollback inserted entry

Hi,

We have a test that executesALTER TABLE table_name AUTO_INCREMENT = 2, and then inserts a row in table_name. However, we find that the inserted row is still in the table_name after the test is done. If we don't do the alter, then the entry gets deleted, but ofcourse, the ids are incremented.

We are using mysql.

test with gorm return error: Error 1146: Table '*.users' doesn't exist

I'm trying to test gorm with go-txdb, but always get error.

code

type User struct {
	gorm.Model
	UserId int64  `json:"user_id,omitempty" gorm:"unique_index:uid_name_service_id"`
	ServiceId   int64  `json:"service_id,omitempty" gorm:"unique_index:uid_name_service_id"`
	UnitName    string `gorm:"unique_index:uid_name_service_id"`
}

func TestUserDao_GetUser(t *testing.T) {
	// dsn serves as an unique identifier for connection pool
	txdb.Register("txdb", "mysql", "root:123456@/infosec")
	db, err := sql.Open("txdb", "root:123456@/infosec")
	if err != nil {
		log.Fatal(err)
	}
	defer db.Close()

	DB, err := gorm.Open("mysql", db)
	if err != nil {
		t.Fatal(err)
	}
	var u User

	findDb := DB.Model(&User{}).First(&u)
	err = findDb.Error
	t.Log(u, err)
}

log:

=== RUN   TestUserDao_GetUser
    TestUserDao_GetUser: user_test.go:30: {{0 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC <nil>} 0 0 } Error 1146: Table 'infosec.users' doesn't exist

(/Users/snm/dev/gomod/infosec/model/user_test.go:28) 
[2020-05-26 17:48:03]  Error 1146: Table 'infosec.users' doesn't exist 
--- PASS: TestUserDao_GetUser (0.03s)
PASS

I don't understand why it keeps looking for Table 'infosec.users'.

go-txdb does not satisify the 'driver.Execer' interface

This leads to problems with some statements in some databases. For instance, attempting to execute a LOCK TABLES in MySQL fails with the following error:

Error 1295: This command is not supported in the prepared statement protocol yet

This appears to be due to go-txdb not fulfilling the Execer interface, which forces the sql library to fallback to using a prepared statement which does not work in this case.

I imagine many other statements, such as CREATE TABLE, DROP TABLE, and ALTER TABLE would likely have similar restrictions on a number of databases.

(NOTE: I realize that LOCK TABLES within a transaction in MySQL doesn't really work, but I think that's beyond the scope of this library)

txDriver.realConn is not closed.

In this commit, txDriver.realConn holds a new connection, but it's not closed anywhere.

c0f89b5

We got Too many connections in our tests with the latest go-txdb.

TODO: make separate project for sqlite3 safe concurrency sql driver

currently all sqlite3 drivers I've found do not provide efficient concurrency support using go standard sql driver interface. This may be done by wrapping sqlite3 sql driver under a concurrency safe driver implementation, would work for any of those libraries. (though in general should be very similar to txdb logic)

Auto increment values does not reset completely

When running tests though auto increment values get cleared after closing the db connection, next time the value of auto increment field is the continuation of the previous one.

this issue is found in postgres:9.6.11-alpine

Can´t get it to work with Gorm V2.

Hello.

I am trying to use txdb in my database integration tests together with Gorm V2, but it´s not going well.

Here is my test code:

func TestPgRepository_CreateTodo(t *testing.T) {
	db := testutil.GetTestDBConnection()

	defer testutil.CloseDb(db)

	repo := todo.NewPgRepository(db)

	td, err := repo.CreateTodo(todo.Todo{
		Description: "some-todo",
		CreatedAt: time.Now(),
	})

	assert.Nil(t, err)
	assert.NotNil(t, td.ID)
}

func TestPgRepository_FindAll(t *testing.T) {
	db := testutil.GetTestDBConnection()

	defer testutil.CloseDb(db)

	repo := todo.NewPgRepository(db)

	// Create test data
	r := db.Create(&todo.GormTodo{
		Description: "TODO 1",
		CreatedAt: time.Now(),
	})

	fmt.Println(r.Error)
	result, err := repo.FindAll()

	assert.NotNil(t, err)
	assert.Len(t, result, 1)
}

When testing the FindAll method it returns 2 records, so the transaction is not being rollback.

And here is the code of my GetTestDBConnection

func GetTestDBConnection() *gorm.DB {

	dbHost := os.Getenv("DB_HOST")
	dbPort := os.Getenv("DB_PORT")
	dbUser := os.Getenv("DB_USER")
	dbPassword := os.Getenv("DB_PASSWORD")
	dbName := GetTestDBName()

	dsn := fmt.Sprintf("host=%v user=%v password=%v dbname=%v port=%v sslmode=disable",
		dbHost,
		dbUser,
		dbPassword,
		dbName,
		dbPort,
	)

	once.Do(func() {
		txdb.Register("tx_test", "postgres", dsn)
	})

	db, err := gorm.Open(gormPG.Open(dsn), &gorm.Config{})

	if err != nil {
		panic(err)
	}

	return db
}

Gorm changed the Open method signature, so now has to receive a Dialect struct instead the driver name. The only link between txdb and GORM in this example is the dsn. not sure if it´s enough.

I tried to use a persistent connection instead of creating a new one for each est but then I can´t close the connection and thus the transaction are not rollback, otherwise the subsequence tests will complain the connection is closed

I also tried to register a txdb for each test with a random name to avoid conflicts but it also doesnt seem to work. It should IMO, since each test is opening and closing a connection, so I guess it´s really GORM not using txdb correctly.

Any ideas?

Thank you

DATA RACE in concurrent testing

WARNING: DATA RACE
Read at 0x00c000081380 by goroutine 63:
  github.com/DATA-DOG/go-txdb.(*txDriver).Open()
      /home/runner/work/coinex_push_backend/coinex_push_backend/vendor/github.com/DATA-DOG/go-txdb/db.go:145 +0x17d
  database/sql.dsnConnector.Connect()
      /opt/hostedtoolcache/go/1.16.9/x64/src/database/sql/sql.go:707 +0x8c
  database/sql.(*dsnConnector).Connect()
      <autogenerated>:1 +0x2e
...........
Previous write at 0x00c000081380 by goroutine 65:
  github.com/DATA-DOG/go-txdb.(*conn).Close()
      /home/runner/work/coinex_push_backend/coinex_push_backend/vendor/github.com/DATA-DOG/go-txdb/db.go:181 +0xda
  database/sql.(*driverConn).finalClose.func2()
      /opt/hostedtoolcache/go/1.16.9/x64/src/database/sql/sql.go:592 +0x8b
  database/sql.withLock()
      /opt/hostedtoolcache/go/1.16.9/x64/src/database/sql/sql.go:3294 +0x7e
  database/sql.(*driverConn).finalClose()

First, thanks for your great code library.

When I testing my code in concurrent mode, I found a data race warning as above.

After some research, I found the issue appear in the Open() method as below:

	c, ok := d.conns[dsn]
	if !ok {
		c = &conn{dsn: dsn, drv: d, savePoint: &defaultSavePoint{}}
		for _, opt := range d.options {
			if e := opt(c); e != nil {
				return c, e
			}
		}
		d.conns[dsn] = c
	}
	c.opened++
	return c, nil

which connections have not been protected by a lock, so I write this issue to report that.

And, a PR also been created for this.

DB CURRENT_TIMESTAMP is always the same

Hello, thanks for the library!

I have been using it in tests for some time but today I noticed a strange behaviour.
In one of my test I insert 3 records one-by-one with created_at set by DB as DEFAULT CURRENT_TIMESTAMP.
Later I select those records with ORDER BY created_at. And records order is always messed up.
I investigated it a little bit and I noticed that all records have the same created_at value whereas with read DB it differs in few milliseconds.

I ended up with code snippet which demonstrates the problem:

func main() {
	const dsn = "postgres://postgres@localhost/test_db?sslmode=disable"
	txdb.Register("txdb", "postgres", dsn)

	db, err := sql.Open("txdb", "anywordhere")
	//db, err := sql.Open("postgres", dsn)
	if err != nil {
		log.Fatal(err)
	}

	probe(db)
	time.Sleep(1 * time.Second)
	probe(db)
}

func probe(db *sql.DB) {
	var ts time.Time
	err := db.QueryRow(`SELECT now()`).Scan(&ts)
	if err != nil {
		log.Fatal(err)
	}

	fmt.Println(time.Now(), ts)
}

# => 2019-09-24 19:13:08.429564 +0300 MSK m=+0.009807600 2019-09-24 16:13:08.432757 +0000 UTC
# => 2019-09-24 19:13:09.436596 +0300 MSK m=+1.016829028 2019-09-24 16:13:08.432757 +0000 UTC

As you can see DB time is always the same.

Could you please help me to figure out what is going on? Is there any cache?
Thank you.

Tag a release

Would be nice to have a tagged release. :)

Ping ends up in error

When trying to Ping the database to check if there is a connection I receive the following error:

https://pastebin.com/6bnAztAk

Looks like Ping calls itself recursively: https://github.com/DATA-DOG/go-txdb/blob/master/db_go18.go#L96-L98

At least it could return with nil, even if it is not implemented for the moment.

hangs when same primary key used in different tests

When you insert the same primary key in two different transactions your tests will hang.

With the code below, the following happens

A inserting
A inserted
A db closed
B inserting <-- hangs here

db schema

-- CREATE DATABASE example;
CREATE TABLE example_table (
  name text PRIMARY KEY
);

test file example_test.go

package main

import (
	"database/sql"
	"fmt"
	"log"
	"testing"
	"time"

	txdb "github.com/DATA-DOG/go-txdb"
	_ "github.com/jackc/pgx/stdlib"
)

var (
	dsn    = "postgres://postgres@localhost/example?sslmode=disable"
	query  = "insert into example_table (name) values ('hello world')"
	driver = "pgx"
)

func init() {
	txdb.Register("txdb", driver, dsn)
}

func TestA(t *testing.T) {
	// setup
	db, err := sql.Open("txdb", "A")
	if err != nil {
		panic(err)
	}
	defer func() {
		db.Close()
		fmt.Println("A db closed")
	}()

	// insert
	fmt.Println("A inserting")
	_, err = db.Query(query)
	if err != nil {
		t.Fatal(err)
	}
	fmt.Println("A inserted")
}

func TestB(t *testing.T) {
	// setup
	db, err := sql.Open("txdb", "B")
	if err != nil {
		panic(err)
	}
	defer func() {
		db.Close()
		fmt.Println("B db closed")
	}()

	// insert
	fmt.Println("B inserting")
	_, err = db.Query(query)
	if err != nil {
		panic(err)
	}
	fmt.Println("B inserted")
}

I have tried this code with the postgres drivers _ "github.com/jackc/pgx/stdlib" and _ github.com/lib/pq

Closing prepared statement after error in nested transaction prevents transaction rollback in v0.1.6

First of all, thank you for go-txdb! It's been a huge help for us testing our project.

v0.1.6 introduced some new behavior that I believe to be a bug. The following test passes on v0.1.5 but not v0.1.6. When using postgresql, it fails. It seems closing a prepared statement automatically rolls back the nested transaction?

I took a look at the changes in v0.1.6 and wasn't able to untangle the behavior, but I was hoping this test case might help someone else figure it out.

Please let me know if I can provide any more information.

func TestShouldAllowClosingPreparedStatementAndRollback(t *testing.T) {
	for _, driver := range drivers() {
		db, err := sql.Open(driver, "rollback")
		if err != nil {
			t.Fatalf(driver+": failed to open a connection, have you run 'make test'? err: %s", err)
		}
		defer db.Close()

		// do a query prior to starting a nested transaction to
		// reproduce the error
		var count int
		err = db.QueryRow("SELECT COUNT(id) FROM users").Scan(&count)
		if err != nil {
			t.Fatalf(driver+": prepared statement count err %v", err)
		}
		if count != 3 {
			t.Logf("Count not 3: %d", count)
			t.FailNow()
		}

		// start a nested transaction
		tx, err := db.Begin()
		if err != nil {
			t.Fatalf(driver+": failed to start transaction: %s", err)
		}
		// need a prepared statement to reproduce the error
		insertSQL := "INSERT INTO users (username, email) VALUES(?, ?)"
		if strings.Index(driver, "psql_") == 0 {
			insertSQL = "INSERT INTO users (username, email) VALUES($1, $2)"
		}
		stmt, err := tx.Prepare(insertSQL)
		if err != nil {
			t.Fatalf(driver+": failed to prepare named statement: %s", err)
		}

		// try to insert already existing username/email
		_, err = stmt.Exec("gopher", "[email protected]")
		if err == nil {
			t.Fatalf(driver + ": double insert?")
		}
		// The insert failed, so we need to close the prepared statement
		err = stmt.Close()
		if err != nil {
			t.Fatalf(driver+": error closing prepared statement: %s", err)
		}
		// rollback the transaction now that it has failed
		err = tx.Rollback()
		if err != nil {
			t.Logf(driver+": failed rollback of failed transaction: %s", err)
			t.FailNow()
		}
	}
}

Breaks go's sql driver semantics

https://golang.org/pkg/database/sql/driver/#Driver says that Open's returned connection is only used by one goroutine at a time, but because the txdb Driver returns the same connection multiple times, you can end up in a situation where the same connection is used in multiple concurrent threads of execution.

The only correct thing to do is to return an error/panic if another connection is opened while the connection is already being used. It points to a programming error, for example an sql.Rows not being closed, or the *sql.DB being used while a transaction has started.

Is it possible to run a transaction within txdb?

Hello there,

I'm trying to use your driver together with the underlying mysql driver, a mysql database and sqlx. Unfortunately whenever I create a transaction in my production code, it fails tx.Commit() or tx.Rollback(), usually with the error saying that the save point does not exist for tx_0 for example. Is it possible to run transactions within a (hidden) transaction started by your txdb driver?

Thanks!

postgres PrepareContext returns a nil `sql.Stmt` pointer

Issue

I am trying to test some code that uses a prepared statement (using postgres/sqlx), small example below.

The code is running in to nil pointer exceptions stemming from a nil statement pointer.

Is this a known shortcoming of lib? If so, are there any common workarounds?

Potentially related to #3?

Definitely happy to contribute if I can get pointed in a direction :)

Example

func init() {
	txdb.Register("pgx", "postgres", "my connection string")
}

func NewTestDB(t *testing.T) *sqlx.DB {
	db, err := sqlx.Open("pgx", t.Name())
	if err != nil {
		t.Fatal(err)
	}
        return db
}

func TestFoo(t *testing.T) {
       db := NewTestDB(t)
       stmt, err := db.PrepareContext(ctx, pq.CopyInSchema(
             "someschema", "sometable", "somecolumn"
       ))
      // stmt is nil here, no err
}

Is there any way to support intended rollback?

Background

I am using go-txdb for unit tests in my project, and I expect that I can keep the same status of my database before running each test case. Currently it seems like that go-txdb rollback only when the sql.DB intends to close all driver connections so that it can not meet my requirement.

I have read the code in this repository but can not find any function or method to solve my problem. Have I missed something? If not, I think I can provide a PR to support this.

Proposal: manual rollback support

Besides original rollback strategy, we can add a new method of txdb.txDriver, for example, txDriver.ManualRollback(dsn string), and extract a new method of txdb.conn, for example, conn.Rollback.

txDriver.ManualRollback is responsible for searching the corresponding conn and then delegate the request to conn.Rollback method.

In conn.Rollback, it just needs to do the below 2 steps:

1. Rollback the underlying db connection;
2. reset the tx field of conn.
   That is, it does exactly like the partial of the original conn.Close method, see

   go-txdb/db.go

   Lines 183 to 186 in 8216b38

   	if c.tx != nil {
   	c.tx.Rollback()
   	c.tx = nil
   	}

Besides, we need to make txdb.Register returns the txdb.txDriver instance to the caller so that the latter is possible to call its ManualRollback method.

Savepoint does not exists if using several transactions

In my project we prepare several different transactions before executing and committing them

func MultipleTx() {
	db, _ := sql.Open("postgres", "dsn")
	
	txs := make([]*sql.Tx, 0)
	
	for _ = range []string{"p", "o", "s", "g", "e", "s"} {
		tx, _ := db.Begin()
		txs = append(txs, tx)
		tx.Exec("some query")
	}
	
	...
	
	for _, tx := range txs {
		tx.Commit()
	}
}

It works in production, but fails in tests with error savepoint "tx_2" does not exist.

Here is test example

func TestPostgresMultipleTx(t *testing.T) {
	// make sure drivers are registered first
	driver := drivers()[1]
	db, err := sql.Open("psql_txdb", "multiple_tx")
	if err != nil {
		t.Fatalf("psql: failed to open a postgres connection, have you run 'make test'? err: %s", err)
	}
	defer db.Close()

	var count int

	tx, err := db.Begin()
	if err != nil {
		t.Fatalf(driver+": failed to begin transaction: %s", err)
	}
	otherTX, err := db.Begin()
	if err != nil {
		t.Fatalf(driver+": failed to begin transaction: %s", err)
	}

	{
		_, err = tx.Exec(`INSERT INTO users (username, email) VALUES('txdb', '[email protected]')`)
		if err != nil {
			t.Fatalf(driver+": failed to insert an user: %s", err)
		}
		err = tx.QueryRow("SELECT COUNT(id) FROM users").Scan(&count)
		if err != nil {
			t.Fatalf(driver+": failed to count users: %s", err)
		}
		if count != 4 {
			t.Fatalf(driver+": expected 4 users to be in database, but got %d", count)
		}

	}

	err = db.QueryRow("SELECT COUNT(id) FROM users").Scan(&count)
	if err != nil {
		t.Fatalf(driver+": failed to count users: %s", err)
	}
	if count != 4 {
		t.Fatalf(driver+": expected 4 users to be in database, but got %d", count)
	}

	{
		_, err = otherTX.Exec(`INSERT INTO users (username, email) VALUES('other txdb', '[email protected]')`)
		if err != nil {
			t.Fatalf(driver+": failed to insert an user: %s", err)
		}
		err = otherTX.QueryRow("SELECT COUNT(id) FROM users").Scan(&count)
		if err != nil {
			t.Fatalf(driver+": failed to count users: %s", err)
		}
		if count != 5 {
			t.Fatalf(driver+": expected 5 users to be in database, but got %d", count)
		}
	}

	if err = tx.Commit(); err != nil {
		t.Fatalf(driver+": failed to rollback transaction: %s", err)
	}
	if err = otherTX.Commit(); err != nil {
		t.Fatalf(driver+": failed to commit transaction: %s", err)
	}

	err = db.QueryRow("SELECT COUNT(id) FROM users").Scan(&count)
	if err != nil {
		t.Fatalf(driver+": failed to count users: %s", err)
	}
	if count != 5 {
		t.Fatalf(driver+": expected 5 users to be in database, but got %d", count)
	}
}

Use testcontainers (or similar solution) for tests

The current test suite depends on locally running MySQL and PostgreSQL instances, and fails rather spectacularly when it cannot connect:

    db_test.go:322: mysql_txdb: could not prepare - Error 1698 (28000): Access denied for user 'root'@'localhost'
--- FAIL: TestShouldReopenAfterClose (0.00s)
    db_test.go:387: mysql_txdb: could not prepare - Error 1698 (28000): Access denied for user 'root'@'localhost'
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
        panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x545f18]

I suggest we use go-testcontainers, or a similar tool, to run these tests so to ease the maintenence and writing of tests.

Postgres Example

I have tried everything and for the life of me I cannot get Postgres to work. Can I get an example for connecting using postgres?

Thanks!

Deadlock found when trying to get lock

For some number of months I've been encountering intermittent deadlock errors while using txdb in tests and I want to open an issue to see if anyone else has encountered this issue and might be able to shed light on this.

Error 1213: Deadlock found when trying to get lock; try restarting transaction

Unfortunately I don't have a reproducible test case, but it is a regular issue. Tests recently moved to run on a faster host and the rate of this issue increased.

Consider supporting the sql.Connector interface

If this package were to satisfy the driver.Connector interface, it would no longer be necessary to register a driver for every new instance, which could make certain test patterns much simpler.

I'd be happy to submit a PR if it's likely to be accepted.