go-testfixtures / testfixtures Goto Github PK

I created the following yml file and ran it, but I am getting an error

users.yml

- id: 1
  warehouse_ids:
    - 1
    - 2

error

&testfixtures.InsertError{
  Err: &pq.Error{
    Severity:         "ERROR",
    Code:             "22P02",
    Message:          "malformed array literal: \"[1,2]\"",
    Detail:           "Missing \"]\" after array dimensions.",
    Hint:             "",
    Position:         "",
    InternalPosition: "",
    InternalQuery:    "",
    Where:            "",
    Schema:           "",
    Table:            "",
    Column:           "",
    DataTypeName:     "",
    Constraint:       "",
    File:             "arrayfuncs.c",
    Line:             "292",
    Routine:          "array_in",
  },
   SQL:    "INSERT INTO \"users\" (\"id\", \"warehouse_ids\") VALUES ($1, $2)",
 Params: []interface {}{
    1,
    "[1,2]",
  },
}

How can I solve this?

Hi there,
my postgres is version 10.10 and my golang is 1.15.
I used the following config to initial fixtures,

f, err := testfixtures.New(
	testfixtures.Database(db),
	testfixtures.Dialect("postgresql"),
	testfixtures.Directory("../fixtures"),
	testfixtures.DangerousSkipTestDatabaseCheck(),
	testfixtures.UseDropConstraint(),
)

with UseDropConstraint still, I'm getting foreign key constraint

17:37:27 XXXXXXXXX.go:412: testfixtures: error inserting record: ERROR: insert or update on table "XXXXXXXXX" violates foreign key constraint "XXXXXXXXX_fk" (SQLSTATE 23503)
( X are masked character)

any thought and idea why it's not applied and I see the error.

Thanks

In our use-case we have some common test data that we want in a table named "feature" for example.

If we load multiple files named 'feature.yml' from multiple paths, only the data from the last one added is present.

Instead I would have expected the data from both files to be aggregated before being inserted.

Environment

• PostgreSQL 10
• testfixtures 2.4.3
• Go 1.8.3

Error message

2017/12/20 19:43:41 pq: update or delete on table "users" violates foreign key constraint "environments_created_by_users_id_foreign" on table "environments"

SUPERUSER + &testfixtures.PostgreSQL{} => Success
SUPERUSER + &testfixtures.PostgreSQL{UseAlterConstraint: true} => Failure
USER + &testfixtures.PostgreSQL{} => Failure
USER + &testfixtures.PostgreSQL{UseAlterConstraint: true} => Failure

It seems ALTER CONSTRAINT is ineffective.
Do you have any information?

Reference: #30 (comment)

We can move from the .sql file to an YAML file (since it supports arrays for multiple statements) or to a Go function/interface

For some testing scenarios it can be handy to generate a binary that is able to populate it's own test database. A prerequisite for this is to bundle the fixture files with the binary. One possible way to do so is packr. But there is no straightforward way to make the bundled files available for the testfixtures package.

Luckily packr implements http.FileSystem in order to represent the bundled files in the binary. I therefore suggest to add the following functions to testfixtures:

func NewFileSystem(db *sql.DB, helper Helper, fs http.FileSystem) (*Context, error) 

func LoadFixtureFileSystem(db *sql.DB, helper Helper, fs http.FileSystem) error

This would allow testfixtures to work together with packr or other sources for fixture files that implement the http.FileSystem interface.

Originally asked in #23, but I've being thinking about it for some time.

Today, this lib cleans the database, load everything and then commit the changes for every test run. Rails has an option called "transactional fixtures" (or "transactional tests"), where fixtures are loaded once before all tests, and then each test runs inside a transaction. This would allow concurrent running of tests that touch the database.

Needs investigation on implementation and the better public interface to expose this.

Ideas and help would be very much appreciated.

It would be better if you can support the following format.

-
  id: 1
  slice_of_struct:
     -
       name: myname1
       value: myvalue1
     -
       name: myname2
       value: myvalue2

I generated the fixtures using GenerateFixtures API with my postgres development database. Got the fixtures under the testdata/fixtures folder.

After that I have been trying to load those fixtures into a test database and run my tests but fixtures.Load() call is failing as I don't see any relations in the database and thus all my tests are failing.

Do suggest if I am missing anything.

I get the following error when using "go get ":

can't load package: package github.com/go-testfixtures/testfixtures: code in directory /home/im
ad/.go/src/github.com/go-testfixtures/testfixtures expects import "github.com/go-testfixtures/t
estfixtures/v3"

Now when importing data, the data in the relevant table will be deleted. This approach is a bit rude (for example, some data I don’t want to delete or others want to use the data), can it be modified to delete the data that conflicts with the imported data (For example, specify a single primary key？). I know this is a bit more complicated than deleting the entire table, but generally speaking, we expect to delete this part of the data that we imported.

@andreynering and contributors, this is a dope project, thanks for working on it!

I have a test database with two schemas:

1. public

• table_a

2. exampleschema

• table_b

Currently, postgresql.go searches for sequences to reset via

"SELECT relname FROM pg_class WHERE relkind = 'S'"

(https://github.com/go-testfixtures/testfixtures/blob/master/postgresql.go#L86-L107)

Unfortunately, when I try to load fixtures via fixtures.Load(), this error crops up

momer@ubuntu:~/.golang/src/my-domain/my-company/myproject$ go test -v
{"level":"info","msg":"Successfully opened DB Connection","time":"2017-07-19T14:06:01-05:00"}
=== RUN   TestInsertIntoTableA
{"level":"fatal","msg":"pq: relation \"schema_namespaced_table_b_id_seq\" does not exist","time":"2017-07-19T14:06:01-05:00"}
exit status 1
FAIL    my-domain/my-company/myproject   0.039s

with this output on PostgreSQL's stdout

postgres    | ERROR:  relation "schema_namespaced_table_b_id_seq" does not exist at character 15
postgres    | STATEMENT:  SELECT SETVAL('schema_namespaced_table_b_id_seq', 10000)

I've put together a pull request which resolves these issues, and will comment it below! Test suite for PostgreSQL still passes via go test -tags postgresql

Hi,

running the following command results in this weird error:

err = testfixtures.LoadFixtures(FixturesPath, db.DB(), &testfixtures.PostgreSQLHelper{})
if err != nil {
    log.Fatal(err)
}

2016/07/22 22:20:43 pq: permission denied: "RI_ConstraintTrigger_a_19013" is a system trigger

Any ideas?

This is an issue to track changes to be made in v3.

Work in progress: #57

Breaking changes

• Move from gopkg.in/testfixtures.v2 to github.com/go-testfixtures/testfixtures/v3
• Remove Oracle support
• Rewrite the public API to use functional options

New features

• Add option to choose a specific time zone to parse date/time objects. On the absence of this option, testfixtures should use time.Location as it does today
• Have and CLI and publish it using GoReleaser (#54)
• [Maybe?] Implement transaction fixtures (#24)
• [Maybe?] Allow templating in YAML files (#25)

I want to use like ERB/Ego to create large data.

<% 1000.times do |n| %>
user_<%= n %>:
  username: <%= "user#{n}" %>
  email: <%= "user#{n}@example.com" %>
<% end %>

Hi! In YAML i configured an object

- order_code: 123456
  created_at: 2020-11-19 12:29:10
  updated_at: 2020-11-19 12:38:36
  data:
      AfterUpdateData: 
        PickupTimestamp: 2020-11-24T08:34:25.46251+03:00
        AtClientTimestamp: 2020-11-19T15:38:35.806412+03:00
        AtVendorTimestamp: 2020-11-24T08:34:25.46251+03:00
      BeforeUpdateData: 
        PickupTimestamp: 2020-11-24T08:34:25.46251+03:00
        AtClientTimestamp: 2020-11-24T08:34:25.46251+03:00
        AtVendorTimestamp: 2020-11-24T08:34:25.46251+03:00

Appliyng of this fixture leads to an error

testfixtures: error inserting record: pq: unsupported jsonb version number 123, on file: delivery.yml, index: 0, sql: INSERT INTO "delivery" ("order_code", "created_at", "updated_at", "data") VALUES ($1, $2, $3, $4), params: [123456 2020-11-19 12:29:10 +0300 MSK 2020-11-19 12:38:36 +0300 MSK map[AfterUpdateData:map[AtClientTimestamp:2020-11-19T15:38:35.806412+03:00 AtVendorTimestamp:2020-11-24T08:34:25.46251+03:00 PickupTimestamp:2020-11-24T08:34:25.46251+03:00] BeforeUpdateData:map[AtClientTimestamp:2020-11-24T08:34:25.46251+03:00 AtVendorTimestamp:2020-11-24T08:34:25.46251+03:00 PickupTimestamp:2020-11-24T08:34:25.46251+03:00]]]

Same problem described here lib/pq#528 . The solution - to cast []byte or json.RawMessage to string before inserting. Is it possible?

Currently the only time sequences are modified is after fixtures are loaded. I propose that, in addition to the current behavior, to set sequences to 1 before loading fixtures.

When omitting the serial columns from the fixture files, subsequent test runs result in fixture records with serial values greater than ResetSequencesTo. Thus when the sequences are reset, it is reset to a value before the fixture records.

That was confusing to type. Example, run 1:

  id   |       name        
-------+-------------------
     1 | 1) fixture
 10000 | 2) from test
 10001 | 3) also from test

Run 2:

  id   |       name        
-------+-------------------
 10002 | 1) fixture
 10000 | 2) from test
 10001 | 3) also from test

As soon as you increase the number of rows generated by a given run, conflict.

I've worked around this by specifying ids in the fixture files. This is probably a good practice anyway, and the current behavior acts as a reasonable forcing function. An alternative fix for this issue might be simply documenting a best practice of specifying values for serial columns.

A robust fix would be to take the max of the serial columns, though that's a bit more involved. The query in this question would be a good starting point though: https://dba.stackexchange.com/questions/47098/how-do-i-determine-if-a-column-is-defined-as-a-serial-data-type-instead-of-an-in

TLDR, please do one of:

1. Reset sequences to 1 before loading fixtures then to ResetSequencesTo afterwards.
2. Update docs to recommend that all serial columns have values and that they be less than ResetSequencesTo
3. At least in the case of postgres, set ResetSequencesTo automatically using the result of select max(<serial column>)+1 from <schema>.<table>.

Sometimes it is need to make db schema initialization before texture uploading.
And this method will be helpful to unify database name checking.

Can this product truncate table before insert yaml?
Now table's auto increments is not cleared.
I want reset auto increment value when insert yaml.

I'm using mysql.

thanks.

I use SQLite3 to store small binary data (data type blob). Call GenerateFixtures() and obtain a .yml file like following:

- type: 0
  value:
  - 8
  - 128
  - 8
  - 16
  - 13
...

When I use Load() to read this *.yml file and overwrite the whole DB entries, the values are not the same as they were when I generated the fixtures. Except for blob, other data types remain intact.

Before GenerateFixtures():

INSERT INTO logs VALUES(2,1024,1,X'088008100d18a4f3f488a8c3aad015','2019-05-21 17:55:01.882239396+08:00',NULL,0,0);

After Load():

INSERT INTO logs VALUES(2,1024,1,X'5b382c3132382c382c31362c31332c32342c3136342c3234332c3234342c3133362c3136382c3139352c3137302c3230382c32315d','2019-05-21 17:55:01.882239396+08:00',NULL,0,0);

I believe the inconsistency is caused by json.Marshal in jsonArray's receiver function Value(). It will automatically convert []int to marshaled []byte, but the result is unexpected.

An easy workaround is like this:

func (a jsonArray) Value() (driver.Value, error) {
    var arr []byte
    for _, v := range a {
        arr = append(arr, byte(v.(int)))
    }
    return arr, nil
}

However, it just considers int array case.

Any suggestions?

HI,
I'm looking to use templates fixtures but can't figure out how it works.
The given example is not extremely explicit.

I am currently using two fixture files

{{range $u := $.Users}}
  - username: {{$u.username}}
{{end}}

{{range $a := $.Articles}}
- title: {{$a.title}}
  content: {{$a.content}}
{{end}}

And trying to load it like

testfixtures.New(
		testfixtures.Template(),
                (...)
		testfixtures.TemplateDelims("{{", "}}"),
		testfixtures.TemplateData(map[string][]map[string]string {
			"Users": {{"username": "test"}, {"username": "test2"}},
			"Articles": {{"title": "test", "content": "test"}, {"title": "test2", "content": "test2"}},
		}),
		)

I get the error testfixtures: fixture is not a slice or map. I'm not too sure what data structure I should put in to allow to load slices of maps of data.
Thanks

I would like to change time.Location of date, but cannot change because of this code.

So, I fixed my yaml file the following temporary.

-  datetime: 2018-01-01 18:00:00
+  datetime: RAW="2018-01-01 18:00:00"

Is there other solution?

It's available for PG, not sure for other databases:

https://www.postgresql.org/docs/9.6/static/sql-truncate.html

• Homebrew
• GoDownloader

load mongo data from yaml

I've run into a situation where my build fails depending on the order of the tests and the nature of their foreign key relationships. I am using the UseAlterConstraint method with Postgres, and not running tests in parallel.

For example:

Test 1 creates fixtures for Table A & Table B. Table A has a record which references some record in Table B.
Test 2 only creates fixtures in Table B. Because there is leftover data in table A from test 1, now test 2 fails when testfixtures tries to clear Table B.

What I had to do to solve this is to copy the logic to make the constraints deferrable (I am using Postgres). Then I can do a teardown at the end of each test where I delete all the fixture data.

So what I am looking for is a method something like:
fixtures.Unload()

This would remove everything created by fixtures.Load(), to avoid potential conflicts with subsequent tests.

Hi,

I'm using Gorm as ORM and I wanted to know if I could test my postgres queries in sqlite with testfixtures ?

Thanks for this nice plugin !

Hi
I'm struggling to find out how can I use a PostGIS field in my fixtures. For example, for updating my model I need to run this command:

update my_table set polygon = ST_GeomFromText("SRID=4326; POLYGON ((11 22, 33 44, 55 66))") where id = 1;

and in my fixtures I have:

- id: 1
  polygon: ST_GeomFromText("SRID=4326; POLYGON ((11 22, 33 44, 55 66))")

But in the SQL command it executes, it wraps ST_GeomFromText... in quotations, which results in error.
How can I tell it not to wrap some value in quotations?

Thanks.

Whenever I add to a fixture a map or array, I get an error like:

"sql: converting Exec argument #3's type: unsupported type map[interface {}]interface {}, a map"

Any chance we can support JSONB?

Ideally I want to be able to do

# TestUserService_OnePost.yaml
users:
    - id: 2
      name: Joe
posts:
    - id: 1
      user_id: 2
      content: "hi"

The example in the README shows you have to make one file per table, what I want is to have individual files for each test case which contain all the necessary data for each table as shown in the example. Currently I've been using https://github.com/romanyx/polluter which does the job but isn't maintained. I'd like to migrate to this eventually.

hello,
does it work with timescaledb?

load data to clickhouse
driver: https://github.com/kshvakov/clickhouse

Should this be "mysql"?

Somehow, primary key conflict occurs when sequence are used as primary key default value.
It happens when tests are run several time.
As workaround an option has been added to the postgres helper to no reset sequences #38
This issue is opened to track the root cause of the problem which have not been completely identified.

I ran into an issue recently where my database had run out of connections but I received an error from testfixtures that my database name didn't include "test." This is a bit misleading and took me some time to track down.

How do you feel about modifying the helper interface to return (string, error)? This matches other methods on the interface and while it would be a breaking change it could help folks track down errors more easily when they occur.

Happy to PR this if you think it's a sane move. Thanks for the library!

I like test fixtures but encountered a minor (documentation) problem:

When I use this:

...
testfixtures.Database(db.DB()),
testfixtures.Dialect("sqlite"),
testfixtures.Directory("testdata/fixtures"),
testfixtures.Template(),
testfixtures.TemplateFuncs(template.FuncMap{
	"now": now,
}),

I get an error:

testfixtures: could not unmarshal YAML: yaml: invalid map key: map[interface {}]interface {}{"now":interface {}(nil)}

When I move the line with Directory() down a bit, everything works fine:

testfixtures.Database(db.DB()),
testfixtures.Dialect("sqlite"),
testfixtures.Template(),
testfixtures.TemplateFuncs(template.FuncMap{
	"now": now,
}),
testfixtures.Directory("testdata/fixtures"),

I could not see this in the documentation. It might be worth mentioning?

Like

fixtures.Delete(), which will clean all the tables

Now, koad a fixtures with null yaml file will lead to "testfixtures: fixture is not a slice or map" error

I want a delegate method to clear the database

When trying to add fixtures to a Postgres database on AWS, I get the following error:
"Error adding fixture "pq: permission denied: "RI_ConstraintTrigger_a_26452" is a system trigger"

Evidently, Amazon reserves the "Superuser" role for their internal system. https://serverfault.com/questions/661661/why-cant-i-create-a-superuser-in-aws-postgresql-instance#answer-661685

Any suggestion on how to make this work with AWS?

Using Postgres I can connect to the database fine using sql.Open, but passing the variable to testfixtures produces this error:

dial tcp [::1]:5432: connect: connection refused

Here is my code:

dbConnectionString := os.Getenv("DATABASE_TEST_CONNECTION_STRING")
db, err := sql.Open("postgres", dbConnectionString)
if err != nil {
	log.Println("DEBUG: ERROR ", err)
}

_, err = testfixtures.NewFolder(db, &testfixtures.PostgreSQLHelper{}, "testdata")
 if err != nil {
     log.Println("DEBUG: ", err)
 }

Database: Postgres 9.6

I didn't want to make my test user role a superuser, so I tried the UseAlterConstraint approach.

When I was testing, I found that one of my tables was always empty, even though it should have been populated with my fixtures.

My guess is that ON DELETE CASCADE on my foreign keys is a causing this problem.

Example:

parent
 - id

child
  - id
  - parent_id (fkey with on delete cascade)

Now this is largely dependent on the order that the data is deleted and inserted. So if child is processed before parent, then the child table will be empty due to the fkey cascade.

DELETE FROM "child"
INSERT FIXTURES into "child"
DELETE FROM "parent" (A casade appears to delete from child)
INSERT FIXTURES into "parent"

Looking at the codebase, I wonder why the deletes aren't done together. Then after that perform all the necessary inserts? Is there a reason why we need to delete/insert for each table like this?

It would be great to have a command line interface with basic fixtures load/unload commands.

This would allow users to seed database using same fixtures and use them in development environment. It would be useful for manual testing or as demonstration data.

Go version: go1.11.2 linux/amd64
DB: MariaDB 5.5.60
testfixtures: v2.5.1

I'm running into some weird issues where it seems that func (c *Context) Load() either isn't blocking the tests from running until it finishes or silently errors out. In a couple of my tests I have a yaml file with 7 rows defined, sometimes running my tests everything works as expected the table is correctly cleaned of old data and the new data is populated and my tests passes. Other times only 1-2 rows may get loaded which causes my tests to be wrong due to the missing data.

I'm not too sure of an easy way to trace down what exactly is going on though that is causing this error.

Coming from the rails world I'm just baffled as to why this, the most popular golang database fixture package, only has 151 stars. Do gophers not test with databases? Is everybody just mocking database queries (which is insane)? Do they just not test?

Does it make sense to have the option of passing a sql.Tx so tests can be run in parallel? (and disable the automatic database cleaning - which, by the way, is greatly appreciated)

I'm currently using pgx (https://github.com/jackc/pgx) instead of pq I would like to use this driver/dialect in testfixtures, but it's not currently supported. Seems like I can just add pgx to the helperForDialect function. I tested it out and it seemed to work. It also passed the unit tests.

There was one slight difference that I noticed. In one of my fixtures I have a zip code string field, which I populate like this 11111. In postgres/pq it gets converted to a string automatically. But I noticed in pgx I need to explicitly make the zip code a string (e.g. '11111'). There may be more differences, but haven't tested it too extensively so far.

func helperForDialect(dialect string) (helper, error) {
	switch dialect {
	case "postgres", "postgresql", "timescaledb", "pgx":
		return &postgreSQL{}, nil
	case "mysql", "mariadb":
		return &mySQL{}, nil
	case "sqlite", "sqlite3":
		return &sqlite{}, nil
	case "mssql", "sqlserver":
		return &sqlserver{}, nil
	default:
		return nil, fmt.Errorf(`testfixtures: unrecognized dialect "%s"`, dialect)
	}
}

Some of my tests require that my queries return zero results because the DB is empty.

If I don't call load on my first test, I can get this behaviour, however on subsequent runs, due to having loaded in a later test, this test now fails (more than zero results) as the sqlite DB still exists.

It'd be helpful to have an unload/reset or a way to load in an empty table using the NewFiles for this purpose.

It doesn't appear this functionality is available. Can I suggest it as a feature?

Example (just add a singularized version of the existing LoadFixtures method)

testfixtures.LoadFixture("../fixtures/<my-table>.yml", db, &testfixtures.MySQLHelper{})