boostport / migration Goto Github PK

I enjoy working with your simple and easy migration library, thanks for creating it. 😃

However I ran into an issue where I had named my migrations like this:

1-init.up.sql

Instead of:

1_init.up.sql

This was a frustrating bug to fix because it took me a while to see that I had used a hyphen instead of an underscore. The hyphen doesn't match the regex the library uses to look for migrations. I think requiring an underscore is a good idea, but it would be helpful to docment the actual regex that the migration file names must match.

Hi,

I have tried to implement a db migration setup with this package - without success to this point.

I experimented a bit and for my setup it seems like the Driver Implementations in the ./driver package don't implement the Driver interface.

My code doesn't recognize the driver instance I created using the postgres.NewFromDB() method as a valid implementation of the driver interface.

When I open your code directly the driver implementations aren't recognized as valid implementations of the driver interface either. Whereas the mockDriver you implemented in the package root is recognized correctly.

I experimented a little bit and it seems like if I modify the driver interface slightly to explicitly import the package itself for the PlannedMigration parameter of the Migrate() method all interface implementations are recognized correctly.

To me this seems like a strange behaviour since there shouldn't be an issue with the original interface declaration and implementations as they are.
So the question is wether my setup is just misbehaving or if you want to adapt this change. Unfortunately this breakes the mockDriver implementation. So one would have to find a solution for that.

I still think that Intellij is getting something wrong. Please let me know if you have any advice on how to use this lib properly.

Kind regards,
Felix

Firstly, thank you for this wonderful library! We've been happily using it for over 2 years.

I have a feature request. Sometimes we have multiple instances of our application start a the same time, and they all try to run the same DB migration. Suffice to say, only one instance is successful and the others often crash. Worst case scenario, they could interfere with each other and break the migration.

I was thinking it would be useful to have an option to lock the schema_migration table in order to avoid such issues (or even do it by default). This would be the order of operations:

1. lock schema_migration
2. select version from the DB
3. plan migrations, apply migrations
4. insert new version
5. unlock schema_migration

This way if multiple applications try to run migrations at the same time, only one will get the lock. After the migration completes, the other ones can then get the lock, select versions, and realize that there's nothing to migrate anymore.

I'm thinking of either implementing locking in our applications or contributing the feature here, but I don't know when I'll have the time to do it. Perhaps someone else would like to implement it? For reference, we're using the postgres driver.

Experiencing some import problems with the package due to a malformed filepath in test-migrations/3_add_column.up..sql there are multiple dots.

This is the error message I get is the following:

C:\Users\felix\source\felixcolaci\worf\shared>go mod tidy
go: extracting github.com/Boostport/migration v0.12.0
-> unzip C:\Users\felix\go\pkg\mod\cache\download\github.com\!boostport\migration\@v\v0.12.0.zip: malformed file path "test-migrations/3_add_column.up..sql": double dot

I have been wondering for quite some time why in our projects we end up with a lot of linter dependencies in go.mod. It seems they are included transitively through Boostport/migration. They seem to be imported in tools.go. What is it? Would it be possible to keep go.mod clean of linter dependencies?

It seems that Golang migrations doesn't integrate well with other regular database migrations.

To reproduce the issue I'm facing, imagine an scenario where I have a migrations folder with a single sql migration:

• 1_step1.up.sql
• 1_step1.down.sql

I've migrated this up using db, so I have one row 1_step1 contents in the schema_migration table.

Then I want to build a go migration as a second step like this:

src := golang.NewSource()
src.AddMigration("2_step2", migration.Up, func() error {
    fmt.Println("I'm migrating up")
}
src.AddMigration("2_step2", migration.Down, func() error {
    fmt.Println("I'm migrating down")
}

I've used the postgres driver to return the contents of the schema_migration table like this:

drv, err := postgres.New(dbURL)
if err != nil {
	return err
}
applied := func() ([]string, error) {
	return drv.Versions()
}

drv.Versions() returning only 1 on this point, 1_step1 which is fine.

If I try to migrate the go 2_step2 over the first db step_1 however it doesn't apply anything.

• getMigrations() returns only the go source manually added files 2_step2 in this case.
• appliedMigrations has only db entries 1_step1.
• migrationsToApply is empty, so the go migration 2_step2 migration never gets applied.

Something I misunderstood or doing wrong?

Btw I have some improvements in mind for the docs, took me a while to understand some points.

CC\ @F21

Additional support for triggers within MySQL/MariaDB statements. Since triggers have a statement enclosed in the syntax which can contain a ; it breaks the normal parsing of the migration.

Using a common double semi-colon ;; that is a result of various sql dump methods a basic approach could be something like the following before driver/mysql.go:L71 :

import(
  ...
  "regexp"
  ...
)
...
if strings.Contains(content, "END;;") {

    triggerRegexp := regexp.MustCompile(`(?sm)^CREATE\sTRIGGER.*?END\;\;`)

    triggerStmts := triggerRegexp.FindAllString(content, -1)

    for i, _ := range triggerStmts {

      if _, err := driver.db.Exec(triggerStmts[i]); err != nil {
        return fmt.Errorf("Error executing statement: %s\n%s", err, triggerStmts[i])
      }

      // Remove the trigger statement from the migration content
      content = strings.Replace(content, triggerStmts[i], "", -1)
  }
}

Basic trigger example:

CREATE TRIGGER some_delightful_trigger
BEFORE UPDATE 
ON `some_magical_table` FOR EACH ROW
BEGIN
    INSERT INTO `some_magical_table_history` SET 
    `some_magical_table_id` = OLD.id,
    `a_column` = OLD.a_column;
END;;

Including code-quality checks should be a must for every go project.

We tend to use in our projects the following ones: goimports, golint, vet, megacheck, errcheck.

They han proven to be very useful.

Possibly add more?

I was thinking of adding a Makefile, refactor the code to pass all the checks, and possibly integrate that Makefile check as a Travis CI building stage?

I would take over this.

Thoughts?