file does not exist error about migrate

[dhui]

Did you create the migration files using migrate create? The migration file names should match the regex: ^([0-9]+)_(.*)\.(down|up)\.(.*)$

[slax0rr]

Hello,
I just had this problem yesterday, and it was due to manual removal of some migration files that were already executed with up, but not taken down with down.

Steps to reproduce:

1. Create a migration file
2. Run the migration with up
3. Remove the previously created migration file
4. Try running migration with up again, and you should get the same error

[jkonarze]

Here is my docker-compose.yaml, which solved the issue for me:

version: '3.8'
services:
  db:
    image: postgres
    environment:
      POSTGRES_DB: mydb
      POSTGRES_USER: myuser
      POSTGRES_PASSWORD: mypass
    ports:
      - "5432:5432"
  migrate:
    image: migrate/migrate
    volumes:
      - ./migrations:/migrations
    depends_on:
      - db
    command: -source=file://migrations -database postgres://myuser:mypass@db:5432/mydb?sslmode=disable up
    restart: on-failure
  adminer:
    image: adminer
    restart: always
    ports:
      - "8081:8080"
    environment:
      ADMINER_DEFAULT_SERVER: db
    depends_on:
      - db

[jonleopard]

I got it working! The issue was on my part, I had named the sql files incorrectly:

1_create_tables_up.sql -> 1_create_tables.up.sql
1_create_tables_down.sql -> 1_create_tables.down.sql

If anyone needs a working docker-compose.yml file, feel free to reach out to me.

Sorry for the noise!

[yiGmMk]

I got it working! The issue was on my part, I had named the sql files incorrectly:

1_create_tables_up.sql -> 1_create_tables.up.sql
1_create_tables_down.sql -> 1_create_tables.down.sql

If anyone needs a working docker-compose.yml file, feel free to reach out to me.

Sorry for the noise!

works for me,tanks

[dhui]

Removing the migration file referenced by the schema_migration table (or whatever your specific DB driver uses) will cause this error. Modifying/replacing the migration file won't work without forcing the schema version to the previous version using migrate force. Don't forget, if your migrations should aren't atomic (e.g. wrapped in a transaction if supported by the DBMS), then you'll need to manually backout your borked migration.

[davidsbond]

Hello, I too am getting this error when trying to do dry runs of migrations against a new PostgreSQL instance running in a CI pipeline:

error: first /migrations/: file does not exist

The pipeline starts an instance of Postgres then does the following command:

docker run -v /database/src:/migrations --network host migrate/migrate -path=/migrations/ -database postgres://postgres:integration@localhost:5432/postgres?sslmode=disable up

The current working directory is set to /database in my pipeline, that's where the code is checked-out. In my git repository I have this folder structure:

./
  src/
    1_initial.up.sql
    1_initial.down.sql 

Is there some initialization command I need to be running first?

[bnfinet]

I ran into this today when I tried to migrate a file named 20191120T2259000Z_init.up.sql which I had created with the argument -format 20060102T150405000Z

Since the regex for a legit migration is ^([0-9]+)_(.*)\.(down|up)\.(.*)$ that failed

@dhui would you be open to a PR to adjust that regex to include -,a-z,A-Z on the left

or possibly to modify the error message to reference the applicable regex?

Thanks much for the fine work on migrate! i'm glad to have it working in my project.

[suryapandian]

Here is my docker-compose.yaml, which solved the issue for me:

version: '3.8'
services:
  db:
    image: postgres
    environment:
      POSTGRES_DB: mydb
      POSTGRES_USER: myuser
      POSTGRES_PASSWORD: mypass
    ports:
      - "5432:5432"
  migrate:
    image: migrate/migrate
    volumes:
      - ./migrations:/migrations
    depends_on:
      - db
    command: -source=file://migrations -database postgres://myuser:mypass@db:5432/mydb?sslmode=disable up
    restart: on-failure
  adminer:
    image: adminer
    restart: always
    ports:
      - "8081:8080"
    environment:
      ADMINER_DEFAULT_SERVER: db
    depends_on:
      - db

thanks @jkonarze , this worked for me as well.

[dhui]

@haskaalo try -source=file:///migrations in the command
I think you're using a relative path instead of an absolute one

[wotzhs]

hey guys, i also ran into the same issue earlier, however after trying with various path, it worked for me below is my code (it lives in the wallet/internal/db/postgres.go):

and my folder structure is as such:

hopefully it helps

[dhui]

The -path option works (I've been using it).
Is file does not exist the full error message? Is your path relative or absolute?

[zwhitchcox]

error: first /go/src/github.com/zwhitchcox/consolidator/migrations: file does not exist is the full error message

The output of ls /go/src/github.com/zwhitchcox/consolidator/migrations is

1526653772_init_down.sql 1526653772_init_up.sql

As I said, maybe I just don't understand what the argument -path is for. I'm giving the path to my sql migrations, but that is not working.

[zwhitchcox]

This is the full command:
migrate -path /go/src/github.com/zwhitchcox/consolidator/migrations -database postgres://postgres:password@db:5432/dev?sslmode=disable up

[kostyaVyrodov]

I have the same issue.
My project contains a directory with all migrations and there are 2 files with SQL scripts:

- <root-of-project>/schema/201802221525_init_schema.up.sql
- <root-of-project>/schema/201802221525_init_schema.down.sql

I try to run following command from the 'root-of-project':
$ migrate -database "mysql://root:root@(0.0.0.0:3306)/databasename" -source file://schema up
and I got such message:
error: file does not exist

[dhui]

@kostyaVyrodov what's the full error message? e.g. does it start with error: first?
Also, is the directory listable and are the files readable by the user you're running migrate with?

[kostyaVyrodov]

@dhui ,
Full message is:

Files are readeable:

By the way, I noticed that I use https://github.com/mattes/migrate v.3.0.1 instead of your fork. Maybe there is no such issue with your library. I just inattentive and missed that this repo is a fork of the library... Sorry :)
But in any way, there is a similar issue and I hope that I can help you.

[dhui]

@kostyaVyrodov Try using this repo's CLI but I don't expect a difference since the source/file tree hasn't changed

[tomspeak]

Am also getting this issue - anyone figure out the cause?

[hawarir]

I had the exact problem, rebuilding the cli binary with adding file tags works for me:

go build -tags 'postgres file' -o /usr/local/bin/migrate github.com/golang-migrate/migrate/cli

What baffles me is after that I tried to rebuild the binary again without the file tag, and it still works.

EDIT:
I can't find file build constraints anywhere, even though it's used in the Makefile as one of the possible build tags (that's where I got the idea). So I think the solution here is simply try to rebuild the cli binary as usual.

[dhui]

I can't find file build constraints anywhere, even though it's used in the Makefile as one of the possible build tags (that's where I got the idea). So I think the solution here is simply try to rebuild the cli binary as usual.

The file source is automatically built with the CLI (it's imported in cli/commands.go). For re-building using the -a flag may help.

Is everyone using Go 1.9.x or 1.10.x?

Also, since v3.3.0, you can see the source and db drivers that are loaded in the help text.

[nutmix]

We have the same error. It worked once, then when we try to run it again (nothing changed), files present and correct, correct path etc. we get "Error: File does not exist"

This means we can no longer update our production system.

[francisdb]

> curl -L https://github.com/golang-migrate/migrate/releases/download/v3.4.0/migrate.darwin-amd64.tar.gz | tar xvz
> ls ./migrations
1534505409_create_table_merges.down.sql 1534505409_create_table_merges.up.sql
> ./migrate.darwin-amd64 -source file://./migrations -database clickhouse://localhost:9000 down

no change
> ./migrate.darwin-amd64 -source file://./migrations -database clickhouse://localhost:9000 up

1534505409/u create_table_merges (13.128535ms)
> ./migrate.darwin-amd64 -source file://./migrations -database clickhouse://localhost:9000 down

1534505409/d create_table_merges (8.1552ms)
> ./migrate.darwin-amd64 -source file://./migrations -database clickhouse://localhost:9000 up

error: file does not exist

Maybe as a start that error could indicate what file it is looking for?

SELECT *
FROM schema_migrations

┌────version─┬─dirty─┬────────────sequence─┐
│ 1534505409 │     1 │ 1534505488799259560 │
│ 1534505409 │     0 │ 1534505488809850593 │
│ 4294967295 │     1 │ 1534505611427091886 │
│ 4294967295 │     0 │ 1534505611431268021 │
└────────────┴───────┴─────────────────────┘

4 rows in set. Elapsed: 0.002 sec.

somehow when you go all the way down it introduces a version 2^32 − 1 instead of 0? Causing it too look for 4294967295_{title}.down.{extension}?

[dhui]

@francisdb Not sure how 4294967295 ended up in the schema_migrations table. It's odd b/c on a 64-bit system, uints should underflow to 2^64 - 1. See the Migration and Migrations structs in source/migration.go. Also, I see protection for underflow in Migrations.Prev().
I don't see anything obviously broken in Migrate.read() in migrate.go. Can you reproduce the issue with -verbose enabled?

[francisdb]

> ./migrate.darwin-amd64 -source file://./migrations -verbose -database clickhouse://localhost:9000 up

2018/08/20 09:08:44 no change
2018/08/20 09:08:44 Finished after 2.93045ms
2018/08/20 09:08:44 Closing source and database
> ./migrate.darwin-amd64 -source file://./migrations -verbose -database clickhouse://localhost:9000 down

2018/08/20 09:08:54 Start buffering 1534505409/d create_table_merges
2018/08/20 09:08:54 Read and execute 1534505409/d create_table_merges
2018/08/20 09:08:54 Finished 1534505409/d create_table_merges (read 6.722404ms, ran 6.281131ms)
2018/08/20 09:08:54 Finished after 22.784606ms
2018/08/20 09:08:54 Closing source and database
> ./migrate.darwin-amd64 -source file://./migrations -verbose -database clickhouse://localhost:9000 down

2018/08/20 09:09:03 error: file does not exist
> ./migrate.darwin-amd64 -source file://./migrations -verbose -database clickhouse://localhost:9000 up

2018/08/20 09:09:08 error: file does not exist

[moelius]

Hi. Have same issue with clickhouse only. postgres is ok. I'm using cli.
After migrate create I have files like 20180903190206_add_...... But clickhouse writes 3146827521 to schema_migrations table. I thin somewhere cli uses wrong types. Temporarily I solved this by renaming files to 3146827521_add_.....
Also it can be solved with option -format unix. I think this option must be enabled by default.

[cydonknight88]

Any updates on this?

I've been experiencing this for the past few days and the only solution for me was to nuke the db and repopulate everything from scratch for it to work correctly. I'm just fortunate as that particular db was part of a new application deployment but I can see issues arising when trying to update db versions...It seems like @slax0rr might be right about the cause, as I experienced the error mostly as he described in his steps to reproduce:

1. migrate up (in my case this failed because of an issue parsing the sql files, which led me to step 2)
2. remove/modify/replace migration file(s)
3. migrate up
4. error

[n1koo]

I started debugging a similar issue.

1. Run migrate to up 2 first

# ls -lha /sql/foobar
total 28K
drwxr-xr-x 2 root root 4.0K Jan 17 21:10 .
drwxr-xr-x 1 root root 4.0K Jan 17 21:10 ..
-rw-rw-rw- 1 root root   95 Jan 17 19:05 1_initial_schema.down.sql
-rw-rw-rw- 1 root root  422 Jan 17 19:05 1_initial_schema.up.sql
-rw-rw-rw- 1 root root  596 Jan 17 19:05 2_add_col_and_index.down.sql
-rw-rw-rw- 1 root root  673 Jan 17 19:05 2_add_col_and_index.up.sql

Everything seems good, checking schema_migrations we see that we are on 2 and clean

backend_uas=> SELECT * from schema_migrations;
 version | dirty
---------+-------
       2 | f

2. then tried to do noop migration (as part of CI)

# /sql/migrate -path /sql/foobar -database postgres://${USER_DB_NAME}:${USER_DB_PASS}@${USER_DB_HOST}:5432/${USER_DB_NAME}?sslmode=disable up 2
error: file does not exist

but

# /sql/migrate -path /sql/foobar -database postgres://${USER_DB_NAME}:${USER_DB_PASS}@${USER_DB_HOST}:5432/${USER_DB_NAME}?sslmode=disable up
no change

Looking at the code for some reason we return os.ErrNotExist in https://github.com/golang-migrate/migrate/blob/master/migrate.go#L562-L566 but only when limit is positive. Without limit set it defaults to -1 which explains why that worked.

I don't understand the intent of os.ErrNotExist in this case, seems like we should returning ErrNoChang just like in https://github.com/golang-migrate/migrate/blob/master/migrate.go#L551-L555 ?

[dhui]

@n1koo Your issue is slightly different as you're running up with the number of expected migrations specified.

The standard usage is to use up in automatic mode. e.g. without the number of migrations specified

However, you have the option to use up in manual mode, which should generally only be used to fix any issues with migration(s) that failed to apply correctly. When using up in manual mode, the assumption is that you, the user, knows what's going on to fix the issue. e.g. you're telling migrate how many migrations you expect to be migrated

[dhui]

@davidsbond What's docker run --rm -v /database/src:/migrations --entrypoint ls migrate/migrate -la /migrations print?

[davidsbond]

drwxr-xr-x    2 root     root          4096 Jan 21 22:54 .
drwxr-xr-x   29 root     root          4096 Jan 21 22:54 ..

Looks like the issue must be down to how the docker volume is being set, although I don't think I'm setting it incorrectly. Will keep digging

Here's the output of ls /database/src, not sure why my files aren't turning up

1_initial.down.sql  1_initial.up.sql

[davidsbond]

@dhui Apprently my CI provider doesn't support mounting files this way, I'll have to change the pipeline to use the CLI directly instead!

[adrianforsius]

I found myself with the same problem today I managed to get around the issue by --mount ing instead of using volume, this might not be a sufficient substitution but it might give insight to what the problem is.

[dhui]

Are people still hitting this issue with migrate v4.3.1?
If so, please provide the error message(s) and the list of migration files. e.g. info on how to reproduce the issue

[haskaalo]

Any update?

[haskaalo]

@dhui Simply run this docker compose file with some sql migration files in ./migrations (wait a few second) and you'll see this pop up on docker logs postgres_migration

error: first /migrations: file does not exist

version: '3'

services:
  postgres:
    container_name: postgres_server
    image: postgres:alpine
    networks:
      - app-network
    ports:
      - 5432:5432
    environment:
      - POSTGRES_USER=auser
      - POSTGRES_DB=adb
      - POSTGRES_DATABASE=totalrandompassword
  postgres_migration:
    container_name: postgres_migration
    image: migrate/migrate
    restart: on-failure
    networks:
      - app-network
    volumes:
      - ./migrations:/migrations
    depends_on:
      - postgres
    command: -source=file://migrations -verbose -database postgres://auser:totalrandompassword@postgres:5432/adb?sslmode=disable up

networks:
  app-network:
    driver: bridge

[haskaalo]

@dhui Just tried, it doesn't work :/

[ashtonian]

I ran into this in 4.6.1 when trying to use two different migrations sources on the same db. Had to rename schema_migrations for this to work.

[nejtr0n]

just need to add migration table name config variable. This will allow to run multiple migrations on same database.

[dhui]

@dhui would you be open to a PR to adjust that regex to include -,a-z,A-Z on the left
or possibly to modify the error message to reference the applicable regex?

@bnfinet Yeah, go for it! For full support of the -format option, we'd need to match [^_].
Don't forget to add appropriate tests.

[dvaldivia]

I'm running into the problem is I'm running the migrations function from my /tests folder, since the migrations live in /lib/migrations I'm forced to do file://../lib/migrations when referring the source from my tests and file://lib/migrations when referring to them from the main package, any way around this?

[jonleopard]

Any update on this?

I'm trying to run this via the CLI and also docker-compose. the command doesn't work in either environments. If it helps, I'm on a macOS running the latest version of golang-migrate.

Directory Structure:

.
├── Makefile
├── docker-compose.yml
├── go.mod
├── go.sum
├── goreddit.go
└── migrations
    ├── 1_create_tables_down.sql
    └── 1_create_tables_up.sql

Via CLI:

migrate -source file://migrations -database "postgres://mydbuser:mydbpwd@localhost:5432/mydbname?sslmode=disable" up 2 

Returns error: first : file does not exist

docker-compose.yml

version: '3.8'

services:
  db:
    image: postgres
    environment:
        POSTGRES_DB: mydbname
        POSTGRES_USER: mydbuser
        POSTGRES_PASSWORD: mydbpwd
    ports:
      - "5432:5432"
  
  migrate:
    image: migrate/migrate
    volumes:
      - .:/migrations
    depends_on:
      - db
    command: -path=/migrations/ -database postgres://mydbuser:mydbpwd@db:5432/mydbname?sslmode=disable up 2

  adminer:
    image: adminer
    restart: always
    ports:
      - "8081:8080"
    environment:
      ADMINER_DEFAULT_SERVER: db
    depends_on:
      - db

Container returns error: first : file does not exist

Any input would be great!

[jonleopard]

As @dhui suggested, I've tried putting -source=file:///migrations in my command script. I've also changed the volume mapping to - ./migrations:/migrations. It's still throwing the error: first : file does not exist error .

Is there a way to pause the migrate container so I can shell in and verify if the files are getting copied or not?

[jkonarze]

@jonleopard would you mind sharing your docker-compose.yml ?

[DudeWhoCode]

I ran into this in 4.6.1 when trying to use two different migrations sources on the same db. Had to rename schema_migrations for this to work.

@ashtonian I am also facing the same issue while trying to have two sets of migrations under same database. But if you rename the schema_migrations table how will this library keep track of the 2 different sets of migrations?
Did you fork this repo and made changes to maintain multiple reference tables for multiple migration sets?

[alhamsya]

remover query param in string database

[marlongerson]

1_create_tables_up.sql -> 1_create_tables.up.sql
1_create_tables_down.sql -> 1_create_tables.down.sql

This fixed the issue

[afaderoTextNow]

yiGmMk

This really was the key for me, I had named the files incorrectly all this time, thanks so much!

[mattrltrent]

remover query param in string database

Worked for me 🥂❤️