GithubHelp home page GithubHelp logo

coldze / mongol Goto Github PK

View Code? Open in Web Editor NEW
8.0 3.0 2.0 99 KB

Migration tool for MongoDB (liquibase like)

License: MIT License

Go 99.57% Dockerfile 0.43%
mongodb liquibase migration golang go mongodb-migration docker rollback mongodb-support migrate

mongol's Introduction

MongoDB + Liquibase = mongol

Liquibase-like tool for MongoDB. Manipulating structure and data of MongoDB in migration-based way.

##How-to

Structure:

  • Mongol expects that migrate and rollback commands will be provided with a file (default ./changelog.json) in JSON format:
Main changelog file format:
{
	"connection":"mongodb://username:password@mymongodb:27017/myauthdb",
	"dbname":"mydbname",
	"migrations": [
		{
			"include": "some-folder/changelog.json",
			"relativeToChangelogFile": true
		}
	]
}
  • connection - Required. Full connection string to your MongoDB database
  • dbname - Required. Database name inside MongoDB to which migrations will be applied
  • migrations - Required. List of migrations to apply. Several formats are acceptible (see below)
  • include - Required. Path to migration changelog file. Full or relative to this changelog file, according to relativeToChangelogFile
  • relativeToChangelogFile - Optional. Indicates whether include path should be treated as relative to this changelog file. Default: true

Following formats are acceptable:

  • Migrations is of type string. relativeToChangelogFile is true, by default.
{
	"connection":"mongodb://username:password@mymongodb:27017/myauthdb",
	"dbname":"mydbname",
	"migrations": "some-folder/changelog.json"
}
  • Migrations is of type array of strings. relativeToChangelogFile is true, by default.
{
	"connection":"mongodb://username:password@mymongodb:27017/myauthdb",
	"dbname":"mydbname",
	"migrations": ["some-folder/changelog.json", "another-folder/changelog.json"]
}
  • Full format, ommiting relativeToChangelogFile.
{
	"connection":"mongodb://username:password@mymongodb:27017/myauthdb",
	"dbname":"mydbname",
	"migrations": [
		{
			"include": "some-folder/changelog.json"
		}
	]
}
  • Full format.
{
	"connection":"mongodb://username:password@mymongodb:27017/myauthdb",
	"dbname":"mydbname",
	"migrations": [
		{
			"include": "some-folder/changelog.json",
			"relativeToChangelogFile": true
		}
	]
}
Migration changelog file format:
{
  "id": "migration_id",
  "changes": [
    {
      "migration": {
        "include": "00001_first_migration.json",
        "relativeToChangelogFile": true
      },
      "rollback": {
        "include": "00001_first_migration_rollback.json",
        "relativeToChangelogFile": true
      }
    }
  ]
}
  • id - required. Migration's ID.
  • changes - required. List of changes to apply. Contains an object with 2 fields migration - forward migration, that is applied by migrate command; rollback - backward migration, that is applied by rollback command.
  • migration - required. Lists direct commands to apply during forward migration. Has the same format as migrations tag from main changelog file (see above).
  • rollback - optional. Lists direct commands to apply during backward migration. Has the same format as migration tag.

Following formats are acceptable:

{
  "id": "20190101_00001_initial_migration",
  "changes": [
    {
      "migration": {
        "include": "00001_first_migration.json",
        "relativeToChangelogFile": true
      }
    }
  ]
}
{
  "id": "20190101_00001_initial_migration",
  "changes": [
    {
      "migration": {
        "include": "00001_first_migration.json",
      }
    }
  ]
}
{
  "id": "20190101_00001_initial_migration",
  "changes": [
    {
      "migration": "00001_first_migration.json"
    }
  ]
}
{
  "id": "20190101_00001_initial_migration",
  "changes": [
    {
      "migration": ["00001_first_migration.json", "00002_second_migration.json"]
    }
  ]
}
Migration file format:
{
  "cmds" [
    {
      //first command in extended json format
    },
    {
      //second command in extended json format
    }
    ...
  ]
}

Example (single command). Following migration creates collection collection_name with validator:

{
  "create": "collection_name",
  "validator":{
    "$jsonSchema":{
      "bsonType":"object",
      "required":["name"],
      "properties":{
        "name":{
          "bsonType":"string",
          "description":"must be a string and is required"
        }
      }
    }
  }
}

Example (several commands). Following migration creates collection collection_name with validator and fills it up with test value:

{
  "cmds" [
    {
      "create": "collection_name",
      "validator":{
        "$jsonSchema":{
          "bsonType":"object",
          "required":["name"],
          "properties":{
            "name":{
              "bsonType":"string",
              "description":"must be a string and is required"
            }
          }
        }
      }
    },
    {
      "insert": "collection_name",
      "documents":[
        {
          "name": "Test value",
          "bsonType":"object",
          "required":["name"],
          "properties":{
            "name":{
              "bsonType":"string",
              "description":"must be a string and is required"
            }
          }
        }
      ]
    }
  ]
}

Example

Can be found here

Run

Docker-way:

  • forward migrations
docker run --rm -v /path/to/src:/mongol/src coldze/mongol:latest mongol migrate --path=/mongol/src/changelog.json 
  • backward migrations
docker run --rm -v /path/to/src:/mongol/src coldze/mongol:latest mongol rollback --path=/mongol/src/changelog.json 

Compile from source code & run:

  1. Install go: https://golang.org/doc/install
  2. Don't forget to add $GOBIN to your $PATH
  3. Run the following:
go get -u github.com/kardianos/govendor //govendor tool to handle dependecies, see vendor/vendor.json
go get github.com/coldze/mongol
cd $GOPATH/src/github.com/coldze/mongol
govendor sync
go install
mongol --help

Docker-hub:

https://hub.docker.com/r/coldze/mongol/

Functionality

  • forward migrations:
mongol migrate --path=/path/to/changelog.json --count=123
  • backward migrations/rollbacks. You must specify amount of migrations to rollback. Count takes into consideration ALL migrations, specified in changelog.json, whether they were applied or not. So if you have 10 migrations in total, but only 5 were applied and you want to rollback last 2, you will have to specify count=7 (last 5 missing, 2 to rollback):
mongol rollback --path=/path/to/changelog.json --count=7
  • MongoDB supports JavaScript starting from 3.0 up to 3.6, in 4.0 it was deprecated, so you're able to use eval in migrations for MongoDB 3.x versions.

Sample

Folder structure
./changelog.json //contains information about connection to mongo, includes sub-changelogs
./migrations/20190101/changelog.json //sub-changelog with migrations created at 2019-01-01
./migrations/20190101/0001_fill_data.json
./migrations/20190101/0001_fill_data_rollback.json
Migration file
Changelog file
Root changelog file

Running a test

Change mongo-db settings in $GOPATH/src/github.com/coldze/mongol/test/changelog.json and execute:

mongol migrate --path=$GOPATH/src/github.com/coldze/mongol/test/changelog.json

mongol's People

Contributors

coldze avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

Forkers

bomb0069 irkkmr

mongol's Issues

Facing issue while executing changeset

C:\Users\rajasekarm\go\src\github.com\coldze\mongol>mongol migrate --path=C:\Users\rajasekarm\go\src\github.com\coldze\mongol\test\changelog.json
2019/04/03 19:38:57 [INFO] Migration path: 'C:\Users\rajasekarm\go\src\github.com\coldze\mongol\test\changelog.json'
2019/04/03 19:40:01 [FATAL] App run failed with error: Run failed. Error: C:/Users/rajasekarm/go/src/github.com/coldze/mongol/commands/migrate.go:67
Error: Changelog validation failed.

C:/Users/rajasekarm/go/src/github.com/coldze/mongol/engine/changelog.go:460
Error: Failed to apply changes.

C:/Users/rajasekarm/go/src/github.com/coldze/mongol/engine/changelog.go:469
Error: Failed to process changeset '20180318_00001_00001'

C:/Users/rajasekarm/go/src/github.com/coldze/mongol/primitives/mongo/validator.go:63
Error: Failed to check change-set with ID 20180318_00001_00001

C:/Users/rajasekarm/go/src/github.com/coldze/mongol/primitives/mongo/validator.go:86
Error: Failed to get change from DB. Error: server selection error: server selection timeout
current topology: Type: ReplicaSetNoPrimary
Servers:
Addr: app1:27017, Type: Unknown, State: Connected, Avergage RTT: 0, Last error: dial tcp: lookup app1: no such host

Following is the changelog json file.

{
"connection": "mongodb://aos:ChgMeNOW@localhost:3339/nc",
"dbname": "nc",
"migrations": [
{
"include": "20180318_00001/changelog.json",
"relativeToChangelogFile": true
},
{
"include": "20180523_00001/changelog.json",
"relativeToChangelogFile": true
}
]
}

Base version for applying change

Hello Roman, this is less an issue but more a question.

  1. Do you create and update something like DATABASECHANGELOG collection with already applied patches to prevent failures or damages by accidentially repeating of not idempotent changesets?
  2. Is it possible to declare the base state (version/timestamp from the above DATABASECHANGELOG), for what a patch/changeset may be applied?

Regards, Gena

Support variables in changelogs and migrations

I need to run migrations for different environments - so at least different connection strings are required.
It will be easier if "mongol" will support variables int he same manner as liquidbase.
Changelog parameters

Currently I have to use workaround with placeholders and sed which generate env-tied changelog.

Thank you.

to execute the mongodb query with javascript function

Describe the bug
Need to execute the following query which are executed properly in mongodb command line.
But facing issue while placing and executing from Mongol framework file.

Here is the script details:

db.runCommand(
{
"aggregate": "me_me",
}).result.forEach(
function(e,i) {
var existingvalue = e.ctr.opstatus.ovalstts;
var newValue = (existingvalue === "uknwnall") ? "not started" : "started";
e.ctr.cldepsts1 = e.ctr.cldepsts1.replace('##ctr.opstatus.ovalstts##', newValue);
db.me_me.save(e);
})

Invoking function works in db.runCommand where it fails in the while invoking via mongol tool.

Below is my run command and works fine perfectly.
db.runCommand( {
"aggregate": "inventory2",
"pipeline": [
{ "$addFields":{"name": { "$concat": [ "$item", convertIndiviualState("$item"), "$item" ] }}},
{ "$out": "inventory2" }
],
"cursor": { }
})

But the same one I am placing it in json file and invoking via mongol facing the issue.

{
"cmds": [
{
"aggregate": "inventory2",
"pipeline": [
{ "$addFields":{"name": { "$concat": [ "$item", convertIndiviualState("$item"), "$item" ] }}},
{ "$out": "inventory2" }
],
"cursor": { }
}
]
}

Below is the error:

Error: Failed to generate migration from file 'deployment.json'. Error: C:/Users/rajasekarm/go/src/github.com/coldze/mongol/engine/decoding/ext_json.go:23
Error: Failed to decode ext-json. Error: invalid JSON input. Position: 122. Character: c

The script complains for the invocation of function.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.