square / shift

shift is an application that helps you run schema migrations on MySQL databases

shift

shift is an application that makes it easy to run online schema migrations for MySQL databases

Who should use it?

shift was designed to solve the following problem - running schema migrations manually takes too much time. As such, it is most effective when schema migrations are taking up too much of your time (ex: for an operations or DBA team at a large organization), but really it can be used by anyone. As of writing this, shift has had no problem running hundreds of migrations a day or running migrations that take over a week to complete.

Features

• safe, online schema changes (invokes the tried-and-true pt-online-schema-change)
  • supports all "ALTER TABLE...", "CREATE TABLE...", or "DROP TABLE..." ddl
• a ui where you can see the status of migrations and run them with the click of a button
• self-service - out of the box, any user can file and run migrations, and an admin is only required to approve the ddl (this is all configurable though)
• shard support - easily run a single migration on any number of shards

Components

shift consists of 3 components. Each component has its own readme with more details

• ui: a rails app where you can file, track, and run database migrations
• runner: a go agent that consumes jobs from an api exposed by the ui
• pt-osc patch: a patch for pt-online-schema-change from the percona toolkit

Demo

Watch a demo video here

Installation

Read the installation guide here

License

Copyright (c) 2016 Square Inc. Distributed under the Apache 2.0 License. See LICENSE file for further details.

shift's Issues

logo.img

Can't locate YAML/Syck.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at /usr/bin/pt-online-schema-change line 3644.

when i try to start the migartion its failing with
Can't locate YAML/Syck.pm in @inc (@inc contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at /usr/bin/pt-online-schema-change line 3644.

Patch failed for latest pt-online-schema-change

Patch command is getting failed while applying the patch file for the latest percona toolkit (3.0.13).

Logs:

root@localhost:/opt/shift/files# patch 3.0 ptosc-patch/0001-ptosc-square-changes.patch 
patching file 3.0
Hunk #1 succeeded at 28 (offset 1 line).
Hunk #2 succeeded at 3701 (offset 157 lines).
Hunk #3 succeeded at 3787 (offset 157 lines).
Hunk #4 succeeded at 5728 (offset 346 lines).
Hunk #5 succeeded at 5743 (offset 346 lines).
Hunk #6 succeeded at 5864 (offset 365 lines).
Hunk #7 succeeded at 6089 (offset 372 lines).
Hunk #8 succeeded at 6101 (offset 372 lines).
Hunk #9 succeeded at 8358 with fuzz 1 (offset 253 lines).
Hunk #10 succeeded at 8956 (offset 365 lines).
Hunk #11 succeeded at 9191 (offset 371 lines).
Hunk #12 succeeded at 9223 (offset 371 lines).
Hunk #13 succeeded at 9257 (offset 371 lines).
Hunk #14 succeeded at 9349 (offset 371 lines).
Hunk #15 succeeded at 9471 (offset 371 lines).
Hunk #16 succeeded at 9513 (offset 371 lines).
Hunk #17 succeeded at 9538 (offset 371 lines).
Hunk #18 succeeded at 9560 (offset 371 lines).
Hunk #19 succeeded at 9578 (offset 371 lines).
Hunk #20 succeeded at 9770 (offset 371 lines).
Hunk #21 succeeded at 9806 (offset 371 lines).
Hunk #22 succeeded at 9912 (offset 381 lines).
Hunk #23 succeeded at 9940 (offset 381 lines).
Hunk #24 succeeded at 9962 (offset 381 lines).
Hunk #25 succeeded at 10006 (offset 381 lines).
Hunk #26 succeeded at 10195 (offset 424 lines).
Hunk #27 succeeded at 10236 (offset 424 lines).
Hunk #28 succeeded at 10251 (offset 424 lines).
Hunk #29 succeeded at 10641 (offset 505 lines).
Hunk #30 succeeded at 10656 (offset 505 lines).
Hunk #31 FAILED at 10256.
Hunk #32 succeeded at 10860 (offset 568 lines).
Hunk #33 succeeded at 10882 (offset 566 lines).
Hunk #34 succeeded at 10911 (offset 566 lines).
Hunk #35 FAILED at 10359.
Hunk #36 succeeded at 11255 (offset 612 lines).
Hunk #37 FAILED at 10709.
Hunk #38 succeeded at 11684 (offset 773 lines).
Hunk #39 succeeded at 12460 (offset 842 lines).
Hunk #40 succeeded at 12537 (offset 879 lines).
Hunk #41 succeeded at 12569 with fuzz 2 (offset 891 lines).
Hunk #42 succeeded at 12691 with fuzz 2 (offset 944 lines).
Hunk #43 succeeded at 12843 with fuzz 2 (offset 993 lines).
Hunk #44 succeeded at 12914 with fuzz 2 (offset 1015 lines).
Hunk #45 succeeded at 12986 (offset 1016 lines).
3 out of 45 hunks FAILED -- saving rejects to file 3.0.rej

3.0.rej file:

--- pt-online-schema-change       2017-07-17 09:46:31.000000000 -0700
+++ pt-online-schema-change       2017-07-17 09:46:31.000000000 -0700
@@ -10256,6 +10583,16 @@
    elsif ( $o->get('execute') ) {
       print ts("Swapping tables...\n");
 
+      if ( $o->get('swap-table-name') ) {
+         $table_name = $o->get('swap-table-name');
+         $table_name =~  s/%T/$orig_tbl->{name}/g;
+         use POSIX 'strftime';
+         my $timestamp = strftime("%Y%m%d%H%M%S", localtime(time()));
+         $table_name =~  s/%D/$timestamp/g;
+      } else {
+         $table_name = $prefix . $table_name;
+      }
+
       while ( $name_tries-- ) {
          $table_name = $prefix . $table_name;
 
@@ -10359,7 +10720,10 @@
            . ' LIKE ' . $q->literal_like($orig_tbl->{tbl});
    PTDEBUG && _d($sql);
    my $triggers = $dbh->selectall_arrayref($sql);
-   if ( $triggers && @$triggers ) {
+   # Don't quit on existing triggers if running from a saved state. The triggers
+   # are expected to be present.
+   if ( $triggers && @$triggers &&
+         (!defined $loaded_step || (defined $loaded_step && !grep { $_ eq $loaded_step } qw(triggers copy rename)))) {
       die "The table $orig_tbl->{name} has triggers.  This tool "
          . "needs to create its own triggers, so the table cannot "
          . "already have triggers.\n";
@@ -10709,8 +11074,11 @@
    @drop_trigger_sqls = ();
    foreach my $trg ( @triggers ) {
       my ($name, $sql) = @$trg;
-      print $sql, "\n" if $o->get('print');
-      if ( $o->get('execute') ) {
+      print "$name trigger: $sql\n";
+      # Don't create triggers if running from a saved state. They should already
+      # be there.
+      if ( $o->get('execute') &&
+         (!defined $loaded_step || (defined $loaded_step && !grep { $_ eq $loaded_step } qw(triggers copy rename)))) {
          osc_retry(
             Cxn     => $cxn,
             Retry   => $retry,

Please let me know how to patch it without getting failed.

Consider publishing a Docker image

Thanks for building and sharing this great product!

Would you consider creating a Docker image for this?

I tried going through https://github.com/square/shift/wiki/Installation-Guide, but the setup errors out on the latest master, because of version drifts over time, and native extensions failing to build on Ruby 2.2. Even after bumping to Ruby 2.5, packages like json, mysql2, and therubyracer need version tweaks, which triggers a need for cascading changes all over the place.

I'm more than happy to share an attempt in progress so we can discuss better. Thanks!

Getting this error intermittently while running migration

Tried a simple migration, got the below error.

Runner logs:

I0305 06:57:00.697588 00015 migration.go:350] mig_id=1: Running query 'SELECT DATA_LENGTH, INDEX_LENGTH, TABLE_ROWS FROM information_schema.tables WHERE table_schema=? AND table_name=?' (args: [world sbtest1]).
I0305 06:57:00.747675 00015 migration.go:357] mig_id=1: Query response was 'map[DATA_LENGTH:[22593536] INDEX_LENGTH:[2637824] TABLE_ROWS:[98712]]'
E0305 06:57:02.613776 00015 runner.go:363] mig_id=1: RestError [NextStep]: invalid character '<' looking for beginning of value.
I0305 06:57:03.552564 00015 runner.go:373] mig_id=1: Successfully failed migration.
I0305 06:57:13.576634 00015 runner.go:240] Getting staged migrations.
I0305 06:57:13.661886 00015 runner.go:247] No new staged migrations.

pt-osc logs:

[2020-03-05 06:56:58] stdout: Operation, tries, wait:
[2020-03-05 06:56:59] stdout:   analyze_table, 10, 1
[2020-03-05 06:56:59] stdout:   copy_rows, 10, 0.25
[2020-03-05 06:56:59] stdout:   create_triggers, 10, 1
[2020-03-05 06:56:59] stdout:   drop_triggers, 10, 1
[2020-03-05 06:56:59] stdout:   swap_tables, 10, 1
[2020-03-05 06:56:59] stdout:   update_foreign_keys, 10, 1
[2020-03-05 06:56:59] stdout: Starting a dry run.  `world`.`sbtest1` will not be altered.  Specify --execute instead of --dry-run to alter the table.
[2020-03-05 06:56:59] stdout: Creating new table...
[2020-03-05 06:56:59] stdout: Created new table world._sbtest1_new OK.
[2020-03-05 06:56:59] stdout: Altering new table...
[2020-03-05 06:56:59] stdout: Altered `world`.`_sbtest1_new` OK.
[2020-03-05 06:56:59] stdout: Not creating triggers because this is a dry run.
[2020-03-05 06:56:59] stdout: Not copying rows because this is a dry run.
[2020-03-05 06:56:59] stdout: Not swapping tables because this is a dry run.
[2020-03-05 06:56:59] stdout: Not dropping old table because this is a dry run.
[2020-03-05 06:56:59] stdout: Not dropping triggers because this is a dry run.
[2020-03-05 06:56:59] stdout: 2020-03-05T06:56:59 Dropping new table...
[2020-03-05 06:56:59] stdout: 2020-03-05T06:56:59 Dropped new table OK.
[2020-03-05 06:56:59] stdout: Dry run complete.  `world`.`sbtest1` was not altered.

If possible please share your skype id / slack details so that we exchange the logs instead of creating the issues here.

Cannot use final insert for migrations that create a new table

It seems that Shift tries to validate the migration and fails, because a table doesn't exist, but it would have existed after the DDL statement execution.

It would be nice if Shift could recognize this and appropriately validate the input.

Error I saw:

Error message: migration: invalid final insert statement: Error 1146: Table 'some_app_staging.ar_internal_metadata' doesn't exist.

My DDL statement:

 CREATE TABLE
	`ar_internal_metadata` (
		`key` VARCHAR (255) CHARACTER
	SET
		utf8 COLLATE utf8_unicode_ci NOT NULL
		,`value` VARCHAR (255) COLLATE utf8mb4_unicode_ci DEFAULT NULL
		,`created_at` datetime NOT NULL
		,`updated_at` datetime NOT NULL
		,PRIMARY KEY (`key`)
	) ENGINE = InnoDB DEFAULT CHARSET = utf8mb4 COLLATE = utf8mb4_unicode_ci

My final insert:

INSERT INTO `ar_internal_metadata` (`key`, `value`, `created_at`, `updated_at`) 
    VALUES ('environment', 'staging', NOW(), NOW())

Patch Failing

Have been trying to apply patch to pt-online schema change but it keeps failing.
Have tried Percona 2.2.15, 16 and 17 and it seems to fail. The patch gets applied to 15 but when running the command gives following errors:
Percona 2.15
Can't locate YAML/Syck.pm in @inc (you may need to install the YAML::Syck module) (@inc contains: /home/porfirio/perl5/lib/perl5/5.26.1/x86_64-linux-gnu-thread-multi /home/porfirio/perl5/lib/perl5/5.26.1 /home/porfirio/perl5/lib/perl5/x86_64-linux-gnu-thread-multi /home/porfirio/perl5/lib/perl5 /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.26.1 /usr/local/share/perl/5.26.1 /usr/lib/x86_64-linux-gnu/perl5/5.26 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.26 /usr/share/perl/5.26 /home/porfirio/perl5/lib/perl5/5.26.0 /home/porfirio/perl5/lib/perl5/5.26.0/x86_64-linux-gnu-thread-multi /usr/local/lib/site_perl /usr/lib/x86_64-linux-gnu/perl-base) at /home/porfirio/perl5/bin/pt-online-schema-change line 3644.
BEGIN failed--compilation aborted at /home/porfirio/perl5/bin/pt-online-schema-change line 3644.

As mentioned it the issues, tried to install previous versions of perl but no success
Percona 2.2.16 and 2.2.17
Perl: 5.22.0 ,5.24.0 and 5.26.1
Hunk #31 FAILED at 10256.

Have DBI and DBD installed.

Improve Documentation

Hi,

I'm not seeing any detail configuration steps for this tool, I've updated following file

/home/ec2-user/shift/ui/config/database.yml

with local mysql credentials (but no host name) same EC2 where Shift is installed, I also update config.x.mysql_helper.db_config paramter with username & password in following file.

/home/ec2-user/shift/ui/config/environments/development.rb

now after run following command

"bundle exec rake db:setup"

and run server by issuing follwoing command

RAILS_ENV=production
rails server -b 0.0.0.0 -p 3000

I see databases in database drop down. Do I need to install Shift on same machine where my staging DB is located or how can I connect it to remote mysql host ? In my case its RDS.

Thank you

[GOLANG] Runner switch to vendoring

Is there a reason why the workspace approach was choosen and not the vendoring one?

Would you be interested in a PR changing that based on Godep?

Demo video is private

The demo video linked from readme is private:

• https://www.youtube.com/watch?v=u5L7PqIk--k

Unknown option: save-state

After the first check which passed , I did try the actual change to the DB.
Error before copy records:
redundant argument in printf at /usr/bin/pt-online-schema-change line 3520

Data is missing because of rename migration

Data is missing if the rename migration takes time inbetween.

Below is the screenshot of the migration:

Before starting migration row count - 986,408
After completing migration row count - 986,473

help

pt-osc patch fails on 2.2.17

Patching pt-online-schema-change fails on 2.2.17:

root@toolbox:/root/shift/ptosc-patch# patch /usr/local/bin/pt-online-schema-change /root/shift/ptosc-patch/0001-ptosc-square-changes.patch
patching file /usr/bin/pt-online-schema-change
Hunk #2 succeeded at 3703 (offset 79 lines).
Hunk #3 succeeded at 5572 (offset 240 lines).
--snip--
Hunk #28 succeeded at 10285 (offset 414 lines).
Hunk #29 FAILED at 9976.
Hunk #30 succeeded at 10465 (offset 453 lines).
--snip--
Hunk #37 succeeded at 11834 (offset 524 lines).
Hunk #38 succeeded at 11874 with fuzz 2 (offset 528 lines).
Hunk #39 succeeded at 11906 with fuzz 2 (offset 540 lines).
Hunk #40 succeeded at 11975 with fuzz 2 (offset 540 lines).
Hunk #41 succeeded at 12086 (offset 548 lines).
Hunk #42 succeeded at 12153 (offset 566 lines).
1 out of 42 hunks FAILED -- saving rejects to file /usr/local/bin/pt-online-schema-change.rej

Question: did you ever consider migrating to a different data store?

Such as Postgres, that provides O(1) migrations, rather than develop a system and use online-schema-change that anyways copies the entire table to do modifications (so is still slow, right?).
If you're doing this 100 times a day I can't shake the feeling that this is a lot of new tables and data shuffling that increase risk and churn on database servers.
Am I missing something?

Thanks!

Shift creating shadow table

After completion of migration, the shift is creating a shadow table like below

+---------------------------+
| Tables_in_test            |
+---------------------------+
| 20181004125700887_sbtest1 |
| 20181004130253090_sbtest5 |
| 20181004130952087_sbtest5 |
| 20181005112052953_sbtest5 |
| 20181005112924812_sbtest5 |
| 20181005112924891_sbtest5 |
| 20181005112924892_sbtest5 |
| 20181005113204700_sbtest5 |
| 20181005113431865_sbtest5 |
| 20181005121503073_sbtest5 |
| sbtest1                   |
| sbtest2                   |
| sbtest3                   |
| sbtest4                   |
| sbtest5                   |
+---------------------------+```

Do we have any parameters to stop this ?

Error executing the go runner

Hi,

I'm trying to setup Shift but i'm getting the next error when starting the runner:

I0627 09:54:37.607786 14755 runner.go:173] Getting staged migrations.
E0627 09:54:37.607975 14755 runner.go:176] Failed to get staged migrations (error: rest: error getting staged migrations)

I'm sure that the API url is configured correct in the configuration.

Any idea how to debug this issue?

Shift-api-bounce

How to change from localhost?

Hi,

I have installed Shift on an EC2 instance and when I run 'bundle exec rails server it starts successfully on localhost. However, with port 3000 open in the EC2 security group, I am unable to open the interface.
Can I tell Shift to be available on an IP address instead of localhost?

Question: Would this work with AWS RDS?

Would this work on mySQL (or MariaDB) database's hosted on AWS RDS?

i

The UI for creating meta requests is weird IMO

This is a nit, but can you make meta requests more easily accessible? I would like it if there was a "Create Meta Request" button next to the "Create Request" button at the top.

support gh-ost

gh-ost seems to be a better solution than pt-online-schema, it'd be amazing if it were available as an option

Runner script dropping the existing triggers

The runner script has dropped the old triggers for the table.

Logs:

I0118 13:00:53.844432 06964 runner.go:775] mig_id=9: cleaning up.
I0118 13:00:53.845156 06964 migration.go:322] mig_id=9: cleaning up triggers.
I0118 13:00:53.845193 06964 migration.go:350] mig_id=9: Running query 'SELECT trigger_name FROM information_schema.triggers WHERE trigger_schema=? AND event_object_table=?' (args: [stage t1]).
I0118 13:00:53.849166 06964 migration.go:357] mig_id=9: Query response was 'map[trigger_name:[check_trigger]]'
I0118 13:00:53.849207 06964 migration.go:364] mig_id=9: Running query 'DROP TRIGGER IF EXISTS `stage`.`check_trigger`'.
I0118 13:00:53.850382 06964 migration.go:328] mig_id=9: cleaning up shadow table.

is that expected behavior ?

No results found on database list for new install

I'm a Rails noob:

Configured database.yml - added a development database.

Running db:setup created the database (though did not populate cluster).

Added a cluster, per the insert statement.

Edited config/environments/development.rb to point to the DB server.

But creating a New Migration Request "fails". I can see the cluster, but looking at databases shows "No results found"

did someone work on xtradb Cluster 5.7 successful ?

hello,
did someone work on xtradb Cluster 5.7 successful ?