getdave / grunt-deployments Goto Github PK

A simple way to exclude table rows from certain tables?

ie.
SELECT * FROM wp_posts WHERE post_type LIKE '%revision%'

I've been using this across several sites over the last year and some databases are 100mb+ so being able to exclude 'padding' or unwanted data would save a lot of time and space.

Discovered potential error when using the db_pull command. Target database db dump file has contents of:

/bin/sh: 8bN[N7Nwe.1: command not found

Not sure what's going on.

I'm using AMPPS as my local mysql and apache server and now trying to use this package to sync my local db with the remote dbs.

I can connect with no error to the dbs locally and remotely, but when trying to dump or import locally I get

/bin/sh: mysql: command not found
/bin/sh: mysqldump: command not found

Surely there's something simple I'm missing.

Had to remove the " in my db password in order to make it work.

Error happened in line 217

var output = shell.exec(cmd, {silent: false}).output;

is this due to compression?
my db is quite huge...

these is the output:

grunt db_pull --target='prd'
Running "db_pull" task

Pulling database from 'PRD' into Local
Creating DUMP of remote database
FATAL ERROR: CALL_AND_RETRY_0 Allocation failed - process out of memory

This could come in handy where user/customer data could possibly be changed between a push and pull that you wouldn't want to risk losing

$$ grunt db_pull --target="prod"
Running "db_pull" task

Pulling database from 'Production' into Local
Creating DUMP of remote database

Database DUMP succesfully exported to:
backups/prod/20140730/00-07-49/db_backup.sql
Replacing 'www.production.com' with 'www.local.com' in the database.
sed: can't read s#www.production.com#www.local.com#g: No such file or directory
Database references succesfully updated.

It appears to be treating the edit string like the file name.

Despite the fact that it says references were updated, there is no output file created.

I'm testing the feature/advanced-search-replace branch and when I run:

grunt db_pull --src="production" --dest="local"

I get the following error:

/bin/sh: node_modules/search-replace-db/srdb.cli.php: No such file or directory

From what I can see, is this something to do with scriptPath not having the correct path to srdb.cli.php ?

Currently I'm not positive the Plugins is using the "options" format expected by the Grunt documentation.

Revisit this.

Hi

Looks like Gulp is gaining some interest these days. Could you be able to port your plugin ? It's the most useful to me, so until Gulp has something equivalent, I won't switch :)

Thanks.

Hello,
I am getting a sed error when replacing database URLs. The problem is easy to understand, there's no escape for the URLs I have into the Gruntfile.js so the sed command will be:

sed -i '' 's/http://aaa.com/http//bbb.com/g' <%= path %>

This is a problem because the slash inside the command are not escaped and makes confusion on sed. A very quick fix should be the use of another sed delimiter, like #instead of /.

Although in the scenario that the Plugin was designed for requires SSH access to pull/push to remote DBs, some hosts allow direct MYSQL connections via the -h option.

mysql -h 192.168.1.8 -u root -p

This is uncommon but it happens. We need to allow for this scenario. This means we cannot use the prescence of ssh_host to determine the "local" environment. Needs a separate boolean flag?

• modify test which determines the "local" environment
• modify DB commands to cater for usage via SSH or otherwise
• modify code to run S&R on tmp files only. Preserve original DB dumps

Currently only very poor basic tests exist. To ensure this repo can continue to develop better tests are required.

I'm currently using Vows.

Causes database imports to fail.

I'm using MySQL 5.6, which has improved security, which might be the problem.

Fixed by modifying tpls strings to set password variable:

     mysqldump: "MYSQL_PWD=<%= pass %> mysqldump -h <%= host %> -u<%= user %> <%= database %>",

     mysql: "MYSQL_PWD=<%= pass %> mysql -h <%= host %> -u <%= user %> <%= database %>",

Question:

This might be obvious but I couldn't figure out how to do it. Is there a way to run the db_pull command as a task in GruntFile.js?

I figured something like the following would work

deployments: {
local: {...},
staging: {...},
pull_db_staging: {
db_pull: {
target: 'staging'
}
}
}

grunt.registerTask('pull-stg', ['deployments:pull_db_staging','rsync:pull_uploads_staging']);

mysqldump returns 'bash: mysqldump: command not found', but launching it directly on server works well.

Currently we expect the user to want to run a search and replace on a single ref in the database. This is url because WordPress (which the Plugin is designed for) stores siteurl in the database and it needs updating when you migrate the DB to a new environment.

However some scenarios might call for several DB refs to be replaced. Suggest renaming the option to be searchreplace and allow the user to define a series of search and replace statements.

Suggestion:

searchreplace: {
    ['search_term1','replace_term1'],
    ['search_term2','replace_term2']    
    ['search_term3','replace_term3']    
}

This plug-in is working as expected until I place an absolute path in a Text Widget and run the db_push task. For example, if I have place an image in a text widget:

<img src="http://example.com/wp-content/uploads/test.jpg" alt="Test Image" />

or have a link to an internal page:

<a href="http://example.com/internal-page/">Internal Page</a>

the database will successfully push to the development target except none of the widget data from the local database will migrate over. If I change the paths to be relative:

<img src="/wp-content/uploads/test.jpg" alt="Test Image" />

<a href="/internal-page/">Internal Page</a>

everything will migrate over perfectly.

I have not been able to figure out a way to use db_pull or db_push in a list of tasks because of the need for the "--target=" parameter.

Is there a reason not to use the "standard" grunt task syntax, e.g. task:target?

I solved it for my purposes by modifying db_pull and db_push to use:

var target = grunt.option('target') || (this.args.length >= 0 && this.args[0]) || task_options['target'];

so that --target has precedence but task:target can be used also.

While importing dumped (Errored file content) sql file, i am having "You have an error in your SQL syntax" but programm moves forward and writes "Operations completed. Done, without errors."

I have a Windows 8.1 with Wamp locally and I work with a remote linux platform. Your grunt plugin is fantastic but it didn't worked for me until I made some changes which can interest you.

To make it work on Win platform, there is 2 problem:

1. Sed doesn't work on Windows platform.
   As shelljs is a dependencie, you can use:

       //Execute SED
       shell.sed("-i", search, replace, output_file);
   

   to replace:

   
        var cmd = grunt.template.process(tpls.search_replace, {
            data: {
                search: search,
                replace: replace,
                path: output_file
            }
        });
   .....
       // Execute cmd
        shell.exec(cmd);
   

   in db_replace function and you don't need search_replace template either.

2. Problem w/ command backslashes in db_dump function

   cmd = tpl_ssh + " \\ " + tpl_mysqldump;
   

   I just had to remove it (\\ ) to make it work for me. If not, mysql_dump is not launched. I don't know if it's crucial for other platforms.

To stick to JS conventions the backups_dir option should be camel case.

I had a testing setup where my mysql user on the development server had an empty/no password. The mysqldump function did not work in this case.

I'm testing the feature/advanced-search-replace branch and when I run:

grunt db_pull --src="production" --dest="local"

I get the following error:

Error: Missing argument, -n or --name is required

I realise that argument is required by advanced-search-replace, but shouldn't the script already have that info from the Gruntfile?

I tried ``grunt db_pull --src="production" --dest="local" -n {database-name}` and a few combinations of that, but now get:

>> Local Npm module "{database-name}" not found. Is it installed?

Some VPS (MediaTemple) don't use localhost as the mysql host. This causes the mysqldump to fail silently. An error is reported when importing to local db.

I ran into an issue with a password that contained special characters.

This can be somewhat mitigated against by using the long hand for the password option

--password='mypassword9808'

This is the error I'm getting when I try grunt db_push --target="stage"

Importing DUMP into remote database
stdin: is not a tty
ERROR 1064 (42000) at line 1: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '/bin/sh: mysqldump: command not found' at line 1

Any ideas?

Alex

i got this error when running db_pull "cannot read property 'options' of undefined use".

Any one could tell me what's wrong or what's causing this ?

I am running it on windows without the replace url thing.

My remote database dumps were hanging here:

dnewton@newtron ~/Sites/gh/great 👍  grunt db_pull --target="dev"
Running "db_pull" task

Pulling database from 'Dev DB' into Local
Creating DUMP of remote database

I traced this to deployments.js line 208, cmd = tpl_ssh + " \\ " + tpl_mysqldump;. My system was not executing the remote command after connecting. By changing the line to cmd = tpl_ssh + " '" + tpl_mysqldump + "'";, it was able to execute the remote command.

I'd do a PR, but this is obviously working fine for some people, so I'm guessing it has something to do with the version of SSH or the OS. My local machine is running OS X Yosemite, and the SSH version is "OpenSSH_6.2p2, OSSLShim 0.9.8r 8 Dec 2011".

It's too easy to forget to version bump. Use Grunt Bump to enable this.

IN the readme the backups_dir option is written as backup_dir. Fix

Does the find/replace part of this task account for serialized data?

My backups folder is getting pretty big on several of my projects and would be great if we could tidy during a pull or push.

Rename "db_pull" to "db_migrate" because it's confusing!

In a team scenario it's likely that each person will have a different local environment. Therefore it's imperative that the configuration options for the local target must be outside of VCS.

This is easily achievable via a separate JSON file which is included in the configuration. This external file can then be ignored and omitted from VCS.

This can be done already, but the user would need to have a relatively advanced understanding of Grunt to be able to execute. Make this a requirement of the plugin or provide detailed instructions.

Under heading: Other Environment Targets

Lines containing url definitions have missing commas at the end of their lines.

e.g.

"url": "development_db_url" 

should be

"url": "development_db_url",

As a definition proceeds it (ssh_host)

When working in a team, it would be nice to have the possibiltiy to setup multiple local configurations for each team meber. You could then maybe select the local database with another optional parameter.

It is reasonably common for hosting services to use a non-standard SSH port, e.g. 2222.

Hi. It would be great to be able to ignore some tables. For example, I've a Wordpress with the Relevanssi plugin installed, which is kind of a big search index; it doesn't have any value for the developer and it's really slowing down the process.

I'd love something like :

options {
    ignoreTables: 'wp-relevanssi,[...]',
    ...
}

Do you think it's possible ?
Thanks.

Currently we build up manual SSH commands and execute them via shelljs. Often this involves quite complex templating and can become confusing. It also doesn't provide that much protection in terms of errors and feedback for the user.

Consider removing all manually executed SSH commands in favour of

https://github.com/mscdex/ssh2

Commands can be executed in series directly on the remote machine via a single SSH connection.

It would be useful to have an external file (as done in github.com/zonak/grunt-ftp-deploy) to save private data (password) outside of the gruntfile (and repositories).

Even though there are defaults for all options, if there are no options specified I get:

Warning: Cannot read property 'backups_dir' of undefined Use --force to continue.

Aborted due to warnings.

It would be more flexible if it was possible to specify a "privateKey" option, as is done by grunt-rsync and grunt-rsyncwrapper (and, in fact, rsync itself).

This could eliminate the need for a separate key-agent process.

At the moment the script creates directories for sql dumps based on the date and time at the point the script was run.

However it might also be useful to store them by git commit hash.

In which case we should allow the script to be configurrable enough to allow this.

Btw, a slick one liner to get the last commit hash is

git log -n1|head -n1|grep -o '[[:xdigit:]]\{7,\}'

Receiving error:
"Importing DUMP into remote database
stdin: is not a tty
ERROR 1064 (42000) at line 1: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '/bin/sh: mysqldump: command not found' at line 1"

Any idea how to fix this? Much appreciated!

The preferred method for sharing options across multiple tasks is as per.

https://github.com/gruntjs/grunt/wiki/Frequently-Asked-Questions#dynamic-alias-tasks