wearejh / stripped-db-provider

An M2 module that provides stripped databases on a scheduled basis to a storage location of choice

Jh Stripped Db Provider

This module can be configured to automatically send stripped databases periodically into an Amazon S3 Bucket in the cloud.

Installation

Add the following repositories to the repositories section of your composer.json

{"type": "vcs", "url": "[email protected]:WeareJH/stripped-db-provider.git"},

Then install like any regular composer package

composer require wearejh/stripped-db-provider

Configuration

There are two areas of configuration. The first is project specific config, which is set in config.php and gets commited to the repository. The second is environment specific configuration which is added to the env.php file directly on the server.

Project Specific Configuration

All databases are stripped by default but you can define project specific tables whose data you wish to not be included in the dump by adding the following to your config.php file:

'system' => [
        'default' => [
            'stripped_db_provider' => [
                'dump' => [
                    'project_ignore_tables' => ['example_table_a', 'example_table_b']
                ]
            ]
        ]
    ]

Where project_ignore_tables is a list of project specfic tables.

Environment Specific Configuration

Once Installed and deployed to production, you will want to configure it so that it automatically sends stripped DBs to the S3 Bucket. Edit the env.php file manually and set the following config (values below are examples):

'system' => [
        'default' => [
            'stripped_db_provider' => [
                'general' => [
                    'enabled' => '1',
                    'project_name' => 'example-project',
                    'cron_expr' => '0 4 * * 0'
                ],
                'storage' => [
                    'bucket_name' => 'example-bucket-name',
                    'region' => 'example-bucket-region',
                    'access_key_id' => 'example-bucket-access-key',
                    'secret_access_key' => 'example-bucket-secret-access-key'
                ]
            ]
        ]
    ]

Values are described in the following table:

config	value
enabled	One of {0,1}
project-name	String. Unique name for the project (used to name the db dump)
cron_expr	A valid cron expression. Eg. "0 0/5 * * *"
bucket_name	String. AWS S3 Bucket name
region	String. AWS S3 Bucket region
access_key_id	String. AWS S3 Bucket access key id
secret_access_key	String. AWS S3 Bucket secret access key

The Amazon S3 Bucket Details should be in LastPass. Ask your fellow developers if you can't find it or need help setting up.

After you have edited the env.php file, run the following command immediately :

bin/magento app:config:import

Run upload manually

You can also manually trigger the stripped database upload from the command line by running the following command :

bin/magento wearejh:stripped-db-provider:upload-to-remote 

To do a full DB dump

bin/magento wearejh:stripped-db-provider:upload-to-remote --full

Import a remote dump locally

You can use the module to import a dump from S3 directly to your local. It will back up your local admin accounts and reimport them

bin/magento wearejh:stripped-db-provider:import-from-remote PROJECT NAME 

To skip admin accounts backup

bin/magento wearejh:stripped-db-provider:import-from-remote PROJECT NAME --no-admin-backup=1

Issues / Feature Request

Please open github issues for any issues you encounter or feature requests you want to see. PRs are of course welcomed.

Troubleshooting

If any project has 0.3.3 version installed of this module, getting a dependency error related to module: ifsnop/mysqldump-php ^2.12 while upgrading module to latest version then remove the below line from composer.json file of project root folder: { "type": "vcs", "url": "[email protected]:maciejslawik/mysqldump-php.git" }

stripped-db-provider's Issues

Not all installations have every table

I got the following in my dump:
mysqldump: Couldn't find table: "catalogsearch_fulltext_cl"

Not a huge deal, can just remove the line. But would be nice for the module to check existence of tables.

Db has to be emptied first

Database has to be emptied first + constant is missing

Starting the Database import
SQLSTATE[42S01]: Base table or view already exists: 1050 Table 'admin_adobe_ims_webapi' already exists
Undefined constant Symfony\Component\Console\Command\Command::SUCCESS#0 /var/www/html/vendor/symfony/console/Command/Command.php(255): Jh\StrippedDbProvider\Console\ImportFromRemoteCommand->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#1 /var/www/html/vendor/magento/framework/Interception/Interceptor.php(58): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#2 /var/www/html/vendor/magento/framework/Interception/Interceptor.php(138): Jh\StrippedDbProvider\Console\ImportFromRemoteCommand\Interceptor->___callParent('run', Array)
#3 /var/www/html/vendor/magento/framework/Interception/Interceptor.php(153): Jh\StrippedDbProvider\Console\ImportFromRemoteCommand\Interceptor->Magento\Framework\Interception\{closure}(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#4 /var/www/html/generated/code/Jh/StrippedDbProvider/Console/ImportFromRemoteCommand/Interceptor.php(77): Jh\StrippedDbProvider\Console\ImportFromRemoteCommand\Interceptor->___callPlugins('run', Array, Array)
#5 /var/www/html/vendor/symfony/console/Application.php(1021): Jh\StrippedDbProvider\Console\ImportFromRemoteCommand\Interceptor->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#6 /var/www/html/vendor/symfony/console/Application.php(275): Symfony\Component\Console\Application->doRunCommand(Object(Jh\StrippedDbProvider\Console\ImportFromRemoteCommand\Interceptor), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#7 /var/www/html/vendor/magento/framework/Console/Cli.php(116): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#8 /var/www/html/vendor/symfony/console/Application.php(149): Magento\Framework\Console\Cli->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#9 /var/www/html/bin/magento(23): Symfony\Component\Console\Application->run()
#10 {main}

Is there any clever workaround for it? Or separate command could be added to override existing db dynamically?

Add B2B tables to skipped tables

B2B tables should be skipped by default as long as customers are skipped. Right now, customers are removed but companies not and that leads to following problem when one tries editing company in admin panel:

Require confirmation before importing from remote

I know this is maybe kind of silly, but I am always scared I copy wrong command and by accident that could let me override whole live db for example. I think we should block import command somehow - for example for env with production mode or at least have a some questions and warnings in cli fr this command

src/Console/ImportFromRemoteCommand.php

Command to just dump db or atleast print command

Sometime I wanna get a prod db but they so god dam big. It would be nice to able to use this tool to get a stripped db using the exclusions already configured for the project without uploading to s3.

[FEATURE REQUEST] Allow dumping a full DB, without skipping tables

Add a flag to the upload command which will dump the full DB, without skipping the contents of some of them.

Now the tables https://github.com/WeareJH/stripped-db-provider/blob/master/src/Model/Db/DbTables.php#L13 are always dumped as structure only.

I'd like to have the option to specify a flag for the CLI upload command to dump all tables, e.g. --full

Common 3rd party modules

We already do ignore tables if they don't exist, this means we're able to provide extra tables without worry.

I'm proposing we add common 3rd party modules tables to prevent having to repeatidly add them in project specific ignores.

e.g. nosto, klevu, amazon