flowingis / idephix

💥 Automation and deploy tool - ALPHA release - things can catch fire in any moment!

Idephix - Automation and Deploy tool

Idephix is a PHP automation tool useful to perform remote and local tasks. It can be used to deploy applications, rotate logs, synchronize data repository across server or create a build system. If you want to learn more about how to use it read the docs.

Installation / Usage

1. Download the idephix.phar executable and init your idxfile.

   $ curl -LSs http://getidephix.com/idephix.phar > idephix.phar
   $ chmod a+x idephix.phar
   $ idx initFile

2. Now you can define tasks just defining php functions in your idxfile.php

   <?php
   
   /**
    * Execute the touch of a file specified in input
    * @param string $name the name of the file to be touch-ed
    * @param bool   $go   if not specified the script execute a dry-run
    */
   function testParams(\Idephix\Context $context, $name, $go = false)
   {
        $context->local('touch /tmp/'.$name);
        $context->remote('touch /tmp/'.$name.'_remote');
   }

For more information about how to define tasks, configuration for multiple environments and much more read the docs. Global installation of Idephix

Deploying with Idephix

Idephix is good for many different jobs, but we like to use it especially for application deployment. In fact out of the box your idxfile.php will be initialized using our recipe for PHP application deployment that you can use as a starting point for your projects.

Requirements

PHP 5.3.2 or above, >=5.3.12 recommended

Authors

• Manuel 'Kea' Baldassarri [email protected]
• Michele 'Orso' Orselli [email protected]
• Filippo De Santis [email protected]
• other contributors

License

Idephix is licensed under the MIT License - see the LICENSE file for details

idephix's Issues

global settings

There are few settings which could be tweaked via conf like:

• default timeout for commands: now you have to specify it in every local invocation
• failonerror: the default behaviour is to stop when something goes wrong, maybe there are some use cases where the script should continue

makes sense? I'm for creating a single array with all the needed configuration

Shippable is broken

Shippable is broken for all PR. The problem is on php 5.3.3 build.

I think that problem is on phpenv-install.

https://app.shippable.com/builds/552c3d4e6206490c0097364f

How we could fix it ?

Deploy recipe

As discussed in #71 we're going to remove from the extension library all the stuff that is highly related to the user's project, like Deploy. Instead of providing an Extension that tries (and fails) to do everything we'll provide some recipes to use as customizable boilerplate code in every project.

A recipe is easily adaptable to the user's specific need, but still provide a quick and reliable starting point.

This is my proposal for a deploy recipe (will be part of cookbook #78):

function prepareDeploy(\Idephix\IdephixInterface $idx)
{
    $remoteBaseDir = "/var/www/testidx";

    //prepare remote structure, if needed
    $idx->remote("
        mkdir -p {$remoteBaseDir}/releases && \\
        mkdir -p {$remoteBaseDir}/shared
    ");
}

function deploy(Idephix\IdephixInterface $idx)
{
    $shared = [
        'app/config/parameters.yml',
        'cache',
        'logs',
        'web/assets'
    ];
    $sshHost = "labs.ideato.it";

    $remoteBaseDir = "/var/www/testidx";
    $nextRelease = "$remoteBaseDir/releases/" . time();
    $linkedRelease = "$remoteBaseDir/current";
    $rsyncExclude = "./rsync_exclude.txt";
    $rsyncInclude = "./rsync_include.txt";
    $localArtifact = ".deploy";
    $repository = "file:///Users/ftassi/workspace/testidx";

    //prepare the deployable release (locally)
    $idx->local(
        "
        rm -Rf {$localArtifact} && \\
        git clone {$repository} {$localArtifact} && \\
        cd {$localArtifact} && \\
        git fetch && \\
        git checkout --force origin/master && \\
        composer install --no-dev --prefer-dist --no-progress --optimize-autoloader --no-interaction
    "
    );

    //copy the current release into the next release so rsync will not transfer unmodified file
    try{
        $idx->remote("cd $remoteBaseDir && cp -pPR `readlink {$linkedRelease}` $nextRelease");
    }catch (\Exception $e){
        $idx->output()->writeln("<info>First deploy, sending the whole project</info>");
    }

    //sync next release
    $idx->local(
        "rsync -rlpDvcz --delete --exclude-from={$rsyncExclude} --include-from={$rsyncInclude} {$localArtifact}/ {$sshHost}:{$nextRelease}"
    );

    //prepare shared items for next release
    foreach ($shared as $item) {
        $idx->remote("ln -nfs $remoteBaseDir/shared/$item $nextRelease/$item");
    }

    //link next release as current release
    $idx->remote("
        cd $remoteBaseDir && \\
        ln -nfs $nextRelease current
    ");
}

It is not final yet, there is some logic that we may want to extract to a Deploy extension. What I need now is a feedback about the overall procedure.

wrap settings in a class

The configuration defined in $target inside an idxfile.php is now available inside idephix as an array. It could be useful to wrap the configuration settings in a class which:

• hides the actual implementation, simplifying future changes
• makes easy to get values, allowing for default values eg:

$config->get('ssh_params.user', 'kea')

vs

if (isset($config['ssh_params'] && isset($config['ssh_params']['user'])

Compatibility with PHP 5.3

We are using the new array syntax that is breaking the compatibility with php 5.3:
https://travis-ci.org/ideatosrl/Idephix/jobs/132074103

Move releases to S3

Right now we're pushing the archive (each deployed release) to the website repo. This will make it heavy.

Shall we store phars on S3 instead?

ssh_params port doesn't work

I try to use

'ssh_params' => array( 'user' => 'my_user', 'port' => 8888 ),

but ssh connection is always on port 22.

Define a cookbook

We should define a set of recipes for common tasks such as deployment or build.

This is even more importante given that on #71 we discussed about dropping some extensions

move idx object creation outside idxfile

With the current settings the developer is in charge of creating an idephix object inside the idxfile. While this solution is very effective is has some downsides:

• it exposes end users to BC breaks on the Idephix class
• binds the Idephix object to a given idxfile: implementing an idxfile lookup mechanism a la vagrant or specifiing a given idxfile can be done only outside Idephix itself

Idxfile should be only about specifing configuration and task, all the heavy lifting should be done outside it
My proposal is to inject $idx object in the idxfile a la codeception

Looking forward to feedback

Fix generated idxfile

generated idxfile has some issues:

• Missing type hinting for $idx params
• Missing variables in idxrc
• Integrate it with new deploy recipe: see #71

Rsync excluded folder from Target settings

The method rsyncProject() contained within the Idephix/Extension/Project/Project.php class accepts as third parameter a file used for exclude patterns from the sync.

It would be useful to look up the file even from the target settings if the parameter is not explicitly declared.

Something like that:

if( !$exclude ) {
    $exclude = $target->get('deploy.rsync_exclude_file');
}

add tag to released versions

idephix version is identified by the commit hash eg:

Idephix version d123fb7faa23338918bd84f6c0aceff51ed72ced released 2016-07-13 09:16:31

since now we are tagging releases we can also add this reference using the placeholder git-tag

Command with arguments doesn't work with new style configuration

The problem that has been shown in #62 (comment) is that idx commands with arguments are broken, actually the tests fail with error similar to:

Fatal error: Uncaught exception 'RuntimeException' with message 'Too many arguments.' in /home/travis/build/ideatosrl/Idephix/vendor/symfony/console/Symfony/Component/Console/Input/ArgvInput.php on line 181
RuntimeException: Too many arguments. in /home/travis/build/ideatosrl/Idephix/vendor/symfony/console/Symfony/Component/Console/Input/ArgvInput.php on line 181

Call Stack:
0.0001     239520   1. {main}() /home/travis/build/ideatosrl/Idephix/bin/idx:0
0.0032     645272   2. Idephix\run() /home/travis/build/ideatosrl/Idephix/bin/idx:21
0.0148    2254016   3. Symfony\Component\Console\Input\Input->bind() /home/travis/build/ideatosrl/Idephix/src/Idephix/bootstrap.php:14
0.0148    2253472   4. Symfony\Component\Console\Input\ArgvInput->parse() /home/travis/build/ideatosrl/Idephix/vendor/symfony/console/Symfony/Component/Console/Input/Input.php:61
0.0149    2255712   5. Symfony\Component\Console\Input\ArgvInput->parseArgument() /home/travis/build/ideatosrl/Idephix/vendor/symfony/console/Symfony/Component/Console/Input/ArgvInput.php:90

The exit status of the process should be coherent

Actually when I run test with idephix the exit status is always successful even when the tests fail.
This causes problem with jenkins that cannot undestand when the build fails or not.

The workaround found by @dymissy is to force the exit status in the task:

$idx->add('asdasda',
    function ($message = null, $go = false) use ($idx) {
        try {
            $idx->doSomething();
        } catch(\Exception $e) {
            $idx->output->writeln(sprintf("<error>Exception: \n%s</error>", $e->getMessage()));
            exit(1);
        }
    })->

Maybe we could find a way to do it automatically.

Local command failure

In the actual implementation of "idx->local()" (see https://github.com/ideatosrl/Idephix/blob/master/src/Idephix/Idephix.php#L263 ), and exception is thrown when a command fails. This happens even for command like "rm <not_existing_filename>".

What do you think of adding a parameter to the local method to skip the exception in case of command failure?

Libraries VS Extensions, we need consistence

We should pick one :D
Right now we have an Extension namespace and a \Idephix\IdephixInterface::addLibrary method.

I prefer Extension over Library, what do you thing @kea ?

cache:clear is invoked after migrations execution

cache:clear is invoked after migrations execution, this means that if install and configure the doctrine migration bundle after the first installation the new config.yml file is not loaded until the cache is cleared and the parameters describing migration paths are not accessibile resulting in a runtime error, this could happen on any command relying on configuration parameters that are loaded from cache.

Autocomplete on the command line

I could be useful to get autocompletion for task names (and maybe options) on the command line. I did a quick research and there are few ways we can achieve that

https://github.com/stecman/symfony-console-completion
https://github.com/bamarni/symfony-console-autocomplete
https://github.com/robbyrussell/oh-my-zsh/blob/master/plugins/symfony2/symfony2.plugin.zsh
https://raw.githubusercontent.com/jaytaph/SFConsole/master/console_completion.sh

what do you guys think?

Restore --go option

➜  Idephix git:(master) bin/idx --go
PHP Fatal error:  Uncaught exception 'RuntimeException' with message 'The "--go" option does not exist.' in /Users/ftassi/workspace/Idephix/vendor/symfony/console/Symfony/Component/Console/Input/ArgvInput.php:213
Stack trace:
#0 /Users/ftassi/workspace/Idephix/vendor/symfony/console/Symfony/Component/Console/Input/ArgvInput.php(154): Symfony\Component\Console\Input\ArgvInput->addLongOption('go', NULL)
#1 /Users/ftassi/workspace/Idephix/vendor/symfony/console/Symfony/Component/Console/Input/ArgvInput.php(86): Symfony\Component\Console\Input\ArgvInput->parseLongOption('--go')
#2 /Users/ftassi/workspace/Idephix/vendor/symfony/console/Symfony/Component/Console/Input/Input.php(61): Symfony\Component\Console\Input\ArgvInput->parse()
#3 /Users/ftassi/workspace/Idephix/src/Idephix/bootstrap.php(14): Symfony\Component\Console\Input\Input->bind(Object(Symfony\Component\Console\Input\InputDefinition))
#4 /Users/ftassi/workspace/Idephix/bin/idx(21): Idephix\run()
#5 {main}
  thrown in /Users/ftassi/workspace/Idephix/vendor/symfony/console/Symfony/Component/Console/Input/ArgvInput.php on line 213

Fatal error: Uncaught exception 'RuntimeException' with message 'The "--go" option does not exist.' in /Users/ftassi/workspace/Idephix/vendor/symfony/console/Symfony/Component/Console/Input/ArgvInput.php on line 213

RuntimeException: The "--go" option does not exist. in /Users/ftassi/workspace/Idephix/vendor/symfony/console/Symfony/Component/Console/Input/ArgvInput.php on line 213

Call Stack:
    0.0016     236040   1. {main}() /Users/ftassi/workspace/Idephix/bin/idx:0
    0.0183     526168   2. Idephix\run() /Users/ftassi/workspace/Idephix/bin/idx:21
    0.0438    1840440   3. Symfony\Component\Console\Input\Input->bind() /Users/ftassi/workspace/Idephix/src/Idephix/bootstrap.php:14
    0.0438    1840168   4. Symfony\Component\Console\Input\ArgvInput->parse() /Users/ftassi/workspace/Idephix/vendor/symfony/console/Symfony/Component/Console/Input/Input.php:61
    0.0439    1840528   5. Symfony\Component\Console\Input\ArgvInput->parseLongOption() /Users/ftassi/workspace/Idephix/vendor/symfony/console/Symfony/Component/Console/Input/ArgvInput.php:86
    0.0439    1841096   6. Symfony\Component\Console\Input\ArgvInput->addLongOption() /Users/ftassi/workspace/Idephix/vendor/symfony/console/Symfony/Component/Console/Input/ArgvInput.php:154```

Env VS Target

I think we should try to be more coherent on naming.
We ask the user to define targets in his configuration file, but then we ask him to specify a --env parameter to select one.

What do you think?

Write a "contribute" guide

We need to ease contributions from our users, this is very important.

We should write a brief introduction on how to contribute, which comprises tech stuff like coding standards but also info about our waffle kanban board, gitter chat and all the ways they can communivate with the @Ideato team.

new Idephix($config, ...) instead of new Idephix($targets...)

Why not inject a config assoc array (containg targets) instead an arrray of tagets? Sometime you need to set configuration keys for tasks that play with the local environment only... for example.

[Insight] The composer.json file should be valid

in composer.json

description : is missing and it is required
Posted from SensioLabsInsight

document "init-idx-file" and "selfupdate"

even if the name sounds obvious let's add a description showed in the help

I would like to preserve my current deploy dir after $releasesToKeep subsequent failing deploys

I don't know if it is a feature or a joke, but after $releasesToKeep +1 subsequent failing deploys, the current directory will be erased.

Use SymfonyStyle methods also for manage input

Symfony's doc refers to SymfonyStyle as $io because it can be used both for input and output, shouldn't we do the same?

Allow to specify idxfile

In a CI/CD environment splitting build files by stage is a common practice. In idephix this is not possible atm since filename is hardcoded.

I propose to add a new option -f / --file in order do specify a given idxfile. The syntax would be something like that

idx -f myidxfile.php [args]

what do you think? @kea @ftassi @ricfrank @francescotrucchia

better config management

As for now the whole idephix configuration is managed by a plain array eg.

$targets = array(
    'test' => array(
        'hosts' => array('127.0.0.1'),
        'ssh_params' => array('user' => 'kea')
    ),
);

which is fine as long you don't need to modify dinamically one of the entries. One use case would be finding hosts ip dinamically by invoking aws api. One approach to solve this could be wrapping all the configurations in a Config o Context class which can ben injected in task a al symfony controllers

$idx-> add('idephix:test-params', function (Context $ctx, $go = false) use ($idx) {
         $ctx->setHosts('192.168.56.101');
});

waiting for feedback @francescotrucchia @ftassi @ricfrank @kea

idephix is no more compatible with the standard file format

Now that http://getidephix.com/idephix.phar is updated, my continuous integration fail because:

PHP Catchable fatal error:  Argument 1 passed to Idephix\Idephix::__construct() must be an instance of Idephix\Config, array given, called in /..../idxfile.php on line 18 and defined in phar:///..../idephix.phar/src/Idephix/Idephix.php on line 42

We can choose to stay compatible or not, but maybe we should add some documentation on how to migrate to the new format, or even better a hint inside idephix when is called in the wrong way.

Exception "Destination file or directory exists" when switching from no shared_folders to some shared_folders

If you start doing releases without shared_folders and then you add something to it you get Exception.

[Exception]
Unable to link shared directory "/[...]/app/logs". Destination file or directory exists.

Fixing StyleCi?

We have StyleCi enabled on this project but I think is not properly configured, for instance https://styleci.io/analyses/zY6p9A (we cannot use short array syntax unless we drop php 5.3)

I think we have 2 choices:

• enforce coding standard ad a build step (dropping styleci)

• configure styleci properly

  what do you guys think?

General commands such as "selfupdate" do not work without a idxfile.php

When I istall idephix.phar globally (ie. in /usr/bin/), and I run "idephix.phar serlfupdate" it asks me to generate the idxfile.php file.
To get this to work I have to be in a folder where there is a idxfile.php.

Show user defined tasks in a different section

When showing the command line help user defined tasks are mixed with default idephix tasks such ad "selfupdate", "list", "init-id-file".

Showing user defined tasks in a separate section could add clarity, what do you guys think?

it should be nice to see a progress bar with the status for the current task

such as http://symfony.com/doc/current/components/console/helpers/progressbar.html

Multiple hosts and autoexit

Ciao, I was just playing a bit with Idephix. I wanted to see if I could deploy my project on 2 remote servers.

So I set hosts for my environment like this

'hosts' => array('4.5.6.7', '1.2.3.4'),

but didn't work, the code was sent only to the first server.

After a bit of debugging I've noticed that Application is, by default, set to exit after run (see property autoExit ) so the script was always stopping at this line https://github.com/ideatosrl/Idephix/blob/master/src/Idephix/Idephix.php#L175

A couple of questions:

• Is my description correct or I'm missing something ?
• Shouldn't autoExit be set to false by default in Idephix/Application ? Even if you have one host the method closeRemoteConnection is not run.

Thank

simplify the release of a new idephix version

Managing code and docs in two different branches is a pain in the a**. Releasing a new idephix phar is unnecessary complicated as it involves:

1. create the phar in the code branch
2. switch branch
3. add and commit the new phar
4. update the documentation, remembering what you've done in the other branch

I have some solution in mind

• drop github pages and switch to readthedocs or similar, managing only one repo
• split docs and code repo

And in general, automating the release process with a Idephix script