proboci / probo Goto Github PK

The build steps that we have above execute in the order:

• install node
• npm install
• es lint
• sass lint
• install test suite
• run tests

However, when looking on probo.ci the install node task is showing up after sass lint.

Most Wordpress sites have custom MySQL table prefixes other than those starting with "wp_". Ideally the constant for $table_prefix should be allowed to be changed in the probo.yml possibly the same way the databaseName is configurable.

The example config here https://docs.probo.ci/plugins/wordpress-plugin/ would change to something like:

  assets:
    - dev.sql.gz
    steps:
    - name: Site setup
      plugin: WordPressApp
      database: dev.sql.gz
      databaseName: wordpress
      tablePrefix: wp_abcxyz
      databaseGzipped: true
      subDirectory: code
      devDomain: example.com
      devHome: example.com/home
    - name: Flush the cache
      plugin: WordPressApp
      flushCaches: true

My client hasn't cancelled their app.probo.ci account yet, so I can see that my PR's are still building on that platform, as well as my new app2.probo.ci account.

Builds on app are completing in 3 minutes, builds on app2 are completing in 8 to 11 minutes. I can't predict what will alter the times - but I can see that it's almost all in database import. app and app2 are both working from the same database export.

When I add a Github label to a PR that already had a probo site spinned up, the labeling causes probo to rebuild. If someone was testing with the first probo build, and someone else added a label to the PR, then the tester's work would get wiped out and that shouldn't happen.

Steps to reproduce:

• Create a Github PR for a repo using probo
• Wait for the probo build on the PR to complete successfully
• Then add a Github Label to that PR

Actual Result:

• See that probo rebuilds (wiping out the probo site build previously created)

Expected Result:

• Probo shouldn't rebuild for that PR because no commits or changes were made to the PR.

Right now all components communicate synchronously via HTTP requests (it's node, so they're async calls but one service waits on another and the message fails to pass if the service goes down). We can (and should) be able to use a message queue to make this more reliable and decrease the risk of overwhelming the workers. This will also enable us to restart the container manager service without the risk of orphaning a builds half way through (and while we're at it we should store what containers need to be stopped and when so that we can still shut down old containers after a restart).

My approach is to create a PublishEvent interface that can fire off post requests, issue events through an EventEmitter (for in process event transmission especially useful for mocking in unit tests), and enable us to use a message queue like kafka.

I see the new release v3.11.0 added 2 hours ago, and after that my probo.ci project get failed with the following errors on Drupal provision step (Mysite site provision) with the following errors:

[error]  SQLSTATE[42S02]: Base table or view not found: 1146 Table 'drupal.undefinedkey_value' doesn't exist: DELETE FROM {key_value} 
63WHERE (name IN (:db_condition_placeholder_0)) AND (collection = :db_condition_placeholder_1); Array
(
    [:db_condition_placeholder_0] => drupal_css_cache_files
    [:db_condition_placeholder_1] => state
)

here is my .probo.ci file

basicAuth:
  username: user
  password: pass
assets:
  - db.sql.gz
steps:
  - name: Upgrade to php 7
    plugin: Script
    script: |
      echo 'exit 0' > /usr/sbin/policy-rc.d
      apt-get update
      apt-get install -y software-properties-common language-pack-en-base
      LC_ALL=en_US.UTF-8 add-apt-repository ppa:ondrej/php
      apt-get update
      DEBIAN_FRONTEND=noninteractive apt-get install -y php7.1 libapache2-mod-php7.1 php7.1-gd php7.1-curl php7.1-json php7.1-mbstring php7.1-mysql php7.1-mcrypt php7.1-imagick php7.1-dev php7.1-gmp php7.1-xml
      cp /etc/php5/mods-available/general_settings.ini /etc/php/7.1/apache2/conf.d/20-general_settings.ini
      a2dismod php5
      a2enmod php7.1
      a2enmod mpm_prefork
      apache2ctl graceful
  - name: opcache
    plugin: Script
    script: |
      echo 'opcache.enable=1' >> /etc/php/7.1/mods-available/opcache.ini
      echo 'opcache.enable_cli=1' >> /etc/php/7.1/mods-available/opcache.ini
      echo 'opcache.memory_consumption=192' >> /etc/php/7.1/mods-available/opcache.ini
      echo 'opcache.interned_strings_buffer=16' >> /etc/php/7.1/mods-available/opcache.ini
      echo 'opcache.max_accelerated_files=3907' >> /etc/php/7.1/mods-available/opcache.ini
      echo 'opcache.validate_timestamps=0' >> /etc/php/7.1/mods-available/opcache.ini
      echo 'opcache.fast_shutdown=1' >> /etc/php/7.1/mods-available/opcache.ini
      apache2ctl graceful
  - name: Mysite site provision
    drupalVersion: 8
    plugin: Drupal
    databaseGzipped: true
    databaseUpdates: true
    clearCaches: false
    database:db.sql.gz
    databaseName: drupal
    databaseUser: drupal
    subDirectory: docroot
    settingsAppend: |
      $config_directories = ['sync' => 'sites/default/config/sync'];
  - name: Post deploy commands
    plugin: Script
    script: |
      drush cim -y
      drush en -y stage_file_proxy
      drush -y config-set stage_file_proxy.settings origin 'http://url.com/'
      bin/drush -y config-set system.image toolkit 'gd'
      drush -y config-set system.file path.temporary '/tmp'
      drush cr
  - name: Uninstall Node
    plugin: Shell
    command: 'apt-get remove -y nodejs'
  - name: Add nodesource debs
    plugin: Shell
    command: 'wget -qO- https://deb.nodesource.com/gpgkey/nodesource.gpg.key | apt-key add - ; echo "deb https://deb.nodesource.com/node_4.x trusty main" > /etc/apt/sources.list.d/nodesource.list ; echo "deb-src https://deb.nodesource.com/node_4.x trusty main" >> /etc/apt/sources.list.d/nodesource.list'
  - name: Install nodejs
    plugin: Shell
    command: 'apt-get update ; apt-get install --fix-missing -y nodejs'
  - name: Install gulp-cli
    plugin: Shell
    command: 'npm install -g gulp; npm install -g gulp-cli'
  - name: Install bower
    plugin: Shell
    command: 'npm install -g bower'
  - name: Pull down node modules
    plugin: Shell
    command: 'cd $SRC_DIR/docroot/themes/mytheme; npm install;'
  - name: Install bower dependecies
    plugin: Shell
    command: 'cd $SRC_DIR/docroot/themes/mytheme; bower --verbose --allow-root install;'
  - name: Compile frontend assets
    plugin: Shell
    command: 'cd $SRC_DIR/docroot/themes/mytheme; gulp build;'

We should send a build failure to Github if .probo.yaml can't be parsed in the github handler.

I would expect if it's deleted, to not be able to delete/pin/view site for a build, but all of those buttons are still present.

I am seeing documentation for a GitHub implementation. Is there any existing support or documentation for BitBucket?

Thank You;
Michael

On app2.

What I did:
I have three different pull requests A B C, totaling 9.03G out of my 10G limit. All were active and none should be significantly larger than another. One build on C is pinned. I pushed a new commit to A, then a new commit to B.

What I expected to happen:
For reaping purposes, I expected one of the older PR's on B to be reaped, since now that one would be stale.

What actually happened:
The newest build on A that I had just built was reaped, as was the previous build on A. None of the builds on B were reaped, so I ended up with a stale B build I didn't need instead of the fresh build on A I just pushed.

Sometimes I only want a pinned build to last until the PR is closed (via merged or otherwise) so that clients can review it, but then I don't care about it anymore.

Currently, we have a number of pinned builds which don't need to be pinned but are still pinned. This is giving us a system where everything is pinned, therefore nothing is really pinned. It would be convenient to have a selectable duration for pins - forever/until PR closes/etc so that outdated pinned builds can be reaped again.

In Drupal 8 there's a new setting called trusted_hosts. This is interfering with probo builds. We should consider having the Drupal plugin append a snippet in settings.php for Drupal 8 sites like:

$settings['trusted_host_patterns'] = array(
  '^localhost$',
  '^.+\.probo\.build$',
);

In order to prevent deployments or other build steps, it would be great to have the option to stop the build process upon receiving an error.

Current behavior:
In some cases when a build step fails, for example when a behat test fails, the build process continues to the next step.

Proposed behavior:
Allow a configuration option that denotes when an error in a build step should terminate the process.

Allow a setting in the YAML file to turn off logging/stdout or at least turn it down to improve performance. This should be a standard setting across all executables.

https://github.com/ProboCI/probo/blob/master/lib/plugins/TaskRunner/Drupal.js#L37

It pulls from constants instead of from the LAMPApp options.

According to this video it's an idea so good that your boss was asking you about it :)

In this sample probo config, the database old-database.sql.gz is correctly identified and downloaded during site building, but new-database.sql.gz is not!

assets:
  - old-database.sql.gz
steps:
  - name: Setup Drupal site
    plugin: Drupal
    database: new-database.sql.gz
    subDirectory: public
    databaseGzipped: true
    databaseUpdates: true
    revertFeatures: true

To make this work properly, you must specify the database in both locations:

assets:
  - new-database.sql.gz
steps:
  - name: Setup Drupal site
    plugin: Drupal
    database: new-database.sql.gz
    subDirectory: public
    databaseGzipped: true
    databaseUpdates: true
    revertFeatures: true

Gitlab is one of the most used git storage.
Gitlab interaction stopped working on both app and app2 servers.
Assuming the open source version is not working either.

https://github.com/ProboCI/probo/blob/master/lib/plugins/TaskRunner/WordPressApp.js#L78
https://github.com/ProboCI/probo/blob/master/lib/plugins/TaskRunner/WordPressApp.js#L27

The actual database name that is used when building is hard coded.

Sending status updates to Github quickly for the same sha and context can lead to them being processed out of order and a success update being handled and saved before a pending update. Sending GH one status update at a time shouldn't have too big of a performance hit while making the system perform as expected and send updates in order.

I’ve made my way through the quick start guide and on the release/tag v3.3.2. Github is touching my server by way of the webhook as well.

After I submit a pull request from my Github repo, stdout on the server running Probo returns an error "Cannot read property 'build' of undefined"

Here is a snippet of the error log:

{"name":"probo","hostname":"li171-190","pid":22686,"component":"http","req_id":"470bb3c0-8fe9-11e6-aad3-ffe2a13c958f","level":20,"event":"status updated","data":{"context":"ProboCI-local/env","status":{"state":"error","description":"[■] Cannot read property 'build' of undefined","context":"ProboCI-local/env","target_url":"http://55ed4e903fc00000.probo.nodesafe.com:3050/","user":"brianyoungblood","repo":"probo_wordpress","sha":"153584b716cd99a72f56fe9a9aa25e395c378da2"}},"msg":"build event for build 55ed4e903fc00000: status updated","time":"2016-10-11T19:31:16.614Z","src":{"file":"/opt/probo/lib/ContainerManager.js","line":40,"func":"emitBuildEvent"},"v":0}
{"name":"probo","hostname":"li171-190","pid":22686,"component":"http","req_id":"470bb3c0-8fe9-11e6-aad3-ffe2a13c958f","level":30,"status":{"state":"error","description":"[■] Cannot read property 'build' of undefined"},"msg":"status updated","time":"2016-10-11T19:31:16.615Z","src":{"file":"/opt/probo/lib/ContainerManager.js","line":320,"func":"updateStatus"},"v":0}
{"name":"probo","hostname":"li171-190","pid":22686,"component":"container-manager","level":50,"err":{"message":"Cannot read property 'build' of undefined","name":"TypeError","stack":"TypeError: Cannot read property 'build' of undefined\n    at Container.<anonymous> (/opt/probo/lib/Container.js:193:39)\n    at /opt/probo/lib/ContainerManager.js:359:47\n    at next (native)\n    at onFulfilled (/opt/probo/node_modules/co/index.js:65:19)\n    at bound (domain.js:280:14)\n    at runBound (domain.js:293:12)\n    at runCallback (timers.js:574:20)\nFrom previous event:\n    at next (/opt/probo/node_modules/co/index.js:100:51)\n    at onFulfilled (/opt/probo/node_modules/co/index.js:69:7)\n    at /opt/probo/node_modules/co/index.js:54:5\n    at co (/opt/probo/node_modules/co/index.js:50:10)\n    at /opt/probo/lib/safeco.js:9:12\n    at Immediate.<anonymous> (/opt/probo/lib/ContainerManager.js:354:5)\n    at runCallback (timers.js:576:20)\n    at tryOnImmediate (timers.js:554:5)\n    at processImmediate [as _immediateCallback] (timers.js:533:5)"},"msg":"Uncaught co Exception","time":"2016-10-11T19:31:16.615Z","src":{"file":"/opt/probo/lib/ContainerManager.js","line":142},"v":0}

Let me know if anyone has thoughts or ideas on this one. Happy to test plausiable solutions!

-Brian

There is a an action item for every probo step in the check details panel on Github PRs. This seems to be unnecessarily verbose, especially when contrasted to similar services that only post one link (or limit the number of posts to actual full runs):

Refreshing the build page (not the build itself, but inside the web app) will fix the problem.

We're exited about Probo, but one worry is that it only builds PRs. We're used to every branch being build on push, as is common.

Any plans for branch building? If not build-all-branches, then at least being able to configure develop to always being build?

Got a Call to undefined function mcrypt_encrypt error with one of my builds. Can you update your docker image to include the mcrypt php module?

When running tests, I would like to have several different Drupal installs. The use case for this is I want to test an install profile with a couple different sql dumps, plus a fresh install. Each one would be a different "siteFolder".

The way I see this working is by simply entering "siteFolder: site1" into the Drupal plugin that it creates a multisite setup so that the url is: https://site1.UUID.probo.build.

Then, in the same .probo.yaml file, I could add a second Drupal site install for "siteFolder: site2".

Add a level of basic security by hiding them behind a configurable BA challenge.

There are often scenarios in which we'd want to rebuild manually. For instance, a build dependency has changed but the main repo has not. Most CI tools have a button to re-do the build using the same parameters. This would be a great addition.

I remember this was a sporadic issue on app, but on app2 it is much more common.

Since bower is usually globally installed by npm install -g bower can that be provided in the base image, rather than having to install it through the yaml file?

Minor issue: If I follow a link to a build from Github and am not logged into Probo, it takes me to a login screen and then just takes us to my top-level list of projects rather than the build I originally clicked on. Not a big deal at all, I go back to the PR on Github and click through again. But it does happen to me once a day or so, and for people new to the service it is often confusing.

I sent a client a link about a week ago for review. She logged in and made some changes (I know, I know) for me to look at. When I tried to log in last night, the database was presumably gone because I was sent to the drupal install page.

UPDATE: see first comment below, I'm pretty sure this issue was too specific at first

@dzink @lliss

Proposal for saving the error details of YAML loading steps.

Steps in the process of processing a PR where there is a YAML load error:

1. Our webhook receives the PR
2. It is sent to GithubHandler's processRequest method
3. processRequest tries to fetch the YAML file
4. fetchProboYamlConfigFromGithub is called to get the YAML
5. Error callback is found
6. Build up an error and post back to Github

I propose that we also save the update that was sent to GH in the local build store. This will also provide a point to emit this error via our eventbus to other services or webhooks that anyone using this project wants. The build object here should have a steps array, where this error should be saved. We'd just make sure that the step we save there is the same format as other steps.

This is due to a node 4.1.2+ change that broke how node handles unix sockets closing their connections prematurely. Creating this issue to track upstream:

Node issue (where fix will happen): nodejs/node#3947
Dockerode issue: apocas/dockerode#193

I'm trying out Probo CI today for the first time. I have to say, it's a relief to have a CI platform that actually includes a web server! And it works with PHP without any hassles! Definitely the easiest set up I've tried.

I ran into a wall when using the LAMPApp plugin: http://probo.ci/docs/plugins/lamp-plugin/

I specified a databaseName and databaseUser, which seemed to have worked fine. But then I got stuck. There's no databasePassword! How am I supposed to access the created database if I don't have a password? I tried just omitting the password and using the username as the password, neither worked.

It seems like we need these in the base image.

• php5-imagick
• imagemagick

How come that CI tool itself has coverage 53%?

Every BitBucket Pull Request has a step that indicates it never finishes

The BitbucketDownloader task is never marked as complete, despite all the other steps completing successfully.

After updating our Drupal 7 Probo config to use the new supported Docker images (for https://blog.probo.ci/probo-1404-and-1604-images-removed-security), our existing drush make based configuration no longer works and these errors can be seen in the ProboCI console (even after adding drupalVersion: 7 to our config):

$ drush make build-ua_quickstart.make.yml /var/www/html --no-cache
bash: line 7: drush: command not found
$ rsync -a /src/ /var/www/html/profiles/ua_quickstart
bash: line 8: rsync: command not found

At least part of the problem seems to be that the drush symlinks that was recently added in #187 are created until the addScriptAppendSettingsPHPSettings() step in the populateScriptArray() method of the Drupal.js Probo plugin which is after the addScriptRunMakeFile() step is run.

I have no idea why the rsync: command not found bash error is happening.

When running a Backdrop (or Drupal) SimpleTest suite, the --color option is quite helpful to quickly find failures in the terminal output. In the web UI for Probo, the colors are not decoded and instead display as their escape characters:

�[32mActions: Actions executing in a potentially infinite loop (ActionLoopTestCase) 13 passes, 0 fails, 0 exceptions, and 1 debug message
�[0m�[32mAJAX: AJAX forms on cached pages (AJAXFormPageCacheTestCase) 33 passes, 0 fails, 0 exceptions, and 10 debug messages
�[0m�[31mAdministration bar: Link types (AdminBarLinkTypesTestCase) 21 passes, 0 fails, 3 exceptions, and 4 debug messages
�[0m�[32mBatch API: Batch percentages (BatchPercentagesUnitTestCase) 18 passes, 0 fails, and 0 exceptions

Something like this might help: http://pablomarin-garcia.blogspot.com/2011/04/converting-ansi-to-html-how-to-convert.html

The white space from the terminal output in the web UI seems to be collapsed. My local output looks like this:

Status    Group      Filename          Line Function                            
--------------------------------------------------------------------------------

But Probo's UI is missing the spaces:

 Status Group Filename Line Function
-------------------------------------------------------------------------------- 

This is more of a proposal than an actual issue.

I would like to suggest that over time we move our projects to follow a set standard... http://standardjs.com/

While it is true there are aspects of the code we would have to adjust and that there are parts of the standard we might not actually like, I think the rules are quite good and sensible. The added advantage is that we don't have to maintain and enforce our own standard and maintain eslint files which is not something I find to be either rewarding, enjoyable, or a particularly good investment of time.

I'm not suggesting that we move everything at once, but some of the smaller projects could be converted pretty quickly, I'd guess, and over a few months of PRs, I'm sure the others would get sorted out pretty easily as well.

We would like to allow users to be able to upload a (g)zipped file of assets that will be extracted to Drupal's public files directory. This will prevent users from having to manually extract files in a separate step.

At least for us, our development process often involves working on several branches and repos during a single session of work. In our homegrown QA system (which we're trying to replace with Probo), sites get a subdomain for the repo they're built from and a sub-subdomain for the branch name. Probo's domain names are just a hash and I've often been confused about what build I'm actually looking at.

IE,

http://my-branch-name.repo.our-qa-server.com

vs

https://99a085dd-041d-419d-9bb7-bffe54ccb52c.probo.build

The --force-complete option for drush make is currently hard-coded into the Drupal Probo plugin.

This causes problems for us because, if we introduce a breaking change to our drush make file, the Drupal plugin step in our Probo CI build configuration still reports as "successful" even though drush make had problems and then our other Probo.CI config steps have problems due to missing modules, etc. The underlying problem is that the --force-complete option for drush make generates drush warnings but does not exit with an error status, which would be preferable for our purposes.

We would prefer for the hard-coded --force-complete to be be removed from the Drupal plugin and possibly even the addition of a makeArgs option that could be used to provide drush make options.

Currently we provide a set of environment variables in builds but they are somewhat incomplete. We provide the commit that was built but not the branch it belonged to - we should add this and evaluate what other metadata we can/should provide.

No matter if this.options.siteFolder /settings.php exists or not a default $databases array is written there which means Drupal DB prefixing is unsupported. See https://github.com/ProboCI/probo/blob/master/lib/plugins/TaskRunner/Drupal.js#L108

How can prefixing be supported within Probo?

A couple options:

1. plugin option for specifying prefix e.g. this.options.dbPrefix
2. plugin option for skipping default db array write e.g. this.options.useOwnSettings

I prefer option 2 because it allows more control over the full databases array. Its more of an advanced option so would mean that if the committed settings.php file was missing $databases tho that the step would fail.