GithubHelp home page GithubHelp logo

square / kochiku Goto Github PK

View Code? Open in Web Editor NEW
599.0 34.0 51.0 3.86 MB

Shard your builds for fun and profit

License: Apache License 2.0

Ruby 88.87% JavaScript 1.09% HTML 7.61% Shell 0.06% CSS 2.37%

kochiku's Introduction

Kochiku - Distributed tests made easy

Kochiku is a distributed platform for test automation. It has three main components:

  • A web server, which lets you inspect builds and manage repositories
  • Background jobs that divide builds into distributable parts
  • Workers that run individual parts of a build

A single machine typically runs the web server and background jobs, whereas many machines run workers.

Use Kochiku to distribute large test suites quickly and easily. It's language agnostic; Use it for Ruby, Rails, Node.js, Ember, Java, C, C++ or anything else that runs in a unix environment.

Git integration

Kochiku currently integrates with git repositories stored in Github (including Github Enterprise) or Atlassian Bitbucket (formerly known as Stash). This lets Kochiku automatically run test suites for pull requests and commits to the master branch. Kochiku can also build any git revision on request.

Support for headless git servers is coming soon.

User Guide

kochiku's People

Contributors

bakineggs avatar bbtochi avatar beeflamian avatar bunkyoneko avatar capnhook avatar cheister avatar connorhd avatar denis avatar eventualbuddha avatar jackdanger avatar jakewharton avatar jawspeak avatar jdavisp3 avatar jeversmann avatar jkingdon avatar lossosatsquare avatar mariechatfield avatar matthewalbani avatar nicolasleger avatar nolman avatar robolson avatar shenil avatar simoleone avatar square-build-bot avatar squarenerd avatar tamird avatar tochi-square avatar xaviershay avatar zachblock avatar zachhaitz avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

kochiku's Issues

Error on service hook update.

If the service hook ID stored in the database ends up wrong (say because the hooks were edited by hand), the attempt to update the hook will fail and builds will not proceed.

response: Net::HTTPUnprocessableEntity body: {"message":"Validation Failed","errors":[{"resource":"Hook","code":"custom","message":"Hook already exists on this repository"}]}

Drop the deprecated on_success_script column

This feature moved into the kochiku.yml config file last summer. Anyone using it has had time to migrate so we can drop the column from the table and any code that references it now.

Should send an email after first build part failure

Currently Kochiku sends a build failure email after all build parts have completed. This means the email can get held up by a really long running build part. Developers would like quicker feedback, therefore we should switch from a build failed email to a build is doomed email.

Perhaps provide this as an option at the repository level. Short builds might prefer the current behavior whereas long builds would not.

What should Kochiku's default partitioning strategy be?

Currently if you don't specify a partitioning strategy in your kochiku.yml, your parts will be partitioned with the "alphabetically" strategy. I am thinking that the "round_robin" strategy makes more sense as the default.

The problem that alphabetically suffers from is that, for instance, if you have the same number of model and controller test, the models test will be in one part and the controller test in the other. If all of your controller test are slow and your models test are fast this defeats the partitioning strategy. "round_robin" would equally distribute the model and controller tests across the build parts.

Chef/Puppet recipe

What about recipe to automate server deployment?

Which system community prefer?

If chef is good enough I can write something like this recipe .

error during cap deploy: unable to create a directory

I'm trying to do a deploy and i'm getting the following:

** transaction: start
*** [err :: localhost] cp:
*** [err :: localhost] cannot create directory `/home/kochiku-web/kochiku/releases/20130908064030'
*** [err :: localhost] : No such file or directory
*** [err :: localhost]
*** [deploy:update_code] rolling back
failed: "env PATH=/usr/local/bin:$PATH sh -c 'cp -RPp /home/kochiku-web/kochiku/shared/cached-copy /home/kochiku-web/kochiku/releases/20130908064030 && (echo d7c117f > /home/kochiku-web/kochiku/releases/20130908064030/REVISION)'" on localhost

I'm installing on localhost just to test what exactly is deployed and installed.

I know that the cap environment can create directories because it created /home/kochiku-web/shared and all of it's sub-directories.

I worked around the issue by creating the releases/ directory manually.

Refresh checkbox does not actually work

The Elapsed Time column seems to increment, but the status does not actually update. I'm pretty sure the Elapsed Time column doesn't increment correctly either (the minute is off).

GitHub authentication for private repos

How would you recommend handling authentication for private GH repos during the clone operations for both the kochiku server and workers? The GitHub remote server model (https://github.com/square/kochiku/blob/bea210e11d350e3f01495996d71e14078e189a57/app/models/remote_server/github.rb) doesn't accommodate for a password file and auth methods like the stash version.

I have also found that SSH key-based authentication with GH doesn't work in the context of the resque workers.

Thanks in advance for your insight.

Pull requests from forks will not build correctly

Kochiku workers have been updated to support this however I am 99% certian that the mainline repository does not enqueue a job with the correct (ie the fork) repository url.

I have a branch that starts to add support for this however I haven't had a chance to finish it, wanted to make an issue to record this problem for the public.

Error with git submodules ?

Hello,
I am trying to make Kochiku CI work for a git repository that contains a submodule.

For some reason, the BuildPartioningJob fails. Here is what the Resq error queue gives me:

non-0 exit code pid 13187 exit 1 returned from [git submodule --quiet update]
/app/kochiku/kochiku/releases/20130919182453/lib/git_repo.rb:76:in `run!'
/app/kochiku/kochiku/releases/20130919182453/lib/git_repo.rb:30:in `block (2 levels) in inside_copy'
/app/kochiku/kochiku/releases/20130919182453/lib/git_repo.rb:17:in `chdir'
/app/kochiku/kochiku/releases/20130919182453/lib/git_repo.rb:17:in `block in inside_copy'
/usr/local/rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/tmpdir.rb:88:in `mktmpdir'
/app/kochiku/kochiku/releases/20130919182453/lib/git_repo.rb:13:in `inside_copy'
/app/kochiku/kochiku/releases/20130919182453/app/jobs/build_partitioning_job.rb:17:in `perform'
/app/kochiku/kochiku/releases/20130919182453/app/jobs/job_base.rb:22:in `perform'

I can trigger a green build for my repository without the submodule.
Any idea on what could go wrong? Do I need any special configuration to make Kochiku work with submodules?
Thanks

error during cap deploy: unable to access log file

*** [err :: localhost] Rails Error: Unable to access log file. Please ensure that /home/kochiku-web/kochiku/releases/20130908071331/log/production.log exists and is chmod 0666. The log level has been raised to WARN and the output directed to STDERR until the problem is fixed.
*** [err :: localhost] rake aborted!
*** [err :: localhost] No such file or directory - /home/kochiku-web/kochiku/releases/20130908071331/log/bullet.log

why does it not create the log directory itself?

Merge on Success uses Kochiku Robot as the author

Merge on Success uses Kochiku Robot as the author for the merge commit, it should use the author from the commits that it is merging in.
Since it shows that the author of the merge is Kochiku Robot, a lot of git tools like git blame and IntelliJ's Annotate Lines become much less useful when your team members are using Merge on Success:
screen shot 2017-10-02 at 2 52 57 pm
This should be pretty simple to solve, git authors can be set to whatever you like with the flag --author="John Doe <[email protected]>" in most git commands.

log_file_globs setting inside of targets does not work

The implementation of individual log_file_globs for each target inside of a kochiku.yml does not currently work. The last log file glob specified will always win and overwrite all of the others.

Example:

test_command: 'script/ci'
targets:
  - type: unit
    glob: spec/unit/**/*_spec.rb
    log_file_globs:
      - log/test.log
  - type: integration
    glob: spec/integration/**/*_spec.rb
    log_file_globs:
      - log/integration.log

Here the unit target will receive a log_file_globs value of "log/integration.log" because that is the last one specified.

This will be difficult to fix so I propose removing the log_file_globs key from the targets section. Users can specify all of their log files in the global key instead.

Support reversing an abort action

Currently click the "Abort" button on a build it goes into the aborted state permanently. Kochiku should allow the user to reverse this action if they wish by clicking a "Reactivate build" button. Once the build is active again they can enqueue build parts.

Issues with public github URL's

I think they changed the url format from api.github.com/api/v3 to just api.github.com. There needs to be a change from enterprise to regular github. It worked when I hacked in the url locally. I'll try and do a proper PR in a bit.

cap deploy failing with a Gemfile.lock required error

I forced the deploy to use a release dir of "bear" and created the log directory just so I could get past the previous blocking point, but now I get this:

  • 2013-09-09 08:48:26 executing `deploy:cold'
  • 2013-09-09 08:48:26 executing `deploy:update'
    ** transaction: start
  • 2013-09-09 08:48:26 executing `deploy:update_code'
    updating the cached checkout on all servers
    executing locally: "git ls-remote https://github.com/square/kochiku.git master"
    command finished in 419ms
  • executing "if [ -d /home/kochiku-web/kochiku/shared/cached-copy ]; then cd /home/kochiku-web/kochiku/shared/cached-copy && git fetch -q origin && git fetch --tags -q origin && git reset -q --hard d7c117f && git clean -q -d -x -f; else git clone -q -b master https://github.com/square/kochiku.git /home/kochiku-web/kochiku/shared/cached-copy && cd /home/kochiku-web/kochiku/shared/cached-copy && git checkout -q -b deploy d7c117f; fi"
    servers: ["localhost"]
    [localhost] executing command
    command finished in 1666ms
    copying the cached version to /home/kochiku-web/kochiku/releases/bear
  • executing "cp -RPp /home/kochiku-web/kochiku/shared/cached-copy /home/kochiku-web/kochiku/releases/bear && (echo d7c117f > /home/kochiku-web/kochiku/releases/bear/REVISION)"
    servers: ["localhost"]
    [localhost] executing command
    command finished in 49ms
  • 2013-09-09 08:48:29 executing deploy:finalize_update' triggering before callbacks fordeploy:finalize_update'
  • 2013-09-09 08:48:29 executing `deploy:assets:symlink'
  • executing "ls -x /home/kochiku-web/kochiku/releases"
    servers: ["localhost"]
    [localhost] executing command
    command finished in 18ms
  • executing "rm -rf /home/kochiku-web/kochiku/releases/bear/public/assets && mkdir -p /home/kochiku-web/kochiku/releases/bear/public && mkdir -p /home/kochiku-web/kochiku/shared/assets && ln -s /home/kochiku-web/kochiku/shared/assets /home/kochiku-web/kochiku/releases/bear/public/assets"
    servers: ["localhost"]
    [localhost] executing command
    command finished in 24ms
  • 2013-09-09 08:48:29 executing `bundle:install'
  • executing "cd /home/kochiku-web/kochiku/releases/bear && bundle install --gemfile /home/kochiku-web/kochiku/releases/bear/Gemfile --path /home/kochiku-web/kochiku/shared/bundle --deployment --quiet --without development test"
    servers: ["localhost"]
    [localhost] executing command
    ** [out :: localhost] The --deployment flag requires a Gemfile.lock. Please make sure you have checked
    ** [out :: localhost] your Gemfile.lock into version control before deploying.
    command finished in 354ms
    *** [deploy:update_code] rolling back
  • executing "rm -rf /home/kochiku-web/kochiku/releases/bear; true"
    servers: ["localhost"]
    [localhost] executing command
    command finished in 25ms
    failed: "env PATH=/usr/local/bin:$PATH sh -c 'cd /home/kochiku-web/kochiku/releases/bear && bundle install --gemfile /home/kochiku-web/kochiku/releases/bear/Gemfile --path /home/kochiku-web/kochiku/shared/bundle --deployment --quiet --without development test'" on localhost

Automerge and success email should be combined.

Sending a separate email for an auto-merged build in addition to the success email is noisy (especially when you get a third email from the code review tool as well). The success email should be augmented to specify the fact that a build was also merged.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.