GithubHelp home page GithubHelp logo

remind101 / conveyor Goto Github PK

View Code? Open in Web Editor NEW
221.0 79.0 24.0 8.88 MB

A fast build system for Docker images.

Home Page: http://engineering.remind.com/introducing-conveyor/

License: BSD 2-Clause "Simplified" License

Makefile 0.88% Shell 0.71% Go 98.21% Ruby 0.21%

conveyor's Issues

Streaming logs

Conveyor should have some UI like tugboat and support streaming the build logs. Could use pgstream to do the streaming.

S3 sucks because the logs aren't treated as IO streams.

Update Docker to More Recent Version

Every once in a while, a build fails with Layer already being pulled by another client. Waiting.. This issue was supposedly fixed on 1.9.

Are there any plans to move towards newer Docker Engine versions? I know Docker's not the most stable, so it'd definitely be a slow process. Let me know if you'd like help with it.

Bitbucket Support

I was just wondering if support for Bitbucket webhooks is something that might be integrated in the future and/or whether people think this would be something useful to develop? Looking at github.go, I think it should be a fairly straightforward addition.

can we try catch the calls to github status change?

i am getting this error, it is just because github has some weird problems with updating status travis-ci/travis-ci#1283

2015/12/30 18:20:11 Starting build: id=7a16a452-2475-49d2-96b3-66d2731d4ce0 repository=cordoval/convey-test branch=master sha=a4371fb913c8366bb2bf95b622b60d15081731c2
reporting err: POST https://api.github.com/repos/cordoval/convey-test/statuses/a4371fb913c8366bb2bf95b622b60d15081731c2: 422 Validation Failed [{Resource:Status Field:target_url Code:custom}]
2015/12/30 18:20:11 POST https://api.github.com/repos/cordoval/convey-test/statuses/a4371fb913c8366b

I wonder if we can wrap the calls with try catch equivalent to avoid this

Document working without SQS (aka memory queue) or fix SQS error

I am getting this error but i don't know how to avoid setting or using SQS

2015/12/30 03:34:44 sqs error: %v AccessDenied: Access to the resource https://sqs.us-east-1.amazonaws.com/ is denied.
    status code: 403, request id: 1e909b68-b47c-56e8-b9ae-d595e7688a02
panic: runtime error: invalid memory address or nil pointer dereference

do you have a simple configuration for avoiding SQS usage?

Shutdown builds when conveyor shuts down.

Docker is gonna SIGKILL the process after 10-30 seconds so the best thing to do is probably kill the builders. Should just use context.Context to do this and maybe set a deadline at the top to timeout long builds from running forever.

error on restarting docker run

This happens when i restart the docker run

ubuntu@ip-10-0-0-86:~$ docker ps -a
CONTAINER ID        IMAGE                       COMMAND              CREATED             STATUS                      PORTS               NAMES
b17ce773be35        remind101/conveyor:master   "/go/bin/conveyor"   24 seconds ago      Exited (2) 22 seconds ago                       conveyor
ubuntu@ip-10-0-0-86:~$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
ubuntu@ip-10-0-0-86:~$ docker logs b17ce773be35
Starting 4 workers
Starting server on 8080
2015/12/28 21:28:54 sqs error: %v InvalidParameter: 1 validation errors:
- missing required parameter: Entries
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x51767c]

goroutine 9 [running]:
github.com/remind101/conveyor.(*SQSBuildQueue).handleError(0xc20803b920, 0x7ff16ccb2030, 0xc20825df80)
    /go/src/github.com/remind101/conveyor/queue.go:186 +0x15c
github.com/remind101/conveyor.func·004()
    /go/src/github.com/remind101/conveyor/queue.go:119 +0x7f
created by github.com/remind101/conveyor.(*SQSBuildQueue).Subscribe
    /go/src/github.com/remind101/conveyor/queue.go:122 +0xe4

goroutine 1 [chan receive]:
main.mainAction(0xc2080b00b0)
    /go/src/github.com/remind101/conveyor/cmd/conveyor/main.go:66 +0x297
github.com/codegangsta/cli.(*App).Run(0xc20805c300, 0xc20800a000, 0x1, 0x1, 0x0, 0x0)
    /go/src/github.com/remind101/conveyor/Godeps/_workspace/src/github.com/codegangsta/cli/app.go:130 +0xbb9
main.main()
    /go/src/github.com/remind101/conveyor/cmd/conveyor/main.go:47 +0x3d6

goroutine 5 [syscall]:
os/signal.loop()
    /usr/src/go/src/os/signal/signal_unix.go:21 +0x1f
created by os/signal.init·1
    /usr/src/go/src/os/signal/signal_unix.go:27 +0x35

goroutine 6 [chan receive]:
github.com/honeybadger-io/honeybadger-go.func·003()
    /go/src/github.com/remind101/conveyor/Godeps/_workspace/src/github.com/honeybadger-io/honeybadger-go/buffered_worker.go:12 +0x86
created by github.com/honeybadger-io/honeybadger-go.newBufferedWorker
    /go/src/github.com/remind101/conveyor/Godeps/_workspace/src/github.com/honeybadger-io/honeybadger-go/buffered_worker.go:25 +0x13d

goroutine 7 [chan receive]:
main.runWorker(0x7ff16ccb2580, 0xc20803b920, 0xc2080b00b0, 0x0, 0x0)
    /go/src/github.com/remind101/conveyor/cmd/conveyor/worker.go:86 +0x40a
main.func·001()
    /go/src/github.com/remind101/conveyor/cmd/conveyor/main.go:59 +0x43
created by main.mainAction
    /go/src/github.com/remind101/conveyor/cmd/conveyor/main.go:60 +0x1cd

goroutine 8 [select]:
main.runServer(0x7ff16ccb2580, 0xc20803b920, 0xc2080b00b0, 0x0, 0x0)
    /go/src/github.com/remind101/conveyor/cmd/conveyor/server.go:60 +0x54d
main.func·002()
    /go/src/github.com/remind101/conveyor/cmd/conveyor/main.go:63 +0x43
created by main.mainAction
    /go/src/github.com/remind101/conveyor/cmd/conveyor/main.go:64 +0x26f

goroutine 11 [IO wait]:
net.(*pollDesc).Wait(0xc2080791e0, 0x72, 0x0, 0x0)
    /usr/src/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc2080791e0, 0x0, 0x0)
    /usr/src/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).accept(0xc208079180, 0x0, 0x7ff16ccb1c58, 0xc20800bd58)
    /usr/src/go/src/net/fd_unix.go:419 +0x40b
net.(*TCPListener).AcceptTCP(0xc208038178, 0x475264, 0x0, 0x0)
    /usr/src/go/src/net/tcpsock_posix.go:234 +0x4e
net/http.tcpKeepAliveListener.Accept(0xc208038178, 0x0, 0x0, 0x0, 0x0)
    /usr/src/go/src/net/http/server.go:1976 +0x4c
net/http.(*Server).Serve(0xc2080428a0, 0x7ff16ccb3b30, 0xc208038178, 0x0, 0x0)
    /usr/src/go/src/net/http/server.go:1728 +0x92
net/http.(*Server).ListenAndServe(0xc2080428a0, 0x0, 0x0)
    /usr/src/go/src/net/http/server.go:1718 +0x154
net/http.ListenAndServe(0xc20800bd50, 0x5, 0x7ff16ccb2a68, 0xc208086420, 0x0, 0x0)
    /usr/src/go/src/net/http/server.go:1808 +0xba
main.func·003()
    /go/src/github.com/remind101/conveyor/cmd/conveyor/server.go:57 +0xb4
created by main.runServer
    /go/src/github.com/remind101/conveyor/cmd/conveyor/server.go:58 +0x445

goroutine 16 [select]:
github.com/remind101/conveyor.(*Worker).Start(0xc2080351c0, 0x0, 0x0)
    /go/src/github.com/remind101/conveyor/worker.go:102 +0x4f3
created by github.com/remind101/conveyor.Workers.Start
    /go/src/github.com/remind101/conveyor/worker.go:18 +0x6c

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
    /usr/src/go/src/runtime/asm_amd64.s:2232 +0x1

goroutine 31 [select]:
net/http.(*persistConn).writeLoop(0xc2080b0630)
    /usr/src/go/src/net/http/transport.go:945 +0x41d
created by net/http.(*Transport).dialConn
    /usr/src/go/src/net/http/transport.go:661 +0xcbc

goroutine 18 [select]:
github.com/remind101/conveyor.(*Worker).Start(0xc208035200, 0x0, 0x0)
    /go/src/github.com/remind101/conveyor/worker.go:102 +0x4f3
created by github.com/remind101/conveyor.Workers.Start
    /go/src/github.com/remind101/conveyor/worker.go:18 +0x6c

goroutine 19 [select]:
github.com/remind101/conveyor.(*Worker).Start(0xc208035240, 0x0, 0x0)
    /go/src/github.com/remind101/conveyor/worker.go:102 +0x4f3
created by github.com/remind101/conveyor.Workers.Start
    /go/src/github.com/remind101/conveyor/worker.go:18 +0x6c

goroutine 20 [select]:
github.com/remind101/conveyor.(*Worker).Start(0xc208035280, 0x0, 0x0)
    /go/src/github.com/remind101/conveyor/worker.go:102 +0x4f3
created by github.com/remind101/conveyor.Workers.Start
    /go/src/github.com/remind101/conveyor/worker.go:18 +0x6c

goroutine 30 [runnable]:
net/http.(*persistConn).readLoop(0xc2080b0630)
    /usr/src/go/src/net/http/transport.go:928 +0x9ce
created by net/http.(*Transport).dialConn
    /usr/src/go/src/net/http/transport.go:660 +0xc9f

Successful/Failed build notifications

It would be a really nice feature to be able to add a web hook to notify something like a slack channel that a build succeeded or failed. Right now I'm just checking s3 for the new log files manually. If there is already an easier way I'd love to hear it.

I tried to start working on this locally but I can't get everything to play nicely together. Are the docs on getting this running locally up to date? If I can get this running I will start working on it.

Thanks for the great work on this and empire btw.

AMI no longer exists

Parameter validation failed: parameter value ami-2d097847 for parameter name AmiId does not exist. Rollback requested by user.

Use captain to build

Using captain in the default builder image would be nice for a couple of reasons:

  1. It supports a captain.yml file to build multiple Dockerfiles.
  2. captain could be used locally to build as well.

Only thing that prevents this atm is that captain tags the docker image with a short sha instead of the full sha.

Improve readme installation instructions

I'm trying to get Conveyor running for the first time but I'm running into problems with lack of documentation :S

In particular I'm not quite sure how to get converyor running on a swarm, there is mention that this can be done in the readme but I haven't figured out how to do it yet.

I realise that the way you guys do it is using cloudformation but I'm looking for a way to avoid cloudformation and use Conveyor on a simple docker host, is this possible? Could you improve the readme with some more detailed instructions/guidance for installation?

Thanks!

Use Alpine-based base image in Dockerfile

Using golang:${VERSION}-alpine (ex: golang:1.6-alpine) cuts the size of the final image by at least 50%.

Also, it would be great if you could use CMD instead of ENTRYPOINT (but that's a minor annoyance).

using obsolete docker API (removed in docker 1.12.*)

conveyor uses a few old API calls, for example:

conveyor_1 | 2016/10/22 22:37:12 Starting build: id=d4841527-eea1-4bbd-a785-455867c1482e repository=remind101/acme-inc branch=master
sha=827fecd2d36ebeaa2fd05aa8ef3eed1e56a8cd57
conveyor_1 | reporting err: start container: API error (400): {"message":"starting container with HostConfig was deprecated since v1.
10 and removed in v1.12"}
conveyor_1 |

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.