balanced / bob Goto Github PK

Bob the Builder - package your software into a deb/rpm/tar, publishes to a repository, and notifies you all by using a simple yaml file

Home Page: https://balancedpayments.com/

Python 99.72% Shell 0.28%

bob's People

Contributors

Stargazers

Watchers

Forkers

ascii1011 jeff-lewis soitun jna

bob's Issues

Exclude directives are not excluded because find is using -name instead of -regex

regex types: http://www.gnu.org/software/findutils/manual/html_node/find_html/Regular-Expressions.html

find /root/work/bob -depth -regex ".*\.pyc" -delete

These aren't https://github.com/balanced/precog/blob/omnibussed/build.yml#L8-L9 globs

Precog build is slow due to compiling Numpy & Scipy

Ideally these resources would be cached somehow so it doesn't have to compile the dependencies every time we want to deploy. Perhaps we could use python wheels so that we aren't wasting so much time building.

Implement hooks

Currently the after_install, before_install hooks are ignored.

Specifiy provides

https://github.com/jordansissel/fpm/wiki

--provides PROVIDES What this package provides (usually a name). This flag can be specified multiple times.

allows us to give hints when a package is not installed.

Bob clobbers itself when it is using itself to build an upgraded debian file.

When bob is used to build bob, there is no going back if it fails to build. This is due to https://github.com/balanced/bob/blob/master/bob/builders/ubuntu/__init__.py#L81-L85

        for directory in (self.tmp_dir, self.target):
            self.run_command('sudo rm -rf {}'.format(directory))
            self.run_command('sudo mkdir -p {}'.format(directory))
            self.run_command('sudo chown {} {}'.format(
                getpass.getuser(), directory)
            )

Bob will blow away /opt/bob when it's trying to build the debian file.

After YAML deserialization, map the payload to pilo

Use @bninja/pilo

warp with a webserver that can handle travis hooks

http://docs.travis-ci.com/user/notifications/

Bob build will not stop when pip install fails

Looks like even the pip install fails, the build continues, please see

No distributions at all found for balog[sqs]==1.0.1 (fro m justitia==0.0.0)

in following log records

21:28:38,265 INFO  [bob.builders] [13439] [MainThread] [0.4.5] [guru_id=] Downloading/unpacking balog[sqs]==1.0.1 (from justitia==
0.0.0)
21:28:38,265 INFO  [bob.builders] [13439] [MainThread] [0.4.5] [guru_id=]   Could not find any downloads that satisfy the requirem
ent balog[sqs]==1.0.1 (from justitia==0.0.0)
21:28:38,266 INFO  [bob.builders] [13439] [MainThread] [0.4.5] [guru_id=] Cleaning up...
21:28:38,266 INFO  [bob.builders] [13439] [MainThread] [0.4.5] [guru_id=] No distributions at all found for balog[sqs]==1.0.1 (fro
m justitia==0.0.0)
21:28:38,266 INFO  [bob.builders] [13439] [MainThread] [0.4.5] [guru_id=] Storing debug log for failure in /root/.pip/pip.log
21:28:38,267 DEBUG [bob.builders] [13439] [MainThread] [0.4.5] [guru_id=] [u'\n        fpm -s dir -t deb -n justitia -v 1.0.36 -x 
"*.pyc"             --iteration=1              -d libxml2 -d libxslt1.1 -d libpq5 -d ipython              --after-install=/root/wo
rk/justitia/scripts/after-install.sh --before-remove=/root/work/justitia/scripts/before-remove.sh /opt/justitia\n        ']
21:28:38,267 DEBUG [bob.builders] [13439] [MainThread] [0.4.5] [guru_id=] {}
21:28:39,101 INFO  [bob.builders] [13439] [MainThread] [0.4.5] [guru_id=] {:timestamp=>"2014-10-01T21:28:39.068233+0000", :message
=>"Created package", :path=>"justitia_1.0.36-1_amd64.deb"}
21:28:39,101 DEBUG [bob.builders] [13439] [MainThread] [0.4.5] [guru_id=] [u'\n            depot -s s3://apt.vandelay.io -c precis
e -k 277E7787                 --component=unstable --no-public --force                 justitia_1.0.36-1_amd64.deb\n            ']
21:28:39,102 DEBUG [bob.builders] [13439] [MainThread] [0.4.5] [guru_id=] {'env': {u'AWS_ACCESS_KEY_ID': 'xxxxxxxxxxxxxxxxxxxxxxx', u
'AWS_SECRET_ACCESS_KEY': 'xxxxxxxxxxxxxxxxxxxxxxx'}}
21:29:02,475 INFO  [bob.builders] [13439] [MainThread] [0.4.5] [guru_id=] Uploading package justitia_1.0.36-1_amd64.deb
21:29:02,476 INFO  [bob.builders] [13439] [MainThread] [0.4.5] [guru_id=] Uploading metadata
21:29:03,243 INFO  [bob.builders] [13439] [MainThread] [0.4.5] [guru_id=] built justitia version v1.0.36 and uploaded to unstable
21:29:03,243 INFO  [bob.builders] [13439] [MainThread] [0.4.5] [guru_id=] ***finished***

Use Fabric for some of these commands

https://github.com/balanced/bob/blob/master/bob/builders/__init__.py#L17-L32

Bob needs to learn to clean up after itself

delete those debs and build tmp files after each upload

yaml file needs a version attribute

for future modifications

Investigate parameterised tests

1f64a42#commitcomment-7408122

Our test fixtures look like they will get unwieldily quickly.

How to stream log output to response

Trying to figure out the easiest way to stream log output to the response since building is a slow process.

I'm thinking of

Spin up a process and write data to a log file on disk and then read the file back as it's written.
Spin up a background thread via gevent and run the builder there and pass a logger to it. Kind like https://stackoverflow.com/questions/24892418/background-thread-logging-to-a-server-sent-event-stream-in-a-flask-app

        messages_to_yield = []

        if result['build']:
            builder = forms.background_build(
                result['organization'], result['name'], result['commit'],
                log_stream=logger
            )

        class Handler(logging.Handler):

            def emit(self, record):
                msg = self.format(record)
                messages_to_yield.append(msg)

        logger.addHandler(Handler())

        def iterate_response():
            import time
            builder.start_build()
            while builder.building:
                time.sleep(1)
                for msg in messages_to_yield.pop():
                    yield msg

        return api.Response(app_iter=iterate_response())

Any opinions about the simplest way to do this?

Config

How to setup the configuration? Should be able to to read from --config=/path/to/settings.ini

balanced / bob Goto Github PK

bob's People

Contributors

Stargazers

Watchers

Forkers

bob's Issues

Recommend Projects

Recommend Topics

Recommend Org

Jobs