GithubHelp home page GithubHelp logo

mapbox / stork Goto Github PK

View Code? Open in Web Editor NEW
10.0 102.0 4.0 289 KB

This project has been deprecated and is no longer maintained.

Shell 1.64% Logos 0.39% Roff 1.02% JavaScript 96.94%
testing-1 manual testing-2 testing-3 artifact-management

stork's Introduction

⚠️ Deprecation Notice ⚠️

Stork has been deprecated for use internally at Mapbox and will no longer be maintained.

stork Build Status

About

Stork is a continuous integration system that runs on AWS CodeBuild. Its primary usage is to build .zip bundles for use in Lambda functions each time a commit is pushed to a Github repository. It can also be used as a more generic tool for running a CodeBuild project on each commit.

Usage

These instructions spell out how to use stork bundles from the application developer's perspective. They depend on there already being a stork stack running in your AWS account. To learn about how to bootstrap a stork stack, please see these docs instead.

Ask stork to watch your repository

There are several pieces of information that need to be collected prior to configuring a repository to be watched by stork. At Mapbox, we've written an internal CLI tool that knows these values for our account, and configures the repository for our production stork stack. If you're a Mapbox employee, please refer to mbxcli documentation.

Stork provides a Node.js function and a CLI tool that can be used to set this up if you've gathered the following information:

  • The region that your account's stork stack runs in
  • The suffix of your account's stork stack, as in the stork-${suffix} stack.
  • Your personal access token which will be used to make github requests to configure the repository for stork to watch.
  • The org name of the repository you wish stork to watch
  • The repository name for stork to watch

The Github token provided here must have the following scopes:

  • user: for adding the repository to your stork github app
  • repo: for reading repository data
  • admin:repo_hook: for adding the webhook to the repository

The token will only be used once to set up webhooks, and after that you can delete the token if you wish.

With those information in hand, you can chose to configure stork to watch your repository using either a CLI command or by writing a Node.js script.

The following example connects my-repo owned by mapbox to a production stork stack in us-east-1:

via CLI

$ ./bin/hook.js \
>   --regions us-east-1 \
>   --suffix production \
>   --org mapbox \
>   --repo my-repo \
>   --token xxx

via Node.js

const hook = require('stork').setupHook;

const options = {
  region: 'us-east-1',
  suffix: 'production',
  token: 'xxx',
  org: 'mapbox',
  repo: 'my-repo'
};

hook(options)
  .then(() => console.log('Linked to webhooks in us-east-1'));

Using the Lambda bundles stork created

A running stork stack is configured to write .zip files to a specific S3 bucket and prefix. For example, if the stork stack writes to my-bucket and my-bundles, and you were to make a commit to my-repo with a SHA of abc, then the bundle will be located at:

s3://my-bucket/my-bundles/my-repo/abc.zip

Each time you push a commit to my-repo, another .zip file will be written with the commit's SHA. This predictable naming scheme helps you manage Lambda functions defined in CloudFormation templates, where the Lambda function code might change from commit to commit.

Stork's default environment

By default, stork will try and bundle your application assuming it will be running on a Lambda function on the nodejs6.10 runtime. If this is true, then no further configuration on your part is required.

Note: Stork uses npm v5.3.0 to install your libraries' --production dependencies. If your repository includes a package-lock.json file, it will be respected during bundling.

Building a bundle for a different Lambda runtime

There are four base images provided by stork. These correspond directly to Lambda runtime environments and are meant to reproduce that runtime environment.

  • nodejs6.x (default)
  • nodejs4.3
  • nodejs8.10
  • python2.7
  • python3.6

For each of these environments, see the related Dockerfile in the Dockerfiles folder for the specifics of the build environment. Furthermore, the respective .yml files in the buildspecs folder define the steps that will be taken to bundle your repository.

If you wish to build bundles for a runtime other than nodejs6.10, follow these steps:

  1. Include a .stork.json file at the top-level of your repository
  2. The file should be a JSON object, and the image property should indicate which of the stork images to use.

Here's an example .stork.json file to bundle an application intended for the python2.7 Lambda runtime:

{
  "image": "python2.7"
}

Deeper customization of the stork build process

You can further override stork to run a customized CodeBuild project on each commit to your repository. This may mean taking additional steps prior to building a Lambda bundle, or you could use stork as a trigger for an entirely different build process using AWS CodeBuild. You can override stork's default procedures by adding either of these files to your repository:

  • .stork.json: Allows you to choose from a set of images that stork provides, or set the build to use a custom Docker image. Note that the image you choose must be either one of stork's defaults (see above), or any public image.
  • buildspec.yml: Allows you to determine exactly the build steps performed by CodeBuild after pulling your code. If you put this file in your repository, you will completely override stork's default bundling procedure. You are responsible for making sure that this file uploads any artifacts from the build to S3.

.stork.json

This file has the following structure:

{
  "image": "a stork default image name or the full URI for a public image",
  "size": "one of small, medium, or large"
}

See the AWS CodeBuild documentation for details about the differences between the small, medium, and large build environments.

buildspec.yml

This file, if provided, determines what actions will be taken during a CodeBuild run on each commit. By defining this file in your repository, you take complete control over the CodeBuild actions, and can use it to take whatever build actions you'd like to.

See the AWS CodeBuild documentation to understand this file's syntax and capabilities.

stork's People

Contributors

agius avatar dputtick avatar emilymdubois avatar jseppi avatar kellyoung avatar manaswinidas avatar ramshackle-jamathon avatar rclark avatar springmeyer avatar vincentsarago avatar vsmart avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

stork's Issues

Engineering standards inventory

Required Elements

If any elements in the below list are not checked, this repo will fail standards compliance.

  • Not running node 4 or below
  • Has at least some test coverage?
  • Has a README?
  • Has no hard-coded critical secrets like API keys?

Rubric

  • 1 pt Is in Version Control/Github ✅ (free points)
  • 1-2 pt node version:
    • 2 pt Best: running node 8+ 🏅
    • 1 pt Questionable: node 6
    • 0 pt Not ok: running node4 or below ⛔️
  • 1 pt No hard-coded config parameters?
  • 1 pt No special branches that need to be deployed?
    • Unsure of context on branch g++
  • 1 pt All production stacks on latest master?
  • 1 pt No hard-coded secrets like API keys?
  • 1 pt No secrets in CloudFormation templates that don’t use [secure]?
  • 1 pt CI enabled for repo?
  • 1 pt Not running Circle CI version 1? (Point awarded if using Travis)
  • 1 pt nyc integrated to show test coverage summary?
  • 1-3 pt test coverage percentage from nyc?
    • 3 pt High coverage: > 90%
    • 2 pt Moderate coverage: between 75 and 90% total coverage
    • 1 pt 0 - 74% test coverage
  • 1-2 pt evidence of bug fixes/edge cases being tested?
    • 2 pt Strong evidence/several instances noted
    • 1 pt Some evidence
  • 1 pt no flags to enable different functionality in non-test environments?
  • 1 pt Has README?
  • 1-2 pt README explains purpose of a project and how it works to some detail?
    • 2 pt High (but appropriate) amount of detail about the project
    • 1 pt Some detail about the project documented, could be more extensive
  • 1 pt README contains dev install instructions?
  • 1 pt README contains CI badges, as appropriate?
  • 1-2 pt Code seems self-documenting: file/module names, function names, variables? No redundant comments to explain naming conventions?
    • 2 pt Strongly self-documented code, little to no improvements needed
    • 1 pt Some evidence of self-documenting code
  • 1 pt No extraneous permissions in IAM roles?
  • 1 pt Stack has alarms for AWS resources used routed to PagerDuty? (CPU utilization, Lambda failures, etc.)
  • 1 pt Stack has other appropriate alarms routed to PagerDuty? (Point awarded if no other alarms needed)
  • 1 pt Alarms documented?
    • In progress: #29
  • master branch protected?
    • 1 pt PRs can only be merged if tests are passing?
    • 1 pt PRs must be approved before merging?
  • 2 pt BONUS: was this repo covered in a deep dive at some point?

Total possible: 30 points (+2 bonus)
Grading scale:

Point Total Qualitative Description Scaled Grade
28+ points Strongly adheres to eng. standards 5
23-27 points Adheres to eng. standards fairly well 4
18-22 points Adheres to some eng. standards 3
13-17 points Starting to adhere to some eng. standards 2
9-12 points Following a limited number of eng. standard practices 1
< 9 points Needs significant work, does not follow most standards 0

Repo grade: 25 - Scaled Grade 4

/cc @mapbox/assembly-line

Stork fails post on missing gitsha's

Ran into a stork alarm today with some stork errors regarding an "unprocessable entity"

sample:

{ HTTPError: Response code 422 (Unprocessable Entity)
    at stream.catch.then.e (/var/task/node_modules/got/index.js:123:13)
    at process._tickDomainCallback (internal/process/next_tick.js:135:7)
  message: 'Response code 422 (Unprocessable Entity)',
  host: 'api.github.com',
  hostname: 'api.github.com',
  method: 'POST',
  path: <repo path>,
  statusCode: 422,
  statusMessage: 'Unprocessable Entity' }

@rclark determined that the end of that path is a sha, and some quick investigating it became clear that sha was not present on github.

It looks like stork will error when a sha is force pushed or deleted.

cc: @rclark

edited: removed repo path (@kellyoung)

trigger function errors when .yml is named incorrectly

Today, we got an alarm because the trigger function error'ed when an incorrectly named .yml file was used.

2017-11-02T18:36:28.441Z    bc7ecdd9-bffc-11e7-aba8-f192b47c1978
{
    "errorMessage": "ENOENT: no such file or directory, open '/var/task/buildspecs/node4.3.yml'",
    "errorType": "Error",
    "stackTrace": [
        "Error (native)",
        "Object.fs.openSync (fs.js:641:18)",
        "Object.fs.readFileSync (fs.js:509:33)",
        "getDefaultBuildspec (/var/task/lambda.js:318:13)",
        "githubToken.then.then.then.then (/var/task/lambda.js:396:33)",
        "process._tickDomainCallback (internal/process/next_tick.js:135:7)"
    ]
}

We should modify the trigger function so that it handles errors such as spelling errors in a different way.

cc/ @mapbox/platform

Single stack output to multiple regions (accounts?)

This is set up right now such that you could run a bundle-shepherd stack in each AWS region you want to write bundles to, and then subscribe your repository to all of the webhook urls.

It would be better if one stack could write to several locations. This can't be done with the out-of-the-box codebuild artifacts settings. It has to be done via custom code.

App-developer-centric readme

  • I want to hook up my node.js repo for lambda bundles
  • I want to hook up my python2.7 repo for lambda bundles
  • Where are my bundles?
  • I want to hook up a custom codebuild continuous integration service

add Github token to codebuild project env variables

Right now we only pass the NPM_ACCESS_TOKEN variable in the codebuild project to be able to install nodejs package from public and private repo.

stork/lambda.js

Line 124 in b6d8990

{ name: 'NPM_ACCESS_TOKEN', value: options.npmToken }

When it comes to Python sadly there is not such private and public packages, to overcome this we can use private GitHub repo (pip3 install git+https://$GITHUB_ACCESS_TOKEN:[email protected]/blaabla/blabla.git), but to be able to do so within the codebuild project we need to have access to the GITHUB_ACCESS_TOKEN.

@rclark Do you think we could add this variable into the project ?

Support targeting repo directories

Does Stork allow for targeting subdirectories with their own package.json for zipping?

Use Case

I have a repo that is using step functions to create and monitor the progress of ecs-watchbot jobs. the repo has a single package.json which includes the dependencies for both the lambda step functions and the ecs-watchbot worker. Unfortunately the ecs-watchbot worker has dependencies that have build steps that depend on system libs that are not on the stork images.

Current work arounds

Using a custom stork image or moving the install of culprit dependencies into the ecs-watchbot Dockerfile. I would love to be able to have stork use an image representative of the lambda runtime, keep all my dependencies in a package.json and not have stork zip any dependencies not used by my lambda functions.

Proposed change

If I could keep all of my lambda functions in their own directory with its own package.json and have stork target that directory I could avoid having to do any workaround nasty-jams and keep any ecs-watchbot only node_modules out of the lambda .zips

add git as a dependency for node bundling

According to this documentation, git URLs are valid npm dependencies in a package.json file: https://docs.npmjs.com/files/package.json#git-urls-as-dependencies

However, stork errors with this message:

[Container] 2017/12/21 15:16:02 Running command npm install --production
npm ERR! code ENOGIT
npm ERR! No git binary found in $PATH
npm ERR! Failed using git.
npm ERR! Please check if you have git installed and in your PATH.

Would it make sense to include git as a dependency in the node dockerfiles? Or should I use a custom buildspec.yaml in that situation?

Trigger function error parsing github API response

A very rare occurrence, from the trigger function's logs:

2017-10-05T20:35:40.161Z	bfa880b4-aa0c-11e7-b661-3b257937c083
{
    "errorMessage": "Unexpected token < in JSON at position 0 in \"https://api.github.com/app\": \n<!DOCTYPE html>\n<html>\n  <head>\n    <meta http-equiv=\"Content-type\" content=\"...",
    "errorType": "ParseError",
    "stackTrace": [
        "<!DOCTYPE html>",
        "<html>",
        "  <head>",
        "    <meta http-equiv=\"Content-type\" content=\"...",
        "stream.catch.then.e (/var/task/node_modules/got/index.js:118:14)",
        "process._tickDomainCallback (internal/process/next_tick.js:135:7)"
    ]
}

The error occurred somewhere in this bit of code, which involves getting a github token, and then checking for buildspec.yml and/or .stork.json in the repository.

Next steps

  • why does this stack trace suck so hard?
  • where can we catch this parse error so we can either surface the github api error or retry the request?

cc @kellyoung

Stork can 404 on losing access to a repo

We saw 404's on stork today when:

  • there had been a recent commit to to-fix-internal
  • @rclark removed stork from to-fix-internal shortly after the commit

End result was

2017-10-30T20:40:44.102Z    99a31736-bdb2-11e7-8ec9-fd593e309025    { HTTPError: Response code 404 (Not Found)
at stream.catch.then.e (/var/task/node_modules/got/index.js:123:13)
at process._tickDomainCallback (internal/process/next_tick.js:135:7)
message: 'Response code 404 (Not Found)',
host: 'api.github.com',
hostname: 'api.github.com',
method: 'POST',
path: '/repos/mapbox/to-fix-internal/statuses/0eb154f84c9ed67ae8892dd754516cb69689ace2',
statusCode: 404,
statusMessage: 'Not Found' }

That gitsha exists, but stork couldn't see it any longer because it no longer had access to to-fix-internal

The errors subsided on its own after the initial trouble; posting this ticket to document another github api edge case where stork can run into trouble.

cc: @rclark

Java support

A while ago we built a demo that runs an Alexa Skill on a Lambda, this was built with Java because we used Mapbox Java Services (https://github.com/mapbox/mapbox-java). The code for the Lambda can be found on https://github.com/mapbox/mapbox-assistant-example.

Right now, we're manually building the .zip file every time there's a new release (make build in the root folder) which is far from ideal. We'd love to use Stork instead.

By looking at the current documentation, looks like only Node and Python are supported, so I'd like to explore the option of supporting Java as well (if this is possible). From the docs, looks like I should start by contributing a new Dockerfile that enables Java. As I'm not super-familiar with Stork, is there anything else that I should do? Is this possible at all?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.