deis / workflow-manager-api

Deis Workflow Manager API

	Deis Workflow is no longer maintained. Please read the announcement for more detail.
09/07/2017	Deis Workflow [v2.18][] final release before entering maintenance mode
03/01/2018	End of Workflow maintenance: critical patches no longer merged
	Hephy is a fork of Workflow that is actively developed and accepts code contributions.

Deis Workflow Manager Service API

The Workflow Manager Service API is responsible for interfacing with Deis Workflow data. It is a golang https implementation that speaks JSON. The API is the source of authority for the following:

• metadata about unique Deis clusters that have checked in: which components are installed at the time of check-in, and which versions
• released version history for each Deis cluster component

Additionally, the API is the official interface for accepting Workflow Manager data, e.g.:

• Deis cluster component version CRUD operations
  • i.e., centrally store latest stable version information for all Deis cluster components
• Deis unique cluster anonymous registration
  • i.e., receive Deis Workflow Manager client usage statistics

Usage

To download dependencies:

$ make bootstrap

To test:

$ make test

Note: if you prefer to run tests with go test, you'll need glide on your PATH. Once you have it, run go test -tags testonly $(glide nv) to run tests. To build:

$ IMAGE_PREFIX=$MY_DOCKERHUB_ACCOUNT make build docker-build docker-push

(All of the above operations assume a local Docker environment.)

Status

A working, minimal API is currently live at https://versions.deis.com.

workflow-manager-api's Issues

Audit timezone implementation

We expect all timestamps to be represented in PDT. Ensure that this is so.

remove timestamp properties from `clusters` db table

We have all the time-based data we need in the clusters_checkins table.

consider replacing literal SQL statement string concatenation w/ prepared statements

Versions data schema redesign

In light of recent design discussions, we want to capture the following information in component version records:

• The version "train" (e.g., "production" or "beta")
• The particular release date associated with the component + train + version string

We will "probably" adhere to semver for versions once we cut the production "2.0.0" release, but we aren't currently doing that, and so we need a strategy in the absence of a clear semantic version implementation to determine which in a set of version records is the most recent; a timestamp will work for that.

The current version table looks like this:

"CREATE TABLE IF NOT EXISTS versions ( component_name varchar(64) PRIMARY KEY, last_updated timestamp, data json )"

I'm proposing we change this to the following:

"CREATE TABLE IF NOT EXISTS versions ( release_id bigserial PRIMARY KEY, release_time timestamp, component_name varchar(64), release_train varchar(16), release_version varchar(32), data json )"

In the example above we'll keep the generic json data field to enable general purpose docstore-type metadata. But we want to keep all the data we expect to query as fields in the schema for optimization.

wrap HTTP handlers in http.HandlerFunc

Proposal: remove "high level" data functions

Functions like this one:

func GetVersion(componentVersion types.ComponentVersion, db *sql.DB, v Version) (types.ComponentVersion, error) {

Are pure pass-through calls to the underlying implementation (in this case, Version). Unless their scope or functionality are going to be increased now, they should be removed and the underlying calls should just be made.

This is an important pre-requisite for #48 because it simplifies the DB access to a single set of functions in the data package, separating them from the structure of the DB (i.e. the schemas). Doing this refactor will enable a much finer-grained transition toward using Gorm for DB access, which will lower the complexity and size of each Gorm-related PR

define the API with swagger

instead of, or in addition to having the api-architecture.md document, we should have a swagger spec to define the API so that we can predictably change the API and publish documentation if we need to.

Implementation details:

• go-swagger is a useful tool for generating Go code from Swagger 2.0 specifications, but other tools can be used instead. The only requirement is that the tool must be easy to install and easy for developers to work with
  • For example, the go-swagger tool is written in Go, so it's a self contained binary with good documentation
• The swagger specification should mirror the endpoints that are currently in doc/api-architecure.md
• The swagger specification should live in a swagger directory under the deis/workflow-manager repository
• The Makefile in this repository must have a build target to generate server "stub" code
• The Makefile in deis/workflow-manager must have a build target to generate client code
• The implementation in this server should be moved into the server stubs (#138)

Also, the items in the above bullet points should be split up into smaller, logical units of work and submitted as small, easy to review & test PRs.

Design and implement a "cluster age" interface

• get a cluster count of clusters that are n days stale (clusters that haven't checked in for n days)
• get a cluster count of clusters that are n days fresh (cluster that have been active for n days)
• clusters that have been created within a range of n days

depends on #42 and #43

Decide on URL path convention for selecting component, train and version

For example, should we do .../{train}/{component}/{version} or .../{component}/{train}/{version}?

rationalize error responses: return nil instead of a zero value type slice literal

suggestion: enable simple "cluster age" queries

Our current API accepts "cluster age" filter criteria as follows:

• created before a certain date
• created after a certain date
• checked in before a certain date
• checked in after a certain date

From a convenience standpoint, it would be nice to enable just one, or two, or three of those from the API consumer (more closely matching a human semantic expression), e.g., "clusters that have checked in since last week". This will require either overloading the existing "cluster age" API interface to allow the ommission of certain filter inputs, or to create novel API interfaces for simpler requests.

Validate that common data parameter values between URL dynamic route segments and POST data are equal

/healthz endpoint

makes sense to add a /healthz endpoint that runs a ping + dumps out db.Stats so we can keep an eye on it

Implement endpoint to multi-get a collection of latest releases

POST /:apiVersion/versions/latest

Put the operations in SetVersion into a txn

there's a read-modify-write in there, so it should be in a txn.

also, since the SetVersion function is really a set-or-update, maybe it should be called "Upsert" or something similar.

clusters_checkins table is incorrectly schematized

we don't need uniqueness constraints in this table

versions.deis.com CI updates story

As a versions.deis.com user I expect current version information matching current builds to be present in the data that the API represents in order to get correct, up-to-date version info for all deis components.

We should include in the standard CI build processes a POST to versions.deis.com of the CI-built component version.

CA-stamped TLS cert

It's important that deis client connect to a verified, trusted API. We're currently assuming versions.deis.com as the FQDN that the API will respond to, but until we have a cert that's subject to comment.

Proposal: use an ORM for DB interaction

Given that the scope of database usage is growing fast, I think it'd be a good idea to look at gorm as an interface to our database. It includes a lot of high level features that I think we could already make use of.

This issue is only for running data queries on the DB, not for initializing it.

update README.md to reflect DB changes

Proposal: remove the 'DB' interface

An audit of the codebase shows that DB is a factory for *sql.DB types, but its Get func is only called on server startup, suggesting we don't need a generic way to create multiple *sql.DB instances. If that's the case, can we remove this interface and just pass around *sql.DB instances (we do this already in many places, in fact)

Write end to end tests to check impossible filter errors

#65 introduces cluster age filtering by checkin and created time, and that introduces the possibility for invalid filters. we have unit tests for invalid filters at the data layer, but it would be nice to add tests at the query string (end-to-end) layer.

grab explicit db instance, not just the first instance in the array from the endpoint connection

convert http handlers to use json encoders

instead of json.Marshaling into a string, then writing the string to the http.ResponseWriter, json.NewEncoder(w).Encode(...) reduces allocations

Audit unit test coverage

As of this writing, coverage is just over 50%. We should intelligently look for ways to increase it, then write the tests to do so.

Implement endpoint to get latest component & train release

GET /:apiVersion/versions/:train/:component/latest

This endpoint could be solved with a query to get latest timestamp or similar

apply new versions db schema

update cluster mysql convenience function has bad SQL

test leakage 👎

multi-get implementation

For a given deis cluster, we're going to assume a particular collection of components, each with their own release train, will represent the components+versions "set" of data that is requested. Rather than require each component+train+version to be a single API request, we should accommodate a multi-get request to fulfill the set criteria.

Auto-deploy all merges to master to staging

Create swagger spec for the workflow-manager-api endpoints

This can become a formally specified version of the markdown written in #29

investigate dropping the dependency to workflow-manager/components

github.com/deis/workflow-manager/components depends upon k8s.io packages

Split data.go into separate files

Proposed split:

• Put each interface into its own file
• Put each interface implementation into its own file, where the filename is similar to its associated interface file, but indicates what kind of implementation it is. For example, version_from_db.go
• Group DB initialization functions into a single file
• Group other utility functions into one or more other files

Such a split will make the codebase easier to follow, lower cognitive load and reduce the impact of conflicts.

Split handlers into separate files

One handler per file makes the codebase easier to follow, lowers cognitive load and reduces the impact of conflicts.

refactor "component" + "train" dynamic route segments

We want the hierarchical order to reflect component --> train --> version on all API route endpoints. This refactor will result in the following changes (as of this writing):

• GET /{apiVersion}/versions/{train}/{component} becomes GET /{apiVersion}/versions/{component}/{train}
• GET /{apiVersion}/versions/{train}/{component}/latest becomes GET /{apiVersion}/versions/{component}/{train}/latest
• GET /{apiVersion}/versions/{train}/{component}/{version} becomes GET /{apiVersion}/versions/{component}/{train}/{version}
• POST /{apiVersion}/versions/{train}/{component}/{version} becomes POST /{apiVersion}/versions/{component}/{train}/{version}

Use the MAX function in the (data/Versions).Latest func

Currently, we're sorting by timestamp and getting the first row. It's cleaner SQL to use the MAX function. See http://www.w3schools.com/sql/sql_func_max.asp

Unit tests

Codebase was recently decomposed and existing unit tests should be considered obsolete.

Split server_test.go into separate files

One or a few related tests per file makes the codebase easier to follow, lowers cognitive load and reduces the impact of conflicts

workflow-manager cluster checkins intermittently happening really frequently

see these cluster_id values:

• dfa0c19a-2eba-4631-b92f-d884c50fe04e
• 8b88f34c-d4ca-4344-92f0-273891e6b9a0

Add functional tests

These tests should run the data layer against a local Postgres instance, to test each query against a closer representation of the database that this codebase will use in production

Reformat api architecture docs to put each endpoint under a header

doing so will allow us to reference each endpoint via a permanent link

cluster age filters fail in real world

It seems our sqlite tests didn't cover this. Here's an example URL against the prod API:

https://versions.deis.com/v2/clusters/age?created_before=2016-05-02T00%3A00%3A00Z&created_after=2016-05-01T00%3A00%3A00Z&checked_in_before=2016-05-02T00%3A00%3A00Z&checked_in_after=2016-05-01T00%3A00%3A00Z

Response is a 500 with "database error" in the document body.

App error log looks like this:

2016/05/02 23:10:27 Error filtering clusters by age (pq: could not identify an equality operator for type json)

implement "firstSeen" and "lastSeen" cluster properties

There are really two issues:

• we're not doing anything useful with the existing "firstSeen" and "lastSeen" properties (the properties are inheriting their zero value, documented here: https://golang.org/pkg/time/#Time)
• we're polluting the data properties in the cluster db record

A proposal: get rid of these two properties in the Cluster types we're using to submit to the database (or use another type definition); and then add these metadata in a meaningful way using a different type struct.

Paginate the cluster age filtering API

The new endpoint in #65 can return a lot of results, so we should paginate them

document persistent data schema

A quick write-up of what data these API endpoints service would be nice.

(auth) validate deis cluster requests

This is a high-level issue to address the following concern:

• How do we verify that a write-request (HTTP POST) originates from a real deis cluster?

We don't want to overly complicate things and, say, store user credentials. But we would like, longer term, to disallow random folks from reverse engineering our elegantly designed API and sending us bogus data.

Add more tests for the cluster age filter DB query

After #65 is done, the DB-level tests run through multiple filters but add a single cluster to the database on each run. Then, when the cluster is added, each run checks for its existence in the filter query's result or not.

This test should improve to add multiple clusters to the database on each iteration. Some of the clusters should be in the filter and some shouldn't, and each run should verify that the correct clusters were returned.

Check error types in the cluster age filter tests

After #65, the cluster age filter represents important business logic, and invalid queries are possible. This issue is for making tests check the error messages from the cluster age filter creation and validation functions, since those messages will be returned to the client.

rationalize json marshaling error responses

Standardize the way we respond to json.Marshal errors. It seems sometimes we're just printing out a simple message, and sometimes we're returning the error to the caller. Let's be consistent.

Hook up to CI

Since this is a private repo, this may require a paid travis account (do we have that?)

I believe we need this before beta.