zalando-stups / kio Goto Github PK

Variable	Default	Description
HTTP_PORT	`8080`	TCP port to provide the HTTP API.
HTTP_CORS_ORIGIN		Domain for cross-origin JavaScript requests. If set, the Access-Control headers will be set.
DB_SUBNAME	`//localhost:5432/kio`	JDBC connection information of your database.
DB_USER	`postgres`	Database user.
DB_PASSWORD	`postgres`	Database password.

Example:

$ docker run -it \
    -e HTTP_CORS_ORIGIN="*.zalando.de" \
    -e DB_USER=kio \
    -e DB_PASSWORD=kio123 \
    stups/kio

Building

$ lein uberjar
$ lein docker build

Releasing

$ lein release :minor

Developing

Kio embeds the reloaded workflow for interactive development:

$ lein repl
user=> (go)
user=> (reset)

License

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

kio's People

Contributors

Stargazers

Watchers

Forkers

ashuchopra mawenzel taojang rafaelcaricio bogdanaioanei maxim-tschumak ethercrow oporkka amosciatti mikkeloscar scherniavsky

kio's Issues

Add "is_public_endpoint" information to apps

It should be possible to mark applications as public endpoints.

This will skip the OAuth check in fullstop

org.postgresql.util.PSQLException: ERROR: syntax error in tsquery: "team:foobar"

It seems that colons inside search queries produce the following postgresql exception:

org.postgresql.util.PSQLException: ERROR: syntax error in tsquery: "team:octopus"

Allow admins to edit all apps

Some admins should be allowed to maintain apps, even if they are no members of the owning team.

Wrong team is compared

When updating an application, the team contained in the payload is used for authorization. It should be the team in the database.

Support audit criticality levels for application

Each application should have an audit criticality level 1, 2 or 3.
1: Not audit relevant
2: Audit relevant
3: PCI

The criticality level defines the number of required approvers (Level 1 -> 1 Approver, Level 2 and 3 -> 2 approvers)

The default value for each new application is 2. Only a few people should be able to change this.

Require scm_url, documentation_url, specification_url

Make them mandatory fields

scm_url
documentation_url
specification_url

"TODO-FIXME" in approvals

Use user from provided OAuth token.

Auditing fields for application

created, last_modified.

no special uids for criticality level anymore

Every team can set it for their own apps

Allow /apps to be filtered by team

This would help for zalando-stups/yourturn#386 and general curl usage.

Field to configure n-eyes-principle

required_approvers. On application for simplicity (as opposed to app version).

implement caching

Currently any "u/require-internal-user request" results in a call to the users api, this means for some parts in yourturn the users api is called at least 10-20 times in a second for the same data.

Publish the API specification

as required by https://opensource.zalando.com/restful-api-guidelines/#192

Extend [/apps] endpoint to return only apps registered after a certain timestap

zalando/kontrolletti#12 Is a job responsible for importing scm-url's from new applications registered with zalando-stups/kio.
Can you extend the /apps-endpoint to accept a timestamp filter parameter such that we could request only for applications registered/updated after a certain time?
The result should only contain apps that were updated or created after the passed timestampparameters.

Extend returned App model with scm_url in [/apps] endpoint

Right now to retrieve all the SCM-URL's from zalando-stups/kio we need to call 2 endpoints.

First we call /apps which includes all the id's of all applications.
Second we call /apps/id for each id we got on the first call.

It would be great if zalando/kontrolletti#60 could call /apps endpoint once to get all registered SCM-url's in the App json-object.
Is it possible to extend the returned json object to include the scm-url and time-created?

Search seems to have issues with dashes

Because for search foo the app foo-bar is returned, but not for the exact match query foo-bar.

Kio search errror

When the search contains spaces, kio throw an error.

GET https://kio.address/apps?search="ab"

GET https://kio.address/apps?search="a b"

Applications & Components: required changes

To accommodate Zalando's new Applications & Components definition, the following changes are needed in the Kio application registry:

Remove the obsolete/unused required_approvers field
Remove the obsolete/unused criticality_level field
Remove the obsolete/unused specification_type field
Remove the publicly_accessible field as it lost its purpose
Remove the scm_url field as it lost its purpose (application components can come from multiple git repos)
Migrate/rename the field service_url to url as it now denotes the main (production) application URL, e.g. for UI web apps. The field stays optional.
Add a new field incident_contact to denote the responsible 24x7 team (#99)

Create fulltext search query capability

Add a search endpoint to do fulltext search of application names, description and their versions with descriptions:

GET /search?q=docker+registry

adjust application criticality regarding the highest scope criticality

An Application should have the highest criticality of all the it used scopes.

Add last_modified to versions again

It breaks sorting of versions in YOUR TURN. Was removed in #10

Load Testing fields in Kio/Yourturn

As we agreed in email, we would like to add 2 more fields in Kio/Yourturn about Load Testing.
Field # 1 “Link to load test scripts”
Which can have one of 3 values:

not yet: The application must be load tested, but they do not have a load testing tool yet. 
doesn't apply: The application doesn't need to be load tested, e.g. native application.
{Load_test_artefacts_URL}: GHE link to load test script when using scripted load testing tool / GHE link to load test configuration files when using non-scripted tool / link to internal docs that explain how to run those load tests when using a non-scripted tool.

Field # 2 “Request Per Second (RPS)”
Will be added for each git revision id / docker image tag git rev-parse HEAD | cut -c-10 meaning git rev id 671ee80c08 and can handle 80 RPS

Improve logging for approvals

Right now, we only log which use, approves which application in which version but we do not know which kind of approval was given.

Improve logging

...especially for the authorization logic, which is really complex:

(defn require-write-authorization
  "If user is employee, check that is in correct team.
   If user is service, check that it has application_write.all scope OR has application.write and is correct team"
  [request team]
  (require-uid request)
  (let [has-auth? (auth/get-auth request team)
        realm (from-token request "realm")
        is-robot? (= "/services" realm)
        is-human? (= "/employees" realm)
        has-scope? (set (from-token request "scope"))]
    (if is-human?
      (when-not has-auth?
        (api/throw-error 403 "Unauthorized")))
    (if is-robot?
      (if-not (has-scope? "application.write_all")
        (when-not (and
                    (has-scope? "application.write")
                    has-auth?)
          (api/throw-error 403 "Unauthorized"))
        (require-special-uid request)))))

Kio returns 503 HTTP status for unique key violations (e.g. existing approvals)

Kio currently returns the 503 status code for failing DB queries (as SQL is wrapped with Hystrix). This is annoying esp. for unique key violations such as when trying to "overwrite" existing approvals.

Add attribute "specification_type"

An application should have an additional attribute "specification_type".
It is free text, e.g. "jira", "github", "post-its", ...

Specify API id in the API specification

Please allow Content-Type and Accept headers in CORS requests

E.g. in Node I do it like this:

res.header('Access-Control-Allow-Headers', 'Origin, X-Requested-With, Content-Type, Accept, Authorization');

Add OAuth 2.0 capability

Allow all operations only in team scope.

Allow all users to write

Currently it checks the uid in the special list:
https://github.com/zalando-stups/kio/blob/master/src/org/zalando/stups/kio/api.clj#L42

This should be allowed to any user that has the correct scope.

please, @prayerslayer :)

Please return a 401 if the provided OAuth access token is invalid

See section 3.1 here.

Application version must be immutable

Change PUT method to a PUT idempotent as the application version must be immutable

Partial application updates will reset existing fields to default values

When updating an application with partial data (i.e. leaving out some fields), Kio will currently set these fields to default values instead of keeping existing field values.

Allow service users to create applications

But safely without god mode, ie. only for their team, same as humans.

Use magnificent via friboo

Instead of fetching teams by itself, Kio should just use the central auth server "magnificent" via Friboo.

Support release stages workflow

As Release-engineer
I want reduce redundant steps while deploying new version on release stage
So that reduce deploy time for new version on release stage and keep me audit compliant

Currently every single time when I deploy release stage I could receive a lot of violations:

APPLICATION_VERSION_NOT_PRESENT_IN_KIO
MISSING_VERSION_APPROVAL
VERSION_APPROVAL_NOT_ENOUGH
EC2_WITH_A_SNAPSHOT_IMAGE

To be audit compliant I should have no violations at all.

To deploy new version into release stage without violations I should:

Manually create a version
Approve the version version
Ask at least one colleague to approve the version
Make a deployment
Go to violations and resolve EC2_WITH_A_SNAPSHOT_IMAGE manually

Unfortunately, only one of these steps makes sense for release stages

Search for applications broken: column "a_matched_rank" does not exist

Kio should offer a search for application feature (/apps?search=..)

Use case: other services offering users to input application names can implement auto completion and similar features with search.

Produce only violations that make sense

As Deploy engineer
I want to see violations that make sense
So that reduce violations count and make process much cleaner

Currently I could receive these violations together:

MISSING_VERSION_APPROVAL
VERSION_APPROVAL_NOT_ENOUGH

Unfortunately second one has no matter if first one already presented

Update Docker base image, Clojure version

Query argument mismatch

java.lang.AssertionError: Assert failed: Query argument mismatch.
Expected keys: (:description :documentation_url :specification_url :publicly_accessible :service_url :name :criticality_level :scm_url :last_modified_by :active :id :subtitle :team_id :created_by :specification_type)
Actual keys: (:description :documentation_url :specification_url :publicly_accessible :service_url :name :scm_url :last_modified_by :active :id :subtitle :team_id :created_by :specification_type)
Missing keys: (:criticality_level)

zalando-stups / kio Goto Github PK

kio's Introduction

Kio

Download

Requirements

Configuration