GithubHelp home page GithubHelp logo

iver-wharf.github.io's Introduction

Wharf documentation

Codacy Badge

This repo contains a Docsify.js site that's hosted via GitHub Pages over at https://iver-wharf.github.io/.

Build locally

  1. Install Node.js: https://nodejs.org/en/download/

  2. Install Docsify.js dependencies using NPM:

    npm install

  3. Start Docsify's local server (that supports hot-reloading!)

    npm start

  4. Visit http://localhost:3000

Linting

make deps # download linting dependencies

make lint

make lint-md # only lint Markdown files

Some errors can be fixed automatically. Keep in mind that this updates the files in place.

make lint-fix

make lint-fix-md # only lint and fix Markdown files

Maintained by Iver. Licensed under the MIT license.

iver-wharf.github.io's People

Contributors

alexamakans avatar applejag avatar fredx30 avatar pikabanga avatar

Stargazers

avatar avatar

Watchers

avatar

iver-wharf.github.io's Issues

POC - Load certificates by mounting into the containers instead of into the Docker images

This is an old ticket moved from our internal ticketing system. The idea was to use OPA but if we could find a different way to solve this without introducting yet another dependency then that would be swell.

This deserves an RFC once POC is working

Currently the certificate added by the Wharf build into the images when building with kaniko is loaded via the following lines:

We want to add the certificates into the containers inside kubernetes instead, leaving the images intact of any self-signed CAs.

Suggestion is to use OPA (Open Policy Agent) in our kubernetes cluster to dynamically add in the mounting of our CA certs, for example via a configmap.

Steps:

  • RnD how to use OPA in k8s (tip: https://www.openpolicyagent.org/docs/v0.12.2/kubernetes-admission-control)
  • The rules we want to impose are something like:
    1. If not containing some flag, for example an attribute or label on the pod or on the container with name & value wharf-inject-certs: false (to be able to turn it off)
    2. Add configmap containing certs as volume to pod
    3. Add mounting of volume to container into designated paths
  • Alternatively we want rules like this:
    1. (same as above) If not containing some flag, for example an attribute or label on the pod or on the container with name & value wharf.iver.com/inject-certs: false (to be able to turn it off)
    2. (same as above) Add configmap containing certs as volume to pod
    3. Add init-container to each container that prepares the /etc/ssl/certs folder in a temp volume using the configmap volume of our certs.
    4. Mount the temp volume into /etc/ssl/certs on all containers.

After some research (mostly by looking in the update-ca-certificates script), certs needs to go into

# certs are imported from here
/usr/local/share/ca-certificates/**/*.pem

# certs are stored here
/usr/share/ca-certificates/**/*.pem

# conf file tracking all added and ignored certs
/etc/ca-certificates.conf

# all certs in one file
/etc/ssl/certs/ca-certificates.crt

# all certs one by one, name hashed via `openssl rehash .`
/etc/ssl/certs/{HASH-OF-CERT}

That's why it's probably best to do it with an init-container, or maybe an operator. Needs further RnD!

This issue is requesting a tracerbullet. Just something that works good enough for regular GNU/Linux containers. Disregard windows containers for now.

Document access key permissions: azuredevops

When importing from either provider, you need an access key.
The access this key needs has to be documented, as it's an obvious question that arises.

Todo:

  • Look at the code (https://github.com/iver-wharf/wharf-provider-azuredevops), see what it's doing, and coming up with the assumed permissions
  • Do some manual testing to verify with a key that only has those permissions
    • importing new project
    • refreshing the project
  • document the required permissions.

We should make it possible to update the credentials for a project in the future when we need it. Such as when we need to set commit statuses, we then need permission for that. But let's follow the "principle of least privilage"

Canary manual builds to our internal test environment

This is a multi-step process as concluded in a previous meeting. Meeting notes: https://iver-wharf.github.io/wharf-notes/2021-11-08-biweekly

The meeting notes states that this should be done after the Jenkins move. But I (@jilleJr) consider it such a low hanging fruit to get at least something up and running, to dismiss it.

To begin with, we need a way to build and deploy any branch from the Wharf projects and then deploy them to our internal test environment.

Associated PRs

Document access key permissions: gitlab

When importing from either provider, you need an access key.
The access this key needs has to be documented, as it's an obvious question that arises.

Todo:

  • Look at the code (https://github.com/iver-wharf/wharf-provider-gitlab), see what it's doing, and coming up with the assumed permissions
  • Do some manual testing to verify with a key that only has those permissions
    • importing new project
    • refreshing the project
  • document the required permissions.

We should make it possible to update the credentials for a project in the future when we need it. Such as when we need to set commit statuses, we then need permission for that. But let's follow the "principle of least privilage"

Automatic artifact uploads

Regards the "container" step

This is no longer relevant as we're transitioning away from Jenkins. However, the feature itself is still heavily requested.

Once the cmd repo has been transferred (see #32) we can add a new issue there with the equivalent feature request, as well as closing this issue.

Todo

  • Feed build with URL to Wharf API
  • Add to trigger Jenkinsfile to give artifacts property to container step
  • Add to container Jenkinsfile to upload specified file (or files in folder) to given Wharf API for its BUILD_REF
  • Add logging of which files were uploaded

Example step config

container:
  image: mcr.microsoft.com/dotnet/core/sdk:3.1
  cmds:
  - dotnet test /repo/src/Iver.Spark.sln --results-directory /test-results
  artifacts:
  - /test-results

To consider: curl and wget will not exist on every image that's used. It's really bad habit to assume the image using is either Linux, have certain tools preinstalled, or even has permission to do some stuff.

Good option is to mount a second container, for example how they do it in their example found in the README.md:
https://github.com/jenkinsci/kubernetes-plugin#container-group-support

Trimmed down example of just that:

podTemplate(containers: [
    containerTemplate(name: 'main', image: '$CONTAINER_IMAGE', ttyEnabled: true, command: 'cat'),
    containerTemplate(name: 'artifacts', image: 'ubuntu:20.04', ttyEnabled: true, command: 'cat')
  ]) {

    node(POD_LABEL) {
        stage('Run container commands') {
            git 'https://github.com/jenkinsci/kubernetes-plugin.git'
            container('maven') {
                sh CONTAINER_CMDS
            }
        }

        if (CONTAINER_ARTIFACTS) {
            stage('Upload artifacts') {
                container('golang') {
                    sh"""
                    for file in $CONTAINER_ARTIFACTS/*; do
                        curl $WHARF_API/something/artifacts
                    done
                    """
                }
            }
        }
    }
}

Notification sending component with persistence

After #33 is deployed and tested, we then need to consider persistence.

Add persistence of messages that are going to be send with email notifications. Messages should be first saved to the persistence and after that the module should attempt to send them. Messages should be deleted from persistence only after they are successfully sent.
We do not need to keep them in a database, yaml/json files should be enough.
Number of retries should be configurable (configmap from initial task). Exponential delay on the retries.

Document coding styles and IDE integrations

We have some Go coding style rules that we decided on in the Wharf team a while ago, but this was before we went open source.

These notes needs to be moved to iver-wharf.github.io, cleaned up, and then possibly linked in each repo's CONTRIBUTING.md

Ask me, @jilleJr, where to find these private notes if you wish to take on this task.

GitHub actions for running tests

Issue #72 regards adding vulnerability scans, but this only regards unit testing and formatting checks

For all Go repos:

  • Add unit testing action
  • Add goimports diffing, making sure everything is formatted (via git diff --exit-code)

For Angular repo:

  • Add unit testing action

This needs to be added as GitHub Actions to the applicable repos to be run on new or updated pull requests.

Codacy already deals with linting and some static code checks

Sub-issues

PRs

The future plan is to have Wharf run and validate pull requests, but as Wharf lacks that functionality we have to settle with GitHub Actions, as some is better than none

Document access key permissions: github

When importing from either provider, you need an access key.
The access this key needs has to be documented, as it's an obvious question that arises.

Todo:

  • Look at the code (https://github.com/iver-wharf/wharf-provider-github), see what it's doing, and coming up with the assumed permissions
  • Do some manual testing to verify with a key that only has those permissions
    • importing new project
    • refreshing the project
  • document the required permissions.

We should make it possible to update the credentials for a project in the future when we need it. Such as when we need to set commit statuses, we then need permission for that. But let's follow the "principle of least privilage"

Include third-party-licenses notices in projects

We are depending on lots of libraries, none of which we give credit for. This is a violation of the open source licenses that they have, and actually needs to be addressed ASAP, for legal reasons.

We may need to remove previous released versions that does not have this, or backport these changes to the previous released versions.

Needs to be investigated:

  • How to extract licenses automatically on build, for example via https://github.com/google/go-licenses

  • What requirements does the licenses have? Can we just add the licenses next to the binaries in the Docker images, or do we need CLI argument to extract the licenses, or do we need new endpoints to return the licenses? For the frontend I think just a "third party licenses" page would suffice, but that needs to be investigated as well.

  • Are we using any dependencies with incompatible licenses to our MIT license?

  • Can we get an actual open source lawyer to look over the upcoming RFC for this, to see if our solution is compliant to the different licenses?

This needs an RFC of the proposed solution for how we solve this for all of our code repos.

Investigate fetching deployment status

This is about fetching deployment status from kubernetes clusters and displaying this on Wharf frontend.

Task is an investigation of how to get that information from the cluster.
There is multiple scenarios of how to do that.

What is the most convienient for the user?
What is the easiest to implement by us?

Should we monitor per build?
Should we monitor per environment for a project?

After investigation we need to make a meeting with Bjorn and agree on the particular solution for this issue.

Possible solution

Investigate escalation hacks for container step

Some shell syntaxes allow escalation of priviliges, such as the ; & | combo.

Need to investigate these security holes and if it can damage us. It's not unthinkable that a customer sells Wharf as a service and allows arbitrary commands to be executed. This is an attack vector that needs to be investigated if it can be abused to hurt us or our customers.

This was originally mentioned by Niklas E. Could perhaps consult with him for more information about this and ways to solve it.

Move remaining repos

Remaining repos:

Rules for moving these:

  • Redact all company references
  • No Git history. Safest way is just copying all files except the .git folder into the new repo

Once these has been moved we can update the "About Wharf" page in this repo so it no longer says "goal is to be open sourced by end of 2021"

Authentication & Authorisation

This concerns the implementation of OpenID Connect as a mechanism for authentication and eventually possibly also and implementation of authorisation.
TODOs ->

  • Review callback url:s for the azure app reg.
  • Investigate getting delegated controll for the app reg.
  • Write down thoughts about nice-to-haves.
  • Write down must haves.

Documentation

  • #91
  • Document - Wharf Configuration Process
    • Auth config - via helm-values. iver-wharf/wharf-helm#45
    • Auth config - via docker-compose
    • Auth config - running as a binary

Changes for swagger

  • Enable auth for interaction
  • Whitelist swagger app

Changes wharf web

Changes for the api regarding authentication

Changes to wharf-helm

Changes api to implement authorization

  • Paths/Path-groups need to individually verify scopes from the token.
  • Communicate with admins for the app reg about creating groups and delegating controll to controll over included poeple.

Flesh out step types docs

The step types docs are very basic at the moment. Using solely a code block to describe everything.

This is fine for us internally, but does not show a greater grasp over how these step types work.

Suggested convention:

<!-- panels:start -->

<!-- div:left-panel -->

# Container step

Short description of what it does.

Longer description of how it works.

<!-- div:right-panel -->

## Minimal example

```yaml
myStage:
  myStep:
    container:
      image: alpine:latest
      cmds:
        - echo 'hello world'
```

<!-- panels:end -->

<!-- panels:start -->

<!-- div:left-panel -->

## Parameters

### `image`

Type: `string`

Short description of what it does

```yaml
image: alpine:latest
```

### `cmds`

Type: `string array`

Short description of what it does

```yaml
cmds:
  - echo 'hello world'
```

<!-- div:right-panel -->

## Full example

```yaml
myStage:
  myStep:
    container:
      image: alpine:latest
      cmds:
        - echo 'hello world'
      os: linux/windows
      shell: /bin/sh
      secretName: mysecret
      serviceAccount: default
      certificatesMountPath: /usr/local/share/ca-certificates
```

<!-- panels:end -->

Preview:

Screenshot_2021-05-20_09-43-40

Fix communication between API ↔ providers

Currently providers can talk to the API, but not the other way around. We want to change that.

Most providers support checks/statuses that can reject PRs and such (see iver-wharf/wharf-provider-azuredevops#5)

The previous attempt was to solve this with a message queue (see https://github.com/iver-wharf/messagebus-go) just to get around the circular-dependency issue. This has since been concluded as overkill.

What we instead can do is use dependency-inversion, where we create a standardized API for the providers that they have to fulfill, and then the main API can talk to them through this standardized interface. By doing this, we can also let the main API do all the redirection of endpoints so the end user does not talk to the provider/import APIs directly, and by doing that we can populate the data that the providers get as well, such as the project name when the user only provided the project ID.

This needs an RFC. Especially regarding how this standardized API should look. If we stick to REST or transition over to an asyncronous protocol like gRPC or websockets is up for discussion. Personally (from @jilleJr's perspective) I find gRPC the most enticing.

v2+ Go modules

Based on RFC: iver-wharf/rfcs#26
Published: https://iver-wharf.github.io/rfcs/published/0026-v2-go-modules-release

Need to update the module names in the following repos to include the major version as suffix:

Motivation

We need to stay compatible with Go modules to be able to use this library from other projects. Go modules almost requires it's own doctrine, but this PR is based on some statements from here: https://github.com/golang/go/wiki/Modules#releasing-modules-v2-or-higher

Example errors that appear as we haven't been Go modules compatibility up until now:

$ go get github.com/iver-wharf/wharf-api/pkg/[email protected]
go get github.com/iver-wharf/wharf-api/pkg/[email protected]:
  github.com/iver-wharf/[email protected]:
    invalid version:
      module contains a go.mod file, so major version must be compatible:
        should be v0 or v1, not v4

(I've wrapped the text in console output just to be more easily readable)

Document unit-test naming convention

Our unit tests are not that coherent in their names. We use a mix of different naming conventions, as they are written by different developers.

The simple answer is just to name them:

func Test{NameOfTypeOrFunction}(t *testing.T)

// Example
func Add(a, b int) int

func TestAdd(t *testing.T)

But it gets diffuse when we start having multiple tests targetting the same function/type/method but with slight variations.

An example format of this (though not that great IMO):

func Test{NameOfTypeOrFunction}_{StateOfType}_{ExpectedResult}(t *testing.T)
func TestAdd_GivenNegativeValues_ReturnsNegative(t *testing.T)

Suggest to write an RFC of this.

Track version in repos

Currently each program's version is only set when building via the docker Makefile target or via Wharf.

Would be nice if for example go install github.com/iver-wharf/wharf-api contained the version already, instead of our current "local dev" version.

Suggest to:

  • Get version from version.yaml (compared to now where it's an empty string in a freshly cloned repo)

  • Get Git commit from BuildInfo.Settings["vcs.revision"] (debug.ReadBuildInfo, package runtime/debug)

  • Use Git commit date from BuildInfo.Settings["vcs.time"] (it's not the build date, but closest available. Maybe add that as another field to app.Version as Version.CommitDate)

  • Leave the Version.BuildRef as unset, and only populate it from Wharf builds, as is done already.

This needs to first be implemented into wharf-core, then info all of the dependent repos.

Notification sending component

Basic API (REST) that can send notifications based on some configuration.

To begin with, you configure the component with SMTP credentials and then the API can tell this component to send an email based on its config from the database

This component only needs a basic HTTP endpoint that will tell it to send an email. The API implementation comes later in iver-wharf/wharf-api#8

Having persistence and remembering notifications that failed to send is out of scope for this issue. That comes later. Make this just a dumb endpoint for sending emails to begin with.

Add "committish" parameter to Jenkins build

An improvement of usability would be to be able to tell the Jenkins build which git committish to build.

Currently it will build the latest commit on the target branch. You cannot build or deploy an older version. That should be the default behaviour, but the freedom of being able to declare which committish could improve usability a lot.

What needs to be done:

  • Add GIT_COMMITTISH parameter to /trigger and default-build/trigger builds
  • Add logic in the trigger Jenkinsfile to checkout the committish instead of the designated branch.

This will probably be irrelevant once we get the cmd project up and running, but the same applies there so we're keeping this issue for that sake

Allow GPG-signed-only enforcement

Enforcing GPG signed Git commits is a great security feature for anything that deals with Git repos.

Suggested implementation:

  • When enabled, reject builds on commits that are unsigned or unverified.

  • Users provide Wharf with a list of public keys that it has in its own storage that cmd project then uses with a basic git verify-commit HEAD call, preferably with a simple --verify-commit flag or something to the cmd command.

    • To extend later (in a different WO): Wharf refreshes its DB of GPG keys every now and then as a CRON job by populating it with keys from the different providers and removing old ones. This refresh should then also need to be activateable by a user to force refresh for a given project.

  • Wharf API shall provide all GPG keys for a given project on demand for the builder to use for verifying the commits in a single endpoint. For example: GET /projects/{projectid}/gpg/keyring to just get all GPG keys together in PEM format (one directly after the other) so they can all be imported into GPG at once by doing:

    $ curl "https://wharf.local/api/projects/${PROJECT_ID}/gpg/keyring" > keyring.pub
$ gpg --import keyring.pub

    We can later add a GET /projects/{projectid}/gpg/keys endpoint to receive each key with some metadata in JSON format instead, similar to how GitLab does it. Would be nice to add just for the flexability.

  • Public GPG keys will be stored in the DB. Columns needs to be

    1. (PK) incremental int
    2. (indexed) GPG key ID
    3. GPG key in PEM format (named public_key, PublicKey, or something like that)
    4. Timestamp of when it was added
    • There shall be 1:M mapping table for the project, as a key can be reused by multiple projects, and when we fetch the keys we only want keys for a given project. This comes more into effect with GitHub as you cannot simply "import ALL keys from everyone", and therefore we need to have it scoped on a per-project level, which is the lowest common denominator in this. You can figure out which providers a key exists in by looking at it's project mappings and those projects provider mappings.

This was originally a comment from Niklas Engvall, and I [kalle] presume it comes from his usage of Argo CD which has this feature.

Problem pages

We're introducing problem types, according to IETF RFC-7087, and need problem describing pages for the following:

  • docs/prob/api/internal-server-error.md (#27)
  • docs/prob/api/invalid-param.md (#29)
  • docs/prob/api/invalid-param-int.md (#27)
  • docs/prob/api/invalid-param-uint.md (#27)
  • docs/prob/api/missing-param-string.md (#29)
  • docs/prob/api/project/cannot-change-group.md (#29)
  • docs/prob/api/project/run/params-deserialize.md (#29)
  • docs/prob/api/project/run/params-serialize.md (#29)
  • docs/prob/api/project/run/trigger.md (#29)
  • docs/prob/api/provider/invalid-name.md (#29)
  • docs/prob/api/record-not-found.md (#29)
  • docs/prob/api/unexpected-body-read-error.md (#29)
  • docs/prob/api/unexpected-db-read-error.md (#29)
  • docs/prob/api/unexpected-db-write-error.md (#29)
  • docs/prob/api/unexpected-multipart-read-error.md (#29)
  • docs/prob/build/run/invalid-input.md (#29)

See api!45 and api!46 MRs for more info (sorry, not open sourced yet)

Add provider Go template repository

GitHub supports template repositories. Once we get the providers a bit more stable and how we want them, we can create a repository, named iver-wharf/wharf-provider-template-go, and then we can add more providers.

Top prio on new providers:

  • Bitbucket
  • Gitea
  • Gogs

Spin up and tear down a new kubernetes environment for tests

Our future planned workflow for running integration/system tests:

  1. Wharf can detect new PRs
  2. Wharf creates a new build from that PRs branch
  3. That PR build creates a new namespace and deploys everything to there (See "Create "kube-copy" wharf step type")
  4. When the PR is merged or closed, Wharf should be able to start up a new build to tear down the setup (ex: kubectl delete namespace)

What would aid in this particular use case is for tighter integration with Kubernetes. We already have Kubernetes integration planned (see "Investigate deployment status") but what this ultimately needs is a way to configure Wharf to wait until all the deployed resources are fully deployed and running (ex: All deployments have 100% of their pods in Ready state) before proceeding with running the tests. Though this can be scripted and is not directly dependent on any Wharf functionality. Wharf can only add QoL features to this.

Add Trivy vulnerability scans to all docker-released repos

Best case:

  1. Once a day, a GitHub Action builds the docker image and scans it
  2. If there are vulnerabilities, an issue is created with the label security
  • How to handle if issue already exists? Create yet another one? Update the existing one? Do nothing while such a vulnerability scan issue already exists?

The issues should contain description of how to build and scan the docker image yourself locally. Such as:

$ docker build . -t wharf-web

$ docker save wharf-web -o image.tar

$ trivy image --input image.tar

There are some alternatives available, just searching the internet for "trivy github action" yields lots of good alternatives.

Suggest to add this to one repo, and once that is reviewed and merged, first then start applying it to the rest of the repos.

Repos that need this:

In the future we can translate this to a Wharf build, but as Wharf lacks this kind of integration right now we should start the work using GitHub Actions.

Publish Docker images publicly

Based on RFC-0011: quay.io

Docker images are currently stored in our internal docker repo. We want these to be publicly available.

We shall host them over at https://quay.io
Both are free for open source repos. Quay.io includes RedHat's Clair that gives industry leading security scanning.

Todo

  1. Register account on quay.io using [email protected] with the username iver-wharf (same as on GitHub)
  2. Setup our build pipeline to deploy images to quay.io
  3. Optionally add a badge in the README.md (ex: https://quay.io/repository/iver-wharf/wharf-api/status)
  4. Update docs on how to get and use docker images
  5. Update docker-compose.yml in iver-wharf/wharf-docker-compose to reference the new image names so you don't need to build them locally, but still can

Conflicts

Certificates! Oh the joy!

We need a better way to import our internal self-signed CAs than our current procedure where we're embedding them into the images. But this deserves a different GitHub issue and nothing we should have to worry about here. It can be looked into in #44

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.