depscloud / deploy

Deploy deps.cloud to containerized environments.

deps.cloud

⚠️ After much internal conflict, I've decided to move this project into maintenance mode. This comes after a long 3+ years of working on this project in open source with little to no involvement from others. After trying to present this at several conferences, it's clear either the community isn't ready for or does not require such a building block. Most individuals that have come to the project open issues, but have not seemed interested in contributing anything beyond a ticket. IF interest picks up again, I'm always happy to take the project off the back burner. For now, I'm just too burnt out managing a project that doesn't seem wanted / needed / desired.

deps.cloud is a tool to help companies understand what libraries and projects their systems use. It works by detecting dependencies defined in common manifest files (pom.xml, package.json, go.mod, etc). Using this information, we’re able to answer questions about project dependencies.

• What versions of k8s.io/client-go do we depend on?
• Which projects use eslint as a non-dev dependency?
• What open source libraries do we use the most?

To start using deps.cloud

See our documentation on deps.cloud.

To start developing deps.cloud

Take a look at our contributing guidelines and project board.

# setup a workspace for all depscloud
mkdir depscloud && cd $_

# clone repository
git clone [email protected]:depscloud/depscloud.git

Support

Join our mailing list to get access to virtual events and ask any questions there.

We also have a Slack channel.

Checks

Branch

Release

Image

License

deploy's Issues

Allow additional labels to be passed into the charts.

Right now, we include a default set of labels that are added to all resources created by the chart. It's often useful to pass along additional labels.

This should be a pretty easy change and a good introduction to Helm.

Move to gitflow

Now that renovate runs over helm charts, it seems moving this project to a gitflow type model would be beneficial from a publication standpoint.

Support pod and service annotations

Right now, our helm charts don't support adding pod or service annotations which can be popular for service mesh solutions.

• Add podAnnotations to each of the helm chart
• Add service.annotatations to each helm chart

Pull generated dashboards into helm charts

It would be really useful to pull the generated dashboards from jsonnet into the helm release. The dashboards should be turned into a config map. The dashboard shouldn't be deployed by default (only if .metrics.dashboard.enabled=true, default false). The dashboard should attach a label with the key of .metrics.dashboard.label (default to grafana_dashboard) with a value of "1". The namespace should be able to be overridden using .metrics.dashboard.namespace but should default to the namespace of the release.

Problem enabling built in ingress

I keep running into:

Error: template: idepscloud/charts/depscloud/templates/NOTES.txt:13:45: executing "depscloud/charts/depscloud/templates/NOTES.txt" at <.Values.ingress.tls>: wrong type for value; expected bool; got []interface {}
helm.go:81: [debug] template: depscloud/charts/depscloud/templates/NOTES.txt:13:45: executing "depscloud/charts/depscloud/templates/NOTES.txt" at <.Values.ingress.tls>: wrong type for value; expected bool; got []interface {}

or

Error: template: depscloud/charts/depscloud/templates/NOTES.txt:13:45: executing "depscloud/charts/depscloud/templates/NOTES.txt" at <.Values.ingress.tls>: wrong type for value; expected bool; got []interface {}
helm.go:81: [debug] template: depscloud/charts/depscloud/templates/NOTES.txt:13:45: executing "depscloud/charts/depscloud/templates/NOTES.txt" at <.Values.ingress.tls>: wrong type for value; expected bool; got []interface {}

From what I can tell:
This expects a boolean

deploy/charts/depscloud/templates/NOTES.txt

Line 13 in e4d47fe

{{- $scheme := ternary "https" "http" .Values.ingress.tls -}}

and this expects an array/slice

deploy/charts/depscloud/templates/ingress.yaml

Line 21 in e4d47fe

{{- range .Values.ingress.tls }}

Move over to using release assets

Right now, all the artifacts from a release are sent over to a gh-pages branch. It would be better if we could use a release asset format instead of needing to maintain this directory. Right now, our approach needs to use git-lfs in order to handle the artifacts. Using releases would drastically simplify this.

However, we still want to preserve the index.yaml file. We simply want to move the artifact location to be as github assets.

Bump helm charts to 0.2.34

Don't auto-mount service account tokens

Right now, we allow service account tokens to be auto-mounted. This poses a potential security risk as anyone in the pod now has a token and access to the Kubernetes API. None of the deps.cloud processes need to communicate with the Kubernetes API so there's no real value in mounting the token.

Add cert-manager certificates to charts

The charts expect these to be externalized, but it would be nice if they came with a quick way to enable mTLS between processes using cert-manager certificates.

https://cert-manager.io/docs/usage/certificate/

There are two perspectives to consider. The first is from using the umbrella depscloud chart and the second is using each chart individually.

We should really only need to add a create and issuerRef value. Everything else should be able to be encapsulated by the chart.

tls:
  create: false
  secretName: ""
  issuerRef: {}

Similarly, we will want to update the clients (indexer, gateway) to use a similar structure. Right now, secretName is right under the service key but it should be fine to move it.

extractor:
  tls:
    ...

tracker:
  tls:
    ...

secretName for clients should behave similarly to how the address is set up.

tls.create and tls.issuerRef should be overridable by .global.tls.create and .global.tls.issuerRef respectively.

The depscloud umbrella chart should add a tls.issuer section to facilitate the creation of the cert-manager issuer resource. The issuer should be scoped to a namespace

https://cert-manager.io/docs/configuration/selfsigned/

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• chore(deps): update cockroachdb/cockroach docker tag to v21.2.17
• chore(deps): update postgres docker tag to v13.15
• chore(deps): update actions/checkout action to v4
• chore(deps): update actions/setup-go action to v5
• chore(deps): update cockroachdb/cockroach docker tag to v24
• chore(deps): update helm release mysql to v11
• chore(deps): update helm release postgresql to v15
• chore(deps): update mariadb docker tag to v11
• chore(deps): update peaceiris/actions-gh-pages action to v4
• chore(deps): update postgres docker tag to v16
• Click on this checkbox to rebase all open PRs at once

Detected dependencies

docker-compose

docker/cockroachdb/docker-compose.yaml

• cockroachdb/cockroach v21.2.7

docker/mariadb/docker-compose.yaml

• mariadb 10

docker/mysql/docker-compose.yaml

• mysql 8

docker/postgres/docker-compose.yaml

• postgres 13.6

docker/sqlite/docker-compose.yaml

docker/tls/docker-compose.yaml

github-actions

.github/workflows/branch.yaml

• actions/setup-go v2
• actions/setup-node v2-beta
• actions/checkout v3
• peaceiris/actions-gh-pages v3

helm-values

charts/extractor/values.yaml

charts/gateway/values.yaml

charts/indexer/values.yaml

charts/tracker/values.yaml

helmv3

charts/depscloud/Chart.yaml

• mysql 8.8.30
• postgresql 10.16.2

jsonnet-bundler

monitoring/jsonnetfile.json

---

• Check this box to trigger a request for Renovate to run again on this repository

Update sqlite configuration

The default configuration for tracker depends on sqlite3 as a parameter. In a recent change, we swapped over to using sqlite as the storage driver.

Update README.md and Chart.yaml for all charts

Now that the project is listed on artifacthub, we should update the project's README.md files to include some information about the project, calls to action, and any metadata. The parameter section, while convenient is a bit clunky and difficult to maintain.

Setup tests across multiple versions of kubernetes

To ensure our deployment configuration works across multiple versions of kubernetes, we should set up some tests to ensure the deployment configuration applies cleanly. We should be able to do this pretty easily with Github actions and a matrix. Take a look here to see how I did this with tracker across storage providers. The test doesn't have to do anything fancy to start, just cleanly bring up the system using the new deployment configuration.

https://github.com/depscloud/depscloud/blob/main/.github/workflows/tracker-integration.yaml#L23-L30

Versions to test: 1.16, 1.17, 1.18, 1.19 (should eventually build in a support model)

Tests should only be run if the charts change.

I can help point someone the right way on this once they get started.

Add ingress to gateway chart

Right now, we expect ingress definitions to be added outside the context of the deployment. Shouldn't be too hard to generate a new helm chart and copy it's ingress block over.

Make sure you're using Helm v3

Move .travis.yml to github workflows

See depscloud/deps.cloud#9

Action Required: Fix Renovate Configuration

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

Error type: undefined. Note: this is a nested preset so please contact the preset author if you are unable to fix it yourself.

Add prometheus-pushgateway for the indexer process.

Requires completion of depscloud/depscloud#108

Once the indexer has support for the pushgateway, we should make it easy for folks to deploy one with the depscloud chart. By default, it shouldn't be enabled (similar to the mysql and postgres integration). In addition to that, the indexer chart needs to be updated to know the location of the pushgateway.

Information on the pushgateway chart:
https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus-pushgateway

Add postgres to deployment configuration

Now that tracker has been published with postgres support, we'll want to add some example deployment configuration.

1. Add a docker/postgres folder and include a simple docker-compose.yaml file for running the postgres integration
2. Add a step to scripts/generate.sh to generate a postgres.yaml file similar to that of mysql.
   • You can probably use either of bitnami/postgres or bitnami/postgres-ha since we already have the bitnami repo tapped for mysql.
3. Update readme to call out to availability of postgres resources

Add support for HPA

Not sure what the right way to do some of this is, but I'd like to add an HPA to the ecosystem to help it optimize usage.

Enabling Jekyll breaks YAML

https://github.com/depscloud/deploy/blob/main/.github/workflows/branch.yaml#L28

When jekyll is enabled, it transforms YAML files, often breaking some of the definitions. We were able to put a solution into place that mitigates that. It would be nice to use a simple markdown converter to turn the README.md file into a single HTML file during build and then set this property to false.

Add better landing page for gh-pages

I put something quick in but it would be good to actually have a landing page. It's also tempting to replace this approach with a proper artifactory.

Fix target templating

Right now, targets are hardcoded to depscloud-{extractor,tracker}. This means if the depscloud chart is released with anything other than depscloud, the addressing breaks.

Document database connection string pattern

This uses a standard Golang connection string. We should document it for clarity.

Convert dashboard to jsonnet

The current dashboard likely isn't portable. Now that more direct support for prometheus has been added, it should be pretty easy to start in on converting this and building out some more in-depth dashboards.

Automate more of the release process

Right now, a lot of the release process in this repo is error-prone. It would be good to automate as much as possible so that mistakes aren't made. I think moving over to tags has helped, but I need to do some more of it.

Update content for group rename

See for more details: depscloud/deps.cloud#41

Add NOTES.txt to charts

After people install the project using Helm, they are shown a brief section of notes that describes how to use the system. We currently do not provide any notes and it would be good to include a snippet about how to access the API and configure the CLI to point to the system.

Hosted MySQL YAML missing service account

It's not clear why, but the YAML being generated by bitnami/mysql does not contain a service account. This is blocking pods from starting up for the statefulset.