jchesterpivotal / concourse-build-resource Goto Github PK

Right now put / out is a no-op.

What it should do is trigger a build.

This would first require authentication logic, which is a whole thing.

Not that I particularly like being on the bleeding edge of things, but GOPATH is basically easy-CI kryptonite. So I am going to go through this in order to make #7 easier.

The problem

Currently, the resource injects 4 metadata (its release version, its git ref, the timestamp and the remote Concourse version).

It injects these into all of the JSON files it produces. This makes it easy to get basic provenance of a record inserted into a database.

What I would like to have is a trustworthy unique identity for any particular get. None of the existing metadata, singly or in combination, is able to do this. In particular there's a temptation to assume that together they will be unique, but that's not true: I could be running jobs in parallel. Or I might be running the resource in multiple places and pooling the data.

A solution

To guarantee uniqueness, add a UUID to the injected metadata. This simplifies insert logic and lets me tie together all the information ever retrieved on a particular occasion.

Problem

As released in v0.11.0.

When I compare the browser readout of events with the events.json file written by the resource, I see that the latter is missing a lot of information I would very much like to read.

In particular, the event type (eg, log, finish-put) is missing.

Solution

The root of the problem seems to be that what I get from Concourse's stream handling code is the atc.Event type. What I actually want is the atc.Envelope type, which has the data put into atc.Event as well as the event type/type version information that's currently missing.

Turns out there's an endpoint I overlooked.

Will probably be a little tedious to go through and backfill the various files and whatnot, but the additional of access to the triggering info is worth the effort IMV.

• job.json
• job-12345.json
• team_pipeline-name_job-name.json
• tasks/show-job
• Readme

Description

One of my purposes is to capture data about builds and resource versions for further analysis. In practice this means I will be feeding a database. I imagine other folks will want to do this too.

As well as JSON, spitting out one or more files with SQL inserts would be useful. The files could be picked up by subsequent tasks or resources for further action.

Open questions

• What kind of testing?
• One file with everything? Several files with different stuff in them?
• How DB-specific? In practice, my view is that PostgreSQL is awesome and that I like to use its features.
• What schema is provided or implied? Do I provide migrations?
• Is it a good idea to develop some kind of postgresql-database or perhaps gcp-bigquery resource, or is that too much effort vs value?

Todos

• in
  • cmd/in
  • pkg/in
• Provide show-sql-inserts utility task

Problem

Job logs/events are often private, even if the overall pipeline is public. In the original events.log implementation, receiving a "not authorized" message from the ATC would lead to skipping writing the events.log file.

Because the logic for events.json follows a distinct path, it is not correctly skipping. Instead, it collects events and stores them in a null, which is faithfully written out to disk. This is because whereas the events.log file is handled in a single specialised method, events.json is handled by two routines following the convention of other endpoints: a fetch method and a write method.

The presence of an incomplete events.json file confuses downstream tools which expect the existence/non-existence of the file to signal whether the events could be retrieved.

Solution

Stop doing that. The logic which fetches events for encoding as a JSON array embedded in events.json should somehow signal that the file is not to be written.

For example, it seems quite easy to hang on fetching event streams.

The problem

This resource scrapes remote Concourses. It relies on go-concourse, for which the upstream code generally tracks the latest Concourse release. While the API should be stable, we live in a world of chaos; incompatibilities may well arise.

For debugging and record-keeping purposes, users may wish to snapshot the version of Concourse their data was scraped from.

This information is visible in the x-concourse-version header in API responses.

Todos

• Inject Concourse version data into JSON under the concourse_build_resource object.
• Add a concourse_version K-V file.

I believe this happens because, when event streams are unauthorised (ie, the logs are not public), it returns a nil. Then I defer a .Close() call, which is invoked during error-handling. Kablammo.

The problem

Right now I basically ignore the business of keeping in sync with the upstream code from Concourse (go-concourse, atc etc). This is in part because Concourse itself is sprawled over multiple repositories that are not really synced with each other within the repositories. The syncing is achieved as a by-product of manufacturing BOSH releases, but not applied back to the repositories in a way that go modules can easily consume.

At the moment I have a little bit of a Frankenstein's go.mod file to try and approximately pin versions. This works, but has problems:

• I'm trying to work out what to pin to at a git-commit level. I might get it wrong.
• Concourse keeps moving forward and so too might my upstream dependencies, but many users will want to scrape older versions of Concourse. What's the compatibility matrix going to be like? Or will I take the fly policy and wash my hands of it?

A solution

The Concourse team are aware of the problems of multi-repo sprawl and are working towards consolidating into a monorepo.

When that work lands and stabilises, I anticipate that all my direct dependencies on Concourse code will be safely consumable as a single version line in go.mod. I will be able to introduce pipeline jobs to monitor and update this resource to keep in sync with the upstream.

Open questions

• Do I abandon vendoring? It provides a useful guarantee of stability and reproducibility, but doesn't seem to be the "approved" way to work with go modules. It also makes the repo much heavier than it needs to be.
  • Following from which: if I drop vendoring, do I prune history or accept the cloning cost forever?

Todos

None currently; the issue is blocked on the upstream work.

The problem

Suppose I have a massive directory full of JSON files. Which version of concourse-build-resource is responsible? Perhaps there is a bug and I need to redo some scraping. Or perhaps a later version has information an earlier version didn't capture, and I want to know that.

Questions

• How? I take it I can snapshot environment variables at build time.
• What else? HTTP user-agent?

• Move to dockerhub
• Move bucket
• Stand up new Concourse to run pipeline

Problem

If I set initial_build_id to 100, but the earliest global build number visible for a team/pipeline/job combination is 101, the resource will "hang" after fetching version 100. This is because I use since as my search parameter, which returns an empty array for build IDs that are below its range.

Solution

Use limit=1 and until=1 to find the earliest version visible for a given team/pipeline/job. If it is above initial_build_id, use that number instead.

Description

The versioned resource types endpoint for a pipeline returns very useful information about the configuration of resources at the time they were used.

The distinction may vanish in future, but for now it's very helpful.

This will be helpful for #19 and (posthumously) #32.

As a lazy person, I don't want to write code to read JSON so I can do smart things with it. I want someone else to explode my incoming data into a variety of shapes that I can use easily.

• Add files which encode information in their filename (for ease of putting to file stores using glob patterns)
  • <thing>-team-pipeline-job-number.json
  • <thing>-<global build number>.json
• Add files which encode single values in the file contents (for ease of using cat et al in simple task scripts)
• Update build-pass-fail to use the status file instead of parsing JSON.

The problem

In the current design, a user fully specifies the team, pipeline and job they wish to follow:

- name: build
  type: concourse-build
  source:
    concourse_url: https://concourse.example.com
    team: main
    pipeline: example-pipeline
    job: some-job-you-are-interested-in

This is good if I want to just follow a single job. But it is a hassle when scaling up for two reasons.

1. Repetition. If I am following twenty jobs, I need to create a resource for every job. But these resources will have repeated configuration for URL, team and pipeline. This is unwieldy in terms of YAML and also, means I am spamming Concourse with 20 check containers instead of a single check container.

2. Dynamism. If I want to keep my scraping up to date, I will need to manually reconcile my scraping pipeline with target jobs. If a job is added on the target system, it does not get watched automatically. If the job is removed, my pipeline breaks.

A solution

Under the hood this resource uses go-concourse to interact with Concourse. Based on an eyeball inspection, it will be possible to loosen the restriction, so that one can track jobs to an arbitrary level of the hierarchy.

Afterwards, these would all be legal configurations:

# Track builds of a single job in single pipeline in single team
- name: build
  type: concourse-build
  source:
    concourse_url: https://concourse.example.com
    team: main
    pipeline: example-pipeline
    job: some-job-you-are-interested-in

# Track builds of all jobs in a single pipeline in a single team
- name: build
  type: concourse-build
  source:
    concourse_url: https://concourse.example.com
    team: main
    pipeline: example-pipeline

# Track builds of all jobs in all pipelines in a single team
- name: build
  type: concourse-build
  source:
    concourse_url: https://concourse.example.com
    team: main

# Track builds of all jobs in all pipelines in all teams (ie, track everything)
- name: build
  type: concourse-build
  source:
    concourse_url: https://concourse.example.com

Todos

• Jobs in pipeline
  • cmd/check
  • pkg/check
• Jobs in pipelines in team
  • cmd/check
  • pkg/check
• Jobs in pipelines in teams
  • cmd/check
  • pkg/check
• Add metadata to pkg/in response to more easily distinguish job builds in resource view

The Problem

When a user provides a URL of the form https://example.com/ instead of https://example.com, it leads to a whole bunch of // in the API paths assembled by go-concourse.

For versioning

Right now the versioning is based purely on the build ID. But the build ID is not a single, immutable snapshot of the external state, because a build can meaningfully evolve under the same ID number.

Instead, the version should be ID + a build status.

For example, 123+pending is different from 123+started, which is different again from 123+failed.

For filtering

Users of the resource will probably only be interested in one or two statuses. So it should be possible to filter -- "only show me new passes", etc, with "show me everything for every build" as the default behaviour. I'd like feedback on whether that's better as a get param or as a source param.

They come back from a remote Concourse in the form of 3, 2, 1 etc; but check needs to turn this into 1, 2, 3.

As a consequence of copying and pasting great slabs due to impatience.

Description

Currently, the resource renders events out to events.log. This is useful for folks wanting raw loglines.

It would be nice to have access to the original JSON form (say events.json), however. This would allow analysis of timestamps and grouping into the various steps.

To save on database usage, Concourse operators can configure their ATC to only retain a certain number of build logs. This appears to upset the logic I use when catching up from initial_build_id.

Often the README is showing features that don't yet exist in the latest release. For users who pull the latest docker image, this can lead to confusing behaviour.

Problem

The response from the endpoint is an array, not an object, so my hacky trick of using string substitution on parentheses instead injects into the first resource in the array:

[
  {
    "concourse_build_resource": {
      "release": "0.10.0",
      "git_ref": "d3767db",
      "get_timestamp": 1537644501,
      "concourse_version": "[redacted]",
      "get_uuid": "[redacted]"
    },
    "name": "bosh-deployment",
    "type": "docker-image",
    "source": {
      "repository": "[redacted]"
    },
    "privileged": false,
    "check_every": "",
    "tags": null,
    "params": null,
    "version": {
      "digest": "sha256:[redacted]"
    }
  },
  {
    "name": "cron-resource",
    "type": "docker-image",
    "source": {
      "repository": "[redacted]"
    },
    "privileged": false,
    "check_every": "",
    "tags": null,
    "params": null,
    "version": {
      "digest": "sha256:[redacted]"
    }
  }
]

Solution

I can see two possibilities:

1. Add it to the front of the array. It's not a typed array, after all, so I can put any old thing in there.
2. Creating a wrapping type which holds the array and the metadata inside it.

I'm inclined to prefer option 2, as it doesn't require someone iterating over the array to add special-case logic to skip the metadata object.

Description

Currently, the resource only works for public jobs in public pipelines.

This is rather limiting.

Something like:

- name: build
  type: concourse-build
  source:
    concourse_url: https://concourse.example.com
    authentication:
      # ???

Open questions

• How does this actually work? Authentication has lots of knobs and dials. Presumably I would need to implement some logic per scheme -- one for github, one for CF auth etc.
• Limit to 4.x and above? Authentication changed radically in 4.0, which is why there's a 4.0.
• How would tokens work? Do we log in every time?
• What happens when authentication fails?
• What information is recorded about auth? Do I add it to version metadata? Print it to stderr? Create more K-V files? All of these?

Todos

Assuming I focus on github only and split out other auth options into standalone issues.

• check
  • cmd/check
  • pkg/check
• in
  • cmd/in
  • pkg/in
• Readme

Currently the resource outputs team, pipeline, job etc. It also outputs url, which is the full build URL.

But actually, there are a bunch of different URLs I might want:

• concourse_url: the base URL of the Concourse. This would be the same as the concourse_url passed into source.
• team_url: the URL for the team
• pipeline_url: the URL for the pipeline
• job_url: the URL for the job (defaulting to latest build)
• build_url: the URL for the build -- what url is currently.

Description

At the moment, the resource begins with the most recent build from the remote Concourse. But I might actually want to start with a historical version and then work forwards.

Suppose I have a Concourse with 100 builds. If I have this configuration:

- name: build
  type: concourse-build
  source:
    concourse_url: https://concourse.example.com

Then the resource will start at build 100: [{"build_id","100"}].

But what I want to be able to do is start with, say, build 95:

  source:
    concourse_url: https://concourse.example.com
    initial_build_id: 95

Then the resource would return [{"build_id","95"}] for the first invocation (ie, it would not dial out at all, but simply return the configured value). When invoked the second time it would detect the difference and return the rest: [{"build_id","96"}, {"build_id","97"}, {"build_id","98"}, {"build_id","99"}, {"build_id","100"}].

Open questions:

• What happens if that build ID doesn't exist / isn't visible?
• How do I clearly document the difference between build_id (the actual version) and the build name / build number shown in the web UI for each individual job?

Todos

• add key
• use Since for API paging
• update readme

Problem

Suppose I get this resource.json:

{
  "inputs": [
    {
      "name": "stable-pool",
      "resource": "stable-pool",
      "type": "pool",
      "version": {
        "ref": "7a11a45a5f3f62e42d5d5e50df993699fe9ddd5f"
      },
      "metadata": [
        {
          "name": "lock_name",
          "value": "bellatrix"
        },
        {
          "name": "pool_name",
          "value": "cf-deployment/stable"
        }
      ],
      "pipeline_id": 9,
      "first_occurrence": true
    }
  ],
  "outputs": [
    {
      "id": 0,
      "type": "",
      "metadata": null,
      "resource": "stable-pool",
      "version": {
        "ref": "950a4bdde704c758b9184b2092895227a7ae10a6"
      },
      "enabled": false
    }
  ]
}

Here I see that stable-pool appears twice because it appears in both get and put steps of the underlying build. But information is missing from the output that was present in the input. In this case I can deduce this information from reading the input description, but it is common to have a put with no corresponding get in the same job. That means I will be stuck with less information than I want.

It appears that this is due to the current implementation in Concourse. The input array is populated with PublicBuildInput type, but there is no corresponding PublicBuildOutput type. Instead the outputs are populated with VersionedResource objects. In turn the information for both is fetched from the database in build.Resources(), which appears to be intended to populate the PublicBuildInput type, as it selects out of the build_inputs table but not the build_outputs table. Separately there is a build.GetVersionedResources() method which selects from both build_inputs and build_outputs, but which returns the VersionedResource type. It is not used by the endpoint that resources.json is based on.

Solution

Concourse also exposes a toplevel /api/v1/resources endpoint. This endpoint is agnostic to input or output status and shows all (publicly-visible) resources across the entire Concourse instance. The data in this endpoint can fill some of the missing information for the output entries (eg, name, type) but not all (version metadata).

It's not ideal, but it's better than the current state of affairs.

It is confusing and annoying.

Switch the current team-pipeline-job files to team_pipeline_job. This is because pipelines and jobs are typically named using hyphens, which means the filenames are ambiguous to a casual glance.

So, for example, plan-main-a-pipeline-name-a-job-name-123.json would become plan_main_a-pipeline-name_a-job-name_123.json. A little jarring, but at least consistent.

So after the first page is processed, things stop proceeding, because I switched to using Since paging.

This is not quite a regression from #13, but it defeats the purpose of having initial_build_id.

The problem seems to be that sometimes builds come back in different orders: sometimes ascending, sometimes descending. So my simple tactic of reversing the order isn't always going to work.

Problem

Suppose I have two independent groups with independent Concourses: Upstream and Downstream.

Upstream produces software in their Concourse pipelines which is then uploaded to a repository. In the Downstream pipelines, these uploads are detected and the Downstream team performs more processing. An example is having a central team performing security scanning, legal reviews, integration testing for major products etc.

When Downstream is watching only one Upstream, it is possible for these teams to coordinate. Downstream can add the Upstream pipeline to their monitors. But when Downstream is watching many outputs from many groups, this becomes untenable.

What's missing?

A feedback loop from Downstream's pipeline to Upstream's pipeline. Upstream want to know if particular resource versions they emitted from a pipeline caused a Downstream failure. To do so, Upstream needs a Job that will fail when Downstream fails.

Solution

This resource provides one part of the solution (fetching build results). But it's not enough. Also required is a way to convert that information into a pass/fail signal.

So: add a task that can be dropped into existing pipelines.

As a Pivot
I want to have tested code
So that I may be received in polite company

Given a codebase which is untested
When I backfill the tests
Then I should feel less guilty

• build-pass-fail
• prettyjson wrapper commands (show-build, show-plan, show-resources)
• prettyjson
• show-logs
• check cmd
• check pkg
• in cmd
• in pkg

Currently, all the fetches and writes performed during in are done sequentially.

Golang is meant to be good at this concurrency stuff. Maybe parallelise the fetch/write for each endpoint.

Right now, the README is edited by hand and the utility task YAMLs are all swinging wild and free without any versioning at all.

A task that edits these files to substitute in the release version, with a commit made back to the repo, should probably go in shipit.

Currently it's hardcoded to 50. This is the value used in go-concourse.

This limits the use of fly check-resource --from to collect historical build information.