Support for `pre_compile` and `post_compile` steps about buildpacks-python

[ipmb]

Thanks for the thoughtful write-up @edmorley! I see where you're coming from and the value in following a standard. If good examples are provided, the project.toml is bearable, but heroku buildpacks:set feels far more user friendly 🙂

[edmorley]

Thank you! Yeah this is on the list of things to consider - though it's possible I'll be dropping support for this in favour of people using a "run" buildpack instead, since it's not ideal for each language to reimplement custom run support. (For languages where there are built-in hooks, it's perhaps still worth supporting those, but for Python post_compile is proprietary so doesn't seem worth preserving in the CNB.)

I'll leave this open to track that decision, and if the feature is dropped we'll need to add warning (or error) messages to ease the transition regardless.

[luzfcb]

I haven't tried this new way of deployment yet, but I would like to leave a comment on the way I use post_compile today.
My use case for post_compile is as follows:

I have an e-commerce platform made with Django + Frontend application in React. The deployment process has integration with Sentry releases.

In Heroku Classic, you do not have access to GIT and repository, just an environment variable that informs which git commit hash of a particular deploy flow has started.
You also do not have access to information when using pipelines and use functionality to promote to Production a Heroku Slug running on Staging

The application has 2 independent react applications (one uses old things and another uses new and incompatible things, and we are gradually migrating to the new application)

We use pre_compile for:
1 - Perform compilation tasks for the 2 react applications, so that everything is prepared for when python/django compilation tasks are performed
2 - Upload to Sentry releases of the Source Maps JS files generated in the previous step
3 - Since Heroku Classic has no direct access to the Git Repository, Sentry-Cli is not able to automatically get all the information needed to create a new release in Sentry Releases, so I must do so manually by running a custom script to consult the Sentry API to find out which one was the hash of last previous deployment made with successful status (That is, a finalized release) and then generate the range of commits to which this deploy possibly covers and manually starts the creation of a new release on Sentry Releases

post_compile:

1 - We perform Django's Collectstatic
2 - If all previous steps are executed sucessfully, we execute the Finalize release step of Sentry Releases to confirm that the application was successfully implanted. https://docs.sentry.io/product/cli/releases/#finalizing-releaseseses

[edmorley]

For CNBs, the standardised way of running custom commands before/after buildpacks is now using the new built-in inline buildpacks feature, which is documented here:
https://buildpacks.io/docs/app-developer-guide/using-inline-buildpacks/

For language ecosystems where there is a standardised convention (for that language) way of running commands, I think it makes sense for the buildpacks for that language to keep supporting those conventions (for example, the Node.js buildpack supporting the package.json scripts feature for things like the build command etc).

However, for language ecosystems like Python where there is no such convention, I don't think it makes sense to preserve the classic Python buildpack's proprietary bin/{pre,post}_compile steps, given there is a built in feature for custom commands in the CNB spec (inline buildpacks).

As such, I think I may not end up adding support for this in the Python CNB. Though I may end up adding some explicit error handling to ease the transition. (Which could for example print an example inline buildpack config to the build log, which users can then copy directly into their project.toml.)

[ipmb]

I understand the motivation here, but project.toml feels like a big regression in DX/ergonomics.

Previously I could do something like this:

pack build myimage -b heroku/nodejs,heroku/python -B heroku/buildpacks:20

If I'm understanding the inline buildpacks, I now need to add a project.toml like this:

[_]
schema-version = "0.2"
id = "io.buildpacks.my-app"

[[io.buildpacks.group]]
id = "heroku/nodejs"
version = "1.1.2"

[[io.buildpacks.group]]
id = "heroku/python"
version = "0.4.0"

[[io.buildpacks.group]]
id = "me/post-compile"

  [io.buildpacks.group.script]
  api = "0.9"
  inline = "echo hello"

and then I can run:

pack build myimage -B heroku/builder:22

Versions seem required in project.toml so you lose auto-updating. It also requires more knowledge of buildpack internals (schema_version, api, io.buildpacks...).

Maybe this is outside of Heroku's scope, but a thought I had was that a separate "run-script" buildpack could be published which executes a script from a predefined location (or location provided by env var)?

[edmorley]

The buildpack version in project.toml is an optional field, and defaults to "latest", so auto-updating of buildpack versions won't be affected. See:
https://buildpacks.io/docs/reference/config/project-descriptor/#iobuildpacks-_table-optional_

I agree that users may get the syntax of the project.toml file incorrect, however, we can and should document it clearly + have clear validation error messages (either upstream in the CNB lifecycle, or else in Pack CLI + the Heroku next gen build system).

Ref having to create a project.toml - in your example you would already have had to create such a file, since that app uses both Python and Node.js, and the CNB auto-detection would have only picked one language. (Just like currently Heroku only auto-detects a single language when no buildpacks are set, and for anything else you have to set explicit buildpacks using heroku buildpacks:set.)

Whilst the project.toml file won't be mandatory, it will be needed for any app that:

• uses more than one official language buildpack (as above)
• uses a third party buildpack
• wants to pin to a non-latest version of an official buildpack
• wants to use a non-default Heroku stack (or more correctly, CNB builder), or wants to control when the stack is updated rather than relying on the new Heroku default builder concept

In addition, app.json may eventually end up being merged into project.toml, meaning that a project.toml file may be needed for the Heroku CI / Review App use-cases too.

As such, I think having a project.toml will be fairly common for anything but beginner apps. And beginner apps don't tend to use bin/{pre,post}_compile.

In general though, I much prefer having buildpacks/stacks be configured via code rather than heroku buildpacks:set / heroku stack:set, since implicit differing state between apps is a regular source of user confusion ("why does my staging app work but my production app not, I'm git pushing the same source to both" etc).

[edmorley]

a thought I had was that a separate "run-script" buildpack could be published which executes a script from a predefined location

Yeah this is another viable solution too.

Though perhaps a third option would be to ask for a simpler project.toml syntax for "run commands" that don't require the full verbosity of the current "inline buildpack" concept?
https://github.com/buildpacks/rfcs#proposal

[ipmb]

The buildpack version in project.toml is an optional field, and defaults to "latest", so auto-updating of buildpack versions won't be affected.

Using the latest pack, I get this error if I remove the versions from the heroku buildpack groups:

$ pack build myimg -B heroku/builder:22
22: Pulling from heroku/builder
Digest: sha256:c9f45adaaf3f91eb2f126b29c8e947f1375bae560a725511393b6a197cfe765d
Status: Image is up to date for heroku/builder:22
22-cnb: Pulling from heroku/heroku
Digest: sha256:0c422f13416d589a199e3fbf9fa17c1a75b1a120fc60425cd6a20216c432f026
Status: Image is up to date for heroku/heroku:22-cnb
ERROR: failed to build: Invalid buildpack defined in project descriptor

If I use "latest" I get:

$ pack build myimg -B heroku/builder:22
22: Pulling from heroku/builder
Digest: sha256:c9f45adaaf3f91eb2f126b29c8e947f1375bae560a725511393b6a197cfe765d
Status: Image is up to date for heroku/builder:22
22-cnb: Pulling from heroku/heroku
Digest: sha256:0c422f13416d589a199e3fbf9fa17c1a75b1a120fc60425cd6a20216c432f026
Status: Image is up to date for heroku/heroku:22-cnb
ERROR: failed to build: downloading buildpack: error reading heroku/nodejs@latest: invalid locator: InvalidLocator

What am I missing?

[edmorley]

What am I missing?

Oh strange! I know there are some heuristics around interpreting a plain URI identifier (ie: to work out if it's a Docker Hub reference, or say a CNB registry reference; xref https://buildpacks.io/docs/app-developer-guide/specify-buildpacks/#uri-examples), perhaps there is a bug around that and the default of latest? I personally haven't tried defining buildpacks in project.toml yet (we're still very early stages in the overall CNB / CNB tooling journey), so maybe the best place to ask for now might be https://github.com/buildpacks/pack/issues or https://slack.cncf.io/ (in one of the #buildpacks / #buildpacks-* channels).

but heroku buildpacks:set feels far more user friendly 🙂

I agree it is seemingly more friendly at first glance, but it actually does cause quite a few issues that likely do not surface for experienced/diligent users (such as yourself :-)) but often trip beginners up - such as:

• the classic footgun of "buildpacks:set" clobbering existing buildpacks
• users not realising buildpacks have an order, or knowing about --index to buildpacks:set or buildpacks:add
• users updating the buildpacks in app.json after an app was first created (rather than running buildpacks:{add,set}), and not knowing why the app doesn't run the new buildpacks. (Particularly for Review Apps, since the buildpacks in app.json are used for the first build, but not for subsequent)
• different apps in their account running the same codebase but showing different behaviour and users not knowing why (either because of them having completely different buildpacks, or even more subtle cases of one app being pinned to an older version of an official buildpack compared to another)

In addition, with CNBs we finally have a much better story for running a build locally that matches production in the form of pack build etc. However, given that heroku buildpacks:set stores state remotely in Heroku's API/database for a specific app, how would pack build know what buildpacks to run, unless that's defined in the codebase?

[ipmb]

In case anyone else comes across this, a uri is required, not an id for it to work without a version:

[[io.buildpacks.group]]
uri = "urn:cnb:builder:heroku/nodejs"

[edmorley]

In case anyone else comes across this, a uri is required, not an id for it to work without a version:

@ipmb That seems like a bug - would you mind reporting it upstream? https://github.com/buildpacks/pack/issues

[ipmb]

✅ buildpacks/pack#1862

[edmorley]

Thank you!

[jose-fully-ported]

As such, I think having a project.toml will be fairly common for anything but beginner apps. And beginner apps don't tend to use bin/{pre,post}_compile.

I've long used bin/{pre,post}_compile files in heroku apps to install binaries I needed in apps at runtime that were annoying to install otherwise. I don't think its something that is "optional" by any means, but maybe Heroku has telemetry on it's usage?

[jose-fully-ported]

Would it be best for someone (me, you, someone) to publishpre-compile and post-compile buildpacks that detects based on the existence of the respective files? I don't know if there is a way to control order here and wrap all other buildpacks other than specifying it in a particular order within project.toml (or --buildpack flag if using pack).

[edmorley]

The issue here is that bin/{pre_post}_compile are an entirely proprietary Heroku classic Python buildpack thing - they are not used by any other Heroku language buildpack, nor are they used by any other non-Heroku Python ecosystem tool.

IMO one of the big wins of the migration from classic buildpacks to CNBs (for all Heroku languages) is that of switching from proprietary features/concepts to open standards and modern best practices. For example:

• slugs -> OCI images
• closed source tooling -> open source tools that can be run locally to build the exact same image (pack build)
• app build configuration (eg stack, list of buildpacks) being stored in an external API -> taking more of an infrastructure as code approach (project.toml), so your local builds match production
• For Python specifically:
  • runtime.txt -> .python-version (coming soon)
  • bin/{pre_post}_compile -> upstream inline buildpack feature

For some of these transitions (for example runtime.txt), the Python CNB will support both the old and the new method - but will recommend the new method in docs and show a deprecation warning for the old approach in the build logs.

For some other differences between the classic Python buildpack and the CNB, the feature will be dropped and result in an error that explains how to migrate.

I haven't decided yet which approach to use for bin/{pre_post}_compile (hard error with migration advice, or deprecation warning) - however, I don't feel "support it as the preferred/recommended approach" (or replacing it with a custom pre-compile buildpack replacement) is even an option we should be considering as a best practice moving forwards.

Ultimately "run a custom command before/after one of my buildpacks" is not a Python specific feature - and should be something handled by the upstream CNB project (and already is - though there are likely UX improvements that could be made).

If you have suggestions for UX improvements to how the upstream inline buildpack feature work, I'd strongly recommend opening some issues or starting a discussion upstream:
https://github.com/buildpacks/community/discussions
https://github.com/buildpacks/rfcs

Ultimately end users being able to influence the design of buildpack APIs and tooling is another of the advantages of switching to an open standard - so please do take advantage of that! 😄

[ipmb]

It's not as pretty, but adding a project.toml like this now works and gives me a workaround for post_compile

[_]
schema-version = "0.2"
id = "io.buildpacks.my-app"

[[io.buildpacks.group]]
id = "heroku/nodejs"

[[io.buildpacks.group]]
id = "heroku/python"

[[io.buildpacks.group]]
id = "my-app/post-compile"

  [io.buildpacks.group.script]
  api = "0.9"
  inline = "bin/post_compile"

If your post_compile script is a shell script, this will get you a header that matches the others in the build output:

echo -e "\n\e[1;35m[post_compile]\e[0m"

[edmorley]

Thank you for the example - glad to see that works!

Using an inline table allows for reducing the boilerplate a bit more (depending on personal taste for TOML style):

[[io.buildpacks.group]]
id = "my-app/post-compile"
script = { api = "0.9", inline = "bin/post_compile" }

I'll open a PR against the upstream docs to make them use the inline table approach in the rake package example on:
https://buildpacks.io/docs/app-developer-guide/using-inline-buildpacks/