conda-incubator / conda-recipe-manager Goto Github PK

Split from this rattler-build issue: prefix-dev/rattler-build#829

We need to add a fn -> file_name transformation. This happens under the source object and I think a few other paths.

Pixi is a modern alternative to a Makefile and natively integrates with the conda-ecosystem to install dependencies. I think it would work nicely for this project. You can even use the pyproject.toml support as of recently.

I can give it another go if you are up for it :)

Kicking the can on this, I can't currently figure this out.

These two bioconda integration tests seem to cause the GitHub runner to lock-up with a yellow spinning wheel. The tests are marked as failed, without completing the post-run stages. The timeout checks for rattler-build don't seem to trigger.

Here is an example run:
https://github.com/conda-incubator/conda-recipe-manager/actions/runs/8482951360/job/23243473204?pr=15

Work on these unit tests is still on-going in #15

These tests appear to work fine on my Mac locally.

A summary section like this (found in r-highr-feedstock):

  summary: Provides syntax highlighting for R source code. Currently it supports LaTeX and HTML
    output. Source code of other languages is supported via Andre Simon's highlight
    package (<http://www.andre-simon.de>).

Appears to be valid YAML. The conversion process (or more likely, the parser) turns the above string into:

  summary:
    - Provides syntax highlighting for R source code. Currently it supports LaTeX and HTML
    - "output. Source code of other languages is supported via Andre Simon's highlight"
    - package (<http://www.andre-simon.de>).

^ Which causes rattler-build to fail as the value must be a scalar type for summary.

The R series of packages on conda-forge (along with a few other packages) all use an environ["PREFIX"] JINJA variable in their /about/license_file field that causes the parser to fail.

A quick search shows that there are many recipes in our integration tests that fail from using environ["<STRING>"] in other places.

NOTE:

CEP-14 does seem to indicate the syntax might have changed in the new format:

  # the script environment. You can use Jinja to pass through environment variables
  # with the `env` key (`${{ env.get("MYVAR") }}`).
  env: {string: string}

Example

Here is an example from the integration test work (r-rann-feedstock):

$ convert conda_forge_recipes_01/r-rann-feedstock/recipe/meta.yaml 
EXCEPTION: An exception occurred while parsing the recipe file
while parsing a flow mapping
  in "<unicode string>", line 1, column 4:
    - {{ environ["PREFIX"] }}/lib/R/sha ... 
       ^
expected ',' or '}', but got '['
  in "<unicode string>", line 1, column 13:
    - {{ environ["PREFIX"] }}/lib/R/share/licens ... 
                ^
0 errors and 0 warnings were found.

The /build/script section is currently not being upgraded. In simple scenarios, this works out fine. In complex situations, we need to use the new Script object

This will be a pretty involved upgrade process that will need to be heavily conditionalized. From a brief search, there are already many variants AND we have to parse-out shell variable syntax in the existing script_env field.

From the conda-build element chat:

  build:
    - python                                 # [build_platform != target_platform]
    - cross-python_{{ target_platform }}     # [build_platform != target_platform]

is not quite corretly translated (only the first of the two elements makes it into the then part of the condition.

boto3-stubs-feedstock in the Anaconda Recipes test data is a good example of this (so is curl-feedstock). When a multi-output recipe has a requirements section at the top level, rattler will throw an error. This is not supported in the V1 format.

I think in this scenario the top-level requirements applies to all the multi-output requirements sections but I would have to double check on the expected behavior.

A few things:

• It would be beneficial if the conversion script could be run across multiple projects at once
• As part of this, the script should be able to output to a JSON format for easy parsing by other scripts
• The JSON output should include statistics about runtime and errors encountered so bulk operations can be easily understood.

cctools-ld64-feedstock contains a recipe that has a selector on a variable. In this particular case, the selector is unnecessary and is selectively used anyways.

{% set native_compiler_subdir = 'linux-64' %}  # [linux]

We should add support for a warning as a human will need to determine the intent/danger of doing this. In (hopefully) most cases, we can ignore the selector and just keep the variables. This will at least allow a human to correct the issue.

At @kenodegard's suggestion, we should run this code analysis tool to assist in finding opportunities of improvement in this project.

Patching:

foo:
  - bar

With {"op": "replace", "path": "/foo", "value": None} will result in:

foo:
  - null

Instead of foo:

I had this idea while writing #2

At some point in the near future, we should consider "eating our own dog food" and use rattler-build to package this project.

For a period of time, we might want to consider having two build-recipe workflows on PRs/merges; 1 for conda-build and one for rattler-build

This will enable much faster testing if we can easily convert many repos at once.

We should also be able to better format the error output in large test runs. Maybe we should output to JSON with a flag?

From Wolf, over chat:

I would also say that if a package did not specify an imports in the tests, we don't need to deal with the pip check either
Although I did recently find out that the imports can also be used to test Perl imports :'(

From r-highr-feedstock:

Error:   × Parsing: failed to parse SPDX license: unknown term
  │ See <https://spdx.org/licenses> for the list of valid licenses.
  │ Use 'LicenseRef-<MyLicense>' if you are using a custom license.
    ╭─[75:12]
 74 │ about:
 75 │   license: GPL-3
    ·            ──┬──
    ·              ╰── here
 76 │   summary:
    ╰────

Ideas:

• We should throw a warning when a license is not recognized in the conversion phase
• We could potentially auto-correct the issue(?)

As discussed in this PR: #55 we can take the JSON data SPDX publishes in combination with Python's difflib.get_close_matches and see if we can get better license matches without maintaining our own look-up table.

skip: ${{ true if linux }}

Should be:

skip: linux

This will likely need more complex logic that parses-out selectors that have boolean expressions. Listed skip attributes are logically OR'd together.

This comes out of a logical misinterpretation of the CEP-14:
https://github.com/conda-incubator/ceps/blob/main/cep-14.md#python-test-element

pip_check defaults to true IFF the package is a Python package. For non-Python packages, this is an error.

TODO: We need a way to classify recipes as building "python" packages that is reliable and consistent.

@wolfv brought this up in conversation, and it would be consistent with some of the other tools we have at Anaconda.

We should move to use conda-recipe-manager as the base for current and future tools. Although it is more to type, it would lend itself to providing a better user experience, aggregating all the commands into 1 namespace.

Example:

conda-recipe-manager convert <args>
conda-recipe-manager <tool> <args>

Wolf suggested that we can use the Pydantic schema found in this repo to validate our recipe output.

This should be used in unit testing and/or return an error in the conversion process.

There are many changes to the build/ section that are not currently supported by the conversion script. We need to perform a thorough audit of this section and support all the conversions.

In the not too distant future, I would like to start developing some kind of integration testing with rattler-build.

Here are some starting goals:

• On PRs/merges, take a series of recipes that we know that we can successfully convert from the v0 -> v1 recipe format.
• Once converted, run these recipes through the latest release of rattler-build
• Ideally, we may also want the ability to trigger these integration tests independently of a PR/merge

This should give us some confidence that we work well with our target build platform.

protobuf-feedstock contains a recipe file that has an edge case that is not being handled properly.

There are a few commands listed that contain selectors. When the selectors get patched to the new if/then blocks, the command string gets mangled. The : in the command statement appears to be parsed as YAML at some point and that causes the conversion script to emit the wrong output.

Problematic lines:

    - python -c "from google.protobuf.internal import api_implementation; assert api_implementation.Type() == 'cpp'"  # [unix and python_impl != "pypy"]
    - python -c "from google.protobuf.internal import api_implementation; assert api_implementation.Type() == 'python'"  # [win or (unix and python_impl == "pypy")]

Debug tree:

      |- <Collection Node>
        |- if
          |- unix and python_impl != "pypy"
        |- then
          |- python -c "from google.protobuf.internal import api_implementation; assert api_implementation.Type()
          |- == 'cpp'"
      |- <Collection Node>
        |- if
          |- win or (unix and python_impl == "pypy")
        |- then
          |- python -c "from google.protobuf.internal import api_implementation; assert api_implementation.Type()
          |- == 'python'"

We need to ensure the about/ section is being properly upgraded from v0 -> v1

The about section is only checked at the top-level currently. We need to add support for multi-output recipes that contain multiple about sections.

We are missing a transformation of the merge_build_host field from CEP-14:

  # merge the build and host environments (used in many R packages on Windows)
  # was `merge_build_host`
  merge_build_and_host_envs: bool (defaults to false)

rattler-build error:

Error:   × Parsing: invalid field `merge_build_host`.
    ╭─[50:3]
 49 │   noarch: generic
 50 │   merge_build_host: $${{ true if win }}
    ·   ────────┬───────
    ·           ╰── here
 51 │   rpaths:
    ╰────

I've never had a project with those before, but this one should look professional and I want to learn.

When conda-recipe-manager-feedstock exists (see this PR) we should consider redacting the recipe file in this project.

I don't want to have to maintain two sources of truth. This comes with some caveats:

• The GHAs that call out to conda-build and rattler-build will need to be updated to pull from the feedstock?
• This might have a bootstrapping issue when it comes time to release a new version(?). If a dependency is added, you would need to accept that the builds will fail until a new version is fully released.

rattler-build will fail a dry-run build if environment variables referenced by env.get() are not present.

For our purposes, this is analogous to a false-positive and potentially obfuscates other test failures as this check appears to happen before some other validation checks.

We should find a way to mitigate this issue. At the very least, this issue affects all R recipe files.

Licenses we can't match to a known SPDX compliant license should be flagged as custom per the SPDX spec. This should make rattler-build happy with any remaining license files we can't automatically fix. Some may actually be custom licenses.

r-highr-feedstock contains a list of strings under /build/rpaths:

build:
  merge_build_host: True  # [win]
  # If this is a new build for the same version, increment the build number.
  number: 0
  # no skip
  noarch: generic

  # This is required to make R link correctly on Linux.
  rpaths:
    - lib/R/lib/
    - lib/

This does not appear in CEP-13 nor CEP-14 and rattler-build does not currently support this path. There is a new /build/dynamic_linking/rpaths but that is a different transformation which we recently added support for.

rattler-build error:

Error:   × Parsing: invalid field `rpaths`.
    ╭─[51:3]
 50 │   merge_build_host: $${{ true if win }}
 51 │   rpaths:
    ·   ───┬──
    ·      ╰── here
 52 │     - lib/R/lib/
    ╰────

After a brief discussion with @beeankha , we noticed that the new recipe format replaces outdated terminology for new inclusive language in a handful of places, primarily with allow/deny lists in recipe files.

I don't think the current conversion work addresses these changes yet, but we will need to in order to pass the schema checks in rattler-build.

We need to ensure that the extras section can be correctly upgraded from v0 -> v1

55 recipes in the Anaconda Recipes test data (mostly R packages) have conditionalized JINJA variables that do not translate into something rattler-build supports.

Here is what is commonly used:

{% set posix = 'm2-' if win else '' %}
{% set native = 'm2w64-' if win else '' %}

NOTE: Sometimes p and n are used as variable names instead.

Here is what is conda-recipe-manager converts that to:

context:
  posix: "'m2-' if win else ''"
  native: "'m2w64-' if win else ''"

Here's the full debug dump:

{
      "error": "Error:   \u00d7 Parsing: failed to parse match spec: ''m2-'' is not a valid package name. Package names can only contain 0-9, a-z, A-Z, -, _, or .",
      "recipe_count": 55,
      "recipes": [
        "ccache-feedstock/recipe/recipe.yaml",
        "cmake-no-system-feedstock/recipe/recipe.yaml",
        "double-conversion-feedstock/recipe/recipe.yaml",
        "r-ape-feedstock/recipe/recipe.yaml",
        "r-brglm-feedstock/recipe/recipe.yaml",
        "r-checkmate-feedstock/recipe/recipe.yaml",
        "r-coin-feedstock/recipe/recipe.yaml",
        "r-diffobj-feedstock/recipe/recipe.yaml",
        "r-fansi-feedstock/recipe/recipe.yaml",
        "r-fastmap-feedstock/recipe/recipe.yaml",
        "r-feather-feedstock/recipe/recipe.yaml",
        "r-fs-feedstock/recipe/recipe.yaml",
        "r-gert-feedstock/recipe/recipe.yaml",
        "r-git2r-feedstock/recipe/recipe.yaml",
        "r-glmnet-feedstock/recipe/recipe.yaml",
        "r-glue-feedstock/recipe/recipe.yaml",
        "r-gsl-feedstock/recipe/recipe.yaml",
        "r-hexbin-feedstock/recipe/recipe.yaml",
        "r-influencer-feedstock/recipe/recipe.yaml",
        "r-irkernel-feedstock/recipe/recipe.yaml",
        "r-irlba-feedstock/recipe/recipe.yaml",
        "r-isoband-feedstock/recipe/recipe.yaml",
        "r-kohonen-feedstock/recipe/recipe.yaml",
        "r-magrittr-feedstock/recipe/recipe.yaml",
        "r-mapproj-feedstock/recipe/recipe.yaml",
        "r-mime-feedstock/recipe/recipe.yaml",
        "r-mnormt-feedstock/recipe/recipe.yaml",
        "r-modelmetrics-feedstock/recipe/recipe.yaml",
        "r-pcapp-feedstock/recipe/recipe.yaml",
        "r-pki-feedstock/recipe/recipe.yaml",
        "r-ps-feedstock/recipe/recipe.yaml",
        "r-pspline-feedstock/recipe/recipe.yaml",
        "r-purrr-feedstock/recipe/recipe.yaml",
        "r-randomforest-feedstock/recipe/recipe.yaml",
        "r-readr-feedstock/recipe/recipe.yaml",
        "r-reticulate-feedstock/recipe/recipe.yaml",
        "r-rgl-feedstock/recipe/recipe.yaml",
        "r-rjsonio-feedstock/recipe/recipe.yaml",
        "r-rodbc-feedstock/recipe/recipe.yaml",
        "r-roxygen2-feedstock/recipe/recipe.yaml",
        "r-rserve-feedstock/recipe/recipe.yaml",
        "r-rsqlite-feedstock/recipe/recipe.yaml",
        "r-sf-feedstock/recipe/recipe.yaml",
        "r-sourcetools-feedstock/recipe/recipe.yaml",
        "r-survival-feedstock/recipe/recipe.yaml",
        "r-terra-feedstock/recipe/recipe.yaml",
        "r-tidygraph-feedstock/recipe/recipe.yaml",
        "r-urca-feedstock/recipe/recipe.yaml",
        "r-xfun-feedstock/recipe/recipe.yaml",
        "r-xml2-feedstock/recipe/recipe.yaml",
        "r-xts-feedstock/recipe/recipe.yaml",
        "r-yaml-feedstock/recipe/recipe.yaml",
        "r-zip-feedstock/recipe/recipe.yaml",
        "r-zoo-feedstock/recipe/recipe.yaml",
        "the_silver_searcher-feedstock/recipe/recipe.yaml"
      ]
}

While investigating some integration testing logs, I found that, on occasion, recipe files use capitalized field names, like Skip instead of skip.

This obviously breaks schema validation in rattler-build but presumably conda-build doesn't care.

We could easily correct for this BUT it does not appear this issue happens particularly often, so it is a lower priority issue.