[FEATURE] Support conda-lock files about jake

[bollwyvl]

@itamarst This would be great!

I am fairly certain it contains the same information as conda list.

Yes, this following are mostly the same:

• conda-lock [--file ...] (where file can be a couple of formats)
• conda list --explicit in an existing environment

A notable departure: conda-lock adds a few more comment lines which capture the relevant platform. It can also make use of mamba, which can be quite a bit snappier, especially for complex windows solves.

extra dependencies being installed and not scanned for vulnerable releases.

Right: the format of the list of packages from either tool is interesting, as it's not only a set of packages, but also a topological sorting of their install order, which can be exploited for caching schemes, resolving duplicate paths, etc.

As you're calling out conda-forge... it doesn't look like jake can even handle those packages yet... this issue might be mis-filed, but has some of the thoughts we came up with, as well as this discussion issue.

And, of note, one can install jake<1 from conda-forge... we're working on jake==1 but will have some back-filling to do for various new upstreams.

[DarthHater]

This is a cool idea, all for it. Stoked y'all dig the tool! @allenhsieh and I wrote this a few years ago because we really like jake the snake (just kidding, or maybe?!)

[madpah]

@itamarst - FYI we've added Conda support in jake when generating an SBOM:

conda list --explicit | jake sbom -t CONDA

We're looking next into supporting Conda and other input formats when running ddt and iq subcommands (which currently just reads what's installed in your current Python Environment).

FYI: @DarthHater , @bollwyvl

[bollwyvl]

Nice: I'm making some progress towards getting the conda-forge package up and running. Of note, during a self-test, I found some more exotic package names aren't very well supported:

https://conda.anaconda.org/conda-forge/linux-64/_libgcc_mutex-0.1-conda_forge.tar.bz2

See conda-forge/jake-feedstock#3 (comment)

[madpah]

Thanks @bollwyvl - will take a look at that package... - can you share a complete output that includes the above package from either conda list --explicit and/or conda list --json?

Thanks

[bollwyvl]

Sure, here are a bunch of widely-used lockfiles that get deployed thousands of time a day:

https://github.com/jupyterhub/repo2docker/blob/main/repo2docker/buildpacks/conda/environment.py-3.9.lock

[madpah]

@bollwyvl - I've done a bit more digging on this, and specifically the example you've provided above.

FYI - the parsing of Conda lock files is actually handled by a parent library to jake - cylconedx-python.

This project includes a parser for parsing conda lock files and already has a unit test specifically for the example you have above, which passes: https://github.com/CycloneDX/cyclonedx-python/blob/master/tests/test_utils_conda.py#L112

Am I missing something, or have you perhaps provided the incorrect example (before I go down a rabbit hole!)?

Thanks

[bollwyvl]

Yeah, as a downstream packager of these packages, I'm only just keeping up with the recent spate of package renamings and versions, and haven't evaluated whether lockfiles work in a while. Once these land, I'll have a better idea:

• conda-forge/staged-recipes#17533
• conda-forge/jake-feedstock#14

I've added the test case i tried in october to the latter, so we'll probably know more later this week.

[bollwyvl]

Well, we've shipped jake 1.4.0 on conda-forge, and it looks like it can successfully generate an sbom for its own environment... sorta.

Of note, there are a great number of packages that aren't python-related in conda(-forge), so blanket assuming a lot of stuff is in the pypi namespace is probably inaccurate, a la pkg:pypi/[email protected] (or more humorously, pkg:pypi/[email protected]), but that's probably more akin to the even further-upstream problem.

Meanwhile, when a package does correspond to one in pypi, but has a different name, there is a semi-authoritative mapping. I don't see any good examples here, but its common for things where the pypi name is a pun for the underlying c library, e.g. msgpack -> msgpack-python.

[madpah]

Thanks @bollwyvl - as ever, super insightful info and feedback. I'll ponder the two key points and see if there are any options we can employ to help.

[madpah]

On the point about generating an SBOM for it's own environment, can you share a little more, or can we consider #66 closed?