twu / skjold Goto Github PK

View Code? Open in Web Editor NEW

63.0 3.0 12.0 504 KB

Security audit Python project dependencies against security advisory databases.

Home Page: https://pypi.org/project/skjold/

License: MIT License

Makefile 1.09% Python 98.91%

pip poetry pipenv vulnerabilities vulnerability-detection github-security-advisories safety safety-db gemnasium pyup

skjold's People

Contributors

Stargazers

Watchers

Forkers

ghtyrant brondsem inksong chubbymaggie tbjers bangil0 borismih fabaff pombredanne st0rmnt85 aigeneratedusername pjacobsen-utilimarc

skjold's Issues

How do you call audit programatically?

I need to skip the cli and call audit_ directly from my package. Not sure how I am supposed to do that since you use click decorators. Any help would be appreciated!

Currently what I am trying:
pyproject.toml

[tool.skjold]
    sources = ["gemnasium"]
    cache_dir = "./tmp/skjold_cache"

from skjold.cli import audit_

audit_(["-r", "-o", "cli", "-i=", "-s", "gemnasium", "Pipfile.lock"])

Yet it fails since in pyproject.toml I have changed the location of cache:

Traceback (most recent call last):
..........
  File "/home/xxx/Documents/Projects/yyy/projects/ppp/lib/python/cli/commands/check_packages.py", line 19, in check_packages
    audit_([
  File "/home/xxx/Documents/Projects/yyy/projects/ppp/.virtualenv/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/xxx/Documents/Projects/yyy/projects/ppp/.virtualenv/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/xxx/Documents/Projects/yyy/projects/ppp/.virtualenv/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/xxx/Documents/Projects/yyy/projects/ppp/.virtualenv/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/xxx/Documents/Projects/yyy/projects/ppp/.virtualenv/lib/python3.9/site-packages/click/decorators.py", line 84, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/home/xxx/Documents/Projects/yyy/projects/ppp/.virtualenv/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/xxx/Documents/Projects/yyy/projects/ppp/.virtualenv/lib/python3.9/site-packages/skjold/cli.py", line 184, in audit_
    findings = audit(config, packages, ignore=ignore)
  File "/home/xxx/Documents/Projects/yyy/projects/ppp/.virtualenv/lib/python3.9/site-packages/skjold/tasks.py", line 221, in audit
    if source.has_security_advisory_for(package_name):
  File "/home/xxx/Documents/Projects/yyy/projects/ppp/.virtualenv/lib/python3.9/site-packages/skjold/sources/gemnasium.py", line 130, in has_security_advisory_for
    return package_name.strip().lower() in self.advisories.keys()
  File "/home/xxx/Documents/Projects/yyy/projects/ppp/.virtualenv/lib/python3.9/site-packages/skjold/models.py", line 101, in advisories
    self.update()
  File "/home/xxx/Documents/Projects/yyy/projects/ppp/.virtualenv/lib/python3.9/site-packages/skjold/sources/gemnasium.py", line 126, in update
    with open(self.path, "wb") as fh:
FileNotFoundError: [Errno 2] No such file or directory: '.skjold_cache/gemnasium.cache'

Looks like if I try to call audit_ directly it looses all configuration settings.

Latest OSV schema update breaks `pypa` and `osv` sources.

As of last week pypa and osv stopped working due to some breaking changes/updates in the OSV schema. See Open Source Vulnerability Version Format 0.8 (August 24, 2021).

2021-08-17 Support multiple packages per entry by moving packages, ecosystem_specific and database_specific into affected. The affected field is intentionally named differently to the previous affects field to make migration easier. Also use "events" containing single versions to represent affected version ranges instead.

Inconsequent ignoring

Let's take a look at this example:

% echo "py==1.11.0" | skjold audit -s pyup -s gemnasium -
Warning: No advisory sources configured!

py==1.11.0 (<=1.11.0) via gemnasium as CVE-2022-42969 found in <stdin>

Regular expression Denial of Service. The py library through 1.11.0 for Python
allows remote attackers to conduct a ReDoS (Regular expression Denial of
Service) attack via a Subversion repository with crafted info data, because the
InfoSvnCommand argument is mishandled.
https://nvd.nist.gov/vuln/detail/CVE-2022-42969

https://nvd.nist.gov/vuln/detail/CVE-2022-42969
https://pypi.org/project/py
https://github.com/pytest-dev/py/blob/cb87a83960523a2367d0f19226a73aed4ce4291d/py/_path/svnurl.py#L316
https://github.com/pytest-dev/py/issues/287
--

py==1.11.0 (<=1.11.0) via pyup as pyup.io-51457 found in <stdin>

Py throughout 1.11.0 allows remote attackers to conduct a ReDoS (Regular
expression Denial of Service) attack via a Subversion repository with crafted
info data, because the InfoSvnCommand argument is mishandled.
https://github.com/pytest-dev/py/issues/287
https://pyup.io/vulnerabilities/CVE-2022-42969/51457/

--
Found 1 vulnerable package(s)!

Both pyup and gemnasium finds this package as a vulnerability. However when using pyup as a source, it's identified as pyup.io-51457, not as CVE-2022-42969, like it is when using gemnasium. Thus, if you want to ignore this finding, you will have to run skjold ignore py CVE-2022-42969 and skjold ignore py pyup.io-51457 to ignore both of these.

I'm guessing there's just some error (or inconsistency) in how the data from pyup is parsed, as the raw data contains the CVE.

Changing this behaviour can of course break some existing ignore files, but would still be nice if it would work intuitively and following the examples on how to ignore the finding.

False Positive for Patched pyyaml From `osv` Source

Skjold flags pyyaml==5.4.1, but the vulnerability exists at pyyaml<5.4.

skjold........................................................................Failed
- hook id: skjold
- exit code: 1


pyyaml==5.4.1 (<5.4) via osv as PYSEC-2021-142

A vulnerability was discovered in the PyYAML library in versions before 5.4,
where it is susceptible to arbitrary code execution when it processes untrusted
YAML files through the full_load method or with the FullLoader loader.
Applications that use the library to process untrusted input may be vulnerable
to this flaw. This flaw allows an attacker to execute arbitrary code on the
system by abusing the python/object/new constructor. This flaw is due to an
incomplete fix for CVE-2020-1747.
https://bugzilla.redhat.com/show_bug.cgi?id=1860466

https://bugzilla.redhat.com/show_bug.cgi?id=1860466
-- 
Found 1 vulnerable package(s)!

Invalid Specifier on Gemnasium ranges

In an environment with dbt-core and using the gemnasium database I get the error:

packaging.specifiers.InvalidSpecifier: Invalid specifier: '>=1.6.0 <1.6.13'

It appears related to this alert:
GHSA-p72q-h37j-3hq7

Which makes it into the gemnasium database here

https://gitlab.com/gitlab-org/security-products/gemnasium-db/-/blob/master/pypi/dbt-core/GHSA-p72q-h37j-3hq7.yml

It seems like skjold is expected commas between range specifiers, but gemnasium uses a space

Display helpful message if Github Token is not found/set.

Present a more helpful message to the user if SKJOLD_GITHUB_TOKEN was not found and the user is trying to use the github source.

It looks like you're trying to use the github source without providing an access token. Skjold needs a token to access the Github GraphQL API. You can generate a token at https://github.com/settings/tokens without giving it any permissions and make it available to skjold by setting SKJOLD_GITHUB_TOKEN in your environment.

Enable Python `"3.10"` in workflows.

Invalid specifier error

Hi @twu !! I got the bellow error. Can I help to fix this?

specifiers.SpecifierSet(f"=={x}", prereleases=True) File "/home/bernardo.abreu/.cache/pre-commit/repow4ikj4am/py_env-python3/lib/python3.10/site-packages/packaging/specifiers.py", line 700, in __init__ parsed.add(Specifier(specifier)) File "/home/bernardo.abreu/.cache/pre-commit/repow4ikj4am/py_env-python3/lib/python3.10/site-packages/packaging/specifiers.py", line 234, in __init__ raise InvalidSpecifier(f"Invalid specifier: '{spec}'") packaging.specifiers.InvalidSpecifier: Invalid specifier: '==0.7.1.fix1'

Wrong option name in documentation

From section "Usage":

$ pip freeze | skjold -v audit --source gemnasium -

This command outputs: Error: no such option: --source Did you mean --sources?

Please bump to packaging major 22 if possible or loosen spec up

skjold/pyproject.toml

Line 48 in 39bf17a

packaging = "^21.0"

Error parsing a github source, on 0.4.0

With skjold 0.4.0 and a requirements.txt file containing waitress==1.4.4 as a package, the following error occurs

Traceback (most recent call last):
  File "/var/local/env-sfpy-py37/lib/python3.7/site-packages/packaging/specifiers.py", line 652, in __init__
    parsed.add(Specifier(specifier))
  File "/var/local/env-sfpy-py37/lib/python3.7/site-packages/packaging/specifiers.py", line 105, in __init__
    raise InvalidSpecifier(f"Invalid specifier: '{spec}'")
packaging.specifiers.InvalidSpecifier: Invalid specifier: '= 1.4.2'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/local/env-sfpy-py37/bin/skjold", line 8, in <module>
    sys.exit(cli())
  File "/var/local/env-sfpy-py37/lib/python3.7/site-packages/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "/var/local/env-sfpy-py37/lib/python3.7/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "/var/local/env-sfpy-py37/lib/python3.7/site-packages/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/var/local/env-sfpy-py37/lib/python3.7/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/var/local/env-sfpy-py37/lib/python3.7/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/var/local/env-sfpy-py37/lib/python3.7/site-packages/click/decorators.py", line 84, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/var/local/env-sfpy-py37/lib/python3.7/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/var/local/env-sfpy-py37/lib/python3.7/site-packages/skjold/cli.py", line 184, in audit_
    findings = audit(config, packages, ignore=ignore)
  File "/var/local/env-sfpy-py37/lib/python3.7/site-packages/skjold/tasks.py", line 223, in audit
    package_name, package_version
  File "/var/local/env-sfpy-py37/lib/python3.7/site-packages/skjold/sources/github.py", line 196, in is_vulnerable_package
    if candidate.is_affected(package_version):
  File "/var/local/env-sfpy-py37/lib/python3.7/site-packages/skjold/sources/github.py", line 71, in is_affected
    return version in self.vulnerable_version_range
  File "/var/local/env-sfpy-py37/lib/python3.7/site-packages/skjold/sources/github.py", line 62, in vulnerable_version_range
    self._json["node"]["vulnerableVersionRange"], prereleases=True
  File "/var/local/env-sfpy-py37/lib/python3.7/site-packages/packaging/specifiers.py", line 654, in __init__
    parsed.add(LegacySpecifier(specifier))
  File "/var/local/env-sfpy-py37/lib/python3.7/site-packages/packaging/specifiers.py", line 271, in __init__
    super().__init__(spec, prereleases)
  File "/var/local/env-sfpy-py37/lib/python3.7/site-packages/packaging/specifiers.py", line 105, in __init__
    raise InvalidSpecifier(f"Invalid specifier: '{spec}'")
packaging.specifiers.InvalidSpecifier: Invalid specifier: '= 1.4.2'

With skjold 0.3.2 it runs ok. I believe this github vulnerability is the one that it is erroring on: GHSA-73m2-3pwg-5fgc

click >= 8

Hi there!

Does skjold really need click version >= 8?
Looks like it breaks celery with its hardcoded click == 7.1.2. That's celery's problem, of course, but I just wanted to know if 7.1.2 would be fine for skjold too.

Migrate to `packaging` or `semver` packages.

Summary: poetry-semver looks like it wasn't updated in a while. poetry is now using the vendor-ed version in poetry-code and its probably best to migrate to something that is more widely used and maintained.

How will it work?:

For now the two possible options are:

Use semver or semver3 (See API Reference) and .match (See API Reference).
Use packaging. See Specifiers - packaging and packaging/test_specifiers.py at main · pypa/packaging · GitHub.

Links:

Running against all sources by default?

The readme's Introduction section says this about sources:

Unless configured explicitly skjold will run the given packages against all of them

However if I run skjold audit with a requirements file it says:

Error: Please specify or configure at least one advisory source.

(by the way, this tool seems great overall. I'll have a few other issues to report but this tool seems to do what nothing else offers. Thanks!)

Pre-commit hook fails if multiple lock or requirements files are modified at same time

If you in your repo have multiple files that should be analyzed by skjold and use it as a pre-commit hook, the hook fails if you modify multiple lock/requirements files in the same commit.

The reason to the problem is that pre-commit tries to invoke skjold audit with multiple files as arguments at once and the skjold audit command only accepts one file at a time.

A minimal POC can be obtained by just creating an empty git repo (mkdir skjold-poc; cd skjold-poc; git init), enabling pre-commit (pre-commit install), adding a .pre-commit-config.yaml containing

repos:
  - repo: https://github.com/twu/skjold
    rev: v0.4.1
    hooks:
    - id: skjold
      verbose: true

and then creating two (or more) requirements files (touch requirements.txt requirements-dev.txt) and then running git add .

If you then try to commit with git commit -m "Adding requirements files" you will be greeted with an error saying:
Error: Got unexpected extra argument (requirements.txt). If you add even more different lock files or requirements-something.txt files the message will be expanded with more extra arguments.

It's probably not that common people would have both poetry.lock and Pipfile.lock files, but possible people want to have multiple requirements.txt files (one for dev and one for prod) or multiple lock files in different directories of the same repo and get them all analyzed. It's of course possible to somewhat well avoid the problem by modifying only one lock file at a time, but the same issue also affects pre-commit run --all-files that will scan all the matched files regardless of if they've been changed or not.

pypa audits raise ScannerError: mapping values are not allowed here

pypa audits started failing in the past day or so. Here's an example:

$ rm -rf .skjold_cache/
$ echo 'bottle==0.12.23' | skjold audit -s pypa -
Warning: No 'pyproject.toml' found!
Traceback (most recent call last):
  File "../env/bin/skjold", line 8, in <module>
    sys.exit(cli())
  File "../env/lib/python3.7/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "../env/lib/python3.7/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "../env/lib/python3.7/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "../env/lib/python3.7/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "../env/lib/python3.7/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "../env/lib/python3.7/site-packages/click/decorators.py", line 84, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "../env/lib/python3.7/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "../env/lib/python3.7/site-packages/skjold/cli.py", line 184, in audit_
    findings = audit(config, packages, ignore=ignore)
  File "../env/lib/python3.7/site-packages/skjold/tasks.py", line 225, in audit
    if source.has_security_advisory_for(dependency):
  File "../env/lib/python3.7/site-packages/skjold/sources/pypa.py", line 65, in has_security_advisory_for
    return dependency.canonical_name in self.advisories.keys()
  File "../env/lib/python3.7/site-packages/skjold/core.py", line 123, in advisories
    self.populate_from_cache()
  File "../env/lib/python3.7/site-packages/skjold/sources/pypa.py", line 43, in populate_from_cache
    doc = yaml.load(obj_fh, Loader=yaml.SafeLoader)
  File "../env/lib/python3.7/site-packages/yaml/__init__.py", line 81, in load
    return loader.get_single_data()
  File "../env/lib/python3.7/site-packages/yaml/constructor.py", line 49, in get_single_data
    node = self.get_single_node()
  File "../env/lib/python3.7/site-packages/yaml/composer.py", line 36, in get_single_node
    document = self.compose_document()
  File "../env/lib/python3.7/site-packages/yaml/composer.py", line 55, in compose_document
    node = self.compose_node(None, None)
  File "../env/lib/python3.7/site-packages/yaml/composer.py", line 84, in compose_node
    node = self.compose_mapping_node(anchor)
  File "../env/lib/python3.7/site-packages/yaml/composer.py", line 127, in compose_mapping_node
    while not self.check_event(MappingEndEvent):
  File "../env/lib/python3.7/site-packages/yaml/parser.py", line 98, in check_event
    self.current_event = self.state()
  File "../env/lib/python3.7/site-packages/yaml/parser.py", line 428, in parse_block_mapping_key
    if self.check_token(KeyToken):
  File "../env/lib/python3.7/site-packages/yaml/scanner.py", line 116, in check_token
    self.fetch_more_tokens()
  File "../env/lib/python3.7/site-packages/yaml/scanner.py", line 223, in fetch_more_tokens
    return self.fetch_value()
  File "../env/lib/python3.7/site-packages/yaml/scanner.py", line 579, in fetch_value
    self.get_mark())
yaml.scanner.ScannerError: mapping values are not allowed here
  in ".skjold_cache/pypa.cache", line 2, column 98

Latest schema update breaks `pypa` source.

Excerpt from pyproject.toml:

  [tool.skjold]
  sources = ["pypa"]

Excerpt from pre-commit output when running git commit:

skjold........................................................................Failed
- hook id: skjold
- exit code: 1

Traceback (most recent call last):
  File "~/.cache/pre-commit/repo0vl87024/py_env-python3/bin/skjold", line 8, in <module>
    sys.exit(cli())
  File "~/.cache/pre-commit/repo0vl87024/py_env-python3/lib/python3.9/site-packages/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "~/.cache/pre-commit/repo0vl87024/py_env-python3/lib/python3.9/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "~/.cache/pre-commit/repo0vl87024/py_env-python3/lib/python3.9/site-packages/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "~/.cache/pre-commit/repo0vl87024/py_env-python3/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "~/.cache/pre-commit/repo0vl87024/py_env-python3/lib/python3.9/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "~/.cache/pre-commit/repo0vl87024/py_env-python3/lib/python3.9/site-packages/click/decorators.py", line 84, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "~/.cache/pre-commit/repo0vl87024/py_env-python3/lib/python3.9/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "~/.cache/pre-commit/repo0vl87024/py_env-python3/lib/python3.9/site-packages/skjold/cli.py", line 184, in audit_
    findings = audit(config, packages, ignore=ignore)
  File "~/.cache/pre-commit/repo0vl87024/py_env-python3/lib/python3.9/site-packages/skjold/tasks.py", line 221, in audit
    if source.has_security_advisory_for(package_name):
  File "~/.cache/pre-commit/repo0vl87024/py_env-python3/lib/python3.9/site-packages/skjold/sources/pypa.py", line 59, in has_security_advisory_for
    return package_name.strip().lower() in self.advisories.keys()
  File "~/.cache/pre-commit/repo0vl87024/py_env-python3/lib/python3.9/site-packages/skjold/models.py", line 104, in advisories
    self.populate_from_cache()
  File "~/.cache/pre-commit/repo0vl87024/py_env-python3/lib/python3.9/site-packages/skjold/sources/pypa.py", line 40, in populate_from_cache
    self._advisories[advisory.package_name.lower()].append(advisory)
  File "~/.cache/pre-commit/repo0vl87024/py_env-python3/lib/python3.9/site-packages/skjold/sources/osv.py", line 80, in package_name
    return str(self._json["package"]["name"]).strip()
KeyError: 'package'

Drop Python `3.6.x` support.

Python 3.6 is EOL since Dec. 2021.

Allow ignoring CVEs

Motivation:
One might need a way to ignore a vulnerability

temporarily
until a migration to a new library would take place
so that safety check does not fail every day until you migrate
permanently
for example, if you faced a CVE like this: https://nvd.nist.gov/vuln/detail/CVE-2020-28463

feat: set cache directory via environment variable

What

Allow to define a cache path in any way a user wants: via "pyproject.toml" or via an environment variable.
Environment variable name proposal: SKJOLD_CACHE_PATH

It will be required to document which setting takes higher priority during config merging: env config or file config. E.g. Removate uses env variables if both env and file setting were set.

Why

Simplify CI/CD setup. A developer will not need to modify a pyproject.toml file in every repository one by one.

For example, I have dozens of repositories. Every has its own pyproject.toml file. Environment variables in CI/CD allow me easily share common cache locations for different tools.

How other tools do

If you will take a look at Black, Mypy, Poetry, Renovate, then you will see that all tools allow to set a cache path via an environment variable.

UPDATE

I found a workaround. I can use --configuration-file FILE option with a path to shared file. I share common configuration files across many repositories via a single Git submodule.
This way I may avoid modifying "pyproject.toml" files in all repositories.

`Warning: No advisory sources configured!` must not be raised if advisory sources were passed via CLI

Reproducible sample

SKJOLD_GITHUB_API_TOKEN=<YOUR_TOKEN> skjold -v audit poetry.lock -s pyup -s gemnasium -s pypa -s osv -s github

Versions

skjold, version 0.5.1

Related source code

skjold/src/skjold/tasks.py

Line 60 in 1de5a66

"Warning: No advisory sources configured!", err=True, fg="yellow"

Possible fix

Check CLI arguments before raising the warning.

Newly introduced `$meta` field breaks `pyup` source.

Newly introduced $meta field breaks pyup source.

src/skjold/models.py:105: in advisories
    self.populate_from_cache()
src/skjold/sources/pyup.py:94: in populate_from_cache
    obj = PyUpSecurityAdvisory.using(package_name, advisory)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

cls = <class 'skjold.sources.pyup.PyUpSecurityAdvisory'>, name = '$meta', json_ = 'advisory'

    @classmethod
    def using(cls, name: str, json_: dict) -> "PyUpSecurityAdvisory":
        obj = cls()
        obj._json = json_
>       obj._json["name"] = name
E       TypeError: 'str' object does not support item assignment

src/skjold/sources/pyup.py:21: TypeError

 tests/test_pyup.py ⨯

pypa: TypeError: string indices must be integers, not 'str'

Not sure if this is a skjold or pypa data issue, but running this command gives the following error. Just started recently

echo 'aiohttp-session==2.12.0' | skjold audit -s pypa -f requirements.txt -
triggers

Traceback (most recent call last):
  File "py311venv/bin/skjold", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "py311venv/lib/python3.11/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "py311venv/lib/python3.11/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "py311venv/lib/python3.11/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "py311venv/lib/python3.11/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "py311venv/lib/python3.11/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "py311venv/lib/python3.11/site-packages/click/decorators.py", line 84, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "py311venv/lib/python3.11/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "py311venv/lib/python3.11/site-packages/skjold/cli.py", line 186, in audit_
    findings = audit(config, packages, ignore=ignore)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "py311venv/lib/python3.11/site-packages/skjold/tasks.py", line 221, in audit
    if source.has_security_advisory_for(dependency):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "py311venv/lib/python3.11/site-packages/skjold/sources/pypa.py", line 65, in has_security_advisory_for
    return dependency.canonical_name in self.advisories.keys()
                                        ^^^^^^^^^^^^^^^
  File "py311venv/lib/python3.11/site-packages/skjold/core.py", line 123, in advisories
    self.populate_from_cache()
  File "py311venv/lib/python3.11/site-packages/skjold/sources/pypa.py", line 44, in populate_from_cache
    advisories = OSVSecurityAdvisory.using(doc)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "py311venv/lib/python3.11/site-packages/skjold/sources/osv.py", line 60, in using
    "name": affected_package["package"]["name"].strip(),
            ~~~~~~~~~~~~~~~~^^^^^^^^^^^
TypeError: string indices must be integers, not 'str'

I tried to debug a bit and it looked like affected_package was just a string "package" which seemed weird.

error on LOW priority result

I'm not sure yet which package is an example of this, but at least one in my big requirements.txt file is LOW priority and causes this error:

  File "/Users/brondsem/tmp/py3venv/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/Users/brondsem/tmp/py3venv/lib/python3.6/site-packages/skjold/cli.py", line 182, in audit_
    report(config, results)
  File "/Users/brondsem/tmp/py3venv/lib/python3.6/site-packages/skjold/tasks.py", line 82, in report
    }[result["severity"]]
KeyError: 'LOW'

report-only and report_format not implemented ?

Hello,
Thanks for this program.
I do tried the report_only mode with report_format as json but cannot make it work, it seems the program just ignore it.
I check a bit into the code but i didn't found the part of the code that does make these parameter work, so i'm thinking that maybe an unimplemented feature ?

checking `pyspark` against `gemnasium` throws an exception

command to reproduce the issue: echo 'pyspark==3.1.2' | skjold -v audit --sources gemnasium -

Result:

Warning: No 'pyproject.toml' found!
sources: []
report_only: False
report_format: cli
verbose: True
cache_dir: .skjold_cache
cache_expires: 43200
ignore_file: .skjoldignore
Using .skjold_cache as cache location
Using .skjoldignore as ignore file
Checking 1 package(s).
Using ['gemnasium'] as source(s).
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/packaging/specifiers.py", line 634, in __init__
    parsed.add(Specifier(specifier))
  File "/usr/local/lib/python3.8/site-packages/packaging/specifiers.py", line 98, in __init__
    raise InvalidSpecifier(f"Invalid specifier: '{spec}'")
packaging.specifiers.InvalidSpecifier: Invalid specifier: '('

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/merge/.local/bin/skjold", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 84, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/merge/.local/lib/python3.8/site-packages/skjold/cli.py", line 184, in audit_
    findings = audit(config, packages, ignore=ignore)
  File "/merge/.local/lib/python3.8/site-packages/skjold/tasks.py", line 222, in audit
    is_vulnerable, advisories = source.is_vulnerable_package(
  File "/merge/.local/lib/python3.8/site-packages/skjold/sources/gemnasium.py", line 140, in is_vulnerable_package
    if candidate.is_affected(package_version):
  File "/merge/.local/lib/python3.8/site-packages/skjold/sources/gemnasium.py", line 81, in is_affected
    affected_versions = map(allows_, self.vulnerable_version_range)
  File "/merge/.local/lib/python3.8/site-packages/skjold/sources/gemnasium.py", line 66, in vulnerable_version_range
    return [
  File "/merge/.local/lib/python3.8/site-packages/skjold/sources/gemnasium.py", line 67, in <listcomp>
    specifiers.SpecifierSet(x, prereleases=True)
  File "/usr/local/lib/python3.8/site-packages/packaging/specifiers.py", line 636, in __init__
    parsed.add(LegacySpecifier(specifier))
  File "/usr/local/lib/python3.8/site-packages/packaging/specifiers.py", line 253, in __init__
    super().__init__(spec, prereleases)
  File "/usr/local/lib/python3.8/site-packages/packaging/specifiers.py", line 98, in __init__
    raise InvalidSpecifier(f"Invalid specifier: '{spec}'")
packaging.specifiers.InvalidSpecifier: Invalid specifier: '('

Pre-commit hooks only checks files in root of repo

The pattern for files in .pre-commit-hooks.yaml are set up to only check any poetry.lock, Pipfile.lock and requirements*.txt files in the root of the repo. However it's not really uncommon that you have such files elsewhere in the repo. I'd suggest changing the pattern so that it'll scan such files elsewhere as well.

If you agree on the idea, I can create a PR, or you can just update the line to be:
files: (^|/)(poetry\.lock|Pipfile\.lock|requirements.*\.txt)$

Links to pyup.io point to 404 page

Here's an example of an issue reported by skjold:

cryptography==38.0.1 (<39.0.0) via pyup as pyup.io-51159 found in poetry.lock

Cryptography 39.0.0 drops support for C library "LibreSSL" < 3.4, as these
versions are not receiving security support anymore.
https://pyup.io/pyup.io-51159

The link at the last line points to https://pyup.io/pyup.io-51159, however it should point to https://pyup.io/vulnerabilities/CVE-2021-41581/51159/

It's likely appending the id instead of the more_info_path to the domain when constructing the link.

More flexibility in requirements.txt format

If a requirements file has anything extra, this error is raised:

  File "/Users/brondsem/tmp/py3venv/lib/python3.6/site-packages/skjold/cli.py", line 169, in audit_
    packages = extract_package_list_from(config, file, file_format)
  File "/Users/brondsem/tmp/py3venv/lib/python3.6/site-packages/skjold/formats.py", line 80, in extract_package_list_from
    for package in reader_func(file):
  File "/Users/brondsem/tmp/py3venv/lib/python3.6/site-packages/skjold/formats.py", line 47, in read_requirements_txt_from
    package_name, package_version = line.strip().split(" ")[0].split("==")
ValueError: not enough values to unpack (expected 2, got 1)

The most common example that can cause this is # for comments.

I also have requirements.txt files with hashes specified and spanning multiple lines, like this:

beautifulsoup4==4.8.0 \
    --hash=sha256:05668158c7b85b791c5abde53e50265e16f98ad601c402ba44d70f96c4159612 \
    --hash=sha256:25288c9e176f354bf277c0a10aa96c782a6a18a17122dba2e8cec4a97e03343b \
    --hash=sha256:f040590be10520f2ea4c2ae8c3dae441c7cfff5308ec9d58a0ec0c1b8f81d469

There are other things that can be in a valid requiremens.txt file too, like extra pip options. https://pip.pypa.io/en/latest/reference/pip_install/#requirements-file-format

These can all be worked around by massaging the requirements file first and passing it as stdin something like: cat requirements.txt | sed 's/#.*//' | skjold audit -s gemnasium - But it would be very handy for skjold to handle any requirements.txt file that is valid for pip

Support `pypa/advisory-db` source.

Summary

I will start playing around with the specification today and see if I get something to work. I haven't found information on how final the current API and specification is so I consider all of this experimental and won't put it into master just yet. Personally I prefer the offline approach instead of asking an API for every installed package.

How will it work?

skjold will either query the osv.dev API directly or will fetch the master tarball from the pypa/advisory-db similarly to how the Gemansium source works (or both).

Intended outcome

Skjold supports fetching and checking data from the PyPA Advisory Database using -s pypa and/or -s osv.

Links

Ensure `skjold` works with `click` `7.x` and `8.x`.

The following works, however I'm not yet sure how stupid this approach is yet.

diff --git a/src/skjold/tasks.py b/src/skjold/tasks.py
index 8c8b30b..ce9da4d 100644
--- a/src/skjold/tasks.py
+++ b/src/skjold/tasks.py
@@ -17,7 +17,10 @@ def default_from_context(attr: str, cls: object) -> Type[click.Option]:
     class OptionDefaultFromContext(click.Option):
         def get_default(self, ctx: Any, call: bool = False) -> Any:
             self.default = getattr(ctx.find_object(cls), attr)
-            return super().get_default(ctx, call)
+            try:
+                return super().get_default(ctx, call)  # click>=8
+            except TypeError:
+                return super().get_default(ctx)  # click<8

     return OptionDefaultFromContext