dependabot / smoke-tests

A collection of manifest files for various package managers and is used to perform end-to-end tests for Dependabot.

This repo contains manifest files for various package managers and is used to perform end-to-end tests for Dependabot.

You're welcome to use this repo to test Dependabot functionality. However, we aren't accepting contributions at this time.

How does this work?

Dependabot CLI has the capability to generate and consume test files.

To generate a test, run dependabot update go_modules dependabot/smoke-tests -o my-test.yml. This will create my-test.yml containing the inputs you provided, plus some ignore conditions to make the test more reproducible, and also contain the outputs the CLI recorded.

To run the test, run dependabot test -f my-test.yml. This will run Dependabot with the inputs provided in the test, and assert the outputs match what was previously recorded.

Caching

To further aid in reproducibility, the CLI can instruct the Proxy to enable caching with the --cache tmp/cache option. Simply pass that along with a path to use for the cache during testing, and the Proxy will dump messages it received there.

If there are already cache files present, the Proxy will use them during the run instead of making a network call. Neat!

Checking failures

Sometimes a test will fail. To more easily check the difference, use the -o option to write the new outputs for an easy diff:

dependabot test -f input.yml -o output.yml
diff input.yml output.yml

Or if the file is checked into Git, just overwrite the existing file:

dependabot test -f test.yml -o test.yml

Workflows

This repo has several workflows used to manage the caches that are stored as build artifacts:

• Cache one
  • This allows us to run a single smoke test and cache the results.
• Cache all
  • This runs all of the smoke tests and cache the results, which is useful when a large change happens.
• Smoke
  • This test downloads the cache from the build artifacts before running each test in parallel. This was designed to run after each Pull Request.

To see the percentage of caching on each test, go to the Smoke tests summary view. If the test has low cache coverage then it is more likely to fail in the future. Rerun the Cache one workflow to recache it, or debug why it is uncachable.

Regenerating tests

Sometimes after a test has been uncached for a while, it will break because the dependencies have changed, and recaching won't fix it. Also some package-managers seem to get around the caching after a while.

In this case we will need to regenerate the failing test locally with dependabot test -f tests/smoke-bundler.yaml -o tests/smoke-bundler.yaml then push it up, and recache that test with the "Cache" workflows.

Where possible try to add additional ignore_conditions and allowed_updates so even when uncached the tests will not fail.

For convenience there's a script/run-all.sh which will regenerate all of the tests.

How to add new tests

To create a new test:

1. Add new manifest files, and push the commit to this repo in a branch
2. Run dependabot update $ecosystem dependabot/smoke-tests --directory $dir --commit $previous_commit_sha -o tests/smoke-$ecosystem-$testname.yaml substituting the appropriate values:
   • $ecosystem is the directory name in dependabot-core for the ecosystem
   • $dir is the relative path to the manifest files
   • $previous_commit_sha is the commit SHA that contains the manifest files under test
   • $testname is a descriptive name for the test
3. Commit the resulting test
4. Run the Cache-One Workflow to cache the test, so it won't break due to dependency changes before the PR merges
5. When merging, be sure to make a merge commit, don't squash or rebase! This will change the $previous_commit_sha.

The reason we do it this way is the commit under test is in the API calls in the output section. So we must pin the test to a commit or a change to this repo will break all the tests.

Also, the manifests must be pushed up before the test can be run, because Dependabot will attempt to fetch them.

If you are contributing from a fork, perform the steps above replacing dependabot/smoke-tests with your fork, and we can do the rest.

Secrets

To keep secrets out of test files, the CLI can use environment variables instead:

credentials:
  - type: npm_registry
    registry: https://npm.pkg.github.com
    token: $MY_TOKEN

ARM64 machines

Since we currently don't produce ARM images, it is recommended you use Codespaces to run and regenerate tests. It runs much faster than emulation!

smoke-tests's Issues

Add an explicit license.

At the moment, the closest to this is the README saying

You're welcome to use this repo to test Dependabot functionality.

But it would be nice if the licensing was less vague and ephemeral.

I ask because I'd like to duplicate the files here in a set of tests to run github's linguist on. Linguist can't run on submodules, and I'd like for the unit tests to not have to rely on an external repo, or multiple versions of it given the pinned versions that the smoke tests here guarentee to be runnable with dependabot.

Can external contributors re-generate the tests?

I ran ./dependabot test -f tests/smoke-npm-group-multidir.yaml -o tests/smoke-npm-group-multidir.yaml but that failed with various auth errors

The following comment from the readme mentions that you do not accept contributions

You're welcome to use this repo to test Dependabot functionality. However, we aren't accepting contributions at this time.

But it is not necessarily true in all situations. For example of tests that don't need regeneration like #59

Could you please clarify who can regenerate these?

Move repo to the dependabot-fixtures org

We should consider moving this repo into https://github.com/dependabot-fixtures

No rush on this, as there's a chance we may decide to go the other way and move the fixtures into this org and then tag the repos with something like test / fixtures for easy identification... (part of me actually leans that way for simplification--then everything public that's "dependabot-related" is in this org)

Mostly I'm opening this issue as a placeholder reminder that it's inconsistent that we have some test fixtures in this org and some in the other org. I'm not aware of any current guidelines that result in "smoke tests go this org, but other fixtures in the other org".

Also, we shouldn't move this repo until we've thought through how we want to handle the test-fixture-related repos in this org, such as https://github.com/dependabot/vgotest, https://github.com/dependabot/dummy-packages, https://github.com/dependabot/monolog etc

Testability of the head commit on default branch

Hi; wanted to use this to test some dependabot functionality to confirm my use of the file_fetcher classes.

I forked it and cloned it, and ran a script to test using all the file_fetcher classes on the head commit, which seemed reasonable to assume should be in a valid state, because there wasn't anything abundantly visible warning that wouldn't be the case.

Assuming that, when the bundler file_fetcher errored on the bundler folder missing a gemspec (as the gemfile lock has the relative path), I started investigating other possible causes i.e. looking into the dependabot updater image's job classes and cli to see if there was any way that when it was run as a live service it would side step the Dependabot::PathDependenciesNotReachable error, believing that the head of the default branch here should be in a valid state.

After exhausting all those other areas, I came back to the smoke tests here which were all passing to see in the logs that the bundler smoke test was retrieving a gemspec; and I saw that the ref wasn't the current head ref -- which was the first point in time I had the inclination to go and check the smoke test configs themselves and see

smoke-tests/tests/smoke-bundler.yaml

Line 18 in 0e13b71

commit: 832e37c1a7a4ef89feb9dc7cfa06f62205191994

to realise they weren't targetting the head commit.

Whether or not the job configs in the smoke test requires that commit ref to be specified and thus targetting a previous state of the repository after the jobs config with a new ref is committed, is there any reason why the head of the default branch isn't also expected to pass the smoke tests, and be in a valid state?

If it's not required to be in a valid state, it'd be nice if there was a visually obvious warning near the top of the readme that the head commit isn't necessarily the "valid" snapshot reference, and pointing to the fact that the commits referred to in the smoke test job's configs is the actual source of truth for the "valid" state to test against.