qiskit / qiskit-bot

Github bot to automate various aspects of qiskit development

qiskit-bot

This repo contains a wsgi app for running for GitHub automation of the Qiskit organization. It performs many functions to automate the workflow of the various aspect of managing the repositories in the Qiskit GitHub organization. For example, release automation to automatically generate a GitHub release with a full changelog from just a git tag and then generate a PR to bump the meta-repository based on that tag. The goal of this project is to minimize the number of manual actions needed as part of daily maintenance of qiskit.

As of right now the bot concentrates on release process automation. It handles 3 key aspects of that. First it generates release notes based on the git log and pull request tags. This generated changelog is added to the GitHub releases page. The next step is branch creation. If a project is configured to branch on minor releases, the bot will automatically create the branch. These steps are triggered by tag creation. This means with the bot all that is required for a Qiskit project maintainer is to push a git tag to GitHub. When coupled with pypi artifact (wheel and sdist) CI jobs pushing a tag becomes the only required manual step to push a release everything else will now be done automatically.

The other main feature the bot offers is to automatically leave a message on all new PRs when they are opened. This can be useful to leave a comment to set expectations for contributors but also be used to notify particular people to review the PR. The bot is configurable for each project so that subsets of GitHub users can be mentioned in this comment automatically based on the files changed in the PR.

In the future, the bot may be expanded to automate additional aspects of the GitHub workflow for the Qiskit community.

Configuration

Github side configuration

You will need to configure the bot to work with an GitHub user account. This account will need permission to create releases, push to the repo, etc.

There are two pieces of information needed from that account for the bot to work. First, you need the GitHub bot user to have an ssh key configured for the local user the daemon is run as. This is needed for git based commands like pushing branches. Then an access key is needed to be setup for the GitHub api access. This is used for API access to GitHub for the repositories.

The last piece of GitHub side configuration needed is to configure the webhook. You need to configure a webhook for each repository that uses the bot. Setup the webhook to send all necessary event types to the endpoint where the bot is running. Two things to remember is that make sure you send the webhook events to the /postreceive endpoint off of the server's address and that the Content type is set to application/json.

Per repo configuration

qiskit-bot gives projects some local configuration options that can be set in the repository. To set a local configuration file a file qiskit_bot.yaml must be created in the root of the git repository. If this file is present then qiskit-bot will read it before every action and adjust behavior based on its contents. Currently this configuration file is used to control two things: the changelog generation behavior, and whether the bot will leave comments on new pull requests when they're opened (and the exact behavior of that comment). An example of a fully populated configuration file is:

---
categories:
    "Changelog: Custom": Special category
    "Changelog: Custom 2": Less special category
    "Nothing": null
notifications:
    ".*":
        - @core-team
    qiskit/transpiler:
        - @user1
        - @user2
    qiskit/transpiler/passes:
        # You can escape usernames with ` so that they don't get
        # GitHub notifications.
        - `@user3`
        - `@user4`
always_notify: true
notification_prelude: |
    This is a custom prelude

    I include whitespace:

The details on each option are as follows:

• categories: This contains a nested mapping of GitHub labels to changelog sections. If specified at release time when qiskit-bot generates the changelog it will look at each merged PR in the release and if any have any matching labels that commit summary message will be put under the corresponding sections in the changelog used for the release page. If a value for any label is set to null this means that this label is counted as matching but will not be included in the generated changelog. By default any labels outside this set will not be included in the changelog but when the tools/generate_changelog.py script is run it flags any merged PRs for a pending release that don't have a matching label. By setting one or more labels to null these PRs will not show up in that script.

  If this field is not specified the following default values are used:

  "Changelog: Deprecation": Deprecated
  "Changelog: New Feature": Added
  "Changelog: API Change": Changed
  "Changelog: Removal": Removed
  "Changelog: Bugfix": Fixed
  "Changelog: None": null

• notifications: This contains a mapping of path regexes to a list of usernames to notify if an opened PR touches files that match a particular regex (as found by Python's stdlib re.search() function). For example if you set the path regex to ".*" this would match everything, but using a regex gives you control over exactly how and what matches a particular group. If a path matches, the listed usernames will be listed in the notification comment left by the bot on a new PR being opened. The specified usernames will receive a GitHub mention notification, so you can instead escape the usernames with ` so that instead the reviewer only knows those are relevant people to ping if necessary. The matching is additive, so if there is more than 1 match the users from all the matches will be listed in that comment. If this is not specified (and always_notify is not set) then no comment will be left by the bot when new PRs are opened.

• always_notify: If this is specified, a notification/comment is always left on PR opening even if there are no matching notification paths. In the case of no matching paths just the notification prelude is used.

• notification_prelude: If this is specified, the text used for this field will be used as the beginning of every notification comment. If this is not specified the following prelude is used:

  Thank you for opening a new pull request.
  
  Before your PR can be merged it will first need to pass continuous
  integration tests and be reviewed. Sometimes the review process can be slow,
  so please be patient.
  
  While you're waiting, please feel free to review other open PRs. While only a
  subset of people are authorized to approve pull requests for merging,
  everyone is encouraged to review open pull requests. Doing reviews helps
  reduce the burden on the core team and helps make the project's code better
  for everyone.
  

qiskit-bot's Issues

Notify first time contributor / community contributors to fulfill PR checklists

• ask to add release notes if needed
• ask to add test if needed
• qiskit-bot to tag Documentation and Changelog: None if the PR is only contain documentation changes

Changelog generation fails for versions >= 1.0.0

Since the release of qiskit 1.0.0 the changelog generation that runs as part of release process has been failing. Looking at the logs the git log command that's generated is not valid and getting confused by the concept of major versions (which to be fair for almost it's entire life weren't a thing). For example:

2024-05-16 20:02:49,238: 350 ERROR qiskit_bot.git [-] Failed to get git log
stdout:
b''
stderr:
b"fatal: ambiguous argument '1.1.0...0.0.0': unknown revision or path not in the working tree.\nUse '--' to separate paths from revisions, like this:\n'git <command> [<revision>...] -- [<file>...]'\n"
Traceback (most recent call last):
  File "/qiskit-bot/qiskit_bot/git.py", line 77, in get_git_log
    res = subprocess.run(['git', 'log', '--oneline', sha1],
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['git', 'log', '--oneline', '1.1.0...0.0.0']' returned non-zero exit status 128.
Process Process-3:
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.11/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/qiskit-bot/qiskit_bot/release_process.py", line 259, in _finish_release__changelog_process
    create_github_release(repo, log_string, version_number,
  File "/qiskit-bot/qiskit_bot/release_process.py", line 213, in create_github_release
    changelog = _generate_changelog(repo, log_string, categories)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/qiskit-bot/qiskit_bot/release_process.py", line 142, in _generate_changelog
    git_log = git.get_git_log(repo, log_string).decode('utf8')
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'str' object has no attribute 'decode'

We should fix the release processing to correctly generate the git log from 1.1.0...1.0.0 as intended.

Bump each element version after release

The only manual step that's still part of the release workflow (besides reviewing, approving, and merging PRs) is bumping the element version after a successful release. For example, see:

Qiskit/qiskit#3803

We should add support to the release automation path of qiskit-bot to do this automatically. This way it's less error prone and something we won't forget after pushing a release.

Make notifications varied so that people don't automatically ignore it

In private discussion with @mtreinish, we realize that people tend to ignore the qiskit-bot PR notification (for example: Qiskit/qiskit#8592 (comment)). A wild idea we have is to use some kind of AI service to generate the same message using different tones each time to make it more varied so that people will pay attention to it.

The simplest way to implement this is just to generate a bunch of these messages offline and store it in this repo and the bot randomly pick one version of it each time.

Add support for auto tagging and releasing meta package versions

Qiskit bot already watches the meta-package repo when it opens up a version bump pr like:

Qiskit/qiskit#802

so it can close it when the commit is finished. The next step of automation is that the bot should automatically create a release from the merged pr (which will trigger the travis job to upload the sdist). It also already has all the information for the release description with the updated versions since it uses that as the commit message. Right now this is a manual process and error prone and it should be automated.

Add support for publishing pre-releases

Looking to future releases it would be good to support doing release candidate releases prior to publishing a final release. To support this qiskit-bot will need to be improved to recognize a prerelease tag (vs a real release flag). On a pre-release tag qiskit-bot should create the stable branch, and the changelog, but not update the metapackage. Then on subsequent release candidates from the stable branch we should only generate the changelog. Then on the full release we will treat it like a stable point release but for the changelog generation we will want the from the previous full release to the new one (not just the incremental from the last release candidate) and to update the metapackage. We also probably should add a config flag to say whether the project supports release candidates or not because it does influence stable branch creation.

Don't post "Thank you for opening a new pull request" for frequent contributors

This automated comment is useless and noisy if the contributor has already made fifty PRs. (or five or maybe two).

Is there a way to check if the contributor has already made several PRs to the repo in question (eg. qiskit-terra) ?

Or even keep a static list of frequent contributors? I poked around in the code a bit. But I don't see where the github id that opened a PR is available.

Change release trigger mechanism

Right now qiskit-bot's release automation is triggered by by an authorized contributor pushing a tag to a tracked repo which triggers the changelog generation, github release page creation, etc. However, in practice this has proven to be somewhat error prone, while mistakes in manually tagging aren't common they happen occasionally (typically either tagging the wrong branch or the wrong commit). These mistakes are avoidable though if we just have qiskit-bot do the tagging.

A workflow I've been thinking about since the aer 0.4.2/0.5.0 release (which was caused by an admin on aer at the time tagging 0.4.2 on master instead of the stable branch) is to trigger the release automation from the release prep PR somehow and let qiskit-bot do the tagging (either directly or via github's release api). The two trigger conditions I'm debating between are either a PR merge with a release tag, or a user comment like @qiskit-bot release on a merged PR. I'm leaning towards the latter because it enables us to limit the users via configuration on who are allowed to trigger a release (more users are allowed to tag a PR than just those with write permissions) and it's more explicit in behavior.

Assuming we go with a comment trigger the expected behavior here would be adding a config field to list the authorized usernames on a repo, then on each PR comment check for the expected trigger comment text (probably just @qiskit-bot release) that will then trigger the tag creation from the sha1 of the merged commit for the pr (merge_commit_sha in the PR details). The rest of the release workflow should be the same.

Add support for marking PRs as explicitly not needing a changelog entry

For large releases, being able to tag a PR as explicitly not needing a changelog entry would be helpful in order to triage which PRs have and haven't yet had their changelog status reviewed. This could be from a label like 'Changelog: None' which exclude the labeled PR both from any of the changelog categories and from the ## No changelog entry list.

Add enhancements for external contributor activity

It would be great if we could have the qiskit bot do certain tasks related to activity from external community members. There are a few different parts to this outlined below, any further comments welcome :)

Part 1 - when HW/GFI label is added to an issue

1. create new webhook function for event_type=issues
2. when GFI or HW labels are added to an issue (i.e. data['action']=labeled and data[label]=gfi or hw) do the following:
   • add that issue to contributor monitoring project (github org level project beta)

I don't think PyGithub has support yet for editing github projects (beta), but they are editable via the github graphql api, if we can add a python graphql library or something

useful docs:

• webhook api ref
• github graphql api for updating projects

Part 2 - comments on HW/GFI issues from external contributors

1. create new webhook function for event_type=issue_comment
2. when a comment is left on an issue by a non-member (i.e. data['author_association'] != MEMBER) with GFI/HW (i.e. data['issue']['label']['name]=gfi or hw) do the following:
   • trigger notification for community-reviewers team

useful docs:

• webhook api ref

Drop metapackage handling

While an original motivating feature for the development of the qiskit-bot release automation was to handle the metapackage sync between all the qiskit elements and automate as much of that process as we could. With the release of qiskit 0.44.1 the use of the metapackage has been official retired. Moving forward all qiskit releases are from a single repository and we don't need all the complex logic involved with bumping version numbers and requirements files anymore. While it's not a strong priority right now, we should drop all the metapackage functionality from qiskit-bot to simplify the code base.

comment on visualization changes with binder for visualization testing

It would be great if the bot can comment when somebody modifies qiskit/visualization/ with the binder to test the visualization module: https://mybinder.org/v2/gh/<github_user>/<repo>/<branch>?urlpath=apps/test/ipynb/mpl_tester.ipynb and that can locally be run with jupyter notebook test/ipynb/mpl_tester.ipynb.

Change default_branch default value after qiskit migrates to use main

As pointed out in: #9 (review) a better forward looking default value for the default_branch config options added in #9 is main. Once the qiskit repos currently using qiskit-bot (qiskit/qiskit, terra, aer, ignis, aqua, nature, optimization, finance, and machine-learning) all migrate to use main for their default branch we should change the default value.

False positive for "Community PR" tag

The bot tagged one of my Terra PRs (Qiskit/qiskit#8627) with "Community PR", even though I appear as a member of the Qiskit org. When I curl'd the API hook for that PR, my association appears as "CONTRIBUTOR", not "MEMBER" (what the bot checks for).

I think we might want to swap to making a second API request to the pr["user"]["organizations_url"] API point to ask if the user is part of Qiskit directly, rather than trying to infer it from the association, since contributor seems to take precedence over member.

cc: @javabster (since I couldn't assign you for some reason).

tag as "community PR" when PR author is not in the organization

In order to better track PRs from the Qiskit community, it would be great if qiksit-bot could tag as community PR PRs that are not from members of the organization. The data seems to be available as author_association in the PR object https://api.github.com/repos/Qiskit/qiskit-terra/pulls?state=open

Add support for configurable notifications

Github doesn't natively give granular level of control for notifications on subdirectories. If an active contributor wants to get notified of open PR or issues that only effect a subset of the repo their only option right now is to subscribe to the whole project and get emails for every piece of activity. Using codeowners has been tried to solve this problem for those users who have write access to the repo, but it's not ideal because this couples conditional notifications with approval authority.

Adding a feature to qiskit-bot should not be hard to implement this, it already has all the capabilities to implement it. When qiskit-bot is configured to watch a repo it already knows how to read a local config file (for configurable tags) and it also gets events for every event. The missing piece is the step to take in PR and issue events and check them against the local_config_file for a configured element to see if a user should be notified. If there are users who need to be notified the bot can leave a mention for each configured user as a comment on the PR or issue. This will then generate a notification email for users with github's default notification settings.

Add release notes link to changelog generation output

We should have an option to include a link to the hosted release notes in the generated changelog output. The changlog output is just a subset of the git history for the release and the full release notes contain all the details. To do this we likely need a config flag somewhere to give qiskit-bot a hint to the url for the release notes and then it can generate the version suffix for the url to link to it.