GithubHelp home page GithubHelp logo

oca-github-bot's Introduction

OCA GitHub bot

pre-commit.ci status GitHub CI status

The goal of this project is to collect in one place:

  • all operations that react to GitHub events,
  • all operations that act on GitHub repos on a scheduled basis.

This will make it easier to review changes, as well as monitor and manage these operations, compared to the current situations where these functions are spread across cron jobs and ad-hoc scripts.

Table of contents

Mention declared maintainers that addons they maintain are being modified.

Comment with a call for maintainers if there are no declared maintainer.

Auto-delete pull request branch
When a pull request is merged from a branch in the same repo, the bot deletes the source branch.
Repo addons table generator in README.md
For addons repositories, update the addons table in README.md.
Addon README.rst generator
For addons repositories, generate README.rst from readme fragments in each addon directory, and push changes back to github.
Addon icon generator
For addons repositories, put default OCA icon in each addon that don't have yet any icon, and push changes back to github.
setup.py generator
For addons repositories, run setuptools-odoo-make-defaults, and push changes back to github.

These actions are also run nightly on all repos.

Also nightly, wheels are generated for all addons repositories and rsynced to a PEP 503 simple index or twine uploaded to compatible indexes.

When there are two approvals, set the approved label. When the PR is at least 5 days old, set the ready to merge label.

When the CI in a Pull Request goes green, set the needs review label, unless it has wip: or [wip] in it's title.

One can ask the bot to perform some tasks by entering special commands as merge request comments.

/ocabot merge followed by one of major, minor, patch or nobump can be used to ask the bot to do the following:

  • merge the PR onto a temporary branch created off the target branch
  • merge when tests on the rebased branch are green
  • optionally bump the version number of the addons modified by the PR
  • when the version was bumped, udate the changelog with oca-towncrier
  • run the main branch operations (see above) on it
  • when the version was bumped, generate a wheel, rsync it to a PEP 503 simple index root, or upload it to one or more indexes with twine

/ocabot rebase can be used to ask the bot to do the following:

  • rebase the PR on the target branch

/ocabot migration, followed by the module name, performing the following:

  • Look for an issue in that repository with the name "Migration to version {version}", where {version} is the name of the target branch.
  • Add or edit a line in that issue, linking the module to the pull request (PR) and the author of it.
  • TODO: When the PR is merged, the line gets ticked.
  • Put the milestone corresponding to the target branch in the PR.

See our open issues, pick one and contribute!

The easiest is to look at examples.

New webhooks are added in the webhooks directory. Webhooks execution time must be very short and they should delegate the bulk of their work as delayed tasks, which have the benefit of not overloading the machine and having proper error handling and monitoring.

Tasks are in the tasks directory. They are Celery tasks.

Tasks can be scheduled, in cron.py, using the Celery periodic tasks mechanism.

First create and customize a file named .env, based on environment.sample.

Tasks performed by the bot can be specified by setting the BOT_TASKS variable. This is useful if you want to use this bot for your own GitHub organisation.

You can also disable a selection of tasks, using BOT_TASKS_DISABLED.

docker-compose up --build will start

  • the bot, listening for webhooks calls on port 8080
  • a celery worker to process long running tasks
  • a celery beat to launch scheduled tasks
  • a flower celery monitoring tool on port 5555

The bot URL must be exposed on the internet through a reverse proxy and configured as a GitHub webhook, using the secret configured in GITHUB_SECRET.

This project uses black as code formatting convention, as well as isort and flake8. To make sure local coding convention are respected before you commit, install pre-commit and run pre-commit install after cloning the repository.

To run tests, type tox. Test are written with pytest.

Here is a recommended procedure to test locally:

  • Prepare an environment file by cloning and adapting environment.sample.
  • Load environment in your shell, for instance with bash:
set -o allexport
source environment
set +o allexport
  • Launch the redis message queue:
docker run -p 6379:6379 redis
PATH=/path/to/maintainer-tools/env/bin/:$PATH
  • Create a virtual environment and install the project in it:
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt -e .
  • Then you can debug the two processes in your favorite IDE:
    • the webhook server: python -m oca_github_bot
    • the task worker: python -m celery --app=oca_github_bot.queue.app worker --pool=solo --loglevel=INFO
  • To expose the webhook server on your local machine to internet, you can use ngrok
  • Then configure a GitHub webhook in a sandbox project in your organization so you can start receiving webhook calls to your local machine.

To release a new version, follow these steps: - towncrier --version YYYYMMDD - git commit the updated HISTORY.rst and removed newfragments - git tag vYYYYMMDD - git push --tags

This module is maintained by the OCA.

Odoo Community Association

OCA, or the Odoo Community Association, is a nonprofit organization whose mission is to support the collaborative development of Odoo features and promote its widespread use.

oca-github-bot's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

oca-github-bot's Issues

review reminder

I'd like to use the code sprint to develop a bot that scans open PRs and in case the submitter didn't do enough reviews in this repo (or all of OCA) writes a kind reminder that we need reciprocity. It could also add labels praising PRs by very engaged contributors and (maybe that's bad, we'll have to discuss all of what I write here) mark/close PRs by not so engaged authors as such.

If you're interested, I think this is what we'll have to discuss first. Please see this really as a proposal for discussion, none of this is fixed yet.

  1. identify if a PR is one that creates work or relieves us from work. So the bot should do nothing (or add the great PR label) for a PR that just fixes (=make green) a branch. No idea if the api allows us to parse the results of other tests, but the idea would be that a PR that makes tests succeed that failed before is one that relieves from work, all others create work.

    1.1 tell a good review from a bullshit one. No idea how to do this without giving sick incentives, if possible at all

  2. for a PR that creates work, check how many open PRs the contributor has and how many reviews she did in this repo (or all of OCA?) in the last X days. Then come up with some formula like

    $num_open_prs - 2 * $num_relieving_prs < $num_good_reviews - $num_bullshit_reviews
    

    and if that's true, propose friendly to do some more reviews, and/or set labels.

  3. try to propose some PRs in the above message, take things like size of the PR, amount of reviews into account when proposing

My hope would be that this motivates people to more reviews, but the formula above would also make it possible to never review, but just contribute by doing the grunt work of fixing branches. This should cater to people who just don't want or cannot do reviews.

Reduce the number of commits in the main branch bot

The main branch bot and ocabot merge tend to create many commits (readme, icon, bump version, etc). One way to improve that could be to have these individual scripts/task do a git add instead of a git commit and then let the bot commit them all at once.

merge strategies

Currently, the default and only available merge strategy for the merge command is rebase.

After further reflection, I now think the default merge strategy could be merge.
We can create the merge bot branch by branching off the target branch and merging the PR into it. The rest of the process can be identical and we keep the same benefits (i.e. testing on top of the main branch before merging into it, bumping version as last operation, etc).

Additional benefits of the merge strategy would be that branches which contain merge conflict resolution should be mergeable via the bot command, and GitHub would see PRs as merged.

Other merge stragegies (rebase with autosquash, squash all) can be made available via additional options to the ocabot merge command.

delete merge bot branch when pr is closed or merged

Currently the bot deletes the merge bot branch when merging.

In case a manual merge is done in the meantime, or the merge is re-requested by another user, some merge bot branches can stay pending forever.

So instead of (or in addition to) deleting the branch when merging, also delete it when a PR is merged or closed.

Merge on demand

A new bot task, that would be triggered by a command in a pull request comment.

Command syntax

/ocabot merge (major|minor|patch)?

Actions performed

  • on merge command detected in PR comment

    • !!! check the author of the comment has the right to merge
      • is part of a team with write permission on the repo
      • or, is declared as maintainer in the manifest of all addons modified by the PR
    • rebase on main branch with autosquash
    • generate addon readme and setup.py
    • bump manifest version if (major|minor|patch), looking at all addons changed by the PR
    • push to a new branch named ocabot-merge-pr<pr>-to-<branch> (fail if that branch already exists)
  • on red CI status on ocabot-merge-pr<pr>-to-<branch> branch

    • (ignore coverage/coveralls)
    • add failed comment on PR
    • delete branch, exit
  • on green CI status on ocabot-merge-pr<pr>-to-<branch> branch

    • push to main branch (--no-ff), if fail: rebase on main branch and force push ocabot-merge-pr<pr>-to-<branch>, if rebase fails, delete branch and exit
    • delete branch ocabot-merge-pr<pr>-to-<branch>
    • add a comment on the PR: merged at sha
    • close the PR
    • create wheel and publish to pypi, for the commit of our previous merge (to make sure a non .dev version goes to pypi if we bumped version)

Attention point

  • makepot must somehow run before bumping the version: idea travis does makepot on ocabot-merge-pr#, and a local msgmerge -U is done before building the wheel so the translations in the wheel are aligned with the .pot.
  • maybe we should allow makepot to run and push to PR branches => risk of merge conflict

Tasks

  • create main PR for this issue to which other subtasks can be contributed: #39
  • main method @sbidoul
  • tie main method to command @sbidoul
  • function to bump manifest version
  • function to check if a github login has write access to the repo @fkantelberg
  • webhook @fkantelberg
  • function to detect addons modified by a PR, start here
  • function to parse a comment and detect the merge command, returning {pr, target branch, version bump requested ('major'|'minor'|'patch'|None)}
  • functions to make the merge bot branch name from {pr, target branch} and parse the branch name
    to return {pr, target branch}
  • function to check if a github status webhook call is considered valid for merging (ignore coverage?)

number of approvals

Following discussion in #1

Concerning the number of approvals needed, maybe we should allow a per-repo configuration (in a file or whatever) with a fallback value set to 2.

Any opinions?

Automatic backward/forward porting?

This was discussed long ago at OCA/maintainer-tools#381 but it seems that today the part of version bumping is solved by the bot and... well, we have a bot; so it might be a good moment to reopen the discussion.

One of the worst parts about code maintenance here in OCA is the presence of several versions of the same code across different git branches, and I know it won't change, but today we have this bot, which is a good foundation to finally attack this problem without altering current workflow.

Odoo itself uses the same system, but they actually have a combination of bot and humans to perform the merge+forwardport loop sanely, and a different product and policy to offer, which helps their maintenance.

It would be great if the bot could handle forward ports (backward would be awesome too). To avoid the dangers of automatic code ports, all the bot should do is:

  1. Am I instructed to do the forward port? then:
  2. Try to rebase this patch onto the next version branch. Some special machinery will be needed to ignore conflicts in version bumps. Did it rebase? then:
    1. Open a PR to that new version.
    2. Assign it to whoever instructed the bot to forwardport the original PR.
  3. If it didn't rebase, just add a comment there in the original PR specifying the reasons for not forwardporting. Humans will have to handle this case.

The same for backward ports (that could be left for the future).

A cool thing to have in mind would be massive version migrations. This again can be left for the future, but imagine a scenario where Odoo 13.0 is released, then the 13.0 branches are opened, and then the bot does by itself the commit and code forward ports and opens a PR per module.

This would let humans focus in the corner cases, increasing speed of development and reducing repetitive maintenance tasks 💪, which often get undone due to their nature. 😊

temporary_clone must not change current directory

This prevents running tasks in parallel.

The temporary_clone must yield the temporary directory instead of changing directory.
Code must be reviewed for place that rely on the current directory to be the git clone root.
In particular all subprocess calls must be adapted to have a proper cwd argument.

[RFC] Add to bot instructions through comments for maintaining migration issues

A current problem is that migration issues with the list of modules to migrate is not up to date:

  • Already merged of migration PRs are not checked.
  • Or even worse, they are not noted.
  • New migration PRs are not in the list because they weren't in the repository in the previous version when the branches were created.

I think we can inspire in robodoo from Odoo for putting comments for doing certain automated tasks. A flow can be:

  • A PR for a migration is proposed.
  • A PSC put an instruction comment indicating the module and the version it migrates: @oca-git-bot migration 11.0 module_x.
  • The bot checks the migration issue (through the title "Migration to version X"), and fills the line with the author and the PR number, creating it if it doesn't exist.
  • When the PR is merged, then the bot marks the check.
  • If the PR is closed, the bot strikes-through the part of the text.
  • If a new PR for the same module appears (as superseeded), and a PSC instructs the bot with the same comment in the new PR, the text for the new one is added on the line.

New ocabot merge option: autosquash?

Following the discussion here: #75, and specifically this clarification: #75 (comment).

Personally, when I first red that the bot was going to autosquash commits I loved the feature as I prefer to add fixup commits to have clear diffs as you attends review or do changes. For me, rely on the force push diff is not ideal. Then today I found that it was not actually working and the merge strategy is a different one, so this issue raises the question: is it worth/useful to implement such an option? something like ocabot merge autosquash.

Min PR age

Following a discussion in #1 .

Most of us (I think) are employees (or self-employed) and don't work on the week-ends.

For the computation of the date on which the tag "ready to merge" is set on a PR, what about using a "real" calendar and don't count week-ends.
Proposal: use BusinessDays like in this gist https://gist.github.com/JosefJezek/275b2c667698aabd6969

Any opinion?

merge bot: allow addon maintainers to merge

In addition to checking the push permission, the merge bot could check if the user who requests the merge is a maintainer in all addons modified by the PR.

This would help delegating merge right on large repos without giving push access to the whole repo.

This check must be done on the target branch, not on the PR branch (to avoid someone declaring itself as maintainer to merge).

Auto-update pre-commit configuration

Configuration

  • git url (optional) + subdirectory containing configuration files (in subdirectories named pre-commit-{series}).

Cron

  • nightly

Algorithm

  • in main branch bot
  • git clone
  • copy configuration files to repo root
  • update top level exclude to add addons with installable=False
  • ? preserve known_third_party in .isort.cfg
  • pre-commit run -a (repeat up to 3 times until successful, to cope for generated diffs)
  • if successful, git commit and push to main branch
  • if unsuccessful, git commit (with --no-verify) and push to a temporary branch and create a PR to let maintainers resolve the issue

merge bot: special case bump version patch for new modules

When a new module is merged, the bot currently does not bump version.
In case the commit that set the version number is not the last commit, then a .dev version goes to pypi.

To avoid this, the bot could detect that situation and force a patch version bump if the commit setting the version number is not the last one. This must be done only for new modules being added to the target branch.

CLA Bot

Migrate the CLA bot to here.

Improve logging of subprocess calls

Currently, subprocess are called with subprocess.check_output to capture stdout and stderr in the exception and let sentry display it.

Unfortunately Sentry seems to truncate the output attribute of the CalledProcessError exception.

So a better way is needed so we get both a hint in sentry and a full output in the log.

Repo addons table generator

I don't know if there is already a way to get a list of all the modules in the OCA repositories (with its version and the latest update date), but if it doesn't exist I believe that would be very helpful.

There are more than 150 repositories in OCA, mostly with branches for Odoo 12.0, 11.0, 10.0... Each branch can have several modules, and each module can have several versions and updates. So, keep the trace of all that modules are a hard and tedious work

I don't know if there are already any plan to get a list of all the modules, or something like that. But maybe that github-bot could help to create and update that list.

Would it be useful to have a table with all the OCA modules? And sorting or filtering that modules by repository, Odoo version, last update?

CLA bot: Handling of old users

@gurneyalex there's a problem with CLAbot handling all users. Check the first commit of PR OCA/stock-logistics-workflow#482:

seleccion_064

There's no GitHub account associated with it. Why? Because that person is no longer working on the company and the account was deleted/unlinked to that email address.

I tried to add the contact with the email and with the CLA tag in our DB, but it's not enough for removing the warning.

Can we look for commit e-mail instead of looking for GitHub account in these cases?

seleccion_065

Command to abort a merge command

#82 makes me think we probably want some command to abort a merge. Because if an addon maintainer would have done that by accident s.he would have had no way to cancel it.

Such command simply needs to delete any ocabot-merge-... branch for the corresponding PR.

It could be / ocabot merge abort.

CLA bot on translation commits

@gurneyalex if you look at this PR: OCA/account-analytic#204, you can see that there are commits made by "Harald Panten" that come from Weblate, but they are considered as CLA unknown.

First question: should we force to sign a CLA for translations?

If so, @sbidoul / @max3903 , we have to try to enforce that CLA in Weblate login or translation request.

In any case, I think CLA bot should ignore that commits.

push readme and setup.py to PR branches

When the contributor leaves it's branch open to changes by maintainers, it would be nice to have the bot push the README update and setup.py generation to the PR branch.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.