Comments (8)
My reply:
AFAIK this is not exposed via the GH API and I can think of these three options: [to gain this knowledge]
- hardcode what CIs are required/"more important"
- simulate login as user and scrape the PR page with the received cookie (I have already implemented sth. similar here: #50)
- have a simple plain file within the repo that provides this knowledge (there's no API rate limit on raw files!)
from dlang-bot.
@wilzbach Ah, OK. I may have been misinterpreting L33-50 of your ci.d
file, where I was making the assumption that you were able to query the individual CI providers for status info.
If that's difficult (for now) then I agree that a minimum threshold is a good compromise. In all likelihood this will in practise amount to higher-priority and more stable tests passing (it seems to be stuff like the style and code-coverage tests that fail more frequently and sometimes arbitrarily).
from dlang-bot.
I may have been misinterpreting L33-50 of your ci.d file, where I was making the assumption that you were able to query the individual CI providers for status info.
No you haven't. We do query the APIs of the CI providers for the PR, but we already get the status info as part of the hook payload, see e.g. this file for an example:
https://github.com/dlang-bots/dlang-bot/blob/master/data/hooks/github/dlang_dmd_status_6324.json
Moreover we do query the status
endpoint as well to get an overview of the state of all CI providers, see e.g. this file for an example:
(it seems to be stuff like the style and code-coverage tests that fail more frequently and sometimes arbitrarily).
Yeah this is still due to https://github.com/codecov/support/issues/360 (don't forget to vote for it)
If that's difficult (for now) then I agree that a minimum threshold is a good compromise. In all likelihood this will in practise amount to higher-priority and more stable tests passing
AFAIK @MartinNowak's plan are
- superseed CircleCi with the new Dlang CI system, so that the number of CI goes down
- set all CIs except CodeCov to "enforced" (and maybe even CodeCov)
That's also why I didn't want to put any specific assumption about the CI system into the bot as of now.
from dlang-bot.
All CIs should be created equal, attention is the most precious resource we have.
Any "bad" CIs not worth being noticed or not presenting understandable errors, are rather a burden, as they lower the credibility of "good" CIs.
Requiring people to learn a complex (and undocumented) rating system based on 5 ✓/✗s sounds horrible. It would be great if GH had a 🚥 status including unstable builds, but they don't so we have to live with binary.
IMO there should be a single green or red dot on every PR. CIs that are too flaky, still experimental, or only informative should always be green.
We should also stop abusing required
as a tool to let PRs with red "bad" CIs pass through. That's a conflicting message and thus confusing.
from dlang-bot.
(and maybe even CodeCov)
Even that, coverage is important, we can configure sufficient tolerances. BTW, we agreed to only keep codecov/patch as that has the best SNR.
from dlang-bot.
I would suggest that some CI are more important than others -- so e.g. style CI should not block moving to 'needs review', but test-suite failure should probably see the tag set to 'needs work'
As for the workflow ideas in the OP, such a distinction sounds reasonable.
While all CI failures should mean failure and not mergeable, they have different implications on the review workflow.
We should abstract this a bit, in case we later move linting and unittests to the same infrastructure, though even then we should be able to extract the information from the status message.
from dlang-bot.
Even that, coverage is important, we can configure sufficient tolerances.
We did overwrite CodeCov to be always green.
BTW, we agreed to only keep codecov/patch as that has the best SNR.
While all CI failures should mean failure and not mergeable
Absolutely agreed and for the main D repos all CIs are now enforced to pass. Well except for gigantically slow Travis OSX builds for DMD. It has worked well for the last seven days.
from dlang-bot.
All CIs should be created equal, attention is the most precious resource we have.
This sounds right to me today as well, so I think we can close this issue.
from dlang-bot.
Related Issues (20)
- clear shutdown of CI servers before deleting them
- Smart WIP handling HOT 3
- Bot does not recognize the switch from a draft PR to a full PR
- Bugzilla description partially discarded HOT 2
- Testsuite doesn't run on Alpine Linux / Musl HOT 1
- Don't merge PRs where non-required status are failing HOT 3
- Continuous TLS error on Scaleway API HOT 6
- Deployed Heroku instance should have debug infos HOT 1
- Bot ignore Azure pipelines failures HOT 2
- Bot cannot post on issues with Vibe.d v0.8.6 HOT 1
- The bot wrongly closes bugzilla issues from events generated by dlang-community projects HOT 1
- Query state of GitHub issues HOT 2
- Detect weblinks as reference to bugtracker issue HOT 6
- Update last commit message when merging with links to the PR and its title HOT 1
- Heroku stack is deprecated HOT 1
- dlang-bot for dub didn't tag the latest release HOT 1
- Bot didn't catch second bugfix HOT 2
- Tweaking bot with draft might be possible HOT 1
- Spurious escape in bug description HOT 1
- BUG: The dlang-bot doesn't Resolve Issues whose PRs Target the table branch HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dlang-bot.