Comments (6)
Currently the workflow also fails for repositories having more than ~120 PRs redhat-et/time-to-merge-tool#4 even when the Github API rate limit is not reached. Explore options of replacing data collection notebook with script, to rule out the possibility of timing out because of the jupyter notebook cells running for too long.
from ocp-ci-analysis.
Also evaluate the size of the dataset generated and the storage space available on the github workflow worker
from ocp-ci-analysis.
Currently the workflow succeeds on repos ~400 PRs. During the workflow the Github API token got rate limited, but the workflow continued running and the data download resumed when the API rate was restored in an hour
thoth-station/mi
- 414 PRs - https://github.com/oindrillac/ttmtool/actions/runs/3391777511 - 3 h 10 minsthoth-station/mi
- 414 PRs - https://github.com/oindrilla-chat/ttmtool/actions/runs/3391830064 - 3 h 8 mins
Currently triggered workflows on larger repos with ~750 PRs, awaiting their results
- https://github.com/oindrillac/ttmtool/actions/runs/3394465766
- https://github.com/oindrilla-chat/ttmtool/actions/runs/3394697938
As a next step, will modify the data collection step to collect PRs across an organization and monitor a sample workflow
from ocp-ci-analysis.
Jobs with a large number of PRs example thoth-station/kebechet
with 750 PRs fails each time when the workflow is submitted
- https://github.com/drillachat/ttmtool/actions/runs/3406491796
- https://github.com/oindrilla-chat/ttmtool/actions/runs/3394697938
During the course of running the workflow, it pauses several times upon reaching the rate limit INFO:srcopsmetrics.github_handling:API rate limit REACHED, will now wait for 58 minutes
.
But each time, it failed after running for around 6 hrs, with an Error: The operation was canceled.
when the workflow was paused while rate limited.
from ocp-ci-analysis.
The workflow fails each time after 6 hrs even if the job was continuing to run.
eg: https://github.com/drillachat/ttmtool/actions/runs/3414370231/jobs/5682237181
It seems this is in line with Github's usage limits which does not allow jobs within a workflow to run longer than 6 hrs.
srcopsmetrics
currently does not save intermediary results from collected data which means if the job gets canceled, no data from that PR would be saved.- Since we see that even collecting data from a single repo with (~750 PRs) can take > 6 hrs, we need to find a way to split this into multiple jobs within the workflow and also save intermediary results from a job on S3.
from ocp-ci-analysis.
Even if we split a workflow into multiple workflow or jobs, we will need to save intermediate results from the first job. srcopsmetrics
currently does not do that in the default mode. We can try to see if we can download the data locally and save them manually incrementally on S3 per repo.
However, for repos such as thoth-station/kebechet
which has over 700 PRs, srcopsmetrics
even in the local mode does not save a json per PR as it saves one file per repo thus we will not be able to use srcopsmetrics
to collect data for repos such as thoth-station/kebechet
within a github workflow job
from ocp-ci-analysis.
Related Issues (20)
- Create a new Superset Dashboard with KPIs and metrics specific to the use-case OSP and some relevant metrics like number of tests, time-to-test, mean time to failure, etc. HOT 1
- Scale19x: Data Science Workshop Feedback
- Workshop in Opf DS Meetup - Changes and Tasks to be done
- Explore Kata Containers github repository
- Use base container image for TTM github action workflow HOT 2
- Pass env vars to github action workflow from curl and action secret HOT 3
- Simplify notebooks in TTM workflow
- Create user documentation and architecture diagram for TTM Github action workflow HOT 2
- AI4 OSC: Q4 OKRS
- Save trained model output of github action workflow as an image
- Explore option of creating a container action and using that in the github action workflow
- Export Github action workflow as a job HOT 1
- Encrypt env variables being passed to github workflow HOT 3
- Add model validation nb to github workflow pipeline HOT 1
- Add model deployment to github workflow HOT 2
- Simplify feature engg and model training nb to remove unnecessary experimental code HOT 1
- Change the methodology of choosing the model (it is currently hard-code in yaml). Edit the notebook to pick the best model based on performance metrics HOT 1
- Consider possibly combining feature engineering with model training steps
- No pinned version for package src
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ocp-ci-analysis.