New patch release

Meta-information Indicators (MI)

Overview

MI project collects data from GitHub repositories. You can use it to either collect data stored locally or within Amazon's S3 cloud. For personal usage, checkout <Usage> section.

Together with mi-scheduler, we provide automated data extraction pipeline for data minig of requested repositories and organizations. This pipeline can be scheduled customly, e.g. to run daily, weekly, and so on.

Data extraction request

To request data extraction for repository or organization, create Data Extraction Issue in MI-Scheduler repository. Use this link TODO

Data extraction Pipeline (diagram)

MI pipeline is simple to understand, see diagram below

+---------+
|ConfigMap|
+----+----+
     |

+--+-------+--------+--+ | | | | | | mi-scheduler | | | | | | +------+---+---+-------+ | | | | | | | | | | | | | | | | Argo Workflows | | | | | | | | | | |

+---------------v---v---v---v----v------------------+ +-------------------- +--------------------+ | | | Visualization | | Recommendation | | +---------+ +---------+ +---------+ | +-------------------+ +--------------------+ | | AICoE | | your | | | Project Health | | thoth | | | station| | | | org | | | (dashboard) | | | | +---------+ +---------+ +---------+ | | | | | | | +---------+---------+ +----------+---------+ | | | | | | repos | | thoth-station/mi ^ ^ | X X X X X | | | (Meta-information Indicators) | | | | | | | | | | +-------------+---------------+ | | | | | | | | | | | | | | | | | | +-----------------+-------------------+ | | | | | | | | | | | +---------+ +---------+ +---------+ | | Knowledge Processsing | | | | | +-----------------------+---------------------------+ +-----------------+-------------------+ GitHub repositories | ^ | +--------------------------------------------------------+ | | | | | | | Entities Analysis +-------> Knowledge | | +---------------->-+ +--------------------+ +---------+----------------+----------+------------------+ | Issues | Pull Requests | Readmes | etc........... | | | | | | +---------+----------------+----------+------------------+

What can MI extract from GitHub?

MI analyses entities specified on the srcopsmetrics/entities page Entity is essentialy a repository metadata that is being inspected (e.g. Issue or Pull Request), from which specified features are extracted and are stored to dataframe.

MI is essentialy wrapped around PyGitHub module to provide careless data extraction with API rate limit handling and data updating.

Install

pip

MI is available through PyPI, so you can do

pip install srcopsmetrics

git

Alternatively, you can install srcopsmetrics by cloning repository

git clone https://github.com/thoth-station/mi.git

cd mi

pipenv install --dev

Usage

Setup

Connect to GitHub

To be able to extract data from GitHub, access token must be configured. To generate one, read this

To use the token with mi, set GITHUB_ACESS_TOKEN environment variable to the token value, for example:

export GITHUB_ACESS_TOKEN=<token_string>

or

GITHUB_ACESS_TOKEN=<token_string> python -m srcopsmetrics.cli ...

and etc.

Data Location

To store data locally, use -l when calling CLI or set is_local=True when using MI as a module.

By default MI will try to store the data on Ceph. In order to store on Ceph you need to provide the following env variables:

S3_ENDPOINT_URL Ceph Host name
CEPH_BUCKET Ceph Bucket name
CEPH_BUCKET_PREFIX Ceph Prefix
CEPH_KEY_ID Ceph Key ID
CEPH_SECRET_KEY Ceph Secret Key

For more information about Ceph storing look here

CLI

To view all of the available commands and their description use

python -m srcopsmetrics.cli --help

See some of the general usage examples below

Get repository PullRequest data locally

python -m srcopsmetrics.cli --create --is-local --repository foo_repo --entities PullRequest

which is equivalent to

python -m srcopsmetrics.cli -clr foo_repo -e PullRequest

Get organization PR data locally

python -m srcopsmetrics.cli -clo foo_org -e PullRequest

Get multiple repository PR data locally

python -m srcopsmetrics.cli -clr foo_repo,bar_repo -e PullRequest

Get multiple entity data locally

python -m srcopsmetrics.cli -clr foo_repo -e PullRequest,Issue,Commit

Meta-Information Entities Data

How to load data

Indicators

To know more about indicators that are extracted from data, check out Meta-Information Indicators.

How to contribute

Always feel free to open new Issues or engage in already existing ones!

Custom Entities & Metrics

If you want to contribute by adding new entity or metric that will be analysed from GitHub repositories, feel free to open up an Issue and describe why do you think this new entity should be analysed and what are the benefits of doing so according to the goal of thoth-station/mi project.

After creating Issue, you can wait for the response of thoth-station devs Do not forget to reference the Issue in your Pull Request.

Implementation

Look at Template entity to get an idea for requirements that need to be satisfied for custom entity implementation.

	ContentFile = Schema({"name": str, "path": str, "content": str}) # TODO: Adjust content type

	ContentFiles = Schema({str: ContentFile,})

thoth-station / mi Goto Github PK

mi's Introduction

Meta-information Indicators (MI)

Overview

Data extraction request

Data extraction Pipeline (diagram)

What can MI extract from GitHub?

Install

pip

git

Usage

Setup

Connect to GitHub

Data Location

CLI

Get repository PullRequest data locally

Get organization PR data locally

Get multiple repository PR data locally

Get multiple entity data locally

Meta-Information Entities Data

How to load data

Indicators

How to contribute

Custom Entities & Metrics

Implementation

mi's People

Contributors

Stargazers

Watchers

Forkers

mi's Issues

Command

Notes

Command

Notes

Command

Notes

Command

Notes

When should we update the processed knowledge?

This issue was generated by todo based on a TODO comment in 19017e4. It's been assigned to @goern because they committed the code.

Command

Notes

This issue was generated by todo based on a TODO comment in 54d4dd8 when #164 was merged. cc @goern.

Command

Notes

Command

Notes

Recommend Projects

Recommend Topics

Recommend Org

Jobs

This issue was generated by todo based on a `TODO` comment in `19017e4`. It's been assigned to @goern because they committed the code.

This issue was generated by todo based on a `TODO` comment in `54d4dd8` when #164 was merged. cc @goern.