Bugfix pattern mining about hercules

[EgorBu]

Copy-paste of #188 (comment)

About classification - it's possible to collect some labels from PRs in huge repositories like

• https://github.com/tensorflow/tensorflow/labels and bug related
• https://github.com/pytorch/pytorch/labels and bug related
• and so on

Pattern mining

I like an idea of pattern mining.
I would suggest making commit deduplication + community detection / (or clustering / topic modeling) instead of classification.
Why:

• it's not clear how many classes we have (50+ labels in each repository that I mentioned above).
• it could be quite a big variability even inside one class.
• incredible number of commits

deduplication + community detection

The way how to do it the same as in apollo:

• extract features: textual, structural, etc -> bag-of-something.
• fuzzy deduplication + hyperopt (it may require manual labeling or automatic calculating similarity score and selecting threshold).
• connected components + community detection.
• descriptive statistics of each community / labeling.

So after it will be possible to query and receive communities and their descriptions / labels.

topic modeling

How to - standart topic modeling pipeline:

• extract features: textual, structural, etc -> bag-of-something.
• hierarchical or simple topic modeling using bigartm and vis

So after it will be possible to query and receive topics for commits.

One more point - that the first step (feature extraction) could be reused for several different approaches.