bigcode-project / bigcode-analysis

Repository for analysis and experiments in the BigCode project.

BigCode Analysis

This repository is for the analysis done in BigCode Project. You can find analysis of datasets, models, architecture choices and more.

Contents

• Data analysis: In the folder data_analysis, we provide code for data analysis:

  • Near deduplication
  • Python data analysis:
    • Natural language distribution in comments/docstrings
    • Data decontamination for HumanEval and MBPP benchmarks
    • Percentage of files that can be successfully compiled
    • Percentage of configuration and test files
    • Exploration of unimax sampling on The Stack Some notebooks with some early data and model loss analysis.

• Multi-Query Attention experiments, for details please to multi_query_experiments/README.md)

bigcode-analysis's Issues

Evaluate CodeGen on safe and all-license dataset

cannot import AttentionType from gpt2

Loading multi_query_experiments.profile_hf_generate results in

ImportError: cannot import name 'AttentionType' from 'transformers.models.gpt2.modeling_gpt2'

I find this puzzling since this class is not exported by transformers 4.20.0

Are you using a custom fork of transformers by chance?

Broken link

The link to python-all-license is broken in the data analysis readme.

Reorganize bigcode-data-analysis repository

[Near Deduplication] Benchmark

Provide results on large dataset with different near deduplication methods:

1. minhash + lsh
2. simhash
3. any relevant methods

Details to be included:

• tokenization method
• method parameters
• hardware
• memory usage
• time
• duplication results, examples

[Decontamination] Add readme and instructions to run substring decontamination

Add filtering to the near deduplicated safe dataset

[Near Deduplication] Tokenization

As we extend deduplication to a wide range of languages, what tokenization method to use will have an impact on the final results.

The current script uses a simple regex and uni-gram to perform minhash calculation. What are the consequences using a different configuration?

Rename model names on HF hub

Decontamination

Evaluation datasets like mbpp or HumanEval might be contained in the training set.

• Exact dedup: Remove files containing the exact evaluation code.
• Near dedup: Some of the tasks resemble popular Leetcode questions, how can we identify them at prompt level?

[Exact Substring Deduplication] Analysis

Near deduplication #7 only operates on file level. It is also possible for a file to be

1. a substring of another file, while the minhash/simhash fingerprints being wildly different
2. composed of multiple snippets from different sources

Do we do something about them, knowing they contains large chunks of repeated snippets?

Upload github dataset with license column

[Near Deduplication] Post processing

The current script building clusters of duplicates, but there are cases it might yield unwanted results:

When doc B is clustered under doc A's name, another doc C can also be clustered into B's name (AB, BC, C!~A), thus when we are deleting non "extreme"s from each cluster, we could end up having both A and B kept in the results.

A better way to delete duplicates is to find community within each connected components. This is used in https://github.com/src-d/gemini.

github scraping speed limit

We have a speed limit for scraping github, repo homepages at least.
From one ip address it is around 2 repo per second, but it is only 2-3 times faster from 20 different IP addresses ( from the same datacenter, toolkit). A lot of status code 429, rate limiting events. I wonder if it is general github policy or or datacenter just got lucky?
Experiment code here https://github.com/bigcode-project/bigcode-analysis/blob/github_scraping_test/data_analysis/github_scraping_test/github_scrapping_test.ipynb

Maybe anyone can run this experiment on their ray cluster or just repeat the test any other way form their range of ip addresses?