microsoft / codet

CodeT

This repository contains projects that aims to equip large-scale pretrained language models with better programming and reasoning skills. These projects are presented by Microsoft Research Asia and Microsoft Azure AI.

Projects

• [CodeT]: Code Generation with Generated Tests
• [DIVERSE]: On the Advance of Making Language Models Better Reasoners
• [RepoCoder]: Repository-Level Code Completion Through Iterative Retrieval and Generation

codet's Issues

Model Release Plan

Hi, thanks for your impressive work! I really enjoyed reading the paper. When do you plan to release the model code and weights? Thank you!

Random subsets for reproduction

Thank you for your wonderful work!

I just ran run_pipeline.py and got missing file errors:

FileNotFoundError: [Errno 2] No such file or directory: 'datasets/random-api-completion.test.jsonl'

As far as I understand, the random- prefix denotes split subsets for evaluation, as referred to in Section 3 of the original paper:

Eventually, a total of 1600 test samples are generated for the line completion dataset.

and

From these candidates, we then randomly select 200 non-repetitive API invocations from each repository, resulting in a total of 1600 test samples for the API invocation completion dataset.

For the purpose of reproduction , I would like to ask you about the following four subsets in utils.py:

class FilePathBuilder:
    api_completion_benchmark = 'datasets/random-api-completion.test.jsonl'
    random_line_completion_benchmark = 'datasets/random-line-completion.test.jsonl'
    # short version for codegen
    short_api_completion_benchmark = 'datasets/random-api-completion-short-version.test.jsonl'
    short_random_line_completion_benchmark = 'datasets/random-line-completion-short-version.test.jsonl'

Evaluation on HumanEval

Hi, thanks for the great work! Whether the models were finetuned before using HumanEval, and whether the models and parameters can be provided?

File Not Found Error

While running run_pipeline.py I'm encountering the below error:

Traceback (most recent call last):
  File "run_pipeline.py", line 66, in <module>
    run_RG1_and_oracle_method(CONSTANTS.api_benchmark, repos, window_sizes, slice_sizes)
  File "run_pipeline.py", line 28, in run_RG1_and_oracle_method
    CodeSearchWrapper('one-gram', benchmark, repos, window_sizes, slice_sizes).search_baseline_and_ground()
  File "/local/CodeT/RepoCoder/search_code.py", line 119, in search_baseline_and_ground
    self._run_parallel(query_line_path_temp)
  File "/local/CodeT/RepoCoder/search_code.py", line 105, in _run_parallel
    repo_embedding_lines = Tools.load_pickle(repo_embedding_path)
  File "/local/CodeT/RepoCoder/utils.py", line 118, in load_pickle
    with open(fname, 'rb') as f:
FileNotFoundError: [Errno 2] No such file or directory: 'cache/vector/repos/huggingface_diffusers_ws20_slice2.one-gram.pkl'

Additional Info:

huggingface_diffusers_ws20_slice2.one-gram.pkl file is created in cache/vector/random_api directory however, there is nothing in cache/vector/repo
Also, there is huggingface_diffusers_ws20_slice2.pkl file in cache/window/repo/

Can someone please help with this issue.

Releasing generated data for MBPP

Hi All,

Great work!

I am wondering if you can also release the generated data for MBPP? Similar to the ones you released for HumanEval.

https://github.com/microsoft/CodeT/tree/main/CodeT/data/generated_data

Thank you!

Dead link to cluttr results

in DIVERSE/data/clutrr.md there's a link to

https://dkibdm.blob.core.windows.net/llmpublic/clutrr_code_davinci_002.zip

and it's dead.

Thanks!

CodeGen-16b MBPP pass@1 results

Hi,

Thanks for the great work and releasing the data/code.

I tried to replicate the results using the generated codes/test cases of CodeGen-16b on the MBPP benchmark. My pass@10/100 are quite similar to your reported results but pass@1 is quite far behind when using CodeGen-16b only (the result when using CodeT is similar to the reported number).

Model		pass@1
CodeGen-16b	reported	42.4%
CodeGen-16b	replicate	31.32%
CodeGen-16b + CodeT	reported	49.5%
CodeGen-16b + CodeT	replicate	49.58%

I wonder whether you used a different generation setting for pass@1 (e.g. different. temperatures) than for pass@10/100? Or there were a typo in the reported number.

Data split, prompt, and model predictions for CLUTRR

Hi authors, thanks for the great work! Would it be possible to share more details about the CLUTRR dataset, e.g. the data split, the prompt you used, and the model predictions?

Thanks!

Missing pkl files

Hi,

When I execute:
python run_pipeline.py

I encounter the error:
FileNotFoundError: [Errno 2] No such file or directory: 'cache/vector/repos/repositories/huggingface_diffusers_ws20_slice2.one-gram.pkl'

In fact, there is no pkl file available under: cache/vector/repos/repositories/
Also, no pkl file is present under: cache/window/random_api/gt/repositories/

Please upload the missing pkl files. Or, supply instructions on the method to generate these pkl files.

Thank you

The wrong constant rg was used in RepoCoder

CONSTANTS.gt is groundtruth approach
CONSTANTS.rg is vanilla retrieval-augmented approach

In GroundTruthWindowMaker, it should uses CONSTANTS.gt, but not CONSTANTS.rg. The incorrect constant will result in the inability to proceed with subsequent steps

[RepoCoder] Benchmark test file not found: datasets/random-api-completion.test.jsonl

Hi Authors,

Thank you for the amazing work on RepoCoder!

We wanted to test the performance of just In-file code completion (the "In-file" column in Table 2 (a,b)) - i.e. without any retrieved context, we noticed that the actual benchmark files are present here

CodeT/RepoCoder/utils.py

Line 23 in 35f54d6

api_completion_benchmark = 'datasets/random-api-completion.test.jsonl'

'datasets/random-api-completion.test.jsonl', 'datasets/random-line-completion.test.jsonl'

But we cannot find these files even after unzipping the datasets folder. Could you please upload those files, or suggest if we are doing something wrong here

This repo is missing important files

There are important files that Microsoft projects should all have that are not present in this repository. A pull request has been opened to add the missing file(s). When the pr is merged this issue will be closed automatically.

Microsoft teams can learn more about this effort and share feedback within the open source guidance available internally.

Merge this pull request

This repo is missing a LICENSE file

This repository is currently missing a LICENSE file.

A license helps users understand how to use your project in a compliant manner. You can find the standard MIT license Microsoft uses at: https://github.com/microsoft/repo-templates/blob/main/shared/LICENSE.

If you would like to learn more about open source licenses, please visit the document at https://aka.ms/license (Microsoft-internal guidance).

The version of deepspeed.

Which version of deepspeed are you using, I am using version 0.7.7 but it reports an error in Trainer.

Traceback (most recent call last):
  File "run_ner.py", line 411, in <module>
    main()
  File "run_ner.py", line 346, in main
    trainer.train(Traceback (most recent call last):

  File "/home/songyiping/anaconda3/envs/diverse/lib/python3.8/site-packages/transformers/trainer.py", line 1113, in train
  File "run_ner.py", line 411, in <module>
Traceback (most recent call last):
  File "run_ner.py", line 411, in <module>
Traceback (most recent call last):
  File "run_ner.py", line 411, in <module>
    deepspeed_engine, optimizer, lr_scheduler = deepspeed_init(    
  File "/home/songyiping/anaconda3/envs/diverse/lib/python3.8/site-packages/transformers/integrations.py", line 517, in deepspeed_init
main()
  File "run_ner.py", line 346, in main
    main()
  File "run_ner.py", line 346, in main
    model, optimizer, _, lr_scheduler = deepspeed.initialize(
  File "/home/songyiping/anaconda3/envs/diverse/lib/python3.8/site-packages/deepspeed/__init__.py", line 125, in initialize
        trainer.train(
trainer.train(
    engine = DeepSpeedEngine(args=args,
      File "/home/songyiping/anaconda3/envs/diverse/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 290, in __init__
  File "/home/songyiping/anaconda3/envs/diverse/lib/python3.8/site-packages/transformers/trainer.py", line 1113, in train
  File "/home/songyiping/anaconda3/envs/diverse/lib/python3.8/site-packages/transformers/trainer.py", line 1113, in train
main()
  File "run_ner.py", line 346, in main
    self._configure_distributed_model(model)
  File "/home/songyiping/anaconda3/envs/diverse/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 1106, in _configure_distributed_model
    trainer.train(
    deepspeed_engine, optimizer, lr_scheduler = deepspeed_init(  File "/home/songyiping/anaconda3/envs/diverse/lib/python3.8/site-packages/transformers/trainer.py", line 1113, in train

  File "/home/songyiping/anaconda3/envs/diverse/lib/python3.8/site-packages/transformers/integrations.py", line 517, in deepspeed_init
    deepspeed_engine, optimizer, lr_scheduler = deepspeed_init(
  File "/home/songyiping/anaconda3/envs/diverse/lib/python3.8/site-packages/transformers/integrations.py", line 517, in deepspeed_init
    self._broadcast_model()    model, optimizer, _, lr_scheduler = deepspeed.initialize(
  File "/home/songyiping/anaconda3/envs/diverse/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 1022, in _broadcast_model

  File "/home/songyiping/anaconda3/envs/diverse/lib/python3.8/site-packages/deepspeed/__init__.py", line 125, in initialize
    model, optimizer, _, lr_scheduler = deepspeed.initialize(
  File "/home/songyiping/anaconda3/envs/diverse/lib/python3.8/site-packages/deepspeed/__init__.py", line 125, in initialize
        engine = DeepSpeedEngine(args=args,    
engine = DeepSpeedEngine(args=args,deepspeed_engine, optimizer, lr_scheduler = deepspeed_init(  File "/home/songyiping/anaconda3/envs/diverse/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 290, in __init__


  File "/home/songyiping/anaconda3/envs/diverse/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 290, in __init__
  File "/home/songyiping/anaconda3/envs/diverse/lib/python3.8/site-packages/transformers/integrations.py", line 517, in deepspeed_init
    groups._get_broadcast_src_rank(),
  File "/home/songyiping/anaconda3/envs/diverse/lib/python3.8/site-packages/deepspeed/utils/groups.py", line 338, in _get_broadcast_src_rank
    self._configure_distributed_model(model)
      File "/home/songyiping/anaconda3/envs/diverse/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 1106, in _configure_distributed_model
self._configure_distributed_model(model)
          File "/home/songyiping/anaconda3/envs/diverse/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 1106, in _configure_distributed_model
model, optimizer, _, lr_scheduler = deepspeed.initialize(
return dist.get_global_rank(_get_data_parallel_group(), 0)  File "/home/songyiping/anaconda3/envs/diverse/lib/python3.8/site-packages/deepspeed/__init__.py", line 125, in initialize

  File "/home/songyiping/anaconda3/envs/diverse/lib/python3.8/site-packages/deepspeed/comm/__init__.py", line 22, in get_global_rank
        engine = DeepSpeedEngine(args=args,if hasattr(torch.distributed.distributed_c10d, "get_global_rank"):    self._broadcast_model()


  File "/home/songyiping/anaconda3/envs/diverse/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 1022, in _broadcast_model
AttributeError  File "/home/songyiping/anaconda3/envs/diverse/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 290, in __init__
: module 'deepspeed.comm.torch' has no attribute 'distributed'
    self._broadcast_model()
  File "/home/songyiping/anaconda3/envs/diverse/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 1022, in _broadcast_model
    self._configure_distributed_model(model)
  File "/home/songyiping/anaconda3/envs/diverse/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 1106, in _configure_distributed_model
    groups._get_broadcast_src_rank(),
  File "/home/songyiping/anaconda3/envs/diverse/lib/python3.8/site-packages/deepspeed/utils/groups.py", line 338, in _get_broadcast_src_rank
    groups._get_broadcast_src_rank(),
  File "/home/songyiping/anaconda3/envs/diverse/lib/python3.8/site-packages/deepspeed/utils/groups.py", line 338, in _get_broadcast_src_rank
        return dist.get_global_rank(_get_data_parallel_group(), 0)self._broadcast_model()
  File "/home/songyiping/anaconda3/envs/diverse/lib/python3.8/site-packages/deepspeed/comm/__init__.py", line 22, in get_global_rank

  File "/home/songyiping/anaconda3/envs/diverse/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 1022, in _broadcast_model
        return dist.get_global_rank(_get_data_parallel_group(), 0)if hasattr(torch.distributed.distributed_c10d, "get_global_rank"):

  File "/home/songyiping/anaconda3/envs/diverse/lib/python3.8/site-packages/deepspeed/comm/__init__.py", line 22, in get_global_rank
AttributeError: module 'deepspeed.comm.torch' has no attribute 'distributed'
        if hasattr(torch.distributed.distributed_c10d, "get_global_rank"):
groups._get_broadcast_src_rank(),
AttributeError  File "/home/songyiping/anaconda3/envs/diverse/lib/python3.8/site-packages/deepspeed/utils/groups.py", line 338, in _get_broadcast_src_rank
: module 'deepspeed.comm.torch' has no attribute 'distributed'
    return dist.get_global_rank(_get_data_parallel_group(), 0)
  File "/home/songyiping/anaconda3/envs/diverse/lib/python3.8/site-packages/deepspeed/comm/__init__.py", line 22, in get_global_rank
    if hasattr(torch.distributed.distributed_c10d, "get_global_rank"):
AttributeError: module 'deepspeed.comm.torch' has no attribute 'distributed'

I have not made any changes to the code and think there may be a problem with one of the library versions, so I hope you can help me.

Issue reproducing codex numbers

Dear authors,
Thank you for releasing the data accompanying CodeT. However, I was having some trouble reproducing the Codex numbers reported in the paper. The original paper reports 47.0 @1, 74.9 @10, 92.1 @100 for Codex. I tried to calculate the scores using the data provided in the repository.

First, I converted it to a format compatible with the official OpenAI HumanEval script.

import argparse
import json
from human_eval.data import read_problems

parser = argparse.ArgumentParser()
parser.add_argument('--results_file', default="codet-generations/HumanEval_davinci002_temp0.8_topp0.95_num100_max300_code_solution.jsonl")
parser.add_argument('--outputs_file', default="codet-generations/HumanEval_davinci002_temp0.8_topp0.95_num100_max300_code_solution_outputs.jsonl")
args = parser.parse_args()

with open(args.results_file, "r") as f:
    results = [json.loads(line) for line in f.read().strip().split("\n")]

problems = read_problems()
problem_ids_prompts = [(x, problems[x]["prompt"]) for x in problems]

outputs = []

for res in results:
    # find task_id
    task_ids = []
    def_line = [i for i, x in enumerate(res["prompt"].split("\n")) if x.startswith("def ")][0]
    def_line = "\n".join(res["prompt"].split("\n")[def_line:def_line + 2])
    for task_id1, prompt1 in problem_ids_prompts:
        if def_line in prompt1:
            task_ids.append(task_id1)
    assert len(task_ids) == 1
    for sample in res["samples"]:
        outputs.append(json.dumps({
            "task_id": task_ids[0],
            "completion": sample
        }))

with open(args.outputs_file, "w") as f:
    f.write("\n".join(outputs) + "\n")

After running this through the OpenAI script, I get {'pass@1': 0.1653, 'pass@10': 0.6257, 'pass@100': 0.8537}. What could I be doing incorrectly here? Also, any reason to choose n=100 instead of n=200 like Chen et al. 2021 and CodeGen paper? I'm guessing the pass@100 estimation will be more accurate with n=200?

Thank you!
Kalpesh