wyl-willing / intercode

[NeurIPS 2023 D&B] Code repository for InterCode benchmark https://arxiv.org/abs/2306.14898

🔄 InterCode

Build interactive code environments for interactive code agents.

Please refer to the change log for information on the latest updates to the InterCode environment.

👋 Overview

InterCode is a lightweight, flexible, and easy-to-use framework for designing interactive code environments. Following the popular gym interface definition, InterCode makes it easy to quickly define a code environment and deploy an agent to operate in code within the context of the environment.

For an overview of InterCode, building interactive code tasks with InterCode, and evaluating agents on InterCode environments, please check out our wiki, website, and the original paper:

InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedback
John Yang, Akshara Prabhakar, Karthik Narasimhan, Shunyu Yao

🚀 Quick Start

You can install InterCode as a PyPI package or by building from source.

Note InterCode requires the following installations to run:

• python >= 3.8
• docker: Learn more here to install. Before running the below code, make sure the Docker daemon/application is running locally.

🐍 PyPI Package

1. Install the (pypi package):

pip install intercode-bench

2. Copy + Paste the following code for interacting with the InterCode Bash environment into a python file (i.e. run_bash.py)

import readline
from intercode.assets import bash_build_docker, bash_image_name, bash_test_data
from intercode.envs import BashEnv

if __name__ == '__main__':
    bash_build_docker()
    env = BashEnv(bash_image_name, data_path=bash_test_data, traj_dir="logs/", verbose=True)

    try:
        for idx in range(3):
            env.reset()
            obs, done = env.observation, False
            while not done:
                action = input('> ')
                obs, reward, done, info = env.step(action)
    except KeyboardInterrupt:
        print("Keyboard interrupt detected")
    finally:
        env.close()

3. Run the file (i.e. python run_bash.py)

If InterCode was installed successfully, the InterCode Bash environment should be started successfully and a CLI interpreter should appear, allowing you to enter bash commands to interact with the task setting. You can ^c at any to time to exit the environment. Similar starter code for the InterCode SQL environment is available on the PyPI page.

💽 Build from Source

1. Clone this repository, create a virtual environment, and install necessary dependencies

git clone https://github.com/princeton-nlp/intercode.git
cd intercode
conda env create -f environment.yml
conda activate intercode

2. Run setup.sh to create the docker images for the InterCode Bash, CTF, Python, and SQL environments
3. Run python run_demo.py sql

If InterCode was installed successfully, the InterCode SQL environment should be started successfully and a CLI interpreter should appear, allowing you to enter SQL commands to interact with the task environment. You can ^c at any to time to exit the environment. You can run python run_demo.py [bash|ctf|python|sql] to interact with any of the current four InterCode environments.

python docker/utils/python_server.py

🔎 Learn More

If you'd like to...

• Get a more in depth, but still brief overview of InterCode, see here
• Access the InterCode Bash, CTF, Python, or SQL environment, see here
• Build an interactive code task with InterCode, see here
• Run language and code agents on InterCode based environments, see here

Not seeing what you want? Please feel free to check the wiki and paper for more details, or raise an issue if you still can't find it.

💫 Contributions

We would love to hear from the broader NLP and Machine Learning community, and we welcome any contributions, pull requests, or issues! To do so, please either file a new pull request or issue and fill in the corresponding templates accordingly. We'll be sure to follow up shortly!

Contact person: John Yang

✍️ Citation

If you find this repository helpful, feel free to cite our publication.

@inproceedings{yang2023intercode,
  title = {InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedback},
  author = {Yang, John and Prabhakar, Akshara and Narasimhan, Karthik and Yao, Shunyu},
  booktitle = {ArXiv},
  year = {2023},
  html = {https://arxiv.org/abs/2207.01206}
}

🪪 License

MIT. Check LICENSE.md.