huybery / awesome-code-llm

👨‍💻 An awesome and curated list of best code-LLM for research.

Hey, I'm huybery

class Attributes(huybery):
    @staticmethod
    def contact() -> tuple:
        homepage  = "https://huybery.github.io"
        twitter   = "https://twitter.com/huybery"
        email     = "huybery [at] gmail.com"
        return homepage, twitter, email

    @staticmethod
    def research() -> tuple:
        interesting = ['Large Language Models', 'Executable Language', 'Embodied Agent', 'Dialog Systems']
        interesting.pop()   # chatGPT is all your need.
        paper = "https://scholar.google.com/citations?user=RBb3ItMAAAAJ"
        return interesting, paper

    @staticmethod
    def project() -> list:
        OpenDevin = "https://github.com/OpenDevin/OpenDevin"
        Qwen = "https://github.com/QwenLM/Qwen"
        CodeQwen = "https://github.com/QwenLM/CodeQwen1.5"
        OctoPack = "https://github.com/bigcode-project/octopack"
        Awesome_Code_LLM = "https://github.com/huybery/Awesome-Code-LLM"
        project_lst = [OpenDevin, Qwen, CodeQwen, OctoPack, Awesome_Code_LLM]
        return project_lst

awesome-code-llm's Issues

tiny typo on GPT-4 "Reflexion" in leaderboard

First off, if you put this all together yourself I salute you! Great resource!

Secondly, any reason why Claude/Claude 2 were left out of the ranking? me thinks it would be 3rd on your list:)

MetaGPT Results?

Have you added the results for MetaGPT (https://arxiv.org/pdf/2308.00352.pdf) to the leaderboard?

Link for GPT-4 in README is incorrect

Link for GPT-4 in README points to DeepSeek-Coder

Remove the Steve Jobs quote

Steve Jobs wasn't a coder, but don't take my word for it.

https://web.archive.org/web/20160322073045/http://www.woz.org/letters/does-steve-jobs-know-how-code

Is the score in the Leaderboard reliable and real-time?

Hi there,

Thanks for your great curated list of Code LLMs. I really love it. 🙏
I wanna know where the data in the Leaderboard (Sort by HumanEval Pass@1) comes from. Is it reliable and real-time?
I find the score of GPT4 (67.0 Pass@1) is different from paperswithcode (86.6 Pass@1) https://paperswithcode.com/sota/code-generation-on-humaneval.
Could you check it one more time? Thanks a lot. 😄

Can you add DeepSeek Coder?

Can you add DeepSeek Coder?

Suggestions on searching for RLHF-ed code models

Hi,

Thank you for putting up this repo; it's extremely helpful for someone like me who's new to this field.

Sorry that I kinda abuse issues for asking questions, but I was wondering if you are aware of, or have any suggestions for finding, instruction-based code models that have been trained with RLHF? I only know that codellama is RLHF-ed, but I was wondering if there are more of them.

Thank you for your time.

Magicoder in the list?

Magicoder seems perform pretty good in humaneval and mbpp benchmarks:

https://arxiv.org/pdf/2312.02120.pdf

Perhaps the leadership list can be updated.

Can we initiate a leaderboard section instead of showing single one?

I found there are a lot of leaderboards on code generation with LLMs.
Such as https://huggingface.co/spaces/bigcode/bigcode-models-leaderboard, https://intercode-benchmark.github.io/, and more.
Rather than statically showing a single leaderboard, we add a table to include as many as links instead to redirect users to refer based on their preference, just like tjunlp-lab/Awesome-LLMs-Evaluation-Papers#13.