triagelab / triage

An AI assistant for open-source communities

trIAge

An AI assistant for open source communities

We develop trIAge, an AI-driven assistant that supports users, maintainers and contributors of open source projects.

About

Leveraging the capabilities of Large Language Models (LLMs), trIAge is able to analyze issues, discussions and pull requests on collaboration platforms (e.g. GitHub, GitLab) and provide users and maintainers with hints and suggested solutions by responding on the discussion thread. trIAge has access to the context of the project (code, documentation, guidelines) and can be configured by maintainers to become gradually active as needed. The bot automates part of the workload of maintainers and helps users to solve their issues faster, e.g. by automatically answering questions from documentation, or generating test cases for reported issues. Current chat models like GPT-4 have shown remarkable abilities in the relevant natural language and code understanding tasks. This enables a wide range of automation with respect to issue triage, issue quality control, debugging, user support, testing and documentation. As an open-source project, we are aiming to eventually base the bot’s capabilities on emerging open-source models (e.g. Vicuna, Orca), whose capabilities are rapidly catching up to the current gold standard (GPT-4).

The MVP of trIAge can easily be deployed by open source maintainers on their repositories in the form of a GitHub app, which we host as a service for demonstration purposes. We also create the option for self-hosting of the application. We add a convenient user interface for maintainers to customize the desired capabilities and action triggers of the assistant. Subsequently we plan to support other code hosting platforms (e.g. GitLab) as well.

Skills: The Assistant's Capabilities

under construction

• Support: answer questions based on documentation, generate explanatory code examples
• Issue Quality Control: analyze issues for quality of description and reproducibility, provide feedback for improvement
• Issue Triage: Automatically categorize issues by content (feature request, bug, support) and tag them, detect duplicates, prioritize issues (e.g. security issues), link related issues
• Debugging: Analyze error causes, generate solution suggestions
• Testing: automatic generation of test cases based on issues
• Pull Request Review: Analysis of pull requests regarding code quality, code style, documentation, test coverage, etc.
• Documentation: Generation of documentation proposals from the project context
• Changelogs: Generation of changelogs from the project history

triage's Issues

Skill: Answer from documentation

Skill description

The issue tracker is often used to ask questions that could be answered from documentation already. The bot checks the documentation for the answer and retrieves it if applicable.

Connect to https://goodfirstissue.dev

https://goodfirstissue.dev

should provide a great test set for trIAge.

bot test issue

evaluate: LlamaIndex

https://github.com/jerryjliu/llama_index

Skill: Conform to issue template

Skill description

Many projects provide an issue template. trIAge can check whether a reported issue deviates in a problematic way from this template. trIAge can point this out to the user and suggest improvements to the issue based on the template.

evaluate Haystack

Haystack may be an alternative to langchain

Model: Orca

Evaluate the Orca model

https://www.youtube.com/watch?v=Dt_UNg7Mchg

Skill: Test case generation

Skill description

Given a well-described bug report, the trIAge bot could generate test cases that demonstrate the bug and would pass if it was fixed. Maintainers could incorporate these test cases into their debugging workflow.

Demo

• Pandas bug

Skill: Ask for missing information

Skill description

trIAge should check a submitted issue for clarity and completeness, especially with respect to reproducibility.

If important information is missing, the bot should answer and ask the submitting user to provide the information.

Was an Issue already answered in the README?

Check if an answer is already answered somewhere in the README (or other specified document in the repo). Give the answer here and point to the source file.

assistant test issue

Test

Non-Skill: Refuse requests unrelated to job description

If the bot can be invoked with free text requests, it is important that it stays focused on its scope of tasks, does not expose all capabilities of the underlying assistant model, and refuses requests that are out of scope.

Test Issue

This is for testing bot interactions.

eyeingbot test issue

a test for the bot

eyeingbot test issue

This is a test

Skill: Redirect to correct repository

Skill description

Users may report an issue which is actually better addressed to another repository.

• The trIAge bot evaluates whether the issue is misplaced.
• It may also suggest the correct repository to report the project to.

Skill: Issue Formatting

Skill description

Lack of formatting, or uncommon formatting can often interfere with the readability of an issue. Maintainer's time is not well spent on formatting. LLMs are capable of reformatting the issue into a common format, or even into a custom, project-specific format.

evaluate Bard API

https://github.com/dsdanielpark/Bard-API