metauto-ai / gptswarm

🐝 GPTSwarm: LLM agents as (Optimizable) Graphs

🐝 GPTSwarm is a graph-based framework for LLM-based agents, providing two high-level features:

• It lets you build LLM-based agents from graphs.
• It enables the customized and automatic self-organization of agent swarms with self-improvement capabilities.

News

• 🔥 [05/01] GPTSwarm has been accepted by ICML2024.

• 🔥 [03/01] GPTSwarm can be installed via pip now: pip install gptswarm

• 🔥 [02/27] Our academic paper: Language Agents as Optimizable Graphs is released.

Edge optimization example

Here is the edge optimization process that updates edge probabilities toward improvement of the benchmark score. Notice that within an agent, the edges are fixed, whereas the inter-agent connections are getting optimized towards either edge pruning (value 0, blue) or creation (value 1, red).

About GPTSwarm

At a granular level, GPTSwarm is a library that includes the following components:

Module	Description
swarm.environment	Domain-specific operations, agents, tools, and tasks
swarm.graph	Graph-related functions for creating and executing agent graphs and swarm composite graphs
swarm.llm	Interface for selecting LLM backends and calculating their operational costs
swarm.memory	Index-based memory
swarm.optimizer	Optimization algorithms designed to enhance agent performance and overall swarm efficiency

Quickstart

Clone the repo

git clone https://github.com/metauto-ai/GPTSwarm.git
cd GPTSwarm/

Install packages

conda create -n swarm python=3.10
conda activate swarm
pip install poetry
poetry install

You should add API keys in .env.template and change its name to .env

OPENAI_API_KEY="" # for OpenAI LLM backend
SEARCHAPI_API_KEY="" # for Web Search

Getting started with GPTSwarm is easy. Quickly run a predefined swarm

from swarm.graph.swarm import Swarm

swarm = Swarm(["IO", "IO", "IO"], "gaia")
task = "What is the capital of Jordan?"
inputs = {"task": task}
answer = await swarm.arun(inputs)

or make use of tools, such as the file analyzer

from swarm.graph.swarm import Swarm
swarm = Swarm(["IO", "TOT"], "gaia")
task = "Tell me more about this image and summarize it in 3 sentences."
files = ["./datasets/demos/js.png"]
inputs = {"task": task, "files": files}
danswer = swarm.run(inputs)

Check out the minimal Swarm example in Colab here: .

See how to create a custom Agent and run a Swarm with it here: .

Here is a Youtube video on how to run the demo notebooks:

🔥🔥🔥 See our experiments for more advanced use of our framework.

Class diagram

Example of the Swarm

More Visualizations

Running with a local LLM

We support local LM inference via LM Studio. Download their desktop app for Mac or Windows, choose a model from the Huggingface repository and start the server. Use model_name='lmstudio' in GPTSwarm code to run with the local LLM.

Contributors

• Mingchen Zhuge (PhD Student@KAUST; Project Initiator)
• Wenyi Wang (PhD Student @ KAUST; Initial Participant)
• Dmitrii Khizbullin (Research Engineer Lead @ KAUST; Project Engineer Lead)
• Louis Kirsch (PhD Student @ IDSIA)
• Francesco Faccio (PostDoc@IDSIA; Visiting Researcher@KAUST)
• Jürgen Schmidhuber (Director @ KAUST AI Initiative; Scientific Director @ IDSIA)
• William FH (Engineering @ LangChain)
• Lukas Vierling (BS Student @ RWTH & HKUST)

Please read our developer document if you are interested in contributing.

Citation

Please cite our paper if you find the library useful or interesting.

@article{zhuge2024language,
  title={Language Agents as Optimizable Graphs},
  author={Zhuge, Mingchen and Wang, Wenyi and Kirsch, Louis and Faccio, Francesco and Khizbullin, Dmitrii and Schmidhuber, Jurgen},
  journal={arXiv preprint arXiv:2402.16823},
  year={2024}
}

gptswarm's Issues

Update demo notebooks for the open-source repo

Try move to poetry as package config

Retry Error When running the example

2024-05-01 17:00:25.721 | ERROR | swarm.graph.node:execute:141 - Node DirectAnswer failed to execute due to: RetryError: RetryError[<Future at 0x314e4fcd0 state=finished raised OpenAIError>]
2024-05-01 17:03:04.397 | ERROR | swarm.graph.node:execute:141 - Node DirectAnswer failed to execute due to: RetryError: RetryError[<Future at 0x314e4ff40 state=finished raised OpenAIError>]
2024-05-01 17:07:03.721 | ERROR | swarm.graph.node:execute:141 - Node DirectAnswer failed to execute due to: RetryError: RetryError[<Future at 0x314e84100 state=finished raised OpenAIError>]

---

Showing the above error when running the examples, the local openai connection is okay.

Replace mock_llm with uses_openai pytest marker

Add code coverage with Coveralls + badge

https://coveralls.io/github/metauto-ai/GPTSwarm

Remove find_project_root "automation"

GPTSWARM_ROOT = find_project_root()

@mczhuge FYI this does not work with packages.

Issue with Asynchronous Multi-Agent Commands using LMStudio API Causing Incorrect Output Assignment

Hi,

Thanks for this useful framework!

There is an issue while running multi-agent commands that require asynchronous use of LMStudio API. For instance, running PYTHONPATH=. python experiments/run_mmlu.py --num-truthful-agents=3 --mode=OptimizedSwarm, the outputs of LMStudio seem to be not assigned to the relevant question but to random input questions.

I have not tested this with Open AI API to see if this is specifically an incompatibility issue of LMStudio.

My current temporary solution is not to use the asynchronous implementation but this is annoyingly very slow. Have you encountered this? Is there a way to still use your asynchronous implementation and not experience this issue?

Thanks so much!

Publish the package to PyPI

[BUG] List instead of str in FinalDecision

    output: Foo Bar Asy
2024-02-27 15:46:46.445 | ERROR    | swarm.graph.node:execute:141 - Node FinalDecision failed to execute due to: AttributeError: 'list' object has no attribute 'strip'
2024-02-27 15:46:46.446 | INFO     | swarm.graph.node:log:160 - Memory Records for ID 3BUm:
    operation: GenerateQuery
    files: ['datasets/demos/agi.txt']
    subtask: # Information Gathering for Question Resolution

Double check the initial release

@Obs01ete @Wenyi-AI-Wang Hi, Dmitrii and Wenyi. Thanks for your effort!

These are the remain work for us.

• Please help with the dataset upload (git-lfs)
• Check the GitHub setting of project (automated test)
• Upload the ArXiv link and right format of paper citation