A PyBullet robot-arm simulation controlled by natural-language commands. The "brain" runs entirely on-device via Foundry Local – no cloud APIs, no API keys.

The system uses the Microsoft Agent Framework to orchestrate multiple specialised agents that collaborate to translate natural language into safe, validated robot actions:

The Orchestrator manages the pipeline: Planner → Safety → Executor → Narrator. If the safety check rejects a plan, execution is skipped and the narrator explains why.

Voice mode records audio from the microphone, transcribes it locally using a Whisper model served by Foundry Local, and then feeds the resulting text into the same planning pipeline.

A FastAPI web server provides a browser-based interface with three panels:

• Camera – live view of the PyBullet simulation (auto-refreshing at 2 fps)
• Chat – type commands and see structured responses from the agent pipeline
• Agent Status – real-time cards showing each agent's progress and results

The web UI communicates via REST endpoints and a WebSocket for live updates.

• GPU recommended for faster LLM inference but CPU works fine.
• Voice mode needs a working microphone and the sounddevice library (requires PortAudio on Linux: sudo apt install libportaudio2).

# Windows
winget install Microsoft.FoundryLocal

# macOS
brew tap microsoft/foundrylocal && brew install foundrylocal

# Verify
foundry --version

# List available models
foundry model list

# Fastest option (~5 s per command) – recommended for interactive use
foundry model run qwen2.5-coder-0.5b

# Higher accuracy (~35 s per command)
foundry model run phi-4-mini

The model server stays active in the background. You can switch models at any time via the dropdown in the web UI.

The Whisper model is downloaded automatically by the foundry-local-sdk on first use. No manual step needed – just make sure Foundry Local is installed.

The project uses a .venv virtual environment. Use the provided setup script to create it, activate it, and install all dependencies in one step:

# Windows (PowerShell) – recommended
cd robot-simulator-foundrylocal
.\setup.ps1

# Windows (cmd)
cd robot-simulator-foundrylocal
setup.bat

# macOS / Linux
cd robot-simulator-foundrylocal
chmod +x setup.sh
./setup.sh
source .venv/bin/activate

cd robot-simulator-foundrylocal
python -m venv .venv

# Activate:
#   Windows PowerShell:  .\.venv\Scripts\Activate.ps1
#   Windows cmd:         .venv\Scripts\activate.bat
#   macOS/Linux:         source .venv/bin/activate

pip install --upgrade pip
pip install -r requirements.txt

Note: Always ensure the .venv is activated before running the app. Your prompt will show (.venv) when active.

Use the provided start scripts to launch the application. The default mode opens the web UI in your browser:

# Windows (PowerShell)
.\start.ps1              # Web UI (default) → http://localhost:8080
.\start.ps1 --cli        # CLI mode instead
.\start.ps1 --no-gui     # Web UI without PyBullet window

# Windows (cmd)
start.bat                # Web UI (default)
start.bat --cli          # CLI mode

# macOS / Linux
chmod +x start.sh
./start.sh               # Web UI (default) → http://localhost:8080
./start.sh --cli          # CLI mode

Or run directly with Python:

# Web UI (recommended)
python -m src.app --web

# CLI text mode
python -m src.app

# Voice mode – speak commands into the microphone
python -m src.app --mode voice

# Dry-run mode – see what the planner would do, without executing
python -m src.app --dry-run

# Custom target object
python -m src.app --object path/to/my_object.urdf

# Headless web UI (no PyBullet window)
python -m src.app --web --no-gui

🤖 Enter command (or 'quit'): pick up the cube
🤖 Enter command (or 'quit'): describe the scene
🤖 Enter command (or 'quit'): move to x 0.4 y 0.1 z 0.3 then open gripper
🤖 Enter command (or 'quit'): grab the blue object and place it at x 0.6 y -0.2 z 0.65
🤖 Enter command (or 'quit'): reset
🤖 Enter command (or 'quit'): quit

Run with --mode voice, then speak any of the above commands when prompted. You have 5 seconds (configurable via --record-seconds) to speak.

The simulator supports voice commands through Foundry Local Whisper — all transcription happens on-device with zero cloud dependencies.

  Browser mic ──► MediaRecorder (WebM/Opus)
       │
       ▼
  Client-side resample to 16 kHz mono WAV (OfflineAudioContext)
       │
       ▼
  POST /api/voice  (multipart form-data)
       │
       ▼
  Server: transcribe_with_chunking()
       ├── audio ≤ 30 s → single Whisper pass
       └── audio > 30 s → split into ≤ 30 s chunks, transcribe each, concatenate
       │
       ▼
  Transcribed text sent through the normal agent pipeline
  (Planner → Safety → Executor → Narrator)

1. A Whisper model must be cached or loaded in Foundry Local. The mic button (🎤) only appears in the web UI when this condition is met.

   # Download a Whisper model (first time only — pick one):
   foundry model run whisper-tiny         # 75 MB – fastest (recommended)
   foundry model run whisper-base         # 194 MB – fast, good accuracy
   foundry model run whisper-small        # 542 MB – slower, better accuracy
   foundry model run whisper-medium       # 1.5 GB – slowest, best accuracy

2. Browser microphone permission — the browser will prompt on first use. HTTPS is not required for localhost.

3. CLI voice mode additionally needs sounddevice + PortAudio:

   pip install sounddevice
   # Linux only:
   sudo apt install libportaudio2

1. Start the app: python -m src.app --web
2. Open http://localhost:8080
3. The 🎤 button appears next to the text input if a Whisper model is available
4. Hold the 🎤 button and speak your command
5. Release the button — the audio is sent to the server
6. The transcribed text appears in the chat as "Heard: ..."
7. The command is then automatically executed through the agent pipeline

Tip: Keep commands under 30 seconds. Longer audio is automatically chunked into 30-second segments and transcribed in sequence.

python -m src.app --mode voice
# Speak when you see: 🎤 Recording for 5s – speak now …

# Longer recording window:
python -m src.app --mode voice --record-seconds 10

Overview	Reaching	Above Cube

Gripper Detail	Front View	Side View

Describe Scene	Pick Cube

Move End-Effector	Reset

robot-simulator-foundrylocal/
├── .gitignore
├── AGENTS.md                # AI agent instructions
├── CONTRIBUTING.md          # Contribution guidelines
├── LICENSE                  # MIT licence
├── README.md
├── requirements.txt
├── setup.ps1                # Windows PowerShell setup (venv + deps)
├── setup.bat                # Windows cmd setup
├── setup.sh                 # macOS / Linux setup
├── start.ps1                # Launch app (PowerShell)
├── start.bat                # Launch app (cmd)
├── start.sh                 # Launch app (bash)
├── docs/
│   ├── gallery.html         # Screenshot gallery page
│   └── screenshots/         # Rendered PyBullet images (1920×1080)
├── tests/
│   ├── test_action_schema.py
│   ├── test_agents.py       # Agent framework tests
│   ├── test_config.py
│   ├── test_executor.py
│   ├── test_planner.py
│   ├── test_scene_helpers.py
│   └── test_render_screenshots.py
└── src/
    ├── __init__.py
    ├── __main__.py          # python -m src entry point
    ├── app.py               # main entry – CLI or web mode
    ├── config.py            # CLI args + env vars → Config
    ├── web_ui.py            # FastAPI server (REST + WebSocket + camera)
    ├── agents/
    │   ├── __init__.py
    │   ├── orchestrator.py  # Multi-agent pipeline coordinator
    │   ├── planner_agent.py # NL command → JSON action plan
    │   ├── safety_agent.py  # Validates plans against bounds & schema
    │   ├── executor_agent.py# Dispatches actions to PyBullet
    │   └── narrator_agent.py# Scene description & action narration
    ├── simulation/
    │   ├── __init__.py
    │   ├── scene.py         # ground, table, objects
    │   ├── robot.py         # Franka Panda IK + joint control
    │   └── grasp.py         # pick/place sequences
    ├── brain/
    │   ├── __init__.py
    │   ├── foundry_client.py # Foundry Local OpenAI-compat client
    │   ├── action_schema.py  # strict JSON action schema + validator
    │   └── planner.py        # NL → validated action plan via LLM
    ├── input/
    │   ├── __init__.py
    │   ├── text_input.py     # interactive CLI prompt
    │   └── voice_input.py    # mic recording + Whisper transcription
    ├── executor/
    │   ├── __init__.py
    │   └── action_executor.py # maps actions → simulation calls
    └── static/
        ├── index.html        # Web UI single-page app
        ├── style.css         # Dark-theme styles
        └── app.js            # Camera, chat, agent status logic

# List all available Foundry Local models
foundry model list

# Run a different model
foundry model run phi-3.5-mini

# Tell the app to use it
python -m src.app --model phi-3.5-mini

The SDK auto-selects the best hardware variant (CUDA GPU → QNN NPU → CPU).

Speed depends on model size and your hardware (GPU vs CPU):

Note: Sizes shown are for CPU model variants. GPU (CUDA) variants are smaller: 528 MB, 3.6 GB, and 4.7 GB respectively. The SDK auto-selects the best hardware backend for your system.

Tip: For the fastest experience, use qwen2.5-coder-0.5b. You can switch models at any time via the dropdown in the web UI without restarting.

MIT