Rai220/car_robot

esp32 car expirements

★ 3Forks 0PythonGitHub ↗Compare

README

car_robot

Experiments toward a universal robot driver — an AI that can pilot almost any robot regardless of how it's built — running on a ~$20 ESP32-CAM toy car.

Live dashboard while the car searches for a target

tools/dashboard.py observability UI mid-task: the raw 320×240 camera (left), the Depth-Anything-V2 free-space heatmap with the look-ahead band and floor reference (center), and the live VLM decision log with per-zone openness and on-rug confidence (right).

Background & motivation (RU): «Отпускной пост про DIY-роботов».

Idea

One model isn't enough. Like a person — Kahneman's Thinking, Fast and Slow — the robot needs a fast reflexive system and a slow, deliberate one, and here a third system that watches both and improves them over time. The bet: just as large language models reshaped NLP, a general "robot driver" will do the same for robotics — and the path there is composing fast / slow / self-improving layers rather than training one monolithic end-to-end policy.

Three systems

  1. System 1 — fast & reflexive — tools/reflex_drive.py + tools/depth_perception.py. Depth-Anything-V2 splits each frame into left / center / right, scores where it's most open, and steers in real time. Dumb and fast: it owns the wheels and close-range obstacle avoidance. The same depth signal acts as the emergency bumper inside the ER driver below.

  2. System 2 — slow & smart — tools/er_drive.py, the primary mission driver: a visual-action loop on Gemini Robotics-ER 1.6. ER looks through the camera and emits one bounded parametric command at a time (action / duration / turn-degrees / trim + a target point); the driver executes it, measures what actually happened from the camera, feeds that back, and self-calibrates. Local Qwen2.5-VL / FastVLM remain as offline alternatives inside reflex_drive.py/vlm_local.py. Shared primitives: tools/carlib.py.

  3. System 3 — self-improvement (the ralph loop) — tools/evolve_loop.sh + tools/self_improve_loop.sh. The car drives a bounded episode, then a coding agent (Claude Code or Codex) reads the episode's logs and annotated frames plus the persistent MISSION.md/PROGRESS.md state, and may rewrite the prompts and logic of Systems 1–2 — a psychotherapist for the robot, fixing deep mechanisms rather than momentary decisions. Then the next episode starts with the new code. It is bounded by a file allowlist, before/after source-hash checks, per-iteration backups with automatic rollback on red tests, and a unittest suite it is not allowed to edit.

See docs/self_improvement.md for the harness and run commands. (tools/gemini_drive.py, tools/seek.py, tools/vlm_drive.py are earlier single-loop drivers kept as reference.)

Hardware

A cheap ESP32-CAM toy car (camera + WiFi + I²C motor driver), ~1800 ₽ / ~$20. Firmware lives in firmware/VideoCar/; see firmware/README.md for the build/flash toolchain and the HTTP control API.

Contributors

Rai220

Issues