๐ค LeRobot aims to provide models, datasets, and tools for real-world robotics in PyTorch. The goal is to lower the barrier to entry to robotics so that everyone can contribute and benefit from sharing datasets and pretrained models.
๐ค LeRobot contains state-of-the-art approaches that have been shown to transfer to the real-world with a focus on imitation learning and reinforcement learning.
๐ค LeRobot already provides a set of pretrained models, datasets with human collected demonstrations, and simulation environments to get started without assembling a robot. In the coming weeks, the plan is to add more and more support for real-world robotics on the most affordable and capable robots out there.
๐ค LeRobot hosts pretrained models and datasets on this Hugging Face community page: huggingface.co/lerobot
 |
 |
 |
| ACT policy on ALOHA env | TDMPC policy on SimXArm env | Diffusion policy on PushT env |
- Thanks to Tony Zaho, Zipeng Fu and colleagues for open sourcing ACT policy, ALOHA environments and datasets. Ours are adapted from ALOHA and Mobile ALOHA.
- Thanks to Cheng Chi, Zhenjia Xu and colleagues for open sourcing Diffusion policy, Pusht environment and datasets, as well as UMI datasets. Ours are adapted from Diffusion Policy and UMI Gripper.
- Thanks to Nicklas Hansen, Yunhai Feng and colleagues for open sourcing TDMPC policy, Simxarm environments and datasets. Ours are adapted from TDMPC and FOWM.
- Thanks to Antonio Loquercio and Ashish Kumar for their early support.
Download our source code:
git clone https://github.com/huggingface/lerobot.git && cd lerobotCreate a virtual environment with Python 3.10 and activate it, e.g. with miniconda:
conda create -y -n lerobot python=3.10 && conda activate lerobotInstall ๐ค LeRobot:
pip install .For simulations, ๐ค LeRobot comes with gymnasium environments that can be installed as extras:
For instance, to install ๐ค LeRobot with aloha and pusht, use:
pip install ".[aloha, pusht]"To use Weights and Biases for experiment tracking, log in with
wandb login.
โโโ examples # contains demonstration examples, start here to learn about LeRobot
โโโ lerobot
| โโโ configs # contains hydra yaml files with all options that you can override in the command line
| | โโโ default.yaml # selected by default, it loads pusht environment and diffusion policy
| | โโโ env # various sim environments and their datasets: aloha.yaml, pusht.yaml, xarm.yaml
| | โโโ policy # various policies: act.yaml, diffusion.yaml, tdmpc.yaml
| โโโ common # contains classes and utilities
| | โโโ datasets # various datasets of human demonstrations: aloha, pusht, xarm
| | โโโ envs # various sim environments: aloha, pusht, xarm
| | โโโ policies # various policies: act, diffusion, tdmpc
| | โโโ utils # various utilities
| โโโ scripts # contains functions to execute via command line
| โโโ eval.py # load policy and evaluate it on an environment
| โโโ train.py # train a policy via imitation learning and/or reinforcement learning
| โโโ push_dataset_to_hub.py # convert your dataset into LeRobot dataset format and upload it to the Hugging Face hub
| โโโ visualize_dataset.py # load a dataset and render its demonstrations
โโโ outputs # contains results of scripts execution: logs, videos, model checkpoints
โโโ tests # contains pytest utilities for continuous integration
Check out example 1 that illustrates how to use our dataset class which automatically download data from the Hugging Face hub.
You can also locally visualize episodes from a dataset by executing our script from the command line:
python lerobot/scripts/visualize_dataset.py \
--repo-id lerobot/pusht \
--episode-index 0It will open rerun.io and display the camera streams, robot states and actions, like this:
battery-720p.mov
Our script can also visualize datasets stored on a distant server. See python lerobot/scripts/visualize_dataset.py --help for more instructions.
Check out example 2 that illustrates how to download a pretrained policy from Hugging Face hub, and run an evaluation on its corresponding environment.
We also provide a more capable script to parallelize the evaluation over multiple environments during the same rollout. Here is an example with a pretrained model hosted on lerobot/diffusion_pusht:
python lerobot/scripts/eval.py \
-p lerobot/diffusion_pusht \
eval.n_episodes=10 \
eval.batch_size=10Note: After training your own policy, you can re-evaluate the checkpoints with:
python lerobot/scripts/eval.py \
-p PATH/TO/TRAIN/OUTPUT/FOLDERSee python lerobot/scripts/eval.py --help for more instructions.
Check out example 3 that illustrates how to start training a model.
In general, you can use our training script to easily train any policy. To use wandb for logging training and evaluation curves, make sure you ran wandb login. Here is an example of training the ACT policy on trajectories collected by humans on the Aloha simulation environment for the insertion task:
python lerobot/scripts/train.py \
policy=act \
env=aloha \
env.task=AlohaInsertion-v0 \
dataset_repo_id=lerobot/aloha_sim_insertion_humanThe experiment directory is automatically generated and will show up in yellow in your terminal. It looks like outputs/train/2024-05-05/20-21-12_aloha_act_default. You can manually specify an experiment directory by adding this argument to the train.py python command:
hydra.run.dir=your/new/experiment/dirA link to the wandb logs for the run will also show up in yellow in your terminal. Here is an example of logs from wandb:
You can deactivate wandb by adding these arguments to the train.py python command:
wandb.disable_artifact=true \
wandb.enable=falseNote: For efficiency, during training every checkpoint is evaluated on a low number of episodes. After training, you may want to re-evaluate your best checkpoints on more episodes or change the evaluation settings. See python lerobot/scripts/eval.py --help for more instructions.
If you would like to contribute to ๐ค LeRobot, please check out our contribution guide.
To add a dataset to the hub, you need to login using a write-access token, which can be generated from the Hugging Face settings:
huggingface-cli login --token ${HUGGINGFACE_TOKEN} --add-to-git-credentialThen move your dataset folder in data directory (e.g. data/aloha_ping_pong), and push your dataset to the hub with:
python lerobot/scripts/push_dataset_to_hub.py \
--data-dir data \
--dataset-id aloha_ping_ping \
--raw-format aloha_hdf5 \
--community-id lerobotSee python lerobot/scripts/push_dataset_to_hub.py --help for more instructions.
If your dataset format is not supported, implement your own in lerobot/common/datasets/push_dataset_to_hub/${raw_format}_format.py by copying examples like pusht_zarr, umi_zarr, aloha_hdf5, or xarm_pkl.
Once you have trained a policy you may upload it to the Hugging Face hub using a hub id that looks like ${hf_user}/${repo_name} (e.g. lerobot/diffusion_pusht).
You first need to find the checkpoint located inside your experiment directory (e.g. outputs/train/2024-05-05/20-21-12_aloha_act_default/checkpoints/002500). It should contain:
config.json: A serialized version of the policy configuration (following the policy's dataclass config).model.safetensors: A set oftorch.nn.Moduleparameters, saved in Hugging Face Safetensors format.config.yaml: A consolidated Hydra training configuration containing the policy, environment, and dataset configs. The policy configuration should matchconfig.jsonexactly. The environment config is useful for anyone who wants to evaluate your policy. The dataset config just serves as a paper trail for reproducibility.
To upload these to the hub, run the following:
huggingface-cli upload ${hf_user}/${repo_name} path/to/checkpoint/dirSee eval.py for an example of how other people may use your policy.
An example of a code snippet to profile the evaluation of a policy:
from torch.profiler import profile, record_function, ProfilerActivity
def trace_handler(prof):
prof.export_chrome_trace(f"tmp/trace_schedule_{prof.step_num}.json")
with profile(
activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA],
schedule=torch.profiler.schedule(
wait=2,
warmup=2,
active=3,
),
on_trace_ready=trace_handler
) as prof:
with record_function("eval_policy"):
for i in range(num_episodes):
prof.step()
# insert code to profile, potentially whole body of eval_policy functionIf you want, you can cite this work with:
@misc{cadene2024lerobot,
author = {Cadene, Remi and Alibert, Simon and Soare, Alexander and Gallouedec, Quentin and Zouitine, Adil and Wolf, Thomas},
title = {LeRobot: State-of-the-art Machine Learning for Real-World Robotics in Pytorch},
howpublished = "\url{https://github.com/huggingface/lerobot}",
year = {2024}
}
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent