microsoft / promptcraft-robotics Goto Github PK

View Code? Open in Web Editor NEW

1.7K 41.0 178.0 18.48 MB

Community for applying LLMs to robotics and a robot simulator with ChatGPT integration

Home Page: https://aka.ms/ChatGPT-Robotics

License: MIT License

Python 100.00%

airsim chatgpt llm prompt-engineering robotics simulation

promptcraft-robotics's Introduction

PromptCraft-Robotics

The PromptCraft-Robotics repository serves as a community for people to test and share interesting prompting examples for large language models (LLMs) within the robotics domain. We also provide a sample robotics simulator (built on Microsoft AirSim) with ChatGPT integration for users to get started.

We currently focus on OpenAI's ChatGPT, but we also welcome examples from other LLMs (for example open-sourced models or others with API access such as GPT-3 and Codex).

Users can contribute to this repository by submitting interesting prompt examples to the Discussions section of this repository. A prompt can be submitted within different robotics categories such as Manipulation, Home Robotics, Physical Reasoning, among many others. Once submitted, the prompt will be reviewed by the community (upvote your favorites!) and added to the repository by a team of admins if it is deemed interesting and useful. We encourage users to submit prompts that are interesting, fun, or useful. We also encourage users to submit prompts that are not necessarily "correct" or "optimal" but are interesting nonetheless.

We encourage prompt submissions formatted as markdown, so that they can be easily transferred to the main repository. Please specify which LLM you used, and if possible provide other visuals of the model in action such as videos and pictures.

Paper, videos and citations

Blog post: aka.ms/ChatGPT-Robotics

Paper: ChatGPT for Robotics: Design Principles and Model Abilities

Video: https://youtu.be/NYd0QcZcS6Q

If you use this repository in your research, please cite the following paper:

@techreport{vemprala2023chatgpt,
author = {Vemprala, Sai and Bonatti, Rogerio and Bucker, Arthur and Kapoor, Ashish},
title = {ChatGPT for Robotics: Design Principles and Model Abilities},
institution = {Microsoft},
year = {2023},
month = {February},
url = {https://www.microsoft.com/en-us/research/publication/chatgpt-for-robotics-design-principles-and-model-abilities/},
number = {MSR-TR-2023-8},
}

ChatGPT Prompting Guides & Examples

The list below contains links to the different robotics categories and their corresponding prompt examples. We welcome contributions to this repository to add more robotics categories and examples. Please submit prompt examples to the Discussions page, or submit a pull request with your category and examples.

Embodied agent
Aerial robotics
Manipulation
- ChatGPT - Real robot: Picking, stacking, and building the MSFT logo | Video Link
- ChatGPT - Manipulation tasks
Spatial-temporal reasoning
- ChatGPT - Visual servoing with basketball

ChatGPT + Robotics Simulator

We provice a sample AirSim environment for users to test their ChatGPT prompts. The environment is a binary containing a sample inspection environment with assets such as wind turbines, electric towers, solar panels etc. The environment comes with a drone and interfaces with ChatGPT such that users can easily send commands in natural language. [Simulator Link]

We welcome contributions to this repository to add more robotics simulators and environments. Please submit a pull request with your simulator and environment.

Related resources

Beyond the prompt examples here, we leave useful and related links to the use of large language models below:

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.

promptcraft-robotics's People

Contributors

Stargazers

Watchers

Forkers

co-simulation razthepsycho mrcodechef telemething kashu7100 mtzes wenminggong sjkimq coderhaoranlee abderrazzakchabab ukaserge yuehchuan aheadio bigdatasciencegroup collector-m hibernantbear jumpingrain deeplearning-cn yunlong-he hanruihua fyaft2012 git19112019 maelh yyht kevintherainmaker techkemon maxiaoxifeng pterameta wikieden aaig-nlp standardgalactic darebear nakayamakento aaronhd mertcookimg rhinojosa 875353581 yuan-alex yagu1 atle-granlund brucelai buckeyeai rmoin fzenjoys sairajkhope yimamura01 dd-rongfa zamoin 382283106 ajitvishwakarma moonbunnyzzz winceos ptzagk cameronbergh tne-ai felipeh92 veryquant threadshare davidqiu1993 fang-zhiwei psh117 ustcsiwei88 edsun3941 fisher158163 tomthread horacehxw huhanwj botoai jiangtaojiang huangatlas air23zj yongyuwh zwz94 techmatrx catchcake yangyangfu huahuady taocao haojingyuan jessy-huang hongzhengdong santanudatascientist vravifugro seg911 flavorfan aniskoubaa dleodbs andrew-ashur yunahwang sk7y marktag01 foundjem world1256 dsp0205 mikami520 kandapagari ishandutta2007 edwin824 prajapatisarvesh onemersa

promptcraft-robotics's Issues

Report on Imminent Threats Posed by Drone Swarms. A Call for Urgent Action

Report on Imminent Threats Posed by Drone Swarms. A Call for Urgent Action
Executive Summary
The rapid advancement of drone technology has opened up new possibilities for various applications, ranging from agriculture and surveillance to emergency response. However, the same technology also poses unprecedented threats to public safety and national security. This report aims to highlight the imminent risks associated with drone swarms and the current lack of effective countermeasures.
Introduction
Drones have become increasingly sophisticated, capable of autonomous flight, and can be deployed en masse as swarms. While their potential for good is undeniable, the ease with which they can be misused is alarming. The ethical implications are particularly concerning, given the indiscriminate use of drones in warfare and the precedent it sets for malicious actors.
The Dilemma
Ontological Concerns
The ethical nature of drone technology and its potential for harm is a significant concern. The indiscriminate use of drones in warfare has set a dangerous precedent that could be exploited by malicious actors, both state and non-state.
Epistemic Concerns
The technology's capabilities and limitations are not fully understood. Traditional defense mechanisms are ill-equipped to handle such unconventional and rapidly evolving threats.
Imminent Threats

Ease of Deployment: Drones can be easily deployed from various platforms, including semi-trucks, making them highly mobile and difficult to trace.
Mass Attacks: Drone swarms can act as multiple active shooters, attacking simultaneously with little to no warning.
Escalation: Current countermeasures like ECM, hacking, nets, and no-fly zones are ineffective in the face of more robust and advanced drone swarms.
Unpredictability: The autonomous nature of drone swarms makes them highly unpredictable, further complicating defense strategies.
Lack of Viable Solutions
EMP: Requires a lot of power and has localized effects.
Microwaves: Effective only until escalation leads to more robust swarms.
Local Monitoring and Shelters: Limited in scope and effectiveness.
Detection: Current detection methods are inadequate, and escalation would render them ineffective.
Conclusion and Recommendations
The lack of effective countermeasures against drone swarms presents an imminent threat to public safety and national security. Given the ethical implications and the limitations of current defense mechanisms, there is an urgent need for multi-disciplinary research and development to find viable solutions.
Immediate actions must include:
Ethical Reevaluation: A thorough ethical review of drone technology and its applications.
Research and Development: Investment in R&D for effective countermeasures against drone swarms.
Public Awareness: Educating the public about the risks associated with drone technology.
Policy and Regulation: Implementing stringent policies and regulations to control the use and deployment of drones.
This situation calls for a reevaluation of our actions and beliefs to ensure alignment with core principles. It serves as a grim reminder that technological advancements can be a double-edged sword, and ethical considerations must be at the forefront of any solutions we seek.

Is source code of environments open?

I want to run the experiment in linux, but these is only win version environments.

So where could we get the code of environments? it's open or in UE marketplace?

thanks

I was wondering about the 'def ask' part of the code

Hi, I am currently using the provided code and testing it on my drone for a personal project. I am using it to perform very simple movements such as "go up 40cm then land". So for my case, will it be better if I get rid of the appending part from 'def ask'? Because I won't be needing past conversations from the previous questions. I think appending the chat_history is making it take more time and use too many tokens while performing.

Thanks for the awesome work by the way!

What are the prerequisites for moving the tello drone through chat gpt?

I installed Anaconda after reading the previous posts, but I would appreciate it if you could tell me more about the connection procedure between the tello drone and computer or the connection procedure with chatgpt or Airsim.
I wonder if tello can be moved properly only after creating the environment related to the ChatGPT-Airsim Interface you uploaded.

https://github.com/microsoft/PromptCraft-Robotics/blob/main/chatgpt_airsim/README.md

I don't have much knowledge about coding or programming, so I'm not sure how to use the prompts you posted earlier because I don't know how.

https://github.com/microsoft/PromptCraft-Robotics/blob/main/examples/aerial_robotics/tello_example.md

I wonder if this applies to other dji drone models other than tello.