reidbarber / gen-ui

Use text or image prompts to generate components and apps built with React.

GenUI

Use text or image prompts to generate UI components and apps built with React. Powered by OpenAI's Assistants API and CodeSandbox's Sandpack.

Get Started

Get an OpenAI API Key

Create a .env.local file in the root of the project and add your OpenAI API key:

OPENAI_API_KEY=sk-...

Install

yarn install

Start the app

yarn dev

Open http://localhost:3000 to view it in your browser.

1. Create a new assistant

• These can be generic (e.g. React), or specific to a component library (e.g. React Spectrum)
• These can be managed within GenUI, or on OpenAI's Platform. Use OpenAI's platform to add files for knowledge retrieval (e.g. component library documentation in markdown files).

2. Create a new thread with that assistant

• This represents the app that you're building, and all its iterations.

3. Enter a text prompt and optionally add images.

• Note: Your OpenAI account must have access to the gpt-vision-preview model to upload images.

5. A working React app is generated based on the prompt.

6. Ask for modifications or more features to add to the app, and repeat.

gen-ui's Issues

Revert to previous states

Can save at each state, or better yet use git in the browser: https://github.com/isomorphic-git/isomorphic-git

UI for creating Assistants

https://platform.openai.com/docs/api-reference/assistants/createAssistant

Can use metadata field to specify it to be able to be used by GenUI, and other info.

Send back any errors/lint warnings for revisions

https://sandpack.codesandbox.io/docs/advanced-usage/hooks#usesandpack

Host the bundler

https://sandpack.codesandbox.io/docs/guides/hosting-the-bundler

GPT-4 Vision support

https://platform.openai.com/docs/guides/vision

It will need to use the vision model separately and get a text description to send to the main thread, until Assistants API supports vision inputs natively.

Include screenshots at each step

Option 1: On client

Can use something like html2canvas.

May be difficult to implement with the iframe: codesandbox/sandpack#838

Can look into dispatching via Sandpack or window.postMessage().

Option 2: On server

Using something like puppeteer or playwright.

Send screenshots after each update via GPT 4 Vision

Maybe you could use gpt 4 vision to check the current state of how the website looks, then send that input off together with the prompt so that the ai can understand what its looking at and what the user is referring to

Load existing threads

String replace support for updateFile

It would be much faster if the updateFile function didn't have to include the entire update file, if just a small part of it is being changed.

updateFile could support string replace where RegEx can be used to replace part of the initial file.

Alternatively, using unified diffs may be worth trying: https://aider.chat/docs/unified-diffs.html

UI for uploading files for knowledge retrieval

Assistant editor should all for uploading and managing files for knowledge retrieval: https://platform.openai.com/docs/assistants/tools/knowledge-retrieval

Can currently be done via OpenAI's platform: https://platform.openai.com/assistants