nasa-petal / bidara-deep-chat

BIDARA is a GPT-4o chatbot that was instructed to help scientists and engineers understand, learn from, and emulate the strategies used by living things to create sustainable designs and technologies using the Biomimicry Institute's step-by-step design process.

BIDARA : Bio-Inspired Design and Research Assistant

BIDARA is an OpenAI GPT-4o Assistant that was instructed to help scientists and engineers understand, learn from, and emulate the strategies used by living things to create sustainable designs and technologies.

🚀 Try it online! or Use BIDARA in ChatGPT

BIDARA can guide users through the Biomimicry Institute’s Design Process, a step-by-step method to propose biomimetic solutions to challenges. This process includes defining the problem, biologizing the challenge, discovering natural models, abstracting design strategies, and emulating nature's lessons.

🔥 Features

     ✅ Multiple chats (saved offline)
     ✅ Mobile optimized
     ✅ Write and execute python code to analyze data, create visualizations, or perform calculations (Code Interpreter) more info, filetypes supported
     ✅ Summarize or answer questions about uploaded files (Knowledge Retrieval) more info, filetypes supported
     ✅ Custom Functions
         ☑️ Retrieve academic literature with Semantic Scholar
         ☑️ Generate images with DALL-E
         ☑️ Analyze images you upload or that it has generated with GPT-4o
         ☑️ Detect pABC patterns in images with GPT-4o
         ☑️ Search for US patents with the PQAI API
         ☑️ Search the web with Tavily API (limited to 1000 API calls a month across all BIDARA users)

💻 For developers

bidara-deep-chat is a front-end web app that uses Svelte and the deep-chat web component to connect directly to BIDARA over the OpenAI Assistants API (no backend required). You can easily clone this repository and replace BIDARA with your own custom OpenAI Assistant. Template based on https://github.com/sveltejs/template

Run locally

npm install
npm run dev

Known issues

• deep-chat speechToText: submit command word is sent in message on safari and chrome on iOS.

• deep-chat textToSpeech: doesn't read messages aloud on safari or chrome on iOS.

Nice to haves

• ability rate responses and add feedback.

• ability to send ratings, feedback and chat log to us.

• don't do TtS unless StT has been used.

• Proxy requests to OpenAI through an authenticated API. Users can request access. Ability to generate api keys once authenticated. Authorized API keys required to communicate with API.

• Functions:

• get pdf of paper from semantic scholar link and upload to assistant.thread.messages.files for retrieval. as a temp workaround can use openAccessPdf links to download pdfs on client, and then upload them on client directly to openai assistant.

• get all code interpreter code. useful to check its work.

bidara-deep-chat's Issues

chat messages disappear on safari (osx and ios) after page load

tested on safari on iOS. They reappear after you click on the chat name in the navbar.

duplicate chat thread and missing images

vision branch

I had a converstaion that included an AI generated image, and a user uploaded image.

When I closed the tab and returned, the conversation history was duplicated similar to #60 .

When I changed the thread, and switched back, the duplication was gone, but so were all the images.

When I closed my tab and returned, the duplication was back, and no images.

file attachment area is wider than input box in iOS Safari

create Knowah assistant

Not sure what the output should be. To start could generate a CSV with the nodes and connection information. Maybe later on it could generate a file that can be imported into a graph visualization tool like Gephi.

You might be able to get Code Interpreter to generate a visualization of a knowledge graph using Plotly.
I think Code Interpreter can only output static images of Plotly visualizations. Would be nice if it could output the interactive versions like you get in the native Jupyter notebook environment that we could embed in the web page.

patent search function

patent search - https://developer.uspto.gov/api-catalog/bulk-search-and-download, https://patentsview.org/apis/api-faqs, https://www.npmjs.com/package/uspto-patents-view-api

The code in the python folder can take a OpenAPI spec and convert it into the json schema required by OpenAI.

Mobile sidebar unable to scroll

When chats exceed screen height in sidebar, unable to scroll to view more.

Make it easy for users to "install pwa app"

I'm envisioning a button on the login screen to "add to home screen". Similar to visiting a website of a service that is also available as an app prompts you to "install or open in app". Go to reddit.com for an example of this.

From this post https://stackoverflow.com/questions/38826397/add-to-home-screen-functionality-using-javascript:

According to the Best Practices section of this page: https://developer.chrome.com/multidevice/android/installtohomescreen#best-practices

Do not prompt the user to add your app to the homescreen. There is no way to detect if the app is running installed or not.
It looks like other developers have suggested that you can simply prompt the user with instructions for how to add to the homescreen, rather than try and do it directly.

EDIT: Doing more digging, it seems that since chrome 42, google is introducing app install banners. See: https://developers.google.com/web/updates/2015/03/increasing-engagement-with-app-install-banners-in-chrome-for-android

Your web app will need to meet a bunch of requirements however, including running a service worker, having your site as https, and having the user visit your site at least twice.

multiple empty chat threads allowed in `vision-only`

Tested in iOS Simulator

Was able to create a new thread even though the other thread was empty.

Was this change intended?

download links for files generated by code interpreter don't have filenames or extensions

Might not be fixable, but worth a look. The issue is apparently due to how remarkable is generating the link inside deep-chat.
OvidijusParsiunas/deep-chat#91 (comment)

dark mode

BIDARA fails to consistently call the image generation function

Often it replies with, "I cannot generate images".

image generation is not following the prompt

multiple chats

ability to have multiple chat sessions. conversation history should be persisted to browser storage (either localStorage or indexedDB).

can't rename chat on safari (ios and osx)

tested on safari on iOS.

Multiple responses from assistant aren't put into chat without refreshing

an issue with current version of deep-chat, but a fix is available in deep-chat dev.

This fix comes with other changes on deep-chat dev, so will involve updating some other things along with implementing this fix.

Add survey link to BIDARA

Maybe put it where the bidara logo currently is, or at the bottom of the sidebar? Link text can be something like, "Send Feedback".

https://forms.gle/xDEixG5UJrFBwDKv5

Changing threads and then back, while bidara is "typing" causes the three dots bubble to go away.

Changing the chat title, or changing threads and then back, while bidara is "typing" causes the three dots bubble to go away. The response from BIDARA will not display until a thread is switched to and then back.

Might not be fixable on our end, since the dots bubble is a part of DeepChat.

mobile landscape view cuts off navbar left and right sides

add keyboard support to attachment and mic buttons

Error when logging in `update-save-chats`

Remove all localStorage variables, refresh the page, enter OpenAI key on login screen, then you will get the error.

Also occurs if only the two thread localStorage variables are deleted and the page is refreshed.

Uncaught (in promise) TypeError: Cannot read properties of null (reading 'filter')

have sidebar opened by default on larger screen sizes.

evaluate the new GPT-4 Turbo with Vision `gpt-4-turbo-2024-04-09` model

They recommend this model for both text and vision so could be a replacement for gpt-4-1106-preview and gpt-4-1106-vision-preview that we are currently using for text and vision respectively.
https://platform.openai.com/docs/models/gpt-4-turbo-and-gpt-4

some elements are not keyboard accessible in Safari on MacOS

links in initialMessages, popup menu items, deep-chat's attachment and mic buttons.

disable ability to tab to sidebar elements when sidebar is closed.

Using display:none on the sidebar when closed was suggested as a solution.

`vision-only` does not describe image

I must be doing something wrong.

support streaming messages

I think support for the new Assistants API streaming was added to deep-chat.

OvidijusParsiunas/deep-chat#137

duplicate conversation history in UI after closing and reopening browser.

Semantic Scholar errors not caught in Safari (mac and ios)

on initial page load show loading indicator

Navbar with empty chat title loads in before deep-chat. Show a loading indicator on screen while deep-chat is loading. Might even be best to hide the navbar completely until deep-chat and the active thread have loaded. Here's what displays for a few seconds on initial page load.

manager AI assistant that can delegate tasks to other AI assistants.

This NPM package looks promising!

https://github.com/Mintplex-Labs/openai-assistant-swarm

make a script to delete old files from OpenAI

https://platform.openai.com/docs/api-reference/files/delete

maybe we delete files if they haven't been used in a run in awhile.

might be nice to somehow tag files we create during testing, so they can be deleted, while files generated by actual users can be kept.

images generated using function calling don't persist page refresh.

occasionally active thread doesn't appear to be active

Notice empty chat title, and no blue selected chat in sidebar. Suggested that it might be due to localStorage's openai-active-thread being out of sync with deep-chat active thread. Issue was noticed in OSX Safari 17.3

notch covers chat bubbles in mobile landscape

generated images do not appear in `vision-only`

add blue keyboard focus outline to deep chat elements.

text input, attachment button, and microphone button.

DALLE-3 support for image generation

DALLE-3 support for image generation. (Coming soon to Assistants API, or could implement using Function Calling) https://cookbook.openai.com/examples/creating_slides_with_assistants_api_and_dall-e3

make a requirements assistant

Create an assistant that can help systems engineers perform requirements management:

It should be able to:

1. Given a system description, draft a set of requirements.
2. Given a set of requirements, suggest fixes, improvements, or missing requirements. Make sure to recommend sustainability requirements.

Probably other useful things it should do too like walk them through the process, or help with traceability, or verification.

Requirements Management Resources

This document contains some useful information that may be useful for the assistant's system prompt such as different types of requirements, and the attributes of good requirements.
https://ocw.mit.edu/courses/16-842-fundamentals-of-systems-engineering-fall-2015/7f2bc41156a04ecb94a6c04546f122af_MIT16_842F15_Ses2_Req.pdf

INCOSE SE handbook has sections on requirements: 4.2, 4.4, 7.2. The handbook is on v5, but I found a free pdf for v3.
http://www.las.inpe.br/~perondi/21.06.2010/SEHandbookv3.pdf

NASA SE Handbook
Section 4.2 Technical Requirements Definition
https://www.nasa.gov/reference/4-0-system-design-processes/#hds-sidebar-nav-7
Section 6.2.1 Requirements Management Process
https://www.nasa.gov/reference/6-0-crosscutting-technical-management/#hds-sidebar-nav-6
Appendix C How to write a good requirement
https://www.nasa.gov/reference/appendix-c-how-to-write-a-good-requirement/
Appendix D Requirements verification matrix
https://www.nasa.gov/reference/appendix-d-requirements-verification-matrix/

can't switch threads on iOS (safari or chrome)

Bug found on main and vision-only, which are the only branches I tested.

To reproduce:

• Go to the production site on iPhone or vision-only in iOS simulator.
• Make 3 threads.
• Try switching between threads (nothing happens).

navbar is hidden when keyboard is opened in mobile

It's pretty common for mobile websites, but with the "app like feel" now available via PWA support, would be nice for this behavior to go away.

automated test suite

As the code base grows, it's getting hard to manually test after every change. Should probably start to look into adding some automated tests.

Github Actions might be sufficient or we can look at other CI services.

https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-nodejs

`Cannot read properties of null (reading 'match')` in `threadUtils.js`

"make a simple visualization of the csv data"
Switch to a different thread while it is running
Switch back after it is finished running.

Cannot read properties of null (reading 'match') in threadUtils.js
the string parameter is null.

function findReplaceRegEx(string, regex, replacement) {
  const matches = (string.match(regex) || [])

Semantic Scholar search error

Chats with file links but no attachments link to sandbox (inaccessible links)

This PR comment describes the issue: #80 (comment)

Edit to main PR comment:

Not using load_thread_history from deep-chat.

I mentioned both in the description and in the PeTaL meeting that I'd reverted to use load_thread_history, but this is not the case. I put in functionality to mimic load_thread_history by simply loading threads from api request, and then put in files.

I had done this specific part awhile ago, so I had forgotten exactly what I did for it. It ends up being the same except for one difference:

• messages that do not have files attached, but still link to files, will link to sandbox:/... (the assistant files). These links will not work for users to click.

This is done because using load_thread_history would not allow for adding files (or any chat edits) into the chat.

This is currently worth the tradeoff because:

• it allows for a much more stable version of chat syncing
• messages with file links but no file attachments are uncommon compared to messages with neither or both.
• this has a potential fix I can work on next, but would like to get these current changes in first

I think it is worth still merging the PR with this issue and working on a fix afterwards.

add a function that uses GPT-4V to describe patterns in images.

List of patterns it should detect:
patternABC-poster-24x36-for-TZ-Update-2020.pdf

evaluate upgrading to v2 of the Assistants API

https://platform.openai.com/docs/assistants/whats-new

Migration guide v1 to v2.
https://platform.openai.com/docs/assistants/migration/agents

v2 is required to use the latest gpt-4o model

(no longer) waiting for deep-chat support.
OvidijusParsiunas/deep-chat#190

GPT-4 vision support so it can 'see' images, including ones uploaded by users.

implement using function calling.

user messages containing the filename of uploaded files don't persist page refresh.

When a thread run errors, it locks the thread.

Got a 500 error on a thread run, and then when I submitted another message, it says "Can't add messages to a thread while a run is active". Maybe catch these errors and stop the run so the conversation on the thread can continue.

Use text-to-image GenAI to draw sketches that depict how the biological strategies work

If GenAI is struggling generating good images, could lookup existing engineering design patent diagrams based on a biological strategy.

example design strategy sketches:
https://toolbox.biomimicry.org/methods/abstract/