[Request]: Ingest ChatGPT Conversation JSON about obsidian-ai-research-assistant

[ericrallen]

This is certainly an interesting use case.

Is there a specific tool you're using to export a conversation as JSON? Anyone working on this would probably want to be able to export an example conversation of their own for testing - we might also want to consider generating some sort of kitchen sink demo conversation JSON that includes various situations that a parser might encounter.

It seems like the approach here would be:

1. Create a parser for the ChatGPT JSON export
2. Add an Command to AI Research Assistant that will ingest that JSON format from a new note and execute the parser
3. Format and save the note like other AI Research Assistant notes

Admittedly, this isn't a current use case for me and without a generally agreed upon standard tool for exporting JSON from the UI it probably won't be a high priority for me, but I'd be happy to provide guidance on a PR if anyone is interested in tackling this one.

[blakejwc]

Thank you for considering my feature request.

To clarify the context of this feature: I've been actively using the ChatGPT export option in OpenAI's web app to archive all my chat history. The challenge I've faced is extracting this data programmatically from the JSON format into a more readable and usable form. Your Obsidian AI Research Assistant is exactly what I have been imagining I could do with my conversations if I could get them in a more usable form. The ability to annotate and analyze my previous prompts within Obsidian would be undescribably satisfying.

Regarding the JSON schema I provided, it's something I constructed after examining the data structure across all 30+ MB of my chat history 😵‍💫 It's beyond unmanageable. My hope is the schema, since is isn't a third-party standard but rather a representation based on the consistent structure I've observed in my ChatGPT exports, could be used to help identify the types and organization that would be needed to create a parser. I'd be happy to share all that I've learned and built, as well as anonymized samples of my exported chat for testing purposes. Just let me know.

Also, I completely agree with the approach you outlined: creating a parser for the ChatGPT JSON, adding a command to the AI Research Assistant for ingestion, and then formatting and saving the note. The biggest challenge after parsing will be to represent the ChatGPT web app's conversation tree structure with the MemoryManager. I have analyzed my chat history, and I have some conversations that have a dozen or more branching points. I have the specific numbers if that would be helpful in outlining more details.

I also want to mention that beyond OpenAI's GPT models, I'm also excited about the potential of integrating conversations I've generated using the Oogabooga web UI and the Sillytavern web UI into Obsidian. It seems this aligns well with the broader vision you have for the AI Research Assistant.

I understand that this might not be a priority for you at the moment. However, I believe this feature could be invaluable for many in the community who are looking to integrate their AI chat histories with Obsidian. If there's any way I can assist, whether it's by providing more data samples or collaborating on the parsing logic, please let me know. Your guidance would be invaluable, and I'm eager to contribute to making this feature a reality.

Thank you once again for your openness to this suggestion.

[blakejwc]

Sorry about the close-open, I'm typing on my phone 😅

[ericrallen]

Today I Learned: You can export your ChatGPT history.

Thanks for the additional context.