crackedresearcher / llmverify

Verify outputs generated by LLMs backed with real time data

LLM Verifier

LLM verifier is an open-source project built using Google's Gemini AI model. It helps to verify whether the outputs generated by other LLMs are correct or not, backed with real-time data.

Features

• Verify and check whether outputs generated by LLMs like ChatGPT, Claude 3, and others are hallucination-free.
• Automatically find relevant/real-time data to validate sources.
• Get the percentage accuracy of the correctness of the output backed with real sources.

Built with

1. Gemini AI by Google or type (ai.com)
2. Google Search API
3. Reader API provided by JINA AI
4. Streamlit for frontend
5. ChatGPT to check output generated by it
6. Claude 3 by Anthropic AI

How to Use it

Using the Web App

1. When you open the app, you can enter your Gemini API in the sidebar.
2. If you don't have a Gemini API, head over to Google AI Studio to get yours now. It's free!
3. Now, ask any preferred LLM to generate output on any topic and copy that output (ensure the output isn't very long; processing time will increase if output size is large).
4. Paste the output in the input section and click on the verify button.

Running Locally

1. Fork this repo.
2. Open your terminal and type pip install -r requirements.txt.
3. This will install all the required libraries.
4. Now type streamlit run main.py.
5. This will open the project in your browser.
6. Now you just put your API key and you're done!

NOTE: Since this project is using several APIs like the Reader API, Google Search API, and Gemini API, you can expect it to take approximately 1½ minutes to complete. This time may increase if the data that needs to be verified is large in size.

Also, if the accuracy is greater than 50%, you can consider it accurate. If it's anything less than that, then it's not accurate. All of this depends on the size of the data you need to verify and the relevance of the source

License

This project is under the MIT License.

Project Built by

@0xAyush1 on Twitter