Try out Evaluations - Read Docs - Quickstart Tutorials - Slack Community - Feature Request
UpTrain is an open-source unified platform to evaluate and improve Generative AI applications. We provide grades for 20+ preconfigured checks (covering language, code, embedding use-cases), perform root cause analysis on failure cases and give insights on how to resolve them.
![Data Security](https://private-user-images.githubusercontent.com/43818888/305879439-7b737bfc-b061-4f87-b51e-5bad67879332.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQwOTQwOTgsIm5iZiI6MTcyNDA5Mzc5OCwicGF0aCI6Ii80MzgxODg4OC8zMDU4Nzk0MzktN2I3MzdiZmMtYjA2MS00Zjg3LWI1MWUtNWJhZDY3ODc5MzMyLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MTklMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODE5VDE4NTYzOFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTliYmExNjM3N2MyZDkxODcxNWM3NmRlNWQyYjdhNmNlZjZjZGUwOTA1ZTM4MTRjYmVkN2JiNjZmZmVjNDMzMmEmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.gtZTYLZq_lOSm4n8Yi0xOGrM6dQcpKlzuyr6dsFx5xs)
All the evaluations and analysis run locally on your system, ensuring that the data never leaves your secure environment (except for LLM calls while using model grading checks)
![Experimentation](https://private-user-images.githubusercontent.com/43818888/305879631-fd9c5c7f-3951-4d69-9b77-4fece6ff0830.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQwOTQwOTgsIm5iZiI6MTcyNDA5Mzc5OCwicGF0aCI6Ii80MzgxODg4OC8zMDU4Nzk2MzEtZmQ5YzVjN2YtMzk1MS00ZDY5LTliNzctNGZlY2U2ZmYwODMwLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MTklMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODE5VDE4NTYzOFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTE0ODdkNGNmMWU1MmVhMWI3OTM1MzU1OWVkMzY5YzBiYzFkZDg3YjU3ZDM2NWQ3YWY3ZjU0ZGFkMmEwMmU0MDcmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.phWiehS4olYkEaBBE013cqczTw1pTbVIUX_F6WVbYGs)
Experiment with different embedding models like text-embedding-3-large/small, text-embedding-3-ada, baai/bge-large, etc. UpTrain supports HuggingFace models, Replicate endpoints, or custom models hosted on your endpoint.
![Cost Effective](https://private-user-images.githubusercontent.com/43818888/305879695-8b9bc91b-389a-4664-88f6-11b9c91e1e0f.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQwOTQwOTgsIm5iZiI6MTcyNDA5Mzc5OCwicGF0aCI6Ii80MzgxODg4OC8zMDU4Nzk2OTUtOGI5YmM5MWItMzg5YS00NjY0LTg4ZjYtMTFiOWM5MWUxZTBmLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MTklMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODE5VDE4NTYzOFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTNmNTg2NTUxNzI5ZDg3NWEzZTYyODdmZTIwMjgwNWMzMjBhOGVhNjc4MzhmNWU2Y2ZiN2IxODgxMTI4YWMyMDcmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.6ZezdrUEEQD4raX0KFAfBScjpxxH60x5ToG6fUbTVTA)
By leveraging model grading and introducing an 'Unclear' grade, we are able to leverage GPT-3.5-turbo-1106 as the default evaluator and get high quality yet cost effective scores.
![Root Cause Analysis](https://private-user-images.githubusercontent.com/43818888/305879803-32df049c-c0a4-4377-a807-f74e2efa57b5.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQwOTQwOTgsIm5iZiI6MTcyNDA5Mzc5OCwicGF0aCI6Ii80MzgxODg4OC8zMDU4Nzk4MDMtMzJkZjA0OWMtYzBhNC00Mzc3LWE4MDctZjc0ZTJlZmE1N2I1LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MTklMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODE5VDE4NTYzOFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWVkZjU1NjkxMjVmY2YwY2U5OTI0ZmIxYWIwNmQ5ZDFkMThhNzQ2Y2EzYzgyOGYwZGE1NGI3N2EwYWQ1NjBjYjcmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.LV7pNmHazHSoSxliUOXR-Je3GGQxLB02jXwKReGG1LI)
You can perform root cause analysis on cases with either negative user feedback or low evaluation scores to understand which part of your LLM pipeline is giving suboptimal results. Check out the supported RCA templates.
![Configure your own evaluation LLM](https://private-user-images.githubusercontent.com/43818888/305879990-b87c75b1-b86d-4770-8d37-e8428d5f17c2.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQwOTQwOTgsIm5iZiI6MTcyNDA5Mzc5OCwicGF0aCI6Ii80MzgxODg4OC8zMDU4Nzk5OTAtYjg3Yzc1YjEtYjg2ZC00NzcwLThkMzctZTg0MjhkNWYxN2MyLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MTklMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODE5VDE4NTYzOFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWE2MmEyNTZkYzdiOWQ1ZWIxNDkyOWJhMGU5NDU5MTQ1ZjMzMzFjNGIwMjQ3NGNiZmQ5MWNiMDZjYzFmZjI2MjMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.9gve5j4BYodkDl91_aMsDr5BupuiRPYn5ElIRU0hK1E)
We allow you to use any of OpenAI, Anthropic, Mistral, Azure's Openai endpoints or open-source LLMs hosted on Anyscale to be used as evaluator.
![Customize Evaluations](https://private-user-images.githubusercontent.com/43818888/305879852-167488ec-9feb-48a7-9659-7bb9478584f9.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQwOTQwOTgsIm5iZiI6MTcyNDA5Mzc5OCwicGF0aCI6Ii80MzgxODg4OC8zMDU4Nzk4NTItMTY3NDg4ZWMtOWZlYi00OGE3LTk2NTktN2JiOTQ3ODU4NGY5LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MTklMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODE5VDE4NTYzOFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWE0ODhlNjE0YWYzZDUwMTRkMWE1M2JmMTc1OTNmODg2NDc3Yzc4M2I4MGZmZTgxNGUyYjI4ZWVmNTY2OThiYzkmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.JTb8uPFm1dy26NIsCcp54221kmlyL4wIvaXJJQ6KvEo)
UpTrain provides tons of ways to customize evaluations. You can customize evaluation method (chain of thought vs classify), few shot examples, add scenario description, as well as create custom evaluators.
![40+ Operators Supported](https://private-user-images.githubusercontent.com/43818888/305880054-55d99fb9-ce0a-409f-b898-acb550fa0804.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQwOTQwOTgsIm5iZiI6MTcyNDA5Mzc5OCwicGF0aCI6Ii80MzgxODg4OC8zMDU4ODAwNTQtNTVkOTlmYjktY2UwYS00MDlmLWI4OTgtYWNiNTUwZmEwODA0LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MTklMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODE5VDE4NTYzOFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTE2YmYzYWZjMGNmODdmZjBhMGI2YmIwYWIyMTE2YTg4ZmIzMTEzNmFkYzg3NTdiOGI4NWQwNGVkMmQ3ZjljMTEmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0._j7y6HkXr0tk6U1p9m8fOoKNJn9QTrafRrdlUUuKMls)
Support for 40+ operators such as BLEU, ROUGE, Embeddings Similarity, Exact match, etc.
- Experiment Dashboards
- Collaborate with your team
- Embedding visualization via UMAP and Clustering
- Pattern recognition among failure cases
- Prompt improvement suggestions
![quality of your responses](https://private-user-images.githubusercontent.com/43818888/305879068-654b2289-2799-4310-84be-fcdd071f3e2e.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQwOTQwOTgsIm5iZiI6MTcyNDA5Mzc5OCwicGF0aCI6Ii80MzgxODg4OC8zMDU4NzkwNjgtNjU0YjIyODktMjc5OS00MzEwLTg0YmUtZmNkZDA3MWYzZTJlLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MTklMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODE5VDE4NTYzOFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWQzNjEwMGRlYjQ5MzBiZDkxYjk4MjgxY2M2ZTVhNDcwZDViMjZjMWFlMzU0M2YxNWEyMzExYTZiZGM3OWY1ZjkmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.ysI80hElnliDpeAgZv0DiBAFv6ohw8EG0e_2upa2erA)
Eval | Description |
---|---|
Reponse Completeness | Grades whether the response has answered all the aspects of the question specified. |
Reponse Conciseness | Grades how concise the generated response is or if it has any additional irrelevant information for the question asked. |
Reponse Relevance | Grades how relevant the generated context was to the question specified. |
Reponse Validity | Grades if the response generated is valid or not. A response is considered to be valid if it contains any information. |
Reponse Consistency | Grades how consistent the response is with the question asked as well as with the context provided. |
![quality of retrieved context and response groundedness](https://private-user-images.githubusercontent.com/43818888/305878811-a7e384a3-c857-4a71-a938-7a2a70f8db1e.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQwOTQwOTgsIm5iZiI6MTcyNDA5Mzc5OCwicGF0aCI6Ii80MzgxODg4OC8zMDU4Nzg4MTEtYTdlMzg0YTMtYzg1Ny00YTcxLWE5MzgtN2EyYTcwZjhkYjFlLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MTklMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODE5VDE4NTYzOFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTg1YTdhZWZlZmI1MTgyMGVjZTZkNjYzNjUyOTI2OGM1ZGY5MDU2Nzk1ZGNkODI5MDdmYzE3NTQ1ZTViZmMwMWMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.WaT2jVEsm4kf8NwA_wH35X7pjK18j2uSRmm4Yx5YJ5Y)
Eval | Description |
---|---|
Context Relevance | Grades how relevant the context was to the question specified. |
Context Utilization | Grades how complete the generated response was for the question specified given the information provided in the context. |
Factual Accuracy | Grades whether the response generated is factually correct and grounded by the provided context. |
Context Conciseness | Evaluates the concise context cited from an original context for irrelevant information. |
Context Reranking | Evaluates how efficient the reranked context is compared to the original context. |
![language quality of the response](https://private-user-images.githubusercontent.com/43818888/305878565-776cbc6d-bb4c-4ed1-a892-7a1df38d12d5.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQwOTQwOTgsIm5iZiI6MTcyNDA5Mzc5OCwicGF0aCI6Ii80MzgxODg4OC8zMDU4Nzg1NjUtNzc2Y2JjNmQtYmI0Yy00ZWQxLWE4OTItN2ExZGYzOGQxMmQ1LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MTklMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODE5VDE4NTYzOFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTI2YjZmNzFjNzAyNTA0YzY0N2IzOTE2YzQzMDY4OGUyYWZlZjRmYTNjODBlYTVhZjJiYjdlYjQ0ZmEzODk2YjkmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.3dKJilx3LDfwGZ5JRtkazOBhmcmd03ugjnLz_VGb6WU)
Eval | Description |
---|---|
Language Features | Grades whether the response has answered all the aspects of the question specified. |
Tonality | Grades whether the generated response matches the required persona's tone |
![language quality of the response](https://private-user-images.githubusercontent.com/36454110/305911186-2fba9f0b-71b3-4d90-90f8-16ef38cef3ab.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQwOTQwOTgsIm5iZiI6MTcyNDA5Mzc5OCwicGF0aCI6Ii8zNjQ1NDExMC8zMDU5MTExODYtMmZiYTlmMGItNzFiMy00ZDkwLTkwZjgtMTZlZjM4Y2VmM2FiLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MTklMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODE5VDE4NTYzOFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWFhM2UxNTlkZDk1YTdhMmE4NTYxYjRhNjc0ODRhODA2NDJlZGRmZjI5YzQ1ZWY3ZTdlMzEzYzM0M2MwOGU2YjYmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.M-ZeNaog5ysVxPDocf9X7rRSO9RaOMnXhVLx5pBT-i4)
Eval | Description |
---|---|
Code Hallucination | Grades whether the code present in the generated response is grounded by the context. |
![conversation as a whole](https://private-user-images.githubusercontent.com/43818888/305878623-194f8dd7-26ca-48eb-bdff-028212db9e25.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQwOTQwOTgsIm5iZiI6MTcyNDA5Mzc5OCwicGF0aCI6Ii80MzgxODg4OC8zMDU4Nzg2MjMtMTk0ZjhkZDctMjZjYS00OGViLWJkZmYtMDI4MjEyZGI5ZTI1LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MTklMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODE5VDE4NTYzOFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTFkNGEzYWNhNDc0MzlmYjc3NjhiZjZkOWIzYjI2NjNlYWMzMjViODRhZDVmMDU0MjNlZmQwNGE2OGIzNWZiY2YmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.hvWNzMAbCEMjuaQaMDsZdUz3X3mfpBXcsp2qpK0vwpM)
Eval | Description |
---|---|
User Satisfaction | Grades how well the user's concerns are addressed and assesses their satisfaction based on provided conversation. |
![custom evaluations and others](https://private-user-images.githubusercontent.com/43818888/305878518-0bbc2d82-0f96-49ab-969c-4eec29fef988.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQwOTQwOTgsIm5iZiI6MTcyNDA5Mzc5OCwicGF0aCI6Ii80MzgxODg4OC8zMDU4Nzg1MTgtMGJiYzJkODItMGY5Ni00OWFiLTk2OWMtNGVlYzI5ZmVmOTg4LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MTklMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODE5VDE4NTYzOFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTkxZjAwMDVhZjUxYzM5MDRjMTk0MThjZWM0ZGU4NTFjZTVmZWRhM2ExNDBhNzE0MjM3NDcxZGYwYzBiY2NlNDAmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.Mq_UGnhC-WkvrqMEFoeZWoNNtLE-LgVQYM8ByrRng90)
Eval | Description |
---|---|
Custom Guideline | Allows you to specify a guideline and grades how well the LLM adheres to the provided guideline when giving a response. |
Custom Prompts | Allows you to create your own set of evaluations. |
![compare responses with ground truth](https://private-user-images.githubusercontent.com/36454110/305910995-e3ee22f4-9984-47a5-a7d2-9d8688a090fb.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQwOTQwOTgsIm5iZiI6MTcyNDA5Mzc5OCwicGF0aCI6Ii8zNjQ1NDExMC8zMDU5MTA5OTUtZTNlZTIyZjQtOTk4NC00N2E1LWE3ZDItOWQ4Njg4YTA5MGZiLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MTklMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODE5VDE4NTYzOFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTBlYjlmMWJmMDA0YTNhNDIzMTA4MTgyYjQ1YjZiYzUzODUwZjRmODkxMWVmOWE4MGQyNDEwZGY4NDc2MmI5ZjcmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.sIJnmWWdWqL2jq--6XBMcgEcjNQQT-aaE2jJlE05EEY)
Eval | Description |
---|---|
Response Matching | Compares and grades how well the response generated by the LLM aligns with the provided ground truth. |
![safeguard system prompts and avoid LLM mis-use](https://private-user-images.githubusercontent.com/43818888/305879147-4fcdb1e8-f356-443b-88f0-94146e08829e.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQwOTQwOTgsIm5iZiI6MTcyNDA5Mzc5OCwicGF0aCI6Ii80MzgxODg4OC8zMDU4NzkxNDctNGZjZGIxZTgtZjM1Ni00NDNiLTg4ZjAtOTQxNDZlMDg4MjllLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MTklMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODE5VDE4NTYzOFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTRlMDU4MmQzZjNiM2MwOGZhNDVlMmM5NDdjZGZkOGFiNDU0MmNmOGI2YTcwNzJhMGYzNGIzYmE5ZGU2MzRhMGQmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.k7WkzH3uQUEG-E9QwK9J_8DmNJV49fhXNYr3wMP-7qo)
Eval | Description |
---|---|
Prompt Injection | Grades whether the generated response is leaking any system prompt. |
Jailbreak Detection | Grades whether the user's prompt is an attempt to jailbreak (i.e. generate illegal or harmful responses). |
The UpTrain dashboard is a web-based interface that allows you to evaluate your LLM applications. It is a self-hosted dashboard that runs on your local machine. You don't need to write any code to use the dashboard. You can use the dashboard to evaluate your LLM applications, view the results, and perform root cause analysis.
Before you start, ensure you have docker installed on your machine. If not, you can install it from here.
The following commands will download the UpTrain dashboard and start it on your local machine.
# Clone the repository
git clone https://github.com/uptrain-ai/uptrain
cd uptrain
# Run UpTrain
bash run_uptrain.sh
If you are a developer and want to integrate UpTrain evaluations into your application, you can use the UpTrain package. This allows for a more programmatic way to evaluate your LLM applications.
pip install uptrain
You can evaluate your responses via the open-source version by providing your OpenAI API key to run evaluations.
from uptrain import EvalLLM, Evals
import json
OPENAI_API_KEY = "sk-***************"
data = [{
'question': 'Which is the most popular global sport?',
'context': "The popularity of sports can be measured in various ways, including TV viewership, social media presence, number of participants, and economic impact. Football is undoubtedly the world's most popular sport with major events like the FIFA World Cup and sports personalities like Ronaldo and Messi, drawing a followership of more than 4 billion people. Cricket is particularly popular in countries like India, Pakistan, Australia, and England. The ICC Cricket World Cup and Indian Premier League (IPL) have substantial viewership. The NBA has made basketball popular worldwide, especially in countries like the USA, Canada, China, and the Philippines. Major tennis tournaments like Wimbledon, the US Open, French Open, and Australian Open have large global audiences. Players like Roger Federer, Serena Williams, and Rafael Nadal have boosted the sport's popularity. Field Hockey is very popular in countries like India, Netherlands, and Australia. It has a considerable following in many parts of the world.",
'response': 'Football is the most popular sport with around 4 billion followers worldwide'
}]
eval_llm = EvalLLM(openai_api_key=OPENAI_API_KEY)
results = eval_llm.evaluate(
data=data,
checks=[Evals.CONTEXT_RELEVANCE, Evals.FACTUAL_ACCURACY, Evals.RESPONSE_COMPLETENESS]
)
print(json.dumps(results, indent=3))
If you have any questions, please join our Slack community
Speak directly with the maintainers of UpTrain by booking a call here.
Eval Frameworks | LLM Providers | LLM Packages | Serving frameworks | LLM Observability | Vector DBs |
---|---|---|---|---|---|
OpenAI Evals โ | GPT-3.5-turbo โ | Langchain ๐ | HuggingFace โ | Langfuse ๐ | Qdrant โ |
EleutherAI LM Eval ๐ | GPT-4 โ | Llama Index โ | Replicate โ | Helicone ๐ | Pinecone ๐ |
BIG-Bench ๐ | Claude โ | AutoGPT ๐ | AnyScale โ | Chroma โ | |
Cohere โ | Together ai ๐ | ||||
Llama2 โ | Ollama ๐ | ||||
Mistral โ |
Having worked with ML and NLP models for the last 8 years, we were continuosly frustated with numerous hidden failures in our models which led to us building UpTrain. UpTrain was initially started as an ML observability tool with checks to identify regression in accuracy.
However we soon released that LLM developers face an even bigger problem -- there is no good way to measure accuracy of their LLM applications, let alone identify regression.
We also saw release of OpenAI evals, where they proposed the use of LLMs to grade the model responses. Furthermore, we gained confidence to approach this after reading how Anthropic leverages RLAIF and dived right into the LLM evaluations research (We are soon releasing a repository of awesome evaluations research).
So, come today, UpTrain is our attempt to bring order to LLM chaos and contribute back to the community. While a majority of developers still rely on intuition and productionise prompt changes by reviewing a couple of cases, we have heard enough regression stories to believe "evaluations and improvement" will be a key part of LLM ecosystem as the space matures.
-
Robust evaluations allows you to systematically experiment with different configurations and prevent any regressions by helping objectively select the best choice.
-
It helps you understand where your systems are going wrong, find the root cause(s) and fix them - long before your end users complain and potentially churn out.
-
Evaluations like prompt injection and jailbreak detection are essential to maintain safety and security of your LLM applications.
-
Evaluations help you provide transparency and build trust with your end-users - especially relevant if you are selling to enterprises.
-
We understand that there is no one-size-fits-all solution when it come to evaluations. We are increasingly seeing the desire from developers to modify the evaluation prompt or set of choices or the few shot examples, etc. We believe the best developer experience lies in open-source, instead of exposing 20 different parameters.
-
Foster innovation: The field of LLM evaluations and using LLM-as-a-judge is still pretty nascent. We see a lot of exciting research happening, almost on a daily basis and being open-source provides the right platform to us and our community to implement those techniques and innovate faster.
We are continuously striving to enhance UpTrain, and there are several ways you can contribute:
-
Notice any issues or areas for improvement: If you spot anything wrong or have ideas for enhancements, please create an issue on our GitHub repository.
-
Contribute directly: If you see an issue you can fix or have code improvements to suggest, feel free to contribute directly to the repository.
-
Request custom evaluations: If your application requires a tailored evaluation, let us know, and we'll add it to the repository.
-
Integrate with your tools: Need integration with your existing tools? Reach out, and we'll work on it.
-
Assistance with evaluations: If you need assistance with evaluations, post your query on our Slack channel, and we'll resolve it promptly.
-
Show your support: Show your support by starring us โญ on GitHub to track our progress.
-
Spread the word: If you like what we've built, give us a shoutout on Twitter!
Your contributions and support are greatly appreciated! Thank you for being a part of UpTrain's journey.
This repo is published under Apache 2.0 license and we are committed to adding more functionalities to the UpTrain open-source repo. We also have a managed version if you just want a more hands-off experience. Please book a demo call here.
We are building UpTrain in public. Help us improve by giving your feedback here.
We welcome contributions to UpTrain. Please see our contribution guide for details.