latitude-dev / latitude-llm

Work in progress

The Open-Source Prompt Engineering Platform

See documentation →

Report Bug · Join Our Slack · Roadmap · X

🌈 Why Latitude?

Latitude is a platform that helps developers and product teams build AI features with confidence. Here's what it does:

1. Makes prompts easier to manage: Instead of hiding prompts in code, Latitude lets everyone work on them together.
2. Helps with testing: AI doesn't always return the same answer, which can be tricky. Latitude uses LLMs to check other LLM responses, looking for mistakes or off-topic answers.
3. Provides insights: Offers detailed analytics on LLM request performance, including response times and cost comparisons across various AI models and prompt versions.

Latitude puts all these helpful tools in one place. This means you can add AI to your products without a lot of hassle, and keep an eye on how it's all working.

✨ Features

• 👥 Collaborative prompt manager
• 🚀 Support for advanced features like parameters, snippets, logic, and more
• 🔄 Version control for prompts
• 🔌 API + SDKs for easy integration
• 🔍 Built-in observability
• 📊 Evaluations in batch or real-time
• 🌟 Open-source driven by the community

📚 Table Of Contents

• Quick start
• Community
• Contributing
• Links

⚡ Quick start

Latitude offers two deployment options:

1. Latitude Cloud: A fully managed solution that allows you to get started quickly without worrying about infrastructure.
2. Latitude Self-Hosted: An open-source version that you can deploy and manage on your own infrastructure for complete control and customization.

Choose the option that best fits your needs and follow the corresponding instructions below.

Latitude Cloud

To get started with Latitude, follow these steps:

1. Sign up for Latitude: Visit our website and follow the instructions to create your account.

2. Create a new project: Once logged in, create a new project to organize your prompts and evaluations.

3. Write your first prompt: Navigate to the Editor and create a new prompt. Start with a simple task, like generating a short story or answering a question.

4. Test your prompt: Use the playground to test your prompt with different inputs and see the model's responses.

5. Evaluate in batch: Before deploying, you can upload a dataset and run a batch evaluation to assess your prompt's performance across various scenarios.

6. Deploy your prompt: Once you're satisfied with your prompt's performance in batch evaluation, deploy it as an endpoint for easy integration with your applications.

7. Monitor and evaluate: Use the Logs section to review your prompt's performance over time. Set up ongoing evaluations to systematically assess and improve your prompt's output.

8. Iterate and improve: Based on the evaluation results, refine your prompt or create new versions to enhance its performance.

9. Collaborate with your team: Invite team members to your Latitude workspace to collaborate on prompt engineering and evaluations.

For more detailed information on each step, explore our documentation or join our community for support and discussions.

Latitude Self-Hosted

Follow the instructions in the self-hosted guide to get started with Latitude Self-Hosted.

After setting up Latitude Self-Hosted, you can follow the same steps as in the Latitude Cloud guide to create, test, evaluate, and deploy your prompts.

👥 Community

The Latitude community can be found on Slack where you can ask questions, voice ideas, and share your projects with other people.

🤝 Contributing

Contributions to Latitude are welcome and highly appreciated.

If you are interested in contributing, please join us on our Slack community, open an issue, or contribute a pull request.

🔗 Links

• Home page
• Documentation
• Slack community
• X / Twitter

latitude-llm's Issues

Make Documentation

What?

☝️ That

Move files from one folder to another in Sidebar

What?

We have to allow users to reorganize files. The same as with renaming files we need to take into account that moving a file or renaming it will possibly break others prompts that are referencing the moved file.

TODO

• Think if it's possible to auto-updated references cc @csansoon
• Show modal warning when moving a file that can break things if we don't implement auto update references
• Implement drag & drop in sidebar #fun

Add sentry to gateway and workers also

Extract types in its own package

Rename files from sidebar

What?

Now is not possible to rename files. I think the flow should be similar to create files using the same UI cc @samulatitude ?

Adjust compiler for prompting

Publish SDK in NPM

What?

We need to publish typescript sdk in npm before open beta

Add email parameter to signup

Info – Message when running a prompt without provider

Navigation – Redirect to last draft viewed instead to latest

What?

When editing a draft, if I leave and come back again, I arrive at the latest version instead of the draft I was editing. I think it is better UX to go to the last version I visited.

Cesar: When going back into the project, we should keep the state of the version the user last saw

Deploys break the app because of missing JS chunks

Add Sentry to web and api apps

Run document changes

What?

We want to change how some things are done before start using provider logs

TODO

• documentLog and last providerLog should be created synchronously after streaming is fininshed.
• providerLog should have a polymorphic nullable relation, either to documentLog, evaluationResult, or none.
• documentLog should know which of the providerLog that points to them is the latest step from the chain.
• providerLog should have a timeUntilFirstToken property.

Latitude SDKs placeholders

Navigation – When creating the first document, user should get redirected to it

Add sentry

Kickstart project structure

TODO

• Setup monorepo
• Setup Database + ORM + Migrations
• Setup jobs

Versions – When creating a new version, it should optimistically update the frontend

Add Document editor data to SWR local cache

What?

Now each time we navigate to a document it's fetched from the server because is not cached in our SWR local cache.

Do something similar to what we do with sidebar documents

Versions – First project version should not be live to let users make changes directly

What?

Instead of showing a published version users can't edit we should show a draft called "My first draft" and allowing users create or edit files directly.

Build UI fundamentals

What's the plan?

We want to have centralized in one package all the UI elements and screens for the web app. This package will be divided in two exports ds (Design System) and sections. Following a bit atomic design the idea is that DS holds all the basic elements of the app like buttons, forms, popups, tooltips,...on the other hand screens hold full app screens that are formed with ui elements some structural layouts that are part also of screens export and tailwind styles.

packages/
  web-ui
    src/
      sections/
      ds/

import { EditPrompt } from '@latitude-data/web-ui/sections'
import { Button } from '@latitude-data/web-ui/ds'

This package use under the hood @latitude-data/core for interaction with jobs and database when appropriate. Or the next app can pass the correct server action.

Navigation – Switching to another version should happen when clicking the row in the version list