log10-io / log10

Python client library for improving your LLM app accuracy

log10

⚡ Unified LLM data management to drive accuracy at scale ⚡

Quick Install

pip install log10-io

🤔 What is this?

A one-line Python integration to manage your LLM data.

import openai
from log10.load import log10

log10(openai)
# all your openai calls are now logged - including 3rd party libs using openai

For OpenAI v1, use from log10.load import OpenAI instead of from openai import OpenAI

from log10.load import OpenAI

client = OpenAI()

Access your LLM data at log10.io

🚀 What can this help with?

📝📊 Logging

Use Log10 to log both closed and open-source LLM calls. It helps you:

• Compare and identify the best models and prompts (try playground and llmeval)
• Store feedback for fine-tuning
• Collect performance metrics such as latency and usage
• Perform analytics and monitor compliance for LLM powered applications

Log10 offers various integration methods, including a python LLM library wrapper, the Log10 LLM abstraction, and callbacks, to facilitate its use in both existing production environments and new projects. Pick the one that works best for you.

OpenAI

log10 ver	openai v0	openai v1
0.4	log10(openai) ✅	❌
0.5+	log10(openai) ✅	from log10.load import OpenAI ✅

OpenAI v0 - Use library wrapper log10(openai). Check out examples/logging in log10 version 0.4.6.

import openai
from log10.load import log10

log10(openai)
# openai calls are now logged - including 3rd party libs using openai such as magentic or langchain

OpenAI v1

NOTE: We added OpenAI v1 API support in log10 0.5.0 release. load.log10(openai) still works for openai v1. This also enables logging LLM completions from providers which support OpenAI API, such as Ollama.

from log10.load import OpenAI
# from openai import OpenAI

client = OpenAI()
completion = client.completions.create(model="gpt-3.5-turbo-instruct", prompt="Once upon a time")
# All completions.create and chat.completions.create calls will be logged

Full script here.

Use Log10 LLM abstraction

from log10.openai import OpenAI

llm = OpenAI({"model": "gpt-3.5-turbo"}, log10_config=Log10Config())

openai v1+ lib required. Full script here.

Anthropic

Use library wrapper log10(anthropic). Full script here.

import anthropic
from log10.load import log10

log10(anthropic)
# anthropic calls are now logged

Use Log10 LLM abstraction. Full script here.

from log10.anthropic import Anthropic

llm = Anthropic({"model": "claude-2"}, log10_config=Log10Config())

Open-source LLMs

Log open-source LLM calls, e.g. Llama-2, Mistral, etc from providers. Currently we support inference endpoints on Together.AI and MosaicML (ranked on the top based on our benchmarking on Llama-2 inference providers). Adding other providers is on the roadmap.

MosaicML with LLM abstraction. Full script here.

from log10.mosaicml import MosaicML

llm = MosaicML({"model": "llama2-70b-chat/v1"}, log10_config=Log10Config())

Together with LLM abstraction. Full script here.

from log10.together import Together

llm = Together({"model": "togethercomputer/llama-2-70b-chat"}, log10_config=Log10Config())

Other LLM frameworks

Use Log10 callbacks if you use LangChain's LLM abstraction. Full script here.

from langchain.chat_models import ChatOpenAI
from langchain.schema import HumanMessage

from log10.langchain import Log10Callback
from log10.llm import Log10Config

log10_callback = Log10Callback(log10_config=Log10Config())

messages = [
    HumanMessage(content="You are a ping pong machine"),
    HumanMessage(content="Ping?"),
]

llm = ChatOpenAI(model_name="gpt-3.5-turbo", callbacks=[log10_callback])

Read more here for options for logging using library wrapper, langchain callback logger and how to apply log10 tags here.

🤖👷 Prompt engineering copilot

Optimizing prompts requires a lot of manual effort. Log10 provides a copilot that can help you with suggestions on how to optimize your prompt.

👷🔢 Feedback

Add feedback to your completions. Checkout the Python example or use CLI log10 feedback-task create and log10 feedback create. Please check our doc for more details.

AutoFeedback

Leverage your current feedback and AI by using our AutoFeedback feature to generate feedback automatically. Here’s a quick guide:

• Summary feedback: Use TLDR summary feedback rubics to rate summarization. E.g. log10 feedback predict --task_id $FEEDBACK_TASK_ID --content '{"prompt": "this is article", "response": "summary of the article."}'.
  • You can pass a file containing the context with --file or pass a completion from your Log10 logs with --completion_id.
• Custom Feedback Rubrics: Integrate your own feedback criteria for personalized assessments.
• Getting Started: To explore all options and usage details, use CLI log10 feedback predict --help.

Feel free to integrate AutoFeedback into your workflow to enhance the feedback and evaluation process.

⚖️📊 Model Comparison

Easily benchmark your logged completions using LLM models from OpenAI, Anthropic, Mistral, Meta, etc., by using the log10 completions benchmark_models command in the log10 CLI. Generate detailed reports and gain insights to enhance your model's performance and cost. Please refer to the cli doc or the demo video for details.

🔍🐞 Prompt chain debugging

Prompt chains such as those in Langchain can be difficult to debug. Log10 provides prompt provenance, session tracking and call stack functionality to help debug chains.

🧠🔁 Readiness for RLHF & self hosting

Use your data and feedback from users to fine-tune custom models with RLHF with the option of building and deploying more reliable, accurate and efficient self-hosted models.

👥🤝 Collaboration

Create flexible groups to share and collaborate over all of the above features

⚙️ Setup

1. Create a free account at log10.io
2. Set the following environment variables:

• LOG10_URL=https://log10.io
• LOG10_TOKEN: From the Settings tab in log10.io
• LOG10_ORG_ID: From the Organization tab in log10.io
• OPENAI_API_KEY: OpenAI API key
• ANTHROPIC_API_KEY: Anthropic API key

✅ Run examples and tests

You can find and run examples under folder examples, e.g. run a logging example:

python examples/logging/chatcompletion.py

Also you can run some end-to-end tests with xdocttest installed (pip install xdoctest).

# list all tests
python -m xdoctest log10 list

# run all tests
python -m xdoctest log10 all

# run a single test, e.g.
python -m xdoctest /Users/wenzhe/dev/log10/log10/load.py log10:0

Logging

Few options to enable debug logging:

1. set environment varible export LOG10_DEBUG=1
2. set log10.load.log10(DEBUG_=True) when using log10.load
3. set log10_config(DEBUG=True) when using llm abstraction classes or callback.

💿🧩 Flexible data store

log10 provides a managed data store, but if you'd prefer to manage data in your own environment, you can use data stores like google big query.

Install the big query client library with:

pip install log10-io[bigquery]

And provide the following configuration in either a .env file, or as environment variables:

Name	Description
LOG10_DATA_STORE	Either log10 or bigquery
LOG10_BQ_PROJECT_ID	Your google cloud project id
LOG10_BQ_DATASET_ID	The big query dataset id
LOG10_BQ_COMPLETIONS_TABLE_ID	The name of the table to store completions in

Note that your environment should have been setup with google cloud credentials. Read more here about authenticating.

CLI for completions and feedback

We add CLI to manage your completions and feedback. Read more here.

💬 Community

We welcome community participation and feedback. Please leave an issue, submit a PR or join our Discord. For enterprise use cases, please contact us to set up a shared slack channel.

log10's Issues

Capture exception in a completion logging

Add dependency locking

The current setup with a single requirements.txt files do not provide version for dependencies. This works in the short term, but in the longer term when things stop working it would be really complicated to know what version of each dependency we were using.

The best practice should be to commit a lock file under version control, such as a pipenv.lock. However, we could also take the opportunities to move to poetry, which is more popular (25 k stars vs 4.7k stars compared to hatch), and a better support from tools in the ecosystem (mypy, flake8, etC)

Check credentials on load(), not on library import

A user may have to do some initialization i.e. set up environment variables programatically. Currently, we try to get a session id on library initialization, so that fails if everything isn't ready (URL, token etc) by then.

RFC for Evaluations

We are collecting feedback on Evaluations functionality for log10.io and the client libraries.
The document is here

An example for a code analysis + generation + compilation evaluation use case here

We welcome your input!

Session context length setting

Provide a way to wrap openai calls, and get new log10 session ids:

with log10.session():
  ...

Clean up load.py

local llm support?

any plans to support local backends like llama.cpp, text gen. ui, etc?

Programmatic access to completion URLs

Completion URLs can be printed using the DEBUG_ mode when wrapping openai, like so:

log10(openai, DEBUG_=True)

But is a bit verbose, and not easily grabbed programmatically

Stricter linting and CI/CD

At the moment we do have only a release pipeline, but it would be worth to have at least a linting pipeline to verify the code is somehow correct. Such a pipeline would run an opinionated linter, such as wemake-python-styleguide and encourage better practices.

For example, this code

def camel_agent(
    user_role: str,
    assistant_role: str,
    task_prompt: str,
    max_turns: int,
    user_prompt: str = None,
    assistant_prompt: str = None,
    summary_model: str = None,
    llm: LLM = None,

presents a couple of problems. First of all, given the large number of arguments that have the same type, it's easy to make mistakes in inverting the error, and for those scenarios Python introduced keyword-only named arguments

def camel_agent(
    * ,
    user_role: str,
    assistant_role: str,
    task_prompt: str,
    max_turns: int,
    user_prompt: str = None,
    assistant_prompt: str = None,
    summary_model: str = None,
    llm: LLM = None,

which would force client code to invoke it like so:

camel_agent(
   user_role='biochemist',
  assistant_role='professor',
  ...
)

Also, the code would not pass mypy, because None are not valid types for str. It should rather be Optional[str] like so:

def camel_agent(
    * ,
    user_role: str,
    assistant_role: str,
    task_prompt: str,
    max_turns: int,
    user_prompt: Optional[str] = None,
    assistant_prompt: Optional[str] = None,
    summary_model: Optional[str] = None,
    llm: LLM = None,