schneiderfelipe / chat-splitter

Split chat messages by maximum chat completion token count

chat-splitter

For more information, please refer to the blog announcement.

When utilizing the async_openai Rust crate, it is crucial to ensure that you do not exceed the maximum number of tokens specified by OpenAI's chat models.

chat-splitter categorizes chat messages into 'outdated' and 'recent' messages, allowing you to split them based on both the maximum message count and the maximum chat completion token count. The token counting functionality is provided by tiktoken_rs.

Usage

Here's a basic example:

// Get all your previously stored chat messages...
let mut stored_messages = /* get_stored_messages()? */;

// ...and split into 'outdated' and 'recent',
// where 'recent' always fits the context size.
let (outdated_messages, recent_messages) =
    ChatSplitter::default().split(&stored_messages);

For a more detailed example, see examples/chat.rs.

Contributing

Contributions to chat-splitter are welcome! If you find a bug or have a feature request, please submit an issue. If you'd like to contribute code, please feel free to submit a pull request.

License: MIT

chat-splitter's Issues

Support pruning chat messages

Receive a closure Fn(&M, &[M]) -> bool receiving the oldest message and all the next ones and indicating whether a single oldest message should be pruned. Call it and prune accordingly until it returns false or the total messages in the split is empty. Pruning here means advancing the split position by one unit.

Review, correct and improve documentation

This includes comments in code. It would be nice to review for typos, rewording, etc.

Rename to chat-splitter

Just like text-splitter. This crate only splits chats.

Provide a default implementation of IntoChatCompletionRequestMessage::into_tiktoken_rs

chat-splitter/src/lib.rs

Lines 259 to 261 in c287c0f

	/// Convert to [`tiktoken_rs` chat completion request message
	/// type](`tiktoken_rs::ChatCompletionRequestMessage`).
	fn into_tiktoken_rs(self) -> tiktoken_rs::ChatCompletionRequestMessage;

Release v0.2.0