schneiderfelipe / ellie

echo "It's dangerous to go alone" | ellie

Discover my latest contributions and projects on GitHub.

Explore more of my work at https://schneiderfelipe.github.io/.

ellie's Issues

Consider accepting multiple JSON objects from functions

Currently, the results of a function call are given back to the AI exactly how they are received.

We could try to parse zero, one or more JSON objects out of it and, in case we're able to parse more than one, we might compact each and send a subset of context-relevant objects instead of all of them.

This would free functions from the burden of trying to keep their responses to a minimum, and instead encourages functions to just split their results as much as possible, leaving to ellie to choose the most relevant ones, since ellie is aware of the conversation context.

serde_json::StreamDeserializer is probably the right way of doing it.

ellie/src/functions.rs

Lines 42 to 52 in 487ec27

	#[inline]
	pub(super) fn call(&self, arguments: &str) -> std::io::Result<String> {
	let content = duct::cmd(&self.command, &self.args)
	.stdin_bytes(arguments)
	.read()?;
	// TODO: in the future we might accept multiple json objects and select based on
	// messages.
	// Kind of tricky though.
	// Selection may happen in src/main.rs so return some kind of iterator?
	Ok(try_compact_json(&content))
	}

Ecosystem

Applications such as ripgrep already produce that kind of output with the --json flag:

$ rg todo -i --json
{"type":"begin","data":{"path":{"text":"src/functions.rs"}}}
{"type":"match","data":{"path":{"text":"src/functions.rs"},"lines":{"text":"            // TODO: ignore provider if spec fails,\n"},"line_number":150,"absolute_offset":4585,"submatches":[{"match":{"text":"TODO"},"start":15,"end":19}]}}
{"type":"end","data":{"path":{"text":"src/functions.rs"},"binary_offset":null,"stats":{"elapsed":{"secs":0,"nanos":17536,"human":"0.000018s"},"searches":1,"searches_with_match":1,"bytes_searched":4890,"bytes_printed":297,"matched_lines":1,"matches":1}}}
...

And applications such as jq already accept the same kind of input as above.

Retrieve previous chat context

ellie does not currently know about previous interactions. It should get the previous chat history every time in the future.

It should get the immediate chat context by using chat-splitter, possibly look further in the past for more relevant information, and add a suitable system message right before sending the context to the AI model.

ellie/src/main.rs

Lines 93 to 110 in 74260d1

	/// Get chat messages ending in the given new messages,
	/// essentially building context to them.
	#[inline]
	fn create_chat_messages(
	new_messages: &[aot::ChatCompletionRequestMessage],
	) -> Vec<aot::ChatCompletionRequestMessage> {
	// TODO: actually get previous chat messages,
	// and split/prune to fit the context.
	// Also add other relevant messages and prepend suitable system message.
	// This should always produce a valid context for at least *one* of the MODELS.
	// Tentative workflow:
	// 1. split/prune with the shortest context-length model in mind (chat-splitter)
	// 2. add other extremely relevant messages from past interactions
	// 3. split/prune again with the longest context-length model in mind
	// (chat-splitter)
	// 4. choose a suitable system message as well to prepend.
	new_messages.to_owned()
	}

Consider configuring the client to avoid hanging

ellie currently uses a default client for requesting from the OpenAI's API endpoint. Consider giving more options to it, such as for configuring timeout, back-off strategy and probably caching.

ellie/src/main.rs

Lines 275 to 290 in 74260d1

	// TODO: timeout,
	// backoff,
	// etc.
	// Just to avoid hanging.
	let client = async_openai::Client::new();
	while !matches!(
	new_messages
	.iter()
	.last()
	.expect("there should always be at least one new message")
	.role,
	aot::Role::Assistant
	) {
	let messages = create_chat_messages(&new_messages);
	let request = create_request(messages)?;
	let response = create_response(&client, request).await?;

Store interactions

Currently ellie throws away messages exchanged in each interaction. It should store them for use as context in future interactions (#9). Storing should happen atomically: either all messages in a given interaction are stored or, in case of an error, none are stored (interactions with no function calling have an user-assistant message pair, which is probably the most common case).

ellie/src/main.rs

Lines 273 to 303 in 74260d1

	let mut new_messages = vec![user_message];
	// TODO: timeout,
	// backoff,
	// etc.
	// Just to avoid hanging.
	let client = async_openai::Client::new();
	while !matches!(
	new_messages
	.iter()
	.last()
	.expect("there should always be at least one new message")
	.role,
	aot::Role::Assistant
	) {
	let messages = create_chat_messages(&new_messages);
	let request = create_request(messages)?;
	let response = create_response(&client, request).await?;
	let assistant_message = create_assistant_message(response)
	.await
	.context("creating assistant message")?;
	update_new_messages(&mut new_messages, assistant_message)?;
	}
	// TODO: store new messages with atomicity guarantees: if
	// something fails,
	// nothing is stored,
	// so better store everything at the end.
	// Store them as groups of messages ("interactions or sessions"),
	// which might help with debugging in the future.

Refine previous utterance

After #12 and #9, support a git commit-like interface for modifying the previous interaction (ellie --refine):

• open editor with previous utterance (what was stored)
• maybe incorporate data from the standard input, maybe not, but if so, support both (priority is refining)
• provide both user and assistant messages, but they should modify it so that the assistant message is removed
• present instructions to make things easier
• user modifies the text and it is fed to the model as user message
• the resulting utterance replaces the one stored
• if the user leaves text empty, we abort the procedure
• maybe, just maybe, come up with an extra improved prompt by using the model itself, but always provide the original user message as well

The key thing is to make it extremely easy to

• always use good prompts
• always have good answers
• always have the best utterances stored
• always be able to fix mistakes the moment they appear
• be able to iterate over a single utterance as a microcycle, the macrocycle being the complete conversation

Consider only using relevant funtions based on context

ellie currently sends all known specifications to the AI model. It should mildly select relevant functions based on chat context instead, in a way steering the mode towards more focused decisions and saving token costs.

ellie/src/main.rs

Lines 112 to 138 in 74260d1

	/// Create an `OpenAI` request.
	///
	/// # Errors
	/// If a model could not be chosen for the given messages,
	/// or if functions could not be retrieved.
	#[inline]
	fn create_request(
	messages: Vec<aot::ChatCompletionRequestMessage>,
	) -> eyre::Result<aot::CreateChatCompletionRequest> {
	let mut request = aot::CreateChatCompletionRequestArgs::default();
	request.temperature(TEMPERATURE);
	let model = choose_model(&messages)
	.context("choosing model with large enough context length for the given messages")?;
	log::info!("{model}");
	request.model(model);
	// TODO: choose relevant functions based on the chat messages before collecting.
	let functions = functions::Functions::load()
	.unwrap_or_default()
	.specifications()
	.collect::<Result<Vec<_>, _>>()?;
	if !functions.is_empty() {
	request.functions(functions);
	}
	Ok(request.messages(messages).build()?)
	}

Ignore provider if their specification call throws errors

ellie currently crashes if a provider fails when we request its specification. We should ignore the provider in those cases and give a warning instead. "Ignoring" here means the AI model will never know about the function to start with.

ellie/src/functions.rs

Lines 149 to 162 in 74260d1

	#[inline]
	pub(super) fn specifications(
	&self,
	) -> impl Iterator<Item = eyre::Result<ChatCompletionFunctions>> + '_ {
	self.providers().map(|provider| {
	// TODO: ignore provider if spec fails,
	// but warn the user
	let mut spec = provider.specification()?;
	if let Some(function) = self.get_function(&spec.name) {
	merge(&mut spec, function);
	}
	Ok(spec)
	})
	}

EDIT: This could also apply to malformed specifications¹ (i.e., the provider runs fine, but a bad specification is given). It is fine to ignore said function by the same logic as above.

ellie/src/functions.rs

Lines 170 to 187 in 13e9e66

	#[inline]
	pub(super) fn specifications(
	&self,
	) -> impl Iterator<Item = eyre::Result<ChatCompletionFunctions>> + '_ {
	self.providers().map(|provider| {
	let mut spec = provider.specification()?;
	if let Some(function) = self.get_function(&spec.name) {
	merge(&mut spec, function);
	}
	// TODO: warn/err on the user when a function has a malformed specification and
	// probably ignore said function.
	// The bare minimum seems to be name,
	// parameters,
	// parameters>type,
	// parameters>properties.
	Ok(spec)
	})
	}

Footnotes

1. The minimal specification seems to have name and parameters, and the parameters have a type and properties. ↩