intelligencedev / eternal

Eternal is an experimental platform for machine learning models and workflows.

Eternal

Eternal is an experimental platform for machine learning workflows.

NOTE: This app is a work in progress and not stable. Please consider this repo for your reference. We welcome contributors and constructive feedback. You are also welcome to use it as reference for your own projects.

Eternal integrates various projects such as llama.cpp, stable diffusion.cpp and codapi among many other projects whose developers were kind enough to share with the world. All credit belongs to the respective contributors of all dependencies this repo relies on. Thank you for sharing your projects with the world.

The Eternal frontend is rendered with the legendary HTMX framework.

IMPORTANT:

Configure the quant level of the models in your config.yml appropriately for your system specs. If a local fails to run, investigate the reason by viewing the generated main.log file. The most common reason is insufficient RAM or incorrect prompt template. We will implement more robust error handling and logging in a future commit.

Features

• Language model catalog for easy download and configuration.
• Text generation using local language models, OpenAI GPT-4 an Google Gemini 1.5 Pro.
• Web retrieval that fetches URL content for LLM to reference.
• Web Search to automatically retrieve top results for a user's prompt for LLM to reference. Requires Chrome browser installation.
• Image generation using Stable Diffusion backend.

Getting Started

Basic documentation is provided in the docs folder in this repository.

Showcase

Web Retrieval - Search

Web tools can be enabled via the config.yml file.

• webget: Attempts to fetch a URL passed in as part of the prompt.
• websearch: Searches the public web for pages related to your prompt.

Requires Chrome browser installation.

Code Execution

Execute and edit LLM generated code in the chat view in a secure sandbox. For now, JavaScript is implemented via WASM. More languages coming soon!

Image Generation

Embedded Stable Diffusion for easy high quality image generation.

Configuration

Rename the provided .config.yml file to config.yml and place it in the same path as the application binary. Modify the contents for your environment and use case.

Build

Eternal currently supports building on Linux or Windows WSL using CUDA (nVidia GPU required) or MacOS/Metal (M-series Max required).

To build the application:

$ git clone https://github.com/intelligencedev/eternal.git
$ cd eternal
$ git submodule update --init --recursive
$ make all

Please submit an issue if you encounter any issues with the build process.

Disclaimer

This README is a high-level overview of the Eternal application. Detailed setup instructions and a complete list of features, dependencies, and configurations should be consulted in the actual application documentation.

eternal's Issues

Validate external app dependencies and bootstrap them if not present.

Eternal will require third party application installations for the various tools that will not be embedded. Currently, the web tool requires a headless Chrome install. Chromium would be ideal but it has not been tested yet. We need a method to detect if this app is installed in the end user's OS, and if not, install it.

The Go ChromeDP package may have methods to do this. Otherwise we will implement our own.

If its possible to do this via the package, then the implementation should go in the web tool:

https://github.com/intelligencedev/eternal/blob/main/pkg/web/web.go

Otherwise, a generic method in the main app will suffice to run the process for any third party dependency.

Add a collection of advanced LLM system prompt templates for different agent roles.

The tooling is there to apply custom prompt templates during inference workflows. While the model configs contain local templates that could be configured via the config.yml, we should add more advanced templates such as Chain of Thought, etc. to prepare for the advanced agent workflows that will be reimplemented soon.

Add image generation tool.

Implement the image generation feature. The obvious choice is Stable Diffusion but there is more competition in that space. We will start with Stable Diffusion, but also investigate how we could potentially add support for other image gen models and backends.

Investigate methods for secure code execution. Bonus points if we can run it from the code blocks in the frontend.

We need to implement a code runner solution that is sandboxed in a secure environment. Most applications in this space use a container for this purpose but there may be other options. We should explore an option that does not require a third party OS install such as WASM, but that may not be ideal. Some options to research:

• WASM
• MicroVM
• Container
• A secure sandbox like Firejail or similar alternatives that works across Linux, Mac and Windows. This is unlikely, but just leaving here for reference.

We also want to be able to execute code generated by the LLM via the frontend by triggering an event using a play button that gets injected into code blocks. That may need to be a separate issue but let's leave the idea here for now.

Enhance the code block elements - Close the block as its generated and add a copy button.

The current implementation does not correctly format large code blocks as they are being generated due to the closing tag not being generated by LLM. We need a method to close the blocks as the code is being generated so they display properly from the start.

We also need a copy button on the top right of the block so the end user can easily copy the code.

The copy button will need to be implemented in the chat template:

https://github.com/intelligencedev/eternal/blob/main/public/templates/chat.html

Update model state when it is finished downloading.

When a model finishes downloading it should trigger a callback to update its state in the DB and frontend so we do not have to restart the application to be able to use it.

Configure the CI pipeline for Metal, CUDA, and ROCm.

Eternal currently requires some manual tweaking to build properly due to the lack of proper automation for the build process. This issue serves as a generic reminder that we need to put this on autopilot and have a CI pipeline built to automatically build and publish new releases.