premai-io / prem-daemon Goto Github PK

Description

Enable the possibility to use Plugins in Prem Chat.

Description

Prem Daemon should be able to run, stop, etc the prem services on k8s, docker swarm as well.

Description

Daemon should be able to run binaries as well. Not only Docker images.

Use Tabby as first binary based service: https://tabbyml.github.io/tabby/docs/self-hosting/apple.

Description

Integrate All-MiniLM-L6-v2

If the user doesn't have enough storage in order to download the service. add enoughStorage flag as false, otherwise true.

Description

When fetching system resources, get info from all the GPUs in the system.

Description

• Expose endpoint to encrypt data.
• Encrypt user data
• Backup data in s3 bucket
• Nostr login

Description

Expose the following models using dall-e wrapper.

• CompVis/stable-diffusion-v1-4
• runwayml/stable-diffusion-v1-5
• stabilityai/stable-diffusion-2-base
• stabilityai/stable-diffusion-2-1-base
• OFA-Sys/small-base-diffusion-v0

Description

• Expose an endpoint GET /update-available : bool

Return True if there is a new version available for premd container.

• Expose an endpoint POST /update-daemon

The endpoint should restart premd with the latest version.

Download events sent to the frontend should ignore those status:

 if (
                status == Status.ALREADY_EXISTS.value
                or status == Status.DOWNLOAD_COMPLETE.value
                or status.startswith("Pulling from")
                or status.startswith("Pulling fs layer")
            ):
                continue

I am trying to build a cross-architecture image from the main branch but I have the following problem using docker buildx

docker buildx build --push --file Dockerfile --tag ghcr.io/premai-io/premd:7af3782b00f6176b34840f608327c051502511ce --platform linux/arm64,linux/amd64 .

TRUNCATED

13.71       gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DPSUTIL_POSIX=1 
-DPSUTIL_SIZEOF_PID_T=4 -DPSUTIL_VERSION=595 -DPy_LIMITED_API=0x03060000 -DPSUTIL_ETHTOOL_MISSING_TYPES=1 -DPSUTIL_LINUX=1 
-I/usr/local/include/python3.10 -c psutil/_psutil_common.c -o build/temp.linux-aarch64-cpython-310/psutil/_psutil_common.o
13.71       In file included from /usr/lib/gcc/aarch64-linux-gnu/10/include/syslimits.h:7,
13.71                        from /usr/lib/gcc/aarch64-linux-gnu/10/include/limits.h:34,
13.71                        from /usr/local/include/python3.10/Python.h:11,
13.71                        from psutil/_psutil_common.c:9:
13.71       /usr/lib/gcc/aarch64-linux-gnu/10/include/limits.h:195:15: fatal error: limits.h: No such file or directory
13.71         195 | #include_next <limits.h>  /* recurse down to the real one */
13.71             |               ^~~~~~~~~~
13.71       compilation terminated.
13.71       psutil could not be installed from sources. Perhaps Python header files are not installed. Try running:
13.71         sudo apt-get install gcc python3-dev
13.71       error: command '/usr/bin/gcc' failed with exit code 1
13.71       [end of output]
13.71
13.71   note: This error originates from a subprocess, and is likely not a problem with pip.
13.71   ERROR: Failed building wheel for psutil
13.71   Building wheel for gputil (setup.py): started
13.89   Building wheel for gputil (setup.py): finished with status 'done'
13.89   Created wheel for gputil: filename=GPUtil-1.4.0-py3-none-any.whl size=7392 
sha256=8f6ed156a2f152a786e47693db546f331bd4d07aa8b0ac4c54ba9b1df21cffda
13.89   Stored in directory: 
/tmp/pip-ephem-wheel-cache-cwcq8rvs/wheels/a9/8a/bd/81082387151853ab8b6b3ef33426e98f5cbfebc3c397a9d4d0
13.89 Successfully built gputil
13.89 Failed to build psutil
13.89 ERROR: Could not build wheels for psutil, which is required to install pyproject.toml-based projects
14.12
14.12 [notice] A new release of pip is available: 23.0.1 -> 23.3.1
14.12 [notice] To update, run: pip install --upgrade pip
------
Dockerfile:9
--------------------
   7 |         && rm -rf /var/lib/apt/lists/*
   8 |     COPY requirements.txt .
   9 | >>> RUN pip install --no-cache-dir -r requirements.txt
  10 |
  11 |     COPY . .
--------------------
ERROR: failed to solve: process "/bin/sh -c pip install --no-cache-dir -r requirements.txt" did not complete successfully: exit 
code: 1

Description

Expose endpoint and logic in order to finetune a model. Same interface as OpenAI APIs.

Summary

premd needs to adapt its container orchestration method based on different deployment types. Specifically, premd should consider whether API calls from premapp to premd are coming through a proxy or not. If they are, then premd should start services with specific Docker labels to help Traefik configure routes to prem-services.

DNS-Dependent Labels

If a DNS record has been created for the domain, the following labels should be added when starting the Docker container:

• "traefik.enable=true"
• "traefik.http.routers.dnsd.rule=Host(`{SERVICE_ID}.domain`)"
• "traefik.http.routers.{SERVICE_ID}.entrypoints=websecure"
• "traefik.http.routers.{SERVICE_ID}.tls.certresolver=myresolver"
• "traefik.http.routers.{SERVICE_ID}.entrypoints=websecure"

If a DNS record has not been created:

• "traefik.http.routers.dnsd.rule=Host({SERVICE_ID}.docker.localhost)"
• "traefik.enable=true"

Suggestions

1. Introduce New Variable: Add a new environment variable called PROXY_ENABLED.
2. Method Update: If PROXY_ENABLED=true, update the run_container_with_retries method to send the specific Docker labels as outlined above.
3. DNSD Endpoint: Reaching dnsd should be possible at http://dnsd:8080.
4. Optimization: Cache the first successful check for an existing DNS record in memory to avoid unnecessary, repeated calls to dnsd.

Additional Resources

For a comprehensive list of all relevant deployment types for premd, refer here, also note that for premd it is only relevant if PROXY is enabled or not.

Steps to reproduce:
POST /v1/registries
Body { url: "https://whatever/manifests.json" }

Returns error 400, but the url is still added to the backend.

Description

Cookiecutter template for Prem Services.

NOTES

Potentially prem package should be a CLI module too and handle this part.

Description

Expose the following models

• https://github.com/TabbyML/tabby
• https://github.com/ravenscroftj/turbopilot
• https://github.com/fauxpilot/fauxpilot
• https://huggingface.co/replit/replit-code-v1-3b
• https://huggingface.co/blog/starcoder

Description

Create a simple talk to your data use case with a Python notebook using entirely prem-box.

• Create a new directory notebooks in order to store these examples.

Description

The endpoint should avoid to return duplicates.

Description

Text to video services

Description

Expose services in order to compute summary of a text

Running the latest main branch in a docker container in the logs

2023-11-01 14:29:13 2023-11-01 13:29:13 ERROR    An error occurred: argument of type 'NoneType' is not iterable 

Currently, the percentage doesn't reflect the real state of the download, It's a hacky solution for now.
the best approach would be to move to download the image as tar.gz via the docker save command.

We may need to check what the native SDK of Pythong allows to monitor the advancement of the file download, otherwise, we can check the file size on disk by checking polling the size.

Consider using SSE (Server Side Events to use EventSource browser API) but would be helpful to make sure that if a browser client re-connects to the same image download request, it keeps downloading in the background and return the updated progress.

Description

In order to avoid Docker as a dependency for what concerns the services orchestration, we can use Nomad.

Benefits

• No need to leverage Docker as a dependency on Prem App while running on Desktop. You can just run nomad as a binary and spawn the services as binaries. Check here
• We can still leverage the use of Docker when Prem running on a Server/Cluster.
• Same orchestrator for Local env and Production env.
• It exposes resource management APIs, no need to handle this aspect using docker stats or psutil library. You can check some of the commands here

Task List

• Investigate if we can completely remove the daemon and interact with Nomad from Prem App without missing any functionality both short and long term.
• Explore k8s integration. How Nomad can run in a k8s cluster? What's the currently available integration?
• Reimplement all the methods interacting with the docker client to use Nomad instead.
• In order to avoid breaking changes, keep both docker and nomad methods. Pass ORCHESTRATOR=docker | nomad as the environment variable to decide which orchestrator to use.
• Compile the Daemon using pyinstaller

Notes

There is a nomad python client available here.

Additional Notes related to UX

1. If we run Nomad as binary and the Daemon too, then we can run Prem App in Swarm Mode using the Daemon instead of using Tauri commands.
2. Both Daemon and Nomad can be run using Tauri Sidecar

Description

Expose data augmentation endpoints. Use self-instruct method or similar methods.

Noticed HTTP status codes (e.g. 200, 400, etc.) are hard-coded atm. Would be better to use http.HTTPStatus?

PREM_REGISTRY_URL could be a whitespace-delimited list?

Description

Expose the following models

• https://github.com/facebookresearch/dinov2
• https://github.com/facebookresearch/AnimatedDrawings
• https://github.com/facebookresearch/segment-anything

Note

• Include state of the art models from HF for different computer vision tasks: https://huggingface.co/tasks

Description

Expose diffusers pipelines in order to upscale images

Description

pyinstaller could be an option.

on a system with 2xV100s, the Prem App seems to ignore the second GPU. It thinks there isn't enough GPU RAM for some small models even though the second GPU is unused.

If the url is not a valid registry, deletion will fail.

Summary

In relation to issue 128, the get_service_object function needs an update to return the full URL for prem-service. The URL will be dependent on whether PROXY_ENABLED is true and if a DNS record exists.

DNS Dependent URL Construction

1. If DNS is created: Construct the URL in the format {SERVICE_ID}.domain
2. If DNS is not created: Construct the URL in the format {SERVICE_ID}.docker.localhost

Invoke service based on URL suffix

Applications that are to invoke running prem-service will know, based on if url contains suffix 'docker.localhost', how to 'reach' it.

• With .docker.localhost suffix: Applications should add a Host: {URL} header and send the request to {HOST}.
• Without .docker.localhost suffix: Applications should directly send the request to {URL} without adding a Host header.

Currently if a registry already exists the registry is added anyway.

Description

Return the backendUrl for each service.

Description

Mobile app that uses Mac as a server.

Description

• https://www.mosaicml.com/blog/mpt-7b

Description

An error involving sentry_sdk appeared on ARM hardware. In order to avoid similar issues, we should run the tests on both AMD and ARM in the CI/CD pipeline.

Description

Endpoint to restart a service

Description

Integrate OpenAssistant Llama 30B XOR

Description

Expose the following models:

• https://github.com/openai/whisper (all the dimensions)
• https://github.com/sanchit-gandhi/whisper-jax
• https://github.com/suno-ai/bark

Decription

Currently the download value doesn't fully reflect the status. Find a way to improve UX

On daemon 0.0.24, running without -e PROXY_ENABLED=False

 let _child = thread::spawn(move || {
        let status = Command::new("/usr/local/bin/docker")
            .args(&[
                "run",
                "-d",
                "-v",
                "/var/run/docker.sock:/var/run/docker.sock",
                "-p",
                "54321:8000",
                "--name",
                "premd",
                "-e",
                "PREM_REGISTRY_URL=https://raw.githubusercontent.com/premAI-io/prem-registry/main/manifests.json",
                "--rm",
                image.as_str(),
            ])
            .status();

results in error

2023-09-27 14:29:22 pydantic.error_wrappers.ValidationError: 32 validation errors for ServiceResponse
2023-09-27 14:29:22 response -> 0 -> invokeMethod
2023-09-27 14:29:22   field required (type=value_error.missing)
2023-09-27 14:29:22 response -> 1 -> invokeMethod
2023-09-27 14:29:22   field required (type=value_error.missing)
...

Even though PROXY_ENABLED=False should be the default.

Running with -e PROXY_ENABLED=False results in error:

2023-09-27 14:26:54 2023-09-27 12:26:54 ERROR    An error occurred: HTTPConnectionPool(host='dnsd', port=8080): Max retries exceeded with url: /dns/existing (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xffff6ab4e1d0>: Failed to establish a new connection: [Errno -2] Name or service not known'))
2023-09-27 14:26:54 2023-09-27 12:26:54 ERROR    An error occurred: HTTPConnectionPool(host='dnsd', port=8080): Max retries exceeded with url: /dns/ip (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xffff6ab4e560>: Failed to establish a new connection: [Errno -2] Name or service not known'))

Description

Python package to handle multi modal pipelines

Notes

Some Services to Integrate

• Deepgram
• Play.ht
• 11labs