premai-io / prem-gateway Goto Github PK

Last step, Create GitHub Release, in release GHA fails.

Project Overview

The goal is to develop a platform that integrates with an existing system to safeguard running services by enforcing users to provide a valid API key. The UI/UX should draw inspiration from the ChatGPT API Key Management platform, focusing on credit-based payments without a subscription option.

Target Users

• Admin: Will use the current prem-app as an admin dashboard. In this role, they should be able to create API keys (e.g., without constraints), view usage, etc.
• Users/Developers: Will use a new frontend app to create API keys, top up credit balance, and view usage of API keys.

Core Features

Identity Management

1. Users should be able to register/login to a web app separate from the current prem-app dashboard.

API Key Management

1. Users should be able to generate one or more API keys.
2. API keys should have several levels of access constraints based on requests per minute (rate limit), tokens per minute for each service (service path), and a credit limit, which is a cumulative number of tokens determined by the user balance (available credit amount).

Billing

1. Users should be able to top up their credit balance using various payment methods: credit card, BTC, etc.
2. Users should be able to view usage and remaining credit balance.

Usage/Analytics

1. Users should be able to view the usage of their API keys to understand consumption and remaining access.
2. Administrators should have access to analytics to monitor service usage, identify trends, and optimize offerings.

Integration

• The platform should integrate with the existing system, which consists of services running behind a Traefik proxy.
• The existing Admin frontend application (prem-app) should be enhanced to enable the admin to create API keys and view Billing/Usage analytics.
• Integration should be achieved through Traefik forward auth middleware to ensure authentication and authorization.

Main Flow

1. Admin API Key constraints creation: Admin creates API key constraints per service and configures the price of 1K tokens. Users will access prem-service based on API Key constraints which include:
   • Service Constraints: For each service, there will be a rate limit (number of requests in a minute) and usage limit (number of tokens in a minute).
   • Balance Constraints: Each balance can be converted to an accumulative number of tokens (token credit), based on the price of 1K tokens. For each user, the usage (number of tokens used) will be tracked, and access will be denied if the user of the API key runs out of token balance.
2. User Registration and API Key Creation: User registers/logs in to the Identity Management platform, tops up credit balance with a payment method, and creates an API key. There is a consideration whether users should have the option to create a key for one or more services or if a key should be for all services.
3. Request Verification: User makes a request to prem-service using the API key. The platform checks if the API key exists, whether the related user has enough balance, and whether the rate and usage limits for the desired service path are adhered to.

@tiero @filopedraz

In the installer script, we may ideally want to ask the user if he wants to enable or not basic auth.

I am using Prem Gateway with daemon and app together locally, I notice that when trying to proxy the service it does not use the right published docker port, as I get Gateway Timeout when visiting http://localhost/all-minilm-l6-v2/docs for example

How to replicate

With the latest main branch of prem-app do the following

• Pull latest images

docker-compose -f docker-compose.gateway.yml pull

• Run the services, building premapp

docker-compose -f docker-compose.gateway.yml up -d --build

Then visit http://localhost with the browser and starts All MiniLM L6 v2 service via UI

Logs

• Traefik

192.168.65.1 - - [07/Nov/2023:17:46:29 +0000] "GET /all-minilm-l6-v2/docs HTTP/1.1" 504 15 "-" "-" 126 "all-minilm-l6-v2@docker" "http://172.17.0.3:8000" 30011ms

• Service with published port on 8444

INFO:     Started server process [8]
INFO:     Waiting for application startup.
2023-11-07 17:45:42 INFO     Load pretrained SentenceTransformer: all-MiniLM-L6-v2
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)

it seems the password cannot parse a forward slash. We should escape it?

time="2023-11-03T01:17:17Z" level=fatal msg="failed to create pgdb service: cannot parse `postgresql://root:xxxxxx@dnsd-db-pg:5432/dnsd-db?sslmode=disable`: failed to parse as URL (parse \"postgresql://root:DYyM4/bbNBY=@dnsd-db-pg:5432/dnsd-db?sslmode=disable\": invalid port \":DYyM4\" after host)"

Password is POSTGRES_PASSWORD=DYyM4/bbNBY=

Current app payload {“domain”:“prem.tiero.ai”}

Add a simple service that takes care to poll the versions.json as the source of truth for the latest version of the services to be "stable".

The service should

• expose an endpoint to give the currently running version and the new ones with a newer version for each service.
• expose an endpoint to trigger a docker pull (and also restart?) on the latest images.

Optionally we can pass an AUTO_UPDATE=all|patch|minor|major env var to automatically update to the latest version based on SemVer (it can be exported by the installer script of prem-box after asking user)

In an on-premise setup is common to use services such as ngrok or Cloudflare tunnel to hide the origin-destination or not have to mess with the local firewall.

Using this currently prevents to have a correct mapping the manual DNS record, as we don't really own an A Record on our IP, but we simply run a cloudflare daemon that proxy requests to the machine .

Demo.PremAI.io