dbv111m / bricksllm

🔒 Enterprise-grade API gateway that helps you monitor and impose cost or rate limits per API key. Get fine-grained access control and monitoring per user, application, or environment. Supports OpenAI, Azure OpenAI, Anthropic, LangChain, and open-source LLMs.

BricksLLM: AI Gateway For Putting LLM In Production

Tip

A managed version of BricksLLM is also available! It is production ready, and comes with a dashboard to make interacting with BricksLLM easier. Try us out for free today!

BricksLLM is a cloud native AI gateway written in Go. Currently, it provide native support for OpenAI, Anthropic and Azure OpenAI. We let you create API keys that have rate limits, cost limits and TTLs. The API keys can be used in both development and production to achieve fine-grained access control that is not provided by any of the foundational model providers. The proxy is designed to be 100% compatible with existing SDKs.

Features

• Rate limit
• Cost control
• Cost analytics
• Request analytics
• Caching
• Request Retries
• Failover
• Model access control
• Endpoint access control
• Native support for all OpenAI endpoints
• Native support for Anthropic
• Native support for Azure OpenAI
• Integration with custom models
• Datadog integration
• Logging with privacy control

Roadmap

• Access control via API key with rate limit, cost limit and ttl
• Logging integration
• Statsd integration
• Custom Provider Integration
• PII detection and masking 🚧

Getting Started

The easiest way to get started with BricksLLM is through BricksLLM-Docker.

Step 1 - Clone BricksLLM-Docker repository

git clone https://github.com/bricks-cloud/BricksLLM-Docker

Step 2 - Change to BricksLLM-Docker directory

cd BricksLLM-Docker

Step 3 - Deploy BricksLLM locally with Postgresql and Redis

docker-compose up

You can run this in detach mode use the -d flag: docker-compose up -d

Step 4 - Create a provider setting

curl -X PUT http://localhost:8001/api/provider-settings \
   -H "Content-Type: application/json" \
   -d '{
          "provider":"openai",
          "setting": {
             "apikey": "YOUR_OPENAI_KEY"
          }
      }'   

Copy the id from the response.

Step 5 - Create a Bricks API key

Use id from the previous step as settingId to create a key with a rate limit of 2 req/min and a spend limit of 25 cents.

curl -X PUT http://localhost:8001/api/key-management/keys \
   -H "Content-Type: application/json" \
   -d '{
	      "name": "My Secret Key",
	      "key": "my-secret-key",
	      "tags": ["mykey"],
        "settingIds": ["ID_FROM_STEP_FOUR"],
        "rateLimitOverTime": 2,
        "rateLimitUnit": "m",
        "costLimitInUsd": 0.25
      }'   

Congratulations you are done!!!

Then, just redirect your requests to us and use OpenAI as you would normally. For example:

curl -X POST http://localhost:8002/api/providers/openai/v1/chat/completions \
   -H "Authorization: Bearer my-secret-key" \
   -H "Content-Type: application/json" \
   -d '{
          "model": "gpt-3.5-turbo",
          "messages": [
              {
                  "role": "system",
                  "content": "hi"
              }
          ]
      }'

Or if you're using an SDK, you could change its baseURL to point to us. For example:

// OpenAI Node SDK v4
import OpenAI from 'openai';

const openai = new OpenAI({
  apiKey: "some-secret-key", // key created earlier
  baseURL: "http://localhost:8002/api/providers/openai/v1", // redirect to us
});

How to Update?

For updating to the latest version

docker pull luyuanxin1995/bricksllm:latest

For updating to a particular version

docker pull luyuanxin1995/bricksllm:1.4.0

Documentation

Environment variables

Name	type	description	default
POSTGRESQL_HOSTS	required	Hosts for Postgresql DB. Separated by ,	localhost
POSTGRESQL_DB_NAME	optional	Name for Postgresql DB.
POSTGRESQL_USERNAME	required	Postgresql DB username
POSTGRESQL_PASSWORD	required	Postgresql DB password
POSTGRESQL_SSL_MODE	optional	Postgresql SSL mode	disable
POSTGRESQL_PORT	optional	The port that Postgresql DB runs on	5432
POSTGRESQL_READ_TIME_OUT	optional	Timeout for Postgresql read operations	2s
POSTGRESQL_WRITE_TIME_OUT	optional	Timeout for Postgresql write operations	1s
REDIS_HOSTS	required	Host for Redis. Separated by ,	localhost
REDIS_PASSWORD	optional	Redis Password
REDIS_PORT	optional	The port that Redis DB runs on	6379
REDIS_READ_TIME_OUT	optional	Timeout for Redis read operations	1s
REDIS_WRITE_TIME_OUT	optional	Timeout for Redis write operations	500ms
IN_MEMORY_DB_UPDATE_INTERVAL	optional	The interval BricksLLM API gateway polls Postgresql DB for latest key configurations	1s
STATS_PROVIDER	optional	This value can only be datadog. Required for integration with Datadog.
PROXY_TIMEOUT	optional	Timeout for proxy HTTP requests.
ADMIN_PASS	optional	Simple password authentication for admin endpoints.

Configuration Endpoints

The configuration server runs on Port 8001.

Headers

name	type	data type	description
X-API-KEY	optional	string	Key authentication header.

Get keys: GET /api/key-management/keys

Description

This endpoint is set up for retrieving key configurations using a query param called tag.

Query Parameters

name	type	data type	description
tag	optional	string	Identifier attached to a key configuration
tags	optional	[]string	Identifiers attached to a key configuration
provider	optional	string	Provider attached to a key provider configuration. Its value can only be openai.
keyIds	optional	[]string	Unique identifiers for keys.

Error Response

http code	content-type
400, 500	application/json

Field	type	example
status	int	400
title	string	request body reader error
type	string	/errors/request-body-read
detail	string	something is wrong
instance	string	/api/key-management/keys

Response

Response Body
[]KeyConfiguration

Fields of KeyConfiguration

Field	type	example	description
name	string	spike's developer key	Name of the API key.
createdAt	int64	1257894000	Key configuration creation time in unix.
updatedAt	int64	1257894000	Key configuration update time in unix.
revoked	boolean	true	Indicator for whether the key is revoked.
revokedReason	string	The key has expired	Reason for why the key is revoked.
tags	[]string	["org-tag-12345"]	Identifiers associated with the key.
keyId	string	550e8400-e29b-41d4-a716-446655440000	Unique identifier for the key.
costLimitInUsd	float64	5.5	Total spend limit of the API key.
costLimitInUsdOverTime	float64	2	Total spend within period of time. This field is required if costLimitInUsdUnit is specified.
costLimitInUsdUnit	enum	d	Time unit for costLimitInUsdOverTime. Possible values are [m, h, d, mo].
rateLimitOverTime	int	2	rate limit over period of time. This field is required if rateLimitUnit is specified.
rateLimitUnit	string	m	Time unit for rateLimitOverTime. Possible values are [h, m, s, d]
ttl	string	2d	time to live. Available units are [s, m, h]
allowedPaths	[]PathConfig	[{ "path": "/api/providers/openai/v1/chat/completion", "method": "POST"}]	Allowed paths that can be accessed using the key.
settingId	string	98daa3ae-961d-4253-bf6a-322a32fdca3d	This field is DEPERCATED. Use settingIds field instead.
settingIds	string	[98daa3ae-961d-4253-bf6a-322a32fdca3d]	Setting ids associated with the key.
shouldLogRequest	bool	false	Should request be stored.
shouldLogResponse	bool	true	Should response be stored.
rotationEnabled	bool	false	Should key rotate setting used to access third party endpoints in order to circumvent rate limits.

Create key: PUT /api/key-management/keys

Description

This endpoint is set up for retrieving key configurations using a query param called tag.

Request

PathConfig

Field	required	type	example	description
path	required	string	/api/providers/openai/v1/chat/completion	Allowed path
method	required	string	POST	HTTP Method

Field	required	type	example	description
name	required	string	spike's developer key	Name of the API key.
tags	optional	[]string	["org-tag-12345"]	Identifiers associated with the key.
key	required	string	abcdef12345	API key.
settingId	depercated	string	98daa3ae-961d-4253-bf6a-322a32fdca3d	This field is DEPERCATED. Use settingIds field instead.
settingIds	required	string	98daa3ae-961d-4253-bf6a-322a32fdca3d	Setting ids associated with the key.
costLimitInUsd	optional	float64	5.5	Total spend limit of the API key.
costLimitInUsdOverTime	optional	float64	2	Total spend within period of time. This field is required if costLimitInUsdUnit is specified.
costLimitInUsdUnit	optional	enum	d	Time unit for costLimitInUsdOverTime. Possible values are [m, h, d, mo].
rateLimitOverTime	optional	int	2	rate limit over period of time. This field is required if rateLimitUnit is specified.
rateLimitUnit	optional	enum	m	Time unit for rateLimitOverTime. Possible values are [h, m, s, d]
ttl	optional	string	2d	time to live. Available units are [s, m, h].
allowedPaths	optional	[]PathConfig	2d	Pathes allowed for access.
shouldLogRequest	optional	bool	false	Should request be stored.
shouldLogResponse	optional	bool	true	Should response be stored.
rotationEnabled	optional	bool	false	Should key rotate setting used to access third party endpoints in order to circumvent rate limits.

Error Response

http code	content-type
400, 500	application/json

Field	type	example
status	int	400
title	string	request body reader error
type	string	/errors/request-body-read
detail	string	something is wrong
instance	string	/api/key-management/keys

Responses

Field	type	example	description
name	string	spike's developer key	Name of the API key.
createdAt	int64	1257894000	Key configuration creation time in unix.
updatedAt	int64	1257894000	Key configuration update time in unix.
revoked	boolean	true	Indicator for whether the key is revoked.
revokedReason	string	The key has expired	Reason for why the key is revoked.
tags	[]string	["org-tag-12345"]	Identifiers associated with the key.
keyId	string	550e8400-e29b-41d4-a716-446655440000	Unique identifier for the key.
costLimitInUsd	float64	5.5	Total spend limit of the API key.
costLimitInUsdOverTime	float64	2	Total spend within period of time. This field is required if costLimitInUsdUnit is specified.
costLimitInUsdUnit	enum	d	Time unit for costLimitInUsdOverTime. Possible values are [m, h, d, mo].
rateLimitOverTime	int	2	rate limit over period of time. This field is required if rateLimitUnit is specified.
rateLimitOverTime	int	2	rate limit over period of time. This field is required if rateLimitUnit is specified.
rateLimitUnit	string	m	Time unit for rateLimitOverTime. Possible values are [h, m, s, d].
ttl	string	2d	time to live. Available units are [s, m, h]
allowedPaths	[]PathConfig	[{ "path": "/api/providers/openai/v1/chat/completion", method: "POST"}]	Allowed paths that can be accessed using the key.
settingId	string	98daa3ae-961d-4253-bf6a-322a32fdca3d	This field is DEPERCATED. Use settingIds field instead.
settingIds	string	[98daa3ae-961d-4253-bf6a-322a32fdca3d]	Setting ids associated with the key.
shouldLogRequest	bool	false	Should request be stored.
shouldLogResponse	bool	true	Should response be stored.
rotationEnabled	bool	false	Should key rotate setting used to access third party endpoints in order to circumvent rate limits.

Update key: PATCH /api/key-management/keys/{keyId}

Description

This endpoint is set up for updating key configurations using key id.

Parameters

name	type	data type	description
keyId	required	string	Unique key configuration identifier.

Request

PathConfig

Field	required	type	example	description
path	required	string	/api/providers/openai/v1/chat/completion	Allowed path
method	required	string	POST	HTTP Method

Field	required	type	example	description
settingId	optional	string	98daa3ae-961d-4253-bf6a-322a32fdca3d	This field is DEPERCATED. Use settingIds field instead.
settingIds	optional	string	98daa3ae-961d-4253-bf6a-322a32fdca3d	Setting ids associated with the key.
name	optional	string	spike's developer key	Name of the API key.
tags	optional	[]string	["org-tag-12345"]	Identifiers associated with the key.
revoked	optional	boolean	true	Indicator for whether the key is revoked.
revokedReason	optional	string	The key has expired	Reason for why the key is revoked.
costLimitInUsd	optional	float64	5.5	Total spend limit of the API key.
costLimitInUsdOverTime	optional	float64	2	Total spend within period of time. This field is required if costLimitInUsdUnit is specified.
costLimitInUsdUnit	optional	enum	d	Time unit for costLimitInUsdOverTime. Possible values are [m, h, d, mo].
rateLimitOverTime	optional	int	2	rate limit over period of time. This field is required if rateLimitUnit is specified.
rateLimitUnit	optional	string	m	Time unit for rateLimitOverTime. Possible values are [h, m, s, d]
allowedPaths	optional	[{ "path": "/api/providers/openai/v1/chat/completions", "method": "POST"}]	``	Pathes allowed for access.
shouldLogRequest	optional	bool	false	Should request be stored.
shouldLogResponse	optional	bool	true	Should response be stored.
rotationEnabled	optional	bool	false	Should key rotate setting used to access third party endpoints in order to circumvent rate limits.

Error Response

http code	content-type
400, 500	application/json

Field	type	example
status	int	400
title	string	request body reader error
type	string	/errors/request-body-read
detail	string	something is wrong
instance	string	/api/key-management/keys

Response

Field	type	example	description
name	string	spike's developer key	Name of the API key.
createdAt	int64	1257894000	Key configuration creation time in unix.
updatedAt	int64	1257894000	Key configuration update time in unix.
revoked	boolean	true	Indicator for whether the key is revoked.
revokedReason	string	The key has expired	Reason for why the key is revoked.
tags	[]string	["org-tag-12345"]	Identifiers associated with the key.
keyId	string	550e8400-e29b-41d4-a716-446655440000	Unique identifier for the key.
costLimitInUsd	float64	5.5	Total spend limit of the API key.
costLimitInUsdOverTime	float64	2	Total spend within period of time. This field is required if costLimitInUsdUnit is specified.
costLimitInUsdUnit	enum	d	Time unit for costLimitInUsdOverTime. Possible values are [m, h, d, mo].
rateLimitOverTime	int	2	rate limit over period of time. This field is required if rateLimitUnit is specified.
rateLimitUnit	string	m	Time unit for rateLimitOverTime. Possible values are [h, m, s, d]
ttl	string	2d	time to live. Available units are [s, m, h]
allowedPaths	[]PathConfig	[{ "path": "/api/providers/openai/v1/chat/completion", method: "POST"}]	Allowed paths that can be accessed using the key.
settingId	string	98daa3ae-961d-4253-bf6a-322a32fdca3d	This field is DEPERCATED. Use settingIds field instead.
settingIds	string	[98daa3ae-961d-4253-bf6a-322a32fdca3d]	Setting ids associated with the key.
shouldLogRequest	bool	false	Should request be stored.
shouldLogResponse	bool	true	Should response be stored.
rotationEnabled	bool	false	Should key rotate setting used to access third party endpoints in order to circumvent rate limits.

Create a provider setting: POST /api/provider-settings

Description

This endpoint is creating a provider setting.

Request

Field	required	type	example	description
provider	required	enum	openai	This value can only be openai, anthropic and azure as for now.
setting	required	Setting	{ "apikey": "YOUR_OPENAI_KEY" }	A map of values used for authenticating with the selected provider.
name	optional	string	YOUR_PROVIDER_SETTING_NAME	This field is used for giving a name to provider setting
allowedModels	[]string	["text-embedding-ada-002"]	Allowed models for this provider setting.

Setting

Field	required	type	example	description
apiKey	required	string	xx-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx	This value is required.
resourceName	required	string	YOUR_AZURE_RESOURCE_NAME	This value is required when the provider is azure.

Error Response

http code	content-type
400, 500	application/json

Field	type	example
status	int	400
title	string	request body reader error
type	string	/errors/request-body-read
detail	string	something is wrong
instance	string	/api/provider-settings

Response

Field	type	example	description
createdAt	int64	1699933571	Unix timestamp for creation time.
updatedAt	int64	1699933571	Unix timestamp for update time.
provider	enum	openai	This value can only be openai as for now.
id	string	98daa3ae-961d-4253-bf6a-322a32fdca3d	This value is a unique identifier.
name	string	YOUR_PROVIDER_SETTING_NAME	Provider setting name.
allowedModels	[]string	["text-embedding-ada-002"]	Allowed models for this provider setting.

Get provider settings: GET /api/provider-settings

Description

This endpoint is getting provider settings.

Query Parameters

name	type	data type	description
ids	optional	[]string	Provider setting ids

Error Response

http code	content-type
500	application/json

Field	type	example
status	int	400
title	string	request body reader error
type	string	/errors/request-body-read
detail	string	something is wrong
instance	string	/api/provider-settings

Response

[]ProviderSetting

ProviderSetting

Field	type	example	description
createdAt	int64	1699933571	Unix timestamp for creation time.
updatedAt	int64	1699933571	Unix timestamp for update time.
provider	enum	openai	This value can only be openai as for now.
id	string	98daa3ae-961d-4253-bf6a-322a32fdca3d	This value is a unique identifier.
name	string	YOUR_PROVIDER_SETTING_NAME	Provider setting name.
allowedModels	[]string	["text-embedding-ada-002"]	Allowed models for this provider setting.

Update a provider setting: PATCH /api/provider-settings/:id

Description

This endpoint is updating a provider setting .

Parameters

name	type	data type	description
id	required	string	Unique identifier for the provider setting that you want to update.

Request

Field	required	type	example	description
setting	required	Setting	{ "apikey": "YOUR_OPENAI_KEY" }	A map of values used for authenticating with the selected provider.
name	optional	string	YOUR_PROVIDER_SETTING_NAME	This field is used for giving a name to provider setting
allowedModels	[]string	["text-embedding-ada-002"]	Allowed models for this provider setting.

Setting

Field	required	type	example	description
apiKey	required	string	xx-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx	This value is required.
resourceName	required	string	YOUR_AZURE_RESOURCE_NAME	This value is required when the provider is azure.

Error Response

http code	content-type
400, 500	application/json

Field	type	example
status	int	400
title	string	request body reader error
type	string	/errors/request-body-read
detail	string	something is wrong
instance	string	/api/provider-settings

Response

Field	type	example	description
createdAt	int64	1699933571	Unix timestamp for creation time.
updatedAt	int64	1699933571	Unix timestamp for update time.
provider	enum	openai	This value can only be openai as for now.
id	string	98daa3ae-961d-4253-bf6a-322a32fdca3d	This value is a unique identifier
name	string	YOUR_PROVIDER_SETTING_NAME	Provider setting name.
allowedModels	[]string	["text-embedding-ada-002"]	Allowed models for this provider setting.

Retrieve Metrics: POST /api/reporting/events

Description

This endpoint is retrieving aggregated metrics given an array of key ids and tags.

Request

Field	required	type	example	description
keyIds	required	[]string	["key-1", "key-2", "key-3" ]	Array of ids that specicify the keys that you want to aggregate stats from.
tags	required	[]string	["tag-1", "tag-2"]	Array of tags that specicify the key tags that you want to aggregate stats from.
customIds	required	[]string	["customId-1", "customId-2"]	A list of custom IDs that you want to aggregate stats from.
filters	required	[]string	["model", "keyId"]	Group by data points through different filters(model,keyId or customId).
start	required	int64	1699933571	Start timestamp for the requested timeseries data.
end	required	int64	1699933571	End timestamp for the requested timeseries data.
increment	required	int	60	This field is the increment in seconds for the requested timeseries data.

Error Response

http code	content-type
500	application/json

Field	type	example
status	int	400
title	string	request body reader error
type	string	/errors/request-body-read
detail	string	something is wrong
instance	string	/api/provider-settings

Response

Field	type	example	description
dataPoints	[]dataPoint	[{ "timeStamp": 1699933571, "numberOfRequests": 1, "costInUsd": 0.8, "latencyInMs": 600, "promptTokenCount": 0, "completionTokenCount": 0, "successCount": 1 }]	Unix timestamp for creation time.
latencyInMsMedian	float64	656.7	Median latency for the given time period.
latencyInMs99th	float64	555.7	99th percentile latency for the given time period.

Datapoint

Field	type	example	description
timeStamp	int64	1702504746	Unix timestamp for the data point
numberOfRequests	int64	100	Aggregated number of http requests over the given time increment.
costInUsd	float64	1.7	Aggregated cost of proxied requests in USD over the given time increment.
latencyInMs	int	555	Aggregated latency in milliseconds of http requests over the given time increment.
promptTokenCount	int	25	Aggregated prompt token counts over the given time increment.
completionTokenCount	int	4000	Aggregated completion token counts over the given time increment.
successCount	int	555	Aggregated number of successful http requests over the given time increment.
keyId	int	555.7	key Id associated with the event.
model	string	gpt-3.5-turbo	model associated with the event.
customId	string	customId	customId associated with the event.

Get events: GET /api/events

Description

This endpoint is for getting events.

Query Parameters

name	type	data type	description
customId	optional	string	Custom identifier attached to an event.
keyIds	optional	[]string	A list of key IDs.
start	required if keyIds is specified	int64	Start timestamp.
end	required if keyIds is specified	int64	End timestamp.

Error Response

http code	content-type
500	application/json

Field	type	example
status	int	400
title	string	request body reader error
type	string	/errors/request-body-read
detail	string	something is wrong
instance	string	/api/provider-settings

Response

[]Event

Event

Field	type	example	description
id	int64	1699933571	Unique identifier associated with the event.
created_at	int64	1699933571	Unix timestamp for creation time.
tags	int64	["YOUR_TAG"]	Tags of the key.
key_id	string	YOUR_KEY_ID	Key Id associated with the proxy request.
cost_in_usd	float64	0.0004	Cost incured by the proxy request.
model	string	gpt-4-1105-preview	Model used in the proxy request.
provider	string	openai	Provider for the proxy request.
status	int	200	Http status.
prompt_token_count	int	8	Prompt token count of the proxy request.
completion_token_count	int	16	Completion token counts of the proxy request.
latency_in_ms	int	160	Provider setting name.
path	string	/api/v1/chat/completion	Provider setting name.
method	string	POST	Http method for the assoicated proxu request.
custom_id	string	YOUR_CUSTOM_ID	Custom Id passed by the user in the headers of proxy requests.
request	[]byte	{}	Request in bytes to providers.
custom_id	string	YOUR_CUSTOM_ID	Custom Id passed by the user in the headers of proxy requests.
response	[]byte	{}	Response in bytes from providers.

Create custom provider: POST /api/custom/providers

Description

This endpoint is creating custom providers.

RouteConfig

Field	required	type	example	description
path	required	string	/chat/completion	Path associated with the custom provider route. It must be unique within the custom provider.
target_url	required	string	https://api.openai.com/v1/chat/completions	Proxy destination URL for the custom provider route.
model_location	required	string	model	JSON field for the model in the HTTP request.
request_prompt_location	required	string	messages.#.content	JSON field for the prompt request in the HTTP request.
response_completion_location	required	string	choices.#.message.content	JSON field for the completion content in the HTTP response.
stream_location	required	string	stream	JSON field for the stream boolean in the HTTP request.
stream_end_word	required	string	[DONE]	End word for the stream.
stream_response_completion_location	required	string	choices.#.delta.content	JSON field for the completion content in the streaming response.
stream_max_empty_messages	required	int	10	Number of max empty messages in stream.

Request

Field	required	type	example	description
provider	required	string	bricks	Unique identifier associated with the route config.
route_configs	required	[]RouteConfig	{{ "path": "/chat/completions", "target_url": "https://api.openai.com/v1/chat/completions" }}	Route configurations for the custom provider.
authentication_param	optional	string	apikey	The authentication parameter required for.

Error Response

http code	content-type
500, 400	application/json

Field	type	example
status	int	400
title	string	request body reader error
type	string	/errors/request-body-read
detail	string	something is wrong
instance	string	/api/custom/providers

Response

Field	type	example	description
id	int64	1699933571	Unique identifier associated with the event.
created_at	int64	1699933571	Unix timestamp for creation time.
updated_at	int64	1699933571	Unix timestamp for update time.
provider	string	bricks	Unique identifier associated with the route config.
route_configs	[]RouteConfig	{{ "path": "/chat/completions", "target_url": "https://api.openai.com/v1/chat/completions" }}	Start timestamp for the requested timeseries data.
authentication_param	string	apikey	The authentication parameter required for.

Update custom provider: PATCH /api/custom/providers/:id

Description

This endpoint is updating a custom provider.

RouteConfig

Field	required	type	example	description
path	required	string	/chat/completion	Path associated with the custom provider route. It must be unique within the custom provider.
target_url	required	string	https://api.openai.com/v1/chat/completions	Proxy destination URL for the custom provider route.
model_location	required	string	model	JSON field for the model in the HTTP request.
request_prompt_location	required	string	messages.#.content	JSON field for the prompt request in the HTTP request.
response_completion_location	required	string	choices.#.message.content	JSON field for the completion content in the HTTP response.
stream_location	required	string	stream	JSON field for the stream boolean in the HTTP request.
stream_end_word	required	string	[DONE]	End word for the stream.
stream_response_completion_location	required	string	choices.#.delta.content	JSON field for the completion content in the streaming response.
stream_max_empty_messages	required	int	10	Number of max empty messages in stream.

Request

Field	required	type	example	description
route_configs	optional	[]RouteConfig	{{ "path": "/chat/completions", "target_url": "https://api.openai.com/v1/chat/completions" }}	Route configurations for the custom provider.
authentication_param	optional	string	apikey	The authentication parameter required for.

Error Response

http code	content-type
500, 404, 400	application/json

Field	type	example
status	int	400
title	string	request body reader error
type	string	/errors/request-body-read
detail	string	something is wrong
instance	string	/api/custom/providers

Response

Field	type	example	description
id	int64	1699933571	Unique identifier associated with the event.
created_at	int64	1699933571	Unix timestamp for creation time.
updated_at	int64	1699933571	Unix timestamp for update time.
provider	string	bricks	Unique identifier associated with the route config.
route_configs	[]RouteConfig	{{ "path": "/chat/completions", "target_url": "https://api.openai.com/v1/chat/completions" }}	Start timestamp for the requested timeseries data.
authentication_param	string	apikey	The authentication parameter required for.

Get custom providers: GET /api/custom/providers

Description

This endpoint is for getting custom providers.

RouteConfig

Field	required	type	example	description
path	required	string	/chat/completion	Path associated with the custom provider route. It must be unique within the custom provider.
target_url	required	string	https://api.openai.com/v1/chat/completions	Proxy destination URL for the custom provider route.
model_location	required	string	model	JSON field for the model in the HTTP request.
request_prompt_location	required	string	messages.#.content	JSON field for the prompt request in the HTTP request.
response_completion_location	required	string	choices.#.message.content	JSON field for the completion content in the HTTP response.
stream_location	required	string	stream	JSON field for the stream boolean in the HTTP request.
stream_end_word	required	string	[DONE]	End word for the stream.
stream_response_completion_location	required	string	choices.#.delta.content	JSON field for the completion content in the streaming response.
stream_max_empty_messages	required	int	10	Number of max empty messages in stream.

Request

Field	required	type	example	description
route_configs	optional	[]RouteConfig	{{ "path": "/chat/completions", "target_url": "https://api.openai.com/v1/chat/completions" }}	Route configurations for the custom provider.
authentication_param	optional	string	apikey	The authentication parameter required for.

Error Response

http code	content-type
500	application/json

Field	type	example
status	int	400
title	string	request body reader error
type	string	/errors/request-body-read
detail	string	something is wrong
instance	string	/api/custom/providers

Response

[]Provider

Provider

Field	type	example	description
id	int64	9e6e8b27-2ce0-4ef0-bdd7-1ed3916592eb	Unique identifier associated with the event.
created_at	int64	1699933571	Unix timestamp for creation time.
updated_at	int64	1699933571	Unix timestamp for update time.
provider	string	bricks	Unique identifier associated with the route config.
route_configs	[]RouteConfig	{{ "path": "/chat/completions", "target_url": "https://api.openai.com/v1/chat/completions" }}	Start timestamp for the requested timeseries data.
authentication_param	string	apikey	The authentication parameter required for.

Create routes: POST /api/routes

Description

This endpoint is for creating routes.

StepConfig

Field	required	type	example	description
provider	required	enum	azure	Provider for the step. Can only be either azure or openai.
model	required	string	gpt-3.5-turbo	Model that the step should call. Can only be chat completion or embedding models from OpenAI or Azure OpenAI.
retries	optional	int	2	Number of retries.
params	optional	object	{ deploymentId: "ada-test",apiVersion: "2022-12-01" }	Params required for maing API requests to desired modela and provider combo. Required if the provider is azure
timeout	optional	string	5s	Timeout desired for each request. Default value is 5m.

CacheConfig

Field	required	type	example	description
enabled	required	bool	false	Boolean flag indicating whether caching is enabled.
ttl	optional	string	5s	TTL for the cache. Default value is 168h.

Request

Field	required	type	example	description
name	required	string	staging-openai-azure-completion-route	Name for the route.
path	required	string	/	Unique identifier for.
steps	required	[]StepConfig	apikey	The authentication parameter required for.
keyIds	required	[]string	[]	The authentication parameter required for.
cacheConfig	required	CacheConfig	[]	The authentication parameter required for.

Error Response

http code	content-type
500, 400	application/json

Field	type	example
status	int	400
title	string	request body reader error
type	string	/errors/request-body-read
detail	string	something is wrong
instance	string	/api/routes

Response

Field	type	example	description
id	required	string	9e6e8b27-2ce0-4ef0-bdd7-1ed3916592eb
createdAt	required	string	1699933571
updatedAt	required	string	1699933571
name	required	string	staging-openai-azure-completion-route
path	required	string	/production/chat/completion
steps	required	[]StepConfig	[{"retries": 2, "provider": "openai", "params": {}, "model": "gpt-3.5-turbo", "timeout": "1s"}]
keyIds	required	[]string	["9e6e8b27-2ce0-4ef0-bdd7-1ed3916592eb"]
cacheConfig	required	CacheConfig	{ "enabled": false, "ttl": "5s" }

Retrieve a route: GET /api/routes/:id

Description

This endpoint is for retrieving a route.

Parameters

name	type	data type	description
id	required	string	Unique identifier for the route.

Error Response

http code	content-type
500, 404	application/json

Field	type	example
status	int	404
title	string	request body reader error
type	string	/errors/request-body-read
detail	string	something is wrong
instance	string	/api/routes/:id

Response

Field	type	example	description
id	required	string	9e6e8b27-2ce0-4ef0-bdd7-1ed3916592eb
createdAt	required	string	1699933571
updatedAt	required	string	1699933571
name	required	string	staging-openai-azure-completion-route
path	required	string	/production/chat/completion
steps	required	[]StepConfig	[{"retries": 2, "provider": "openai", "params": {}, "model": "gpt-3.5-turbo", "timeout": "1s"}]
keyIds	required	[]string	["9e6e8b27-2ce0-4ef0-bdd7-1ed3916592eb"]
cacheConfig	required	CacheConfig	{ "enabled": false, "ttl": "5s" }

Retrieve routes: GET /api/routes

Description

This endpoint is for retrieving routes.

Error Response

http code	content-type
500	application/json

Field	type	example
status	int	404
title	string	request body reader error
type	string	/errors/request-body-read
detail	string	something is wrong
instance	string	/api/routes/:id

CacheConfig

Field	required	type	example	description
enabled	required	bool	false	Boolean flag indicating whether caching is enabled.
ttl	optional	string	5s	TTL for the cache. Default value is 168h.

StepConfig

Field	required	type	example	description
provider	required	enum	azure	Provider for the step. Can only be either azure or openai.
model	required	string	gpt-3.5-turbo	Model that the step should call. Can only be chat completion or embedding models from OpenAI or Azure OpenAI.
retries	optional	int	2	Number of retries.
params	optional	object	{ deploymentId: "ada-test",apiVersion: "2022-12-01" }	Params required for maing API requests to desired modela and provider combo. Required if the provider is azure
timeout	optional	string	5s	Timeout desired for each request. Default value is 5m.

RouteConfig

Field	type	example	description
id	required	string	9e6e8b27-2ce0-4ef0-bdd7-1ed3916592eb
createdAt	required	string	1699933571
updatedAt	required	string	1699933571
name	required	string	staging-openai-azure-completion-route
path	required	string	/production/chat/completion
steps	required	[]StepConfig	[{"retries": 2, "provider": "openai", "params": {}, "model": "gpt-3.5-turbo", "timeout": "1s"}]
keyIds	required	[]string	["9e6e8b27-2ce0-4ef0-bdd7-1ed3916592eb"]
cacheConfig	required	CacheConfig	{ "enabled": false, "ttl": "5s" }

Response

[]RouteConfig

OpenAI Proxy

The OpenAI proxy runs on Port 8002.

Headers

name	type	data type	description
x-custom-event-id	optional	string	Custom Id that can be used to retrieve an event associated with each proxy request.

Chat Completion

Call OpenAI chat completions: POST /api/providers/openai/v1/chat/completions

Description

This endpoint is set up for proxying OpenAI chat completion requests. Documentation for this endpoint can be found here.

Embeddings

Call OpenAI embeddings: POST /api/providers/openai/v1/embeddings

Description

This endpoint is set up for proxying OpenAI embedding requests. Documentation for this endpoint can be found here.

Moderations

Call OpenAI moderations: POST /api/providers/openai/v1/moderations

Description

This endpoint is set up for proxying OpenAI moderation requests. Documentation for this endpoint can be found here.

Models

Get OpenAI models: GET /api/providers/openai/v1/models

Description

This endpoint is set up for retrieving OpenAI models. Documentation for this endpoint can be found here.

Retrieve an OpenAI model: GET /api/providers/openai/v1/models/:model

Description

This endpoint is set up for retrieving an OpenAI model. Documentation for this endpoint can be found here.

Files

List files: GET /api/providers/openai/v1/files

Description

This endpoint is set up for list OpenAI files. Documentation for this endpoint can be found here.

Upload a file: POST /api/providers/openai/v1/files

Description

This endpoint is set up for creating an OpenAI file. Documentation for this endpoint can be found here.

Delete a file: POST /api/providers/openai/v1/files/:file_id

Description

This endpoint is set up for creating an OpenAI file. Documentation for this endpoint can be found here.

Retrieve a file: GET /api/providers/openai/v1/files/:file_id

Description

This endpoint is set up for retrieving an OpenAI file. Documentation for this endpoint can be found here.

Retrieve file content: GET /api/providers/openai/v1/files/:file_id/content

Description

This endpoint is set up for retrieving an OpenAI file content. Documentation for this endpoint can be found here.

Images

Generate images: POST /api/providers/openai/v1/images/generations

Description

This endpoint is set up for generating OpenAI images. Documentation for this endpoint can be found here.

Edit images: POST /api/providers/openai/v1/images/edits

Description

This endpoint is set up for editting OpenAI generated images. Documentation for this endpoint can be found here.

Create image variations: POST /api/providers/openai/v1/images/variations

Description

This endpoint is set up for creating OpenAI image variations. Documentation for this endpoint can be found here.

Voices

Create speech: POST /api/providers/openai/v1/audio/speech

Description

This endpoint is set up for creating speeches. Documentation for this endpoint can be found here.

Create transcriptions: POST /api/providers/openai/v1/audio/transcriptions

Description

This endpoint is set up for editting generated images. Documentation for this endpoint can be found here.

Create translations: POST /api/providers/openai/v1/audios/translations

Description

This endpoint is set up for creating translations. Documentation for this endpoint can be found here.

Assistants

Create assistant: POST /api/providers/openai/v1/assistants

Description

This endpoint is set up for creating an OpenAI assistant. Documentation for this endpoint can be found here.

Retrieve assistant: GET /api/providers/openai/v1/assistants/:assistant_id

Description

This endpoint is set up for retrieving an OpenAI assistant. Documentation for this endpoint can be found here.

Modify assistant: POST /api/providers/openai/v1/assistants/:assistant_id

Description

This endpoint is set up for modifying an OpenAI assistant. Documentation for this endpoint can be found here.

Delete assistant: DELETE /api/providers/openai/v1/assistants/:assistant_id

Description

This endpoint is set up for deleting an OpenAI assistant. Documentation for this endpoint can be found here.

List assistants: GET /api/providers/openai/v1/assistants

Description

This endpoint is set up for listing OpenAI assistants. Documentation for this endpoint can be found here.

Create assistant file: POST /api/providers/openai/v1/assistants/:assistant_id/files

Description

This endpoint is set up for creating an OpenAI assistant file. Documentation for this endpoint can be found here.

Retrieve assistant file: GET /api/providers/openai/v1/assistants/:assistant_id/files/:file_id

Description

This endpoint is set up for retrieving an OpenAI assistant file. Documentation for this endpoint can be found here.

Delete assistant file: DELETE /api/providers/openai/v1/assistants/:assistant_id/files/:file_id

Description

This endpoint is set up for deleting an OpenAI assistant file. Documentation for this endpoint can be found here.

List assistant files: GET /api/providers/openai/v1/assistants/:assistant_id/files

Description

This endpoint is set up for retrieving OpenAI assistant files. Documentation for this endpoint can be found here.

Create thread: POST /api/providers/openai/v1/threads

Description

This endpoint is set up for creating an OpenAI thread. Documentation for this endpoint can be found here.

Retrieve thread: GET /api/providers/openai/v1/threads/:thread_id

Description

This endpoint is set up for retrieving an OpenAI thread. Documentation for this endpoint can be found here.

Modify thread: POST /api/providers/openai/v1/threads/:thread_id

Description

This endpoint is set up for modifying an OpenAI thread. Documentation for this endpoint can be found here.

Delete thread: DELETE /api/providers/openai/v1/threads/:thread_id

Description

This endpoint is set up for deleting an OpenAI thread. Documentation for this endpoint can be found here.

Create message: POST /api/providers/openai/v1/threads/:thread_id/messages

Description

This endpoint is set up for creating an OpenAI message. Documentation for this endpoint can be found here.

Retrieve message: GET /api/providers/openai/v1/threads/:thread_id/messages/:message_id

Description

This endpoint is set up for retrieving an OpenAI message. Documentation for this endpoint can be found here.

Modify message: POST /api/providers/openai/v1/files/:file_id/content

Description

This endpoint is set up for modifying an OpenAI message. Documentation for this endpoint can be found here.

List messages: GET /api/providers/openai/v1/threads/:thread_id/messages

Description

This endpoint is set up for listing OpenAI messages. Documentation for this endpoint can be found here.

Retrieve message file: GET /api/providers/openai/v1/threads/:thread_id/messages/:message_id/files/:file_id

Description

This endpoint is set up for retrieving an OpenAI message file. Documentation for this endpoint can be found here.

List message files: GET /api/providers/openai/v1/threads/:thread_id/messages/:message_id/files

Description

This endpoint is set up for retrieving OpenAI message files. Documentation for this endpoint can be found here.

Create run: POST /api/providers/openai/v1/threads/:thread_id/runs

Description

This endpoint is set up for creating an OpenAI run. Documentation for this endpoint can be found here.

Retrieve run: GET /api/providers/openai/v1/threads/:thread_id/runs/:run_id

Description

This endpoint is set up for retrieving an OpenAI run. Documentation for this endpoint can be found here.

Modify run: POST /api/providers/openai/v1/threads/:thread_id/runs/:run_id

Description

This endpoint is set up for modifying an OpenAI run. Documentation for this endpoint can be found here.

List runs: GET /api/providers/openai/v1/threads/runs

Description

This endpoint is set up for retrieving OpenAI runs. Documentation for this endpoint can be found here.

Submit tool outputs to run: POST /api/providers/openai/v1/threads/runs

Description

This endpoint is set up for submitting tool outputs to an OpenAI run. Documentation for this endpoint can be found here.

Cancel a run: POST /api/providers/openai/v1/threads/:thread_id/runs/:run_id/cancel

Description

This endpoint is set up for cancellling an OpenAI run. Documentation for this endpoint can be found here.

Create thread and run: POST /api/providers/openai/v1/threads/runs

Description

This endpoint is set up for creating an OpenAI thread and run. Documentation for this endpoint can be found here.

Retrieve run step: GET /api/providers/openai/v1/threads/:thread_id/runs/:run_id/steps/:step_id

Description

This endpoint is set up for retrieving an OpenAI run step. Documentation for this endpoint can be found here.

List run steps: GET /api/providers/openai/v1/threads/:thread_id/runs/:run_id/steps

Description

This endpoint is set up for listing OpenAI run steps. Documentation for this endpoint can be found here.

Azure OpenAI Proxy

The custom provider proxy runs on Port 8002.

Create Azure OpenAI chat completion: POST /api/providers/azure/openai/deployments/:deployment_id/chat/completions?api-version={API_VERSION}

Description

This endpoint is set up for proxying Anthropic completion requests. Documentation for this endpoint can be found here.

Create Azure OpenAI embeddings: POST /api/providers/azure/openai/deployments/:deployment_id/embeddings?api-version={API_VERSION}

Description

This endpoint is set up for proxying Azure OpenAI completion requests. Documentation for this endpoint can be found here.

Anthropic Proxy

The custom provider proxy runs on Port 8002.

Create Anthropic completion: POST /api/providers/anthropic/v1/complete

Description

This endpoint is set up for proxying Anthropic completion requests. Documentation for this endpoint can be found here.

Custom Provider Proxy

The custom provider proxy runs on Port 8002.

Call custom providers: POST /api/custom/providers/:provider/*

Description

First you need to use create custom providers endpoint to create custom providers. Then create corresponding provider setting for the newly created custom provider. Afterward, you can start creating keys associated with the custom provider, and use the keys to access this endpoint by placing the created key in Authorization: Bearer YOUR_BRICKSLLM_KEY as part of your HTTP request headers.

Route Proxy

The custom provider proxy runs on Port 8002.

Call a route: POST /api/route/*

Description

Route helps you interpolate different models (embeddings or chat completion models) and providers (OpenAI or Azure OpenAI) to gurantee API responses.

First you need to use create route endpoint to create routes. If the route uses both Azure and OpenAI, you need to create API keys with corresponding provider settings as well. If the route is for chat completion, just call the route using the OpenAI chat completion format. On the other hand, if the route is for embeddings, just call the route using the embeddings format.