The latest update is about 3 months ago.

I know this is a stretch... but is it possible to support Nougat inference?

https://huggingface.co/facebook/nougat-base

Summary

Currently we need to pass the prompt_params along with a given request object like we do in claude2_request.

...
messages = [
    {"role": "user", "content": "Write me a short poem"},
]
prompt_params = GenerationParams(messages=messages)
claude2_request = ModelGenerationRequest(
    model_name=ModelsEnum.CLAUDE_INSTANT_12,
    provider_name=ProvidersEnum.ANTHROPIC.value,
    order=1,
    prompt_params=prompt_params,
)
...

While we can possibly update the prompt_params later on by passing a new value by:

claude2_request.prompt_params = new_prompt_params

This makes it a dirty and implicit way of passing messages to the context.

Proposal

Update client.generate to accept prompt_params or messages separately ( assuming that prompt_params are model agnostic ).
Hence the following snippet would look like this:

prompt_params = GenerationParams(messages=messages)
claude2_request = ModelGenerationRequest(
    model_name=ModelsEnum.CLAUDE_INSTANT_12,
    provider_name=ProvidersEnum.ANTHROPIC.value,
    order=1,
)
...
response = client.generate(ordered_generation_requests=[claude2_request], messages=prompt_params)

The benefit of doing this would be:

• Messages are decoupled from Model Request Objects and hence reusing them would be easier, ie one can generate multiple request objects and create a configuration of their request objects and reuse them with the same messages.

claude2_request_prio = ModelGenerationRequest(...)
claude_request_prio = ModelGenerationRequest(...)
openapi_request_non_prio = ModelGenerationRequest(...)
# Generation Request lists that can be reused as per requirement
main_generation_request_list = [
    claude2_request_prio,
    claude_request_prio,
    openapi_request_non_prio
]
...

# Generation request list is passed to `client.generate()` when a message is received
response = client.generate(ordered_generation_requests=main_generation_request_list, messages=prompt_params)

Description

Context

I noticed a discrepancy between the contribution guidelines in the documentation and the actual branch structure in the repository. The guidelines reference a master branch for pull requests, but the repository uses main as its default branch.

Issue

This inconsistency could lead to confusion for new contributors who are following the contribution process outlined in the documentation.

Suggestion/Query

Option 1: Documentation Update
I suggest updating the documentation to reflect the current branch structure of the repository. Specifically, replacing instances of master with main in the contribution guidelines would align it with the actual branch used and clarify the process for new contributors.

Option 2: Branch Re-Naming
Is there an intention to update the repository's default branch to master to align with the current documentation? If not, would it be beneficial to update the documentation to reference the main branch instead?

Additional Notes

I came across this while preparing to contribute to the project. Ensuring consistency between the documentation and the actual repository setup would be helpful for new contributors.

Thank you for addressing this issue.

Summary

npm run build ends up returning multiple errors.

Terminal Response

 npm run build

> [email protected] build
> tsc -p tsconfig.json

src/index.ts:29:26 - error TS2769: No overload matches this call.
  Overload 1 of 3, '(obj: object, msg?: string | undefined): void', gave the following error.
    Argument of type 'unknown' is not assignable to parameter of type 'object'.
  Overload 2 of 3, '(msg: string): void', gave the following error.
    Argument of type 'unknown' is not assignable to parameter of type 'string'.
  Overload 3 of 3, '(arg: number | boolean): void', gave the following error.
    Argument of type 'unknown' is not assignable to parameter of type 'number | boolean'.

29         server.log.error(err);
                            ~~~


src/server.ts:66:26 - error TS2769: No overload matches this call.
  Overload 1 of 3, '(obj: object, msg?: string | undefined): void', gave the following error.
    Argument of type 'unknown' is not assignable to parameter of type 'object'.
  Overload 2 of 3, '(msg: string): void', gave the following error.
    Argument of type 'unknown' is not assignable to parameter of type 'string'.
  Overload 3 of 3, '(arg: number | boolean): void', gave the following error.
    Argument of type 'unknown' is not assignable to parameter of type 'number | boolean'.

66         server.log.error(err);
                            ~~~


src/server.ts:84:44 - error TS7006: Parameter 'request' implicitly has an 'any' type.

84     server.decorate("authenticate", async (request, reply) => {
                                              ~~~~~~~

src/server.ts:84:53 - error TS7006: Parameter 'reply' implicitly has an 'any' type.

84     server.decorate("authenticate", async (request, reply) => {
                                                       ~~~~~


Found 4 errors in 2 files.

Errors  Files
     1  src/index.ts:29
     3  src/server.ts:66

Hi,

I am planning to use GPTRouter but had a few questions before making my decision. Could you please help answer these questions?

Q1: Library vs Router
What is the need for a router as a service instead of a library? I was wondering why you did not go the path of creating a wrapper library from which to invoke the various AI model API.
Here are some reasons I can think of -
a) Manage all the access keys in one place.
b) The library size can get bigger as more and more AI models are integrated, while as a consumer I might be interested in only a one or
2 models.

Q2: Database
Could you also let me know if there is any documentation on what is managed in the Postgres database? Is there ability to switch into a NoSQL database?

Q3: Logging
If I self-host GPTRouter what do I integrate to get all the logging from the GPTRouter for debugging and observability?

Thank you!

Summary

Both Dockerfile and Dockerfile.ECS.Alpha refer to using npm start which in turn executes this script:

  "scripts": {
    "build": "tsc -p tsconfig.json",
    "start": "nodemon --watch 'src/**/*.ts' --exec 'ts-node' src/index.ts"
  }

Here, usage of both --watch and ts-node add extra overhead as there should be no requirement of ts-node and or watch.

As per this thread How to use ts-node in production, we should either use --transpile-only or even better have a prebuild step that precompiles all the typescript code into javascript. This will reduce the extra overhead ts-node and watch adds.

Summary

Currently there's no linting configuration setup for this repository. This causes the issue of code editors/ ides applying the default linting configuration of the contributor due to lack of a linting config. This causes the linting to be changed adding unnecessary changes with every Pull Request done by contributors.

Addition of a linter like ESLint and code formatter like Prettier to ensure better MR and reviews.

Summary

Link provided in README.md mentions a contribution sections and also a link to contribution guidelines CONTRIBUTING.md but there's no contributing.md at that location. it is instead here : contributing.md.