premai-io / dev-portal Goto Github PK

Problems

most (all?) existing comparisons are purely quantitative (perplexity scores)

Objectives

• qualitative comparison (prompt input & outputs)

See Also

• https://arxiv.org/abs/2212.09720
• https://github.com/Hannibal046/Awesome-LLM
• https://tatsu-lab.github.io/alpaca_eval
• https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard
• https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard
• https://paperswithcode.com/task/language-modelling
  • https://paperswithcode.com/sota/code-generation-on-humaneval

• add linting config (e.g. tsc, eslint, prettier, stylelint, etc.)
• add pre-commit (husky) hooks
• add the above to CI checks

Description

We need to provide search capabilities for the blog too.

Description

Include a tutorial step by step on how to contribute to Petals Swarm using Prem App

• Install
  • Install locally (for development or personal usage)
  • Install for production
    • Docker Compose
    • Docker Swarm
    • Kubernetes

Description

Instructions on how to contribute to the developer portal

Description

We need to update the developer portal in order to include instructions/tutorial on how to use Domain / SSL management feature given by Prem Gateway.

Description

• https://github.com/dstackai/dstack

Description

A blog post about deploying a Petals Private Swarm in your infrastructure with an explanation of all the Petals building blocks.

Description

Most of the users get stack when downloading. We need to provide in the documentation a workaround to handle that (ssh in the server and manually pull the docker image with docker pull).

minor, but... ensure -- text is converted into em-dash (—)

Description

Include Alpaca Leaderboard in https://dev.premai.io/blog/evaluating-open-source-llms.

Links:

• https://huggingface.co/spaces/HuggingFaceH4/human_eval_llm_leaderboard
• https://tatsu-lab.github.io/alpaca_eval/

Single Source of Truth which all repo READMEs can link to

• p0: update service creation instructions (depends on premAI-io/prem-services#59)
• p0: architecture/repos & their purposes/relation
  • e.g. merge https://github.com/premAI-io/prem-daemon#prem-architecture & https://dev.premai.io/docs/intro#main-concepts
• p1: document manifests.json spec
• p2: minify/simplify tutorials
  • https://dev.premai.io/blog/serving-falcon-7b-fastapi-docker, ...

• add Daemon
• add App
• add Registry

and review

support convenient syntax e.g.

<img src="foo.png" width="1337" height="42"/>
![](foo.png =1337x42)
![](foo.png){width=1337 height=42}

At the moment the only way to interact with services is to spins up daemon & app as services, but many developers may find easier for testing or work in production to only run the exact container needed for their application, after thei done exploring through the app.

I think is worth to add an extra category here for users, something like "Use with Docker" and will point to ready-made snippets that involves the use of docker CLI. I do imagine

• Run a service (detached, with a nice name, restart on failure, attach GPU etc..)
• Check the logs (with tail of last 200 maybe)
• Stop/Delete a service

A bonus point would be to setup a simple docker swarm and define a number of desired replicas for the service.

Description

Currently is just a list from the most recent to the oldest post

Description

Currently, OpenAI embeddings are not the state of the art. We should write a blog post about it and see if developers can actually use open-source solutions as a real alternative to OpenAI.

• https://huggingface.co/spaces/mteb/leaderboard

NOTE: I noticed the context dimension of the different models while going through the leaderboard. Only the OpenAI model reaches ~8K tokens.