janhq / jan

Jan is an open source alternative to ChatGPT that runs 100% offline on your computer. Multiple engine support (llama.cpp, TensorRT-LLM)

Jan - Turn your computer into an AI computer

Getting Started - Docs - Changelog - Bug reports - Discord

Warning

Jan is currently in Development: Expect breaking changes and bugs!

Jan is an open-source ChatGPT alternative that runs 100% offline on your computer.

Jan runs on any hardware. From PCs to multi-GPU clusters, Jan supports universal architectures:

• NVIDIA GPUs (fast)
• Apple M-series (fast)
• Apple Intel
• Linux Debian
• Windows x64

Download

Version Type	Windows	MacOS		Linux
Stable (Recommended)	jan.exe	Intel	M1/M2/M3/M4	jan.deb	jan.AppImage
Experimental (Nightly Build)	jan.exe	Intel	M1/M2/M3/M4	jan.deb	jan.AppImage

Download the latest version of Jan at https://jan.ai/ or visit the GitHub Releases to download any previous release.

Demo

Realtime Video: Jan v0.4.3-nightly on a Mac M1, 16GB Sonoma 14

Quicklinks

Jan

• Jan website
• Jan GitHub
• User Guides
• Developer docs
• API reference
• Specs

Nitro

Nitro is a high-efficiency C++ inference engine for edge computing. It is lightweight and embeddable, and can be used on its own within your own projects.

• Nitro Website
• Nitro GitHub
• Documentation
• API Reference

Troubleshooting

As Jan is in development mode, you might get stuck on a broken build.

To reset your installation:

1. Use the following commands to remove any dangling backend processes:

   ps aux | grep nitro

   Look for processes like "nitro" and "nitro_arm_64," and kill them one by one with:

   kill -9 <PID>

2. Remove Jan from your Applications folder and Cache folder

   make clean

   This will remove all build artifacts and cached files:

   • Delete Jan extension from your ~/jan/extensions folder
   • Delete all node_modules in current folder
   • Clear Application cache in ~/Library/Caches/jan

Requirements for running Jan

• MacOS: 13 or higher
• Windows:
  • Windows 10 or higher
  • To enable GPU support:
    • Nvidia GPU with CUDA Toolkit 11.7 or higher
    • Nvidia driver 470.63.01 or higher
• Linux:
  • glibc 2.27 or higher (check with ldd --version)
  • gcc 11, g++ 11, cpp 11 or higher, refer to this link for more information
  • To enable GPU support:
    • Nvidia GPU with CUDA Toolkit 11.7 or higher
    • Nvidia driver 470.63.01 or higher

Contributing

Contributions are welcome! Please read the CONTRIBUTING.md file

Pre-requisites

• node >= 20.0.0
• yarn >= 1.22.0
• make >= 3.81

Instructions

1. Clone the repository and prepare:

   git clone https://github.com/janhq/jan
   cd jan
   git checkout -b DESIRED_BRANCH

2. Run development and use Jan Desktop

   make dev

This will start the development server and open the desktop app.

3. (Optional) Run the API server without frontend

   yarn dev:server

For production build

# Do steps 1 and 2 in the previous section
# Build the app
make build

This will build the app MacOS m1/m2 for production (with code signing already done) and put the result in dist folder.

Docker mode

• Supported OS: Linux, WSL2 Docker

• Pre-requisites:

  • Docker Engine and Docker Compose are required to run Jan in Docker mode. Follow the instructions below to get started with Docker Engine on Ubuntu.

    curl -fsSL https://get.docker.com -o get-docker.sh
    sudo sh ./get-docker.sh --dry-run

  • If you intend to run Jan in GPU mode, you need to install nvidia-driver and nvidia-docker2. Follow the instruction here for installation.

• Run Jan in Docker mode

  User can choose between docker-compose.yml with latest prebuilt docker image or docker-compose-dev.yml with local docker build

Docker compose Profile	Description
cpu-fs	Run Jan in CPU mode with default file system
cpu-s3fs	Run Jan in CPU mode with S3 file system
gpu-fs	Run Jan in GPU mode with default file system
gpu-s3fs	Run Jan in GPU mode with S3 file system

Environment Variable	Description
S3_BUCKET_NAME	S3 bucket name - leave blank for default file system
AWS_ACCESS_KEY_ID	AWS access key ID - leave blank for default file system
AWS_SECRET_ACCESS_KEY	AWS secret access key - leave blank for default file system
AWS_ENDPOINT	AWS endpoint URL - leave blank for default file system
AWS_REGION	AWS region - leave blank for default file system
API_BASE_URL	Jan Server URL, please modify it as your public ip address or domain name default http://localhost:1377

• Option 1: Run Jan in CPU mode

  # cpu mode with default file system
  docker compose --profile cpu-fs up -d
  
  # cpu mode with S3 file system
  docker compose --profile cpu-s3fs up -d

• Option 2: Run Jan in GPU mode

  • Step 1: Check CUDA compatibility with your NVIDIA driver by running nvidia-smi and check the CUDA version in the output

    nvidia-smi
    
    # Output
    +---------------------------------------------------------------------------------------+
    | NVIDIA-SMI 531.18                 Driver Version: 531.18       CUDA Version: 12.1     |
    |-----------------------------------------+----------------------+----------------------+
    | GPU  Name                      TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
    | Fan  Temp  Perf            Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
    |                                         |                      |               MIG M. |
    |=========================================+======================+======================|
    |   0  NVIDIA GeForce RTX 4070 Ti    WDDM | 00000000:01:00.0  On |                  N/A |
    |  0%   44C    P8               16W / 285W|   1481MiB / 12282MiB |      2%      Default |
    |                                         |                      |                  N/A |
    +-----------------------------------------+----------------------+----------------------+
    |   1  NVIDIA GeForce GTX 1660 Ti    WDDM | 00000000:02:00.0 Off |                  N/A |
    |  0%   49C    P8               14W / 120W|      0MiB /  6144MiB |      0%      Default |
    |                                         |                      |                  N/A |
    +-----------------------------------------+----------------------+----------------------+
    |   2  NVIDIA GeForce GTX 1660 Ti    WDDM | 00000000:05:00.0 Off |                  N/A |
    | 29%   38C    P8               11W / 120W|      0MiB /  6144MiB |      0%      Default |
    |                                         |                      |                  N/A |
    +-----------------------------------------+----------------------+----------------------+
    
    +---------------------------------------------------------------------------------------+
    | Processes:                                                                            |
    |  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
    |        ID   ID                                                             Usage      |
    |=======================================================================================|

  • Step 2: Visit NVIDIA NGC Catalog and find the smallest minor version of image tag that matches your CUDA version (e.g., 12.1 -> 12.1.0)

  • Step 3: Update the Dockerfile.gpu line number 5 with the latest minor version of the image tag from step 2 (e.g. change FROM nvidia/cuda:12.2.0-runtime-ubuntu22.04 AS base to FROM nvidia/cuda:12.1.0-runtime-ubuntu22.04 AS base)

  • Step 4: Run command to start Jan in GPU mode

    # GPU mode with default file system
    docker compose --profile gpu-fs up -d
    
    # GPU mode with S3 file system
    docker compose --profile gpu-s3fs up -d

This will start the web server and you can access Jan at http://localhost:3000.

Note: RAG feature is not supported in Docker mode with s3fs yet.

Acknowledgements

Jan builds on top of other open-source projects:

• llama.cpp
• LangChain
• TensorRT
• TensorRT-LLM

Contact

• Bugs & requests: file a GitHub ticket
• For discussion: join our Discord here
• For business inquiries: email [email protected]
• For jobs: please email [email protected]

Trust & Safety

Beware of scams.

• We will never ask you for personal info
• We are a free product; there's no paid version
• We don't have a token or ICO
• We are not actively fundraising or seeking donations

License

Jan is free and open source, under the AGPLv3 license.

jan's Issues

fix: llm chat - trim extra space in the response message

feat: Jan Cloud should use Jan

Overview

• cloud.jan.ai should use open source Jan (vs. our older hosted version)
• The intent is for a "multi-tenant Jan" that uses Keycloak realms

Decisions

Should we wait for "Teams share OpenAI Key" to upgrade cloud.jan.ai?

• Might take 2-3 weeks (including implementation of Teams)

feat: User to opt into Analytics and Logging?

Overview

• We have a Posthog instance
• Need to request if user is ok with Analytics (i.e. turn off if they disagree)
• Should this be configurable by an admin?

perf: scary traefik logs when runnign on linux

Env: Linux/ubuntu x86_64
Repro:

1. sucessfully run app
2. see the following errors nonstop

oviderName=docker
jan-traefik-1               | time="2023-09-05T05:43:15Z" level=debug msg="FIXME: Got an status-code for which error does not match any expected type!!!" error="Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get \"http://%2Fvar%2Frun%2Fdocker.sock/v1.24/version\": dial unix /var/run/docker.sock: connect: permission denied" module=api status_code=-1
jan-traefik-1               | time="2023-09-05T05:43:15Z" level=error msg="Failed to retrieve information of the docker client and server host: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get \"http://%2Fvar%2Frun%2Fdocker.sock/v1.24/version\": dial unix /var/run/docker.sock: connect: permission denied" providerName=docker
jan-traefik-1               | time="2023-09-05T05:43:15Z" level=error msg="Provider connection error Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get \"http://%2Fvar%2Frun%2Fdocker.sock/v1.24/version\": dial unix /var/run/docker.sock: connect: permission denied, retrying in 7.117021959s" providerName=docker
jan-traefik-1               | time="2023-09-05T05:43:22Z" level=debug msg="FIXME: Got an status-code for which error does not match any expected type!!!" status_code=-1 error="Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get \"http://%2Fvar%2Frun%2Fdocker.sock/v1.24/version\": dial unix /var/run/docker.sock: connect: permission denied" module=api
jan-traefik-1               | time="2023-09-05T05:43:22Z" level=error msg="Failed to retrieve information of the docker client and server host: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get \"http://%2Fvar%2Frun%2Fdocker.sock/v1.24/version\": dial unix /var/run/docker.sock: connect: permission denied" providerName=docker
jan-traefik-1               | time="2023-09-05T05:43:22Z" level=error msg="Provider connection error Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get \"http://%2Fvar%2Frun%2Fdocker.sock/v1.24/version\": dial unix /var/run/docker.sock: connect: permission denied, retrying in 12.496205234s" providerName=docker

bug: LLM stops responding after 3-4 messages

Env: 4090s on linux x86_64

Convo history:

username
01:47 pm
tell me a lohng story

Llama-2-7B-Chat
01:47 pm
 Absolutely! I'd be happy to listen to a long story. Please go ahead and tell me anything that comes to mind. Is there a particular theme or topic you'd like to explore, or would you prefer me to suggest some prompts?

username
01:48 pm
no you tell me a long story about mermaids please

Llama-2-7B-Chat
01:48 pm
###  Sure! Here is a long story about mermaids:

username
01:48 pm
where did the story go?

Llama-2-7B-Chat
01:48 pm
[never responds]

Logs:
none - or rather the current logs are so busy that I cannot see what even went wrong

docs: Overview

Overview

• Jan needs an "overview" to explain what we are trying to do
• This will be sent to potential customers, partners
• Include Roadmap?
• How does this differ from "Getting Started"? (should Getting Started just be a docs.jan.ai/guides/getting-started?)

Inspirations

• Rocketpool "Overview" is quite good
• https://docs.rocketpool.net/overview/

fix: chat UI last message received is glitchy

To repro:

1. have a few conversations, wait for responses to complete
2. toggle back and forth btw chats
3. see the last ai reply rerender each time.
4. strange

https://www.loom.com/share/c15febd4c81b42738a165ea4e5cf1d77

fix: clean up legacy Jan web app SEO metadata

Right now, when you go to localhost 3000, the web app tab name says "Jan. free blabh blah..." It's legacy seo.

Let's just delete all the SEO metadata and the tab should just say "Jan"

Same thing for the other seo/metadata that I can't think of off of top of my head

bug: hasura image generation timeout

Should convert to event later on

feat: users can use Nitro models

feat: minimally improve login flow

• fill in default username/password (from env)
• login page shouldn't show default Hasura logo, etc.

docs: reflect (and improve) installation & setup steps from readme to docs.jan.ai

success critieria:

• pre reqs
• install
• setup
• run
  steps are all reflected in https://docs.jan.ai/ (getting started page) in greater detail

With:

• Mac/Linux/Windows (in this order, exact copy) instructions in tab view
  • @juzblazin to add the bulk of the content & mac cmds
  • @Its-Alamin-H to work on windows cmds
  • @0xSage to add the linux cmds
• logs & screenshots of the tricker steps?

bug: SyntaxError: Unexpected token ':', ": ping - 2"... is not valid JSON

jan-worker-1                |
jan-worker-1                |
jan-graphql-engine-1        | {"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"172.20.0.13","method":"POST","status":200,"url":"/v1/graphql"},"operation":{"query":{"query":"\nmutation chatCompletions($id: uuid = \"\", $data: messages_set_input) {\n  update_messages_by_pk(pk_columns: {id: $id}, _set: $data) {\n    id\n    content\n  }\n}\n","variables":{"data":{"content":" Not much, just hanging out here"},"id":"2368262c-5a65-4057-9203-9ec64588ec19"}},"query_execution_time":4.6397458e-2,"request_id":"62a9a2c3-b677-4aae-8b83-7d5c760227ce","request_mode":"single","request_read_time":3.5e-6,"response_size":125,"uncompressed_response_size":125,"user_vars":{"x-hasura-role":"admin"}},"request_id":"62a9a2c3-b677-4aae-8b83-7d5c760227ce"},"level":"info","timestamp":"2023-09-03T0SyntaxError: Unexpected token ':', ": ping - 2"... is not valid JSON
jan-worker-1                |     at JSON.parse (<anonymous>)
jan-worker-1                |     at worker.js:135:28
jan-worker-1                |     at async jsonError (worker.js:228:12) {
jan-worker-1                |   stack: SyntaxError: Unexpected token ':', ": ping - 2"...…:135:28
jan-worker-1                |     at async jsonError (worker.js:228:12),
jan-worker-1                |   message: Unexpected token ':', ": ping - 2"... is not valid JSON
jan-worker-1                | }
jan-worker-1                |
jan-worker-1                |

perf: better stdout logging during install and usage

fix: allow users to run other llama OR sd (or both)

Assumption: right now we force users to install the 2 foundational models (llama 2 and sd 1.5).

Ideal: Let's make both AND/OR either models optional

fix: hasura seeds should not yield error on conflict

bug: jan containers wont restart

Env: mac pro arm

To repro:

1. Deploy Jan: docker compose up
2. Stop containers: ctrl+c
3. See the following error

Logs

jan-data-connector-agent-1  | {"timestamp":"2023-09-04T02:13:34.63Z","sequence":111,"loggerClassName":"org.jboss.logmanager.Logger","loggerName":"io.opentelemetry.exporter.internal.grpc.OkHttpGrpcExporter","level":"SEVERE","message":"Failed to export spans. The request could not be executed. Full error message: Failed to connect to localhost/127.0.0.1:4317","threadName":"OkHttp http://localhost:4317/...","threadId":29,"mdc":{},"ndc":"","hostName":"0a623d5d26ef","processName":"super-connector.jar","processId":1}
jan-graphql-engine-1        | {"timestamp":"2023-09-04T02:13:35.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"failed waiting for 9691, try increasing HASURA_GRAPHQL_MIGRATIONS_SERVER_TIMEOUT (default: 30)"}}
jan-traefik-1               | time="2023-09-04T02:13:35Z" level=debug msg="Provider event received {Status:die ID:d955a2c5b1f9e7ed8417640a4fef63bc8654b7e17f27e70024d7827a6fc553f0 From:hasura/graphql-engine:v2.31.0.cli-migrations-v3 Type:container Action:die Actor:{ID:d955a2c5b1f9e7ed8417640a4fef63bc8654b7e17f27e70024d7827a6fc553f0 Attributes:map[com.docker.compose.config-hash:e56b1cc19f80980f5e19c0d2fa1a43635bba84ed845ec7832abfd8ee62622c06 com.docker.compose.container-number:1 com.docker.compose.depends_on:data-connector-agent:service_healthy:false com.docker.compose.image:sha256:5a614e0e760a3f91cee5c093cc7c9e2f676004396af7e8dc9214840976fc5d57 com.docker.compose.oneoff:False com.docker.compose.project:jan com.docker.compose.project.config_files:/Users/nicolezhu/eigenvector/jan/docker-compose.yml com.docker.compose.project.working_dir:/Users/nicolezhu/eigenvector/jan com.docker.compose.service:graphql-engine com.docker.compose.version:2.20.2 desktop.docker.io/binds/0/Source:/Users/nicolezhu/eigenvector/jan/app-backend/hasura/migrations desktop.docker.io/binds/0/SourceKind:hostFile desktop.docker.io/binds/0/Target:/migrations desktop.docker.io/binds/1/Source:/Users/nicolezhu/eigenvector/jan/app-backend/hasura/metadata desktop.docker.io/binds/1/SourceKind:hostFile desktop.docker.io/binds/1/Target:/metadata execDuration:30 exitCode:1 image:hasura/graphql-engine:v2.31.0.cli-migrations-v3 name:jan-graphql-engine-1 org.opencontainers.image.ref.name:ubuntu org.opencontainers.image.version:22.04 traefik.enable:true traefik.http.middlewares.graphql-engine.stripprefix.forceslash:false traefik.http.middlewares.graphql-engine.stripprefix.prefixes:/graphql-engine traefik.http.routers.graphql-engine.entrypoints:jan traefik.http.routers.graphql-engine.middlewares:graphql-engine traefik.http.routers.graphql-engine.rule:(PathPrefix(`/graphql-engine`))]} Scope:local Time:1693793615 TimeNano:1693793615863117304}" providerName=docker

dependency failed to start: container jan-graphql-engine-1 is unhealthy

perf: many jan-web error logs when restarting jan services

To repro:

1. Docker compose up
2. ctrl+c, stop all
3. docker compose up again

It works but for ~1 minute I get the scary logs:

jan-web-1                   | Error refreshing access token TypeError: fetch failed
jan-web-1                   |     at Object.fetch (node:internal/deps/undici/undici:11576:11)
jan-web-1                   |     at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
jan-web-1                   |   cause: AggregateError
jan-web-1                   |       at internalConnectMultiple (node:net:1114:18)
jan-web-1                   |       at afterConnectMultiple (node:net:1667:5)
jan-web-1                   |       at TCPConnectWrap.callbackTrampoline (node:internal/async_hooks:130:17) {
jan-web-1                   |     code: 'ECONNREFUSED',
jan-web-1                   |     [errors]: [ [Error], [Error] ]
jan-web-1                   |   }
jan-web-1                   | }
jan-web-1                   | Error refreshing access token TypeError: fetch failed
jan-web-1                   |     at Object.fetch (node:internal/deps/undici/undici:11576:11)
jan-web-1                   |     at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
jan-web-1                   |   cause: AggregateError
jan-web-1                   |       at internalConnectMultiple (node:net:1114:18)
jan-web-1                   |       at afterConnectMultiple (node:net:1667:5)
jan-web-1                   |       at TCPConnectWrap.callbackTrampoline (node:internal/async_hooks:130:17) {
jan-web-1                   |     code: 'ECONNREFUSED',
jan-web-1                   |     [errors]: [ [Error], [Error] ]
jan-web-1                   |   }
jan-web-1                   | }
jan-web-1                   | Error refreshing access token TypeError: fetch failed
jan-web-1                   |     at Object.fetch (node:internal/deps/undici/undici:11576:11)
jan-web-1                   |     at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
jan-web-1                   |   cause: AggregateError
jan-web-1                   |       at internalConnectMultiple (node:net:1114:18)
jan-web-1                   |       at afterConnectMultiple (node:net:1667:5)
jan-web-1                   |       at TCPConnectWrap.callbackTrampoline (node:internal/async_hooks:130:17) {
jan-web-1                   |     code: 'ECONNREFUSED',
jan-web-1                   |     [errors]: [ [Error], [Error] ]
jan-web-1                   |   }
jan-web-1                   | }
jan-web-1                   | Error refreshing access token TypeError: fetch failed
jan-web-1                   |     at Object.fetch (node:internal/deps/undici/undici:11576:11)
jan-web-1                   |     at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
jan-web-1                   |   cause: AggregateError
jan-web-1                   |       at internalConnectMultiple (node:net:1114:18)
jan-web-1                   |       at afterConnectMultiple (node:net:1667:5)
jan-web-1                   |       at TCPConnectWrap.callbackTrampoline (node:internal/async_hooks:130:17) {
jan-web-1                   |     code: 'ECONNREFUSED',
jan-web-1                   |     [errors]: [ [Error], [Error] ]
jan-web-1                   |   }
jan-web-1                   | }
jan-web-1                   | - wait compiling /api/auth/logout/route (client and server)...
jan-web-1                   | - event compiled successfully in 181 ms (1812 modules)
jan-web-1                   | token is expired, refresh it
jan-web-1                   | token is expired, refresh it
jan-web-1                   | token is expired, refresh it
jan-web-1                   | token is expired, refresh it
jan-web-1                   | token is expired, refresh it
jan-web-1                   | token is expired, refresh it

Moving state management from mobx to Jotai

blogs: ultimate guides to X

We will invest in long-term guides as part of docs site:

• High quality educational resources, vs. "fast food" social media posts
• Target Enterprise customers who want to understand technical depth
• Save people the time of having to trawl through r/localllama, Discords for quality information

Possible Guides

• Ultimate guide to Generative AI
• Ultimate guide to Local AI
• Ultimate guide to LLMs
• Ultimate guide to StableDiffusion

Inspirations

• https://zapier.com/resources/guides
• https://posthog.com/tutorials
• https://gpus.llm-utils.org/nvidia-h100-gpus-supply-and-demand/

feat: Jan should support text2image (StableDiffusion, Controlnet, Upscaler, etc)

feat: Keycloak should be at /auth instead of port number?

Overview

• Currently, Keycloak opens at a different port number (ugly)
• If they are self-hosting, should it be at a URL instead? e.g. /auth
• What is the best practice for this?

Refactor installation script to only handle dependency and installation

Deliverables

• README.md updated
• install.sh script

Ideal Flow:

1. Checking & installation of OS dependencies

• All the things Louis already configured
• homebrew
• Docker
• Docker compose
• (Optional) GPU drivers: nvidia-smi, cuda, nvidia-driver

1. Git clone, submodule pull and other steps

Do not include:

1. Model(s) installation, e.g. wget...
2. Running Jan, e.g. docker compose up
3. Running the Hasura seed migration

Changelog

Previous Instructions

feat: Install with shell script instead of docker

pros from @louis-jan :

• less installation steps
• missing dependencies check
• different arch setup
  • On Mac M1s: https://llama-cpp-python.readthedocs.io/en/latest/install/macos/ , faster inference
  • On other architectures: enable other bare metal stuff that we dont get access to from within a container
• nice progress bar during install

docs: Getting Started

Overview

• We should have very easy, clear "Getting Started" guides
• Clear guides for each platform (e.g. Windows, WSL, Mac, Linux)
• Clear guides for GPU or non-GPU
• Clear guides for Cloud deployments (i.e. Helm)

This will need to be taken on by engineers, due to the technical nature of the content.

feat: User can see elapsed time for each AI inference request

Overview

• Similar to Perplexity https://labs.perplexity.ai/
• We would have this for each AI query

Private Zenhub Image

Additional Ideas

• We can have a ShareGPT equivalent, where we publish the (a) model, (b) infrastructure type, and (c) response times for the model.
• Could be very useful for benchmarking hardware performance, model choice (e.g. 7b, 34b, 70b), and inference speed

UX: Prompt Box should be expandable, collapsible

Private Zenhub Image

bug: llama 7B on llama.cpp on Apple Silicon is responding at 4 tokens per minute

What's the issue?

Env: Mac M1 Pro arm64

Solution: un-dockerize llama.cpp, so we can use Metal

fix: default light mode

Dark mode looks off on first glance. I suggest we go with default light mode instead

Production Deployment Instructions for Jan

Deliverable

• Documentation on Prod Deployment
• Helm Charts?

perf: during install clearly log which service is being scaffolded and why (progress bar)

something like this would be cool:

bug: switching between models got blank page

Force reload the page, switch from LLM to SD convo -> blank page without messages loaded

perf: indicate when all services are ready, print the ports and services

When services are ready, signal to the user that Jan is ready to use.

Log to stdout:

• 👋 Jan is ready!
• This table from README

Service	URL	Credentials
Keycloak	http://localhost:8088	Admin credentials are set via the environment variables KEYCLOAK_ADMIN and KEYCLOAK_ADMIN_PASSWORD
app-backend (hasura)	http://localhost:8080	Admin credentials are set via the environment variables HASURA_GRAPHQL_ADMIN_SECRET in file conf/sample.env_app-backend
web-client	http://localhost:3000	Users are signed up to keycloak, default created user is set via conf/keycloak_conf/example-realm.json on keycloak with username: username, password: password
llm service	http://localhost:8000
sd service	http://localhost:8001

architecture: Jan Cloud multi-tenancy

Overview

• cloud.jan.ai should be a multi-tenant version of Jan
• Cater to users who want a 'managed' Jan
• Cater to users who don't have as high privacy requirements (e.g. share OpenAI key, access cheap open source AI inference)
• Some customers might need LDAP for realm

Architecture Decisions

• What should the approach be for Keycloak? (should we implement this as multi-tenant Keycloak?)

docs: Company

Overview

• "Build in Public" for Jan
• Company Handbook

Inspirations

• https://posthog.com/handbook

readme: Jan Github README should be "designed"

Overview

• Badges
• Jan's "Sales Pitch"
• "Banner" @0xSage
• Gif screen recording of Jan Web (and mobile?) @0xSage
• Diagram explaining Jan product offering @dan-jan

Private Zenhub Image

Resources

• n8n
• Posthog

docs: instructions on how to deploy using Helm

Add hover state to Left Panel shortcuts

Migrated from Linear:

Private Zenhub Image

Jan should use Responsive TailwindUI layout

Overview

• Currently web version is not compatible with mobile device
• Self-css to follow design - should utilize tailwindui layout

Ideal Deliverable

• UI should be "polished" (i.e. "paid" software)
• UX should be similar to ChatGPT

Resources

• Multi column layout: https://tailwindui.com/components/application-ui/application-shells/multi-column#component-65815af4357dfdcf9be55c390221c5fb
  Private Zenhub Image

• Right pane slide overs: https://tailwindui.com/components/application-ui/overlays/slide-overs#component-4d1d575bd79eef81afc9a4f2e1900ecb
  Private Zenhub Image

• Settings page: https://tailwindui.com/components/appl

• ication-ui/page-examples/settings-screens
  Private Zenhub Image

Tasking

• @drakehere to work with @namchuai

feat: users can configure and opt out of services

Current: Jan monorepo is opinionated and heavy with many services, many of which are not critical to the individual enthusiast
e.g. keycloak, minio, services noncritical to the inferencing

Users may not love the many containers, lengthy scaffold time, or they may just not like the particular 3rd party integration tools that we chose.

Ideal: Let's make such services and their respective 3rd integrations hot swappable.

Rough design:

• Services can be turned off via a config yaml in base repo
• Can we turn off the entire app backend if users don't want to persist data?
• [optional] services can be swapped for a different 3rd party provider endpoint, e.g. aws s3 -> gcp store

bug: stop at step 5, no log

Bash script stopped at step 5, without any log

docker info

Client: Docker Engine - Community
 Version:    24.0.6
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2
    Path:     /Users/alandao/.docker/cli-plugins/docker-buildx
WARNING: Plugin "/Users/alandao/.docker/cli-plugins/docker-compose" is not valid: failed to fetch metadata: fork/exec /Users/alandao/.docker/cli-plugins/docker-compose: no such file or directory
WARNING: Plugin "/Users/alandao/.docker/cli-plugins/docker-dev" is not valid: failed to fetch metadata: fork/exec /Users/alandao/.docker/cli-plugins/docker-dev: no such file or directory
WARNING: Plugin "/Users/alandao/.docker/cli-plugins/docker-extension" is not valid: failed to fetch metadata: fork/exec /Users/alandao/.docker/cli-plugins/docker-extension: no such file or directory
WARNING: Plugin "/Users/alandao/.docker/cli-plugins/docker-init" is not valid: failed to fetch metadata: fork/exec /Users/alandao/.docker/cli-plugins/docker-init: no such file or directory
WARNING: Plugin "/Users/alandao/.docker/cli-plugins/docker-sbom" is not valid: failed to fetch metadata: fork/exec /Users/alandao/.docker/cli-plugins/docker-sbom: no such file or directory
WARNING: Plugin "/Users/alandao/.docker/cli-plugins/docker-scan" is not valid: failed to fetch metadata: fork/exec /Users/alandao/.docker/cli-plugins/docker-scan: no such file or directory
WARNING: Plugin "/Users/alandao/.docker/cli-plugins/docker-scout" is not valid: failed to fetch metadata: fork/exec /Users/alandao/.docker/cli-plugins/docker-scout: no such file or directory

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 2
 Server Version: 23.0.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 1fbd70374134b891f97ce19c70b6e50c7b9f4e0d
 runc version: 860f061b76bb4fc671f0f9e900f7d80ff93d4eb7
 init version: 
 Security Options:
  seccomp
   Profile: builtin
 Kernel Version: 6.1.29-0-virt
 Operating System: Alpine Linux v3.18
 OSType: linux
 Architecture: aarch64
 CPUs: 10
 Total Memory: 15.6GiB
 Name: colima
 ID: a16ca123-ffbe-4d18-aaf9-8cc0ac00786f
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

feat: Product Input should be definable by YAML or Gradio-like interface

issue: Some statuses are not running, it's showing Exited

bug: SD 1.5 isn't working at all - remove it from Jan

no logs, no nothing.

env: mac m1
Also
env: linux x86 with 4090s

feat: Helm Chart for Jan

docs: add a changelog tab in blog style

requirement:

• Add Changelog as the last-left-tab on the nav bar
• Use Docusaurus blog feature to do this https://docusaurus.io/docs/blog
• Hint: you should not be installing anything new!

chore: UI should say "me" instead of "user"

Private Zenhub Image

perf: jan-data-connector-agent SEVERE log

What is a data-connector-agent in the first place?
Should we name this service better for users?

Separately:
I know we haven't enabled otel, so maybe we can do something to disable this log. It makes for a very spammy install experience.

Can we:

• turn if off?
• disable the log?
• lower the log severity?
  TBD

jan-data-connector-agent-1  | {"timestamp":"2023-09-05T05:50:26.171Z","sequence":128,"loggerClassName":"org.jboss.logmanager.Logger","loggerName":"io.opentelemetry.exporter.internal.grpc.OkHttpGrpcExporter","level":"SEVERE","message":"Failed to export spans. The request could not be executed. Full error message: Failed to connect to localhost/127.0.0.1:4317","threadName":"OkHttp http://localhost:4317/...","threadId":84,"mdc":{},"ndc":"","hostName":"8ca47fbf537b","processName":"super-connector.jar","processId":1}

product specs: Jan installation flow?

Overview

• Current setup is a bit janky (i.e. admin is imported via Keycloak realm)

Solution?

• Jan first load should guide user through an installation step
• First user (i.e. admin) should provision their own username/password in Keycloak
• Installation step will run Hasura migrations, etc (vs. using Docker)

docs: model table with benchmarks, formats, etc.

Insert a table describing the models supported, performance, backends, etc.

Git submodule update error

Need to install xcode command line tool