brutalcoding / shady.ai

Making offline AI models accessible to all types of edge devices.

ShadyAI

_{My take to get privacy friendly, offline AI models accessible to as many people as possible. Offline first. Privacy first. Open source. Free. What happens in ShadyAI, stays in ShadyAI. No ads. No tracking. No data collection. No registration. No subscription. No in-app purchases. No premium features. Nada. Nothing. That's it. Pure AI, pure intentions—that's the essence of this app.}

Intro

Ever noticed how AI tends to lurk in the shadows? With a sea of startups sporting cryptic names, ShadyAI playfully acknowledges AI's penchant for mystique. It's all a bit of light-hearted banter. In the contrary, behind this playful façade lies a genuine commitment to transparency and ethical AI practices. ShadyAI is fully open source, and is dedicated to bringing clarity, openness, and strong emphasis on privacy.

There are many cutting edge offline AI models, and I want to catch 'em all. Think of Segment Anything, LLaMA and Whisper. What do I mean by that? Well, I want to make it easy for anyone to use cutting edge AI advancements like these without the technical hurdles that come with it.

Sponsors

• My empty wallet
• My free time (I'm a full-time dad, full-time employee, and part-time seeker of getting my work-life balance right)

Progress

Items are subject to change. This list is not exhaustive, I'll update it as I go.

The one thats actively being worked on is marked with a 🚧 emoji.

• Make a slide deck with Flutter (Bonus: contribute back custom changes)
• Generate Dart bindings for llama.cpp
• Generate .dylib for macOS (ARM64) (Bonus: with Metal artifact for GPU acceleration)
• Successfully Load instance of LLaMA with dylib
• Successfully load LLaMA model in GGUF format with default LLaMA context parameters
• Successfully run inference on LLaMA model with custom prompts
• Add a Quick Start experience to the app
• 🚧 Create a cross-platform Flutter FFI plugin to power your own apps the same way (without the countless nights of frustation)
• Extend supported platforms:
  • macOS (ARM64)
  • macOS (x86_64)
  • Windows (x86_64)
  • Linux (x86_64)
  • Android (ARM64)
  • iOS (ARM64)
  • Web (ShadyAI for web should have WebGPU support)
• Week 11 Spike (2 days): text-to-image, segmentation, speech-to-text.
• Publish video(s), blog post(s), add visuals (e.g. in README), and gather early feedback.

Features

Items are subject to change. This list is not exhaustive, I'll update it as I go.

• Ask AI a single question. (Example: "What is the meaning of life?")
• Give AI a single instruction. (Example: "Write a Flutter widget with a button that says 'Hello World'")
• Choose from pre-defined prompt templates.
• Get started quickly by using the built-in tiny-but-mighty AI model. (Note: It's the TinyStories model of < 100 MB, great at generating short stories. I'm keeping an eye on several ~1B models though)
• Ask AI generate an image based on a short description. (Example: "A cat with a hat")
• On-going chat conversation with AI. (Like talking to a friend, with a twist)
• Translate text to another language.
• Copy anyone's voice and say anything you want.
• Make music with AI.
• Make short videos with AI.

If you want to see a feature implemented, please open an issue.

Verified Devices

Here is a list of devices I've tested ShadyAI on. This is not an exhaustive list, your device should work too. If that's not the case, please let me know by opening an issue.

• MacBook Pro (16-inch, 2021).
  • macOS Ventura - Version 13.5.1
  • Chip: Apple M1 Pro
  • Memory: 16 GB

Experimental

ShadyAI might also work by just visiting shady.ai. This is very experimental though, don't get too excited yet!

This website gets freshly baked from the oven every night at 00:00 UTC, as seen on this deployments page.

DIY

Do it yourself. You've got the source code, here's how you can build it

1. Install Flutter and make sure you can run their example app (counter app).
2. Install FVM (Flutter Version Manager) and make sure you can run it (e.g. fvm --version outputs 2.4.1 at the time of this writing).
3. Open your terminal, navigate to the root of this project (e.g. cd ~/my-favorite-projects/shady.ai)
4. Navigate to the frontend: cd shady_ai_flutter
5. Install the same Flutter version: fvm install
6. Finally, run the app with: flutter run (if multiple platforms are detected, e.g. MacOS and your phone, you'll be prompted to pick 1 device)

Benchmark

My dad, the retired butcher in his mid-70s, is going to be the benchmark.

In order for the benchmark to pass, this checklist must be completed.

Checklist:

• Tell dad to download the app.
• Ask him to have a conversation with the app. It's a pass if he's chatting with the AI for at least 5 minutes.
• Ask him to generate an image. It's a pass if he's able to generate an image, he's happy with the result, and he's able to share it with anyone.

If it's not completed, it's a fail. If it's completed, it's a pass.

About my dad.

He's in his mid 70s. Great memory. Always forgets my date of birth, yet somehow remembers details about his customers from 30 years ago. Has a great sense of humour. Oh wait a minute, his memory is not that great after all. Still, he's a great dad.

Say hi to my dad:

Tech Stack

• Flutter - Frontend (in Dart)
• llama.cpp - Allows me to run one of BlinkDL's models on the CPU, fast inference.
• Design System - Design system made by Google. Helps me to get a consisting theme throughout the app in less effort than doing it all myself.
• And many more...
• Serverpod - Backend (in Dart). ShadyAI is currently not using any backend. But I'm planning to use Serverpod to make it easier later to run powerful models that you can deploy on your own server. I bet the folks at r/SelfHosted would love that. I know I would. Just imagine a single command line to run a Docker image that runs a powerful AI model (or multiple models) that you can access from anywhere. That's the dream. I'm not there yet, but I'm working on it.

FAQ

Q: What is so shady about it?

A: Ironically, nothing at all. The name ShadyAI is a playful twist, hinting at the often-mysterious nature of AI. If using it makes you "shady," well, that'll be our little secret. Bottom line? What happens in ShadyAI stays in ShadyAI.

Q: What advice do you have for someone who wants to build something like this?

A: It starts with passion. Jumping on the bandwagon because something's "in vogue" won't sustain you in the long run. Trends are fleeting. Dive deep into the subject — in this instance, AI. Understand its capabilities and think critically about the real-world problems you're eager to tackle. For me, it's not just about talking up AI's potential; it's about demonstrating it, making it tangible and accessible to all. I use a simple benchmark: if my parents start using it voluntarily, I know I'm onto something. It's a journey, and while I'm not there yet, I'm committed to the path.

Disclaimer

This project is not affiliated with Meta, OpenAI or any other company. This is a personal project made by me. ShadyAI, myself, and anyone else involved in this project are not responsible for any damages caused by the use of this app. Use at your own risk.

License

See the LICENSE file.

Contributing

Feel free to open an issue or a pull request. I'm open to any suggestions.

Friends of ShadyAI

• llama.dart - Dart bindings for llama.cpp. Made by me.
• llama.cpp - C++ library for building and running AI models locally.
• Flutter Perth - Perth's Flutter meetup group. I'm the organizer. Join my regular online meetups to learn more about Flutter and AI.

shady.ai's Issues

[BUG] 🐛 - Missing a tokenizer. Probably need to add Dart bindings for HF tokenizer?

Describe the bug
I couldn't wrap my head around last couple of days on what's missing in order to have a conversational AI. A short while ago, I added the AI evaluation feature and successfully performed an eval on a 3B model. In other words, I was able to load it in and run a test against it.

Great, eval done. Chat can be done in less than 15 minutes of work. Right? Yes but no. You see, the problem is that I'm missing an important component: a tokenizer. I've been too blind to notice this until today thanks to a good friend of mine.

Now, that's the theory at least. Will put this to practice and see if I can finish it this week.

PS. I'm not an AI expert, more of an AI noob that's good enough to grasp concepts and simply able to read instructions to run them.

Create video or GIF of initial prototype of MacOS chat model with drag-and-drop

Add chat UI plugin for basic interface

Add drag-and-drop area for at least MacOS client

[BUG] 🐛 - MacOS Shady flutter app is crashing

I’m using a rwkv bin model file. But my app is crashing when the model file is loaded without any specific error. Can you please help me in loading this app first for macOS and then for Android and iOS devices.

I am attaching my error report below for your convenience.
ErrorReport.pdf

Add support for at least 1 more platform (no preference)

What I hope to accomplish is adding this chat experience to:

• MacOS
• Android
• iOS

But if I run into roadblocks, I will simply skip it for now and attempt the remaining platforms instead (Windows, Linux, Web).

Add drag-and-drop area for MacOS, to allow dropping in any supported LLM models

[DOCS] - Create a document about using Fastlane for Flutter

Is your feature request related to a problem? Please describe.
The process of building, signing and deploying apps to their official respective stores can take hours of manual work.

If I were to do this manually, I would have to compile and wait between each platform.

A single platform such as iOS could already take 15min from source code to App Store review submission. This is depending on the complexity of course.

Here's a breakdown of what is involved if I would create an iOS build manually (more than 15min):

• Run tools to analyze code for linter issues, unused code, unused files, unit tests etc.
• Compiling Flutter source code to unsigned iOS app
• Syncing keychain with up-to-date provisioning profiles / certificates
• Analyzing commits and bumping up the version for a semantic release
• Bumping up the version number in all places (source code, changelog, git tag, git release etc)
• Overwrite iOS icon assets with a badge on top of it (note: 10+ images due to various sizes)
• Creating a PR (Pull Request) and fill in all the details
• Uploading git release artifacts (.ipa)
• Upload signed build to the App Store
  ..And so forth..

I definitely missed some steps. Think of:

• Automatically running an E2E (end-to-end) test going through different screens
• Automatically creating screenshots + adding text overlays on top of screenshots
• Upload and replace existing screenshots @ App Store

And that was just 1 flavor for 1 platform. Would be nicer to have alpha, beta and prod flavors right?

Describe the solution you'd like
You can see that DevOps can be frustating, and no matter what tool I end up using, it will be better than doing this manually. I've had good experience with Fastlane to deploy iOS & Android apps, so I will make that an important part of ShadyAI too.

Describe alternatives you've considered
An alternative is do build everything in a Docker container, however, Docker container running MacOS is a bit.. shady.

From my understanding, emulating MacOS in Docker on non-Apple hardware is a no-go. But if you are running it on Apple hardware (e.g. Macbooks), it is allowed to run up to 2 MacOS VM's for purposes such as CI/CD (not commercially).

There's more:

• Cirrus Labs Tart - A pre-configured Flutter+MacOS VM for sillicon based Macs
• Any cloud CI/CD platform such as GitHub Actions, CodeMagic.io or CirrusCI

All the alternatives are interesting, but none of them is a direct 1:1 replacement for Fastlane. I think I should still configure Fastlane within any of the alternatives mentioned. That's why I'm starting off with Fastlane.

[REQUEST] 💡 - Create a prototype of a working local AI model in ShadyAI

Is your feature request related to a problem? Please describe.
I would like to be able to run ShadyAI that interacts with an AI model.

Describe the solution you'd like
Ideally I like to see a version where anyone can download ShadyAI and start interacting with an AI model right out of the box.

Describe alternatives you've considered
Alternatively, if that's too soon, I'd consider these options to be viable for the time being. This list is ranked from best to worst:

1. Shady's Favorites
   An easy to use interface that let's you select any of provided AI models, which then will be downloaded and used.
2. BYOAI - Bring Your Own AI
   If you've got an AI model that you want to try out (a .bin file), ShadyAI must provide a way to let you use that during runtime. Think of a drag-and-drop solution or a file-picker dialog.