occ-ai / obs-polyglot

Realtime local AI translation services for OBS based on CTranslate2

Ployglot - Translation AI assistant OBS Plugin

Introduction

Polyglot translation AI plugin allows you to translate text in multiple languages in real-time and locally on your machine. ✅ No GPU required, ✅ no cloud costs, ✅ no network and ✅ no downtime! Privacy first - all data stays on your machine.

"polyglot" is derived from the Greek words: "poly-" meaning "many"and "glōtta" (or "glōssa") meaning "tongue" or "language".

It's using the excellent CTranslate2 project from OpenNMT.

If this free plugin has been valuable to you consider adding a ⭐ to this GH repo, subscribing to my YouTube channel where I post updates, and supporting my work: https://github.com/sponsors/royshil

Usage Tutorial

Watch a short tutorial on how to use and setup Polyglot on your OBS scene.

https://youtu.be/ZMNILPWDkDw

Current Features:

• Translate in real time using an internal HTTP server
• Choice of CTranslate2 model file

Roadmap:

• Translation directly on OBS text sources, e.g. with a filter
• Translation of live stream or recording captions

Check out our other plugins:

• Background Removal removes background from webcam without a green screen.
• 🚧 Experimental 🚧 CleanStream for real-time filler word (uh,um) and profanity removal from live audio stream
• URL/API Source that allows fetching live data from an API and displaying it in OBS.
• LocalVocal speech AI assistant plugin for real-time transcription (captions), translation and more language functions

Download

Check out the latest releases for downloads and install instructions.

Models

You need to download a CT2 model for the translation service to work.

Here are download links for models that are compatible with the plugin:

• https://forum.opennmt.net/t/nllb-200-with-ctranslate2/5090 NLLB-200 multilingual model (recommended) (List of NLLB language codes)
• https://huggingface.co/models?sort=trending&search=ctranslate CTranslate2 models on HuggingFace

Download models options: (get e.g. model.bin and SPM = SentencePiece Model .model file)

• https://forum.opennmt.net/t/nllb-200-with-ctranslate2/5090 (NLLB)
• https://huggingface.co/JustFrederik/m2m_100_418m_ct2_int8/tree/main (M2M100)
• https://huggingface.co/Rohith04/ct2fast_m2m100_418M/tree/main (M2M100)
• https://huggingface.co/jncraton/m2m100_418M-ct2-int8/tree/main (M2M100)

Building

The plugin was built and tested on Mac OSX (Intel & Apple silicon), Windows and Linux.

Start by cloning this repo to a directory of your choice.

Remember to sync and fetch the submodules before building, e.g.

$ git submodule sync --recursive
$ git update --init --recursive

Mac OSX

Using the CI pipeline scripts, locally you would just call the zsh script. By default this builds a universal binary for both Intel and Apple Silicon. To build for a specific architecture please see .github/scripts/.build.zsh for the -arch options.

$ ./.github/scripts/build-macos -c Release

Install

The above script should succeed and the plugin files (e.g. obs-urlsource.plugin) will reside in the ./release/Release folder off of the root. Copy the .plugin file to the OBS directory e.g. ~/Library/Application Support/obs-studio/plugins.

To get .pkg installer file, run for example

$ ./.github/scripts/package-macos -c Release

(Note that maybe the outputs will be in the Release folder and not the install folder like pakage-macos expects, so you will need to rename the folder from build_x86_64/Release to build_x86_64/install)

Linux (Ubuntu)

Use the CI scripts again

$ ./.github/scripts/build-linux.sh

Copy the results to the standard OBS folders on Ubuntu

$ sudo cp -R release/RelWithDebInfo/lib/* /usr/lib/x86_64-linux-gnu/
$ sudo cp -R release/RelWithDebInfo/share/* /usr/share/

Note: The official OBS plugins guide recommends adding plugins to the ~/.config/obs-studio/plugins folder.

Windows

Use the CI scripts again, for example:

> .github/scripts/Build-Windows.ps1 -Target x64 -CMakeGenerator "Visual Studio 17 2022"

The build should exist in the ./release folder off the root. You can manually install the files in the OBS directory.

obs-polyglot's Issues

overload CPU

Hi, i noted some strange behavior when polyglot http server is started.
Check please attachments:

How can i fix this issue?

It seems that stopping service can crash OBS

1. Install plugin in OBS

2. Setup models:

• model: nllb-200-distilled-600M-int8
• SPM: flores200_sacrebleu_tokenizer_spm.model

3. Start service "Start / Stop HTTP Server"

4. Make some requests to verify it is up and running
   _note: I gave up with testing with plugin: URL Source
   note: I used VS code Thunder client with some requests, but Curl commandline sould do it too.

5. Make sure last request is completed
   _note: some known languages did not transalte at all ie. "ita_Latin"

6. Stop server with Start / Stop HTTP Server button in PolyGlot Translate plugin

Observed: OBS crashed

Unable to load hugging face's CT2 model

I have tried every single Hugging Face CTranslate 2 models under 6GB and I always get the same error : "Failed to load CT2 model. This model cannot be used as a sequence to sequence model.". Once I even got "someexception.json needs to be defined" or smth.
Please help.

error parsing json

Hi, i have this error something when i try to use this plugin.

I use cloud closed captions to get captions and save them in a text source. Those are my settings:

That's polygon settings:

And those are url source settings:

This is the body. I didn't find it on the page so i write it from youtube:
{"text":"{{input}}", "source_lang":"eng_Latn","target_lang":"ita_Latn"}

To test if this works or not i enable the preview on captions and start to talk. You can ses on top captions text, on bottom the translation:

Japanese language support

Hello!

I'm not sure how doable this is as I'm not overly familiar with LASER3, but from what I can tell, there isn't an NLLB code on the provided list for Japanese or Chinese. Is there a way to translate those languages, or alternatively is there a chance the feature could be added?

Error 404 (CORS header ‘Access-Control-Allow-Origin’ missing) when calling /translate

Hello !

I want to build a streamelements widget that use OBS-Polyglot. I have test my request through insomnia, it get me a 200 OK with the translation but when i use this code on a widget streamelements, i've got a CORS Error.

Would it be possible to correct this issue ?
Thank you very much for all your hard work on Polyglot.

const options = {
  method: 'POST',
  headers: {'Content-Type': 'application/json'},
  body: '{"text":"Bonjour monde.","source_lang":"fr_Latn","target_lang":"eng_Latn"}'
};

fetch('http://127.0.0.1:18080/translate', options)
  .then(response => response.json())
  .then(response => console.log(response))
  .catch(err => console.error(err));

On the streamelements widget:

Build release request for GPU and Windows Powershell build script error question

Builds fail frequently in Windows 10.

It is a very convenient tool, but it takes up a lot of CPU space. As it approaches 90% on older PCs, there is a risk in broadcasting. It seemed like the cuda version was included in the recent pull request, so I tried to build it. It would be especially nice to have both versions 11.8 and 12. That's not the problem, you have to build it through a Powershell script on Windows, but when I used a light Windows tweak version, I couldn't resolve the Powershell and winget dependencies. (I'm sorry. I don't use the Windows Market Store because I think Windows is too heavy.) I would like to ask if it is possible to add a build for cuda gpu as a release version. I have little knowledge of Powershell cmake builds, and since I've been doing a lot of broadcasting work lately, it's difficult to have a full-fledged plug-in development environment.

There are two places where the problem occurred.

.github/scripts/Build-Windows.ps1 -Target x64 -CMakeGenerator "Visual Studio 17 2022"

This command says that there is no option called CMakeGenerator. I couldn't find it in the ps1 file, and I thought it was a problem with the development environment, so I installed the latest version of cmake msi and tried opening the Powershell window again with administrator privileges, but it didn't work.

Exclude the Cmake Generator option.
I tried adding only the two x64 Release options, but an error occurred because the winget dependency was not resolved as mentioned before in the Build function execution part.

The 0.0.2 release was recently updated, and I apologize as a former developer for asking this question.

However, from the perspective of OBS Lite users, it seems difficult to build and use the GPU version plugin directly.

My voice text extracted with the localvocal plugin

The goal is to display multi-language subtitles on the screen using polyglot, but the CPU usage is quite high.

The GPU previously used whisper-faster to edit subtitles after video editing, so I installed the nvidia 4070ti, which leaves graphics resources, but it is unfortunate that it runs only on the CPU without hardware acceleration.