occ-ai / obs-cleanstream

CleanStream is an OBS plugin that uses AI to clean live audio streams from unwanted words and utterances

obs-cleanstream

CleanStream is an OBS plugin that cleans live audio streams from unwanted words and utterances using real-time local AI.

If this free plugin has been valuable to you consider adding a ⭐ to this GH repo, rating it on OBS, subscribing to my YouTube channel where I post updates, and supporting my work on GitHub or Patreon 🙏

Check out our other plugins:

• Background Removal remove background (virtual green screen) from video
• Detect will detect and track >80 types of objects in any OBS source
• URL/API Source fetch API data and display it on screen as a video source
• LocalVocal speech AI assistant plugin for real-time transcription (captions), translation and more language functions
• PolyGlot a realtime local translation service based on AI.

Usage

• Add the plugin to any audio-generating source
• Adjust the settings

Download

Check out the latest releases for downloads and install instructions.

Method

This video walkthrough (YouTube) will explain various parts of the code if you're looking to learn from what I've discovered.

Audio processing

The filter is running Whisper in real-time to detect words in small chunks of the incoming audio. For each chunck it produces a decision which then determines if the audio rendering will play the original audio or e.g. a beep or silence. The processing happens in a separate thread and therefore there's a built-in lag/delay mechanism to make sure the audio decision (play, beep, silence) is in-sync with the actual audio playback based on the timestamp. The built-in delay is adaptive since some systems (e.g. with CUDA) can make faster decisions.

Here is an illustration of the process:

Requirements

• OBS version 30+ for plugin versions 0.0.4+
• OBS version 29 for plugin versions 0.0.2+
• OBS version 28 for plugin versions 0.0.1

We do not support older versions of OBS since the plugin is using newer APIs.

Introduction

CleanStream is an OBS plugin that cleans live audio streams from unwanted words and utterances, such as "uh"s and "um"s, and other words that you can configure, like profanity.

See our resource on the OBS Forums for additional information.

It is using a neural network (OpenAI Whisper) to predict in real time the speech and remove the unwanted words.

It's using the Whisper.cpp project from ggerganov to run the Whisper network in a very efficient way.

But it is working and you can try it out. Please report any issues you find. 🙏 (submit an issue or meet us on https://discord.gg/KbjGU2vvUz)

We're working on improving the plugin and adding more features. If you have any ideas or suggestions, please open an issue.

Building

The plugin was built and tested on Mac OSX, Windows and Ubuntu Linux. Help is appreciated in building on other OSs and packages.

The building pipelines in CI take care of the heavy lifting. Use them in order to build the plugin locally.

Start by cloning this repo to a directory of your choice.

Mac OSX

Using the CI pipeline scripts, locally you would just call the zsh script.

$ ./.github/scripts/build-macos.zsh -c Release -t macos-x86_64

Install

The above script should succeed and the plugin files will reside in the ./release folder off of the root. Copy the files to the OBS directory e.g. /Users/you/Library/Application Support/obs-studio/obs-plugins.

To get .pkg installer file, run

$ ./.github/scripts/package-macos.zsh -c Release -t macos-x86_64

(Note that maybe the outputs in the e.g. build_x86_64 will be in the Release folder and not the install folder like pakage-macos.zsh expects, so you will need to rename the folder from build_x86_64/Release to build_x86_64/install)

Linux (Ubuntu-ish)

Use the CI scripts again

$ ./.github/scripts/build-linux.sh

Windows

Use the CI scripts again, for example:

> .github/scripts/Build-Windows.ps1 -Configuration Release

The build should exist in the ./release folder off the root. You can manually install the files in the OBS directory.

> Copy-Item -Recurse -Force "release\Release\*" -Destination "C:\Program Files\obs-studio\"

Building with CUDA support on Windows

CleanStream will now build with CUDA support automatically through a prebuilt binary of Whisper.cpp from https://github.com/occ-ai/occ-ai-dep-whispercpp. The CMake scripts will download all necessary files.

To build with cuda add CPU_OR_CUDA as an environment variable (with cpu, 12.2.0 or 11.8.0) and build regularly

> $env:CPU_OR_CUDA="12.2.0"
> .github/scripts/Build-Windows.ps1 -Configuration Release

Star History

obs-cleanstream's Issues

How to use in other Language?

I don't speak English,however the plugin looks only working on English.
It don't works when I speak Chinese or Japanease.(I changed the Language setting)

Or it needs I input some words in ‘Detect regex’ ?
Chinese and Japanease don't use alphabate, please tell me how to input,Thanks!

And I also find that if input a '|' after the last word in ‘Detect regex’ ,the plugin will indiscriminately attack all words even if they're not swear words.

OBS Cleanstream crashing OBS when adding it to mic

Adding OBS-Cleanstream to my mic force closes OBS

I've been trying to add this plugin to my mic on OBS 29.1.3 and each time i tried to do it, it keeps for closing my OBS and never wants to have it added, but when i tried adding the filter to a media source like a video, it works perfectly fine. My pc is running an intel i7-9750H with an Nvidia GeForce RTX 2060, 12GB of RAM, and on a 64-bit operating system with Windows 10.

To Reproduce

Steps to reproduce the behavior:

1. Go to the github page and download the installer.exe
2. Open OBS after installation of the plugin with the installer
3. Go into OBS settings and in the audio tab, Set Sample Rate to 48kHz and Channels to Mono
4. Go into Mic filters, add Expander, Gain, 3-Band Equalizer, Compressor, Noise Suppression, Limiter, and then the clean stream audio filter
5. See error

Expected behavior

OBS will freeze for about 10 seconds and suddenly force close itself whenever you try to add the source to anything that reads as a audio input device

Log and Crash Report

2023-10-15 07-39-57.txt
2023-10-15 07-39-08.txt

Desktop (please complete the following information):

• OS: Windows 10 64-bit Version 22H2
• Browser: Opera GX
• Plugin Version: 0.0.3
• OBS Version: 29.1.3

Additional context

It only keeps crashing whenever I try to add the filter to a mic source. Any other audio source that isn't a microphone will work perfectly fine

Add GPU support

Using the GPU support from whisper.cpp will be crucial for real-time audio stream processing
It's already supported upstream for CUDA and CoreML: https://github.com/ggerganov/whisper.cpp
OpenVINO is also planned

English-ify Options

Describe the bug

I am not sure if this is part of how the plugin is built but these options needs to be translated to English.

To Reproduce
No reproduction for this one.

Expected behavior

These options are suppose to show in English.

Screenshots

Log and Crash Report

No crashing here

Desktop (please complete the following information):

• OS: Windows 10 Pro 64bit
• Browser: LibreWolf
• Plugin Version: 0.0.3
• OBS Version: 29.1.3

Allow different sounds for words

Add a way to select a custom sound per word(s) found

installed plugin says no properties available

I don't see any options to choose in obs it just says no properties available.

running latest version of obs and MacBook m1

if im missing something can you please explain what im missing or files?

which one for AMD

I am newbie and please tell me which one i can download it for AMD?

• obs-cleanstream-O.O.7-windows-x64-clblast-lnstaller.exe
• obs-cleanstream-O.O.7-windows-x64-clblast.zip
• obs-cleanstream-O.O.7-windows-x64-cpu-lnstaller.exe
• obs-cleanstream-O.O.7-windows-x64-cpu.zip

Add word-level timestamps to segment words in question

Right now I cut or beep the entire buffer, but what if the word in question starts and doesn't end in the buffer, or it doesn't start in the buffer - bits of the word bleed into the next or the last buffer.
Adding exact word timestamps will allow to handle this.

OBS crash on adding filter

Adding the filter to any source crashes OBS. Here is the crash log:

Crash 2023-07-29 15-59-46.txt

OBS crashes after adding Cleanstream filter to a source

OBS crashes after adding the clean stream filter to a source. First crash was after adding the source to a microphone that was added as a Source Clone (from the plugin of that name).

The second crash was when I tried to add it as a filter to a microphone as a normal audio input capture source.

To Reproduce

Steps to reproduce the behavior:

1. Select source
2. Select Filters
3. Add Cleanstream from the add filter menu.
4. After a couple of seconds, OBS crashes.

Expected behavior: Add the filter and use it normally.

Log and Crash Report

LOG:
2023-06-19 17-09-33.txt

Crashes
Crash 2023-06-19 16-36-52.txt
Crash 2023-06-19 16-51-41.txt

• OS: Windows 10 22H2
• Plugin Version: v0.0.1a
• OBS Version: 29.1.2

Explore the VOSK ASR model

https://alphacephei.com/vosk/models

The "tiny" model is about 40Mb

Unable to get the plugin to show in audio filters............

I am using this on Linux Mint 21.2 which is a derivative of Ubuntu. Obs-Studio is installed thru the repo and I have teh plugin inn the same folder for all plugins (which I have obs-backgroundremoval installed and working).

The plugins reside in /home/user/.config/obs-studio/plugins and it is also installed where the .deb file installed it.

I have no cleanstream in the audio filters. I have only Video Delay (async) in the audio filters.

Any ideas?

Plugin Live Test

Describe the bug

This video shows some tests I have done using the plugin. You can hear some oddities.

To Reproduce

Steps to reproduce the behavior:

1. Install the plugin
2. Add the plugin on the Mic Source Filters
3. Make sure all settings are on default.
4. Set Monitoring in Advanced Audio Properties to "Monitor only"

Expected behavior

I wanted a seamless and no fuzz outcome.

Screenshots

Recording of said test [Had to cut the clip short due to 10MB limitations :( ]
https://github.com/royshil/obs-cleanstream/assets/37266538/27d5d605-75c8-4159-82c0-26d1d2099d1e

Log and Crash Report

No crashes occurred here :D

Desktop (please complete the following information):

• OS: Windows 10 Pro 64 bit
• Browser: LibreWolf
• Plugin Version: 0.0.3
• OBS Version: 29.1.3

Additional context

I thought I could deliver some form of testing material to hopefully rat out some bugs and/or glitches.