GithubHelp home page GithubHelp logo

volcomix / virtual-background Goto Github PK

View Code? Open in Web Editor NEW
474.0 22.0 122.0 78.95 MB

Demo on adding virtual background to a live video stream in the browser

Home Page: https://volcomix.github.io/virtual-background

License: Apache License 2.0

HTML 2.60% TypeScript 91.91% Dockerfile 0.16% C++ 2.42% Starlark 2.44% Shell 0.47%
react video stream demo typescript tensorflow tfjs tflite wasm webgl

virtual-background's Introduction

Virtual Background

Demo on adding virtual background to a live video stream in the browser.

👉 Try it live here!

Table of contents

Implementation details

In this demo you can switch between 3 different ML pre-trained segmentation models:

BodyPix

The drawing utils provided in BodyPix are not optimized for the simple background image use case of this demo. That's why I haven't used toMask nor drawMask methods from the API to get a higher framerate.

The drawBokehEffect method from BodyPix API is not used. Instead, CanvasRenderingContext2D.filter property is configured with blur and CanvasRenderingContext2D.globalCompositeOperation is setup to blend the different layers according to the segmentation mask.

The result provides an interesting framerate on laptop but is not really usable on mobile (see Performance for more details). On both devices, the segmentation lacks precision compared to Meet segmentation model.

Note: BodyPix relies on the default TensorFlow.js backend for your device (i.e. webgl usually). The WASM backend seems to be slower for this model, at least on MacBook Pro.

MediaPipe Meet Segmentation

Meet segmentation model is only available as a TensorFlow Lite model file. Few approaches are discussed in this issue to convert and use it with TensorFlow.js but I decided to try implementing something closer to Google original approach described in this post. Hence the demo relies on a small WebAssembly tool built on top of TFLite along with XNNPACK delegate and SIMD support.

Note: Meet segmentation model card was initially released under Apache 2.0 license (read more here and here) but seems to be switched to Google Terms of Service since Jan 21, 2021. Not sure what it means for this demo. Here is a copy of the model card matching the model files used in this demo

Building TFLite to WebAssembly

You can find the source of the TFLite inference tool in the tflite directory of this repository. Instructions to build TFLite using Docker are described in a dedicated section: Building TensorFlow Lite tool.

Canvas 2D + CPU

This rendering pipeline is pretty much the same as for BodyPix. It relies on Canvas compositing properties to blend rendering layers according to the segmentation mask.

Interactions with TFLite inference tool are executed on CPU to convert from UInt8 to Float32 for the model input and to apply softmax on the model output.

WebGL 2

The WebGL 2 rendering pipeline relies entirely on webgl2 canvas context and GLSL shaders for:

  • Resizing inputs to fit the segmentation model (there are still CPU operations to copy from RGBA UInt8Array to RGB Float32Array in TFLite WASM memory).
  • Softmax on segmentation model output to get the probability of each pixel to be a person.
  • Joint bilateral filter to smooth the segmentation mask and to preserve edges from the original input frame (implementation based on MediaPipe repository).
  • Blending background image with light wrapping.
  • Original input frame background blur. Great articles here and here.

ML Kit Selfie Segmentation

Thanks to @RemarkableGuy for pointing out this model.

Selfie segmentation model's architecture is very close to the one of Meet segmentation and they both seem to be generated from the same Keras model (see this issue for more details). It is released under Apache 2.0 and you can find in this repo a copy of the model card matching the model used in this demo (here is the original current model card). The model was extracted from its official artifact.

Unlike what is described in the model card, the output of the model is a single channel allowing to get a float value of the segmentation mask. Besides that, the model is inferred using the exact same pipeline as Meet segmentation. However, the model does not perform as well as Meet segmentation because of its higher input resolution (see Performance for more details), even though it still offers better quality segmentation than BodyPix.

Performance

Here are the performance observed for the whole rendering pipelines, including inference and post-processing, when using the device camera on smartphone Pixel 3 (Chrome).

Model Input resolution Backend Pipeline FPS
BodyPix 640x360 WebGL Canvas 2D + CPU 11
ML Kit 256x256 WebAssembly Canvas 2D + CPU 9
ML Kit 256x256 WebAssembly WebGL 2 9
ML Kit 256x256 WebAssembly SIMD Canvas 2D + CPU 17
ML Kit 256x256 WebAssembly SIMD WebGL 2 19
Meet 256x144 WebAssembly Canvas 2D + CPU 14
Meet 256x144 WebAssembly WebGL 2 16
Meet 256x144 WebAssembly SIMD Canvas 2D + CPU 26
Meet 256x144 WebAssembly SIMD WebGL 2 31
Meet 160x96 WebAssembly Canvas 2D + CPU 29
Meet 160x96 WebAssembly WebGL 2 35
Meet 160x96 WebAssembly SIMD Canvas 2D + CPU 48
Meet 160x96 WebAssembly SIMD WebGL 2 60

Possible improvements

  • Rely on alpha channel to save texture fetches from the segmentation mask.
  • Blur the background image outside of the rendering loop and use it for light wrapping instead of the original background image. This should produce better rendering results for large light wrapping masks.
  • Optimize joint bilateral filter shader to prevent unnecessary variables, calculations and costly functions like exp.
  • Try separable approximation for joint bilateral filter.
  • Compute everything on lower source resolution (scaling down at the beginning of the pipeline).
  • Build TFLite and XNNPACK with multithreading support. Few configuration examples are in TensorFlow.js WASM backend.
  • Detect WASM features to load automatically the right TFLite WASM runtime. Inspirations could be taken from TensorFlow.js WASM backend which is based on GoogleChromeLabs/wasm-feature-detect.
  • Experiment with DeepLabv3+ and maybe retrain MobileNetv3-small model directly.

Related work

You can learn more about a pre-trained TensorFlow.js model in the BodyPix repository.

Here is a technical overview of background features in Google Meet which relies on:

Running locally

In the project directory, you can run:

yarn start

Runs the app in the development mode.
Open http://localhost:3000 to view it in the browser.

The page will reload if you make edits.
You will also see any lint errors in the console.

yarn test

Launches the test runner in the interactive watch mode.
See the section about running tests for more information.

yarn build

Builds the app for production to the build folder.
It correctly bundles React in production mode and optimizes the build for the best performance.

The build is minified and the filenames include the hashes.
Your app is ready to be deployed!

See the section about deployment for more information.

Building TensorFlow Lite tool

Docker is required to build TensorFlow Lite inference tool locally.

yarn build:tflite

Builds WASM functions that can infer Meet and ML Kit segmentation models. The TFLite tool is built both with and without SIMD support.

virtual-background's People

Contributors

volcomix avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

virtual-background's Issues

How to make it work in synchronous mode - wegGL2

this is outstanding work and cudos to OP! I wonder how we can make this work in synchronous mode, rather than asynchronous mode. I am pretty new to webgl stuff. Any insight will be greatly appreciated!

Change blur intensity in the WebGL2 pipeline

How can I decrease the blur intensity in the WebGL2 pipeline? Even manually by changing the code.Non of the controls in the demo seems to affect it.

What does the coverage parameter do? It seems the only parameter passed to BackgroundBlurStage but I don't see any effect in the result when changing it.

Failure to Build a particular version (2.5.0-rc3)

Hi,

I am trying to build version 2.5.0-rc3 and Docker file looks like this

FROM tensorflow/tensorflow:devel
RUN git clone https://github.com/google/mediapipe.git /mediapipe_src
RUN git -C /tensorflow_src checkout refs/tags/v2.5.0-rc3 -b v2.5.0-rc3
COPY entrypoint.sh /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]

And added the following line in entrypoint.sh

sed -i '/":tvos_arm64": COMMON_SRCS + MACH_SRCS + MACH_ARM_SRCS,/a ":emscripten_wasm": COMMON_SRCS + EMSCRIPTEN_SRCS,' /tensorflow_src/third_party/cpuinfo/BUILD.bazel

But I am getting this error:

ERROR: Analysis of target '//:tflite' failed; build aborted: /root/.cache/bazel/_bazel_root/00533da95dbcbd49bca844bdfd8d02a4/external/XNNPACK/BUILD.bazel:4433:26: Configurable attribute "deps" doesn't match this configuration (would a default condition help?).
Conditions checked:
 @XNNPACK//:linux_k8
 @XNNPACK//:linux_arm
 @XNNPACK//:linux_armeabi
 @XNNPACK//:linux_armhf
 @XNNPACK//:linux_armv7a
 @XNNPACK//:linux_aarch64
 @XNNPACK//:macos_x86_64
 @XNNPACK//:macos_arm64
 @XNNPACK//:windows_x86_64_clang
 @XNNPACK//:windows_x86_64_mingw
 @XNNPACK//:windows_x86_64_msys
 @XNNPACK//:windows_x86_64
 @XNNPACK//:android_armv7
 @XNNPACK//:android_arm64
 @XNNPACK//:android_x86
 @XNNPACK//:android_x86_64
 @XNNPACK//:ios_armv7
 @XNNPACK//:ios_arm64
 @XNNPACK//:ios_arm64e
 @XNNPACK//:ios_x86
 @XNNPACK//:ios_x86_64
 @XNNPACK//:watchos_armv7k
 @XNNPACK//:watchos_arm64_32
 @XNNPACK//:watchos_x86
 @XNNPACK//:watchos_x86_64
 @XNNPACK//:tvos_arm64
 @XNNPACK//:tvos_x86_64
 @XNNPACK//:emscripten_wasm
 @XNNPACK//:emscripten_wasmsimd
INFO: Elapsed time: 73.625s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (74 packages loaded, 1523 targets configured)
    Fetching @nodejs_linux_amd64; fetching 12s
    Fetching @emscripten_npm_linux; Restarting. 10s
    Fetching ...8d02a4/external/nodejs_linux_amd64/bin/nodejs; Extracting /root/.cache/bazel/_bazel_root/00533da95dbcbd49bca844bdfd8d02a4/external/nodejs_linux_amd64/bin/nodejs/temp1554428458845878867/node-v12.13.0-linux-x64.tar.xz 9s
error Command failed with exit code 1.

Can you please let me know what I am missing here?

Thanks,

Dipankar.

Huge memory consumption (meet+WebGL2)

First of all thanks for excellent solution. I was checking the demo site as well my implementation in Plug-N-Meet. If I choose meet + WebGL2 in Firefox it use huge memory. If I continue using like almost 20~25 minutes, it used up 12GB+ of RAM & browser got crashed. With meet + Canvas 2D + CPU doesn't have this problem. Is this normal or is there any memory leaks somewhere?

Incorrect response MIME type. Expected 'application/wasm'

Hello.
Nice work of yours! I've used some of your code in my demo in research purposes. Tested it and your demo on my old Linux Mint notebook and these errors appear when webgl processing is trying to start:
wasm streaming compile failed: TypeError: Failed to execute 'compile' on 'WebAssembly': Incorrect response MIME type. Expected 'application/wasm'.
and then as a result:
Uncaught (in promise) TypeError: Cannot read property 'VERTEX_SHADER' of null
OS is:

# lsb_release -a
No LSB modules are available.
Distributor ID:	LinuxMint
Description:	Linux Mint 19.3 Tricia
Release:	19.3
Codename:	tricia

Browser is:

# apt show chromium-browser
Package: chromium-browser
Version: 90.0.4430.93-0ubuntu0.18.04.1

chrome://gpu contains:

Graphics Feature Status
Canvas: Hardware accelerated
Compositing: Hardware accelerated
Multiple Raster Threads: Disabled
Out-of-process Rasterization: Hardware accelerated
OpenGL: Enabled
Rasterization: Hardware accelerated on all pages
Skia Renderer: Enabled
Video Decode: Hardware accelerated
Vulkan: Disabled
WebGL: Hardware accelerated
WebGL2: Hardware accelerated

Does my notebook have some hw or sw incompatibles with wasm?
Thanks.

Color Difference

There is a slight color difference between this and the regular Facetime HD Camera. Do you know what could be the cause of this?

change modelBuffer size in tflite.cc does not work

Hi @Volcomix , thx for the nice project.

I tried to build tflite and tflite-simd wasm with your code, the only thing I have changed is the size of modelBuffer here, as I want to try a float 32 model.

I modified it as char modelBuffer[1024 * 1024];. Thanks for your dockerfile, I can rebuild tflite-simd wasm successfully.

However, when I apply the model, this error appears

Uncaught (in promise) RuntimeError: memory access out of bounds
    at :3001/tflite/tflite-simd.wasm
    at :3001/tflite/tflite-simd.wasm
    at :3001/tflite/tflite-simd.wasm
    at Object.Module._loadModel (tflite-simd.js:9:14734)
    at loadTFLiteModel (bundle.js:558:21)
    at async getModel (bundle.js:39:17)

my model size is

[WASM] Loading model of size: 496468

which is much smaller than the modelBuffer I set 1024*1024 (=1048576)

Do I make any mistakes or miss something?

Virtual background not running in background

your sample is working fine when the browser tab is active, but when I move to another tabs this causes the requestanimationframe to stop. Can I know any workaround for this issue, to make it active, even when moving to another tabs. I have tried using setTimeout instead of request animation frame, video is rendering, but virtual background stops

How to convert custom tflite model to wasm

Hi,

Thank you for your great repo. Below is from the entrypoint.sh, I have a question about how to convert other tflite model to wasm instead from mediapipe, and what should I revise the command in the below. I am a complete novice in webassembly, Thank you so much.

#!/bin/sh
set -e

git -C /tensorflow_src pull --rebase
git -C /mediapipe_src pull --rebase

sed -i 's/"crosstool_top": "//external:android/emscripten"/"crosstool_top": "@emsdk//emscripten_toolchain:everything"/' /tensorflow_src/tensorflow/BUILD

cd tflite

bazel build --config=wasm -c opt :tflite
bazel build --config=wasm -c opt --copt='-msimd128' :tflite-simd

tar xvf bazel-bin/tflite -C ../public/tflite
tar xvf bazel-bin/tflite-simd -C ../public/tflite

question about light wrapping

Hi @Volcomix , thx for this nice work.

I'm interested in the light wrapping function of the post process pipeline and try to rewrite the following function it in python.

void main() {
vec3 frameColor = texture(u_inputFrame, v_texCoord).rgb;
vec3 backgroundColor = texture(u_background, v_backgroundCoord).rgb;
float personMask = texture(u_personMask, v_texCoord).a;
float lightWrapMask = 1.0 - max(0.0, personMask - u_coverage.y) / (1.0 - u_coverage.y);
vec3 lightWrap = u_lightWrapping * lightWrapMask * backgroundColor;
frameColor = u_blendMode * linearDodge(frameColor, lightWrap) +
(1.0 - u_blendMode) * screen(frameColor, lightWrap);
personMask = smoothstep(u_coverage.x, u_coverage.y, personMask);
outColor = vec4(frameColor * personMask + backgroundColor * (1.0 - personMask), 1.0);
}

however, the output result is different with the output of your original code...

I paste my python implementation here

def screen(a, b):
    return 1.0 - (1.0 - a) * (1.0 - b)


def linear_dodge(a, b):
    return a + b


def clamp(x, lowerlimit, upperlimit):
    if x < lowerlimit:
        x = lowerlimit
    if x > upperlimit:
        x = upperlimit
    return x


def smooth_step(edge0, edge1, x):
    # Scale, bias and saturate x to 0..1 range
    x = clamp((x - edge0) / (edge1 - edge0), 0.0, 1.0)
    # Evaluate polynomial
    return x * x * (3 - 2 * x)


def light_wrapping(segmentation_mask, image, background, cov_x=0.6, cov_y=0.8, wrap_cof=0.3, blend_mode='linear'):
    light_wrap_mask = 1. - np.maximum(0, (segmentation_mask - cov_y))/(1 - cov_y)
    light_wrap = wrap_cof * light_wrap_mask[:, :, np.newaxis] * background
    if blend_mode == 'linear':
        frame_color = linear_dodge(image, light_wrap)
    elif blend_mode == 'screen':
        frame_color = screen(image, light_wrap)

    smooth = lambda i: smooth_step(cov_x, cov_y, i)
    vectorized_smooth = np.vectorize(smooth)
    person_mask = vectorized_smooth(segmentation_mask)
    return person_mask, frame_color


def combind_frond_back_ground(person_mask, image, background):
    person_mask = person_mask[:, :, np.newaxis]
    output_image = image * mix_value + background * (1.0 - mix_value)
    output_image = output_image.astype(np.uint8)
    return output_image


def main():
    image_src = "path to input image"
    image = cv2.imread(image_src)   # (720, 1280, 3)
    background_src = "path to background image"
    background = cv2.imread(background_src)  # assuming image and background have the same size (720, 1280, 3)
    segmentation_mask = "segmentation result from model"  # (720, 1280)
    person_mask, frame_color = light_wrapping(segmentation_mask, image, background)
    output_frame = combind_frond_back_ground(person_mask, frame_color, background)
    return output_frame

I feel something might be wrong in smooth_step, but I cannot find more information except for this link.

Could you help me check if there is an obvious error in this python code please?

Thx in advance~

Can 'public/tflite' files be used in other projects? (tflite.wasm, tflite.js)

Hello, I have a question because there are unfamiliar files in the public/tflite path. (Example: tflite.wasm, tflite.js)
If I create a new react project with 'npx create-react-app' command, can I use 'import ... from ...' and use those files as they are?

Suggested improvement: background rendering

Awesome work here. One note, in practice the pipeline suffers from a problem that browsers won't render the canvas when the tab becomes inactive, which is pretty common in a meeting. One way to solve this is to move the rendering of the pipeline into something like an audio context loop instead of relying on animation frames.

function audioTimerLoop(callback, frequency) {

  // AudioContext time parameters are in seconds
  let freq = frequency / 1000;

  let aCtx = new AudioContext();
  // Chrome needs our oscillator node to be attached to the destination
  // So we create a silent Gain Node
  let silence = aCtx.createGain();
  silence.gain.value = 0;
  silence.connect(aCtx.destination);

  let osc;
  let stopped = false;
  
  onOSCend();

  function onOSCend() {
    osc = aCtx.createOscillator();
    osc.onended = onOSCend;
    osc.connect(silence);
    osc.start(0);
    osc.stop(aCtx.currentTime + freq);
    callback(aCtx.currentTime);
    if (stopped) {
      osc.onended = function() {
        return;
      };
    }
  };
  // return a function to stop our loop
  return function() {
    stopped = true;
  };
}```

RGBA Image as background

I want to experiment with rgba images as a background image. That way the rgba image would be over the camera except for the cutout person mask. I can't seem to get the fragment shader to work with rgba.

Any ideas?

Feature Request for Face Skin Smoothing

Hi @Volcomix, thank you for the amazing work! By any chance, have you considered applying "Beauty Filter" or some sort of "face recognition skin smoothing" filter for this? To mainly remove blemishes

Please let me know if you know of any web-based solution to achieve it. Face enhancement is important for some of our users.

Thanks again!

Problem when changing tabs with two users connected via WebRTC

I've integrated this demo into my WebRTC app where two users can be in a videoconference.
If I activate background replacement and change tab then the segmentation is stopped but the stream continue. Is it the intended behaviour, shouldn't the segmentation continue as well?

See the following video:
https://user-images.githubusercontent.com/90264844/226776846-c4e54a83-18be-4bc8-8325-32865d5c518a.mov

My implementation of timerHelper and timerWorker is identical to yours. The difference is in useRenderingPipeline, I removed hooks like useEffect/useState and manually create/destroy the pipeline as I am working on legacy code written in class component/typescript instead of functional component/react. However, I don't think it's the cause of the behavior.

deferInputResizing is true and targetFps equals 65. I am using the WebGL pipeline with MLkit/Meet (Not bodyPix)

I'm not sure if the same problem apply to your hosted demo, it is hard to diagnostic as a standalone user without a WebRTC connection to another user.

I am wondering if using an inaudible sound in background would solve this issue? Or any alternative.
(As mentionned here: #51)

Kind of related issues:
#29
#25

Does not work with safari browser

Works fine with Chrome and Firefox.

I am using Safari version 15.0.
Following errors occur in Safari browser on opening live demo:

[Warning] wasm streaming compile failed: CompileError: WebAssembly.Module doesn't parse at byte 5: can't get Function local's type in group 1, in function at index 19 (tflite-simd.js, line 9)

[Warning] falling back to ArrayBuffer instantiation (tflite-simd.js, line 9)

[Warning] failed to asynchronously prepare wasm: CompileError: WebAssembly.Module doesn't parse at byte 5: can't get Function local's type in group 1, in function at index 16 (tflite-simd.js, line 9)

[Warning] CompileError: WebAssembly.Module doesn't parse at byte 5: can't get Function local's type in group 1, in function at index 16 (tflite-simd.js, line 9)

[Warning] Failed to create TFLite SIMD WebAssembly module. – RuntimeError: abort(CompileError: WebAssembly.Module doesn't parse at byte 5: can't get Function local's type in group 1, in function at index 16). Build with -s ASSERTIONS=1 for more info. (main.0187bf97.chunk.js, line 1)

*RuntimeError: abort(CompileError: WebAssembly.Module doesn't parse at byte 5: can't get Function local's type in group 1, in function at index 16). Build with -s ASSERTIONS=1 for more info.
Screenshot 2022-02-02 at 6 56 49 PM
*

BodyPix tflite model support

@Volcomix , your work is really impressive! Thank you so much for this work.

I saw this comment from @PINTO0309 in this issue #2 (comment)

I don't know if it will be useful for you, but I have converted and quantized it for various frameworks and committed it to my repository.

TFLite Float32/Float16/INT8, TFJS, TF-TRT, ONNX, CoreML, OpenVINO IR FP32/FP16, Myriad Inference Blob

https://github.com/PINTO0309/PINTO_model_zoo
https://github.com/PINTO0309/PINTO_model_zoo/tree/main/109_Selfie_Segmentation

And I'm able to easily play with the different models. Thank you both for your hardwork.

I noticed @PINTO0309 also has bodypix tflite models here but they don't seem to work with @Volcomix pipeline. Getting some logs I noticed the output image has the following:

console.log({
  inputHeight: this._tflite._getInputHeight(),
  inputWidth: this._tflite._getInputWidth(),
  inputChannelCount: this._tflite._getInputChannelCount(),
  outputHeight: this._tflite._getOutputHeight(),
  outputWidth: this._tflite._getOutputWidth(),
  outputChannelCount: this._tflite._getOutputChannelCount(),
});
// Outputs an image with 10x8 resolution, 17 channels

As you can see, the output seems wrong. Any advice on what to adjust? My intention is I want to run bodypix model in wasm hoping to gain more performance than the tfjs one. Is this something you can help with? Thank you in advance!

Installing / adding to project

First of all @Volcomix , thanks for this awesome lib and demo!

My main questions are in terms of implementation - I've forked over the project and started integrating it with mine, as my end goal is to incorporate some of this BG removal functionality in my project.

I haven't had any experience with docker before, so my question is if its required to set up a docker project with the TFLite models in order to get this to work?

Also, in general - what would be the best way to integrate this into an existing project (vs cloning and running locally as a separate one).

Thanks!

Any way to do this via webworkers ?

Lately, I have been experimenting with putting this pipeline on a web-worker to make the experience better.
I have been struggling to send the tflite model to the worker.ts file.
Sending the canvas to the worker.ts was troublesome too but handled it using offscreen canvas.

I also believe putting it on web-worker would fix the issue of Video pausing when browser is in the background #25

The pipeline I have been trying this with is canvas2d using the meet model

Border color change

I'm trying to change the person border color and make it thicker.
Is there any way to do it?

some question about device

Hi, thanks for your excellent job. The web virtual-background is running only with user ? no server, no extra gpu?

Model replica of segm_full leads to wrong output memory buffer

Hey,

I created a keras model that replicates the segm_full google meet model and converted it to tflite.
Link to tflite

Inside python the model behaves exactly like the official one. But when I try to use it inside this repository output memory buffer offset is 0 and it crashes when running inference.

Any ideas on how to solve this? I assume there that the tflite conversion is different, but don't really know how to align it.

Here's the console output:

Model buffer size: 451572
useTFLite.ts:81 Model buffer memory offset: 122720
useTFLite.ts:82 Loading model buffer...
tflite-simd.js:9 [WASM] Loading model of size: 451572
useTFLite.ts:84 _loadModel result: 0
useTFLite.ts:89 Input memory offset: 13684800
useTFLite.ts:93 Input height: 144
useTFLite.ts:94 Input width: 256
useTFLite.ts:95 Input channels: 3
useTFLite.ts:97 Output memory offset: 0
useTFLite.ts:101 Output height: 144
useTFLite.ts:102 Output width: 256
useTFLite.ts:103 Output channels: 2
useRenderingPipeline.ts:97 Animation started: {htmlElement: img.makeStyles-sourcePlayback-7, width: 1280, height: 853} {type: 'image', url: '/virtual-background/backgrounds/architecture-5082700_1280.jpg'} {model: 'meet', backend: 'wasmSimd', inputResolution: '256x144', pipeline: 'webgl2', targetFps: 65, …}
useRenderingPipeline.ts:110 Animation stopped: {htmlElement: img.makeStyles-sourcePlayback-7, width: 1280, height: 853} {type: 'image', url: '/virtual-background/backgrounds/architecture-5082700_1280.jpg'} {model: 'meet', backend: 'wasmSimd', inputResolution: '256x144', pipeline: 'webgl2', targetFps: 65, …}
useRenderingPipeline.ts:97 Animation started: {htmlElement: img.makeStyles-sourcePlayback-7, width: 1280, height: 853} {type: 'image', url: '/virtual-background/backgrounds/architecture-5082700_1280.jpg'} {model: 'meet', backend: 'wasmSimd', inputResolution: '256x144', pipeline: 'webgl2', targetFps: 65, …}
tflite-simd.js:9 Aborted()

Thanks in advance!

Edit: I increased the memory buffer size already. That is not the issue. The model is also barely larger.

v4l2 loopback sink

Hello, is it possible to feed the output to other application as a virtual camera (subj). On linux all video conf software lacks virtual background.

Build failure

Hi,

When trying to build the wasm binaries I am seeing the following errors and the build fails when I run yarn build:tflite:all

INFO: Build option --crosstool_top has changed, discarding analysis cache.
Analyzing: target //:tflite (1 packages loaded, 0 targets configured)
Analyzing: target //:tflite (22 packages loaded, 120 targets configured)
Analyzing: target //:tflite (60 packages loaded, 1195 targets configured)
Analyzing: target //:tflite (72 packages loaded, 2053 targets configured)
ERROR: Traceback (most recent call last):
	File "/root/.cache/bazel/_bazel_root/7c8212d09d20008d9ab2f5e31ef362d8/external/cpuinfo/BUILD.bazel", line 131, column 27, in <toplevel>
		":emscripten_wasm": COMMON_SRCS + EMSCRIPTEN_SRCS,
Error: Duplicated key ":emscripten_wasm" when creating dictionary
ERROR: /root/.cache/bazel/_bazel_root/7c8212d09d20008d9ab2f5e31ef362d8/external/org_tensorflow/tensorflow/lite/kernels/BUILD:335:11: no such target '@cpuinfo//:cpuinfo_with_unstripped_include_path': target 'cpuinfo_with_unstripped_include_path' not declared in package '' defined by /root/.cache/bazel/_bazel_root/7c8212d09d20008d9ab2f5e31ef362d8/external/cpuinfo/BUILD.bazel and referenced by '@org_tensorflow//tensorflow/lite/kernels:cpu_backend_context'
INFO: Repository emscripten_bin_linux instantiated at:
  /tflite_src/WORKSPACE:56:22: in <toplevel>
  /root/.cache/bazel/_bazel_root/7c8212d09d20008d9ab2f5e31ef362d8/external/emsdk/emscripten_deps.bzl:28:21: in emscripten_deps
Repository rule http_archive defined at:
  /root/.cache/bazel/_bazel_root/7c8212d09d20008d9ab2f5e31ef362d8/external/bazel_tools/tools/build_defs/repo/http.bzl:336:31: in <toplevel>
ERROR: Analysis of target '//:tflite' failed; build aborted: Analysis failed
INFO: Elapsed time: 5.551s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (77 packages loaded, 2200 targets configured)
FAILED: Build did NOT complete successfully (77 packages loaded, 2200 targets configured)

Can you please help?

Thanks,

Dipankar.

Wrong path for tflite.wasm

I'm trying to add the virtual background to an angular 11 project and getting an error when trying to load the tflite model:

GET http://localhost:4200/tflite.wasm 404 (Not Found)
tflite.js:9 wasm streaming compile failed: TypeError: Failed to execute 'compile' on 'WebAssembly': HTTP status code is not ok
tflite.js:9 falling back to ArrayBuffer instantiation
GET http://localhost:4200/tflite.wasm 404 (Not Found)
failed to asynchronously prepare wasm: RuntimeError: abort(both async and sync fetching of the wasm failed). Build with -s ASSERTIONS=1 for more info.

This is how I import tflite.js:

import createTFLiteModule from './tflite/tflite';
import createTFLiteSIMDModule from './tflite/tflite-simd';

The path to tflite.wasm is hardcoded in tflite/tflite.js. Is there a config option to set the path without modifying tflite.js?

var wasmBinaryFile="tflite.wasm"

Better performance in bigger video Width

Very useful integration. Congrats on your implementation.

My question is:
Is there a way to increase the performance in bigger video size. I saw that right now it uses 640px video size. When you resize the video to 1980px the performance drops drastically.

So is there a way to bypass this issue .
Thanks in advance.

Mediapipe webassembly

First of all thank you for the efforts you put on to this project. I have gained insights on Tflite-wasm-simd from your project.
I am aware that this question might not be totally relevant to this project, but have you ever tried compiling Mediapipe to WASM from their entire source?

Can't resolve 'fs' and 'path'

I'm trying to add the virtual background to an angular 11 app. When calling 'ng serve' I'm getting the following errors:

Error: ./src/tflite/tflite.js
Module not found: Error: Can't resolve 'fs' in 'src/tflite'

Error: ./src/tflite/tflite.js
Module not found: Error: Can't resolve 'path' in 'src/tflite'

Adding this to package.json resolve the issue. Is there a way to fix the errors without changing package.json or other config files?

"browser": {
    "fs": false,
    "path": false
},

ML Kit Selfie segmentation model

Hi @Volcomix, I am really impressed with your work here! I have been following the work that has been done by multiple parties on this topic but have gotten stuck on the licensing issue surrounding the Google Meet model, until I came across this:
https://developers.google.com/ml-kit/vision/selfie-segmentation

This seems very similar to Google Meet's model at first glance, and when taking a look at the model card, it actually is licensed under Apache 2.0!

Doing some further digging I was able to extract the .tflite file from the Android package without issue:
selfiesegmentation_mlkit-256x256-2021_01_19-v1215.f16.tflite.zip

After that I did an inspection of both the Google Meet model you have in the repo and this new one using https://netron.app/:
Google Meet
segm_full_v679 tflite

ML Kit Selfie segmentation
selfiesegmentation_mlkit-256x256-2021_01_19-v1215 f16 tflite

As you can see their structures are practically identical, the only difference I see is the input and output sizes, the new model using 256x256.

I have tried cloning your repo and loading this model with it by putting it in the public folder with the other models and changing the path used to load it, it seems to load correctly but there is no output visible except a very faint, perhaps stretched outline of the person detected. This leads me to believe using this model with your implementation should be possible, perhaps requiring some adjustment due to the different input/output resolutions (memory offset?), curious to hear your thoughts on this 😄.

Check if simd is supported

A function that check if simd is supported. Based on wasm-check.

Can be used here instead of try/catch.

private simdSupported(): boolean {
  const u32 = (...bytes: number[]) => Uint32Array.of(0x6D736100, 1, ...bytes);
  const u32a = (...bytes: number[]) => u32(1610679297, 33751040, ...bytes, 40239360, 259);

  let wasm = u32a(84344833, 6357249, 17369600, 4259847, 186257917, 1845758464);
  const buffer = wasm.buffer;
  if (WebAssembly.validate(buffer)) {
    try {
      new WebAssembly.Instance(new WebAssembly.Module(buffer)).exports['0']();
      return true;
    }
    catch (_a) {
      return false;
    }
  }
  return false;
}

TFLite fails to build in the Docker container

Hi, I am trying to run your demo on my Mac and when I run yarn build:tflite:all and bazel tries to build tflite, it fails with this error :

$ docker exec -w /tflite_src tflite bazel build --config=wasm -c opt :tflite
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
Loading:
Loading: 0 packages loaded
ERROR: error loading package '': cannot load '@org_tensorflow//tensorflow:workspace.bzl': no such file
INFO: Elapsed time: 4.869s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded)
FAILED: Build did NOT complete successfully (0 packages loaded)
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

I would guess I have to change something in the /tflite_src/WORKSPACE file but I don't know how bazel works yet. I will try to fix the issue and will comment again if I find how to get rid of the error.

TFLite WASM build failing after XNNPACK update

XNNPACK has recently been updated with the latest versions of wasm_simd128.h intrinsics.

As those intrinsics are not yet included in the version of LLVM used by emsdk, TFLite WASM build is failing.

tfjs doesn't have this issue yet when building the wasm backend because it relies on an old version of XNNPACK but I guess it should face the same issue as soon as XNNPACK version will be updated.

Here are some details about the error we have:

[177 / 1,157] Compiling XNNPACK/src/f32-avgpool/9p8x-minmax-wasm-c1.c; 2s processwrapper-sandbox ... (2 actions running)
ERROR: /github/home/.cache/bazel/_bazel_root/0a7049196223eb41bb90aa2e8797b78e/external/XNNPACK/BUILD.bazel:4672:19: C++ compilation of rule '@XNNPACK//:wasm_ukernels' failed (Exit 1): emcc.sh failed: error executing command external/emsdk/emscripten_toolchain/emcc.sh '--sysroot=external/emscripten_bin_linux/emscripten/cache/sysroot' -fdiagnostics-color -fno-strict-aliasing -funsigned-char -no-canonical-prefixes -DNDEBUG ... (remaining 66 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox emcc.sh failed: error executing command external/emsdk/emscripten_toolchain/emcc.sh '--sysroot=external/emscripten_bin_linux/emscripten/cache/sysroot' -fdiagnostics-color -fno-strict-aliasing -funsigned-char -no-canonical-prefixes -DNDEBUG ... (remaining 66 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox
external/XNNPACK/src/x32-pad/wasmsimd.c:32:24: error: implicit declaration of function 'wasm_v128_load32_splat' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
  const v128_t vfill = wasm_v128_load32_splat(fill_value);
                       ^
external/XNNPACK/src/x32-pad/wasmsimd.c:32:24: note: did you mean 'wasm_v16x8_load_splat'?
external/emscripten_bin_linux/lib/clang/13.0.0/include/wasm_simd128.h:70:1: note: 'wasm_v16x8_load_splat' declared here
wasm_v16x8_load_splat(const void *__mem) {
^
external/XNNPACK/src/x32-pad/wasmsimd.c:32:16: error: initializing 'const v128_t' (vector of 4 'int32_t' values) with an expression of incompatible type 'int'
  const v128_t vfill = wasm_v128_load32_splat(fill_value);
               ^       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2 errors generated.
emcc: error: '/github/home/.cache/bazel/_bazel_root/0a7049196223eb41bb90aa2e8797b78e/sandbox/processwrapper-sandbox/1173/execroot/__main__/external/emscripten_bin_linux/bin/clang -DEMSCRIPTEN -fignore-exceptions -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr -Xclang -iwithsysroot/include/SDL -target wasm32-unknown-emscripten -D__EMSCRIPTEN_major__=2 -D__EMSCRIPTEN_minor__=0 -D__EMSCRIPTEN_tiny__=14 -D_LIBCPP_ABI_VERSION=2 -Dunix -D__unix -D__unix__ -flegacy-pass-manager -Werror=implicit-function-declaration --sysroot=/github/home/.cache/bazel/_bazel_root/0a7049196223eb41bb90aa2e8797b78e/external/emscripten_bin_linux/emscripten/cache/sysroot -Xclang -iwithsysroot/include/compat --sysroot=external/emscripten_bin_linux/emscripten/cache/sysroot -fdiagnostics-color -fno-strict-aliasing -funsigned-char -no-canonical-prefixes -DNDEBUG -fomit-frame-pointer -O3 -Wall -DPTHREADPOOL_NO_DEPRECATED_API -iquote external/XNNPACK -iquote bazel-out/wasm-opt/bin/external/XNNPACK -iquote external/FP16 -iquote bazel-out/wasm-opt/bin/external/FP16 -iquote external/FXdiv -iquote bazel-out/wasm-opt/bin/external/FXdiv -iquote external/pthreadpool -iquote bazel-out/wasm-opt/bin/external/pthreadpool -Ibazel-out/wasm-opt/bin/external/FP16/_virtual_includes/FP16 -Ibazel-out/wasm-opt/bin/external/FXdiv/_virtual_includes/FXdiv -Ibazel-out/wasm-opt/bin/external/pthreadpool/_virtual_includes/pthreadpool -isystem external/XNNPACK/include -isystem bazel-out/wasm-opt/bin/external/XNNPACK/include -isystem external/XNNPACK/src -isystem bazel-out/wasm-opt/bin/external/XNNPACK/src -isystem external/FP16/include -isystem bazel-out/wasm-opt/bin/external/FP16/include -isystem external/FXdiv/include -isystem bazel-out/wasm-opt/bin/external/FXdiv/include -isystem external/pthreadpool/include -isystem bazel-out/wasm-opt/bin/external/pthreadpool/include -Wno-error=unused-function -msimd128 -Iinclude -Isrc -std=c99 -O2 -iwithsysroot/include/c++/v1 -iwithsysroot/include/compat -iwithsysroot/include -isystem external/emscripten_bin_linux/lib/clang/13.0.0/include -MD -MF bazel-out/wasm-opt/bin/external/XNNPACK/_objs/wasm_ukernels/1/wasmsimd.d -c -Wno-builtin-macro-redefined -D__DATE__="redacted" -D__TIMESTAMP__="redacted" -D__TIME__="redacted" -Werror external/XNNPACK/src/x32-pad/wasmsimd.c -o bazel-out/wasm-opt/bin/external/XNNPACK/_objs/wasm_ukernels/1/wasmsimd.o' failed (1)
Target //:tflite-simd failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 59.927s, Critical Path: 8.59s
INFO: 52 processes: 6 internal, 46 processwrapper-sandbox.
FAILED: Build did NOT complete successfully
FAILED: Build did NOT complete successfully

Issue created in XNNPACK repository: google/XNNPACK#1630

tflite.js and tflite-simd.js size increased

I've noticed that the size of the tflite files increased significantly after ae6db18
Is there a reason for such increase for the same functionality?

public/tflite/tflite.js
Before = 16.4 KB
After = 72 KB

public/tflite/tflite-simd.wasm
Before = 2.51 MB
After = 2.81 MB

Mabe because --features=wasm_simd was added but -c opt --copt='-msimd128' wasn't removed.
fae12a2
google/XNNPACK@3220551

Cannot Build TFLite Image

Running yarn build:tflite on the latest version of the repository results in the following error:

ERROR: /workspace/tflite/BUILD:25:10: Linking tflite-simd failed: (Exit 1): emcc_link.sh failed: error executing command external/emsdk/emscripten_toolchain/emcc_link.sh @bazel-out/wasm-opt/bin/tflite-simd-2.params

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
stderr (/root/.cache/bazel/_bazel_root/00533da95dbcbd49bca844bdfd8d02a4/execroot/__main__/bazel-out/_tmp/actions/stderr-3561) exceeds maximum size of --experimental_ui_max_stdouterr_bytes=1048576 bytes; skipping
Target //:tflite-simd failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 227.455s, Critical Path: 29.53s
INFO: 1833 processes: 6 internal, 1827 processwrapper-sandbox.
FAILED: Build did NOT complete successfully
error Command failed with exit code 1.

Segment Anything Version

Any thoughts on using Segment Anything? Seems like it might be a good fit to replace the Google models.

Standalone tflite module release?

Hey there!

I was wondering if you'd consider releasing the bundled tflie module as standalone in for example. This would make it easier for those using to get updates, instead of having to vendor the files and check in here every once in a while to see the improvements you've made :-)

Thanks again for a great project!

Regression: SIMD support

Thanks for sharing this amazing project!

I just noticed that SIMD support seems to be broken now. See this console.warn messages
image

Not sure if this is related to recent commits updating the tflite version numbers?

Does not work on ipad

The applied on my website. However, it does not work on ipad. I have check your demo, it does not work on ipad too. Could you please help me to figure out the problem

Working on Safari Browser

@Volcomix Thank you for nice work . I have a question "Whether this web app is supported on the Safari Browser?" . I have tested on the safari browser but it is not working . Thank you

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.