hamidzr / webcam-mods

Face tracking, bg removal, crop, zoom, record & replay, and more webcam mods

Webcam Mods

Tested on Arch Linux.

Checkout my other repository for some ffmpeg-only solutions here

Find installation and a work-in-progress demos here:

• Latest demo https://youtu.be/FfD7lu_A1Dw (recorded on commit 8f08bc68 on Jan 2022). It showcases some of the main features.
• Archived demo https://youtu.be/idp7ei-pF40 (recorded using version cf765)

Included Mods

Face Tracking

Setup your webcam to focus and follow your face by cropping and resizing the frames it receives from your main webcam.

Person Segmentation

Separate the people in the frame from the background using a fast real-time prediction model. The model outputs a mask values between 0 to 1. We have mods based on this to swap the background with:

• a solid color
• another image
• blurred version of the input frame (aka blur my background)

Cropping

Interactively move your camera around with arrow keys ctrl+arrowkeys Resize the cropped frame using ctrl+shift+arrowkeys You can disable this control by defining the environment variable PAN_CONTROL=False.

Padding

Interactively pad your camera output with arrow keys alt+arrowkeys while keeping the output framesize fixed. You can disable this control by defining the environment variable PADDING_CONTROL=False.

Record & Replay

Record and replay your camera feed on the fly. While you're in any of the other modes above press r to start recording and press p to stop recording and start replaying in a loop.

For entertainment purposes only

Installation

Depending on your python setup you might need to include the current directory in your $PYTHONPATH. To do so run the following: export PYTHONPATH="$PYTHONPATH:./"

Dependencies

System dependencies:

• Python 3.8: Some installation options include pyenv conda
• Git
• A virtual camera device: [Linux] v4l2loopback [Windows or MacOS] OBS. Follow pyvirtualcam's instructions to set this up.

If you're just interested in running the released features install the project as a python package using: pip install git+https://github.com/hamidzr/webcam-mods@master (python 3.8 environment) and access the offered features using webcam_mods CLI. This would replace the entry.py mentions in the rest of the documentation.

Python dependencies are listed in Pipfile. Install them using Pipenv (recommended)

[WARN] If you don't use pipenv for dependency and virtual env management you'd need to find replacements wherever you see pipenv mentioned => grep -R pipenv .

1. create a virtual environment: pipenv --python 3.8
2. activate it pipenv shell
3. install the dependencies pipenv install --skip-lock

Setting up a virtual webcam device on Linux

On Linux once you have the v4l2 module installed you can run sudo make add-video-dev to add a virtual camera device with some pre-set flags.

Which executes the following to remove and re-insert the module. You might need root access for this.

pkill gst-launch &> /dev/null || true
rmmod v4l2loopback &> /dev/null || true
modprobe v4l2loopback devices=1 max_buffers=2 exclusive_caps=1 video_nr=10 card_label="v4l2-cam"

Upgrading

If you run into an issue upgrading try removing the old config file at .webcam.conf

Running the Mods

After you've successfully followed installation steps, you can run the different modes by calling python webcam_mods/entry.py --help from within the src directory .

Settings

When you use the interactive controls to move the camera around the resulting parameters are saved in a text file to your disk which is by default located at $HOME/.webcam-mods.conf

Environment Variables

Environment variables are used to configure different parameters. Read more about how to set or persist them here These are mostly defined in the config.py file. To see their default values take a look at that file.

• VIDEO_IN & VIDEO_OUT: If you have multiple video input devices, aka webcams, you can pick the one you want by providing its index through by setting the VIDEO_IN environment variable. eg export VIDEO_IN=0. Same if you have multiple output devices.

• MAX_OUT_FPS: [Default: 30] set an upper limit for output FPS.

• IN_WIDTH [Default: 640], IN_HEIGHT [Default: 480]: Your video input device likely support multiple resolution and FPS settings use these env variables to pick and persist the one you want. v4l2-ctl can list out the different settings your webcam driver supports: v4l2-ctl --list-formats-ext | less

• OUT_WIDTH [Default: 640], OUT_HEIGHT [Default: 480]: similar to IN_HEIGHT and OUT_HEIGHT but for your output device.

• ON_DEMAND [Default: False, Linux only]: set to True to lower cpu usage while the output camera device isn't actively used.

• IN_FORMAT: input video format. This dictates the requested video format from the input video device (webcam) which directly affects picture quality and FPS. If you're looking to get higher a resolution or FPS out of your webcam it's crucial to inspect your camera and driver capabilities and set the appropriate format here.

• PAN_CONTROL [Default True]: Set to False to disable panning/resizing with ctrl+arrowkeys/ctrl+shift+arrowkeys.

• PADDING_CONTROL [Default True]: Set to False to disable padding with alt+arrowkeys.

TODO

house cleaning:

• clean and reorganize the code
• set up a code formatter
• set up a language server for development with Vim and VSCode
• replace the facetracking model with mediapipe
• move the config file to $XDG_CONFIG_HOME
• cli support for settings currently supported by env variables

features:

• more stable edges for person segmentation
• support other video feed formats from webcam eg mjpeg, h264 for higher resolution
• including headphones in the mask
• visualize interactive camera control settings
• zoom support. done through resizing.
  • the controls could be more intuitive
• MacOS support
  • disable ionotify. quartz install
• Windows support? should be there with pyvirtualcam
• hot swap inputs
• [~] add screen as an input
• convert/migrate env variables to cli arguments
• brightness control. (and hue, saturation?)
• [~] smooth bounding box tracking (for facetracking and more)
  • camera/crop size change transition
• overlay on top of video

bugs:

• bug what?

a demo video showcasing the features

Contact

Are you interested in helping improve this tool (hint: look at the TODO section)? Are you looking for a specific feature, or have you found a bug? Use GitHub Issues to reach out to me.

Credits

• Google/mediapipe for their selfie segmentation model.
• fangfufu/Linux-Fake-Background-Webcam For mask post processing and automatic ondemand pause and restart.
• letmaik/pyvirtualcam

webcam-mods's Issues

add support for python 3.8

all the dependencies seem compatible with 3.8 but onnx which is used for facetracking.
3.8 comes default with ubuntu 20

stopping the service can lead to corrupted config that persist

and it shows up as not so easy to debug issues.

#13 (comment)

Unable to change IN_FPS

Thanks for this amazing tool.

I'm able to use only with 10 FPS. How can I increate that value to 30?

I noticed the config.py file with IN_FPS env var, but even with that, I'm not able to go with 30.

This is what I can see in the console when I run it:

2023-11-13 18:29:10.000 | INFO     | webcam_mods.loopback:live_loop:56 - begin passing from #Webcam to #V4l2Cam
2023-11-13 18:29:10.023 | INFO     | webcam_mods.loopback:live_loop:67 - input: {'width': 1280, 'height': 720, 'fps': 10.0}, output: {'device': '/dev/video10', 'width': 1280, 'height': 720, 'fps': 10.0}

The command I'm using is:

export VIDEO_IN=4
export IN_WIDTH=1280
export IN_HEIGHT=720
export OUT_WIDTH=1280
export OUT_HEIGHT=720
export IN_FORMAT=I420
export MAX_OUT_FPS=30
export IN_FPS=30

$HOME/.local/bin/webcam_mods bg-swap --img-path path-to-my-img.jpg

PS: Already tried with formats: I420, JPEG and XRBG without success.

Thanks in advance.

make crop-cam thows a 'std::exception'

Hope you can help.
I have followed the instruction video but when i run the make crop-cam command i am getting the following error:

pipenv run python entry.py crop-cam
starting with config {'crop_dims': [640, 480], 'crop_pos': [0, 0], 'pad_size': [0, 0]}
config file not found at .webcam.conf
begin passing from #Webcam to #VirtualCam
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
Traceback (most recent call last):
  File "entry.py", line 133, in <module>
    app()
  File "/home/rick/.local/share/virtualenvs/webcam-mods-wTpII5o_/lib/python3.8/site-packages/typer/main.py", line 214, in __call__
    return get_command(self)(*args, **kwargs)
  File "/home/rick/.local/share/virtualenvs/webcam-mods-wTpII5o_/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/home/rick/.local/share/virtualenvs/webcam-mods-wTpII5o_/lib/python3.8/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/home/rick/.local/share/virtualenvs/webcam-mods-wTpII5o_/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/rick/.local/share/virtualenvs/webcam-mods-wTpII5o_/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/rick/.local/share/virtualenvs/webcam-mods-wTpII5o_/lib/python3.8/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/home/rick/.local/share/virtualenvs/webcam-mods-wTpII5o_/lib/python3.8/site-packages/typer/main.py", line 500, in wrapper
    return callback(**use_params)  # type: ignore
  File "entry.py", line 48, in crop_cam
    live_loop(base_mod)
  File "/home/rick/Desktop/tmp/webcam-mods/src/loopback.py", line 34, in live_loop
    with fOut as (cam, outp_props):
  File "/home/rick/Desktop/tmp/webcam-mods/src/input/input.py", line 73, in __enter__
    return super().__enter__() # type: ignore
  File "/home/rick/Desktop/tmp/webcam-mods/src/input/input.py", line 24, in __enter__
    return (self, self.setup())
  File "/home/rick/Desktop/tmp/webcam-mods/src/output/pyvirtcam.py", line 28, in setup
    self.cam = pyvirtualcam.Camera(width=self.width, height=self.height,
  File "/home/rick/.local/share/virtualenvs/webcam-mods-wTpII5o_/lib/python3.8/site-packages/pyvirtualcam/camera.py", line 219, in __init__
    raise RuntimeError('\n'.join(errors))
RuntimeError: 'v4l2loopback' backend: std::exception
make: *** [Makefile:7: crop-cam] Error 1

The decribe devices outputs

v4l2-ctl --list-device
v4l-cam (platform:v4l2loopback-000):
	/dev/video10

Integrated_Webcam_HD: Integrate (usb-0000:00:14.0-12):
	/dev/video0
	/dev/video1

v4l2-ctl --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
	Type: Video Capture

	[0]: 'MJPG' (Motion-JPEG, compressed)
		Size: Discrete 1280x720
			Interval: Discrete 0.033s (30.000 fps)
		Size: Discrete 960x540
			Interval: Discrete 0.033s (30.000 fps)
		Size: Discrete 848x480
			Interval: Discrete 0.033s (30.000 fps)
		Size: Discrete 640x480
			Interval: Discrete 0.033s (30.000 fps)
		Size: Discrete 640x360
			Interval: Discrete 0.033s (30.000 fps)
	[1]: 'YUYV' (YUYV 4:2:2)
		Size: Discrete 640x480
			Interval: Discrete 0.033s (30.000 fps)
		Size: Discrete 640x360
			Interval: Discrete 0.033s (30.000 fps)
		Size: Discrete 424x240
			Interval: Discrete 0.033s (30.000 fps)
		Size: Discrete 320x240
			Interval: Discrete 0.033s (30.000 fps)
		Size: Discrete 320x180
			Interval: Discrete 0.033s (30.000 fps)
		Size: Discrete 160x120
			Interval: Discrete 0.033s (30.000 fps)

Which folder is being used?

Hello,
Im wondering which folder/s I have to chown in order to be able to use it without root permissions?
Thank you very much :)

How can I change with custom image?

Hello,
I can't figure out how to change to a custom image. What will be the command and where I should put the image? Can you help me? Thank you :)

loopback problem

HI! I have got:

❯ python src/uses/entry.py --help                                                                                                                   ±[master]
Traceback (most recent call last):
  File "src/uses/entry.py", line 1, in <module>
    from loopback import live_loop
ModuleNotFoundError: No module named 'loopback'

After

❯ pip install loopback 

I get

❯ python src/uses/entry.py --help                                                                                                                   ±[master]
Traceback (most recent call last):
  File "src/uses/entry.py", line 1, in <module>
    from loopback import live_loop
ImportError: cannot import name 'live_loop' from 'loopback' (/home/user/.local/share/virtualenvs/webcam-mods-LF416mUB/lib/python3.7/site-packages/loopback/__init__.py)

Before following installation instructions I just have installed Python 3.7 from AUR (Manjaro).

real-time background removal & virtual background

https://gitlab.com/gersonjferreira/fake-chroma-key

https://github.com/webcamoid/WebcamoidIssues/issues/26

https://github.com/tensorflow/tfjs-models/tree/master/body-pix

request: bounding box tracking

to improve the face tracking feature we need:

given

• A: a stream of fixed size frames, eg camera input
• B: a noisy and moving bounding box coordinates (up to input FPS times per second)

generate a parent bounding box (C) within A per input frame to smoothly follow and focus on B.
some parameters for this bounding box generation could include:

• how much padding to add
• what percentage of C would we want B to cover

things to consider:

• B could disappear or go out of A for one or more frames
• future work: we could have multiple B in one frame

More Detailed Installation Instructions

This project looks great, but it would be great if you could add in step-by-step installation instructions for complete noobs!

ModuleNotFoundError: No module named 'webcam_mods'

Hi,

first of all: i am not really a python guy, though i found your software to get bg blurring working on linux.

when trying to kinda reverse engineer how to actually start the whole thing i end up with this situation:

(webcam-mods) [andreas@world-hq webcam_mods]$ pipenv run python entry.py bg-swap
Traceback (most recent call last):
  File "entry.py", line 2, in <module>
    from webcam_mods.config import DEFAULT_BG_IMAGE
ModuleNotFoundError: No module named 'webcam_mods'

its probably me who does something horribly wrong, could you please shed some light onto how to get it running?

checkout v4l2 py3 module

https://github.com/antmicro/python3-v4l2/

Inpit resolution syntax

Hello,
I'm sorry but I really can't figure out what should be the syntax for setting Inpit resolution. I tried in terminal:
export IN_WIDTH=1920 IN_HEIGHT=1080
but it gives me the same 640x480 output. The main problem is Teams is getting only a part of the image (upper left corner), expands it to fit the camera and the image gets pixelated because of the lower resolution. Or may be it's the output resolution? I don't know but I want it to show the whole image at the original resolution.
Can you help me with this? I'll be very grateful!