The open source VTuber software. ❤
Home Page: https://www.vignetteapp.org
License: Other
The open-source VTuber toolkit.
Please make sure you meet the following prerequisites:
Vignette is Copyright © 2023 Cosyne, licensed under GNU General Public License v3.0 with SDK exception. For the full license text please see the LICENSE file in this repository.
Because of how scuffed DlibDotNet is (#140), we will be ditching DlibDotNet for real time OpenCV detection instead and feed this to a compatible neural network. The work item will be as follows:
Of course we will be locked to PyTorch Tensors for now but I guess the third party integrators can figure out converting their models to get PyTorch instead.
As part of #28, we have discussed how raw data would result on jittery rough data, even if the neural network used is theoretically as precise as a human eye predicting the facial movements of the subject. To compensate for jittery input, we will implement a sort of lag-compensation algorithm.
John Carmack's work with Latency Mitigation for Virtual Reality Devices (source) explains that the physical movement from the user's head up to the eyes is critical to the experience. While the document is designed mainly for virtual reality, one can argue the methodologies used to provide a seamless experience for virtual reality can be applied for a face tracking application, as face tracking like HMDs, are also very demanding "human-in-the-loop" interfaces.
Byeong-Doo Choi, et al.'s work with frame interpolation using a novel algorithm for motion prediction would enhance a target video's temporal resolution, by using Adaptive OBMC. Such frame interpolation techniques according to the paper has been proven to give better results than the current algorithms used for frame interpolation in the market.
As stated on the background, there are many strategies we can perform lag compensation for such raw jittery input from prediction data from the neural network, it is limited to these two strategies:
Byeong Doo-Choi, et al. achieves frame interpolation by the following:
First, we propose the bilateral motion estimation scheme to obtain the motion field of an interpolated frame without yielding the hole and overlapping problems. Then, we partition a frame into several object regions by clustering motion vectors. We apply the variable-size block MC (VS-BMC) algorithm to object boundaries in order to reconstruct edge information with a higher quality. Finally, we use the adaptive overlapped block MC (OBMC), which adjusts the coefficients of overlapped windows based on the reliabilities of neighboring motion vectors. The adaptive OBMC (AOBMC) can overcome the limitations of the conventional OBMC, such as over-smoothing and poor de-blocking
According to their experiments, such method would produce better image quality for the interpolated frames, which is helpful for prediction in our neural network, however it comes with a cost of having to process the video at runtime, as the experiment is only done on pre-rendered video frames already.
John Carmack's work with reducing input latency for VR HMDs suggests a multitude of methods, one of them is View Bypass - a method achieved by getting a newer sampling of the input.
To achieve this, the input should be sampled once but can be used by both the simulation and the rendering task, thus reducing the latency for such. However, the input and the game thread must run in parallel and the programmer must be careful not to reference the game state otherwise it would cause a race condition.
Another method mentioned by Carmack is Time Warping, which he states that:
After drawing a frame with the best information at your disposal, possibly with bypassed view parameters, instead of displaying it directly, fetch the latest user input, generate updated view parameters, and calculate a transformation that warps the rendered image into a position that approximates where it would be with the updated parameters. Using that transform, warp the rendered image into an updated form on screen that reflects the new input. If there are two dimensional overlays present on the screen that need to remain fixed, they must be drawn or composited in after the warp operation, to prevent them from incorrectly moving as the view parameters change.
There are different methods of warping which is forward warping and reverse warping, and such warping methods can be used along with View Bypassing. However, the increased complexity for lag compensation of doing input with the main loop concurrently is possible as the input loop will be independent of the game state entirely.
Such strategies mentioned would allow us to have smoother experience, however, based on my personal analysis, I found that Carmack's solutions would be more feasible for a project of our scale. We simply don't have the team and the technical resources to do from-camera video interpolation as it would be computationally expensive to be implemented with minimal overhead.
Pulling from #4, we have to decide which settings to add in the UI. We already know which we would like to implement but right now I think we should focus on these areas
Model Selection
HoloTrack.Resources or import their own. (Refer to #11).Background Selection
Live2D parameters
Hotkeys
Model Position
Camera Selection
HoloTrack.Vision.in osu!lazer, when we drag and drop a skin or a beatmap, it gets automatically imported. We might want to implement it since there's a need for it at some point.
This isn't a high priority however, since anything below this issue is much more important.
As it stands right now there's some things fundamentally broken which is a major roadblock on Vignette. We're going to fix one of them, and one of those is the draw hierarchy, because right now the inheritance in the hierarchy doesn't make sense.
It seems that it tries to keep searching for Live2DCubismCore, though from what I can tell, we're supposed to be using CubismFramework.
We have the Helen Dataset imported but we haven't actually hadn't started using it. We'll want our program to read our Helen data.
After cloning the repo I had to manually install Humanizer and System.Collections.Immutable due to some Unauthorized errors. Now I'm stuck with a green screen. What am I supposed to do?
Currently we have automated lint tests to ensure code quality. However we will need to automate tests then releases in GitHub Actions.
Right now I still have my code in there regarding Camera enumeration so I might as well get rid of them along with DirectShow.
Since Face recog inferenceing may block the main thread, it's best we implement it is in a seperate thread - may be it a worker or using green threading.
Considering the current UX we have, it feels cramped and unintuitive. In this iteration, we'd focus on visual clarity and as well as follow the branding changes.
Concept:
We want to customize the Layout, and to do that we need to do the following:
Essentially, since we're going to have a layout similar to this:
Currently, the lint and build action will always run when there is a push. We'd only want this to happen on the master branch and if there are any changes to the source code as it doesn't make sense to make a build artifact if there aren't any actual changes to the application itself.
It isn't good UX to force close the app just to only exit it. It may cause unwanted side effects as well.
For some reason our Linux CI is always failing so we might need to investigate further.
To accelerate development, we will need to deploy our version of osu.Framework, preferrably under our GHPR repos since in the future we will be integrating changes that will be impossible to mainline to ppy.osu.Framework. While we want to be upstream as possible, we would also love to do our development without waiting too much on the upstream, especially when it's our own changes so the following groundwork will be done:
Vignette.osu.Framework.-vg_<commmit_hash> suffix).Currently, our Camera implementation exports it into a OpenCV Mat. However, we must convert the Mat to a video stream in order for it to be usable at all for our DLib classification.
As the final task for Milestone 1, we're going to hook up the tracking worker to Live2D and see if we can spot some bugs before we turn in on our release.
In the next week we will be testing Windows 11 so we can test out our integration to Windows 11 early. We should investigate how we can publish to the Store. Right now, as part of the new Windows Apps SDK, we can use the MSIX format for Windows and distribute to MS Store, then Steam for others. This should allow us to be more flexible with our deployment pattern.
On a fresh install, the default theme is incorrect thus failing to load and the entire user interface will be coloured white.
Quoting @LeNitrous from #31 :
Currently there is a bug when there are more than 1 camera where it'll show the wrong name for the camera currently being used. It'll be fixed in a later PR.
We will be assigning this for AB2 since this will be needed to be fixed along with the finalization of the rest of the basic settings UI.
The reason some of these Live2D sample models have additional arm parts is because its required by their baked motion. We shouldn't make these additional arm parts visible if they're not being used.
See title.
Dependabot couldn't authenticate with https://nuget.pkg.github.com/vignette-project/index.json.
You can provide authentication details in your Dependabot dashboard by clicking into the account menu (in the top right) and selecting 'Config variables'.
DlibDotNet is looking at the incorrect paths, as the library seems to be looking for /usr/lib when your .so's are in the same directory as the library.
It has become apparent that we need to support multiple rendering systems in order to reach a wide audience. In order to make this possible, we need to make the puppet implementation abstract and make it modularized for end users to use whatever they see fit for their needs. The following listed below is what we'd want to support initially.
Some parties has raised interest that they want to use our Computer Vision implementation, and out of courtesy, I will be providing a experimental NuGet package in the near future. To do so, I will move out anything that involves o!f out of HoloTrack.Vision.
I'll be forking DirectShowLib and see if we can target it to netstandard 2.1 to the very least.
Since osu!framework does not support multiple windows, we'll have to "write" our way out of the problem, a solution proposed by peppy in our discussion in Discord was this:
Honestly this sounds more modular in structure so I think we'll follow with this approach.
Right now, the way we can do this is the following:
Split the Live2D Window from the "Manager" Window - we do this by having HoloTrack.Desktop launch the Live2D Client (probably put it over at HoloTrack.Live2D so we don't make another package?)
HoloTrack.Live2D is a library, we just tell HoloTrack.Desktop to launch it.
HoloTrack.Live2D will have to ask HoloTrack.Desktop for the current user configuration and use that user configuration to build its scene.
The term "completely open source" is potentially misleading since both Live2D and Bass (used by osu!framework as the audio backend) are not open source.
We want to only target one singular target at the moment, and currently, Face.GetLandmarks() will end up not using the Landmark we provided and will focus on multiple targets. We need to fix that.
Since we did the rewrite in #210, we might as well start on 3D support for osu-framework. Still debating if we should make it out of tree fork or a plugin, but it should be something reproducible later on.
TODO for this task is the following:
Graphics3 namespace (probably won't happen but we'll see).The finalized name for our project is now Vignette. However, we have to reflect it to our repositories and the projects that linked with us. GitHub atm redirects holotrack/holotrack to our new name but that won't be the case for the near future.
This only occurs when running the application from a published build.
We'll have to document more significant parts at some point. We'd want contributors to have an idea how everything works in the back-end after all.
For now we can direct them to osu!framework's Getting Started wiki pages.
Since we want to be able to parse models other than the ones stored in HoloTrack.Resources - I think the best way here is:
If we look at #4, we have a import model, we don't actually import! We just add a reference to the path of the model file. This should lessen the need for us to implement a model interchange format.
Same idea goes for background parsing, we just need to handle references.
If the reference returns an error (aka file moved or what not), we can give the option for the user to remove the reference.
Even if we fulfill #210, there's still some problems on the engine, since there's no clear indication on how to schedule Components. Component by definition is implementation-agnostic, hence we want it to be scheduled to any thread area the game offers, making it a perfect primitive for the entire game.
We'll have to support multiple languages. A good start is looking at Crowdin as a source. We'll support languages by demand but for starters I think we'll support English, Japanese, and Chinese (Simplified and Traditional) given we have people proficient in those languages.
As for implementation, That would be the second part of investigation.
This has been requested by the community; however, this is kinda low priority as we focus most on the core components. The way this works is the following:
MainMenu..dll or a .nupkg which the program can "extract" or "compile" into a DLL, something we can do once we have a better idea with how to dynamically load assemblies.Anyone can propose a better design here since this is a RFC, we appreciate alternative approaches for this.
We should introduce new users in how to use the application on first boot with an option to skip everything as it isn't clear to new users how to first use the app.
Apparently we're getting shader errors in AMD GCN4 graphics cards, this will affect Zen+ microprocessors as well since they carry Vega 8 to Vega 11
Desktop effects are killed globally when vignette is running; Some parts like disabling decorations are fine, but transparency, wobbly windows, smooth animations for actions, etc are all disabled as long as Vignette is running.
We exposed this function which basically opens a VideoCapture however, int index seems ambigous, but if it does refer to the camera index, then we can use this.
CC @LeNitrous can you check a bit if it's actually a index that refers to a camera device? inasmuch as we can use this, I don't think we can enumerate Video Capture devices which is kind of a hassle :/
Register a new virtual camera device so that applications such as Discord, Zoom, OBS (using another option other than Game, Desktop or Window Capture) can use it. This way, we can also control what can be sent to that virtual device avoiding the HUD from being seen at all.
This should help maintain the repository have a uniform standard to coding styles and be able to test before being merged with master.
Optional tasks linked to this: #10
We'll want our users to select from predefined models we have at Resources/ and import their own and save to a directory. Currently we're forcing Haru as our reference model at the moment.
(PS: I have nothing against Haru, she's cute)
This would give us breathing room when we would want to implement other libraries for facial recognition. The hierarchy will be:
FaceTrackerFaceRecognitionDotNetFaceTracker : FaceTrackerTensorFlowFaceTracker : FaceTrackerUnfortunately, our Tracking Backend, which is FaceRecognitionDotNet, which uses DLib and OpenCV, didn't turn out as performant as expected. The delta is too high to make a significant data, and the models currently perform poorly. In light of that, I will have to make a backend we can control directly instead of relying on others' work which we're not sure that has any quality.
Right now we're looking at CNTK and Tensorflow. While CNTK is from Microsoft, there is more laywork on Tensorflow, so we'll have to decide on this.
HoloTrack Packages are just ZIP files that we can just unpack, however there is a strict layout to follow, similar to how .osz files were implemented.
HoloTrack.Live2D.Idea credits from: @LeNitrous
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.