frostkiwi / mirrorball

Multi-Media Adventure of WebApp + Video ( https://youtu.be/rJPKTCdk-WI ) and Summer of Maths Exposition 2023 submission

🔮 Mathematical Magic Mirrorball 🪄

🇬🇧 English	🇯🇵 日本語

A combination of Video and WebApp ( https://mirrorball.frost.kiwi ). While watching, viewers can directly interact with examples and visualizations, on a second device or a separate browser tab. The audience will be taken on a journey through the history and maths on the topic of the "Mirrorball projection".

Deceptively simple, it is an incredible intersection of Computer Science and Mathematics. This project has been made in conjuction with a paper in pre-print, which you can read on arXiv: https://arxiv.org/abs/2308.10991 . The paper was submitted to IJAT.

Features:

Discover the Mirror Ball Projection with colorful visualizations	Export to Equirectangular / Lat-Long and play around with rotations
basic-demo.mp4	equirectangular.mp4

View the projection of a Mirrorball with different colorful visualizations to understand how the math works. You can project a:

• photo
• video
• live webcam stream
• capture card feed

Any format, which your web browser can decode is supported. That should cover all important file types. You can also export as a Equirectangular projection for use in other panorama software.

Photo contribution guide, as per announcement

As shown in the sample Pull Request #13, just commit the photo into any spot and mention how you want to be credited. To do so, you will need to fork this repo in the top right corner first. Then you can drag&drop the photo into the code. This will make GitHub officially credit you as contributor. I will move it to the correct folder (Example commit), crop, rotate the image and credit the author (Example Commit) and generate the thumbnail (Example commit).

If that's too troublesome: Just message me on any Social media or E-Mail me at [email protected].

Submission:

The photos in the "User submissions" tab were provided by:

• @zjquid on GitHub
• Nature's Joints on Discord
• u/wwxxcc on Reddit
• u/MagicList on Reddit / @CigamPower on GitHub
• Anonymous via E-Mail
• Сергей via E-Mail
• Chesapeake / @mothdotmonster on GitHub

Build instructions

If you want to run the code locally:

• Clone the repo
• Install Node.js
• Run npm install in the repo's root directory to install the dependencies as defined by the package.json file
• Run npx vite, so Vite will start a local server to test the code on

Credits:

• The side controlbar is realized via the package lil-gui, package author: @georgealways
• The debug statistics toggled in the settings tab is the evergreen stats.js, package author: @mrdoob
• Debug interface in the settings tab for tracking down bugs especially on iOS is provided by the awesome eruda, package author: @liriliri
• One key insight on state management to get redraws only when the user is actually touching the interface and be a static page otherwise was provided by @Pomax in this StackOverflow answer
• Vector math performed via the package gl-matrix, package author: @toji

mirrorball's Issues

Export to Lat-Long feature

As requested by @taavimetso9178 in this comment. The Maths is already implemented in the latlong_conversion branch, just gotta port it over.

iOS devices can hit a VRAM limit with 8k textures

as tested with the iPad gen9.
Should somehow write a warning about this and auto handle the case, via downsizing.

Work around Firefox using Windows' weak H.264 decoder

Firefox refuses to run some of the high res video samples, because of their use of Window's own decoder, which has an arbitrary limit of 2304 in height.
https://learn.microsoft.com/en-us/windows/win32/medfound/h-264-video-decoder?redirectedfrom=MSDN#format-constraints

Gotta supply 2048x2048 samples as well and an error message in the case of a decode error and a "retry with lower res" feature.

Gamepad code breaks webpage

If the user has a gamepad plugged in that does not have a button index 14, the web page crashes when trying to load media.

Uncaught TypeError: Cannot read properties of undefined (reading 'pressed')
at Ct (index-8efaa98d.js:612:13377)
at i (index-8efaa98d.js:612:17012)

Webcam Labels are empty if no permissions are given prior

Webcam and capture card labels may be hidden due to fingerprinting prevention, so the webcam list is filled but with no names. As per https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/enumerateDevices#examples .
This should be handled by showing Permissions prevent the reading of Webcam names in the webcam combo box or something.

Border subdivision don't cause a redraw

Forgotten onchange propably

Video does not play through to the end under certain conditions

This one I'm confused with.
There is no issue if the video file comes from the user's device, but from the server, only the first couple of seconds are played.
There is probably a conflict with buffering, pausing and the app resuming, that I never saw until now, since I always used the 4k² uncompressed files from local storage.

Not sure if I'll make it in time for the SoME3 deadline with this issue.

Touch pitch rotation not protected against going over 90°

As per title, gotta clamp it like the mouse rotation. Come to think of it, should double check keyboard rotation

Gamepad controls are not frame independant

As with keyboard input, the gamepad function should get time, as not to have sensitivity bound to frame rate

Memory leak with Webcam input

Probably should free the Imagebitmap requested. May also be affecting image mode

Implement Reflection area visualization

Needed to explain how the percentage of area maps to the projection

Lock controls during image loading

Whilst the spinner is active, the controls panel should be locked to prevent some edge cases of changing values, that are not well defined yet or will flip once media loading is complete.

Input sensitivity changes based on DPI

When going fullscreen vs not, the touch controls change sensitivity and need to be scaled based on the device pixel ratio.