UI: Webcam Support about laserweb4

[arthurwolf]

Stargate Atlantis ! ZPM !

On Thu, Nov 10, 2016 at 5:44 PM, Todd Fleming [email protected]
wrote:

Much of Stargate Atlantis was on zero point finding :)

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#30 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAGpFWZ09MoBFPdTNOWrJkzkzTMk6Kc0ks5q80ntgaJpZM4Ku2XI
.

Courage et bonne humeur.

Damn it, now i want to rewatch SG :(

[jchristheodore]

@jorgerobles I was trying to fix the WebGL crash issue. I am thinking that creating a new texture context each time forces it to lose context.
Solution:
Let the texture be defined above the capture method that is polled each time and it should fix the issue.

Any chance you can ship one of those exact ones to Jorge? Might make dev easier than the back and forth of testing on your side and coding on his?

…

On Mar 1, 2017 1:54 AM, "DarklyLabs" ***@***.***> wrote: @jorgerobles <https://github.com/jorgerobles> Looks like the link above doesn't work with this camera. It errors out. The camera we have is a USB2, 5 megapixel (2592x1944) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#30 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHVr2zJ4NFyLYgCi0UU2cmkASeVAmte9ks5rhLOdgaJpZM4Ku2XI> .

[tbfleming]

Much of Stargate Atlantis was on zero point finding :)

http://www.stargate-sg1-solutions.com/wiki/Zero_Point_Module_(ZPM) - well, we just found the module name for that item then

[jorgerobles]

Hmm we could start using https://github.com/peterbraden/node-opencv with tweaks needed to make it work on OSX (UVC patching... I've dabbled on that on the https://github.com/bqlabs/ciclop project).
A serious programmer could be required, @Jesus89 are you there?

[jorgerobles]

And by the way, I already bought this thingie 🍭

[jorgerobles]

This could be of use in UVC streaming / OMR:

• https://github.com/peterbraden/node-opencv
• https://github.com/node-ffi/node-ffi
• https://int80k.com/libuvc/doc/
• https://github.com/romualdoandre/myomr

Support for a basic USB camera, as is in LW3, is what we are looking for.
We have been exploring the glfx library to perform the image manipulations needed to de-skew our camera image and so far it looks promising.

[jorgerobles]

It already uses canvas and glfx, and perspective distort enabled.

Nice! I didnt even know it was that far! So all thats missing is overlaying (underlaying) it behind the 3D grid? (Emblaser's application of the camera is to help position jobs using an overhead camera in the machines lid when opened upright)

…

On Jan 4, 2017 2:54 PM, "jorgerobles" ***@***.***> wrote: It already uses canvas and glfx, and perspective distort enabled. [image: camara] <https://cloud.githubusercontent.com/assets/1706080/21642815/209b65b2-d285-11e6-868e-42e0e3e373d5.gif> — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#30 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHVr2_7rGoshXtVUBCcHZzlA0Xsw90Mdks5rO5Z4gaJpZM4Ku2XI> .

[jorgerobles]

Yes, I'm sure @tbfleming could overlay into the workspace :)

[tbfleming]

Underlay will be easy. Overlay would require a reliable background color to key transparency. Will I get a rectangular bitmap which exactly covers the machine bounds? Or a rectangular bitmap and coordinates for the 2 corners?

@tbfleming We only need underlay. It will be used for material placement and/or design alignment

@jorgerobles How can we access the glfx parameters to create our own specific required de-skew / pinch setup?

[jorgerobles]

@DarklyLabs it's currently "hardcoded" (as is a inner component property), but draggable points as the glfx demo could be done.

Can the params end up in a machine profile?

…

On Jan 5, 2017 11:15 AM, "jorgerobles" ***@***.***> wrote: @DarklyLabs <https://github.com/DarklyLabs> it's currently "hardcoded" (as is a inner component property), but draggable points as the glfx demo could be done. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#30 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHVr29KmYBOcRMw7jI4EDlm1htGVHfiCks5rPLSdgaJpZM4Ku2XI> .

[jorgerobles]

@openhardwarecoza For sure! Any setting could be saved as a profile.
Another thing (to think about) is to unlock some capabilities upon a machine profile, but that would tighten up and cap down generics (unless developed as a vendor plugin, but that's another issue)

With latest commit I've improved the video overlay and have warp dots (trekkies, behold!) to develop the @DarklyLabs requirements, but It would be better if we have some webcam video or images of what we want to achieve. That could be saved as any other setting.

@jorgerobles Thank you for development on this. We will need to apply a number of distortions to our image to achieve what we need. The camera is at 45 degrees to the workspace and also has a very wide angle lens.

What is the best way we can work with you on this. Supplying you an image of the workspace?

There is also a situation where users have more than one camera attached to their computer ie inbuilt camera and machine camera. Chrome does let you select the camera but it would be simpler and neater for customers to be able to select which camera is used as an option in LW.

[jorgerobles]

@DarklyLabs Yes, a capture or two of the workspace with the webcam could help, with maybe the final effect you want to get. That would make me a better idea of the camera position and transformations.

About to choose the camera from inside LW, I don't know if it's possible (yet! https://simpl.info/getusermedia/sources/). Maybe the Electron build could be capable of.

[jorgerobles]

Well, now it has capability of select what device to use, and a second set of warpdots to select Before (rects) and then After (dots) on perspective transformation.

toolUseVideo: boolean refactored to toolVideoDevice: null|deviceId.
Added toolVideoPerspective: null|{before:[8],after[8]} to settings, so is saveable with profiles.
@DarklyLabs I dabbled around with the controls and moving transformation points you could also handle rotation :)

@jorgerobles Great work.
I have attached a pic of our typical workspace. The paper grid is the work area we need to de-skew and present under the viewport grid.

You can see that there is some lens distortion that needs compensating for, along with the perspective.

[arthurwolf]

You guys might want to take a look at openpnp , they do this already and it works pretty well, probably some code to re-use there.

…

On Sun, Jan 8, 2017 at 2:01 AM, DarklyLabs ***@***.***> wrote: @jorgerobles <https://github.com/jorgerobles> Great work. I have attached a pic of our typical workspace. The paper grid is the work area we need to de-skew and present under the viewport grid. You can see that there is some lens distortion that needs compensating for, along with the perspective. [image: e2_workspace2] <https://cloud.githubusercontent.com/assets/14358950/21746464/1ac78edc-d59a-11e6-99f8-2f9fcb25ebdc.jpg> — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#30 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAGpFfdL9CHcfXZPvFRIiIpHHO1DeeSjks5rQDV5gaJpZM4Ku2XI> .

-- Courage et bonne humeur.

@arthurwolf Wow. that looks very impressive. Especially the automatic calibration feature as seen here: https://www.youtube.com/watch?v=LNa2LNSpa68

[jorgerobles]

Thank you @arthurwolf, It's worth a look (or a dozen more 😄)

BTW that's OpenCV (I mentioned it a while ago). There's a compilation for OpenCV on node, so It must be done "serverside". There's a collateral problem with OpenCV on OSX, and UVC devices, that I don't know If has been resolved (I found a patch for python, I need to learn C++ module compiling for Node this year)
@DarklyLabs thank you for the image. We can work on it in the meantime.

@tbfleming I found this https://github.com/dminor/webgl-lens/blob/master/index.html that seems to match your GL wizardry. Could the fishEye be reversed?

[jorgerobles]

Hohoho! https://jywarren.github.io/fisheyegl/example/

@jorgerobles Looks promising. Can this code be included so we basically supply the parameters as shown in the demo to correct for the lens?

Please note, we still need the current perspective adjustments you have in there.

[jorgerobles]

@DarklyLabs yes, I'm trying to adapt it, but I'm no webgl savvy. The fx are stackable and works on GPU should be fluid enough.

[jorgerobles]

@tbfleming, Can you take a look to the GL thingies? I'm stuck at using glfx with this library. I don't know how to make GLFX play nice with other webgleffects (something related with useProgram)
Maybe is better to ditch glfx, as is not much active. I found this as an alternative (http://jsfiddle.net/rjw57/A6Pgy/)
Broken implementation at https://github.com/jorgerobles/laserweb4/tree/WIP_fisheye

[tbfleming]

@jorgerobles My queue is full right now. Any library that takes images in and produces images out should be ok, as long as it doesn't try to take over the UI. If they use WebGL internally, then their WebGL code should create its own WebGL off-screen context; it won't need to hook into our WebGL code.

One thing to look out for: lots of WebGL code (even ReGL) doesn't dispose of the main WebGL context properly, causing Chrome to have issues. The WebGL spec was a little short-sighted. They hid the proper way to dispose inside a poorly-documented extension meant for another purpose: WEBGL_lose_context.

[jorgerobles]

@tbfleming ok then. I will try to look for alternatives, but as I have no idea about webgl any help will be welcome when you get freed.

[jorgerobles]

@tbfleming I've setup an test repository at https://github.com/jorgerobles/ReglTests
With the Barrel and Perspective migrated to REGL. This is as far I can get.

Help wanted:

• Perspective filter mirror in X the image from webcam.
• Improve rendering (MIPMAP thing I have no idea)
• How to chain effects
• Garbage collection
• Anything else :)

[jorgerobles]

After a lot of pain we have a functional Barrel Distort / Perspective correction. It uses REGL 😭 but performs well. Todd, take a look whenever you can, I'm sure it could be heavily optimized.

@jorgerobles Just did a quick test. Looks great! Thank you for the hard work.
We will test it and let you know how it goes.

@tbfleming Understand you are extremely busy. Any eta on when we can see the camera underlaid in the main viewport?

[jorgerobles]

Thank you @DarklyLabs, I have to do further tests, seems perspective distort needs tweaking.

[jorgerobles]

Definitely, something is wrong with the perspective distort. I've adapted from glfx and something slipped thru it. :|

[tbfleming]

I just tried it. The video device dropdown is empty. My cam works on https://simpl.info/getusermedia/sources/

[jorgerobles]

@tbfleming moving DEV chat to https://github.com/LaserWeb/ReglTests/issues/1

[jorgerobles]

This feature is working, but maybe is not enough. The image gets poor output due no full mipmapping on webgl.

@jorgerobles How can we test this?

[jorgerobles]

@DarklyLabs as using the controls on the LW4 UI (will be improved, for sure) I've put a test bench repo at https://github.com/LaserWeb/ReglTests. Please download and run the installation procedure. Tell me if working properly or more instructions needed.

@jorgerobles pretty cool!

For my camera, i think F may need to have it current far right be a center point of the scale, and add a opposite direction too? if thats possible.

Ie at full right F-slider, I wish I could go just a litle further - hope that makes sense

a & b might need smaller steps if thats possible

[jorgerobles]

@openhardwarecoza the barrel is based on https://github.com/jywarren/fisheyegl, and seems have those limits. (I wish I could fully understand those lens maths, but no 😿 )
I've tweaked css to make the sliders wider, so they can finely positioned. I've also tweaked a bit F to get more values, but it only does weird stuff.
I was hoping @tbfleming could take a look when get freed (webgl optimization and so). Seems very busy these days.

[jorgerobles]

OpenCV is known to have some serious algorithms for lens correction, and http://lensfun.sourceforge.net/calibration/ too, but none of them are JS.

[jorgerobles]

@DarklyLabs Did you tested with the test app? how is it working? Any workable result?

@jorgerobles Back at the lab today. Will test and advise. Thanks!

@jorgerobles Our initial tests look very promising! Very well done.
I have two questions:
1: Is it possible to expose numerical values for the perspective settings? The drag controls are perfect for getting into the ball-park but are difficult to fine tune precisely.
2: Is there any control over the image resolution that is captured?

@jorgerobles Also, FovX and FovY appear to have no effect on the image. They can be removed I suspect.

[jorgerobles]

@DarklyLabs I've updated the test bench with number fields to fine grain the transformation.
Dropped FOV as certainly is not working (I will try to contact @jywarren for further research)

About resolution, this is a tricky part as getUserMedia is not stable on all browsers.
As example, https://webrtchacks.github.io/WebRTC-Camera-Resolution/ comes with a patchy way to get the real resolutions supported by chrome. Only to say I've got a FullHD camera and only gets 640x480 on the test.

@jorgerobles The fields are perfect. Well done.
The next step would be to implement this into LW4 so we can start to use to controls to check against artwork and material .

It would be ideal to store these values into a 'machine profile'.

With respect to resolution, since we will be using a known camera and resolution, is it possible to define the resolution and store this in the machine profile?

[jorgerobles]

@DarklyLabs I've moved the test interface to LW4.
About resolution, please do a quick scan on https://webrtchacks.github.io/WebRTC-Camera-Resolution/ and post the CSV, thanks!

@jorgerobles Looking good. One feature you had in the regltest version was the ability to turn on/off the persoective distortion. This made it really easy to initially setup the draggable controls onto the undistorted image. Can this be included?

The webrtc link did not correctly detect the real resolutions for our camera. I have attached the csv in case it is useful:
gumResTestExport.csv.zip

[jorgerobles]

Yes, don't worry will be :)

[jorgerobles]

@DarklyLabs could you run https://webrtchacks.github.io/WebRTC-Camera-Resolution/ on Full Scan? Thanks!

@tbfleming Is there any chance you could assist @jorgerobles in adding the camera display underlay to the workspace?

[tbfleming]

I created #139 to answer his porting questions.

[jorgerobles]

@DarklyLabs I've partial success on workspace video check it out :) No effects yet, though.

@jorgerobles Yeah! Well done. It seems to be displaying well. Thank you.

@jorgerobles Any progress on implementing the distortions onto the webspace canvas?

[jorgerobles]

@DarklyLabs not yet ☹️

@jorgerobles It looks like the latest dev-es6 branch prevents us from selecting the resolution we require for our camera. It limits us to 640x480, which is incorrect for our need.

Can you change this back to what how it was working previously by allowing us to select the resolution we required?

[jorgerobles]

@DarklyLabs I've spent past whole week hunting bugs for getUserMedia. I've tested all the cameras I've got (An 1080 HD zalman, a Logitech 270, a endoscopic chinese one) to consistently check the resolutions. It's definitely a webrtc implementation (Chrome hardware). No matter what sometimes the camera did not get the resolution properly, nor honor constraints in two different canvases. The previous version is equally borked, maybe we have to put a timeout to try the camera stuff after a while, seems chrome browser does not have the capabilities in the first time.
This is very frustrating. All I can do is take another round of camera issues. What camera have you got? (I will bet for an OV5640 )

PS. I'm not pissed off. Just explaining 😄

[jorgerobles]

@DarklyLabs Maybe your resolution is not listed and so the get stripped. Please run https://webrtchacks.github.io/WebRTC-Camera-Resolution/ at full scan

@jorgerobles I can appreciate how frustrating this must be and really want to thank you for sticking with it.
We will run this scan again today and let you know the results. We ran it yesterday and it did not detect the correct camera setting correctly. We will try a different computer today.

@jorgerobles Looks like the link above doesn't work with this camera. It errors out.
The camera we have is a USB2, 5 megapixel (2592x1944)

@openhardwarecoza good idea
@jorgerobles can you email me your address and contact number.

[jorgerobles]

@DarklyLabs Done :)

@jorgerobles We have just run a firmware update on our camera and it is now passing the webrtc-camera-resolution test with 1920x1080.

Browser Device Res Name Ratio Ask Actual Status
chrome 56 PC Camera (058f:3830) 1920x1080 16:9 1920x1080 1920x1080 pass

Any ideas why this resolution isn't offered through LW?

[jorgerobles]

@DarklyLabs That resolution is supported https://github.com/LaserWeb/LaserWeb4/blob/dev-es6/src/lib/video-capture.js#L8
And so, my camera does #30 (comment)

The only thing I wonder is if we have to put a sleep between camera resolution detection, but is now using promises to it should take the time the chrome<->hardware needs.

The other thing I could do is just no forbid any resolution, skipping all detection. If a resolution change does throw an Overconstraint error then disable the option.

@jorgerobles Not sure what is involved from your end, but it may be worthwhile trying.

[jorgerobles]

@DarklyLabs Hmm.. I will go for not forbid resolution first. I will branch and send you the git to your tests.

[jorgerobles]

@DarklyLabs https://github.com/LaserWeb/LaserWeb4/tree/video_tests

@jorgerobles Can you explain how to use this version? When we select the camera it sits there with a spinning cursor on the video resolution field.
Tried to enter a resolution but it still sat there.

[jorgerobles]

@DarklyLabs Shame on me. I pushed the branch too fast. Is now fixed.

I have to find a way to get if the resolution changes are really true or are Chrome-Faked. I've observed no verified changes are done when you choose higher resolutions. ☹️

[jorgerobles]

@DarklyLabs, @openhardwarecoza, Sorry, I give up.

Have no way to consistently prove the resolutions obtained are true. We should ditch the resolution combo, because does nothing real as far as I tested. Has been so long fighting with this issue, maybe we can reopen in later this year, with a new perspective.

Also @tbfleming, I have no clue of WebGL, and I would personally appreciate if you could translate the remaining effects from REGL to LWGL, thanks.

@jorgerobles @tbfleming We have been testing our camera and working with the manufacturer for the last few days. It looks like under windows, the resolutions are reported correctly but there is a problem under OSX for the camera.

With the current functionality you have in dev-es6, all that is missing for us is to have the effects from setup view applied to the viewport view.

This will at least allow our users to use their camera, even if its low resolution at the moment.

[jorgerobles]

Going fresh.