fourms / audiovideoanalysis Goto Github PK

Now, the markers disappear when the mg is inverted and sg is normal.

This would significantly increase the performance of the program so we should try to do most of the processing in GL land.

The camera format actually initializes to "format -1", for some reason. We need to tell the jit.grab to use "format 0" in the initialization process, the same as what is shown in the UI.

This parameter could be beneficial to users if their machines struggle with the load of the application. For instance.

This might be something we want to include/change? Not very important, but nonetheless.

Ive done some searching online, but Im pretty sure the newer versions of Max don't need different [jit.grab] specifications for different operating systems. BUT, I'm not entirely sure. Will need to test the v1.0 on different OS`es to be sure.

Would be good to have an option to choose logarithmetic scale in the spectrogram.

I think this could make the application a little more versatile in that it would accommodate a more varied use.

This is the only "old" external that is still being used to calculate the mean values for the motion-and-videogram generating. Because of this, it might suddenly end up being un-supported down the line. Either way, a custom replacement is safer in any case.

For now, I use the [jit.gl.asyncread] to downsample the texture to a matrix going into the [xray.jit.mean] before re-converting it back to a texture for further processing down the line. This readback method, which downsamples and reads the image from the GPU to the CPU for the mean calculations, seems to be a good practice since the CPU is obviously more optimized to handle these kinds of operations. However, these conversion processes (back and forth between the GPU and CPU) introduces more unnecessary processing which can affect the performance of the app.. maybe..

In any case, I think it would be interesting to find a way to do this on the GPU, so effectively replace [xray.jit.mean] with something in [jit.gl.pix], or something. Alternatively, using the same readback method but with some custom jitter method on the CPU.

This would result in the same "floating" window we have in the VideoAnalysis. I guess this would be a part of issue #9 . It allows the users to freely adjust the window size according to their needs.

According to the reference literature, this is still a thing because of WIN and OSX`es different video engines. This issue is just a note that we need to create two final versions, one with [jit.dx.grab] for the Windows build, and one with [jit.qt/viddll.grab] for the OSX build.

I`ve already taken measures to avoid this, but its still a thing... Must be missing something.

This should be avoided, for sure.

This could be helpful if you have multiple cameras plugged in. We could also implement a "rescan" feature which updates the options if you plug something in while running the app. Related to issue #5

We can also have an option to change the capture dim to smaller resolutions, if necessary.

Add:

• About section #15
• Display options
• Export images #29 #32
• Help (Report a problem to github page).

This was mentioned as a possible enhancement to the VideoAnalysis. We could test this option here and see how it turns out.

When the images are grayscale-inverted, the text on the display is black (because the background is white), but the audio/motion is also very black.

Need to get things working again, 10 years later.

This can be handy if you connect a new video device while the app is running

• Check that everything is correctly initialized.
• Check that we don't bang something an unnecessary amount of times when initializing the app.
• App UI (app-window size, display-window size, no-grow flag, correct paths (documentation), etc.)
• optimize the camera initialization process.

This just means using the same UI colors and stuff from the VideoAnalysis to have more consistency between the applications.

We only use it when pressing ESC for fullscreen, so it's not necessary to have it "hanging around".

Optionally, we could do the same window/viewing-implementation as we did in the VideoAnalysis, which in this case would entail having the Spectrogram and Motiongram in the jit.window.

We only ever care about the world-dim anyway.
And we only use world-dim for most of the other processes that require video dimension.

This might make more sense having the color image be the default.

This is just something I missed in the original implementation.

This happens even when displaying the regular image, so even with no motion averaging or noise reduction happening. Have a feeling its related to issue #18 . An easy fix might be to scale the x-count printing to be slightly slower than the average framerate of the [grab] objects.. or something.. They are about equal now. Will look into this more and try to find a solution.

We can possibly find some smoother way to print the motiongrams and spectrograms. Related to issue #12

Like we have in VideoAnalysis where users can choose between binary and unary noise-reduction.

I think I took this out of the original to "make room" for red and green values to represent differences in frequency energy. However, it's customary to have phase difference represented in a spectrogram so it's probably best to find a visual way to integrate it.

I know this is first and foremost a realtime application, but it would be very handy to be able to export your motiongram or spectrogram as image files on your device. This could go in the menubar, for instance.

At least general frequency markers would be very helpful for analysis. Like this:

But instead of having the marker outside of the display, it would be cool to have them inside the display window.

This is because the draw bang stops when the output_matrix attribute is set to 1. It should be an easy fix.

The idea here is to ensure that the sg and mg always(!) have the same x-count, so as to make it more resilient.

Maybe from green to red where the more energy in a bin the redder the printing. Or something. This kind of color palette also matches basic audio meters.

This would be really nice, I think. Practically it means that a user can click on any location in the spectrogram and get the Hz, amplitude, phase, and time(?) of that particular point.

In addition to the clickpoint info we now get from the spectrogram, it would be nice to have an option to view the freq and time markers in the display, as mentioned in issue #32 .

In light of this, I think it would be cool, and helpful for the analysis side of the app, to have the possibility to view a grid over the display. A grid that cross-sections the time markers on the x-axis with the frequency of the y-axis. It would also change color based on the color-options.

This could also just be a handy feature to add to the recordings before exporting the images since we cant use the clickpoint feature on an exported image.

This is very helpful if you want to record something and do some basic analysis/viewing afterward.

In the current implementation, the colors aren't really representative of the camera feed. This is because it doesn't really take the video (screengrab) colors into account properly.

Experiencing the same TIFF-export issue we experienced when working on VA. Still, with max 8.1.5... However, we made a "hack"(fix) for it which we can just reuse for now.

I`ll leave this issue open because we should continue to check if the Cycling people fix this issue down the line.

This is continuing what was addressed in issue #26 . The spectrogram Y-axis now represents a linear representation of the full sampling size (or half, more specifically, up to 22khz). Therefore, most of the spectrogram "space" in the UI remains blank most of the time, as 10khz is located approximately in the middle of the screen. Although this is totally fine, it's now very user-friendly. It renders a lot of meaningful content almost too small to read.

A few things come to mind as possible solutions here:

1. Make an option to do logarithmic audio scaling. Maybe also let users choose the logarithmic scale value. like mentioned in issue #37
2. Crop the spectrogram Y-axis to a smaller fixed size. f.eks from 0 - 15/12000 or something.
3. Allow users to "zoom in" on the spectrogram in the UI. This zoom would only affect the Y-axis of the spectrogram and would allow you to effectively zoom in to closer inspect certain frequency ranges.

Probably a combination of all of these will be best.

Would be nice to be able to select video device in a menu similar to for the audio device.

I`m thinking that it would be very cool to be able to drag & drop an audio or video file into the application and get the spectrogram and/or motiongram printed in the display. So if you drop a video it would make a spectrogram of the audio and motiongram of the video. And if only an audio file then only the spectrogram.

This would be especially interesting with regards to issue #23 , where you would then be able to view the relationship between the audio and motion in the video in different ways. Like layered, for instance.

I think a kind of vectral blurring would be a nice addition to the spectrogram as it can give a more general overview of the frequency distribution. For instance, this kind of blurring can be great when analyzing lots of short and iterated sounds. I will try this for the 2.0 version.

These options could be nice to have in view (UI).

This is related to issue #12 and #19. When using the [alfablend] layering method, the inverted spectrogram covers the whole frame due its flipped values.