Decouple VideoProcessor from Workers about mediacapture-worker

[mhofman]

BTW, wouldn't the "processing" part be handled by creating a monitor that processes data inside an OffscreenCanvas of which you use it's original canvas's captureStream() to create a MediaStream according to the Media Capture from DOM Elements spec proposal?

This would potentially decouple the content creation part form the consumption, allowing for example mashing up multiple inputs to one output, or have a different framerate.

[rocallahan]

These are possibilities. Going through a canvas and using captureStream() would require solving a few problems.

• For the monitoring use-cases, you want to be notified when a frame is available, and you want to write code that is sure to see all the frames (assuming you can keep up in the long term).
• For the processing use-cases, you want to be able to emit a processed frame with the same timestamp as the input frame, to maintain A/V sync.

I actually agree with your first point though. I think that's probably a good idea.

[mhofman]

Using canvas was just an idea for the production of streams, trying to not duplicate functionality with other parts of the Web Platform.
You're raising an interesting use case of non-live streams though, where you want to generate frames for a given time in a stream. I'm wondering if there's a way to extend the captureStream() feature to support this use case.
In general, support for non-live media in the platform is currently restricted to playback, not generation or processing.

Regarding monitoring, I'm not advocating in this issue to change the consumption model, just to abstract the notification part from the threading model. Ultimately I'd like to find a consumption model that can support both all frames and skip frames use cases (like I already mentioned a few month ago on the mailing list), but I'll open another issue for that another day.

[rocallahan]

Using canvas was just an idea for the production of streams, trying to not duplicate functionality with other parts of the Web Platform.

VideoProcessor as currently specced assumes you'd likely use an OffscreenCanvas to actually do the processing. Though we also want to extend ImageBitmap to provide efficient memory-mapped access to raw image data on platforms that support that. Going through canvas getImageData() is not the most efficient way (e.g. if the data is already available in memory in a format that doesn't match getImageData()).

You're raising an interesting use case of non-live streams though, where you want to generate frames for a given time in a stream.

It's not just about non-live streams. Supporting processing while keeping A/V sync for live streams would also be good to have.

In general, support for non-live media in the platform is currently restricted to playback, not generation or processing.

FWIW this isn't quite true; WebAudio supports "offline" audio generation and processing. Also we're keen to extend this to a general notion of offline MediaStreams.

[mhofman]

Ok I didn't know WebAudio had "offline" generation and processing. I thought it was all real-time.
How do the WebAudio specs handle the manual timing aspect?

The A/V sync issue is more related to the complete separation of audio and video in the Web Platform. A MediaStream actually doesn't guarantee that its tracks are played perfectly in sync, just that they're related.

These would be good questions for a potential Timed Media WG.

[rocallahan]

A MediaStream actually doesn't guarantee that its tracks are played perfectly in sync, just that they're related.

I think it should. It certainly does in Gecko!

How do the audio specs handle the manual timing aspect?

By exposing time and sample rates explicitly. A sample rate isn't very suitable for video though, because of dropped frames and video formats that support irregular frame timing.

[mhofman]

I opened issue #31 to continue the discussion on generation.

Let's keep this issue focused on decoupling monitoring from threading assumptions.

[ChiahungTai]

The API will be looked like below:

partial interface MediaStreamTrack {
    Promise<void>             addWorkerMonitor (VideoMonitor monitor);
    Promise<void>             removeWorkerMonitor (VideoMonitor worker);
    Promise<MediaStreamTrack> addWorkerProcessor (VideoProcessor worker);
    Promise<void>             removeWorkerProcessor ();
};

interface VideoMonitor : EventTarget {
  attribute EventHandler onvideomonitor;
} 

interface VideoProcessor : VideoMonitor {
  attribute EventHandler onvideoprocessor;
} 

// The VideoEvent part is the same.

I am thinking below example, make sense?

Main thread code:

var processor = new VideoProcessor();
var track = mediaStream.getVideoTracks()[0];
var promise = track.addVideoProcessor(processor); //return Promise
promise.then(function(track) {
  newMediaStream.addTrack(track); 
});
var worker = new Worker("processing.js");
worker.postMessage({aCommand : 'pass_processor', aProcessor: processor},
                                  [processor]);

Worker code:

self.onmessage = function(msg) {
  switch (msg.data.aCommand) {
        case 'pass_processor':
                bindProcessor(msg.data.aProcessor)
            break;
        default:
            throw 'no aTopic on incoming message to Worker';
    }
}

function bindProcessor(processor) {
  processor.onvideoprocess = function(event) {
    // Check if the browser supports YUV format.
    var bitmap = event.inputImageBitmap;
    var yuvFormats = ["YUV444P", "YUV422P", "YUV420P", "YUV420SP_NV12", "YUV420SP_NV21"];
    var bitmapFormat = bitmap.findOptimalFormat(yuvFormats);
    if (bitmapFormat == "") {
      console.log("The browser does not support YUV formats.");
      return;
    }
    // Get the need buffer size to read the image data in YUV format.
    var bitmapBufferLength = bitmap.mappedDataLength(bitmapFormat);

    // Create the buffer for mapping data out.
    var bitmapBuffer = new ArrayBuffer(bitmapBufferLength);
    var bitmapBufferView = new Uint8ClampedArray(bitmapBuffer, 0, bitmapBufferLength);

    // Map the bitmap's data into the buffer created in the previous step.
    var promise = bitmap.mapDataInto(bitmapFormat, bitmapBuffer, 0, bitmapBufferLength);
    promise.then(function(bitmapPixelLayout) {
      // Read out the y-channel properties.
      var ywidth  = bitmapPixelLayout.channels[0].width;
      var yheight = bitmapPixelLayout.channels[0].height;
      var yoffset = bitmapPixelLayout.channels[0].offset;
      var ystride = bitmapPixelLayout.channels[0].stride;
      var yskip   = bitmapPixelLayout.channels[0].skip;   // This should be 0.

      // Initialize the buffer for the result gray image.
      var rgbaBufferLength = ywidth * yheight * 4;
      var rgbaBuffer = new ArrayBuffer(rgbaBufferLength);
      var rgbaBufferView = new Uint8ClampedArray(rgbaBuffer, 0, rgbaBufferLength);

      // Convert YUV to Gray.
      for (var i = 0; i < yheight; ++i) {
        for (var j = 0; j < ywidth; ++j) {
          var index = ystride * i + j;
          var y = parseFloat(bitmapBufferView[yoffset + index]);
          rgbaBufferView[index * 4 + 0] = y;
          rgbaBufferView[index * 4 + 1] = y;
          rgbaBufferView[index * 4 + 2] = y;
          rgbaBufferView[index * 4 + 3] = 255;
        }
      }

      // Create a new ImageBitmap from the processed rgbaBuffer and assign to the
      // event.outputImageBitmap.
      var channelR = new ChannelPixelLayout(0, ywidth, yheight, "uint8", ywith * 4, 3);
      var channelG = new ChannelPixelLayout(1, ywidth, yheight, "uint8", ywith * 4, 3);
      var channelB = new ChannelPixelLayout(2, ywidth, yheight, "uint8", ywith * 4, 3);
      var channelA = new ChannelPixelLayout(3, ywidth, yheight, "uint8", ywith * 4, 3);
      var layout = new ImageFormatPixelLayout([channelR, channelG, channelB, channelA]);
      var p = createImageBitmap(rgbaBuffer, 0, rgbaBufferLength, "RGBA32", layout);
      p.then(function(bitmap) {
        event.outputImageBitmap = bitmap;
      }).catch(function(ex) {
        console.log("Call createImageBitmap() failed. Error: " + ex);
      });
    },
    function(ex) {
      console.log("Call mapDataInto() failed. Error: " + ex);
    });
  };
}

What do you think? @rocallahan, @smaug----, @mhofman, @anssiko, @huningxin, @robman

[rocallahan]

Looks fine to me

[ChiahungTai]

Sorry for the bad format. :(

[huningxin]

Looks good to me. Thanks!

[smaug----]

It isn't quite clear to me what the Promise returning methods do - I mean, why they need Promises and, what actually happens in the method.
In the example addVideoProcessor takes a processor which is then immediately transferred to worker.
(so that processor instance becomes neutered). So I assume that is all supposed to just work.
The underlying processor implementation is bound to the track immediately, so doesn't matter that processor instance gets neutered afterwards.

At which point do VideoMonitor/Processor start to get events? asynchronously after calling
add* methods?

About the naming, why add/remove_Worker_...? Nothing forces one to transfer
VideoMonitor or VideoProcessor to worker. What if one just decides to use them in the main thread?
Shouldn't the methods be add/removeVideoMonitor and add/removeVideoProcessor ?

[rocallahan]

Yes, the MediaStreamTrack methods probably shouldn't return promises.

[ChiahungTai]

@smaug---- Thanks for your review. :) I think you are right. The VideoProcessor/Monitor will be neutered once it transferred to the worker.
About the naming of add/removeWorker, it is my bad. I will change to add/removeVideoMonitor and add/removeVideoProcessor.

[anssiko]

LGTM with above comments. @ChiahungTai feel free to proceed with a spec update, I'm happy to review it.

[anssiko]

Submitted a PR to fix, see #53.