Comments (8)
The only minimal example that can point to is the encoding of a video MediaStreamTrack
at Chromium, Chrome using MediaRecorder
(with VP8 or VP9 codec) for an unknown reason (potentially involving NaturalSize()
usage) here outputs a different image at HTML <video>
element at Chromium, the browser that created the file, and Firefox, where the expected output is displayed using the same file. Evidently that change to the source affected captureStream()
which crashes the tab when executed on an HTML <video>
element that contains variable resolution frames. Mozilla browsers do not have those issues.
If input is 400x300, 300x150 spanning two seconds, one second each respectively, the API should ensure (by testing) the output will be 400x300 one second, 300x150 once second. Not only 400x300 for the entire two seconds. Or arbitrary output, e.g., Epiphany Technology Preview, which scales larger frames to lower resolutions and scales smaller frames to larger resolutions when variable resolution frames are encoded in the video track.
The API should not arbitrarily encode the image in such a way as the output will be dissimilar from the input unless explicitly set.
from webcodecs.
Is this issue addressing settings (expectedWidth/expectedHeight)? Or stats?
from webcodecs.
expectedWidth/expectedHeight is just for faster codec initialization (it's slower if you wait for the first frame).
I was thinking something more like for decode:
-
We don't deal with pixel ratios. That would be a problem for rendering, not encode/decode (similar to sample rate changes post decode for audio).
-
The .size on the image is the visible size, ignoring any extra region that's there to make the encoding work.
-
We don't deal with coded size/region until someone asks and says they need it (why would they?)
As for encoding, I need to learn more about what one would expect to happen if you passed in a resolution that doesn't fit macro blocks.
from webcodecs.
Is this issue addressing settings (expectedWidth/expectedHeight)? Or stats?
It's addressing the metadata of the output frame from the decoder.
from webcodecs.
Do not do this Chromium does not dispatch resize event for variable resolution frames and MediaStreamTrack.getSettings() outputs incorrect values for width and height for WebM video output by MediaRecorder at Chromium (arbitrarily change the output to suit only the perspective of the author of the source code, emitting incorrect values relevant to the actual underlying encoded frame).
from webcodecs.
Does this issue cover how values are expressed for the decoder portion of WebCodecs?
// Finally the decoders are created and the encoded media is piped through the decoder
// and into the TrackerWriters which converts them into MediaStreamTracks.
// ...
const videoDecoder = new VideoDecoder({codec: 'vp8'});
Kindly clarify if VideoDecoder()
will be code independent from the implementers' video decoder(s). That is very important if you are expected reliable values to be output for "size" of the current frame.
from webcodecs.
It must be pointed out here that the value of size of the encoded frame will be moot if the decoder does not read each frame when the developer intentionally encodes variable resolution frames intended to be output at HTML <video>
element.
It may be necessary to consider a WebMediaPlayer (with capability to select and de-select the appropriate decoder) to substitute for HTML <video>
element (and implementer default video decoder) if the underlying encoded frame is expected to be displayed as it was input relevant to pixel dimensions instead of arbitrarily scaled to pixel dimensions set at the container metadata only.
An implementer of browser video decoders and HTML <video>
can deliberately ignore potentially variable encoded frames which are written after the initial pixel dimensions of the first frame. The reason for ignoring variable resolution frames could be unclear, though may be based on targeting specific devices which have maximum screen dimensions ("smart" phones; handhelds; tablets). What is interesting when such decisions are made is that invariably when an advertisement or promotion to generate revenue for the parent concern needs to be displayed at HTML <video>
element or custom player as an overlay in the corner or center of the screen the implementer finds a way to display the exact dimensions of the underlying frames.
from webcodecs.
For the root question of "how to expose", please see the solution in draft spec for the VideoFrame interface. I'll give a summary of the current situation below. Please open new bugs for specific issues with this proposal.
- encoded size -> "coded size" (pixel size of decoded frame prior to any cropping or scaling)
- visible size -> "crop size" (cropping applied to coded size)
- natural size -> "display size" (scaling applied to crop size to achieve final display aspect ration)
VideoFrame has all three. Self explanatory.
VideoDecoderConfig has all three. This matches existing generic codec APIs like FFmpeg where not all codecs used have a provision to describe their size in-band. For codecs that do have that capability, the config stuff is just an initial hint.
In chrome, we will use the initial VideoDeocderConfig to pin pixel aspect ratio (PAR) for the stream, while allowing display aspect ratio (DAR) to change per frame. This has been the longstanding behavior of Chrome's
We will not put any size info in the EncodedVideoChunk. We don't have any use for it.
We will put both display size and crop size in VideoEncoderConfig. Crop size is the critical bit, but display size is also nice to have for encoders that want to describe that in-band. Right now the encoder API only has a vague width and height - we'll fix that (#93).
from webcodecs.
Related Issues (20)
- value of optimizeForLatency in emitted decoder configuration HOT 1
- Rounding direction for `AudioData.duration` HOT 1
- What does `allocationSize` return if formats in the source and destination are null? HOT 2
- Chunk data definitions are incomplete (VP9, AV1)
- What's the best way to ensure 1-in 1-out decoding for h264 video? HOT 7
- `contentHint`: notes aren't normative and shouldn't describe normative behaviour HOT 2
- frame.copyTo error in mac-chromeV118 HOT 2
- Probably a silly question: can this encode irregularly spaced frames? HOT 1
- Explicit frame drop signal from VideoEncoder HOT 3
- Long term reference support for AVC/HEVC HOT 9
- isConfigSupported: definition of "invalid" HOT 4
- webcodecs encode h264/h265 Capability set HOT 1
- Support for AV1 switch frames HOT 5
- Fatal encoding/decoding error on Windows 10/11 HOT 2
- Why is isTypeSupported() promise-based? HOT 8
- Receiving Uncompressed Webcam Data without Browser Compression HOT 5
- After how many decode should the codec process the frames? HOT 10
- Clarify `reset()` behaviour when multiple things are being output HOT 3
- key-frame request handling when scalability mode is not L1T1 for encoder HOT 6
- Clarify the `Clone configuration` algorithm HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from webcodecs.