mozilla / mp4parse-rust

Parser for ISO Base Media Format aka video/mp4 written in Rust.

This is an mp4 track metadata parser.

Our primary interest is writing a pure-rust replacement for the track metadata parser needed by Firefox.

API documentation

Project structure

mp4parse is a parser for ISO base media file format (mp4) written in rust.

mp4parse-capi is a C API that exposes the functionality of mp4parse. The C API is intended to wrap the rust parser. As such, features should primarily be implemented in the rust parser and exposed via the C API, rather than the C API implementing features on its own.

Tests

Test coverage comes from several sources:

• Conventional tests exist in mp4parse/src/lib.rs and mp4parse_capi/src/lib.rs as well as under mp4parse/tests and mp4parse_capi/tests. These tests can be run via cargo test.
• Examples are included under mp4parse_capi/examples. These programs should continue to build and run after changes are made. Note, these programs are not typically run by cargo test, so manual verification is required.

Versioning

The master branch represents the last version released to crates.io plus any development since that release. Firefox will ship specific git revisions from the master branch (refer to the mp4parse_capi dependency listed in toolkit/library/rust/shared/Cargo.toml for the currently shipping revision). When sufficient changes to merit a new crates.io release have occurred, the version in Cargo.toml will be bumped and tagged, and the new version published to crates.io.

mp4parse-rust's Issues

CODE_OF_CONDUCT.md file missing

As of January 1 2019, Mozilla requires that all GitHub projects include this CODE_OF_CONDUCT.md file in the project root. The file has two parts:

1. Required Text - All text under the headings Community Participation Guidelines and How to Report, are required, and should not be altered.
2. Optional Text - The Project Specific Etiquette heading provides a space to speak more specifically about ways people can work effectively and inclusively together. Some examples of those can be found on the Firefox Debugger project, and Common Voice. (The optional part is commented out in the raw template file, and will not be visible until you modify and uncomment that part.)

If you have any questions about this file, or Code of Conduct policies and procedures, please see Mozilla-GitHub-Standards or email [email protected].

(Message COC001)

Support rev 1.0 vpcc box

Since FFmpeg's commit 64ad44a381 (FFmpeg/FFmpeg@64ad44a381), Firefox no longer plays vp9 in mp4 files.

This commit updates the vpcc box to be per 1.0 spec:

https://github.com/webmproject/vp9-dash/blob/master/VPCodecISOMediaFileFormatBinding.md

(from https://bugzilla.mozilla.org/show_bug.cgi?id=1365787)

Original format from ProtectionSchemeInfoBox (sinf) is not exposed via C API

This member contains the original format 4cc when dealing with encrypted streams. It would be useful to expose this to aid in handling encrypted media.

The specific case I have in mind is avc1 vs avc3. In avc3 an h264 stream is expected to contain certain information in band, meaning this information may be encrypted and unusable prior to decryption. By exposing the original format, consumers can better reason about what information should be available prior to decryption. In the Gecko use case this will help us understand if we can perform conversions to annex b or not.

Panic when using read_box

extern crate mp4parse;

use std::io::Cursor;

fn main() {
    let mut c = Cursor::new(b"\x30\x30\x30\x30\x66\x74\x79\x70\x30\x30\x30\x30\x30\x30\x30\x30".to_vec());
    let mut context = mp4parse::MediaContext::new();
    let _ = mp4parse::read_box(&mut c, &mut context);
}

panic discovered using afl.rs

Support the ALAC codec

According to its wiki page, the Apple Lossless codec "ALAC" uses mainly MP4 as its container, and the CAF container a bit less commonly.

Check for valid context on C API getters

Functions like mp4parse_get_indice_table access data through a context object within the parser parameter. To help reduce the risk of this unsafe code, the context object could keep some state to indicate whether a a valid parse operation preceded the getter call rather than assuming good data from the C side.

If the C caller passed in uninitialized memory, there's no guarantee that it wouldn't match the field that marked a context as valid, but it would be much less likely.

Implement `meta` parsing

Apple's QuickTime File Format is the closest to a standard to storing metadata in MP4 Boxes, and should be sub boxes of trak, moov or mdia

Publish 0.9.0 on crates.io

It's been a while since we've made a release. This needs:
[x] bump mp4parse version
[x] publish mp4parse_fallible
[x] cargo publish for each crate.
[x] Tag repo.

How do I access track data?

Hello, I am using this crate to read an mp4 container that contains audio tracks I need to decode, I am not sure how to get access to the audio packets of each individual track.

So far I can access the SampleEntry::Audio of each track but in order to decode the packets, I need to be able to calculate the offset each packet in the container file, which eludes me at the moment.

Thank you.

Panic when using read_box

extern crate mp4parse;

use std::io::Cursor;

fn main() {
    let mut c = Cursor::new(b"\x00\x00\x00\x04\xa6\x00\x04\xa6".to_vec());
    let mut context = mp4parse::MediaContext::new();
    let _ = mp4parse::read_box(&mut c, &mut context);
}

coreyf@frewbook-pro /t/mp4 (master) [101]> cargo run
   Compiling mp4 v0.1.0 (file:///private/tmp/mp4)
     Running `target/debug/mp4`
thread '<main>' panicked at 'invalid box size', /private/tmp/mp4parse-rust/src/lib.rs:233
Process didn't exit successfully: `target/debug/mp4` (exit code: 101)

panic discovered using afl.rs

Why isn't LPCM info exposed over the C API?

The library appears to parse all the LPCM information (as far as I can tell so far), why does it not expose the LCPM type over the C-API?

mp4parse-rust/mp4parse_capi/src/lib.rs

Line 106 in 74ac065

pub enum Mp4parseCodec {

Maybe there is so other C header that this library tries to emulate?

Use `std::alloc` functions instead of `(re)alloc` from libc

mp4parse-rust/mp4parse/src/fallible.rs

Lines 22 to 25 in 8239cea

	extern "C" {
	fn malloc(bytes: usize) -> *mut u8;
	fn realloc(ptr: *mut u8, bytes: usize) -> *mut u8;
	}

The use of Vec::from_raw_parts and Box::from_raw with pointers from libc’s malloc or realloc is incorrect, and may cause Undefined Behavior when the Rust standard library’s global allocator is not libc (such as on Windows, or in programs that use #[global_allocator] to override it).

Functions from the std::alloc module should be used instead:

• https://doc.rust-lang.org/std/alloc/fn.alloc.html
• https://doc.rust-lang.org/std/alloc/fn.realloc.html

See also mozilla/mp4parse_fallible#5 and https://phabricator.services.mozilla.com/D71742

cargo check breaks build_ffi_test

If cargo check is run before cargo test, build_ffi_test breaks in the following manner:

$ cargo clean && cargo check && cargo test --all
[…]
test build_ffi_test ... FAILED

failures:

---- build_ffi_test stdout ----
status: exit code: 2
--- stdout ---
rm -f test
rm -f *.a.out
rm -f *.o *.a
rm -f *.d
rustc -g --crate-type staticlib --crate-name mp4parse \
	  --emit dep-info,link=libmp4parse.a \
	  --print native-static-libs \
	  -L /Users/jbauman/src/mp4parse-rust_m-u-current/target/debug/build/mp4parse_capi-45f37232212af4dc/out/../../../deps ../src/lib.rs \
	  2> libmp4parse.a.out || cat libmp4parse.a.out >&2
c++ -g -Wall -std=c++11 -I../include/ -c test.cc
c++ -g -Wall -std=c++11 -I../include/ -o test *.o libmp4parse.a 

-- stderr ---
error[E0464]: multiple matching crates for `mp4parse`
  --> ../src/lib.rs:37:1
   |
37 | extern crate mp4parse;
   | ^^^^^^^^^^^^^^^^^^^^^^
   |
   = note: candidates:
           crate `mp4parse`: /Users/jbauman/src/mp4parse-rust_m-u-current/target/debug/deps/libmp4parse-d0bac2999dfaf13d.rlib

error[E0463]: can't find crate for `mp4parse`
  --> ../src/lib.rs:37:1
   |
37 | extern crate mp4parse;
   | ^^^^^^^^^^^^^^^^^^^^^^ can't find crate

error: aborting due to 2 previous errors

For more information about this error, try `rustc --explain E0463`.
clang: error: no such file or directory: 'libmp4parse.a'
make: *** [test] Error 1

thread 'build_ffi_test' panicked at 'assertion failed: output.status.success()', mp4parse_capi/tests/build_ffi_test.rs:19:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.

See #193 (comment) for some background.

In the meantime, we can work around the issue by running cargo check (for the sake of checking feature-specific code, since we can't run cargo test with the mp4parse_fallible feature) after cargo test.

Encrypted AV1 files do not parse

• We won't parse encrypted AV1 because we don't allow protected sample entries in AV1 configs:

  mp4parse-rust/mp4parse/src/lib.rs

  Line 2917 in 309b9b8

  BoxType::AV1CodecConfigurationBox => {

• We will also want to expose the extra data/config via C as Gecko will use that to config encrypted decoders. I think this can be done around

  mp4parse-rust/mp4parse_capi/src/lib.rs

  Line 995 in 309b9b8

  VideoCodecSpecific::AVCConfig(ref data) | VideoCodecSpecific::ESDSConfig(ref data) => {

• Finally we'll want some test coverage.

Parse `avcc` Box ?

Hi,

Does this library now support parsing avcc box?

I wrote some code to convert the MP4 (H264) file into a H264 stream file.

Since the avcc box parsing is not currently supported,
I cannot extract SPS NAL and PPS NAL from mp4parse::track.

Code: https://gist.github.com/LuoZijun/3e727fd5d337e2e780ba7d68c7bef618#file-mp4_video_samples_to_h264-rs-L259

Raw H.264 Stream:

SEI NAL | SPS NAL | PPS NAL | IDR NAL ....

Update VPxConfigBox field names to match v1.0 of the vpcC spec

@jyavenard noticed the field names of VPxConfigBox don't match the (current v1.0) vpcC spec.

This is because the names are based on v0.0 of the spec (from Netflix/vp9-dash). We should rename the VPxConfigBox fields to match v1.0, since v0.0 was experimental(ish) and is probably dead now. The existing v0.0 support will be retained by mapping the v0.0 fields to v1.0.

Put the capi module behind a feature flag

Servo doesn't need the capi module, and this brings rusty-cheddar and syntex to our already quite big dependency graph. Could that thing be put behind a feature flag?

read_mp4 can expose uninitialised data to third party code (unsound)

read_mp4 calls read_moov which calls read_pssh which calls read_buf which calls allocate_read_buf and passes its result to Read::read.

mp4parse-rust/mp4parse/src/lib.rs

Lines 2320 to 2321 in 26e614b

	if let Ok(mut buf) = allocate_read_buf(size) {
	let r = src.read(&mut buf)?;

But the trait Read is not unsafe, and it is never guaranteed to limit itself to writing to its single argument buf, and the result of allocate_read_buf is a vector of uninitialised bytes (that function should be unsafe, btw).

There were discussions about introducing a new unsafe trait in libstd to signal that a Read implementation doesn't read in the writer it's supposed to write to, I don't know what became of it but I just found out about rust-lang/rust#42788 which seems related.

Replace custom log! implementation with env_logger

This only exists because it was a pain to use other crates inside Gecko when mp4parse was first developed, but now it's easier and we have many existing users of env_logger inside Gecko.

This came up when @bholley asked why we're using this while he was looking at reducing Debug trait use in release logging.

Opus in MP4 file do not play

bugzilla issue https://bugzilla.mozilla.org/show_bug.cgi?id=1482841

fail to parse invalid init segment

From this gecko bug we fail to parse the init segment from http://dash.edgesuite.net/envivio/Envivio-dash2/v4_258-Header.m4s with "unexpected EOF".

This is invalid, but stagefright appears to handle it.

`mif1` compatible_brands interoperability problem

This parser, and therefore Firefox requires ftyp's compatbile_brands to contain mif1. Chrome and libavif don't require this.

AFAIK it's a recommendation in the HEIF spec, but it's not required: https://aomediacodec.github.io/av1-avif/#image-and-image-collection-brand

AVIF images with empty or different compatible_brands work in Chrome, but not in Firefox.

I suggest just dropping the check of compatbile_brands.

rfc: using mp4parse for mp4demuxer

Hello,

I'm about the write video packager and I'd like to use mp4parse as
base for parsing mp4 container. The reading part is pretty good, but
there's some parts which needs tweaks. And before I will send any
patches, I'd like to discussed them.

1. Not all boxes are accessible after reading

pub fn read_mp4
    while let Some(mut b) = iter.next_box()? {
        match b.head.name {
            BoxType::FileTypeBox => {
                let ftyp = read_ftyp(&mut b)?;
                found_ftyp = true;
                debug!("{:?}", ftyp);

		^^^ at this point freed and not accesable

in my PoC, I have extended internal data MediaContext and store box
there. But there are more of them.

2. fullbox's version and flags and not accessible as well

read_mdhd
   let (version, _) = read_fullbox_extra(src)?;


    Ok(MediaHeaderBox {
        timescale,
        duration,
    })

so when you want to implement write_mdhd(), you cannot do that without
version.

There's multiple options to fix this.

a) extend MediaHeaderBox struct with new member version or flags
(depends on box)
b) create new struct FullBoxHeader

   #[derive(Debug, Default)]
   pub struct FullBoxHeader {
       pub version: u8,
       pub flags: u32,
   }

and either return it along with parsed box, e.g.

    Ok(MediaHeaderBox {
        timescale,
        duration,
    }, fullbox_header)

or add new member into struct.

3. some bytes are skipped
   Some of bytes are skipped for good reasons, but some of them will trim
   output after writing box.

fn read_video_sample_entry
    // Skip uninteresting fields.
    skip(src, 50)?;

    // Skip clap/pasp/etc. for now.

4. low hanging fruits, are struct, that are not accessible outside of this module

Please let me know, if any of above changes will work for you.

--
Nikola

Replace potentially allocating functions with fallible alternatives

There are already some instances of fallible alternatives like vec_push, vec_with_capacity, extend_from_slice, and read_to_end. However, there are additional functions which may allocate such as Vec::append and slice::concat which need a similar treatment.

Nightly builds failing since 27th Dec

Looks like the build has been failing on nightly since the 27th Dec weekly cron job. The last good build was the 20th (previous weekly cron job), so that gives us the window for the nightly regression.

Alfredo noticed this in #135.

flac/opus specific box aren't allowed in a protected audio sample entry

This prevent the use of flac and opus in encrypted tracks.

BMO 1509875

Can't play AAC file with explicit SBR (audioObjectType == 5)

Multiple sample description boxes are not handled

When we parse the stsd box, we iterate over the sample descriptions contained in the stsd. Based on the code for parsing at the moment, I suspect that most mp4s contain a single description.

However, while working on support for handling parsing for the cbcs encryption scheme I've noticed that the test media I've created using shaka-packager has multiple description boxes.

Here are video files I created for testing. The init is the primary one of interest, but I've included the other least they're useful: bipbop_cbcs_video_files.zip

The files were created (along with some audio files and a manifest) from mozilla-central's bipbop.mp4 via the following:

packager-win.exe in=bipbop.mp4,stream=audio,init_segment=bipbop_cbcs_audio_init.mp4,segment_template=bipbop_cbcs_audio_$Number$.m4s in=bipbop.mp4,stream=video,init_segment=bipbop_cbcs_video_init.mp4,segment_template=bipbop_cbcs_video_$Number$.m4s --protection_scheme cbcs --enable_raw_key_encryption  --keys label=:key_id=7e571d047e571d047e571d047e571d21:key=7e5744447e5744447e5744447e574421 --iv 11223344556677889900112233445566 --generate_static_mpd --mpd_output bipbop_cbcs.mpd

The first issue I'm hitting on is that the codec_type we set on the track is set based on the last description parsed. This means for the test media above, the video track is labelled as H264, rather than EncryptedVideo.

Furthermore, we only store the first encountered description (contrast this that the type appears to be derived from the last encountered one).

I'm no expert on the format, and am still trying to understand all the ins and outs of how these descriptions are used, but I imagine if both descriptions are present we should store them, as they may be referenced by fragments later.

While I continue trying to get my head around the details, I figured I'd create an issue for discussion on how to handle this. @kinetiknz, @jyavenard, do you have thoughts?

Add non-av01 primary item support

While it's typically the case that the "primary item" (indicated by the pitm box) of an AVIF file is an item of type av01, that is not always the case. Alternatively, it can be an item of type grid (and possibly others).

tkhd should parse track height and width as fixed point numbers

the tkhd parser currently parses the width and height fields of a track as u32.

According to the MPEG4 spec, these fields are fixed-points.

This leads to the parsed values being off by a factor of 65536.

It looks like this is handled in the C API by shifting the number right 16 bits

Unfortunately, Rust doesn't provide built-in fixed-point support but it looks like a couple of crates implement it (see fixed), but this would require adding a dependency. I'm also not sure if that crate provides an interface for parsing fixed-point numbers from binary.

Wiki changes

FYI: The following changes were made to this repository's wiki:

• defacing spam has been removed

• Restricting write access to contributors is strongly encouraged. Please make that change (documentation).

These were made as the result of a recent automated defacement of publically writeable wikis.

Assert in `read_iloc` triggered by fuzzing

When the avif fuzz target is run long enough, eventually the debug_assert! in read_iloc is triggered. It's not urgent, but it'd be nice to avoid this so the fuzzer can continue exploring.

In the case I hit, item_count is 0, so we skip the item reading loop and immediately hit the assert with iloc.remaining() returning 8. Testcase saved locally, if required.

Use nightly's try_reserve when available

Fallible allocation API just went live in nightly and it's breaking the package because of a name conflict with mp4parse_fallible's try_reserve.
Reference: https://bugzilla.mozilla.org/show_bug.cgi?id=1446538

Can this be used for replacing flv.js mp4-generator?

Hi! we are trying to make a wasm version of flv.js, and I'm working on rewriting flv.js' mp4generator.js in rust. Since this is my first time to work on mutimedia, I don't know much about how media is parsed. Would you give me some advice? thanks!

Publish new version?

Would it be possible to publish a new version? I'd like to use the recently implemented alac support in a crate on cargo.io.

read adaptor for mp4_demuxer::Stream

While our initial target is to parse metadata for MSE, the gecko interface expects a stream, so we need to implement some kind of adaptor which implements Read on the rust side and calls back to an mp4_demuxer::Stream on the C++ side.

https://dxr.mozilla.org/mozilla-central/source/media/libstagefright/binding/include/mp4_demuxer/Stream.h

Revisit need for {BUF,TABLE}_SIZE_LIMIT

I tweaked BUF_SIZE_LIMIT up from 1MB to 10MB in #188, but it's not clear that {BUF,TABLE}_SIZE_LIMIT are still necessary and arbitrarily set limits are always at risk of being incorrect.

BUF_SIZE_LIMIT was introduced in dbb6e50
TABLE_SIZE_LIMIT was introduced in 65e24b4

Fallible allocation for "large" allocations was introduced later, in c9a4a1c. With this in place we have the option to rely on allocations failing directly, rather than the prior strategy of trying to avoid OOM panics via the {BUF,TABLE}_SIZE_LIMIT mechanism.

Recommendations for streaming?

I'm thinking about creating a streaming buffer that implements Read, and uses something like fringe or context to suspend mp4parse until more data is available.

The data is coming in asynchronously from a web server and I have to stream chunks into mp4parse-rust (I can not save all of the chunks to disk and pass the complete file to mp4parse).

Any other suggestions?
I was hoping someone could point me to something similar...

Thanks!

Implement ConstructionMethod::ItemOffset

See ISO 14496-12:2015 § 8.11.3

Implement `esds` parsing

The media.rust.test_mode pref in Gecko found another mismatch between mp4parse and libstagefright. From a Facebook live stream, a moov with a video and audio track results in mp4parse returning a samplerate of 44100 and libstagefright returning a samplerate of 22050.

mp4parse is parsing the samplerate directly from the mp4a. libstagefright prefers the value from the esds.

TryBox is unsound

TryBox assumes that Box::drop will use libc::free to deallocate memory. This is not true if a crate uses jemallocator, or any custom allocator.

This could be fixed with:

1. Not using Box for inner, but keeping the raw pointer, and implementing Drop for TryBox that calls libc::free

2. or using std allocator API to use the same allocator as Box

I think option 1 is the simplest and makes least assumptions about Rust inner workings.

There's a more theoretical issue of TryBox not checking alignment. malloc gives alignment for largest platform-native type, but Rust types might ask for bigger alignment. An assert would suffice for this implementation.

std::mem::forget(std::mem::replace can be simplified to ptr::write.

How to get info?

After using the read_mp4 function, what am I supposed to do with the MediaContext struct since all its fields are hidden? And this is despite the fact the doc says:

Metadata is accumulated in the passed-through MediaContext struct, which can be examined later.

Panic when using read_box

extern crate mp4parse;

use std::io::Cursor;

fn main() {
    let mut c = Cursor::new(b"\x00\x00\x00\x01\x30\x30\x30\x30\x00\x00\x00\x00\x00\x00\x00\x00".to_vec());
    let mut context = mp4parse::MediaContext::new();
    let _ = mp4parse::read_box(&mut c, &mut context);
}

coreyf@frewbook-pro /t/mp4 (master)> cargo run
     Running `target/debug/mp4`
thread '<main>' panicked at 'assertion failed: size64 >= 16', /private/tmp/mp4parse-rust/src/lib.rs:273
Process didn't exit successfully: `target/debug/mp4` (exit code: 101)

panic discovered using afl.rs

5.0 release

When do you think the 5.0 version will be released?

Implement AudioSampleEntryV1

There is a new version of the AudioSampleEntry which allows specifying samplerates over 65 kHz. This is apparently uncommon, with current encoders relying on codec override of the container metadata, but we should probably support it for the sake of flac/modern profiles.

Panic when using read_box

extern crate mp4parse;

use std::io::Cursor;

fn main() {
    let mut c = Cursor::new(b"\x00\x00\x00\x13\x66\x74\x79\x70\x30\x30\x30\x30\x30\x30\x30\x30".to_vec());
    let mut context = mp4parse::MediaContext::new();
    let _ = mp4parse::read_box(&mut c, &mut context);
}

panic discovered using afl.rs

Review use of Debug formatting and avoid in release-mode logging

Related to issue #139, @bholley is working on reducing Debug trait use in release builds (for code size reasons). He proposed:

what we do in stylo is use debug! and trace! for stuff where we need/want :?, and then avoid :? in info/warn/log which are compiled in release builds

So we'll need to review existing log! use and replace with debug!/trace! as necessary and remove any use of :? inside log!.

Improve mp4parse_capi test coverage

Since we wrote the C API tests in test.cc and afl-capi.rs, we've added mp4parse_get_indice_table, mp4parse_get_fragment_info, mp4parse_is_fragmented, and mp4parse_get_pssh_info to the C API. This issue covers updating those tests to cover the new API surface.

Longer term, we should probably introduce a policy that requires updating the tests as the API is extended. And/or try to move more of the code into the Rust side and make the C API code as slim as possible.

Related: https://bugzilla.mozilla.org/show_bug.cgi?id=1342848

Update public.rs

We've added a bunch of public structs and members since public.rs was written. Probably we should update that at least so we're clear if we need to bump semver because we removed a public interface.

Republish + regenerate documentation as API has changed

Since the docs were last generated the API has undergone some changes. Notably #163 has moved some data that was previously stored per track to per sample info (offering finer granularity). If we republish the lastest version of the crate the docs could be regenerated for the new version to avoid footguns (which I may have just shot myself with).