zergon321 / reisen

A simple library to extract video and audio frames from media containers (based on libav).

Reisen

A simple library to extract video and audio frames from media containers (based on libav, i.e. ffmpeg).

Dependencies

The library requires libav components to work:

• libavformat
• libavcodec
• libavutil
• libswresample
• libswscale

For Arch-based Linux distributions:

sudo pacman -S ffmpeg

For Debian-based Linux distributions:

sudo add-apt-repository ppa:savoury1/ffmpeg4
sudo apt install libswscale-dev libavcodec-dev libavformat-dev libswresample-dev libavutil-dev

For macOS:

brew install ffmpeg

For Windows see the detailed tutorial.

Installation

Just casually run this command:

go get github.com/zergon321/reisen

Usage

Any media file is composed of streams containing media data, e.g. audio, video and subtitles. The whole presentation data of the file is divided into packets. Each packet belongs to one of the streams and represents a single frame of its data. The process of decoding implies reading packets and decoding them into either video frames or audio frames.

The library provides read video frames as RGBA pictures. The audio samples are provided as raw byte slices in the format of AV_SAMPLE_FMT_DBL (i.e. 8 bytes per sample for one channel, the data type is float64). The channel layout is stereo (2 channels). The byte order is little-endian. The detailed scheme of the audio samples sequence is given below.

You are welcome to look at the examples to understand how to work with the library. Also please take a look at the detailed tutorial.

reisen's Issues

error: 'libavcodec/avcodec.h' file not found

Hi,i wonder to use reisen on my Mac,but unfortunately i encountered some problems ：

Looking forward to your reply, thank you！

Stop the execution

HI!

How I can stop the execution of play?

Thank you

interesting observation regarding framerates

I found a small issue with your player example code:

videoFPS, _ := media.Streams()[0].FrameRate()
spf := 1.0 / float64(videoFPS)
frameDuration, err := time.ParseDuration(fmt.Sprintf("%fs", spf)) 

You are ignoring the denominator here which is fine for your sample video but it should be considered for the general case.

But it gets more interesting yet. Some videos come with valid frame rate of 29.97. In this case the nominator will be 30000 and the denominator will be 1001 (which again is correct). However, if I play a video with that frame rate I observe the increasing drift between video and audio I reported in a previous issue. It gets better: If I manually force the frame rate to 30 instead of 29.97 the drift between audio and video is gone!

I observed the same for videos with 23.98 frame rate: There is a drift unless I force the player to play at 24 frames per second.

Currently, I'm working with mp4 snippets I download from youtube. Of course there is a possibility that the metadata is wrong in my mp4s but I don't think so. Other video players play those videos correctly - but then again those other players may do a better job syncing audio and video and ignore the possibly flawed frame rate data in my videos.

Anyways, just wanted to bring this to your attention and see if you have any insight why I have to "round up" 29.97 to 30 and 23.98 to 24. Thanks!

any mechanism to keep video and audio in sync while playing ?

i checked you video player example , did not found anything related to video and audio syncing according to pts ?

good job ,found another project may be useful .

nice work , simple and clean . i have just read your article on Medium , learned your findings . i used to play with this Go Media Framework . it seems a full-featured binding , but not very active . wanna take a look ?

Re-enable macOS support

Thanks for the great project. I used the tagged version but wanted to contribute so switched to master, where I had some trouble:

On master the project no longer works on darwin, but it does in 0.1.1.
The change for a windows fix moved audio.go (and video.go) to a -windows and -linux suffix, but there is no -darwin so compile fails.

panic: buffered: len(pix) was 8294400 but must be 3686400

Hi! Thank you for this great project! I've tried your player code with your test mp4 and it works as intended. But when I try running it with any other mp4 video file I consistently get the error panic: buffered: len(pix) was 8294400 but must be 3686400. I noticed the factor between those two numbers is exactly 2.25. Any idea what the issue might be here? Thank you!

Support for Windows

Hello,
I have just discovered this repo. Is there any plan to support shared library(by calling .dll, .so, .dylib) and support for windows ? There was a WIP project that has not been updated over a year. I tried it but it has some audio skipping issues.
Thank you for this work

drift between video and audio stream in example player

Another question after spending more time with the video player example.

When playing longer videos I notice a slight drift between the video and the audio stream over time. This becomes even more pronounced when I add some extra image processing. I think I understand what the issue is: Essentially the video frame processing and the audio playing take place on their own go routines without any synchronization between them. So I guess the video processing could fall behind, while the audio processing keeps up? I understand that the player is just a very simple example, so this behavior is probably not surprising. Nevertheless any recommendation how to synchronize audio and video better would be great.

Also, just for my better understanding, I noticed that media.ReadPacket() will return audio and video packets at random for all open streams. Most of the time video and audio packets will alternate perfectly although every once in a while I see 2 video packets for 1 audio packet. What is the rational here in terms of stream synchronization?

"libavcodec/bsf.h" not found

Hello,

When trying to build under Ubuntu 20.04, I encounter the following error:

$ go build .
# github.com/zergon321/reisen
./media.go:8:11: fatal error: libavcodec/bsf.h: No such file or directory
    8 | // #include <libavcodec/bsf.h>
      |           ^~~~~~~~~~~~~~~~~~
compilation terminated.

The required libs are installed:

$ sudo apt install libswscale-dev libavcodec-dev libavformat-dev libswresample-dev libavutil-dev
libavcodec-dev is already the newest version (7:4.2.7-0ubuntu0.1).
libavformat-dev is already the newest version (7:4.2.7-0ubuntu0.1).
libavutil-dev is already the newest version (7:4.2.7-0ubuntu0.1).
libswresample-dev is already the newest version (7:4.2.7-0ubuntu0.1).
libswscale-dev is already the newest version (7:4.2.7-0ubuntu0.1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

But indeed, there is no bsf.h anywhere in /usr/include and subfolders. I copied bsf.h manually from ffmeg/libavcodec source code (as well as a few other needed headers), but the compilation crashes as there is always some source code missing.

No ubuntu 20.04 package seems to supply bsf.h:

$ apt-file search bsf.h
bsh-doc: /usr/share/doc/bsh-doc/html/bsf.html
libc-client2007e-dev: /usr/include/c-client/os_bsf.h
nvidia-cuda-doc: /usr/share/doc/nvidia-cuda-toolkit/html/libdevice-users-guide/__nv_fabsf.html

Could it be because the libavcodec-dev provided by Ubuntu 20.04 is too old? Any suggestion?

Request support for seek to a particular position in video

Currently, reisen can only play the full video from the input file. Can you please add ability to seek to any position in the video?

Can we use Go arena?

This is just an enhancement request.

Go has arena support now:
golang/go#51317
https://www.reddit.com/r/golang/comments/sys1pr/proposal_arena_new_package_providing_memory_arenas/

I am just wondering if using arena would help improve performance. Please do the needful.

Please ignore if this is not relevant or needed.

Support for Pipe Input/Output

Library looks great, I'm wondering if there will be a way to have pipes to somehow work! More specifically for doing this kind of thing https://github.com/pion/example-webrtc-applications/blob/master/gocv-receive/main.go

ffmpeg := exec.Command("ffmpeg", "-i", "pipe:0", "-pix_fmt", "bgr24", "-s", strconv.Itoa(frameX)+"x"+strconv.Itoa(frameY), "-f", "rawvideo", "pipe:1")

Where they read/write the ffmpeg as input/output pipe. Found something over stackoverflow, other or gist but not sure what would be the right approach to support this.

Request support for streaming video

Currently reisen supports only a file. Can you please add streaming support?

OSX example fails with panic: time: invalid duration "NaNs"

tried multiple files, same error.

go run main.go
Duration: 2m30.102176s
Format name: mov,mp4,m4a,3gp,3g2,mj2
Format long name: QuickTime / MOV
MIME type: 
Number of streams: 2

panic: time: invalid duration "NaNs"

goroutine 1 [running]:
main.handleError(...)
        <omitted>/main.go:142
main.main()
        <omitted>/main.go:28 +0x11bf
exit status 2
------
Duration: 4m0.140416s
Format name: matroska,webm
Format long name: Matroska / WebM
MIME type: audio/webm,audio/x-matroska,video/webm,video/x-matroska
Number of streams: 2

panic: time: invalid duration "NaNs"

goroutine 1 [running]:
main.handleError(...)
       <omitted>/main.go:142
main.main()
        <omitted>/main.go:28 +0x11bf
exit status 2

I ran libav as the README noted:

🍺  /usr/local/Cellar/libav/12.3_8: 15 files, 32.4MB
==> Running `brew cleanup libav`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
==> No outdated dependents to upgrade!
==> Checking for dependents of upgraded formulae...
Disable this behaviour by setting HOMEBREW_NO_INSTALLED_DEPENDENTS_CHECK.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
==> Reinstalling 1 dependent with broken linkage from source:
libav
Warning: libav has been deprecated because it is not maintained upstream!

Cant run go get reisen. ERROR: could not determine kind of name for C.av_bsf_receive_packet and C.av_bsf_send_packet

Getting errors when I go get reisen using go get github.com/zergon321/reisen

using brew install libav did not work for me, so tried building ffmpeg from source using brew install ffmpeg --build-from-source which installed all the library and header files, now i get all the following libraries in /usr/local/include and /usr/local/lib

• libavformat
• libavcodec
• libavutil
• libswresample
• libswscale

but I’m getting the following error after running go get github.com/zergon321/reisen
../../go/pkg/mod/github.com/zergon321/[email protected]/media.go:207:12: could not determine kind of name for C.av_bsf_receive_packet

../../go/pkg/mod/github.com/zergon321/[email protected]/media.go:199:12: could not determine kind of name for C.av_bsf_send_packet

I can confirm that all dependencies are install correctly.
please whats happing and can anything be done to resolve this, thank you

Build on Docker Fails

Trying to build on Docker and running into the following issue:

#18 4.671 # github.com/zergon321/reisen
#18 4.671 /go/pkg/mod/github.com/zergon321/[email protected]/audioframe.go:17:27: undefined: Stream
#18 4.671 /go/pkg/mod/github.com/zergon321/[email protected]/frame.go:17:15: undefined: Stream
#18 4.671 /go/pkg/mod/github.com/zergon321/[email protected]/unknown.go:9:2: undefined: baseStream
#18 4.671 /go/pkg/mod/github.com/zergon321/[email protected]/videoframe.go:24:27: undefined: Stream

What I've tried:

• Pegging Golang version to 1.17
• Adding necessary dependencies

Here is what the relevant part of my Dockerfile looks like:

# Use the official Golang image as the base image
FROM golang:1.17 AS builder

# Set the working directory inside the container
WORKDIR /app

# Install necessary packages
RUN apt-get update && apt-get install -y software-properties-common
RUN add-apt-repository ppa:savoury1/ffmpeg4
RUN apt install -y libswscale-dev libavcodec-dev libavformat-dev libswresample-dev libavutil-dev

# Copy the Go module files
COPY go.mod go.sum ./

# Download and cache the Go module dependencies
RUN go mod download

# Copy the source code into the container
COPY . .

# Build the Go application
RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o vidstreamsplit .

Let me know if you've run into this issue before. Happy to try and make a change if you could point me in the write direction.

How should I utilize hardware decoding and encoding with this library?

Thank you very much for providing this library. I encountered a little issue during my learning process. I would like to know how to use hardware decoding and encoding because my device supports hardware decoding. Could you please let me know?

Memory corruption issue

Hi! The following video crashes the player (with width and height constants adjusted to 1024 and 576) quite consistently:

catfight.mp4

It looks like a heap corruption (most likely a buffer overflow somewhere) and happens on Windows and Linux. When it does not crash, one can also see some corrupted pixels at the top of the screen. See screenshot:

The ffplayer utility plays the video without visible issues.

Linux crash message looks like this:

corrupted size vs. prev_size
Thread 8 "player" received signal SIGABRT, Aborted.

The call stack looks like this in GDB/Linux:

#0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=0x6, no_tid=no_tid@entry=0x0) at ./nptl/pthread_kill.c:44
#1  0x00007ffff5e8989f in __pthread_kill_internal (signo=0x6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
#2  0x00007ffff5e3da52 in __GI_raise (sig=sig@entry=0x6) at ../sysdeps/posix/raise.c:26
#3  0x00007ffff5e28469 in __GI_abort () at ./stdlib/abort.c:79
#4  0x00007ffff5e7dc18 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7ffff5fb4689 "%s\n") at ../sysdeps/posix/libc_fatal.c:155
#5  0x00007ffff5e9355a in malloc_printerr (str=str@entry=0x7ffff5fb2161 "corrupted size vs. prev_size") at ./malloc/malloc.c:5531
#6  0x00007ffff5e93eb6 in unlink_chunk (p=<optimized out>, av=0x7fffa4000020) at ./malloc/malloc.c:1626
#7  0x00007ffff5e94040 in malloc_consolidate (av=av@entry=0x7fffa4000020) at ./malloc/malloc.c:4663
#8  0x00007ffff5e95d88 in _int_malloc (av=av@entry=0x7fffa4000020, bytes=bytes@entry=0xa10) at ./malloc/malloc.c:3848
#9  0x00007ffff5e96e9f in _int_memalign (av=av@entry=0x7fffa4000020, alignment=alignment@entry=0x40, bytes=bytes@entry=0x9a7) at ./malloc/malloc.c:4851
#10 0x00007ffff5e9752a in _mid_memalign (alignment=alignment@entry=0x40, bytes=bytes@entry=0x9a7, address=<optimized out>) at ./malloc/malloc.c:3453
#11 0x00007ffff5e98a2f in __posix_memalign (size=0x9a7, alignment=0x40, memptr=0x7fffbe1c03b0) at ./malloc/malloc.c:5557
#12 __posix_memalign (memptr=0x7fffbe1c03b0, alignment=0x40, size=0x9a7) at ./malloc/malloc.c:5541
#13 0x00007ffff7e05aad in av_malloc () at /lib/x86_64-linux-gnu/libavutil.so.57
#14 0x00007ffff7dd2d15 in av_buffer_alloc () at /lib/x86_64-linux-gnu/libavutil.so.57
#15 0x00007ffff7dd2d8e in av_buffer_allocz () at /lib/x86_64-linux-gnu/libavutil.so.57
#16 0x00007ffff7dd3656 in av_buffer_pool_get () at /lib/x86_64-linux-gnu/libavutil.so.57
#17 0x00007ffff6795da4 in  () at /lib/x86_64-linux-gnu/libavcodec.so.59
#18 0x00007ffff679a148 in  () at /lib/x86_64-linux-gnu/libavcodec.so.59
#19 0x00007ffff679eca5 in  () at /lib/x86_64-linux-gnu/libavcodec.so.59
#20 0x00007ffff667e841 in  () at /lib/x86_64-linux-gnu/libavcodec.so.59
#21 0x00007ffff667f4c0 in avcodec_send_packet () at /lib/x86_64-linux-gnu/libavcodec.so.59
#22 0x0000000000573c3b in _cgo_c6952d519846_Cfunc_avcodec_send_packet (v=0xc000233628) at /tmp/go-build/cgo-gcc-prolog:276
#23 0x00000000004722a4 in runtime.asmcgocall () at /usr/lib/go-1.19/src/runtime/asm_amd64.s:844
#24 0x0000000000000001 in  ()
#25 0x0000000000443900 in runtime.fatal at /usr/lib/go-1.19/src/runtime/panic.go:1062
#26 0x0000000000470429 in runtime.systemstack () at /usr/lib/go-1.19/src/runtime/asm_amd64.s:492
#27 0x0000000000000000 in  ()