Low FPS Latency about amazon-kinesis-video-streams-producer-sdk-cpp

[bdhandap]

@francoisdutoit

Unfortunately, using the HTML5 video tag gives you very little control over the buffering behavior of the browser. Even if your MKV is well formed, the browser may have a heuristic to buffer n number of frames (as opposed to n number of seconds) of content before starting playback. To make matters worse, each browser has different heuristics. To gain fine grained control over the buffering behavior (like Amazon Video, Netflix, etc. do), you must use Media Source Extensions. The problem with MSE, is that the content must be mp4/h264 or h265 or webm/VP8 or VP9, not webm/h264 as it appears you are currently doing. So to use MSE, you need to convert the content to MP4 (like what happens behind the scenes in the KVS console), or switch to using VP8 (but the KVS Producer SDK does not really support it). Another possible approach is to render the h264 into jpegs on your backend and stream those to the browser client and render them in a HTML5 canvas element. We will be continuing to improve video playback support, but at this moment these are the options.

[bdhandap]

@francoisdutoit How are you getting the fragments? Are you using the real-time GetMedia API? And the delay which you mentioned is per fragment or per frame?

[francoisdutoit]

@bdhandap
Yes - I am using the GetMedia API for .Net as below

    public async Task<Stream> RetrieveMediaAsync(string dataEndPoint)
    {
        var videoMediaClient = new AmazonKinesisVideoMediaClient(dataEndPoint, _credentials);

        var mediaRequest = new GetMediaRequest()
        {
            StreamARN = _streamARN,
            StartSelector = new StartSelector
            {
                StartSelectorType = StartSelectorType.NOW
            },
        };

        GetMediaResponse getMediaResponse = new GetMediaResponse();

        getMediaResponse = await videoMediaClient.GetMediaAsync(mediaRequest);

        return getMediaResponse.Payload;
    }

Not sure if the delay is per frame of fragment - the video is just about 1 minute behind real time.

[zhiyua-git]

@francoisdutoit Can you provide the log in console so we can get more understanding on this? We can see if there is something in producer SDK causing this high latency.

[francoisdutoit]

@zhiyua-git

Attached is the log as requested

1FpsLog.txt

[MushMal]

@francoisdutoit Looking at the logs you've provided I see no delay - the buffer is 0 so your data is received in real-time as it's produced into the KVS producer. I do see some issues with mapping the retrieved ACKs back to the timeline - can you please confirm whether you are using the latest bits?

We should look on the consuming side whether the timecodes are properly specified.

Also, can you please let us know how do you detect the latency?

[francoisdutoit]

@MushMal

How do a specify timecodes on the consumer side...?

Attached is a video of my app running. The timestamp on the video is about 50 seconds behind my windows clock - so this video is 50 seconds old...... So if i move my hand in front of the camera i will only see it 50 seconds later in my app.

Capture.zip

[MushMal]

Sorry for the delay in my response. I don't see any obvious issues yet - it seems that you are specifying StartSelectorType.NOW so it should be no latency. The frames are all key frames and they are about a second apart.

What's the behavior in the KVS console? Do you still see the same 50 sec delay?

[francoisdutoit]

I will check the delay on the KVS console when I get home - but think it was the same issue. What is weird for me is that at 24fps i don't have the issue on the client side - only when I'm streaming at 1fps....

[francoisdutoit]

@MushMal

So on the console - initially it starts out with about a 10second delay which increases over time - I left it for a while and the console is now 30 seconds behind real time.

I read this post with someone having a similar issue - https://www.wowza.com/community/questions/835/long-buffering-with-1-fps-video.html and they make a mention of an H.264 issue....

However - on the console - when I click the jump to live end it takes the stream back to about seven seconds from real time - It seems like it buffers at some points and then continues from where it was buffering....

[MushMal]

From what I can see is that you have 1 second fragment duration. Each fragment has a single frame. However, the frame duration as I observed in the log is over 1 second - it's close to 1.9 seconds.

I would suggest checking your media pipeline.

[francoisdutoit]

@MushMal
I'm streaming straight from an ip camera....Can you maybe tell me how I can go about checking the media pipeline. What can cause the frame to be longer than one second?

Also - why do you think this is only happening at 1fps and not at 24fps?

[MushMal]

Could you provide your device make/model?

What I am afraid is happening is that many playback engines will need to decode/fill certain threshold in the buffer before they send the decoded frame to A/V sync and hardware surface. This might explain the difference in latency observed between different playback engines (browsers MSE vs JCodec).

I see you are using C# client - we have a Java-based parser library which could give you some understanding of the raw latency of the frame data retrieved from the stream.

https://github.com/aws/amazon-kinesis-video-streams-parser-library

[francoisdutoit]

Below are the device details - but the latency is also experienced in the AWS console.....so dont think the issue is on the client side....

It a Dahua IPC-HFW1120S,

Device Type
IPC-HFW1120S
Software Version
2.420.0000.19.R, Build Date: 2016-06-24
WEB Version
3.2.1.355658
ONVIF Version
2.4.2

[bdhandap]

Thank you. Let me try to find out some info on the specs and hope to get the actual camera to play with. We will also try to configure some other devices to produce low FPS stream here locally. On Mar 23, 2018, at 12:48 AM, francoisdutoit <[email protected]<mailto:[email protected]>> wrote: Below are the device details - but the latency is also experienced in the AWS console.....so dont think the issue is on the client side.... It a Dahua IPC-HFW1120S, Device Type IPC-HFW1120S Software Version 2.420.0000.19.R, Build Date: 2016-06-24 WEB Version 3.2.1.355658 ONVIF Version 2.4.2 - You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#54 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AggA7VKMz1nsvL3XspwnwpY3OSrC_Tx3ks5thKijgaJpZM4S0wO0>.

[bdhandap]

@francoisdutoit Please note that the video played in Kinesis Video streams console is not real-time and it's getting played from the data retention store. However if you are using GetMedia real-time API then you should be seeing frame latency around 25ms and fragment latency around 250ms. We'll take a look at the backend logs and get back to you.

Also how much latency did you face when you were streaming at 24fps?

[francoisdutoit]

At 24fps it was between 4 and 6 seconds. I understand that the AWS console is not real time - but what I see on the aws console and in my client app is about exactly the same....it does not seem to be any more real time in the client than in the AWS console. The code I am using above is for the real time API ...correct?
Thanks again :) Hope you can help me figure out how to get this issue sorted :)

[francoisdutoit]

@bdhandap @unicornss
Hi - Have you been able to replicate the issue?
Thnanks!

[bdhandap]

@francoisdutoit We are not able to reproduce in our lab and our devices are able to stream with low latency. Can you send your aws request id, stream name and account id to [email protected]? I'll take a look at the logs and get back to you.

Thanks
Babu

[francoisdutoit]

@bdhandap

So I have done a bit of investigation on the client side...

If I make a direct call to my API - by entering http://localhost:44375/GetVideoAsync in the browser - the stream data starts to comes through after about 2-3 seconds - So we are getting the data from kinesis pretty quickly. ( see attached stream data from browser )

now - in my html - I have the following in a video tag

video src="http://localhost:44375/GetVideoAsync" type="video/webm" autoplay="autoplay"

With the above - at 24fps - there is a very short delay before the video starts and at 1fps - it takes about 50 seconds - which seems to be constant...it always seems to be about 50 seconds...

So there does seem like some sort of buffering - like the player is waiting for at least 48 frames....

On your side - were you able to stream the low fps in a browser without a delay?
What player are you using?

AwsBrowser.txt

[francoisdutoit]

@bdhandap

Just another observation - if I start my stream at 24fps and open my web application and let the stream start - and then while streaming turn my webcam to 1fps - it switches over seamlessly and the 1fps stream in the web app is only a about 2-3 seconds behind.... which is what i want

What do you think about that behavior?....

[francoisdutoit]

@bdhandap @unicornss

Ok - so it must be the html5 video player - what I can do in javascript is to speed up the video to catch up to "real time". So that means the stream is already available before it starts showing.
I can write some javascript to bring the player to the correct timestamp - i do still however have to wait for 50seconds before the video starts - which is a problem.
I think you can close this case as the problem is obviously on the client side - can you however comment on the below..
1.) On your side - were you able to stream the low fps in a browser without a delay?
2.) What player are you using?
3.) Suggestion for a real time player that can be embedded that wont have the same issue?

Thanks!

[francoisdutoit]

Thanks. I will play around a bit and see what solution works best. Another option could be to prepend the stream in my back-end with 2 seconds of 24fps video....like just blank video to get the stream initialized quicker on the client side. Will keep you posted regarding my solution. Thanks again for the help :)