Large delay (10-20s) in opening stream about homebridge-unifi-protect

[hjdhjd]

@brianmedia Short answer is no. The HomeKit API spec defines what codecs and resolutions must be supported, with a maximum of 1920x1080 (no 4k for us quite yet...grumble...as an owner of half a dozen G4 Pros this gives me no joy).

You can read a longer thread here on others who have struggled with it over the years:
Sunoo/homebridge-camera-ffmpeg#181 (comment)

The TL;DR version: you can't tell the Home app on any platform what resolution it must pull. What you are seeing in the log files is what HomeKit not this plugin is selecting. It's also why I chose to implement some sane defaults for this plugin rather than always go for the highest quality RTSP stream...which believe me, I'd love to do if it made any sense.

The reality is that the best you are going to ever do in HomeKit, today, is 1920x1080 under whatever HomeKit decides is ideal circumstances. I'm going to guess that come this fall, that ideal will be an ethernet-connected Apple TV 4K and in that configuration, perhaps you'll see 1920x1080.

For most people, most of the time, you're going to get something substantially less than that.

Which leads to why for G4-series cameras, I've chosen to leadoff with a Medium quality RTSP stream rather than try to pull a 4K stream and force ffmpeg to try to keep up with transcoding that into what HomeKit expects. It really makes no sense to tax your hardware that hard for something that you'll never see...and it slows things down more, because transcoding isn't free in either CPU cycles or (more crucially at least for me) in responsiveness and speed. You want data streaming as fast as possible to it's final destination, so I try to select the streams that are most likely to help in getting to that goal.

HomeKit is a journey, not a destination, and video quality is definitely a long and winding road. I'm by no means an expert here...I've just done a lot of looking, testing, digging, and experimenting to get to the above conclusions and I'd love to improve my own body of knowledge if anyone definitively knows better in this space.

Ultimately, I want this to be the very best plugin it can be so we can have as close to a first-class experience as you can get within the Protect ecosystem on HomeKit. That means sane defaults and thinking through things like the above...but still giving those who want to, the ability to adjust them to their heart's content.

It's why I spent weeks reverse engineering the realtime notifications API (alright...a little bragging here...but it was so much fun to figure out!) to create the first plugin of it's kind with true, realtime motion event support.

And being able to dynamically add/remove cameras to Protects without having to reboot...and a couple of more things to come, hopefully in the next week or two.

Finally (and I get this is getting to be a long post...)...the ffmpeg delayed streaming issue is a real one. You can also see that it's an issue on the homebridge-camera-ffmpeg issue list as well, which our streaming support heavily (as in nearly completely) borrows from. So be assured that others are working this too.

Hope this helps.

[hjdhjd]

Known issue. It's an artifact of the switch to the new Homebridge APIs. I'm actively working to improve this. It's not an ffmpeg issue and I wouldn't adjust the ffmpeg command line for this. This isn't an ffmpeg problem - I have the same issue and I'm actively working this one and hoping to push an additional update as soon as I finish troubleshooting it.

[hjdhjd]

@Vortec4800 It's in an upcoming update...had the same thought. 😄

[hjdhjd]

@maydaydarkhorse I'm familiar and the dev and I have had a few conversations on the topic. The issues aren't just related to using a copy versus reencoding, though that's a portion of the challenge. As soon as I have something for folks to test, I will share it. As someone with 20 cameras in his environment, believe me, there's nothing I'd like more than to round out this area in the plugin. 😄 Bear with me.

[hjdhjd]

Sounds better. 😄 Closing this issue out.

[tillkruss]

Same here. 10-15s. Homebridge runs on Raspberry Pi 4B.

[hjdhjd]

One more question - what's the Protect platform you guys are using? I'm trying to get a sense of which Protect controllers most people are using (UCK, UDMP, UNVR, etc.)?

[tillkruss]

I'm also using a UniFi Cloud Key Gen2 Plus.

[sonoseco]

I'm having the same issue with a Cloud Key Gen2 Plus. In addition to the delay with the streams, the snapshots are taking about 15-30 seconds to initially appear in the Home app and about 30 seconds between refreshes. Before it was updating the camera snapshot every 10 seconds while the app was opened, not sure if it is related to the same streaming bug.

[markPHL]

I'm also on the UniFi Cloud Key Gen2 Plus and experiencing the same behavior.

[brianmedia]

In my testing the stream is set to "High" in initialization.

If homekit loads a 1920x1080 stream it works
If homekit tries to load a 1280x720 stream it does not.

My Apple TV and iPad both load 1920x1080 streams and works correctly.
My iPhone tries to load a 1280x720 stream and fails.

I've changed config to "Disable.Stream.Medium" (and Low), restart Homebridge, but iPhone still tries to load a Medium stream. I've also tried setting "Enable.StreamOnly.High" but that doesn't work either. iPhone still loads 1280x720

UDM Pro: 1.7.2
Unifi Protect 1.14.11

[maydaydarkhorse]

Same problem here, camera snapshots work but can never get them to stream.

UDM Pro: 1.7.2
Unifi Protect 1.14.11

[hjdhjd]

Thanks @maydaydarkhorse - it's a known issue. I'll post here when there's an update to test against.

[brianmedia]

@hjdhjd do you have any info on how Homekit chooses the stream quality? Below is what my devices request 100% of the time.

iPad - 1920
Apple TV - 1920
iPhone - 1280
Mac Mini / Macbook - 1280

[brianmedia]

Thanks for the great info @hjdhjd

So strange that both my Apple TV 4K and iPad are on wireless connections yet always pull a 1920 feed
Yet I have two Macs wired Ethernet that will always get 1280. And my cameras are obviously hard wired into same UDM Pro as well.

[tronny87]

@hjdhjd , I commend you on your amazing work. I was previously using a different plugin for UnIFi Protect Motion on my RPi but when I switched to my Synology NAS it was no longer supported due to its dependancy of Tensorflow.

I'm glad to finally see someone take advantage of the UniFi API to bring motion alerts to Homebridge.

My only gripe is with the delay in initiating a HomeKit stream as well as the video latency itself. Unfortunately, I've opened issues here regarding these issues before and have had my threads closed with the blame being put on my possibly under-powered NAS processor and ffmpeg config.

Re: the stream start delay and the video latency, (and I know I've suggested it before), maybe you should look at how the homebridge-ring plugin is built, as video stream initiation is very snappy, as well as very good latency in the stream itself (very good for it being streamed via Ring's servers. Another feature that plugin is using that may be beneficial in looking into is the two-way audio. That developer still has some kinks to work out (there is a audio lag he has not been able to quite figure out yet) but it more or less works well.

I recently switched from Ring to a UniFi G4 Doorbell so I very much look forward to you adding full compatibility for that soon! Let me know if you need anything from me in terms of troubleshooting or testing so I can help out where I can.

Using a UDM-Pro on software 1.8 and running Homebridge on a Synology 918+

P.S. One thing I noticed in my Homebridge logs is that the video streams are loading at 1280x720, 30 fps, 299 kbps, even though I set the quality to LOW for all 3 of my cameras. Also, recording FPS within UniFi Protect is set to 15 fps. Could this be contributing to an issue?

[hjdhjd]

All - I just pushed an update out that will definitely improve the quality and speed of snapshots. We now pull snapshots directly off the cameras rather than generate a quick video frame. And...I added a new feature...a security system accessory. Read the details in the documentation and changelog.

[hjdhjd]

@tronny87 Thanks for the compliment. With respect to ring - I've closed the issues because ring is solving a different set of problems than we are and while a cursory glance may lead you to believe that "it's doing it well"...the reality is it contends with many of the same issues this and other plugins do. The primary difference with ring is that the video streams coming from the ring can be mostly used as-is without a lot of extra transcoding or interpolation of channels and streams. It's not the same set of problems that UniFi deals with.

While I genuinely appreciate your enthusiasm, opening issues to say "look at how ring does it" isn't helpful. I've looked at ring, nest, and others. I'm aware of how they approach similar problems. In some cases they've greatly helped inform the choices I've made for this plugin in order to optimize the experience to where I want it to be. That said, even plugins that seem good from afar struggle with this. For instance, while the ring plugin has lower latency in some ways (largely due to the ring video stream itself, not anything related to ffmpeg), they also struggle with something else they innovated - 2-way audio. When you start doing anything with any complexity in HomeKit and ffmpeg in particular, things get a lot tougher. In ring's case, there are timing and latency issues with 2-way audio, which is understandable given the nature of what they're trying to solve.

TL;DR: truly appreciate the support and the feedback. I know about ring and the others. I've looked into it. We're all charting a path here...we'll see what time and creativity brings.

[brianmedia]

The snapshots definitely seem to load faster..

1280x720 streams still do not load/work.

[hjdhjd]

@brianmedia Glad to hear snapshots have improved. 1280x720 works for me...it's laggy to start for sure. I'm working on it. 😄

Here's something fun to try...disable WiFi on your iOS device and try that way. I've had much better luck streaming over cellular.

[Vortec4800]

Thought I would add another datapoint here - forgive me if this isn't useful.

I just updated to the latest version, and snapshots are indeed lightning fast now. Awesome!

Unfortunately, video streams now don't seem to work. They did work on the previous version, but they took a while to start. Now they don't seem to start at all, it just says camera not responding. The log says:

[8/6/2020, 16:48:45] [UniFi Protect] Doorbell: Starting video stream: 1280x720, 30 fps, 299 kbps.

I did try streaming over cellular and that did work, it appears to request a 640x360, 30 fps, 132 kbps stream instead.

Anyway, really impressed overall so far, let me know if I can provide any more info that might help.

[brianmedia]

@hjdhjd found my culprit.

I have Mozilla VPN installed on iPhone and it's really good, but stays connected even when i jump from network to network. so when i get home, it's still on, even though i'm on my home network.

Disabled VPN and now 1280 streams fine... 👍

[hjdhjd]

@brianmedia Yay that.

@Vortec4800 It's flaky but it will work. It's very finicky and network-connection dependent, as you can see. It'll get there...keep experimenting and providing feedback guys.

[hjdhjd]

@Vortec4800 - one more thing to try...force the video quality to the lowest quality and see if things start up quickly over WiFi.

[Vortec4800]

How do I do that exactly?

[hjdhjd]

https://github.com/hjdhjd/homebridge-unifi-protect2/blob/master/docs/FeatureOptions.md

Specifically: Enable.StreamOnly.Low in this case.

[Vortec4800]

That did work over wifi. Console still said: [8/6/2020, 17:07:41] [UniFi Protect] Doorbell: Starting video stream: 1280x720, 30 fps, 299 kbps. but it did connect this time.

[hjdhjd]

@Vortec4800 1280x720 = what HomeKit is requesting. That's never going to change given the same set of circumstances and the device type (iPhone vs iPad vs etc...) you are using.

What's happening is that the stream that UniFi pushes out gets transcoded and repackaged to get pushed out to your iOS device.

The quality of that UniFi stream drives latency / complexity in creating a video stream that can be sent to HomeKit. It's super picky...so while you may think that you should be able to get your fancy G4 Pro to push out 4K video...in reality, it's never, ever, going to get 4K...HomeKit only supports 1080p, max, as of iOS 13.

Reducing the stream quality via Enable.StreamOnly.Low will give you the most optimal case to push a stream out that is least likely (although not impossible!) to have issues in getting a working stream.

I'd like to be able to reasonable 1080p streams going whenever possible, assuming HomeKit makes that request (see earlier in this thread...HomeKit decides video quality, not us). It's a journey!

[Vortec4800]

Ah interesting. I knew HK was responsible for requesting the video size it wants, but I still thought the console log was showing the final negotiated result.

Would it be useful to modify the log output to show requested 1280x720 bla bla bla actual stream: 640x360 bla bla bla? Might help us better understand what's actually happening.

[tillkruss]

@hjdhjd: What happens when HomeKit is requesting 1280, but you're sending 1920?

[hjdhjd]

@tillkruss Unpredictable (and generally bad) behavior.

[maydaydarkhorse]

@hjdhjd - I use the Nest Camera Plugin and it allows for what I would call "passthrough encoding" by having ffmpeg to use the codec "copy". At least for me, not having to re-encode the stream greatly improves the responsiveness of things on my nest cameras (which I would like to replace with UniFi). Is this option available in your plugin? I would love to give it a go!

[hjdhjd]

All, give v3.1.1 a shot. I've significantly improved streaming performance and latency in my test environments. For me, I'm seeing instantaneous responsiveness on G3s and a very minor delay in getting a G4 stream stood up (which is understandable given the higher bitrates in use in my environment). So...pretty close to what I'd expect to be ideal, or nearly so.

Would welcome other people's experience here. A big thanks to the couple of folks who allowed me to experiment on them while I got this one (hopefully!) right.

[tronny87]

Looks like it’s a little snappier in terms of loading up the stream, but it seems inconsistent as well. I rotate starting streams between my 3 cameras, sometimes it’ll load up in as little as 2 seconds, other instances I’m waiting 6-7 seconds for it to start.

In terms of video latency, I’m seeing a major improvement. The lag is now 2 seconds or less vs. the previous 5 seconds and up.

Great progress!

[hjdhjd]

Curious to hear what others are seeing and experiencing.

It's extremely snappy, consistently, on my end at this point with a healthy mix of G4 Pros and G3 Flexs.

What specific Protect cameras are you using, what video quality have you configured, and what iOS or Mac devices are you trying it on @tronny87?

Finally - how are you approaching your testing? The Home app is not without its own quirks and bugs. You should ensure the app is truly killed so you get a new instance when you test to eliminate the quirkiness of the Home app from the equation.

[markPHL]

Huge difference here with v3.1.1 this morning -- thank you!

• Previously, the Home app would load a stream successfully about 80% of the time in 15-20 seconds with several second lag in the video timestamp.

• Now, a stream has successfully loaded with every attempt in anywhere from 3 to 10 seconds (average about 6 seconds) and the timestamp lag is now only about 3 seconds behind real-time. I'm also able to switch between cameras without issue - very snappy.

So far, this testing is on my iPhone 11 Pro (iOS 14 public beta 4) and with the Enable.StreamOnly.Low option enabled. I'll remove that option and test on my Mac and iPad later on.

Cloud Key Gen 2 Plus - 1.1.13
Unifi Protect - 1.13.4
Cameras: 5 G3 Bullets, 3 G3 Flex, 1 G4 Bullet, 1 G4 Doorbell

[hjdhjd]

@mconnolly05 Similar experience on my end. Glad to hear it. Definitely remove the low stream limiter and see how it feels.

Your G3s should be pretty instantaneous I imagine, and the G4s perhaps a small delay. 😄

[Vortec4800]

Major, major improvement for me as well. I removed any special stream quality options so just letting the plugin do whatever it wants, and my G4 Doorbell loads a stream within a couple seconds. Snapshots are also still really fast too.

[hjdhjd]

@Vortec4800 Snapshots should be permanently (I hope???) awesome, given that we’re not calling anything external and just pulling right from the source, but thanks for confirming though.

Glad to hear things are significantly improved. I’m going to play around a bit more and see if I can tweak some things specific to the class of Protect camera...but...I want to hear from more people about their experiences first. I’ve got a few data points, universally good, but I’d like to hear from more people on this thread before I call this one fixed and close it out.

I will continue to refine this over time...your collective feedback on how this is working is crucial to ensuring this plugin continues to evolve. It works great for me now...I want it to work great for everyone else too. 😄

[hjdhjd]

@Vortec4800 FYI - my doorbell should be here next week...so hoping to get doorbell support more fully implemented shortly. 😄

[tronny87]

Curious to hear what others are seeing and experiencing.

It's extremely snappy, consistently, on my end at this point with a healthy mix of G4 Pros and G3 Flexs.

What specific Protect cameras are you using, what video quality have you configured, and what iOS or Mac devices are you trying it on @tronny87?

Finally - how are you approaching your testing? The Home app is not without its own quirks and bugs. You should ensure the app is truly killed so you get a new instance when you test to eliminate the quirkiness of the Home app from the equation.

After another round of testing I do see that it is consistently snappy. I only tested it for a few minutes last night after the update was pushed, so I will monitor and keep you posted, but for now I am seeing the major improvements noted by others.

I am on iOS 14 beta 4 on an iPhone 11 Pro Max, as well as iPadOS 14 beta 4 on a 6th-gen iPad 9.7, macOS Catalina 10.15.6, and watchOS 7 beta 4 (yes, I consistently use HomeKit on all these devices :-P)

As for cameras, I have a G3 Micro, a G3 Flex, and the G4 Doorbell.

For config, Enable.StreamOnly.Low is the only option I'm running for the cameras, as well as -preset ultrafast -tune zerolatency for ffmpegOptions

[Vortec4800]

Very cool, excited to see the new Doorbell goodies. I'm really happy with it so far, came from a Nest doorbell. This is actually my very first (and only) Protect camera so far, but I've got some G4 Pro cameras on the way as well.

One thing I did notice, everything works great on Wifi but less so on cellular. When requesting a stream on cellular, it seems like the plugin is still trying to send the 720p stream and it's not super stable. May be some more testing is necessary there.

[hjdhjd]

Thanks for the update @tronny87, glad to hear it’s working well.

More generally, it’s a good prod to remind folks: I don’t support, nor intend to support, beta versions of iOS, UniFi hardware, or UniFi Protect firmwares / releases. Too much of a moving target for what is often buggy and unfinished (which is fair...it’s why it’s beta after all 😄).

[hjdhjd]

@Vortec4800 Looking forward to it myself. The above (see my beta disclaimer) is why I held off mucking with the G4 Doorbell until it was out of the early access store and live in production, and a couple of significant software updates came out. Ubiquiti's hardware is terrific. Their software gets a lot better with time. Using UniFi betas is definitely a crap shoot...lots of issues and regressions, particularly with more bleeding edge stuff, but cool to get a sneak peak into what's coming.

My goal is to support production/GA hardware, running production/GA software and firmware, in real environments. I've probably got two dozen or so UniFi devices...I learned over time that if I wanted to not spend my weekends troubleshooting why my home network isn't working, I should stick with the production releases, and my life has been much better since. 😄

I wish the G4 Doorbell supported door strikes...it's really an essential feature for me. But, I want to muck with a new toy...and I'm excited to get it up and running in this plugin more completely!

[tronny87]

Thanks for the update @tronny87, glad to hear it’s working well.

More generally, it’s a good prod to remind folks: I don’t support, nor intend to support, beta versions of iOS, UniFi hardware, or UniFi Protect firmwares / releases. Too much of a moving target for what is often buggy and unfinished (which is fair...it’s why it’s beta after all 😄).

Fair statement re: betas. I do find myself in weird situations with UniFi though, since sometimes their beta releases are much more stable than the most recent stable releases (especially in the case of the UDM-Pro, performance on 1.8 has been day and night better compared to 1.7.2 or 1.7.3)

I've spent a good chunk of the last hour testing the HomeKit streams on my various devices, and I must say it is very snappy. I am also running tvOS 14 beta and viewing the streams on my TV is working very well and very quick.

Besides the supposed imminent support for doorbell presses on the G4 Doorbell, I have to say this plugin has grown beautifully over time and is nearing absolute perfection. Great job!

I guess someday in the future, one can hope and dream for two-way audio support on the doorbell and other supported cameras (G3-Micro)

[maydaydarkhorse]

Morning @hjdhjd, you have been busy over night!

3.1.1 is much, much improved here in my environment too. Very snappy stream times on my iPhone 11 and Apple TV (both on beta) although load times on my Mac are around ~8-10 seconds - which seems weird.

I am also seeing lag times of ~3 seconds, and the CPU usage on my RPi 4 is down at around 3-4% so I assume you are now copying the stream through..? It used to be upwards of 90% with much higher lag.

Overall, while load times on my Macs are still a little tardy, this is a very useable version for me.

Thanks!!

[hjdhjd]

@tronny87 Repeating your wishlist will not make it happen any sooner. 😄

@philipsaad, @brianmedia, @tillkruss - can you confirm in your setups please. I'd like to close this issue out today. Thanks!

[hjdhjd]

@maydaydarkhorse Which macOS version are you running? Catalina? Streams are generally funky, even with first-party HomeKit devices on macOS...I'll continue to tweak in future revisions though.

Thanks for the confirmation of improvement.

[maydaydarkhorse]

@hjdhjd my Macs are all on Catalina 10.15.6, and I would agree, the Home app is a pile of steaming garbage.

Confirming my UniFi gear is OS 1.7.2 on DreamMachine Pro with Protect 1.14.11. I learned the hard way to stick to stable UniFi releases too!

[brianmedia]

@hjdhjd all cameras loading... the doorbell seems to load a lot faster than the G3 Flex cameras, about half the time.

G3's avg ~6sec to load
G4 DB ~3secs

[markPHL]

Point well understood about running betas - I just can't resist... but that's totally on me!

I removed the low stream limiter and checked the rest of my devices. All are performing just as well as my initial results this morning on my iPhone 11 Pro (iOS 14 PB 4). This includes a Late 2013 Retina Macbook Pro on Big Sur Public Beta 1, AppleTV 4K on tvOS 14 (whatever the latest beta is), iPad Pro on iOS 14 PB4, and my work iPhone 11 running 13.6.

While all devices are loading up streams really quickly, the AppleTV is almost instantaneous. As far as the cameras go, it feels like the G3 Bullets are fastest, the G4 Doorbell is about as fast as the G3 Bullets, the G4 Bullet takes an extra second or so, and the G3 Flexes are another 1-2 seconds.

Which is all to say - this is awesome. Thank you again!

[hjdhjd]

Thanks @mconnolly05 - appreciate the feedback. Your experience is similar to mine...the G4 Bullet and Pro definitely takes a little longer...it's slight, but imperfection bugs me, so I'm going to play with it a bit more this weekend and see if I can get it to be as instantaneous as the G3s.

Going to close out this issue.

[hjdhjd]

@brianmedia The G3s should load pretty much instantly. Can you provide more specifics about your setup? What controller? What iOS devices? Versions?

In my experience, it's pretty instantaneous (1-2 seconds) for G3s.

[brianmedia]

@hjdhjd

UDM Pro - 1.7.2
Protect - 1.14.11
iPad - 14 beta
iPhone - 13.6
Macbook Pro - 10.15.6

Might have been the first time pulling the streams after the update.. i went back in to test on all devices and now they are loading faster 👍

[ituri]

I'm running the latest version of homebridge-unifi-protect (4.3.5) with a G3 Flex connected to a UDM Pro with the latest firmware installed. Still, it takes around 10-20s for the stream to start. I tried "Enable.Video.Stream.Only.Low" but this made things even worse with the stream taking the same amount of time to load but then skipping multiple seconds of the stream.

Homebridge is running on a Raspberry Pi 4B with 4GB of RAM. The iOS device is a iPhone XS with iOS 14.3 and an iPad Pro with iPad OS 14.3 as well.

The only possible problem I could see is that I'm running homebridge 1.1.7, so I'm on the latest version of the 1.1 branch. Should I be using 1.2.x?

Is there anything else I can try to make the streams load faster?

[github-actions]

This issue is locked to prevent necroposting on closed issues. Please create a new issue for related support requests, bug reports, or feature suggestions.