Display all subtitle entries in the subtitle list? about memento

[Calvin-Xu]

Wow, thank you so much for such detailed reply! I previously had a glance at subtitlelistwidget but assumed that you might be parsing subtitle text separately. I agree that observing mpv is a very robust solution, especially for supporting streamed videos & subs added in ways that bypass Memento.

It might be cool to make a special case for loading an external ass, srt, or vtt file explicitly through Memento, so that the list can be populated all at once. But I understand that it might not work well with the existing implementation, and that you identify other priorities at the moment. Memento is the most performant and feature rich option I have found compared to animebook, asbplayer, mpvacious, etc, and it's incredible that you recreated Yomichan's functionality from scratch & released the macOS port just days ago that already works so well. Unfortunately I am not yet fluent enough in Qt or C++ to contribute to development, but I am very very thankful for your work.

I also want to mention that on macOS, Memento seems to additionally require MuJS that is not listed in the dependencies. But if it's already depended on by one of the others it's my mistake.

[ripose-jp]

It might be cool to make a special case for loading an external ass, srt, or vtt file explicitly through Memento

This is feasible and I don't mind implementing this.

it's incredible that you recreated Yomichan's functionality from scratch

Not that I wanted to. I only did it because I didn't want to upgrade Memento to GPLv3 in order to use Yomichan and I really didn't want Memento to depend on qtwebengine.

I also want to mention that on MacOS, Memento seems to additionally require MuJS that is not listed in the dependencies. But if it's already depended on by one of the others it's my mistake.

Thanks for mentioning this. MuJS isn't a direct dependency of Memento, but I know it is a dependency of mpv. I ran otool on the bundled libmpv dylib and found it linked to my local installation of MuJS. I don't know why macdeployqt missed it, but I'll try to fix the issue and update the release with the fixed binaries.

Edit: I updated the binaries on the release page. I found out the reason this happened is because libmujs is a .so file on macOS instead of a .dylib, so macdeployqt missed it. Don't know why the Homebrew/MuJS maintainers did this, but whatever. Thanks for your help with the MuJS bug. I'm glad you like the software!

[ripose-jp]

Support for populating the subtitle list using external SRT files has been added on master. There will probably be bugs, but I didn't see anything immediately obvious.

I hope to get ASS and VTT working sometime this week. The only work needed is writing parsers.

[ripose-jp]

That would be the ideal way I'd like it to work. The only reason I didn't implement it that way was because it would have greatly increased the difficulty of implementing the feature.

The way it currently works is that Memento observes the sub-text property of mpv, which basically just calls a function whenever a new subtitle appears with the text of that subtitle.

To implement the feature in a way where all the subtitles show up whenever a user selects a subtitle track would require getting all the subtitles and their timings all at once. This is a problem because mpv's API doesn't have any feature that lets me get information about a subtitle that hasn't appeared yet. In fact, I doubt such a thing could easily be added to the existing codebase of mpv considering the limitations of the sub-seek feature. From the mpv docs:

For embedded subtitles (like with Matroska), this works only with subtitle events that have already been displayed, or are within a short prefetch range.

This means the most practical way of implementing the feature is to go outside of mpv. This is really difficulty because it means Memento has to keep some sort of state that is equivalent to mpv's state at all times. If these two states become desynced because mpv changes state for a reason Memento didn't expect, that's a bug.

There's also three different places (that I know of) where subtitles can be located:

1. Inside of the video file
2. In a separate file
3. Remote in the case of a streamed video with a subtitle track

That means Memento has to be able to map mpv subtitle tracks to their actual locations which could be outside even the user's computer. The naive way of thinking about this is that it is feasible because everything must pass through Memento before it gets to mpv, but this is not the case. Due to Memento supporting scripts and configuration files, it's possible that subtitle tracks could get added in ways that bypass Memento completely. If Memento can't find where subtitles are located, it can't populate the subtitle list with those subtitles.

I had tried a hack back when I was initially implementing the feature where there would be in instance of mpv running in the background that mirrors the state of the mpv instance that acts as the player. This mpv instance would just call sub-seek 1 until the end of the video and harvest information about subtitles. Unfortunately, it didn't work well. Plus, there's just no way I would have wanted to release a version of Memento that did since it would have been ridiculously inefficient.

There are other concerns that prevented me from implementing this feature back in v0.3.3, but those were the main problems. I can take a look at it again since I don't have much planned for the next version, but I can't make any promises on this one.

[Calvin-Xu]

Thank you! This is amazing work. I want to report that recent commits have exposed some issues in how Memento does not react correctly to mpv subtitle delays (mpv default binding z / x, which I understand the GlobalMediator emits) in some components.

I tried the latest build a bit, and it seems that since at least 9e91b10, media audio added to Anki (either {audio-media} or {audio-context}) is not generated accounting for the delay. The delay was respected in 7b1c8bd, so I feel it is a regression when introducing {audio-context}. I wonder if I should reopen #38 or if this can be a trivial fix.

Additionally, it seems that navigating subtitles by clicking on entries in the list on the right side does not take the delay into account. With a positive delay, for example, the player often starts at the end of the previous entry, and sometimes a duplicate of the previous entry is added. I think this has always been an issue, but as the list is completely populated on start now & together with the previous regression, I finally noticed it enough to pin it down.

I find interactive timing delays an important feature because subs often are not timed exactly or have uneven offsets, especially for TV version subs that account for ads or sponsor screens in the middle. I think there must be a lot of edge cases, especially there's a lot more complexity going into it and subs are being cached, and I really appreciate your work and the progress of development.

[ripose-jp]

Thanks for reporting the bug with the audio delay. It seems like when I was refactoring some stuff, I forgot to add the delay. I'm pretty certain the bug predates 7b1c8bd but is newer than tag v0.5.1-2-beta.

The subtitle list seeking bug was actually caused by the fact that I only allowed for seeking in whole seconds because the seek method took an integer instead of a float. There's no way to completely fix the issue because doubles are imprecise, but it should only be a frame or two of error and nothing as bad as it was before.

Subtitles getting double added is actually a more complex issue. mpv's sub-text property actually "flattens" multiple subtitles that appear at the same time. This means if there are subtitles 1, 2, and 3 and they appear in order 1 > 2 > 3 and disappear in order 1 > 2 > 3 but all have some time period in which they overlap, mpv's sub-text property will emit these values:
1
12
123
23
3
Because sub-text doesn't emit any extra information about subtitles, there is no way for me to undo this perfectly. That's why some subtitles might appear twice in the subtitle list. Other than that, I did notice an inconsistency in my subtitle parsing code that tried to mimic this flattening behavior. Apparently if a subtitle starts on the same frame one ends, mpv will not flatten that subtitle while my algorithm did.

I believe I've fixed these issues with the last three commits on master.

[Calvin-Xu]

Thank you! I can confirm that audio delay is reenabled. I actually checked out and built each version to confirm the behavior, but I suppose as long as it is fixed now it's all good.

The subtitle list seek behavior is much improved. I can still encounter cases where the entry previous to selected appears for a moment, because as you mentioned doubles are imprecise. Since seeking to the previous entry is probably not any user's desired behavior, would adding a small delta value to ensure landing on the selected line (as done elsewhere in Memento) be appropriate?

I don't think duplicate entries is a huge functional issue as long as each of the duplicated entries has correct timing information & can generate correct cards. Thanks so much for these fast fixes.

[Calvin-Xu]

On a related note, as I'm hopelessly lost about a lot of implementation details at this point, I wonder if Memento still uses sub-text to add entries when it has already populated the list with an external file. If that is the case, would it be more desirable to make the file the single source of truth about subtitles, such that issues like double adding can be avoided?

[ripose-jp]

I can experiment with adding a delta. I'm only a little hesitant because I know there are users that are very particular about not cutting off any audio and would prefer if things are given more breadth instead of less. Also, mpv has a similar issue with seeking a little before a subtitle using sub-seek, so it's not unreasonable for Memento to have similar behavior.

The implementation still does rely on sub-text to know which subtitle is currently being displayed. I could change it to use only the current time of the video, but that would degrade performance because checks would have to be run every frame as opposed to every new subtitle. It would make maintaining the code more difficult since embedded subtitles would need to use the old implementation, while external subtitles would need to use the new implementation. Sounds like a lot of headache for a feature that already works well enough.

[Calvin-Xu]

It would make maintaining the code more difficult since embedded subtitles would need to use the old implementation, while external subtitles would need to use the new implementation.

I can understand this decision and yeah it works well enough. I suppose I am only confused about why duplicated entries still happen in this scenario. As Memento has already populated its own list, wouldn't any additional entries generated during playback be flat out unwanted?

Regarding the delta, I can see it is not perfect, but I don't think most users care about skipping by perhaps one frame, especially as Memento also provides audio padding options as I understand it exported audio should not be affected, only where live playback starts. I think this is the conclusion reached in mpv-player/mpv#5365. Now on the other hand, I don't think most users can notice seeking slightly ahead either if the previous sub is never highlighted in the list. But as it stands, a visual anomaly really stands out and I think people might continue to report it as a bug in the future.

*I think another possibility is to make it that when navigating subtitles in the list, the actual mpv seek functionality is untouched, but the UI does not allow the highlight to leave the clicked entry for small delta window. After the window, the list updates as normal. This does not cut off any audio, but is probably more complicated to implement, and I feel duplicate entries caused by the actual seeking behavior complicate it further.

[ripose-jp]

I suppose I am only confused about why duplicated entries still happen in this scenario.

It was a double precision bug, I fixed it with 1961ad2.

Seeking using the subtitle list is now 28ms ahead of the actual start time. I haven't found a case where this doesn't work, but I didn't look too hard. It doesn't seem to cut off any audio, but I might just be using badly timed subs.

Anyway, now there is support for preloading external ASS, SRT, and VTT subtitles into the subtitle list. I don't have any plans to add support for any more formats since I've never even seen subs that weren't either ASS or SRT.

I'll leave it up to you whether or not you want to close this issue since this feature doesn't work for embedded subtitles. I still don't think adding support for embedded subtitles is feasible for the reasons I've stated previously though.

[Calvin-Xu]

Thank you! As always I'll test it a bit more and report if anything breaks, but now it all looks really good to me. Since using external sub files probably encompasses the majority of use cases, I think this is an quite a milestone for this project.

[Calvin-Xu]

Hello. Please check #46 for some of the changes I have tried to implement while playing with the latest version. Once again, I had almost zero exposure to Qt prior to coming across Memento, so the PR should more be viewed as a proof of concept, and please do any changes that are necessary.

[Calvin-Xu]

For the record, #46 which aimed to fix SubtitleListWidget interface issues was merged into dev still with issues to be developed on. SubtitleListWidget will possibly be refactored in the process.

As "Display all subtitle entries in the subtitle list" has been achieved, I'll close this issue.

This feature does not support embedded subtitles; entries will still be populated during playback from mpv's output. Alternatively one can extract the subtitles and have them in the standard subtitle file formats with other tools.