A few ideas, FrameRate additions. about timecodekit

[orchetect]

@ryanfrancesconi pushed release version 1.0.3 to main with updates.

[ryanfrancesconi]

Timecode(TimeValue(seconds: x), at: fr, limit: ._24hours)

cool, that's essentially what I did though am using the toTimecode convenience.

[orchetect]

Thanks for the interest! It took a few weeks of research and testing to put the library together and ensure its accuracy.

In particular I think a common format list is useful for displaying in a UI. Not very common to list 120 DF for example.

public static var commonFormats: [Timecode.FrameRate] {
  [ ... ]
}

I feel a commonFormats property falls under subjective features. The developer adopting the library can determine what frame rates they want to use or not use, since what constitutes a "common format" is subjective, AFAIK. So you'd be free to have this property as an extension on Timecode.FrameRate as you've illustrated, but in your app.

Also, Avid's Pro Tools shows 119.88/119.88d/120/120d in its list of project frame rates now, so they may become a bit more common as other software suites follow their lead.

That said, I would alter your method to be:

public init?(floatValue: Float, restrictTo: [FrameRate]?, favorDropFrame: Bool = false)

and then just pass in your own commonFormats array to the method whenever you call it:

FrameRate(floatValue: x, restrictTo: FrameRate.commonFormats, favorDropFrame: true)

and if restrictTo is nil, just have the method internally use .allCases as default.

I have the situation where I don't know a frame rate until I open a video and all I've got from that is a floatValue. I'm not actually sure if there is a way to tell if it's drop frame or not?

I'm not sure I fully understand. Whatever API you're using to read/open video files is only supplying float value of the frame rate? The question about drop or non-drop is tricky - is it designed to be deterministic? The way you've implemented definition and discovery of float value seems heuristic in nature. For example, would 29.97 vs 29.97d produce different float values? At the very least I'd check the documentation of the video API you're using, and I'd want to externally generate/acquire video files in all frame rates (known frame rate) then see what your API produces as float values to compare.

I also found that 30df/60df/120df are not treated as actual drop-frame rates in most contexts, so on cursory glance I'm not sure your code is correct.

Is float value a standard in video libraries? Or is it just endemic to the one you're using? That may gauge the viability of including a float value in TimecodeKit. I just want to understand it first and avoid including novel features where they may not be appropriate for most contexts.

But now you've got me curious so I'd love to see this figured out!

[ryanfrancesconi]

oh i didn't notice ProTools added the higher frame rates. Disregard my comment in that regard then!
if i use common formats, then yes that makes sense to make that an application level thing and doesn't need to be in your API.

About the Float float frame rate:

I'm using Apple's AVPlayer for video and that will report a video's frame rate as a Float, yeah. In my particular application, this value isn't super telling given something like this. These FrameRate's of yours all have the same numerical value:

        case ._29_97, ._29_97_drop, ._30_drop:
            return 30.0 / 1.001

        case ._59_94, ._59_94_drop, ._60_drop:
            return 60.0 / 1.001

        case ._120_drop, ._119_88, ._119_88_drop:
            return 120.0 / 1.001

So in that case, how would you know which should be the specific Timecode.FrameRate - or can you? Or, did I miss something? As far as I can tell, all you can get from the AVPlayer is the AVAssetTrack.nominalFrameRate, which is a Float.

    /**
    	@property		nominalFrameRate
    	@abstract		For tracks that carry a full frame per media sample, indicates the frame rate of the track in units of frames per second.
    	@discussion		For field-based video tracks that carry one field per media sample, the value of this property is the field rate, not the frame rate.
    */
    open var nominalFrameRate: Float { get }

So what I'm doing at the moment is limiting the frame rate selection to the ones I'm sure have the approximated base floatValue, and the user could select drop or non-drop. I'm unsure if it's possible to really know more than that in that case. Here's a screenshot of what that looks like:

For the particular video, encoded in Premiere, I encoded as 29.97 - premiere doesn't offer any info on export beyond that, but it does export as 29.97 DF. That's the only export preset that exports DF.

So, opening that video in AVPlayer says the frame rate is "29.970032".

Any advice?

[ryanfrancesconi]

If curious, here's some frame rate selectors I had a look at to see how others are labeling these.

Premiere 2020

Logic

Final Cut Pro

ProTools

[orchetect]

I'm using Apple's AVPlayer for video

Ok, I figured it was 1st-party API. I wouldn't mind playing around with AVPlayer and seeing what it's doing.

These FrameRate's of yours all have the same numerical value

   case ._29_97, ._29_97_drop, ._30_drop:
       return 30.0 / 1.001

   case ._59_94, ._59_94_drop, ._60_drop:
       return 60.0 / 1.001

   case ._120_drop, ._119_88, ._119_88_drop:
       return 120.0 / 1.001

This is your code, not the library. Internally in the library, these are calculated differently and are not quite parallelized or reduced in this way. 30d/60d/120d are calculated differently even though they are labelled as a "drop" frame rate. You'll notice in Logic's frame rate selection list, these rates are in bold and italics to show they are not traditional frame rates but actually modern adaptation rates.

Here's the issue as I see it: The reason why boiling video encoding frame rates down to a float value is obscure is that drop rates technically have fewer "frames" but as you likely know, only certain frames are dropped. What you would end up with would be an average, if it were a float value. But it does not mean that there are that many frames per second in any given second of time. My guess is that AVPlayer may produce a float value that averages the per-second frame count across an entire hour or something, and then flatten it out to a "per-second" frame rate float value. Which would be kind of stupid, but possible I suppose.

Since I'm not familiar with AVPlayer, from a quick Google search I'm guessing nominalFrameRate has to do with the overall video/GPU playback rate which may be misleading. I did read that AVAssetReader can possibly determine media's encoding frame rate with better precision. Have you looked into that?

[orchetect]

As an aside, in retrospect I think I will update the Timecode.FrameRate.stringValue property and add a verbose version so it's more flexible for use in human-readable UI.

I'll update stringValue to produce short Logic-style strings. Then add another property called stringValueVerbose with longer style strings like Pro Tools. Although ultimately this is in the library as a convenience and this nomenclature is up to the developer if it doesn't precisely suit their needs; they can set up their own strings in their app code.

ie:

/// Returns human-readable frame rate string.
public var stringValue: String {
    switch self {
        ...
    case ._29_97:       return "29.97"
    case ._29_97_drop:  return "29.97d"
        ...
    }
}

/// Returns human-readable frame rate string in long form.
public var stringValueVerbose: String {
    switch self {
        ...
    case ._29_97:       return "29.97 fps"
    case ._29_97_drop:  return "29.97 fps drop"
        ...
}

[orchetect]

I played around with AVFoundation a bit and the only mechanism I could find for reading frame rate was the nominalFrameRate property as you mentioned.

Also keep in mind that this value can be an arbitrary value that doesn't correspond to a concrete frame rate, if a video file is encoded as Variable frame rate. For example, I did a screen recording video and it was encoded as AVC Variable rate. When read by AVAssetTrack, it returns 55.473682 as the nominalFrameRate value. Not sure how you want to handle that, but just an FYI in case you haven't run into that yet.

I'll keep digging and see if there's a reasonable solution to the drop/nondrop float value issue.

[orchetect]

I have a video file known to be 29.97d and I ran a few tests.

File meta data's frame rate is shown as "29.970 FPS" as read in Invisor.app

Also, nominalFrameRate is read as exactly float value 29.97.

So from that, you cannot know whether timecode display should be drop or non-drop. Essentially, the user has to specify it.

[ryanfrancesconi]

So from that, you cannot know whether timecode display should be drop or non-drop. Essentially, the user has to specify it.

Right yeah. that was what I had decided as well. Which isn't really such a big deal, but good to know you came to the same conclusion.

[ryanfrancesconi]

As an aside, in retrospect I think I will update the Timecode.FrameRate.stringValue property and add a verbose version so it's more flexible for use in human-readable UI.

oh that's good. Actually I added this same idea as well like this:

    /// Returns human-readable frame rate string.
    public var stringValueUILabel: String {
        switch self {
        case ._23_976:      return "23.976"
        case ._24:          return "24"
        case ._24_98:       return "24.98"
        case ._25:          return "25"
        case ._29_97:       return "29.97"
        case ._29_97_drop:  return "29.97 Drop"
        case ._30:          return "30"
        case ._30_drop:     return "30 Drop"
        case ._47_952:      return "47.952"
        case ._48:          return "48"
        case ._50:          return "50"
        case ._59_94:       return "59.94"
        case ._59_94_drop:  return "59.94 Drop"
        case ._60:          return "60"
        case ._60_drop:     return "60 Drop"
        case ._100:         return "100"
        case ._119_88:      return "119.88"
        case ._119_88_drop: return "119.88 Drop"
        case ._120:         return "120"
        case ._120_drop:    return "120 Drop"
        }
    }

Definitely seeing all those DF and NDF's isn't the best look in a selector.

[orchetect]

I think the conclusion is that float value is not a standard value for timecode expression frame rate, so adding a floatValue property or init to the library will be ambiguous given what we know here.

[ryanfrancesconi]

This is your code, not the library. Internally in the library, these are calculated differently and are not quite parallelized or reduced in this way. 30d/60d/120d are calculated differently even though they are labelled as a "drop" frame rate. You'll notice in Logic's frame rate selection list, these rates are in bold and italics to show they are not traditional frame rates but actually modern adaptation rates.

oh ok. actually i was looking at :
frameRateForRealTimeCalculation

In that switch the values are as I have them. I am doing a lot of realtime conversion myself as I'm deriving the timecode from a seconds value. Is there another way to do that?

[ryanfrancesconi]

I think the conclusion is that float value is not a standard value for timecode expression frame rate, so adding a floatValue property or init to the library will be ambiguous given what we know here.

that's fine. though, regardless of that, I still have to do it. But it doesn't need to be in your API, I just wanted to bring it up as something I'm looking at. In all videos I've tested the rates are close enough - but that is why in my init for floatValue I'm just taking a truncated version and setting the selector to the possible rates. Given that, it doesn't have to be a 100% confident result, but limiting the frame rate in this way gets it at least to the rates that matter.

I hope.

[orchetect]

I am doing a lot of realtime conversion myself as I'm deriving the timecode from a seconds value. Is there another way to do that?

Assuming you are supplied a seconds float for current video playback position, this is the most obvious way:

// x == position in seconds
// fr == FrameRate

Timecode(TimeValue(seconds: x), at: fr, limit: ._24hours)

[orchetect]

I'll push the stringValue / stringValueVerbose update shortly.

Also while I'm at it, I will flesh out the README.md with some basic code examples since I've been meaning to get around to it.

And thanks again for bringing up the float fps issue - something I hadn't dug into before and it was a good consideration to explore.

If you have any other ideas for improvements or additions, feel free to post more Issue tickets.

[ryanfrancesconi]

cool, thanks!