fillLineGap doesn't always fill the gap about imscjs

[pjcherriman]

I had seen a similar linegap issue on a TV using dash.js for playback

It seems the issues is the TV (like many TVs) uses a devicePixelRatio of 1.5

[poolec]

This problem, at least as far as the line gap is concerned, appears to be specific to the case where window.devicePixelRatio is greater than 1 (i.e. where the graphics are rendered at a higher resolution than the co-ordinate system). This is fairly common on TVs and mobiles and indeed in desktop browsers where high resolution displays are used.

Having investigated this further, I believe the problem arises because the implementation of fillLineGap pads each line to an integer vertical co-ordinate in the CSS pixel co-ordinate space. When the device is using a higher resolution for rendering, this does not necessarily align with a device pixel boundary. If that happens, then when the browser draws the background boxes and the positions are not integer positions, gaps can arise e.g. due to independent rounding of positions and sizes in the rendering process. (Overlaps may also occur but would not be apparent with the opaque backgrounds that we use).

I believe there is a fairly straightforward fix to this by rounding to device pixel boundaries. e.g. instead of applying Math.round in the applyFillLineGap function, use a function such as this:

function roundToDevicePixel(position) {
        return Math.round(position * window.devicePixelRatio)/window.devicePixelRatio;
}

[nigelmegitt]

Worth checking against this test in particular:

• TTML: https://github.com/w3c/imsc-tests/blob/main/imsc1/ttml/fillLineGap/FillLineGap005.ttml
• Expected result: https://github.com/w3c/imsc-tests/blob/main/imsc1/png/FillLineGap005/0.000000.png

This test uses background colours with around 50% opacity, so if there are overlapping background boxes then they look like darker lines at the overlap locations. If this fix doesn't break that test, I expect that it's probably fine. Obviously should pass all the other tests too.

[poolec]

I tried that test. At 1.0 devicePixelRatio it works fine. At other devicePixelRatios, the generated PNG always comes out the same size so is not representative of the browser display. With or without my proposed fix, the generated image shows gaps.

The generated HTML shows no overlaps when viewed on the same browser at the same devicePixelRatio. I do still see a gap in one devicePixelRatio, even with the proposed fix, so will dig into that one a bit further.

[poolec]

I've now done some more detailed investigation using this test.
What I see in Chrome can be summarised as follows:

• When window.devicePixelRatio is 1.0, getBoundingClientRect returns values that are precise multiples of 1/64 pixel
• When devicePixelRatio is 1.0, calculated padding values are precise multiples of 1/64 pixel and once padded, the resulting span has an integer vertical size
• When devicePixelRatio is 1.25, getBoundingClientRect returns values that are close to multiples of 0.2 pixels but +/- something of the order of 0.00001
• When devicePixelRatio is 1.25, calculated padding values are close to multiples of 0.2 pixels but +/- something of the order of 0.00001
• When the bottom padding value is even a tiny bit less than the nearest multiple of 0.2 pixels, the background is rendered with a gap. If the bottom padding is a tiny bit more than a multiple of 0.2 pixels, there is no gap. For example: 2.799999237061 gives a cap. 2.8 does not.
• When a gap is visible, the new span height returned by getBoundingClientRect after padding has been applied is ~1/64 pixels smaller than expected

The 1/64 figure I am seeing correlates with the fixed point representation with 6 fractional bits that appears to be used within Blink (and came originally from WebKit). It appears that some calculations within the Chrome browser are rounding down to the nearest 1/64th of a pixel in cases where rounding to nearest might appear more appropriate.

I also observe that whilst 1.25 (the devicePixelRatio I tested) can be precisely represented in floating point, 1/1.25 (0.8), and multiples of it, cannot. Thus I wonder whether some loss of precision in the layout calculations is contributing to the problem.

Based on these new observations, I have just tested a different workaround in which I add 0.0001 to each calculated bottom padding value in the implementation of applyFillLineGap so as to prevent the calculated padding values ever ending up slightly less than the 'expected' value.

Initial results suggest that the problem then does not arise (nor is there any overlap) but more tests would be needed to confirm.

This is, of course, not a very satisfactory fix for imscJS.

[nigelmegitt]

@JibberJim did some work on this before to make it work with devicePixelRatio != 1 on the BBC fork. I did a diff between the BBC fork and the main repo and could not see any significant differences that would explain why it might work in one context but not in another.

[poolec]

Having now investigated this issue further, it's clear that different browsers are showing different rounding behaviours when rendering subtitles on displays where devicePixelRatio is not 1.0. This causes gaps between lines at certain vertical positions. In some cases, there are even gaps between adjacent spans.

I do not believe there are any concrete requirements for the user agent covering rounding behaviour in such cases. As such it can't confidently be argued that it is a user agent problem. In my mind, it therefore falls to imscJS to do its best to avoid these gaps being visible to the end user across the range of browsers in use. (Separately, there may be a case for making the specs more precise.)

To my mind, the best solution would be to introduce a small (~1 pixel) overlap in the text backgrounds in the case where (i) fillLineGap is enabled, (ii) the background colours of adjacent lines are both opaque and (iii) devicePixelRatio is not 1.0. This would not prevent gaps in cases where semi-transparent backgrounds are used but in that case, there is a risk of creating visible overlaps instead which would probably look just as bad. It would address the problem fully for the opaque background case. It could be achieved by an additional 1 pixel of bottom padding and -1 pixel of bottom margin.

A similar approach could be taken between adjacent spans to eliminate vertical lines appearing between words.

It's possible that an overlap of less than 1 pixel may address the problems observed. Maybe 0.5 pixel or 1/devicePixelRatio or 0.5/devicePixelRatio might be sufficient. However, I'm not sure it is worth trying to find the absolute minimum that would solve the problems seen today since (i) future devices might be affected in different ways and (ii) the visible difference would be negligible.

This approach would not result in any text characters being clipped in the same way they would if, for example, line spacing were artificially reduced.

I plan to prototype this properly when I have time, but in the mean time, I'd welcome any comments or suggestions for improvement.

[palemieux]

Thanks for all the sleuthing!

From an imscJS main branch perspective, I feel that the energy spent chasing UA shortcomings could be better spent either

• relentlessly bugging UA folks to support missing CSS features so that imscJS and others do not have to implement fragile workarounds; or
• fixing the shortcomings directly in the UA code.

For example, inline-sizing would make implementing fillLineGap less fragile.

[ed.: I do think imscJS needs workarounds for critical features, but there is diminishing returns in addressing all idiosyncrasies.]

[poolec]

I understand the sentiment. However, the problem with that approach is that it really doesn't address the TV market. TVs invariably have browsers that are some way behind the cutting edge of desktop browsers (typically 2 years or more), very rarely receive updates and are infrequently replaced by their users.

As a content provider wishing to ensure our viewers get a great experience with subtitles on their TVs, we have to work with what is out there, much as we might wish it to be better. Since the majority of TV viewing is on a TV, and we're seeing the problems described here on many of them, I think it is entirely appropriate that imscJS acknowledge that reality and do what's reasonably possible to address it in the interest of subtitle users, provided that the fix doesn't break something else.

I don't want to waste time on a fix for this if you would reject it on principle. However, I think it could be written in a fairly self contained way and be made configurable. Would that be acceptable?

[palemieux]

@poolec Can you share the compatibility requirements for those TV browsers? In particular, what is the lowest ECMA Script version support? The idea is to figure out a strategy for legacy support, esp. in light of the ongoing work on imscJS 2.0.

[poolec]

Sure. They're supporting either HbbTV 2.0.3 or, for very recent models, HbbTV 2.0.4.
HbbTV 2.0.3 references the CTA WAVE Web Media API Snapshot 2018
HbbTV 2.0.4 references the CTA WAVE Web Media API Snapshot 2021
In terms of ECMAScript, they reference ECMAScript 6 and ECMAScript 2021 respectively, albeit with some exceptions.

[nigelmegitt]

Hypothesis: when devicePixelRatio is not 1 the UA is incorrectly calculating the bottom or right edges of areas, because, in general:

round(origin * devicePixelRatio) + round(extent * devicePixelRatio) != round((origin + extent) * devicePixelRatio)

In other words, the theory is that the UA is using the wrong algorithm to calculate the edges in scaled views.

I put together a codepen to discover how frequently this might arise, at https://codepen.io/nigelmegitt/full/pomKoww - if this theory is correct, there are a lot of opportunities to generate 1px gaps or overlaps when devicePixelRatio is not 1.

Just to emphasise, it's a hypothesis only at this point.