ITV failure to download some programmes (viewable on-line) about get-flash-videos

[njtaylor]

I can repeat the problem here, of the 5 episodes available 3 download, 2 fail.

[baffled-mart]

Thank you for your near immediate response. I wish I could help but wouldn't have a clue where to start.
If it helps, I found the same occurs with (including the 2nd Protectors);
https://www.itv.com/hub/grantchester/2a2958a0002
https://www.itv.com/hub/the-protectors/ENT1315a0006
https://www.itv.com/hub/the-widower/2a1391a0003
https://www.itv.com/hub/white-right-meeting-the-enemy-exposure/2a5133a0001

[baffled-mart]

Ah, I see that similar has been commented on "youtube-dl".
"...
That hasn't been for all the ITV videos, it just seems to be a tiny but annoying fraction, heh. I believe they're not always using secure-mercury now.
However, those videos all still play in the browser, so something must be working right. Looking at the network requests, it seems maybe there's a shift to a new playlist service?
..."
ytdl-org/youtube-dl#14944

If that is any help.

[Vangelis66]

@baffled-mart wrote:

I found the same occurs with (including the 2nd Protectors);
https://www.itv.com/hub/grantchester/2a2958a0002
https://www.itv.com/hub/the-protectors/ENT1315a0006
https://www.itv.com/hub/the-widower/2a1391a0003
https://www.itv.com/hub/white-right-meeting-the-enemy-exposure/2a5133a0001

However, those videos all still play in the browser, so something must be working right.

What you get in a compatible desktop browser is encrypted (DRM'ed) MPEG-DASH streams (see also the discussion in #211), whereas the itv.pm plugin of GFV relies upon unencrypted RTMPE (aka Flash) streams (downloadable by RTMPDump).

ytdl-org/youtube-dl#14944

If you look closely at ytdl-org/youtube-dl#14944 (comment), you'll be able to identify the (encrypted) MPEG-DASH manifest (mpd, i.e. Media Presentation Description):

"Base" : "https://itvpnpdotcom-a.akamaihd.net/",
      "MediaFiles" : [
        {
          "Href" : "2-2271-0033-001/18/1/VAR011/2-2271-0033-001_18_1_VAR011.ism/.mpd?hdnea=st=1513091350~exp=1513112950~acl=/*~hmac=13c038ae3549a09a864295735957c6627be0e40a1856b31adfd09803f268c512",
          "KeyServiceUrl" : "https://itvpnp.live.ott.irdeto.com/ClearKey/getkeys?CrmId=itvpnp&AccountId=itvpnp&ContentId=2-2271-0033-001_18"

i.e. the full mpd URI is
https://itvpnpdotcom-a.akamaihd.net/2-2271-0033-001/18/1/VAR011/2-2271-0033-001_18_1_VAR011.ism/.mpd?hdnea=st=1513091350~exp=1513112950~acl=/*~hmac=13c038ae3549a09a864295735957c6627be0e40a1856b31adfd09803f268c512 which is tokenised (expiry+authentication) and uses the ClearKey DRM encryption...
GFV has still no support for MPEG-DASH streams, and, AFAIK, the ClearKey encryption hasn't been cracked (at least publicly...).

@njtaylor

The yt-dl dev has tackled this issue via ytdl-org/youtube-dl@3fae11ac00 ; that is Python, while GFV is written in Perl; I hope you can "translate"; however, the fixed yt-dl version DOES NOT RETURN ANY FLASH streams for the ITVHub listings affected by this issue, only AppleHLS streams are identified:

youtube-dl --version 
2017.12.14
youtube-dl -F "https://www.itv.com/hub/the-widower/2a1391a0003"
[ITV] 2a1391a0003: Downloading webpage
[ITV] 2a1391a0003: Downloading XML
[ITV] 2a1391a0003: Downloading JSON metadata
[ITV] 2a1391a0003: Downloading m3u8 information
[info] Available formats for 2a1391a0003:
format code  extension  resolution note
hls-136      mp4        audio only  136k , mp4a.40.2@128k
hls-569      mp4        512x288     569k , avc1.64001F@ 408k, 25.0fps, mp4a.40.2@128k
hls-754      mp4        512x288     754k , avc1.64001F@ 582k, 25.0fps, mp4a.40.2@128k
hls-1001     mp4        896x504    1001k , avc1.64001F@ 816k, 25.0fps, mp4a.40.2@128k
hls-1286     mp4        896x504    1286k , avc1.64001F@1084k, 25.0fps, mp4a.40.2@128k
hls-1562     mp4        896x504    1562k , avc1.64001F@1345k, 25.0fps, mp4a.40.2@128k (best)

FWIW, the audio stream is mistakenly reported as being AAC LC@128kbps, while in reality is just @96kbps, as in the rest of the downloadable programmes in FLV container...
If, indeed, Flash streams are AWOL for these programmes, then am afraid it's time you implemented AppleHLS support for ITV, as I hinted back in the summer here...

ITV issues aside, I wish Merry Christmas to all those that observe it 😄

[baffled-mart]

@Vangelis66
Thanks, as always, for your learned comments (g_i_p too).
Just a bit beyond my understanding. :-)
MC to you too.

[njtaylor]

Initial support for HLS AES-128 added to GFV still being tested. Worked for couple of downloads from ITV, needs more work - restarts, command line options to force use.

ITV is not consistent in supporting some downloads are HLS only, many new programmes have RTMP downloads and HLS. The basic HLS support is now there for ITV.

[baffled-mart]

Thanks for the update and all your hard work.  Much appreciated.  

>>>

Initial support for HLS AES-128 added to GFV still being tested. Worked for couple of downloads from ITV, needs more work - restarts, command line options to force use. ITV is not consistent in supporting some downloads are HLS only, many new programmes have RTMP downloads and HLS. The basic HLS support is now there for ITV. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.