GithubHelp home page GithubHelp logo

futzu / m3ufu Goto Github PK

View Code? Open in Web Editor NEW
28.0 28.0 3.0 183 KB

The Most Pythonic M3U8 Playlist Parser and Downloader Allowed by Law. Automatic AES Decryption. All HLS Tags Are Supported. All SCTE-35 Tags are Supported.

Python 100.00%
aes-decryption hls m3u8 playlists python scte35 video

m3ufu's People

Contributors

futzu avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar

Forkers

239144498 brahimmade alteoxcom

m3ufu's Issues

Clarification of equals character being dropped on #EXT-OATCLS-SCTE35 messages

I have been failing to decode some #EXT-OATCLS-SCTE35 messages with m3ufu.

My assumption is that I am doing something incorrect, but I would welcome clarification on expected behavior.

Because I do not trust myself, I'll use a SCTE message from Adrian's examples/timesignal/time_signal_placement_opportunity_end.py. I appreciate that this is not a CUE-OUT message, I'm just using a tried and trusted cue message from the threefive samples as an exemplum.

Here's the cue from the threefive repo...

/DAvAAAAAAAA///wBQb+dGKQoAAZAhdDVUVJSAAAjn+fCAgAAAAALKChijUCAKnMZ1g=

Lets run this cue through threefive. It decodes the command nicely (and of course it should, because it is one of Adrian's examples)...

$ threefive /DAvAAAAAAAA///wBQb+dGKQoAAZAhdDVUVJSAAAjn+fCAgAAAAALKChijUCAKnMZ1g=

{
    "info_section": {
        "table_id": "0xfc",
        "section_syntax_indicator": false,
        "private": false,
        "sap_type": "0x3",
        "sap_details": "No Sap Type",
        "section_length": 47,
        "protocol_version": 0,
        "encrypted_packet": false,
        "encryption_algorithm": 0,
        "pts_adjustment_ticks": 0,
        "pts_adjustment": 0.0,
        "cw_index": "0xff",
        "tier": "0xfff",
        "splice_command_length": 5,
        "splice_command_type": 6,
        "descriptor_loop_length": 25,
        "crc": "0xa9cc6758"
    },
    "command": {
        "command_length": 5,
        "command_type": 6,
        "name": "Time Signal",
        "time_specified_flag": true,
        "pts_time": 21695.740089,
        "pts_time_ticks": 1952616608
    },
    "descriptors": [
        {
            "tag": 2,
            "descriptor_length": 23,
            "name": "Segmentation Descriptor",
            "identifier": "CUEI",
            "components": [],
            "segmentation_event_id": "0x4800008e",
            "segmentation_event_cancel_indicator": false,
            "program_segmentation_flag": true,
            "segmentation_duration_flag": false,
            "delivery_not_restricted_flag": false,
            "web_delivery_allowed_flag": true,
            "no_regional_blackout_flag": true,
            "archive_allowed_flag": true,
            "device_restrictions": "No Restrictions",
            "segmentation_message": "Provider Placement Opportunity End",
            "segmentation_upid_type": 8,
            "segmentation_upid_type_name": "AiringID",
            "segmentation_upid_length": 8,
            "segmentation_upid": "0x2ca0a18a",
            "segmentation_type_id": 53,
            "segment_num": 2,
            "segments_expected": 0
        }
    ]
}

Lets put it in a bare-bones m3u8 wrapper and put that message in an #EXT-OATCLS-SCTE35 tag...

$ cat ./time_signal_placement_opportunity_end.m3u8 

#EXTM3U
#EXT-X-PLAYLIST-TYPE:VOD
#EXT-X-VERSION:3
#EXT-X-TARGETDURATION:30
#EXT-X-MEDIA-SEQUENCE:0

#EXT-OATCLS-SCTE35:/DAvAAAAAAAA///wBQb+dGKQoAAZAhdDVUVJSAAAjn+fCAgAAAAALKChijUCAKnMZ1g=
#EXTINF:30.000,
segment01.ts

#EXT-X-ENDLIST

Now lets run m3ufu on that guy...

$ m3ufu -i ./time_signal_placement_opportunity_end.m3u8 

{
    "headers": {
        "#EXTM3U": "",
        "#EXT-X-PLAYLIST-TYPE": "VOD",
        "#EXT-X-VERSION": "3",
        "#EXT-X-TARGETDURATION": "30",
        "#EXT-X-MEDIA-SEQUENCE": "0"
    },
    "media": [
        {
            "media": "segment01.ts",
            "duration": 30.0,
            "cue": "/DAvAAAAAAAA///wBQb+dGKQoAAZAhdDVUVJSAAAjn+fCAgAAAAALKChijUCAKnMZ1g",
            "tags": {
                "#EXT-OATCLS-SCTE35": {},
                "#EXTINF": 30.0
            }
        }
    ]
}

I was expecting to see:

  • "cue_data":{} to be populated. It is not.
  • "cue": payload to match that of the #EXT-OATCLS-SCTE35. There is an equals sign missing from the B64 payload in the output of m3ufu.

But, lets manually put an extra equals character on the end in the HLS manifest, on the assumption that is is getting dropped in the parsing of m3ufu and prior to decode by threefive...

$ cat ./time_signal_placement_opportunity_end_withextraequals.m3u8

#EXTM3U
#EXT-X-PLAYLIST-TYPE:VOD
#EXT-X-VERSION:3
#EXT-X-TARGETDURATION:30
#EXT-X-MEDIA-SEQUENCE:0

#EXT-OATCLS-SCTE35:/DAvAAAAAAAA///wBQb+dGKQoAAZAhdDVUVJSAAAjn+fCAgAAAAALKChijUCAKnMZ1g==
#EXTINF:30.000,
segment01.ts

#EXT-X-ENDLIST

Boom! m3ufu fully decodes the cue this time... One equals character is still dropped, but this time:

  • "cue_data":{} is populated and reflects the output of threefive
  • "command":{} data now includes what I expected to see.
  • "cue": still does not match that of #EXT-OATCLS-SCTE35
$ m3ufu -i ./time_signal_placement_opportunity_end_withextraequals.m3u8 
{
    "headers": {
        "#EXTM3U": "",
        "#EXT-X-PLAYLIST-TYPE": "VOD",
        "#EXT-X-VERSION": "3",
        "#EXT-X-TARGETDURATION": "30",
        "#EXT-X-MEDIA-SEQUENCE": "0"
    },
    "media": [
        {
            "media": "segment01.ts",
            "duration": 30.0,
            "cue": "/DAvAAAAAAAA///wBQb+dGKQoAAZAhdDVUVJSAAAjn+fCAgAAAAALKChijUCAKnMZ1g=",
            "cue_data": {
                "info_section": {
                    "table_id": "0xfc",
                    "section_syntax_indicator": false,
                    "private": false,
                    "sap_type": "0x3",
                    "sap_details": "No Sap Type",
                    "section_length": 47,
                    "protocol_version": 0,
                    "encrypted_packet": false,
                    "encryption_algorithm": 0,
                    "pts_adjustment_ticks": 0,
                    "pts_adjustment": 0.0,
                    "cw_index": "0xff",
                    "tier": "0xfff",
                    "splice_command_length": 5,
                    "splice_command_type": 6,
                    "descriptor_loop_length": 25,
                    "crc": "0xa9cc6758"
                },
                "command": {
                    "command_length": 5,
                    "command_type": 6,
                    "name": "Time Signal",
                    "time_specified_flag": true,
                    "pts_time": 21695.740089,
                    "pts_time_ticks": 1952616608
                },
                "descriptors": [
                    {
                        "tag": 2,
                        "descriptor_length": 23,
                        "name": "Segmentation Descriptor",
                        "identifier": "CUEI",
                        "components": [],
                        "segmentation_event_id": "0x4800008e",
                        "segmentation_event_cancel_indicator": false,
                        "program_segmentation_flag": true,
                        "segmentation_duration_flag": false,
                        "delivery_not_restricted_flag": false,
                        "web_delivery_allowed_flag": true,
                        "no_regional_blackout_flag": true,
                        "archive_allowed_flag": true,
                        "device_restrictions": "No Restrictions",
                        "segmentation_message": "Provider Placement Opportunity End",
                        "segmentation_upid_type": 8,
                        "segmentation_upid_type_name": "AiringID",
                        "segmentation_upid_length": 8,
                        "segmentation_upid": "0x2ca0a18a",
                        "segmentation_type_id": 53,
                        "segment_num": 2,
                        "segments_expected": 0
                    }
                ]
            },
            "tags": {
                "#EXT-OATCLS-SCTE35": {},
                "#EXTINF": 30.0
            }
        }
    ]
}

Question 1: _Is m3ufu intentionally dropping the final equals sign when the tag type is #EXT-OATCLS-SCTE35? It looks like m3ufu is dropping the final '=' because it is also a delimiter.

Question 2, I'm seeing a similar issue with #EXT-X-CUE-OUT-CONT not getting decoded into cue_data, both because of the equals issue above and because self._do_cue() is not called. Here's a test case that I knocked up.

#EXTM3U
#EXT-X-PLAYLIST-TYPE:VOD
#EXT-X-VERSION:3
#EXT-X-TARGETDURATION:30
#EXT-X-MEDIA-SEQUENCE:0

#EXT-X-CUE-OUT-CONT:TEST1=helloWorld,TEST2=helloWorld2=,TEST3=helloWorld3==,TEST4=hello=World4,SCTE35=/DAvAAAAAAAA///wFAVIAACPf+/+c2nALv4AUsz1AAAAAAAKAAhDVUVJAAABNWLbowo,SCTE35=/DAvAAAAAAAA///wFAVIAACPf+/+c2nALv4AUsz1AAAAAAAKAAhDVUVJAAABNWLbowo=,SCTE35=/DAvAAAAAAAA///wFAVIAACPf+/+c2nALv4AUsz1AAAAAAAKAAhDVUVJAAABNWLbowo==
#EXTINF:30.000,
segment01.ts

#EXT-X-ENDLIST

Here's the output, with TEST2,3,4 showing that is the payload contains equals, then BadStuffHappens.

            "tags": {
                "#EXT-X-CUE-OUT-CONT": {
                    "SCTE35=/DAvAAAAAAAA///wFAVIAACPf+/+c2nALv4AUsz1AAAAAAAKAAhDVUVJAAABNWLbowo=": "",
                    "SCTE35=/DAvAAAAAAAA///wFAVIAACPf+/+c2nALv4AUsz1AAAAAAAKAAhDVUVJAAABNWLbowo": "",
                    "SCTE35": "/DAvAAAAAAAA///wFAVIAACPf+/+c2nALv4AUsz1AAAAAAAKAAhDVUVJAAABNWLbowo",
                    "TEST4=hello": "World4",
                    "TEST3=helloWorld3=": "",
                    "TEST2=helloWorld2": "",
                    "TEST1": "helloWorld"
                },

And to prove this, I modified the #EXT-X-CUE-OUT-CONT handler at

m3ufu/m3ufu.py

Line 331 in faea00b

if "#EXT-X-CUE-OUT-CONT" in self.tags:
to:

  1. throw on an extra = onto self.cue. (I am a bad person and I will go to hell).
  2. added self._do_cue() which was missing from the #EXT-X-CUE-OUT-CONT code.
        if "#EXT-X-CUE-OUT-CONT" in self.tags:
            # print(self.tags)
            # #EXT-X-CUE-OUT-CONT tag does not get parsed correctly if the payload contains =.  Adding manually...
            if "SCTE35" in self.tags["#EXT-X-CUE-OUT-CONT"]:
                self.cue = self.tags["#EXT-X-CUE-OUT-CONT"]["SCTE35"] + "="
                self._do_cue()
                return

With the above crime-against-python, #EXT-X-CUE-OUT-CONT successfully decoded. It is a dirty proof of concept, but I did manage to get it to produce cue_data for CONT messages. Please excuse me - as you have been painfully aware from our various back and forth, I am not a developer... And I am illiterate when it comes to python. But I hope the above at least demonstrates the issue.

So I suspect that there's a wider issue with cues that contain equals character, due to python treating any equal character as being a delimiter during tag parsing. The trouble is that a lot of base-64 SCTE-35 cues contain the equals character.

For ref, here's my current versions...

$ pip3 show m3ufu
Name: m3ufu
Version: 0.0.55
$ pip3 show threefive
Name: threefive
Version: 2.3.69

As ever, BigRespect for your work and the suite of tools.

Error While Parsing the m3u8 Url

Traceback (most recent call last):
File "/usr/local/bin/m3ufu", line 5, in
cli()
File "/usr/local/lib/pypy3.6/dist-packages/m3ufu.py", line 624, in cli
fu.decode()
File "/usr/local/lib/pypy3.6/dist-packages/m3ufu.py", line 611, in decode
if not self._parse_line():
File "/usr/local/lib/pypy3.6/dist-packages/m3ufu.py", line 594, in _parse_line
self._do_media(line)
File "/usr/local/lib/pypy3.6/dist-packages/m3ufu.py", line 560, in _do_media
self._add_media(media)
File "/usr/local/lib/pypy3.6/dist-packages/m3ufu.py", line 537, in _add_media
segment = Segment(self.chunk, media, self._start, self.base_uri)
File "/usr/local/lib/pypy3.6/dist-packages/m3ufu.py", line 258, in init
self.relative_uri = media_uri.removeprefix(base_uri)
AttributeError: 'str' object has no attribute 'removeprefix'

Sponsor

Hi Adrian,

I'm working on an experimental project related to inserting SCTE-35 markers in a HLS feed.

Would you be open to a sponsorship opportunity? If so, what's the best way to contact you?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.