futzu / m3ufu Goto Github PK
View Code? Open in Web Editor NEWThe Most Pythonic M3U8 Playlist Parser and Downloader Allowed by Law. Automatic AES Decryption. All HLS Tags Are Supported. All SCTE-35 Tags are Supported.
The Most Pythonic M3U8 Playlist Parser and Downloader Allowed by Law. Automatic AES Decryption. All HLS Tags Are Supported. All SCTE-35 Tags are Supported.
I have been failing to decode some #EXT-OATCLS-SCTE35 messages with m3ufu.
My assumption is that I am doing something incorrect, but I would welcome clarification on expected behavior.
Because I do not trust myself, I'll use a SCTE message from Adrian's examples/timesignal/time_signal_placement_opportunity_end.py. I appreciate that this is not a CUE-OUT message, I'm just using a tried and trusted cue message from the threefive samples as an exemplum.
Here's the cue from the threefive repo...
/DAvAAAAAAAA///wBQb+dGKQoAAZAhdDVUVJSAAAjn+fCAgAAAAALKChijUCAKnMZ1g=
Lets run this cue through threefive
. It decodes the command nicely (and of course it should, because it is one of Adrian's examples)...
$ threefive /DAvAAAAAAAA///wBQb+dGKQoAAZAhdDVUVJSAAAjn+fCAgAAAAALKChijUCAKnMZ1g=
{
"info_section": {
"table_id": "0xfc",
"section_syntax_indicator": false,
"private": false,
"sap_type": "0x3",
"sap_details": "No Sap Type",
"section_length": 47,
"protocol_version": 0,
"encrypted_packet": false,
"encryption_algorithm": 0,
"pts_adjustment_ticks": 0,
"pts_adjustment": 0.0,
"cw_index": "0xff",
"tier": "0xfff",
"splice_command_length": 5,
"splice_command_type": 6,
"descriptor_loop_length": 25,
"crc": "0xa9cc6758"
},
"command": {
"command_length": 5,
"command_type": 6,
"name": "Time Signal",
"time_specified_flag": true,
"pts_time": 21695.740089,
"pts_time_ticks": 1952616608
},
"descriptors": [
{
"tag": 2,
"descriptor_length": 23,
"name": "Segmentation Descriptor",
"identifier": "CUEI",
"components": [],
"segmentation_event_id": "0x4800008e",
"segmentation_event_cancel_indicator": false,
"program_segmentation_flag": true,
"segmentation_duration_flag": false,
"delivery_not_restricted_flag": false,
"web_delivery_allowed_flag": true,
"no_regional_blackout_flag": true,
"archive_allowed_flag": true,
"device_restrictions": "No Restrictions",
"segmentation_message": "Provider Placement Opportunity End",
"segmentation_upid_type": 8,
"segmentation_upid_type_name": "AiringID",
"segmentation_upid_length": 8,
"segmentation_upid": "0x2ca0a18a",
"segmentation_type_id": 53,
"segment_num": 2,
"segments_expected": 0
}
]
}
Lets put it in a bare-bones m3u8 wrapper and put that message in an #EXT-OATCLS-SCTE35
tag...
$ cat ./time_signal_placement_opportunity_end.m3u8
#EXTM3U
#EXT-X-PLAYLIST-TYPE:VOD
#EXT-X-VERSION:3
#EXT-X-TARGETDURATION:30
#EXT-X-MEDIA-SEQUENCE:0
#EXT-OATCLS-SCTE35:/DAvAAAAAAAA///wBQb+dGKQoAAZAhdDVUVJSAAAjn+fCAgAAAAALKChijUCAKnMZ1g=
#EXTINF:30.000,
segment01.ts
#EXT-X-ENDLIST
Now lets run m3ufu on that guy...
$ m3ufu -i ./time_signal_placement_opportunity_end.m3u8
{
"headers": {
"#EXTM3U": "",
"#EXT-X-PLAYLIST-TYPE": "VOD",
"#EXT-X-VERSION": "3",
"#EXT-X-TARGETDURATION": "30",
"#EXT-X-MEDIA-SEQUENCE": "0"
},
"media": [
{
"media": "segment01.ts",
"duration": 30.0,
"cue": "/DAvAAAAAAAA///wBQb+dGKQoAAZAhdDVUVJSAAAjn+fCAgAAAAALKChijUCAKnMZ1g",
"tags": {
"#EXT-OATCLS-SCTE35": {},
"#EXTINF": 30.0
}
}
]
}
I was expecting to see:
"cue_data":{}
to be populated. It is not."cue":
payload to match that of the #EXT-OATCLS-SCTE35
. There is an equals sign missing from the B64 payload in the output of m3ufu.But, lets manually put an extra equals character on the end in the HLS manifest, on the assumption that is is getting dropped in the parsing of m3ufu and prior to decode by threefive...
$ cat ./time_signal_placement_opportunity_end_withextraequals.m3u8
#EXTM3U
#EXT-X-PLAYLIST-TYPE:VOD
#EXT-X-VERSION:3
#EXT-X-TARGETDURATION:30
#EXT-X-MEDIA-SEQUENCE:0
#EXT-OATCLS-SCTE35:/DAvAAAAAAAA///wBQb+dGKQoAAZAhdDVUVJSAAAjn+fCAgAAAAALKChijUCAKnMZ1g==
#EXTINF:30.000,
segment01.ts
#EXT-X-ENDLIST
Boom! m3ufu fully decodes the cue this time... One equals character is still dropped, but this time:
"cue_data":{}
is populated and reflects the output of threefive"command":{}
data now includes what I expected to see."cue":
still does not match that of #EXT-OATCLS-SCTE35
$ m3ufu -i ./time_signal_placement_opportunity_end_withextraequals.m3u8
{
"headers": {
"#EXTM3U": "",
"#EXT-X-PLAYLIST-TYPE": "VOD",
"#EXT-X-VERSION": "3",
"#EXT-X-TARGETDURATION": "30",
"#EXT-X-MEDIA-SEQUENCE": "0"
},
"media": [
{
"media": "segment01.ts",
"duration": 30.0,
"cue": "/DAvAAAAAAAA///wBQb+dGKQoAAZAhdDVUVJSAAAjn+fCAgAAAAALKChijUCAKnMZ1g=",
"cue_data": {
"info_section": {
"table_id": "0xfc",
"section_syntax_indicator": false,
"private": false,
"sap_type": "0x3",
"sap_details": "No Sap Type",
"section_length": 47,
"protocol_version": 0,
"encrypted_packet": false,
"encryption_algorithm": 0,
"pts_adjustment_ticks": 0,
"pts_adjustment": 0.0,
"cw_index": "0xff",
"tier": "0xfff",
"splice_command_length": 5,
"splice_command_type": 6,
"descriptor_loop_length": 25,
"crc": "0xa9cc6758"
},
"command": {
"command_length": 5,
"command_type": 6,
"name": "Time Signal",
"time_specified_flag": true,
"pts_time": 21695.740089,
"pts_time_ticks": 1952616608
},
"descriptors": [
{
"tag": 2,
"descriptor_length": 23,
"name": "Segmentation Descriptor",
"identifier": "CUEI",
"components": [],
"segmentation_event_id": "0x4800008e",
"segmentation_event_cancel_indicator": false,
"program_segmentation_flag": true,
"segmentation_duration_flag": false,
"delivery_not_restricted_flag": false,
"web_delivery_allowed_flag": true,
"no_regional_blackout_flag": true,
"archive_allowed_flag": true,
"device_restrictions": "No Restrictions",
"segmentation_message": "Provider Placement Opportunity End",
"segmentation_upid_type": 8,
"segmentation_upid_type_name": "AiringID",
"segmentation_upid_length": 8,
"segmentation_upid": "0x2ca0a18a",
"segmentation_type_id": 53,
"segment_num": 2,
"segments_expected": 0
}
]
},
"tags": {
"#EXT-OATCLS-SCTE35": {},
"#EXTINF": 30.0
}
}
]
}
Question 1: _Is m3ufu intentionally dropping the final equals sign when the tag type is #EXT-OATCLS-SCTE35
? It looks like m3ufu is dropping the final '=' because it is also a delimiter.
Question 2, I'm seeing a similar issue with #EXT-X-CUE-OUT-CONT
not getting decoded into cue_data
, both because of the equals issue above and because self._do_cue()
is not called. Here's a test case that I knocked up.
#EXTM3U
#EXT-X-PLAYLIST-TYPE:VOD
#EXT-X-VERSION:3
#EXT-X-TARGETDURATION:30
#EXT-X-MEDIA-SEQUENCE:0
#EXT-X-CUE-OUT-CONT:TEST1=helloWorld,TEST2=helloWorld2=,TEST3=helloWorld3==,TEST4=hello=World4,SCTE35=/DAvAAAAAAAA///wFAVIAACPf+/+c2nALv4AUsz1AAAAAAAKAAhDVUVJAAABNWLbowo,SCTE35=/DAvAAAAAAAA///wFAVIAACPf+/+c2nALv4AUsz1AAAAAAAKAAhDVUVJAAABNWLbowo=,SCTE35=/DAvAAAAAAAA///wFAVIAACPf+/+c2nALv4AUsz1AAAAAAAKAAhDVUVJAAABNWLbowo==
#EXTINF:30.000,
segment01.ts
#EXT-X-ENDLIST
Here's the output, with TEST2,3,4 showing that is the payload contains equals, then BadStuffHappens.
"tags": {
"#EXT-X-CUE-OUT-CONT": {
"SCTE35=/DAvAAAAAAAA///wFAVIAACPf+/+c2nALv4AUsz1AAAAAAAKAAhDVUVJAAABNWLbowo=": "",
"SCTE35=/DAvAAAAAAAA///wFAVIAACPf+/+c2nALv4AUsz1AAAAAAAKAAhDVUVJAAABNWLbowo": "",
"SCTE35": "/DAvAAAAAAAA///wFAVIAACPf+/+c2nALv4AUsz1AAAAAAAKAAhDVUVJAAABNWLbowo",
"TEST4=hello": "World4",
"TEST3=helloWorld3=": "",
"TEST2=helloWorld2": "",
"TEST1": "helloWorld"
},
And to prove this, I modified the #EXT-X-CUE-OUT-CONT
handler at
Line 331 in faea00b
self.cue
. (I am a bad person and I will go to hell).self._do_cue()
which was missing from the #EXT-X-CUE-OUT-CONT
code. if "#EXT-X-CUE-OUT-CONT" in self.tags:
# print(self.tags)
# #EXT-X-CUE-OUT-CONT tag does not get parsed correctly if the payload contains =. Adding manually...
if "SCTE35" in self.tags["#EXT-X-CUE-OUT-CONT"]:
self.cue = self.tags["#EXT-X-CUE-OUT-CONT"]["SCTE35"] + "="
self._do_cue()
return
With the above crime-against-python, #EXT-X-CUE-OUT-CONT
successfully decoded. It is a dirty proof of concept, but I did manage to get it to produce cue_data
for CONT messages. Please excuse me - as you have been painfully aware from our various back and forth, I am not a developer... And I am illiterate when it comes to python. But I hope the above at least demonstrates the issue.
So I suspect that there's a wider issue with cues that contain equals character, due to python treating any equal character as being a delimiter during tag parsing. The trouble is that a lot of base-64 SCTE-35 cues contain the equals character.
For ref, here's my current versions...
$ pip3 show m3ufu
Name: m3ufu
Version: 0.0.55
$ pip3 show threefive
Name: threefive
Version: 2.3.69
As ever, BigRespect for your work and the suite of tools.
Traceback (most recent call last):
File "/usr/local/bin/m3ufu", line 5, in
cli()
File "/usr/local/lib/pypy3.6/dist-packages/m3ufu.py", line 624, in cli
fu.decode()
File "/usr/local/lib/pypy3.6/dist-packages/m3ufu.py", line 611, in decode
if not self._parse_line():
File "/usr/local/lib/pypy3.6/dist-packages/m3ufu.py", line 594, in _parse_line
self._do_media(line)
File "/usr/local/lib/pypy3.6/dist-packages/m3ufu.py", line 560, in _do_media
self._add_media(media)
File "/usr/local/lib/pypy3.6/dist-packages/m3ufu.py", line 537, in _add_media
segment = Segment(self.chunk, media, self._start, self.base_uri)
File "/usr/local/lib/pypy3.6/dist-packages/m3ufu.py", line 258, in init
self.relative_uri = media_uri.removeprefix(base_uri)
AttributeError: 'str' object has no attribute 'removeprefix'
Hi Adrian,
I'm working on an experimental project related to inserting SCTE-35 markers in a HLS feed.
Would you be open to a sponsorship opportunity? If so, what's the best way to contact you?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.