fnep / mtv_dl Goto Github PK
View Code? Open in Web Editor NEWMediathekView Downloader
License: MIT License
MediathekView Downloader
License: MIT License
Guten Abend
Seitdem die Progressbar mit der neuen Bibliothek (Paket rich) implementiert wurde, erscheint sporadisch diese Fehlermeldung bei einem beliebigen Download:
2020-05-17T22:05:56 CRITICAL <class 'ZeroDivisionError'>: float division by zero File "mtv_dl.py", line 1168, in <module> main() File "mtv_dl.py", line 1151, in main downloader.download(quality_preference, # type: ignore File "mtv_dl.py", line 1005, in download subtitles_xml_path = list(self._download_files(temp_path, [self.show['url_subtitles']]))[0] File "mtv_dl.py", line 814, in _download_files yield destination_file_path File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "mtv_dl.py", line 246, in progress_bar yield progress File "/home/schui/.local/lib/python3.8/site-packages/rich/progress.py", line 420, in __exit__ self.stop() File "/home/schui/.local/lib/python3.8/site-packages/rich/progress.py", line 402, in stop self.refresh() File "/home/schui/.local/lib/python3.8/site-packages/rich/progress.py", line 550, in refresh self.console.print(self._live_render) File "/home/schui/.local/lib/python3.8/site-packages/rich/console.py", line 720, in print extend(render(renderable, render_options)) File "/home/schui/.local/lib/python3.8/site-packages/rich/console.py", line 495, in render yield from self._render(renderable, options or self.options) File "/home/schui/.local/lib/python3.8/site-packages/rich/console.py", line 471, in _render for render_output in iter_render: File "/home/schui/.local/lib/python3.8/site-packages/rich/live_render.py", line 35, in __console__ lines = console.render_lines(self.renderable, options, style, pad=False) File "/home/schui/.local/lib/python3.8/site-packages/rich/console.py", line 523, in render_lines lines = list( File "/home/schui/.local/lib/python3.8/site-packages/rich/segment.py", line 110, in split_and_crop_lines for segment in segments: File "/home/schui/.local/lib/python3.8/site-packages/rich/segment.py", line 65, in <genexpr> return ( File "/home/schui/.local/lib/python3.8/site-packages/rich/console.py", line 495, in render yield from self._render(renderable, options or self.options) File "/home/schui/.local/lib/python3.8/site-packages/rich/console.py", line 475, in _render yield from self.render(render_output, render_options) File "/home/schui/.local/lib/python3.8/site-packages/rich/console.py", line 495, in render yield from self._render(renderable, options or self.options) File "/home/schui/.local/lib/python3.8/site-packages/rich/console.py", line 471, in _render for render_output in iter_render: File "/home/schui/.local/lib/python3.8/site-packages/rich/table.py", line 324, in __console__ yield from self._render(console, options, widths) File "/home/schui/.local/lib/python3.8/site-packages/rich/table.py", line 488, in _render lines = console.render_lines( File "/home/schui/.local/lib/python3.8/site-packages/rich/console.py", line 523, in render_lines lines = list( File "/home/schui/.local/lib/python3.8/site-packages/rich/segment.py", line 110, in split_and_crop_lines for segment in segments: File "/home/schui/.local/lib/python3.8/site-packages/rich/segment.py", line 65, in <genexpr> return ( File "/home/schui/.local/lib/python3.8/site-packages/rich/console.py", line 495, in render yield from self._render(renderable, options or self.options) File "/home/schui/.local/lib/python3.8/site-packages/rich/console.py", line 471, in _render for render_output in iter_render: File "/home/schui/.local/lib/python3.8/site-packages/rich/padding.py", line 83, in __console__ lines = console.render_lines(self.renderable, child_options, style=style) File "/home/schui/.local/lib/python3.8/site-packages/rich/console.py", line 523, in render_lines lines = list( File "/home/schui/.local/lib/python3.8/site-packages/rich/segment.py", line 110, in split_and_crop_lines for segment in segments: File "/home/schui/.local/lib/python3.8/site-packages/rich/segment.py", line 65, in <genexpr> return ( File "/home/schui/.local/lib/python3.8/site-packages/rich/console.py", line 495, in render yield from self._render(renderable, options or self.options) File "/home/schui/.local/lib/python3.8/site-packages/rich/console.py", line 471, in _render for render_output in iter_render: File "/home/schui/.local/lib/python3.8/site-packages/rich/bar.py", line 65, in __console__ complete_halves = int(width * 2 * completed / self.total)
Aktuelles Beispielt wäre: 2020-05-17T22:39:26 INFO Saved show 'Extra 3 vom 13.05.2020 mit Christian Ehring' (ARD, 'extra 3', 2020-05-13 21:10:00, 39190c75c9c)
Danke für den Fix.
Gruss schui95
Today I encountered this error
2023-04-15T10:27:27 DEBUG Downloading subtitles for 'Michel in der Suppenschüssel' (ARD, 'Astrid Lindgren', 2023-03-26 16:40:00, 0e2f6a7e798) from 'https://api.ardmediathek.de/player-service/subtitle/ebutt/urn:ard:subtitle:9696fedef144f1d8'. Downloading 'Michel in der Suppenschüssel' (ARD, 'Astrid Lindgren', 2023-03-26 16:40:00, 0e2f6a7e798)
2023-04-15T10:27:30 ERROR Download of 'Michel in der Suppenschüssel' (ARD, 'Astrid Lindgren', 2023-03-26 16:40:00, 0e2f6a7e798) failed: [Errno 22] Invalid argument: 'C:\\PUBLIC\\.tmpbwqlmlek\\urn:ard:subtitle:9696fedef144f1d8'
Colons : are not supported in Windows filenames.
Using mtv_dl==0.22.0
.
The download of subtitles failed. In most cases it's fine, but for the following movie the subtitles file .srt is created but empty.
python3 mtv_dl.py download --oblivious --target="{start} {title}{ext}" hash=2f2bad3ed7c
Downloading 'Der erste April (26)' (ARD, 'Meister Eder und sein Pumuckl', 2020-07-05 04:40:00, 2f2bad3ed7c) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% 0:00:00
2020-07-14T20:07:55 INFO Saved show 'Der erste April (26)' (ARD, 'Meister Eder und sein Pumuckl', 2020-07-05 04:40:00, 2f2bad3ed7c) to PosixPath('2020-07-05 04_40_00 Der erste April (26).mp4').
Downloading 'Der erste April (26)' (ARD, 'Meister Eder und sein Pumuckl', 2020-07-05 04:40:00, 2f2bad3ed7c) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% 0:00:00
2020-07-14T20:07:56 INFO Saved subtitles 'Der erste April (26)' (ARD, 'Meister Eder und sein Pumuckl', 2020-07-05 04:40:00, 2f2bad3ed7c) to PosixPath('2020-07-05 04_40_00 Der erste April (26).srt').
The file 2020-07-05 04_40_00 Der erste April (26).srt
is created but empty.
The original subtitles file in the mediathek has content:
https://classic.ardmediathek.de/subtitle/411383
a search like
mtv_dl list topic="Die Sendung mit der Maus" duration=30m
does not produce any results, even though it is stated that it should work:
'=' Pattern is a regular expression case insensitive search within the field value.
Available for the fields 'description', 'start', 'duration', 'age', 'region',
'size', 'channel', 'topic', 'title', 'hash' and 'url'.
Searching without duration shows that there should be search results.
Btw, great work! I am very excited about having found this!
I installed mtv_dl in a virtual_env
It was working fine on another host with ubuntu but here it doesnt work.
I do see:
(mtv) iUser@TP-W500-iUser:~/winshares/media/video/serien/Babylon Berlin/Season 3$ mtv_dl history
2023-10-03T00:10:37 CRITICAL <class 'sqlite3.OperationalError'>: no such table: main.show
File "/home/iUser/mtv/bin/mtv_dl", line 8, in
sys.exit(main())
File "/home/iUser/mtv/lib/python3.10/site-packages/mtv_dl.py", line 1461, in main
showlist = Database(filmliste=cw_dir / FILMLISTE_DATABASE_FILE, history=cw_dir / HISTORY_DATABASE_FILE)
File "/home/iUser/mtv/lib/python3.10/site-packages/mtv_dl.py", line 471, in init
self.initialize_filmliste()
File "/home/iUser/mtv/lib/python3.10/site-packages/mtv_dl.py", line 377, in initialize_filmliste
cursor.execute("DELETE FROM main.show")
Guten Abend
Vielen Dank für das einfache und schlanke Tool!
Ich habe Probleme neuere Sendungen vom SRF zu finden und dann herunterzuladen.
Beispiele wären:
channel=SRF topic=#SRFglobal duration+20m
channel=SRF topic=Deville duration+20m
channel=SRF topic=Einstein duration+45m
Kann es sein, dass Geoblocking oder so aktiviert ist und somit nicht die vollständige Datenbank heruntergeladen wird oder verwende ich das Tool einfach falsch. Für Sendungen im ZDF oder ARD funktioniert es bisher einwandfrei.
Vielen Dank für jede Hilfe.
Gruss schui95
Hi,
first of all thanks for the work put into this. I recently moved mtv to my pi4 and it works really nice.;)
Not sure if this is inteded behaviour. But taking the 'famous' 'Tatort' as an example:
The show is being aired on multiple channels. At different times they show different (or same) episodes.
If i set a filter title!='Audiodeskription' title!='klare Sprache' title!='Trailer' topic='Tatort' duration+70m
i see a lot of duplicates with different hashes because of the channel-hash (even the start-time is identical).
Is this inteded? Are there shows with identical topic,title,size and time but different content?
Or could the channel-hash be removed to prevent redownloading?
Or is there another possibility?
regards,
strowi
The exclude-future option (respective not setting the --include-future option) hides all show of the current day, so also the shows in the past of the current day.
Steps to reproduce: (assume current time is 21:00):
mtv_dl.py list age-20h
will not show a single show.
mtv_dl.py list --include-future age-20h
will show a lot of shows in the past of the current day.
Proposal: Consider also the hour of the day, and not just the date, when filtering the shows.
Change
where.append("date(show.start) < date('now')")
to
where.append("datetime(show.start) < datetime('now')")
When using pytest
to test #61 I get this warning:
.venv/lib/python3.9/site-packages/ijson/compat.py:48
/home/kuehnelc/git/mtv_dl/.venv/lib/python3.9/site-packages/ijson/compat.py:48: DeprecationWarning:
ijson works by reading bytes, but a string reader has been given instead. This
probably, but not necessarily, means a file-like object has been opened in text
mode ('t') rather than binary mode ('b').
An automatic conversion is being performed on the fly to continue, but on the
other hand this creates unnecessary encoding/decoding operations that decrease
the efficiency of the system. In the future this automatic conversion will be
removed, and users will receive errors instead of this warning. To avoid this
problem make sure file-like objects are opened in binary mode instead of text
mode.
warnings.warn(_str_vs_bytes_warning, DeprecationWarning)
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
Thx for building this great tool 👍 ! It fits my use case much better than the Mediathek!
After downloading a video, I would like mtv_dl
to trigger a local script. In my case I want the script to do 2 things: Send me an email about the new download and trigger Jellyfin to update the media library.
The script should be called with configurable arguments, similar to the path configuration, in the config file:
post-download: /some/path/myscript "{title}" "{path}"
Design proposal, in case you would like me to implement this:
post-download
option is set:
subprocess.call
the script with shell=True
mtv_dl
should continue it's work even if the post-download script fails.What do you think?
Would be great if a .nfo file for each downloaded movie is created.
This should be compatible for Kodi https://kodi.wiki/view/NFO_files#Metadata_nfo
Hi!
I just installed mtv_dl on an Alpine Linux VM.
I was able to do a couple of list commands, and then all of a sudden, this error message popped up:
2020-11-05T10:34:15 CRITICAL <class 'sqlite3.OperationalError'>: unable to open database file
File "/home/matti/.local/bin/mtv_dl", line 8, in <module>
sys.exit(main())
File "/home/matti/.local/lib/python3.8/site-packages/mtv_dl.py", line 1202, in main
showlist = Database(filmliste=cw_dir / FILMLISTE_DATABASE_FILE,
File "/home/matti/.local/lib/python3.8/site-packages/mtv_dl.py", line 414, in __init__
self.connection = sqlite3.connect(filmliste_path.absolute().as_posix(),
Rebooting didn't change anything, also there is enough space on the drive.
Hi,
i just ran into this issue on a clean pi4 with v0.23.0:
~> mtv_dl list -r title='Letzte Spur'
Downloading database
....
━━━━━━━━━━━━━━━━━━━━━━━╺ 99.6% 0:00:02Reading database items ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% 0:00:00
2023-06-02T21:15:05 INFO Initializing History database in PosixPath('/home/osmc/mtv/.Filmliste.0.23.0.sqlite').
2023-06-02T21:15:05 CRITICAL <class 'sqlite3.OperationalError'>: near "(": syntax error
File "/usr/local/bin/mtv_dl", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.9/dist-packages/mtv_dl.py", line 1461, in main
showlist = Database(filmliste=cw_dir / FILMLISTE_DATABASE_FILE, history=cw_dir / HISTORY_DATABASE_FILE)
File "/usr/local/lib/python3.9/dist-packages/mtv_dl.py", line 473, in __init__
self.initialize_history()
File "/usr/local/lib/python3.9/dist-packages/mtv_dl.py", line 420, in initialize_history
cursor.execute(
Deleting the sqlite-files, downgrading to v0.22.0 and running the command again works. And after that i can upgrade to v0.23.0 which shows an additional info
2023-06-02T21:40:10 INFO Upgrading history database schema, adding columns for season and episode
and then it succeeds.
Looks to me as if sth. changed in the initialisation of the db?
Hi Frank,
since last few days I get an overflowError:
$ mtv_dl -v list channel=ZDF topic=Heute
2022-02-18T18:23:24 DEBUG Opening Filmliste database
PosixPath('/smb/video/MediathekView/.Filmliste.0.18.1.sqlite').
DEBUG Opening History database PosixPath('/smb/video/MediathekView/.History.sqlite').
DEBUG Initializing Filmliste database in
PosixPath('/smb/video/MediathekView/.Filmliste.0.18.1.sqlite').
DEBUG Opening database from 'https://liste.mediathekview.de/Filmliste-akt.xz'.
Downloading database ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% 0:00:00
2022-02-18T18:23:29 DEBUG Loading database items.
Reading database items ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 64.1% 0:00:48
2022-02-18T18:25:21 CRITICAL <class 'OverflowError'>: timestamp out of range for platform time_t
File "/home/pi/.local/bin/mtv_dl", line 10, in <module>
sys.exit(main())
File "/home/pi/.local/lib/python3.7/site-packages/mtv_dl.py", line 1239, in main
history=cw_dir / HISTORY_DATABASE_FILE)
File "/home/pi/.local/lib/python3.7/site-packages/mtv_dl.py", line 434, in __init__
self.initialize_filmliste()
File "/home/pi/.local/lib/python3.7/site-packages/mtv_dl.py", line 387, in
initialize_filmliste
""", self._get_shows())
File "/home/pi/.local/lib/python3.7/site-packages/mtv_dl.py", line 540, in _get_shows
start = datetime.fromtimestamp(int(show['start']), tz=utc_zone).replace(tzinfo=None)
The code is running on Raspian Linux 10 32bit, mtv_dl installed via pip3.
Could you please have a look?
Thx
Maik
First off: An amazing tool you have build there, @fnep ! I'm using it daily (well, my cron scheduler is 😉).
I've noticed that some TV series e.g. from ZDF are now using (S01E01)
like episode markers in the title
field. So we now have some data that we could use to guess TV series and slightly change the behavior of mtv_dl
. It would be awesome if mtv_dl
would use that information so that the Series appears nicely in the media centers (Emby, Kodi, Jellyfin, ...).
What I had in mind was:
--series
for the download
command to enable special treatment of TV series.
title
, using a regex \(S(\d+)/E(\d+)\)
for the first case:
mtv_dl -r23 dump topic="Nachtschicht" title=S01
mtv_dl -r23 dump topic="Leopard, Seebär"
, here always assume season=1
<episodedetails>
instead of <movie>
<season>XXX</season><episode>XXX</episode>
<topic>
<topic> - SxxExx - <title>.<extension>
NFO
file containing some general series information (not sure where to get the details from)I didn't check if that series metadata would also be available somewhere in the database so that we don't need to guess it from the title...
If you're interested in such a feature, I could take a look in a couple of weeks and propose a pull request...
Near line 1164:
logger.info('Marked %s from %s as downloaded.', downloader.label)
-->
logger.info('Marked %s as downloaded.', downloader.label)
Observation:
If the download failes due to not writable target, the movie is marked as downloaded in the history.
Expectation:
If the anything of the download process failes, do not mark as downloaded in the history.
Hi,
i had not run the tool for a while, and now fetched the ˋlatestˋ container image.
After start the program crashes with this error:
2023-06-10T12:32:29+0000 CRITICAL <class 'sqlite3.OperationalError'>: table main.show has 16 columns but 18 values were supplied
File "<string>", line 1, in <module>
File "/mtv_dl.py", line 1462, in main
showlist.initialize_if_old(refresh_after=int(arguments["--refresh-after"]))
File "/mtv_dl.py", line 618, in initialize_if_old
self.initialize_filmliste()
File "/mtv_dl.py", line 380, in initialize_filmliste
cursor.executemany(
I also tried the ˋmasterˋ image version. Same error was reported.
It looks like the new columns are missing:
# sqlite3 .Filmliste.0.0.0.sqlite
SQLite version 3.34.1 2021-01-20 14:10:07
Enter ".help" for usage hints.
sqlite> .schema main.show
CREATE TABLE show (
hash TEXT,
channel TEXT,
description TEXT,
region TEXT,
size INTEGER,
title TEXT,
topic TEXT,
website TEXT,
new BOOLEAN,
url_http TEXT,
url_http_hd TEXT,
url_http_small TEXT,
url_subtitles TEXT,
start TIMESTAMP,
duration TIMEDELTA,
age TIMEDELTA
);
tl;dr: my problem was that I had 2 .History files at that time, one in cwd
and one in {dir}
and that mtv_dl deleted the files from the database in cwd
and not in {dir}
, because I used mtv_dl history --remove
and not mtv_dl history --remove --config=./.mtv_dl.yml
.
I have just stumbled upon this wonderful piece of software and I am testing it right now.
While playing with settings and filters, I had do re-download a tv show, which wasn't possible.
[jan@two-arch bin]$ ./mtv_dl download --config=./.mtv_dl.yml --sets=./mtv_tatort.txt
2021-10-21T09:07:46 DEBUG Opening Filmliste database PosixPath('/home/jan/Videos/Tatort/.Filmliste.0.18.1.sqlite').
DEBUG Opening History database PosixPath('/home/jan/Videos/Tatort/.History.sqlite').
DEBUG Database age is 0:17:00.
DEBUG Applying filter: channel=ard, topic=tatort, title!=audiodeskription, age-1d, duration+80m (limit: None)
2021-10-21T09:07:47 DEBUG Skipping 'Tatort: Altes Eisen' (ARD, 'Tatort: Altes Eisen', 2021-10-20 20:00:00, c80523597cd) (already loaded on 2021-10-21 06:52:14)
DEBUG Skipping 'Absturz' (ARD, 'Tatort', 2021-10-20 20:10:00, 642cb991142) (already loaded on 2021-10-21 06:53:56)
I then removed both shows from the history:
[jan@two-arch bin]$ ./mtv_dl history --remove=642cb991142
2021-10-21T09:08:58 INFO Removed 642cb991142 from history.
[jan@two-arch bin]$ ./mtv_dl history --remove=c80523597cd
2021-10-21T09:09:09 INFO Removed c80523597cd from history.
but the result was the same:
[jan@two-arch bin]$ ./mtv_dl download --config=./.mtv_dl.yml --sets=./mtv_tatort.txt
2021-10-21T09:09:14 DEBUG Opening Filmliste database PosixPath('/home/jan/Videos/Tatort/.Filmliste.0.18.1.sqlite').
DEBUG Opening History database PosixPath('/home/jan/Videos/Tatort/.History.sqlite').
DEBUG Database age is 0:19:00.
DEBUG Applying filter: channel=ard, topic=tatort, title!=audiodeskription, age-1d, duration+80m (limit: None)
2021-10-21T09:09:15 DEBUG Skipping 'Tatort: Altes Eisen' (ARD, 'Tatort: Altes Eisen', 2021-10-20 20:00:00, c80523597cd) (already loaded on 2021-10-21 06:52:14)
DEBUG Skipping 'Absturz' (ARD, 'Tatort', 2021-10-20 20:10:00, 642cb991142) (already loaded on 2021-10-21 06:53:56)
Then I tried to reset the history:
[jan@two-arch bin]$ ./mtv_dl history --reset
but still the same:
[jan@two-arch bin]$ ./mtv_dl download --config=./.mtv_dl.yml --sets=./mtv_tatort.txt
2021-10-21T09:09:52 DEBUG Opening Filmliste database PosixPath('/home/jan/Videos/Tatort/.Filmliste.0.18.1.sqlite').
DEBUG Opening History database PosixPath('/home/jan/Videos/Tatort/.History.sqlite').
DEBUG Database age is 0:19:00.
DEBUG Applying filter: channel=ard, topic=tatort, title!=audiodeskription, age-1d, duration+80m (limit: None)
DEBUG Skipping 'Tatort: Altes Eisen' (ARD, 'Tatort: Altes Eisen', 2021-10-20 20:00:00, c80523597cd) (already loaded on 2021-10-21 06:52:14)
DEBUG Skipping 'Absturz' (ARD, 'Tatort', 2021-10-20 20:10:00, 642cb991142) (already loaded on 2021-10-21 06:53:56)
The database itself when used manually is working as expected:
[jan@two-arch bin]$ sqlite3 .History.sqlite
SQLite version 3.36.0 2021-06-18 18:36:39
...
sqlite> .headers on
sqlite> select * from downloaded;
hash|channel|description|region|size|title|topic|website|start|duration|downloaded
c80523597cd8b5e45644c85e64f43f99aecd8e74|ARD|Bei ihren Ermittlungen ...|DE|1135|Tatort: Altes Eisen|Tatort: Altes Eisen|https://www...337|2021-10-21 07:17:28
642cb99114281235c64bf72ef3105097f37d825a|ARD|Eigentlich ...|DE|1262|Absturz|Tatort|https://www...MGY|2021-10-20 20:10:00|5312|2021-10-21 07:19:10
sqlite> select hash from downloaded where hash like 'c80523597cd%';
hash
c80523597cd8b5e45644c85e64f43f99aecd8e74
sqlite> delete from downloaded where hash='c80523597cd8b5e45644c85e64f43f99aecd8e74';
sqlite> select * from downloaded;
hash|channel|description|region|size|title|topic|website|start|duration|downloaded
642cb99114281235c64bf72ef3105097f37d825a|ARD|Eigentlich ...|DE|1262|Absturz|Tatort|https://www...MGY|2021-10-20 20:10:00|5312|2021-10-21 07:19:10
mtv_dl.py download hash=9bafd1df3bb
The movie has a invalid subtitle url "/static/avportal/untertitel_mediathek_preview/24429544.xml"
This causes the script to abort with a critical error.
Unfortunately this prevents downloading of additional movies of the same query.
Proposal: Change the logic, that in case of a failure of download, the other movies in the same query are still downloaded.
The script does not respect the option "logfile" in a configfile.
Assumed cause: the logging_handler is defined, before the configfile is read.
# ISO8601 logger
if arguments['--logfile']:
logging_handler = logging.FileHandler(Path(arguments['--logfile']).expanduser())
logging_handler.setFormatter(logging.Formatter("%(asctime)s %(levelname)-8s %(message)s",
"%Y-%m-%dT%H:%M:%S%z"))
else:
logging_handler = RichHandler(console=console)
logging_handler.setFormatter(logging.Formatter(datefmt="%Y-%m-%dT%H:%M:%S%z "))
logging_handler._log_render.show_path = False
logger.addHandler(logging_handler)
sys.excepthook = lambda _c, _e, _t: logger.critical('%s: %s\n%s', _c, _e, ''.join(traceback.format_tb(_t)))
# config handling
arguments = load_config(arguments)
line 41:
-o, --oblivious Download even if the show alredy is marked as downloaded.
-->
-o, --oblivious Download even if the show already is marked as downloaded.
line 87:
age-1mm (age is older then 1 month)
-->
age-1mm (age is younger then 1 month)
Hi there,
i just wanted to know if it would be possible to add an option or if it would be generally a good idea to sanitize the filesnames generated by mtv_dl.
For example when i download the current Mord auf Shetland Episodes there is a | added to the filename which makes problems when you try to transfer the file via scp or SMB.
Same problem when there is a : in the filename etc.
Having a typo in the filter parametername (e.g. uppercase instead of lower case)
python3 mtv_dl.py list Title='extra 3'
leads to a confusing error message:
ERROR Invalid operator '=' for 'Title'.
But not the operator = is invalid, the whole parametername is invalid.
Hi !
thank you very much for this great peace of software. Did you ever think about to make a docker container for this ?!
There does not seem to be an option to download subtitles, although doing a JSON dump of a certain program hash does reveal a working URL for the subtitles.
Downloading the same movie twice with option '--oblivious" into two different locations result in sql-unique-error:
CRITICAL <class 'sqlite3.IntegrityError'>: UNIQUE constraint failed: downloaded.hash
File "mtv_dl.py", line 1175, in <module>
main()
File "mtv_dl.py", line 1161, in main
showlist.add_to_downloaded(item)
File "mtv_dl.py", line 575, in add_to_downloaded
""", show)
Having a opening bracket in the title search results in critical error.
$mtv_dl list title="("
CRITICAL <class 'sqlite3.OperationalError'>: user-defined function raised exception
File "/home/blomi/bin/mtv_dl", line 8, in
sys.exit(main())
^^^^^^
(And thanks for this great tool!)
tqdm
automatically adjusts the size of the progressbar to match the terminal width. However if the description desc
is too long (which it commonly is), it will overflow the line and cause the terminal to scroll.
Downloading 'NEO MAGAZIN ROYALE - History I - DAS steckt wirklich dahinter!' [ZDF, 'NEO MAGAZIN ROYALE', 2020-02-28 05:1
Downloading 'NEO MAGAZIN ROYALE - History I - DAS steckt wirklich dahinter!' [ZDF, 'NEO MAGAZIN ROYALE', 2020-02-28 05:1
Downloading 'NEO MAGAZIN ROYALE - History I - DAS steckt wirklich dahinter!' [ZDF, 'NEO MAGAZIN ROYALE', 2020-02-28 05:1
Downloading 'NEO MAGAZIN ROYALE - History I - DAS steckt wirklich dahinter!' [ZDF, 'NEO MAGAZIN ROYALE', 2020-02-28 05:1
Downloading 'NEO MAGAZIN ROYALE - History I - DAS steckt wirklich dahinter!' [ZDF, 'NEO MAGAZIN ROYALE', 2020-02-28 05:1
Downloading 'NEO MAGAZIN ROYALE - History I - DAS steckt wirklich dahinter!' [ZDF, 'NEO MAGAZIN ROYALE', 2020-02-28 05:1
Downloading 'NEO MAGAZIN ROYALE - History I - DAS steckt wirklich dahinter!' [ZDF, 'NEO MAGAZIN ROYALE', 2020-02-28 05:1
Downloading 'NEO MAGAZIN ROYALE - History I - DAS steckt wirklich dahinter!' [ZDF, 'NEO MAGAZIN ROYALE', 2020-02-28 05:1
Downloading 'NEO MAGAZIN ROYALE - History I - DAS steckt wirklich dahinter!' [ZDF, 'NEO MAGAZIN ROYALE', 2020-02-28 05:1
5:00+08:00, 8d45aebd]: 4%| | 32.2M/854M [02:10<53:39, 255kB/s]
It would be great if the description could be limited automatically, so that the progress output matches the terminal width.
You already added the possibility to run mtv_dl inside a Docker container. What is missing in my opinion is the published image on Docker Hub for even easier use. Right now, you have to clone the repository in order to build the image. I would prefer to simply pull the image off the Docker Hub repository and begin to use it.
I would suggest using an automatic build pipeline which can be connected to this Git repository. You can define that the "latest" image and an image containing the version tags is built whenever a new release is published. Additionally I would suggest building a "dev" image on each commit to the main-branch. This way newest features are available asap.
Problem: How to capture each new "Tatort" aired on Sunday 20:15, but not all the repetitions during the week?
Solution: Have additional command line filter parameters for the start field, like weekday, hour, minute.
My current workaround would be to define age-1h and just call the script once on Sunday at 20:30. (But this does not play well with one filter set file).
From the command line, no problem:
/usr/local/bin/mtv_dl -v download topic='terra' channel='zdf' age-30d duration+30m --high --dir=/mnt/NAS/dvds/recordings
2019-10-15T21:04:47+0200 DEBUG Using cache from PosixPath('/mnt/NAS/dvds/recordings/.Filmliste-akt.xz.cache').
2019-10-15T21:04:48+0200 DEBUG Database age is 0:41:00.
2019-10-15T21:04:48+0200 DEBUG Applying filter: topic=terra, channel=zdf, age-30d, duration+30m
2019-10-15T21:04:49+0200 DEBUG Skipping 'Im Bann der Astrologie - Faszination Universum mit Harald Lesch' [ZDF, 'Terra X', 2019-09-29 19:30:00+02:00, e7da8281] (already loaded on 2019-10-01 09:11:00)
2019-10-15T21:04:49+0200 DEBUG Downloading 'Unser kosmisches Schicksal - Faszination Universum mit Harald Lesch' [ZDF, 'Terra X', 2019-10-13 19:30:00+02:00, d944a6b8] from 'https://rodlzdf-a.akamaihd.net/none/zdf/19/10/191013_fu_schicksal_tex/3/191013_fu_schicksal_tex_3328k_p36v14.mp4'.
However, if I run the same command from within a sudo Python3.7 script using subprocess, I get this:
sudo python3.7 /mnt/NAS/dvds/recordings/pvr.py
2019-10-15 21:03:58.100566
running: /usr/local/bin/mtv_dl -v download topic='terra' channel='zdf' age-30d duration+30m --high --dir=/mnt/NAS/dvds/recordings
out: ''
err: '2019-10-15T21:03:58+0200 DEBUG Using cache from PosixPath('/mnt/NAS/dvds/recordings/.Filmliste-akt.xz.cache').
2019-10-15T21:03:59+0200 DEBUG Database age is 0:40:00.
2019-10-15T21:03:59+0200 DEBUG Applying filter: topic='terra', channel='zdf', age-30d, duration+30m
'
exit: 0
2019-10-15 21:04:00.606802
This is the Python code I use to run an external command
def run(cmd):
print(datetime.datetime.now())
print("running:", " ".join(cmd))
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdout, stderr = proc.communicate()
proc.stdout.close()
proc.stderr.close()
print("out: '{}'".format(stdout.decode("utf8")))
print("err: '{}'".format(stderr.decode("utf8")))
print("exit: {}".format(proc.returncode))
print(datetime.datetime.now())
print("-------------------")
run(["/usr/local/bin/mtv_dl", "-v", "download", "topic='terra'", "channel='zdf'",
"age-30d", "duration+30m", "--high", "--dir=/mnt/NAS/dvds/recordings"])
Do you have any idea what could cause this?
Hallo fnep,
vielen Dank für dieses tolle Paket. Sicherlich ist das kein Issue, sondern eher ein Bedienfehler.
Ich habe meine Probleme etwas zu downloaden.
Dazu nutze ich die folgende Zeile mit dem Ziel die Videos in den aktuellen Pfad zu kopieren. Ich hatte auch schon eine absolute Pfadangabe nach dem Muster 'E:\Videos' probiert. Aber mit derselben Fehlermeldung.
mtv_dl.exe download topic='Elefant, Tiger' channel='MDR' -h --target='./{title}'
Daraufhin bekomme ich dann die folgenden Fehlermeldung
Ich würde mich freuen, wenn du mir auf die Sprünge helfen kannst.
Vielen Dank und Grüße,
Marcus
The file extension {ext} should automatically be added to the --target.
If the user specifies "--target=test.mp4" the movie will be downloaded and then overwritten with the subtitles and then overwritten with the nfo-file, because all three files write to test.mp4 (see log below).
So the user has to specify {ext} in the target, otherwise the output will be useless. But if the user has to specify it, why not add it automatically?
Just adding {ext} to each --target will be not backwards compatible to older configurations.
Proposal 1: Automatically add {ext} to --target in the script, if {ext} is not existing in --target.
Proposal 2: Give a warning if {ext} is not in --target and subtitles or nfo are generated.
python3 mtv_dl.py download -o --target=test.mp4 hash=2f2bad3ed7c
2020-07-16T14:52:08 INFO Saved show 'Der erste April (26)' (ARD, 'Meister Eder und sein Pumuckl', 2020-07-05 04:40:00, 2f2bad3ed7c) to PosixPath('test.mp4').
2020-07-16T14:52:09 INFO Saved subtitles 'Der erste April (26)' (ARD, 'Meister Eder und sein Pumuckl', 2020-07-05 04:40:00, 2f2bad3ed7c) to PosixPath('test.mp4').
INFO Saved nfo 'Der erste April (26)' (ARD, 'Meister Eder und sein Pumuckl', 2020-07-05 04:40:00, 2f2bad3ed7c) to PosixPath('test.mp4').
Is the list shown with command "list" anyhow sorted?
Proposal 1 (sufficient for me): Have a default sorting by age.
Proposal 2 (more general): Add an additional command line parameter to define the column to sort by.
And by the way: Many thanks for this cool script and even more for the extremely fast reactions.
The subtitles for some videos prevents VLC and Kodi from playing the file.
If the the subtitle file .srt is deleted, the file plays fine.
Example: Hash 58887acac00, resolution --high
Output of VLC with subtitle:
[00007f16e8c9fe20] main demux error: option sub-original-fps does not exist
[00007f16e8001170] ts demux error: libdvbpsi error (PSI decoder): TS discontinuity (received 2, expected 11) for PID 0
[00007f16e8001170] ts demux error: libdvbpsi error (PSI decoder): TS discontinuity (received 0, expected 10) for PID 4096
[00007f16e8001170] ts demux error: libdvbpsi error (PSI decoder): TS discontinuity (received 3, expected 14) for PID 0
[00007f16e8001170] ts demux error: libdvbpsi error (PSI decoder): TS discontinuity (received 0, expected 12) for PID 4096
[00007f16e8001170] ts demux error: libdvbpsi error (PSI decoder): TS duplicate (received 0, expected 1) for PID 17
[00007f16e8001170] ts demux error: libdvbpsi error (PSI decoder): TS discontinuity (received 4, expected 15) for PID 0
[00007f16e8001170] ts demux error: libdvbpsi error (PSI decoder): TS discontinuity (received 0, expected 12) for PID 4096
[00007f16e8001170] ts demux error: libdvbpsi error (PSI decoder): TS duplicate (received 0, expected 1) for PID 17
[...]
Hi,
please add two naming options for downloaded files:
--- mtv_dl.py 2022-09-20 18:18:03.000000000 +0200
+++ mtv_dl.py 2022-09-20 18:39:00.826512389 +0200
@@ -900,6 +900,10 @@
ext=file_extension,
date=self.show['start'].date().isoformat(),
time=self.show['start'].strftime('%H-%M'),
+ year=self.show['start'].date().year,
+ monthwozero=self.show['start'].date().month,
+ month=self.show['start'].strftime('%m'),
+ day=self.show['start'].strftime('%d'),
**escaped_show_details))
destination_file_path.parent.mkdir(parents=True, exist_ok=True)
Background: For Jellyfin/Emby I need to have a special Naming of the Files with Season and Episode to get recognized correctly. Currently I am doing this with some manual effort and was not able to adapt the python code in a way that it is working.
E.g. the output file shall look like: Die_Sendung_mit_der_Maus_S2022-E9-25_Die_Sendung_vom_25.09.2022.mp4
Thanks and best regards
Ben
Hi there,
ive just seen that some months ago you switch to sqlite for the history db.
Before i update (i think i use the version from end of last year) i would like to know if my old history will be migrated?
Else i would have to edit my shows.txt to only download new shows after the current date.
Hello again,
I have a weird issue with two separate computers:
When I start a command like
mtv_dl list topic="Lenas Ranch"
nothing happens until i hit ctrl+C and then it starts executing.
see here pasted from my terminal:
mtv_dl download -v topic="Lenas Ranch"
^C2019-01-31T22:05:18+0100 DEBUG Opening database from 'http://verteiler2.mediathekview.de/Filmliste-akt.xz'.
Now I assume I am missing something very obvious, but I don't know what it is.
So I just started moving a few services from different raspberryPis onto a single 10W TDP NUC, to have lower power usage, better peak performance, and a single point of failure. Yeah me.
I plugged the external drive with the library and history file etc. into the new system, ran the usual download script and mtv_dl
wanted to redownload everything. The hashes differed everywhere, which shouldn't be.
cd6670d4e3c │ ARD │ Extra 3 vom 25.08.2022 im Ersten │ extra 3 │ 429 │ 2022-08-25T22:50:00+02:00 │ 44m 3s │ None │ DE │ 2022-08-26T01:37:44+02:00
90b260f6adc │ ARD │ Extra 3 vom 25.08.2022 im Ersten │ extra 3 │ 429 │ 2022-08-25T22:50:00+02:00 │ 44m 3s │ None │ DE │ 2022-09-12T16:58:53+02:00
I apparently had different locale and TZ settings. After setting the locale from C to German UTF-8 and the TZ from London to Berlin, and rebooting, (and probably redownloading the mediathek database file) mtv_dl
had the correct hashes and picked up the history.
I don't know which part (I feel timezone, even though at first glance the code looks agnostic to it and the formatted start times are the same as well), but somehow the hash value is not completely agnostic of some environment settings.
I guess to replicate:
/usr/local/bin/mtv_dl dump title="FIFA und die WM 2022 in Katar"
List count parameter -c ambiguously
python3 mtv_dl.py list -c 5
-c is specified ambiguously 2 times
Hi there,
i have just stumbled upon the problem that my mtv_dl install stopped downloading since the 11. June.
After digging a bit around i found auf that several of the URL in FILMLISTE_URLS seem to be down (verteiler2 to verteiler6). I tried it with my homeconnection and a VPS on both i get an unknown host message when i try to ping them.
Looks like the TV stations removed the series/episode information from the title
fields and now store it in some other fields (example https://www.zdf.de/serien/solo-fuer-weiss) . However I didn't see those other fields in the data provided by the MediathekView bundle.
So I guess we need to wait until MediathekView has that sorted out in their export...
Many users may run mtv_dl
on a host with limited resources, such as typical home-server, NAS or microcomputer platforms. On these, free memory is even today often scarce. mtv_dl
, when parsing the filmlist-JSON
, may use up to ~2.5gb of memory with the current size of the filmlist. This frequently leads to MemoryError
s when trying to download shows.
As the items from the json
are iteratively inserted into the SQLite
database, it is easy to use json streaming instead of loading the whole file into memory.
I wrote a small proof of concept using ijson
, which works, albeit at the expense of not having a progress bar anymore. Please see the attached memory profiler runs. As soon as i have wrestled with poetry to get it to add the dependency I’ll open a pull-request … we can discuss what to do with the progressbar there.
The download runs smooth and the downloads are shown with pretty title and plot in Kodi with the .nfo file! Very cool!
Unfortunately the skins in Kodi don't use the parameter in the .nfo file but instead the file modification time to show and sort the files by date.
Therefore following improvement: Have an additional parameter --set-file-mod-time. If this parameter is set, after downloading the file modification time is set to the aired time.
See PR #32 below.
The script should not start downloading of the database when the command is history.
How to exactly specify the parameters in the option --target on the commandline?
All variants I tried still resulted in using the placeholder-name but not replaced with the content.
python3 mtv_dl.py download --oblivious --target={{dir}}/{{channel}}/{{topic}}/{{start}}_{{title}}{{ext}} hash=2d5b18d8d7c
Downloading 'Wählen Sie Ihren Lieblingsfilm!' (3Sat, '3satZuschauerpreis', 2019-11-23 13:00:00, 2d5b18d8d7c) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% 0:00:00
2020-07-14T10:53:18 INFO Saved show 'Wählen Sie Ihren Lieblingsfilm!' (3Sat, '3satZuschauerpreis', 2019-11-23 13:00:00, 2d5b18d8d7c) to PosixPath('{dir}/{channel}/{topic}/{start}_{title}{ext}').
From the manual:
-t, --target=<path> Directory to put the downloaded files in. May contain
the parameters {{dir}} (from the option --dir),
{{filename}} (from server filename) and {{ext}} (file
name extension including the dot), and all fields from
the listing plus {{date}} and {{time}} (the single parts
of {{start}}).
[default: {{dir}}/{{channel}}/{{topic}}/{{start}} {{title}}{{ext}}]
start = datetime.fromtimestamp(int(show['start']), tz=utc_zone).replace(tzinfo=None)
fails with an OSError: [Errno 22] Invalid argument
for negative start dates.
I added some logging
try:
start = datetime.fromtimestamp(int(show['start']), tz=utc_zone).replace(tzinfo=None)
except OSError:
logger.error("Error parsing show %s: start=%s", title, show['start'])
continue
which showed lots of shows with negative dates
ERROR Error parsing show Europlast: start=-127635900
ERROR Error parsing show Bundespostminister Stücklen eröffnet automatische Vermittlungsstelle für Auslands-Ferngespräche: start=-127473420
ERROR Error parsing show Nerz und Brillanten unterm Weihnachtsbaum: start=-127549260
ERROR Error parsing show Museen ohne Mittel? Von 35 Millionen Lottogewinn gingen 5 Millionen an staatliche Museen.: start=-127645680
The question probably is: Why are there negative values in the first place?
EDIT: Because the Filmliste is originally created by Java, it probably means
A negative number indicates the number of milliseconds before January 1, 1970, 00:00:00 GMT.
https://docs.oracle.com/javase/7/docs/api/java/sql/Date.html
Hello once again,
I now have the problem that the file is not moved to the new location on the mounted NFS drive with
[Errno 18] Invalid cross-device link
This seems to be due to the usage of Path.rename in _move_to_user_target (line 719) which apparently does not work across different filesystems.
Is it possible to use some other method for moving the file (like shutil.move) in case Path.rename fails?
Being a non-software developer, I can only guess what needs to be done and my solution would not be as robust as anything you could come up with.
Thanks for making this great tool, I really hope I can use it for my use cases :)
Being honest i just wanted to say Thanks! because your tool saved me a lot of time downloading shows for my Kids.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.