AniList to TvDB or TheXem DB support about jellyfin-ani-sync

[vosmiic]

I'll add this work to the main project and get to work on it when I can, which should be in a couple of weeks (I am currently away from my dev PC).
I'll keep you updated on the progress.

[Andy2244]

Ok did hop into the #xem channel in the sonarr discord and yes it seems there is no way via xem-api to directly get the anidb-id's out of the api mappings. They did recommend using anime-list xml files for that.

So here is the pipeline:

• get series main tvdb-id, get season/episode numbers
• lookup mapping via xem /all? or parse anime-list xml directly (NOTE: seems xem could be skipped, the offsets/mappings are also in anime-list)
  example:

<anime anidbid="16177" tvdbid="267440" defaulttvdbseason="4" episodeoffset="16" tmdbid="" imdbid="">
    <name>Shingeki no Kyojin: The Final Season (2022)</name>
</anime>

• use anime-list to search for tvdb-id, match season to defaulttvdbseason and get the anidb-id
• use anime-offline-db to directly map anidb-id to anilist-id, myanime, kitsu....
• directly scrobble using the id's + corrected/offset ep numbers

[Andy2244]

Try installing the anilist/anidb plugins, just to confirm this, since it would save unnecessary user setting options. If we get a Anilist id, we can also directly use this id as is and mapping from anidb-id to anilist-id/myanime/kitsu is also a shorter path.

All three options/paths should ideally work.
It also seems we can avoid using any name/alias/synonym lookups, since all should work with just the metadata id's, maybe if some part fails there is a need for name lookup, yet at this point i would rather ask the question why did the lookup per id fail?

Just checked anime-offline-db json files, you need to extract the id's manually, what you get is this:

      "sources": [
        "https://anidb.net/anime/1835",
        "https://anilist.co/anime/989",
        "https://anime-planet.com/anime/shinshaku-sengoku-eiyuu-densetsu-sanada-juu-yuushi-the-animation",
        "https://anisearch.com/anime/1519",
        "https://kitsu.io/anime/879",
        "https://livechart.me/anime/7955",
        "https://myanimelist.net/anime/989",
        "https://notify.moe/anime/cX5JcKmiR"
      ],

At least all use the "/anime/id" formatting, so extracting and matching the id's should be trivial.

[vosmiic]

I have changed the update schedule from every day at midnight to every 24 hours, hopefully this means that the XML file will update if it detects your clock has gone past 24 hours.

I have tested the new functionality to some extent, I think I am going to now merge to master.
I will further test the changes just to make sure that it is 100% working, but I doubt any more changes involving this will be needed.
I'll let you know when the changes will be released (probably either tomorrow or over the weekend). I'll also close this issue. If you find any issues with the changes after they have been released, please do let me know.
Just wanted to say thank you for the help with this, it has been the largest change in how we detect anime since the plugin's release and it would not have been possible without your massive help.

[vosmiic]

Just letting you know that the update that implements these new features has now been released: https://github.com/vosmiic/jellyfin-ani-sync/releases/tag/v2

If you have installed the plugin via the manifest link, you can simply run the "Update Plugins" task in the "Scheduled Tasks" menu.

[vosmiic]

We use a different method to identify anime in this plugin, where we query the provider database using the name in Jellyfin (which is set by whichever metadata provider you use).
While this leads to the anime detection being slightly unreliable, it requires no manual work from the user or other parties.
The advantage to using a dedicated list is obviously the detection being basically 100% accurate but requires manual entry by users.
That being said I could quite easily integrate a list of anime mappings to AniList (like the ones linked) into the plugin, and fall back to querying the API database if the show is not present in the list.
I will ask if the owner of the repo is willing to share the list (I don't feel right just using it without permission), but I entirely understand if they are not willing to give us access.
I don't think that maintaining our own list of mappings to the separate providers (AniList, MAL and Kitsu) will be very achievable at the moment considering how many users we have. Of course that may change later on.
I will keep this open and let you know of the progress.

[Andy2244]

The main issue is that the name of TvDB is not always the same as Anilist, which uses AniDB + synonyms. Also AniDB/Anilist is not season aware. That's where the list-mappings comes in and can map different TVDB seasons to the separate Anilist entries.

Example:

  - title: "Attack on Titan"
    synonyms:
      - "Shingeki no Kyojin"
    seasons:
      - season: 1
        anilist-id: 16498
      - season: 2
        anilist-id: 20958
      - season: 3
        anilist-id: 99147
      - season: 3
        anilist-id: 104578
        start: 13
      - season: 4
        anilist-id: 110277
      - season: 4
        anilist-id: 131681
        start: 17

In this example we also have "sub" seasons, that start at different absolute numbering in TVDB.

[vosmiic]

The main issue is that the name of TvDB is not always the same as Anilist, which uses AniDB + synonyms

Yeah the plugin currently queries the API db with the title stored in Jellyfin, and then checks all of the returned anime's titles and synonyms. This is basically relying on the API db containing the title somewhere. While it usually does find the right series, it would be good to not be dependent on the API db data.

Also AniDB/Anilist is not season aware

We had quite a few challenges with this. As you probably already know Jellyfin organizes shows into one title with multiple seasons, while most API's have separate items for each season. We attempt to combat this problem by querying the first season of the show, and then checking to see if a related "sequel" title exists. Jellyfin returns which season you are watching, so we can count through each related "sequel" to find the season.
So if you were watching season 2 of a show, the plugin would look up the first season and get the first related "sequel".
If you were watching season 3 of a show, it would query the first season, get the first related "sequel", and then query that items first related "sequel". This will continue to loop through each sequel until it finds what it believes to be the season you are watching.
While this method causes a lot of calls to be made, it seems to work relatively well (at least it did during testing).
It would be nice to get a solid way of finding different seasons though.

[Andy2244]

As you probably already know Jellyfin organizes shows into one title with multiple seasons, while most API's have separate items for each season.

I mean thats not a Jellyfin issues, it only depends on which metadata source you use for Anime. If you use the AnidB/AniList metadata plugins you get no seasons at all. Yet this also requires you to name your Anime accordingly and you cant mix it with TvDB metadata.
So if you use the default TvDB/TMDB based metadata, you will get seasons and now your names/ep numbers will be often incompatible with AniDB/AniList.

It would be nice to get a solid way of finding different seasons though.

Yeah i noticed that for "Ascendance of a Bookworm" S3 the plugin tried to find a valid entry, but gave up at season 2. I was actually surprised that some "season" logic even existed.

If you look at the "Attack on Titan" example, this "smart" way of trying to find seasons breaks down if there is a "Sub" Seasons on TvDB, like season 4 part 1 + 2.

The absolute BEST way that would work 99% of the time, is to use thexem, as example here https://thexem.info/xem/show/4469 the API docu is here. You can easily map between TvDB/AniDB and get the correct AniDB entries for each TvDB season/episode. Than you can use the exact AniDB "season" name + correct/offset episode number to query/scrobble AniList/MyAnilist.

The second best is to use user maintained mapping files, like the plex plugin does. Those only include the most complicated mapping for multi "seasons" shows or offset single seasons.

PS: It would also help to have a setting to tell the plugin what metadata source was actually used, in AniDB/AniList plugin mode it can just scrobble as is, while in TvDB mode it needs the mapping information. This should be a user setting i think.

[Andy2244]

Btw there is also the Anime offline DB, if you want to quickly convert from one Anime service to the next. You just download/update the db file, so no extra API/HTTP calls needed. The project already runs for several years and updates nearly daily.

https://github.com/manami-project/anime-offline-database

[vosmiic]

So if you use the default TvDB/TMDB based metadata, you will get seasons and now your names/ep numbers will be often incompatible with AniDB/AniList.

I see, I started development using an alpha build of 10.8 which I believe defaults to using the AniList metadata source, so during testing all seasons would be part of a single entity in Jellyfin. In hindsight I should have tested using multiple metadata sources.

PS: It would also help to have a setting to tell the plugin what metadata source was actually used, in AniDB/AniList plugin mode it can just scrobble as is, while in TvDB mode it needs the mapping information. This should be a user setting i think.

I will look into adding this, good idea.

Btw there is also the Anime offline DB

This looks quite promising, and I believe the license allows free usage.
I don't see a reason why we couldn't use a combination of these methods, so perhaps if the user is using TvDB/TMDB metadata it uses thexem first and then fallback to other detection methods, while other providers will check the database and then fallback to the current method of trawling through the API.

Just wanted to say thank you for all of the help and resources you have added to this thread, this will most likely lead to a positive overhaul of the plugin to provide much more reliable results.

[Andy2244]

alpha build of 10.8 which I believe defaults to using the AniList metadata source

yeah i also noticed that, if you have the AniDB/List metadata plugin installed it always puts those first in the "new lib" list settings... you have to manually reorder or uninstall the plugins. Many users will use tvdb/tmdb for shows/movies as default and only switch to AniDB/Anilist "if" they have "correctly" named there Anime folders... which addressing the elephant in the room will often not be the case (sonarr + scene naming = tvdb names + seasons).

As noted if you can utilize thexem api/DB to correctly map tvdb to anidb this would be perfect, its more robust than relying on the custom user yaml files.
On that note in the source it clearly states:

You do not need to run your own copy of XEM. It's a web service that is completely free to use.

For any other metadata source the anime-offline-database files can be downloaded locally and parsed, than used for searches + aliases and extra data.

Also thank you for trying to improve the scraper!

PS: On a sidenote, if you have/get the tvdb/tmdb id's/names, you are also good to go to scrobble/sync to trakt.tv see docs. Trakt has a free api endpoint and a commercial one.

[vosmiic]

Already hit into some trouble - looks like Jellyfin does not store the season tvdb ID, only the single episode ID in the database. Did a full search of the database and could not find any trace of a season ID.
Since thexem requires you to supply a valid season ID, not an episode ID, I'm not entirely sure how we could implement thexem support.

[Andy2244]

Maybe a lookup from episode ID to season ID is reasonable via TVDBid ? https://github.com/thetvdb/v4-api

[Andy2244]

ok just did a quick check, i think we have to use this query:
https://thexem.info/map/all?id=267440&origin=tvdb

This uses the base series tvdb-id and gives the full list and mapping of a series, in this example for attack on titan. You find the tvdb entry with the matching season/ep nr. and than check the corresponding anidb season/ep and use this for scrobble.

[vosmiic]

Maybe a lookup from episode ID to season ID is reasonable via TVDBid

Just had a quick search and it looks like the tvdb API comes at a cost, which I'm sure most if not all will not be ready to pay for.

i think we have to use this query

Yeah that is the query I was using when briefly testing how we would use thexem, and it worked well. However I think we are still back at not being able to retrieve the series tvdb ID.
Kind of a shame as it seems we are really close.

[Andy2244]

not being able to retrieve the series tvdb ID

Whats the issue here? Jellyfin does store the tvdb id per series, you can check and edit the metadata on a series and see whats there.

[vosmiic]

Ah okay just worked it out, right you are, it seems I was looking in the wrong area.
Will continue with this and let you know of the progress.

[Andy2244]

mhh i did some test and while i can get the mappings between tvdb<->anidb just fine via the /all? query, there seems no obvious way to get the actual anidb-id for the mapped season?
I can see those on the website entries, but the query is not returning them? Did you find a way around this?
Maybe there is a undocumented query for this? (Trying xem irc atm for help on this)

Ultimately we want to use the anidb-id/anilist-id too directly mark the season/episode, without any string/name lookup or searches.

Don't forget we have several maintained lists to help us, we can just download locally and check for updates via size changes:

1. AnidDB data dump xml list and third party web api for it: http://anisearch.outrance.pl
2. anime-list maps anidb and tvdb
3. anime-offline-db maps all anime lists to each other

PS: Did you test what happens if you select all meta providers, anilist/anidb/tvdb and point it on a random library with mixed file/folder names? Will Jellyfin try lookup all providers or stop after the first match? So does a series get only anilist-id, anidb-id, tvdb-id or a mixture of all three if matches are found? If we get only one this would directly "hint" what pipeline to use for the match, otherwise we would need to guess or need user settings per library.

[vosmiic]

Did you test what happens if you select all meta providers, anilist/anidb/tvdb and point it on a random library with mixed file/folder names?

I currently have multiple meta providers enabled (not anilist or anidb, but I can't see it being any different for them)

and only the tvdb provider is being stored, so I believe it finds the first match and sticks with it.

(NOTE: seems xem could be skipped, the offsets/mappings are also in anime-list)

I'll have a look at both and see which suits the plugin better

use anime-list to search for tvdb-id, match season to defaulttvdbseason and get the anidb-id
use anime-offline-db to directly map anidb-id to anilist-id, myanime, kitsu....
directly scrobble using the id's + corrected/offset ep numbers

Sounds like a good plan, utilizing all of the services we have at hand. I'll get onto this asap.

[vosmiic]

I can see what you mean in not needing thexem, since they have tvdbid and season/offset values we could just directly look them up.
Unfortunately just had a look at the anime-list repo and can see that no license is provided, and GitHub state that we will not be allowed to use the data held in the repo.
When you heard about the suggestion to use the xml files in the repo, did they state whether they owned the repo? I would feel much better if we had permission or if they added an open source license to the repo.

[Andy2244]

I would not worry too much, they just did not bother adding a license file in github.
Other plugins like hama for Plex are using this list for years, without any issue or special permission. The list was org. created by ScudLee for his Kodi AniDB scraper plugin, but he is retired now as Kodi dev.

So the org. list is outdated and was forked a year ago by the Anime-List repo, which itself has several forks, but only with minor updates.
So for now the "Anime-List" fork seems to-be the "good" one and is used by a modded Kodi plugin and hama plex plugin.

PS: You can ask via issue, if they can add a license or try PM ScudLee in the Kodi forum.

[vosmiic]

I have asked for permission to use the lists via an issue, will keep you updated on their response

[vosmiic]

They said they are okay with me using the list as long as it is credited, which I was planning on doing anyway to help improve the anime lists.
So far I have created a task that automatically updates the clients local anime list every day at midnight.
Now working on actually utilizing the downloaded copy to lookup the anidb entry and work with the other db to get a link.
Still arguing in my head whether I want to directly download a copy of the anime list and offline db and reference them, or just create a single SQLite file with two tables, one containing the anime list data and the other offline db data.
Makes it tidier but a little bit more work.

[Andy2244]

every day at midnight.

Don't assume that the HTPC/Jellyfin is running 24/7. I directly put my HTPC to sleep after i'm done watching stuff and also have auto sleep power settings setup.

So the update task should also run if its late, than its also a good idea todo a simple http size check before you download, so only download if the size miss-match with the local file and make sure we have gzip compression enabled.

Makes it tidier but a little bit more work.

mhh this may also add more external dependencies and complexity, i guess i would just parse the xml/json lists into a map<id, data> so i can quickly lookup via tvdbid and anidbid as key's.

My issue with a db would be that i don't really want to manage the lists data and just reparse/download on plugin load or update, which should be fast and easy.

In the end do whatever you feel comfortable with, its your baby :p

[vosmiic]

Don't assume that the HTPC/Jellyfin is running 24/7

I believe I can set it so it updates the list every day AND every time the server is started.

size check before you download

Yeah I am using the GitHub API to check the size of the XML file before downloading to decrease unnecessary traffic.

My issue with a db would be that i don't really want to manage the lists data

Yeah it requires extra work trying to parse all of the data into a db, I currently have it so it downloads the files and simply parses them. Think I'll stick with it, no need to make things more complicated if it doesn't improve anything

I have now implemented the reading of the XML file, and it can successfully retrieve the AniDb ID for a show. Now we need to lookup the tracking website links in the offline anime db to get the appropriate IDs.

[Andy2244]

every time the server is started.

Not "started", assume sleep/resume cycles, you only get a missed update aka clock jump. So it could be several days or weeks late, without a actual server restart. So either check for power events directly or do a quick check each time the plugin scrobbles something, so compare the last update date to the current date and if its bigger than a day, trigger a update prior to the scrobble.

[vosmiic]

So either check for power events directly or do a quick check each time the plugin scrobbles something, so compare the last update date to the current date and if its bigger than a day, trigger a update prior to the scrobble.

Hmm might be going beyond what we are able to do with a plugin in Jellyfin, but I'll have a look.

In other news, I have found that we do not need to download the offline anime database, as there is a publicly available API for it: https://github.com/BeeeQueue/arm-server (I think this may have been brought up earlier? I can't remember).
You just need to supply the source and ID and it returns exactly what we need, the ID of each provider.

GET https://relations.yuna.moe/api/ids?source=anidb&id=6327

{
    "anilist": 5081,
    "anidb": 6327,
    "myanimelist": 5081,
    "kitsu": 3919
}

[Andy2244]

there is a publicly available API for it

Ah did not know this existed, but i know yuna.moe desktop app.

[vosmiic]

All is going well, the plugin now by default will check the anime XML file for the show that has been watched, and then skip all of the nasty searching for the show via the providers API calls to get the appropriate ID. It now just uses the provider ID retrieved from the anime offline db to update the progress of said show.
Also if the user is using anilist as a metadata provider it even skips the XML file lookup.
Now getting movies to work with it (shouldn't be any more major work) and then some testing.

[Andy2244]

cool good job

[vosmiic]

Looks like the anime list XML file does not contain any movies, so we will only be able to process movies that already have an anidb ID retrieved from the metadata providers.
I will add all of this data to the plugin readme so users know which metadata providers are best to use (with this specific plugin).