Music scan skips some files in docker setup about raveberry

[raveberry]

I could not reproduce this right now. Note that when searching for a song it will only filter for artist and title, so the path to the file is currently ignored. If the file contains correct metadata, it should be suggested from the input field.
Could you go to djangos admin panel at /admin (login is the same as for Raveberry's admin) and confirm that the songs are not in the 'Archived songs' database table? If not, please list the folder structure that lead to this issue so I can fix it.
Also, I will probably add an additional suggestion filter for the path, so filenames lead to the correct songs as well.

[MickyGX]

I checked in django and they are not there. File structure is:

And thats the song naming structure in one of the folders:

It's not picked up any of the album's so I'm assuming it got stuck at the artist name.

[raveberry]

I just tested the scanning procedure with a two level hierarchy consisting of folders with names taken from the big list of naughty strings. All files in those folders imported correctly, so I'm pretty confident that it has nothing to do with special characters.

I noticed that Raveberry currently only handles path lengths of at most 200 characters, this should probably be raised. Reasoning was that before local file support it only had to store youtube and spotify urls, which could not get that long. Judging from your folder structure, this should not be the problem though. Especially since the longest part of the path is not visible to Raveberry when mounting the folder as a docker volume.

Another thing to consider: Raveberry does not add music files that are directly in the top level folder you specify as your library. This is where other songs are cached, and they should not be scanned. Since you have your music in subfolders, this should also not cause the problem.

Currently it works for you if you directly import the "Guns N' Roses" folder, but their songs are not added if you scan the "music" folder?

[MickyGX]

Currently it works for you if you directly import the "Guns N' Roses" folder, but their songs are not added if you scan the "music" folder?

Thats correct but obviously I only then see them songs. Not sure if it helps at all but I have 7709 tracks all in the same format as described previously so none on the top level, except for the cached ones. When I run scanner this is what I get:

Less than half the tracks are being added so like you said unlikely that all them are because of special characters. I'll have a look to see what else hasn't imported and come back to you to see if there is a pattern

[MickyGX]

Just checked first 10 artists and out of them it appears ony 2 artists were fully added, Anastacia and The Beatles. 1 artist only added 1 song that was Athlete - Wires out of 3 albums. I checked permissions between songs that were added and ones that weren't and they are the same. These are the artists checked:

Path to Athlete song:

As I'm using Spotify this isn't a big deal for me personally as I'm finding the songs in any case but let me know if you want me to try anything further

[raveberry]

After looking into the code again, I found an assertion in the end. Since 'Files Found' is different from 'Files Processed', you should see an Assertion in your log. If not, this indicates that the scanning did not properly complete for whatever reason. Rerunning the scan process might help, as it will try to add songs that were not yet added.

A reason for your 'Files Processed' being lower than 'File Added' might be that these files could not be parsed. There is a check in place to keep non-music files from being added, but maybe actual music files are excluded as well. I copied the code from Raveberry over into this standalone script, please try running it to see if files are deliberately skipped or there is another cause:

pip3 install mutagen
mv scan.txt scan.py
python3 scan.py /path/to/your/Music

[MickyGX]

scan output.txt

Thats the output I could capture from the console but it looks like its picking up the majority if not all of the mp3's.

Edit: I re-ran the scan within Raveberry and no additional mp3 files were added. It stops at 5800 files processed everytime I scan out of 12446 files found, albeit a lot are jpg's etc.

[raveberry]

It skipped some mp3 files because it could not find the mp3 header. The Guns N' Roses folder was not listed as skipped, so it should have been reached.

The interesting thing is that the script apparently processed all of your files, while scanning through the web page does not. This indicates that the background thread performing the scan got killed somehow. Unfortunately, I have no clue what could cause this.

Does it alway stop at exactly 5800 files processed?

[MickyGX]

No it varies, but always seem's to be in the 5000's. Scan done just now:

Then immediately after:

And the opriginal scan from above was 5939.

[raveberry]

Since docker currently only outputs error-level loggings by default, there is another thing you can try:
If your docker setup is running, find the name of the daphne container:
docker ps
For me, it was raveberry_daphne_1Attach to the container with an interactive shell:
docker exec -it raveberry_daphne_1 bash
Now you can inspect the info.log:
cat /opt/raveberry/logs/info.log
This will probably contain a lot of scripts not installed, which you can safely ignore (this just tells you that this is not a system install, but a docker setup).
Maybe you can find some info about the files in there, as not everything is logged to the error log.

[MickyGX]

Full copy of logs:

raveberry_logs.txt

Only thing that jumps out is:

2020-05-11 11:55:58 base.decode:114 WARNING Session data corrupted

Not sure if that is any help

[raveberry]

This just says that the server has a new secret key and your old session is invalid, so it is unrelated.
Unfortunately I don't know where else to look for this error. I will increase the internal path limit, as it caused problems during my testing, but since you have empty docker logs, your problem should be something different.

Maybe somebody else encounters this as well and can provide further information.

[MickyGX]

No problem, thanks for trying to sort and if anything else crops up that might be related I will let you know. Like I said this isn't really an issue for me personally as Spotify is working fine

[MickyGX]

Just updated to latest version and this seems to be fixed now:

All files processed and full library added. Previously skipped songs are now available 😄

[raveberry]

Interesting, because the new version did not change the scanning behavior at all. Maybe the thread did not get terminated early this time.
In any case, great that it works now.