[7z files] System.IO.FileNotFoundException: Could not find file 'C:\users\steamuser\Temp\AngelLoader\FMScan\<abc>.txt' about angelloader

[FenPhoenix]

Your log said "couldn't find file", so I don't think the zero size is anything to do with that. I believe there are some places in the scanner where file size is assumed to be >0 (.mis files for example), but my strategy has been to simply let those throw and fail on purpose because we'd be on a corrupt FM anyway and there'd be no point trying to scan any further.

[FenPhoenix]

Okay, I figured out what's going on. When it iterates the archive to build the files-to-extract list, it only adds readmes that are >0 length. So 0 length readmes don't get extracted. But it still adds those readmes to the "file info" list, and when it goes through that list later to read the readmes in from disk, it doesn't check if the file it's about to read actually exists (it assumes it does). So a slight logic bug. I'm fixing it for the next version.

Note that, of course, you still won't get any useful data out of scanning these corrupt 7z files, but at least it won't fail the entire scan in future.

[FenPhoenix]

Yes, 7z files are much slower to scan due to their nature. They're solid archives (technically they can be non-solid, but in practice they're basically always solid). That means their data is one big compressed blob, so the scanner can't skip around and pluck out the files it needs, and it can't stream them into memory either (no api for it). Instead, it has to just decompress through the entire blob (which could be very large) until it's come across and extracted all the files it needs. This combined with the fact that 7z's LZMA format is much slower to decompress to begin with, and is also not able to be multithreaded during decompression either, makes zip files a far better fit for FMs IMO, in this age of dirt cheap storage. That's my opinion anyway, but if you're cool with making the speed/size tradeoff then go for it. But just know that I won't be able to make 7z files go any faster, I'm already extracting the minimum and calling directly out to the official native 7z.exe for max speed.

[FenPhoenix]

Ah, in that case it makes sense then.

[jonathanlinat]

I'm going to close this issue since it appears to stem from an incorrect conversion from the original Zip format to 7z. Some vital files have been corrupted. Please examine the file sizes, particularly the FortyEight.txt file.

Original Zip file vs Converted 7z file

[FenPhoenix]

The 7z scan first extracts the files it needs to a temp folder, and then later in the scan it reads the files back in from disk, expecting them to be there. The extract obviously must have failed, although I would expect to see a 7z error in the log if that was the case (something like "c:\archives\the_fm.7z: fm is 7z: 7z.exe path: c:\AngelLoader\7z64\7z.exe: [error info]")

I'll look over my error handling code and see if I'm missing some situation in which an error occurs but I don't set the error flag.

[FenPhoenix]

Can you send me one of your broken 7z files? I want to test 7z.exe's error output with it.

[jonathanlinat]

@FenPhoenix Here's the link for reference:

https://ufile.io/sfkg8fph

It's worth mentioning that I can access and extract the so-called "corrupted" 7z file. Yet, AngelLoader struggles to interpret specific files due to their lack of content. This is what I think.

I'm currently troubleshooting my Python conversion script to pinpoint the root of this issue. It's like it is not to be able to do the job as intended. It erases the content of specific files, or it cannot parse them correctly during the conversion process and output empty files.

[jonathanlinat]

Great, thanks Fen.

I've transitioned my Python program over to a Bash script, and it's converting correctly now. The space savings are significant. However, I've noticed that AngelLoader processes 7z files slower compared to Zip and Rar, which seems to be expected.

We can close the issue.

[jonathanlinat]

That's okay. And TY for the explanations. That makes sense.

I'm aiming to save space since I have to store these files on a server I pay for monthly. If I can save upwards of 30GB of data, it's worth it.