rtissera / libchdr Goto Github PK

I can't seem to load CHD files larger than 2GiB, all I get is CHDERR_DECOMPRESSION_ERROR, I used cmake and VS2019 x64 build.
Everything compiles fine and loading smaller CHD files seems to work.

I'd like to consider using it on a python project and it'd be swell if it was on pypi already. My usecase is simple identification of redump dumps of cue/bin and gdi/bin dumps, which necessarily needs to divide the tracks streams, so a way to get the track extents and the stream without file I/O would be nice. The 'data sha1sum' is not sufficient because redump does not have it.

If this ever gets write support, particularly write of delta chds, i'd like to make that xdelta->delta chd child feature in my other RFE if it's not done by this project, because it's a major usability barrier for romhack users, especially if they have to divide the tracks every time. FSM knows MAME upstream probably never will, with the way they think romhacks are for other projects and they 'must' control all dumps themselves.

Threaded read-ahead decompression

• Threaded, so latency isn't added to the current requested read block.
• The Read-ahead thread should wait until the requested block is returned to avoid competing for IO simultaneously versus the current request.
• If a requested location already exists in the decoded read-ahead blocks, then it's returned immediately and any new delta read-ahead block are decoded to the cache.
• Read-ahead distance in KiB is configurable via API (which gets converted to N compressed blocks internally).
• If -1 is passed in for the read-ahead distance, then the read-ahead thread will keep going until it decodes the last block (only pausing to yield to current requests to avoid contention), likewise the read-ahead memory cache will expand to hold all decoded blocks.

This is the most useful compression type of chd, since it allows multiple cd games to minimize duplication.

cd1 is parent of cd2 and cd3 and cd4 in a game with 4 cds is a option and all the textures and models that are duplicated from cd1 are deduplicated without problems of the compression or the fact the files are 'contiguous' to the OS file system interfering.

or game hacks.

It would be cool if the relationship could be a tree. for instance in a game with 2 cds and hack:

cd1
|\
|  \
|    \
|      \
cd2 cd1hack
|
cd2hack

but that is currently unsupported by chdman (it would be really could if it was contributed upstream though)-

As was discussed in libretro/beetle-psx-libretro#587, support for this would be beneficial for various retroarch cores, especially for PS1. Presumably the frontend/core would handle parent locating, so that part at least shouldn't be necessary to implement in this library.

libchdr currently makes an assumption that any compressed hunk can fit within an uncompressed hunk. This assumption can be wrong however for completely valid CHDs, if they were created with low compression levels (as implicitly some internal compression header/footer could be present along with the possibly not very compressed data). It can also be trivially wrong for any maliciously created CHD, since the compressed hunk length is just something read from the hunk map.

Hi,

This is some fallout from the time64 rebuild in Debian with -Werror=implicit-function-declaration

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1069527

I can certainly fix it in some ugly way, I 'll have a look later.

I guess the same bug can be reproduced on a i386 chroot with the right compilation flags.

https://www.gnu.org/software/libc/manual/html_node/File-Positioning.html#index-ftello

https://lwn.net/Articles/938149/

Follow mamedev/mame#10233

Paths that aren't representable with the user's current codepage are unable to be accessed. There needs to be a way to open a chd that eventually calls _wfopen instead of fopen.

Possible solutions:

• New versions of chd_open and chd_read_header are added that take a FILE*, allowing users to open their own files with the appropriate call. chd_open_file is almost this for chd_open, but it's impossible to set (*chd)->owns_file from outside of the library, complicating closing of files.
• New versions of chd_open and chd_read_header are added that take a wchar_t* and call _wfopen on windows
• chd_open and chd_read_header always treat input as utf-8, and internally convert to utf-16 before calling _wfopen on Windows

cmd line used : chdman cratecd -c none -i <src.cue> -o <dest.chd>

As of current libretro libchdr, it does not support uncompressed chd files. updating it will fix this bu not sure if this is known already, but reading audio tracks off from an uncompressed CHD will cause [libretro ERROR] chd_read_sector failed lba=16410 error=14 to be displayed despite game running.

Original post:
https://github.com/libretro/beetle-saturn-libretro/issues/163#issuecomment-696682686

Would it be possible that it could be made so that a libchdr shared library could be built so it can be installed system wide? My main interest is that I would be able to reuse it in downstream projects instead of requiring libchdr to be built statically. This would greatly help in comparing upstream libchdr with downstream implementations.

Additionally it would also be nice to be able to build libchdr with system version of libraries like flac. At least with flac, a system version works fine with libchdr in RetroArch.

LRU cache

• Requests for already decompressed data can be returned immediately instead of burning another decompression round.
• Size of the LRU cache in KiB is configurable via API.

Hi there,

following earlier conversations with @alucryd , I'm fine with resyncing libchdr so that libretro-common, RetroArch and upstream libchdr are all on the same page again.

However, if we may humbly make some tiny requests, there are a couple of things still preventing us from not having to make a single edit to the sources. If these could be addressed then we could get rid of our own fork -

1. Uniquely namespace all source file names. These files get statically linked with libretro cores. There are compilers/toolchains where having two source files with the exact same name creates a problem where only one will be linked. You can see here how we basically renamed all files for libchdr and prefixed 'libchdr_' to it.

https://github.com/libretro/RetroArch/tree/master/libretro-common/formats/libchdr

2. Header includes. On our shallow fork you see that we try pulling from the system directory first -

https://github.com/libretro/RetroArch/blob/master/libretro-common/formats/libchdr/libchdr_bitstream.c#L12

For various libretro cores and for RetroArch itself, we set several include dirs for header files. Again, this is a small thing, but the way it is right now in upstream can pose problems on certain compilers/toolchains we care about.

So far, these are the only two things I can think of. They should be relatively small requests hopefully.

If any additions have been made to libchdr in any of the libretro cores that have not been upstreamed yet, hopefully @alucryd can help us out in that endeavor to make sure that all these missing parts are submitted to you as PRs so that we can finally have one central libchdr version with all the features and we no longer have to make these forks.

Thanks for reading.

People are starting to use the new CHD support in PPSSPP, and are complaining about bad performance.

It turns out that CHDs created with createcd or createraw are very slow to access with PSP-sized sector accesses.

How can I detect such CHDs and warn the user or reject them through the API? I guess chd_getheader, but what fields should I check?

Just check hunkbytes == 2048?

I'm getting a few crash reports such as:
EXCEPTION_ACCESS_VIOLATION_WRITE

Currently using 2781322

I assume these are due to corrupted CHDs but I haven't been able to reproduce the crash, even when hacking the code.
it looks like an overflow of the decoder->huffnode array.

May be add this on line 215?:

	if (repcount + curnode > decoder->numcodes)
		return HUFFERR_INVALID_DATA;

Hi,

Could you publish a release ?
I'd like to package this for Debian and strip the vendored copies from ares, mrboom and retroarch.

Having versionned releases makes it much easier.

Greetings,

I get this from the lr-picodrive buildbot run for the libretro-build-libnx-aarch64 job:

aarch64-none-elf-gcc -c -opico/cd/libchdr/deps/zstd-1.5.6/lib/decompress/huf_decompress.o pico/cd/libchdr/deps/zstd-1.5.6/lib/decompress/huf_decompress.c -Ipico/cd/libchdr/deps/lzma-24.05/include -Ipico/cd/libchdr/include  -I/builds/libretro/picodrive -I platform/libretro/libretro-common/include -I platform/libretro/libretro-common/include/compat -I platform/libretro/libretro-common/include/encodings -I platform/libretro/libretro-common/include/formats -I platform/libretro/libretro-common/include/streams -I platform/libretro/libretro-common/include/string -I platform/libretro/libretro-common/include/vfs -DREVISION=\"-dab9c6e\" -O3 -fomit-frame-pointer -ffast-math -I/opt/devkitpro/libnx/include/ -fPIE -Wl,--allow-multiple-definition -specs=/opt/devkitpro/libnx/switch.specs -D__SWITCH__ -DHAVE_LIBNX -DARM -D__aarch64__=1 -march=armv8-a -mtune=cortex-a57 -mtp=soft -ffast-math -mcpu=cortex-a57+crc+fp+simd -ffunction-sections -Ifrontend/switch -ftree-vectorize -D__LIBRETRO__ -DUSE_LIBRETRO_VFS  -O3 -DNDEBUG -I/builds/libretro/picodrive -falign-functions=2 -DREVISION=\"-dab9c6e\" -DUSE_LIBCHDR -Izlib -DEMU_F68K -D_USE_CZ80 -Ipico/cd/libchdr/deps/zstd-1.5.6/lib -Wno-unused
huf_decompress_amd64.S
pico/cd/libchdr/deps/zstd-1.5.6/lib/decompress/huf_decompress_amd64.S:12: fatal error: opening dependency file /pico/cd/libchdr/deps/zstd-1.5.6/lib/decompress/huf_decompress_amd64.d: No such file or directory
   12 | 
      | 
compilation terminated.

Other aarch64 targets seem to compile fine though.

On error, zlib_codec_init frees the first argument, but that's in various cases not something meant to be freed:

In function 'zlib_codec_init',
    inlined from 'cdfl_codec_init' at libchdr_chd.c:794:8:
libchdr_chd.c:2604:17: warning: 'free' called on pointer 'codec' with nonzero offset 184 [-Wfree-nonheap-object]
 2604 |                 free(data);
      |                 ^~~~~~~~~~
libchdr_chd.c:2604:17: warning: 'free' called on pointer 'codec' with nonzero offset 184 [-Wfree-nonheap-object]
In function 'zlib_codec_init',
    inlined from 'cdzl_codec_init' at libchdr_chd.c:689:8,
    inlined from 'cdzl_codec_init' at libchdr_chd.c:671:18:
libchdr_chd.c:2604:17: warning: 'free' called on pointer 'codec' with nonzero offset 1136 [-Wfree-nonheap-object]
 2604 |                 free(data);
      |                 ^~~~~~~~~~
In function 'zlib_codec_init',
    inlined from 'cdlz_codec_init' at libchdr_chd.c:603:8:
libchdr_chd.c:2604:17: warning: 'free' called on pointer 'codec' with nonzero offset 1176 [-Wfree-nonheap-object]
 2604 |                 free(data);
      |                 ^~~~~~~~~~
libchdr_chd.c:2604:17: warning: 'free' called on pointer 'codec' with nonzero offset 1176 [-Wfree-nonheap-object]

The macros core_fseek and core_ftell are defined in a way that libretro VFS support is broken. Here is a patch that would solve this:

index c8ec622..338a0f0 100644
--- a/include/libchdr/coretypes.h
+++ b/include/libchdr/coretypes.h
@@ -31,7 +31,11 @@ typedef int8_t INT8;
 
 #define core_file FILE
 #define core_fopen(file) fopen(file, "rb")
-#if defined(__WIN32__) || defined(_WIN32) || defined(WIN32) || defined(__WIN64__)
+
+#if defined USE_LIBRETRO_VFS
+       #define core_fseek fseek
+       #define core_ftell ftell
+#elif defined(__WIN32__) || defined(_WIN32) || defined(WIN32) || defined(__WIN64__)
        #define core_fseek _fseeki64
        #define core_ftell _ftelli64
 #elif defined(_LARGEFILE_SOURCE) && defined(_FILE_OFFSET_BITS) && _FILE_OFFSET_BITS == 64```

Hi,

I'm searching to build a (fake) minimal image that I could use in CI test for Debian.
It can't be a real game, it needs to be free data.

Greetings

The ssize_t type is undefined when using Visual Studio 2019.

This can be fixed by defining it in coretypes.h or similar as follows:

#ifdef _WIN32
#ifdef _MSC_VER
typedef size_t ssize_t;
#endif
#endif

Software: PCSX fork using libchdr: https://github.com/DrUm78/pcsx_rearmed
Hardware: FunKey S handheld device with armv7 NEON

Not sure what commit broke it but PS1 ROMs with .bin or .img format do not load anymore, the user is just sent back to the desktop. Nothing relevant in the logs I got from GMenu2X. That works with 045f2a7 but not with last 9882eea.
.PBP and .CHD still work however.
Interesting thing: last commit works well with .bin/.img on PicoDrive emulator on the same platform but breaks .cso file format too.

Steps:

1. Try to load any ROM with .cue/.bin or .cue/.img format
2. Notice that you are sent back to the desktop

AVHuff is the format used for laserdisks.. its used since v3 afaik

Using: cmake -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_BUILD_TYPE=Release -DWITH_SYSTEM_ZLIB=1 -DCMAKE_C_COMPILER=clang -D CMAKE_C_STANDARD=11 -D CMAKE_C_EXTENSIONS=0

-- The C compiler identification is Clang 16.0.2
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/clang - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Found ZLIB: /lib/libz.so (found version "1.2.13")
-- Configuring done (0.2s)
-- Generating done (0.0s)
-- Build files have been written to: /var/tmp/spkg/libchdr-master/build
[ 4%] Building C object deps/lzma-22.01/CMakeFiles/lzma.dir/src/Alloc.c.o
[ 8%] Building C object deps/lzma-22.01/CMakeFiles/lzma.dir/src/Bra86.c.o
[ 12%] Building C object deps/lzma-22.01/CMakeFiles/lzma.dir/src/BraIA64.c.o
[ 16%] Building C object deps/lzma-22.01/CMakeFiles/lzma.dir/src/CpuArch.c.o
[ 20%] Building C object deps/lzma-22.01/CMakeFiles/lzma.dir/src/Delta.c.o
[ 24%] Building C object deps/lzma-22.01/CMakeFiles/lzma.dir/src/LzFind.c.o
[ 28%] Building C object deps/lzma-22.01/CMakeFiles/lzma.dir/src/Lzma86Dec.c.o
[ 32%] Building C object deps/lzma-22.01/CMakeFiles/lzma.dir/src/LzmaDec.c.o
[ 36%] Building C object deps/lzma-22.01/CMakeFiles/lzma.dir/src/LzmaEnc.c.o
[ 40%] Building C object deps/lzma-22.01/CMakeFiles/lzma.dir/src/Sort.c.o
[ 44%] Linking C static library liblzma.a
[ 44%] Built target lzma
[ 48%] Building C object CMakeFiles/chdr-static.dir/src/libchdr_bitstream.c.o
[ 52%] Building C object CMakeFiles/chdr-static.dir/src/libchdr_cdrom.c.o
[ 56%] Building C object CMakeFiles/chdr-static.dir/src/libchdr_chd.c.o
/var/tmp/spkg/libchdr-master/src/libchdr_chd.c:814:10: warning: implicit conversion from enumeration type 'enum huffman_error' to different enumeration type 'chd_error' (aka 'enum _chd_error') [-Wenum-conversion]
return err;
~~~~~~ ^~~
/var/tmp/spkg/libchdr-master/src/libchdr_chd.c:2993:6: error: call to undeclared function 'ftello'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
p = core_stdio_ftell_impl(fp);
^
/var/tmp/spkg/libchdr-master/src/libchdr_chd.c:2987:32: note: expanded from macro 'core_stdio_ftell_impl'
#define core_stdio_ftell_impl ftello
^
/var/tmp/spkg/libchdr-master/src/libchdr_chd.c:2993:6: note: did you mean 'ftell'?
/var/tmp/spkg/libchdr-master/src/libchdr_chd.c:2987:32: note: expanded from macro 'core_stdio_ftell_impl'
#define core_stdio_ftell_impl ftello
^
/usr/include/stdio.h:718:17: note: 'ftell' declared here
extern long int ftell (FILE *__stream) __wur;
^
/var/tmp/spkg/libchdr-master/src/libchdr_chd.c:2994:2: error: call to undeclared function 'fseeko'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
core_stdio_fseek_impl(fp, 0, SEEK_END);
^
/var/tmp/spkg/libchdr-master/src/libchdr_chd.c:2986:32: note: expanded from macro 'core_stdio_fseek_impl'
#define core_stdio_fseek_impl fseeko
^
/var/tmp/spkg/libchdr-master/src/libchdr_chd.c:2994:2: note: did you mean 'fseek'?
/var/tmp/spkg/libchdr-master/src/libchdr_chd.c:2986:32: note: expanded from macro 'core_stdio_fseek_impl'
#define core_stdio_fseek_impl fseeko
^
/usr/include/stdio.h:713:12: note: 'fseek' declared here
extern int fseek (FILE __stream, long int __off, int __whence);
^
/var/tmp/spkg/libchdr-master/src/libchdr_chd.c:3031:9: error: call to undeclared function 'fseeko'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
return core_stdio_fseek_impl((FILE)file->argp, offset, whence);
^
/var/tmp/spkg/libchdr-master/src/libchdr_chd.c:2986:32: note: expanded from macro 'core_stdio_fseek_impl'
#define core_stdio_fseek_impl fseeko
^
1 warning and 3 errors generated.
make[2]: *** [CMakeFiles/chdr-static.dir/build.make:104: CMakeFiles/chdr-static.dir/src/libchdr_chd.c.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:118: CMakeFiles/chdr-static.dir/all] Error 2
make: *** [Makefile:136: all] Error 2

This is a copy of a request at the rust port of this library. I know this is 'overreach' of the purpose of the library of trying to reimplement the format, not chdman, but i fully believe this feature would make chd more popular because the vast vast vast majority of existing hacks are in xdelta, not to mention that i doubt that hacks would ever be distributed as chd child 'roms' in spite of being equivalent in not being the actual game, just for appearances sake, because .chd is a actual rom format as far as the internet is concerned. And if it's implemented downstream maybe MAME will actually get moving on this.

Xdelta (with some outliers in certain platforms of ppf) is the standard way to apply romhacks to cds currently in romhacking.net. It would simplify the life of users a lot, and in the case this library is ever used to create a command line application, if chd had the ability to take a .xdelta patch and apply it to a chd, producing not changes to the chd but a 'child' chd.

If you'd like to go beyond, ppf would be nice too. I believe both ppf and xdelta have checksum verification of the result and maybe the source too.

There is a very unfortunate binary format being used for saturn and dreamcast hacking called sega saturn patcher which is a zip file with several kind of hacks, options and transformations¹ wrapped around a executable which doesn't actually work outside of windows or in wine, which could also be a target, except i fucking hate it (see: doesn't work in linux or wine) and think it would be a mistake to support, since it does many transformations of the original dumps to adapt the source to the result (ie: 'tries not to rely on specific rips').

Anyway and since i don't mind a bit of hypocrisy in a good cause, the usecase is, obviously, not having to decompress and reformat back to cue/bin or toc/bin a file before applying a patch. I'm actually unsure if this is actually possible in all cases since afaik, xdelta patches usually apply to a single track, the binary track. Would be kind of a bummer if they couldn't apply because most patches target the redump format with separate tracks and chd outputs the format with 1 single track. Ideally the target track could be autodetected or as fallback, specified.

Regardless because of that doubt and because of the need to uncompress, you can see why this would make users life easier no? 1 command is easier to explain than 3, and it's not the average user that will feel comfortable extracting a chd, then separating a .bin into tracks, then applying the xdelta, then creating a cue file for them before creating the child chd and deleting all the intermediate files.

¹ A occasional needed conversion in romhacks source discs is turning a 'MODE 1 / 2352' binary track into a iso without checksum data 'MODE 1 / 2048' iirc. This sometimes happens in hacks for sega consoles, for 'some' reason (probably the completely terrible dump situation on those consoles and them using cdi in the older ages, so the hackers first turned the cd image to iso before starting to replace files). This is probably the main 'problem' sega saturn patcher attempts to solve - the alternative to adapt the patch to other dumps is to create the patched rom the usual way, extract the files and attempt to glue them to redump/MAME dumps manually, a complicated process that may require rebuilding the cd image in subtle ways. I wouldn't complain about a switch that did this before applying a patch.

I fixed this locally by adding set(CMAKE_POSITION_INDEPENDENT_CODE ON) to CMakeLists.txt, but I'm not familiar enough with cmake to be certain that would have no drawbacks.

[ 98%] Linking C static library libchdr-static.a
[100%] Linking C shared library libchdr-shared.so
[100%] Built target chdr-static
/usr/bin/ld: libflac-static.a(stream_decoder.c.o): relocation R_X86_64_PC32 against symbol `FLAC__STREAM_METADATA_APPLICATION_ID_LEN' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: Bad value
collect2: error: ld returned 1 exit status
CMakeFiles/chdr-shared.dir/build.make:202: recipe for target 'libchdr-shared.so' failed
make[2]: *** [libchdr-shared.so] Error 1
CMakeFiles/Makefile2:221: recipe for target 'CMakeFiles/chdr-shared.dir/all' failed
make[1]: *** [CMakeFiles/chdr-shared.dir/all] Error 2
Makefile:129: recipe for target 'all' failed
make: *** [all] Error 2

Add VS project files so a static/dynamic library can easily be created.
This would make adding this to other projects easier.

How would one go about doing it to add CHD support to them?

ZStandard Codec Support

• Likely supported via LibZStandard
• Pass-through of API settings for common ZStandard features such as: number of cores, block sizes, compression settings, and so on.
• The API will allow emulator-specific and platform-specific settings to optimize the performance of of the library.

Hello,

PR #93 introduced a compilation error:

pico/cd/libchdr/src/libchdr_flac.c:16:10: fatal error: dr_flac.h: No such file or directory
   16 | #include <dr_flac.h>
      |          ^~~~~~~~~~~
compilation terminated.
make: *** [Makefile:405: pico/cd/libchdr/src/libchdr_flac.o] Error 1

Indeed, dr_flac.h is in deps/dr_libs/ but the include in src/libchdr_flac.c is:

#include <dr_flac.h>

Of course I can fix the include path myself, but just wanted to let you know.

I just played around with Delphi and libchdr and had a problem with data alignment as chd_header was 200 with Visual Studio and 196 in Delphi.
Disabling alignment for the data structures in chd.h fixed the problem.
#pragma pack(push, 1)
...
#pragma pack(pop)