nickelc / hpk Goto Github PK

I'm completely new to rust language and sorry for my weak English, so here is the last output:

$ cargo build --release
   Compiling hpk v0.3.4 (/opt/git/hpk)
error: Could not compile `hpk`.

Caused by:
  process didn't exit successfully: `rustc --edition=2018 --crate-name hpk src/main.rs
  --color always --crate-type bin --emit=dep-info,link -C opt-level=3 -C lto -C
  metadata=efda9f278588f912 -C extra-filename=-efda9f278588f912 --out-dir
  /opt/git/hpk/target/release/deps -L dependency=/opt/git/hpk/target/release/deps --extern
  byteorder=/opt/git/hpk/target/release/deps/libbyteorder-8396a9ff3ced34b0.rlib --extern
  clap=/opt/git/hpk/target/release/deps/libclap-6282fbebe1e3647f.rlib --extern
  filetime=/opt/git/hpk/target/release/deps/libfiletime-0206700f28040b46.rlib --extern
  flate2=/opt/git/hpk/target/release/deps/libflate2-cc230f0d2bf00371.rlib --extern
  glob=/opt/git/hpk/target/release/deps/libglob-e086e0d80c192087.rlib --extern
  hpk=/opt/git/hpk/target/release/deps/libhpk-edf78b3620d574a2.rlib --extern
  lz4_compress=/opt/git/hpk/target/release/deps/liblz4_compress-cb84362fc1ac9e99.rlib
  --extern nom=/opt/git/hpk/target/release/deps/libnom-2fb83814f174885f.rlib --extern
  tempfile=/opt/git/hpk/target/release/deps/libtempfile-97e1e08034b9fee1.rlib --extern
  walkdir=/opt/git/hpk/target/release/deps/libwalkdir-f5079254e823656c.rlib --extern
  zstd=/opt/git/hpk/target/release/deps/libzstd-782b431503605148.rlib -L
  native=/opt/git/hpk/target/release/build/miniz-sys-71c32bff269ebcb6/out -L
  native=/opt/git/hpk/target/release/build/zstd-sys-ab6ec32720866ff2/out`
  (signal: 11, SIGSEGV: invalid memory reference)
$ 

rustc --version: 1.32.0
Linux AMD64 under Debian 7.3.0

I am trying to use hpk to mod Imperium Civitas (Glory of the Roman Empire). So far it has been easy to unpack the files, however I can't manage to repack them again and make it work.

This is how the "original" Packs/data.hpk prints

file is compressed
header:
data_offset: 0x24
fragments_residual_offset: 0x2A299
fragments_residual_count: 49
fragments_per_file: 1
fragments_filesystem_offset: 0x20374
fragments_filesystem_length: 32824
filesystem entries: 4103

and this is what I get when printing the re-packaged data.hpk

file is compressed
header:
data_offset: 0x24
fragments_residual_offset: 0x0
fragments_residual_count: 0
fragments_per_file: 1
fragments_filesystem_offset: 0x1EE164
fragments_filesystem_length: 32824
filesystem entries: 4103

Is there an option I am missing that might be useful to solve this? I have tried playing with the --chunk-size but only changed the fragments_filesystem_offset

Thanks for everything

Garbo

Hi,
Thank you for writing this, it is very handy but:
I recently used the latest version of HPK with an extraction script to decompress JA3 but it seems there are a LOT of textures missing, just in the files I was looking at there were 175 missing textures.
Some of the .mtl files had all of their textures missing, there were no issues noticed during the decompress and I have tried it twice with no luck.
I am presently attempting to list and debug my way through the files doing hand extracts to see if I can find any thing that is blindly obvious as to why its happening but thought I should let you know while Im doing that.

Steam version doesn't like fragmented files compressed with lz4 block format but has no problems with zlib.

I had to modify the cargo for zstd to 4.22+zstd.1.3.8 to get it to build.

Is there a command line sequence to allow me to inject files in a hpk package giving the path within the package?

Example Jagged Alliance 3 -

Say we have a folder C:\something\UI\Mercs unpacked from UI.hpk. I modify the files in \Mercs. I then have a couple of files, and I know their paths. Can I skip repacking the folder?

Down the line I want to create a bat to inject a selection of files into UI.hpk and distribute it with the modified files, but only the modified files. Is there a way to inject files with the .bat or do I have to run commands to unpack, overwrite and repack.

I looked through your documentation but I'm stuck.

Trying to use this tool to extract the lua files from Surviving Mars, however the output file seems to be corrupted. Attached is one of the extracted lua files.

TechTree.zip

when extracting the Surviving Mars .hpk files it appears to work correctly, but if you open any of the resulting files they aren't valid. For example if you open a .Lua file the text is unreadable. If you attempt to open a .tga it is not recognized as a valid picture file.

I downloaded this and ran it fresh today just to verify.

what can i do?
pls check my terminal output:

Compiling hpk v0.3.2 (/home/grinsel/Downloads/hpk)
error: Edition 2018 is unstable and only available for nightly builds of rustc.

error: Could not compile hpk.

To learn more, run the command again with --verbose.
grinsel@ding:~/Downloads/hpk$ cargo build --verbose

Compiling hpk v0.3.2 (/home/grinsel/Downloads/hpk)
Running rustc --edition=2018 --crate-name hpk src/hpk/mod.rs --color always --crate-type lib --emit=dep-info,link -C debuginfo=2 -C metadata=d9d205de6e274196 -C extra-filename=-d9d205de6e274196 --out-dir /home/grinsel/Downloads/hpk/target/debug/deps -C incremental=/home/grinsel/Downloads/hpk/target/debug/incremental -L dependency=/home/grinsel/Downloads/hpk/target/debug/deps --extern byteorder=/home/grinsel/Downloads/hpk/target/debug/deps/libbyteorder-988a048c1905d79a.rlib --extern clap=/home/grinsel/Downloads/hpk/target/debug/deps/libclap-1dd52620fc331773.rlib --extern filetime=/home/grinsel/Downloads/hpk/target/debug/deps/libfiletime-3cd8e69c8d1e50d4.rlib --extern flate2=/home/grinsel/Downloads/hpk/target/debug/deps/libflate2-a4dcf890805f6092.rlib --extern glob=/home/grinsel/Downloads/hpk/target/debug/deps/libglob-dfc6ffbad95a9e84.rlib --extern lz4_compress=/home/grinsel/Downloads/hpk/target/debug/deps/liblz4_compress-431422beb68ecad0.rlib --extern nom=/home/grinsel/Downloads/hpk/target/debug/deps/libnom-6b160a556ef528ca.rlib --extern tempfile=/home/grinsel/Downloads/hpk/target/debug/deps/libtempfile-5a6b6bc7620967b1.rlib --extern walkdir=/home/grinsel/Downloads/hpk/target/debug/deps/libwalkdir-4ad7e55a77bda48b.rlib --extern zstd=/home/grinsel/Downloads/hpk/target/debug/deps/libzstd-0fef2881ee8dcbd5.rlib -L native=/home/grinsel/Downloads/hpk/target/debug/build/miniz-sys-24ade8073a23fac7/out -L native=/home/grinsel/Downloads/hpk/target/debug/build/zstd-sys-e586e6dfbfa0abf4/out
error: Edition 2018 is unstable and only available for nightly builds of rustc.

error: Could not compile hpk.

Caused by:
process didn't exit successfully: rustc --edition=2018 --crate-name hpk src/hpk/mod.rs --color always --crate-type lib --emit=dep-info,link -C debuginfo=2 -C metadata=d9d205de6e274196 -C extra-filename=-d9d205de6e274196 --out-dir /home/grinsel/Downloads/hpk/target/debug/deps -C incremental=/home/grinsel/Downloads/hpk/target/debug/incremental -L dependency=/home/grinsel/Downloads/hpk/target/debug/deps --extern byteorder=/home/grinsel/Downloads/hpk/target/debug/deps/libbyteorder-988a048c1905d79a.rlib --extern clap=/home/grinsel/Downloads/hpk/target/debug/deps/libclap-1dd52620fc331773.rlib --extern filetime=/home/grinsel/Downloads/hpk/target/debug/deps/libfiletime-3cd8e69c8d1e50d4.rlib --extern flate2=/home/grinsel/Downloads/hpk/target/debug/deps/libflate2-a4dcf890805f6092.rlib --extern glob=/home/grinsel/Downloads/hpk/target/debug/deps/libglob-dfc6ffbad95a9e84.rlib --extern lz4_compress=/home/grinsel/Downloads/hpk/target/debug/deps/liblz4_compress-431422beb68ecad0.rlib --extern nom=/home/grinsel/Downloads/hpk/target/debug/deps/libnom-6b160a556ef528ca.rlib --extern tempfile=/home/grinsel/Downloads/hpk/target/debug/deps/libtempfile-5a6b6bc7620967b1.rlib --extern walkdir=/home/grinsel/Downloads/hpk/target/debug/deps/libwalkdir-4ad7e55a77bda48b.rlib --extern zstd=/home/grinsel/Downloads/hpk/target/debug/deps/libzstd-0fef2881ee8dcbd5.rlib -L native=/home/grinsel/Downloads/hpk/target/debug/build/miniz-sys-24ade8073a23fac7/out -L native=/home/grinsel/Downloads/hpk/target/debug/build/zstd-sys-e586e6dfbfa0abf4/out (exit code: 1)

Hi,

Thanks a lot for your work on the tool and especially putting format details into README, saved me a bunch of time figuring it out from the code.

I've noticed that mods uploaded to Steam Workshop via Surviving Mars Mod Editor are packed into .hpk, which can be extracted by the tool, but unpacked files are still compressed with ZSTD signature at the start.

Which are same fragmented files as lz4/zlib-compressed ones, just using zstd instead of these two.

Decompression script example:
https://github.com/mk-fg/games/blob/master/surviving-mars/zstd-dec.py#L39-L64

Not sure if it's also the case for other compression, but had to add "len(buff) != dst_len" check there as for e.g. .png files, despite ZSTD header, fragments are not compressed.

Attached ModContent.hpk.zip has ModContent.hpk for my simple mod, which has .lua and .png files with ZSTD header in it, both should be decompressible by python script above, but not currently handled by hpk tool itself, unfortunately.

ModContent.hpk.zip

• Set last modification date from _filedates file on extraction on windows and linux. (detect Tropico 4/5)
• Generate _filedates file and add option to select the date format on creation.

Trying to unpack savegame gives

Error: Hpk(Io(Error { kind: UnexpectedEof, message: "failed to fill whole buffer" }))

Autosave.savegame.sav.zip

file itself attached

Hello. I've decided to localize the new game Stranded: Alien Dawn.
But I ran into a problem (maybe I'm too dumb).
Unpacking proceeds normally, everything is decoded correctly.
However, I can't pack it back.
An access denied error pops up. What am I doing wrong?

d:\Works\Tools\hpk-v0.3.9-x86_64-pc-windows-msvc>hpk print "E:\SteamLibrary\steamapps\common\Stranded Alien Dawn\Local\English.hpk"
reading file: E:\SteamLibrary\steamapps\common\Stranded Alien Dawn\Local\English.hpk
header:
  data_offset: 0x24
  fragments_residual_offset: 0x0
  fragments_residual_count: 0
  fragments_per_file: 1
  fragments_filesystem_offset: 0x5E138
  fragments_filesystem_length: 24
filesystem entries: 3
filesystem fragments:
  0x5E11F  len: 25
  0x5E10D  len: 18
  0x24     len: 385257
dir:  index=1 depth=0 ""
 fragment: 0x5E11F len: 25
dir:  index=2 depth=1 "CurrentLanguage"
 fragment: 0x5E10D len: 18
file: index=3 depth=2 "CurrentLanguage\\Game.csv"
 fragment: 0x24 len: 385257
 compressed: ZSTD inflated_length=1989815 chunk_size=131072 chunks=16
  chunks: 0x4C     len: 29351
          0x72F3   len: 27035
          0xDC8E   len: 26581
          0x14463  len: 25720
          0x1A8DB  len: 26760
          0x21163  len: 23628
          0x26DAF  len: 20872
          0x2BF37  len: 28986
          0x33071  len: 29952
          0x3A571  len: 26444
          0x40CBD  len: 17184
          0x44FDD  len: 21604
          0x4A441  len: 25649
          0x50872  len: 25496
          0x56C0A  len: 25503
          0x5CFA9  len: 4416

Error message

d:\Works\Tools\hpk-v0.3.9-x86_64-pc-windows-msvc>hpk create D:\Works\Tools\hpk-v0.3.9-x86_64-pc-windows-msvc\pack D:\Works\Tools\hpk-v0.3.9-x86_64-pc-windows-msvc\extracted
Error: Hpk(Io(Os { code: 5, kind: PermissionDenied, message: "Access denied." }))

Here is the archive with localization English.zip

Hello. Could you add support for the first templar game? So that we can extract models, textures, sounds, etc. Thanks!

When extracting armstrong.hpk and packing it again, the game refuses to load into the menu.

Looking at your file format specifications, it seems like something weird is going on:

The original file does have the data of the first file right after the one BPUL header starting at 0x24.
The newly created file though does contain two BPUL headers and then the data for the first file, which then starts later (0x48 I guess?)

here the first 200 bytes of each file:

original

4250554C2400000001000000FFFFFFFF000000000000000001000000CA267A0448050000444B4946000020005650383080073804905F0100010000000000000000000080C80D0000000000000000000030B7019D012A80073804004708858588858488020200061604F70681649F6BDB9B27387B27387B27387B27387B27387B27387B27387B27387B27387B27387B27387B27387B27387B27387B27387B27387B27387B27387B27387B27387B27387B27387B27387B27387B27387B27387B27387B27387B27387B

repacked

4250554C2400000001000000FF0000000000000000000000010000006EB07204480500004250554C2400000001000000FFFFFFFF0000000000000000010000008E763000580000004C5A34200E000000000002001000000072657475726E20313031303738324C5A34208E250A000000020024000000D36200003EB60000BA130100C06F010045D40100FF16734200000014000000456E7469746965732F41636164656D792E656E740D0A4D35000000181F000042446F6F7223006FF7310000002023000050466C

Also I noticed that the first header in the repacked archive does have 0x000000FF instead of 0xFFFFFFFF.

Any idea where this difference comes from and if that could be the cause of the problem?

EDIT:
when running it from terminal, the game seems to crash at loading the DLCs, which this file is part of.

.local\share\Steam\steamapps\common\Surviving Mars\CommonLua\Dlc.lua(702): field LoadDlcs
[string ""](-1):  <>
        --- end of stack

EDIT2:
I also noticed, that there is a size difference between the original and the repacked file (the bak is the original one):

Hi! Thank you for providing hpk unpacker/archiver for Haemimont Engine game files.
I'm currently having a problem when archiving hpk files for localization (Korean language, edited from unpacked Tchinese.hpk file) fonts are missing in-game.

I first assumed it was font issue, but when I archived the unpacked Tchinese.hpk file without editing, and tested it in-game, same issue occurred.

Following is the link to 1)original Tchinese.hpk file, 2)rearchived Tchinese.hpk file from unpacked files

1. https://drive.google.com/file/d/1Vmq7td4Qf1E8_2nr_5ciQzGiaOwMTzPW/view?usp=drive_link
2. https://drive.google.com/file/d/1Ju7h_da_W3UxvuupDLty7YdXxQu3uXYn/view?usp=drive_link

Could it be that I'm somehow doing the archiving process not properly? I've made sure when archiving, the directory order remained same (Tchinese/CurrentLanguage/Game.csv) and used bat file with command line .\hpk.exe create Tchinese Tchinese.hpk