valveresourceformat / valvepak

📦 Fully fledged library to work with Valve's Pak archives in .NET

Valve Pak (vpk) for .NET

VPK (Valve Pak) files are uncompressed archives used to package game content. This library allows you to read and extract files out of these paks.

Usage:

using var package = new Package();

// Open a vpk file
package.Read("pak01_dir.vpk");

// Can also pass in a stream
package.Read(File.OpenRead("pak01_dir.vpk"));

// Optionally verify hashes and signatures of the file if there are any
package.VerifyHashes();

// Find a file, this returns a PackageEntry
var file = package.FindEntry("path/to/file.txt");

if (file != null) {
	// Read a file to a byte array
	package.ReadEntry(file, out byte[] fileContents);
}

Do note that files such as pak01_001.vpk are just data files, you have to open pak01_dir.vpk.

valvepak's Issues

Add method that returns a mmap stream for an entry

See https://github.com/SteamDatabase/ValveResourceFormat/pull/482/files

Instead of reading the whole file into memory, return a memory mapped file stream.

ValveVpk Dispose still occupies memory?

Bug explanation

When using Package.AddFile(), I found that if I add a single file and then destroy it through Package.Dispose(), wait for a while and GC will automatically recycle it.
But if I add multiple files and then destroy them, GC will not recycle them no matter how long I wait.
Even if I manually instruct GC, it will not recycle them.
If the user is operating multiple VPK files, this problem will occupy a large amount of memory and cause significant delays when using the program, thus affecting the overall user experience.

So, is there a good way to read files into memory for recycling, or other methods to read files to reduce memory overhead?
Thanks for taking the time to reply!

Remove `IsDirVPK` check

It's possible for chunked vpks to not have _dir suffix.

I think it should work just fine with this check removed:

ValvePak/ValvePak/ValvePak/Package.Read.cs

Lines 327 to 330 in bf8492f

	if (!IsDirVPK)
	{
	throw new InvalidOperationException("Given VPK filename does not end in '_dir.vpk', but entry is referencing an external archive.");
	}

Add ReadEntry overload that can read into existing output array

For use with cached arrays, e.g. ArrayPool.

Findings based on disassembly

• The metadata that can be included after directory entries is never used as part of the file when reading it.
• There are two special archive IDs: 0x7fff (current archive, for non-_dir archives), and 0xffff (which means "end of list")
• Each VPK entry has one or more archive locations, terminated by 0xffff (which does not have an offset or size after it)
• Archive locations are concatenated to create the file's actual content while reading

Bad VPK Version (4294967041)

Describe the bug

Whether I open the VPK using GCFScape or VRF, you will be prompted with the VPK version problem.
VRF displays "Bad VPK version. (4294967041)".
This is an L4D2 game package file. The game can read it, but the unpacking softwares cannot read the file

What Valve game does this happen in

Left 4 Dead 2

Reference file in a Source 2 game

（Not an official vpk file）

Expected behavior

System.IO.InvalidDataException: Bad VPK version. (4294967041)
at SteamDatabase.ValvePak.Package.Read(Stream input) in /home/runner/work/ValvePak/ValvePak/ValvePak/ValvePak/Package.cs:line 203
at SteamDatabase.ValvePak.Package.Read(String filename) in /home/runner/work/ValvePak/ValvePak/ValvePak/ValvePak/Package.cs:line 157
at GUI.Types.Viewers.Package.Create(VrfGuiContext vrfGuiContext, Byte[] input) in D:\a\ValveResourceFormat\ValveResourceFormat\GUI\Types\Viewers\Package.cs:line 35
at GUI.MainForm.ProcessFile(VrfGuiContext vrfGuiContext, PackageEntry file) in D:\a\ValveResourceFormat\ValveResourceFormat\GUI\MainForm.cs:line 538
at GUI.MainForm.<>c__DisplayClass22_0.b__0() in D:\a\ValveResourceFormat\ValveResourceFormat\GUI\MainForm.cs:line 456
at System.Threading.Tasks.Task`1.InnerInvoke()
at System.Threading.Tasks.Task.<>c.<.cctor>b__272_0(Object obj)
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
the vpk file.zip

What version of VRF are you using? On what platform?

the latest one，Windows

ValvePak Makes the VPK files read only (locked) while opened

This is likely due to FileShare.Read when we open the VPK files

Remove unused code from AsnKeyParser

Increases test coverage too

Encountered OtherMD5Section with size of 0 (should be 48)

The following stack trace is reported when attempting to read test_clock.vpk from this Workshop map:

System.IO.InvalidDataException: Encountered OtherMD5Section with size of 0 (should be 48)
   at SteamDatabase.ValvePak.Package.ReadOtherMD5Section() in /home/runner/work/ValvePak/ValvePak/ValvePak/ValvePak/Package.Read.cs:line 266
   at SteamDatabase.ValvePak.Package.Read(Stream input) in /home/runner/work/ValvePak/ValvePak/ValvePak/ValvePak/Package.Read.cs:line 84
   at GUI.Types.Viewers.Package.Create(VrfGuiContext vrfGuiContext, Byte[] input) in D:\a\ValveResourceFormat\ValveResourceFormat\GUI\Types\Viewers\Package.cs:line 37
   at GUI.MainForm.ProcessFile(VrfGuiContext vrfGuiContext, PackageEntry file) in D:\a\ValveResourceFormat\ValveResourceFormat\GUI\MainForm.cs:line 606
   at System.Threading.Tasks.Task`1.InnerInvoke()
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)

CS2 is able to load this map and so is GCFScape.

The exact test_clock.vpk used when this error was encountered can be downloaded here: https://gist.github.com/roge/2493b95513b8f1b0914e1664d9cee983/raw/0d934400f71018a9547b288e0209802bf983b0b6/test_clock.vpk

Could someone update the ValveResourceFormat version and release?

ValveResourceFormat has been updated to 0.0.13 which contains a critical change that I need for my usage of this tool. Could someone update to the new version and release an update of this tool? Thanks!

Add test case for SmallData

From ValveResourceFormat/ValveResourceFormat#22