frodo45127 / rpfm

Rusted PackFile Manager (RPFM) is a... reimplementation in Rust and Qt5 of PackFile Manager (PFM), one of the best modding tools for Total War Games.

Rusted PackFile Manager

Rusted PackFile Manager (RPFM) is a... reimplementation in Rust and GTK3 Qt5 of PackFile Manager (PFM), one of the best modding tools for Total War Games. Not only it can edit Packs, but also has integrated editors for DB Tables, Loc files, scripts,... and a bunch of different file formats.

Downloads here: https://github.com/Frodo45127/rpfm/releases

Manual here: HERE, READ IT BEFORE ASKING.

Requirements (to use)

• Windows: Just download it, extract it somewhere and execute it.
• Linux:
  • Arch Linux and derivates: it's in the AUR as rpfm-bin.
  • Other distros: Make sure you have Qt5 5.14 or higher, xz, and 7zip installed. DDS files also require you to have the Qt5 Imageformats DDS library installed.
• MacOS: You'll know it when I manage to compile it for Mac.

Requirements (to build)

Check the building instructions here

FAQ

• How can I translate it to my own language?: go to the locale folder, copy the English_en.ftl file, rename it to NameYouWantInTheUI_xx.ftl. For example, for spanish it'll be Español_es.ftl. Translate it. Done.
• Why there is no .exe in the download?: because you downloaded the source code, not the program. Check at the begining of this description, where is says Downloads here.

Credits

• Created and Programmed by: Frodo45127.
• Extra programming work by: Vandy.
• Modern DDS Read support by: Phazer.
• App Icons until v1.6.2 by: Maruka.
• App Icons since v2.0.0 by: Jake Armitage.
• AnimPack research: Marthenil and Frodo45127.
• Ca_vp8 research: John Sirett.
• LUA functions by: Aexrael Dex.
• LUA Types for Kailua: DrunkFlamingo.
• RigidModel research by: Mr.Jox, Der Spaten, Maruka, phazer and Frodo45127.
• RigidModel module until v1.6.2 by: Frodo45127.
• RigidModel module since v2.4.99 by: Phazer.
• TW: Arena research and coding: Trolldemorted.

rpfm's Issues

Schema missing `campaign_maps` table definition

Schema definition is missing for campaign_maps_tables in warhammer schema

RPFM v1.2.0 with latest schema updates and I also verified that it's missing in the master branch.

edit: realized I forgot to mention which schema though I'm unsure if older games implemented the table

BOM in certain Text PackedFiles is not treated properly.

Some Text PackedFiles currently have BOM, shown as ÿþ, or FF FE at the begining of the file. We need to detect that and hide it, so the user can't see it, but RPFM reads it and conserve it when saving.

NOTE: This is specially visible in older games, not so much in newer ones.

Compression Support for PFH5 PackFiles.

PFH5 PackFiles use LZMA for PackedFile compression, but they change the header with a non-standard one. So we need:

• Test different compression levels and see if they work ingame.
• Allow compression per PackFile.
• Allow compression per PackedFile (optional).
• Make the UI show somewhere if a PackedFile is compressed or not.

rpfm fails to run from powershell

using carg run in cmd works fine, but not in powershell:

PS D:\repositories\rpfm_vanilla> cargo run
   Compiling rpfm-code v0.9.4 (file:///D:/repositories/rpfm_vanilla)
warning: unused `std::result::Result` which must be used
    --> src\main.rs:4867:25
     |
4867 | /                         packfile::update_packed_file_data_rigid(
4868 | |                             &packed_file_rigid_model,
4869 | |                             &mut pack_file_decoded,
4870 | |                             data.1
4871 | |                         );
     | |__________________________^
     |
     = note: #[warn(unused_must_use)] on by default
     = note: this `Result` may be an `Err` variant, which should be handled

warning: unused `std::result::Result` which must be used
    --> src\main.rs:4890:33
     |
4890 | /                                 packfile::update_packed_file_data_rigid(
4891 | |                                     &packed_file_rigid_model,
4892 | |                                     &mut pack_file_decoded,
4893 | |                                     index
4894 | |                                 );
     | |__________________________________^
     |
     = note: this `Result` may be an `Err` variant, which should be handled

    Finished dev [unoptimized + debuginfo] target(s) in 54.12s
     Running `target\debug\rpfm-code.exe`
error: process didn't exit successfully: `target\debug\rpfm-code.exe` (exit code: 3221225595)

Make the decoding/encoding PackFile logic independant as a lib.

While investigating TWA's tables I noticed that some packs were not properly unpacked. Luckily I managed to track the problem down: If 1 << 4 is set in the bitmask, every item in the payload starts at a multiple of 8.

twa_pack_lib fix: TotalWar-Modding/tw_pack_lib@c549376 (unfortunately rather big because of major simplifications)

CLI for RPFM.

Someone asked for this, and it's not a bad idea to make it work better with scripts and all that stuff, but needs some planning and investigation on how to do it.

non-library projects should include Cargo.lock for reproducible builds

Otherwise a yanked (or non-backwards compatibly updated) (recursive) dependency will break builds on new git checkouts

Implement a `Live Filter` for the TreeView.

This requires subclassing the filter widget to get it filter only by the last item in the parent-child chain. Because currently it filters by any item in that chain, so if one of the parents doesn't match the filter, it hides him, and everyone of his children, no matter if they matched the filter or not.

Implement `Open Folder after Extracting` setting.

This setting should make RPFM open the folder in the explorer/file manager where a file/folder has been extracted just after the extraction finished. We can try to add it to the "Extraction successful" dialog as an "Open Folder" button instead. Now that I think of it, that would be a better way to do it.

Remove/edit 3 decimal limitation for Float numbers.

Some tables (very few, like sea_surfaces) actually use more than 3 decimals on some cells, and RPFM defaults them to 0.000.

The problem of removing the limitation comes with rounding. Currently RPFM rounds to the third decimal to remove weird precision issues like getting 0.4999999995 instead of 0.5. Removing the limitation means we need to find another way to get rid of the precision problems.

Sorting/Filtering boolean columns in DB Tables/Loc PackedFiles is broken.

The sorting/filtering code sorts and filters by text. These columns have no text. So we need to take a look at the code and change it for these columns.

Implement a `Remove everything not in filter` feature for DB Tables and Loc PackedFiles.

Basically, we can use this to filter parts of a table, then wipe out everything not in screen from that table, so what we see is what's left. Useful for reducing tables.

Reordering columns in DB Tables/Loc PackedFiles breaks the `Ctrl+LClick` on row numbers to deselect rows.

And I have no idea why. Needs investigation.

Hide/Show Columns list should not be closed when clicking on it.

So the user can hide/show multiple columns in one go. So either we find a way to not hide the submenu, or we move it to another place that doesn't dissapear when clicking on it.

Expand the Text PackedFile's Editor.

The idea is to make it into a more complete editor. Features to add:

• 4 spaces tabs.
• Custom font selector (optional).
• Syntax Highlighting (very needed).
• Auto completion (at least for LUA).

Revise encoder/decoder helpers performance.

Recently I found out that, dividing a file into smaller vectors, then adding them to a large one, can hurt performance by a lot. The idea is to find if there is any place other than crypto where this can be changed so we can win some extra performance.

Warhammer I support

What do I need to support warhammer I?
is it a matter of new json schema?

rpfm doesn't cope with malformed input

If you want to fix that within rpfm's pack module you have to

• stop trusting the sizes you read in the header
• ensure the payload-position variable does not overflow (twa_pack_lib uses checked_add)
• enforce an opened file you lazily read from is not modified
• enforce open_dependency_packfile is using a valid file, or stop using unwrap() all the time

For the sake of simplicity, I'd also recommend you move all pack-related filesystem usage into the same module.

Sorting numerical columns in DB Tables is broken.

RPFM sorts them alphabetically, not using the numbers. For example, RPFM will consider 10 to go before 2. This needs some investigation, as I have never touched the sorting code.

Revised Dark Theme.

The Dark Lord is not pleased with:

• Checkboxes are almost invisible if they're not checked.
• Blue text it's hard to read.
• Disabled text is hard to read.

Allow adding multiple folders at once.

This require us to use a trick in the FileDialog that I don't know if it'll work on Linux and Windows, as for those RPFM defaults to the native FileChooser.

Implement `Open Containing Folder` feature for PackFiles.

Just a simple option in the Right-Click menu of the PackFile.

Investigate the unknown boolean in DB Tables/Loc PackedFiles Headers.

There is a little boolean there whose use we don't know. This calls for an investigation. Not really affecting how the tables work currently, but I don't like "unknown" parts in the code.

Make the `TableState` stuff selectable by the user as individual settings.

TableState has grown from 1-2 mildly annoying settings to 3-4, and it's expected to receive more in the future. The idea behind this is to make a setting in the settings window per setting, so the user can enable/disable only what they want.

Certain .tsv Files exported by RPFM fail in reimporting them immediately afterwards

There is probably a bug in the .tsv or .loc parsing logic somewhere. The following bug can be reproduced with the NordoGerman mod for Rome II, which contains a .loc file with a multitude of translations.

I can open the pack file with RPFM and export the .loc as .tsv. But when I reimport the same exported .tsv file immediately afterwards, the parser complains with the following error:

This TSV file has an error in the row 4747, field 2 (both starting at 1). Please, check it and make sure the value in that field is a valid value for that column.

Which is strange because it had no problem opening and exporting the same file before.
The respective row contains the following string: (The [TAB]s represents a Tab whitespace)

event_log_descriptions_event_content_diplo_war[TAB]%S - %S[TAB]true

Maybe there is a problem with the leading % symbol in the second column? Pack File Manager 5.0 can handle this exported .tsv without a problem and reimports it successfully (Apart from stripping all leading " like it always does -.-)

This occurs on the newest version 1.2.2. of RPFM. I hope you can look into it, as I much prefer to edit .loc files with RPFM if it weren't for this bug. Thanks in advance and keep up the good work!

support TWA's tables

The tables are encrypted indeed, but I think I managed to get the key. You can use twa_table_decrypt or twa_table_decrypt_lib to decrypt them, but it could be that the ends of the decrypted tables are a bit off because I didn't understand the paddings correctly.

Do you think the decrypted tables look ok? Naturally it would be great if rpfm would display the tables, so I assume we have to create a schema?

Improved editing controls.

This is a list of possible suggested improvements to make editing stuff faster.

• Ctrl+Delete/BackSpace for deleting everything up to the last/next space or underscore.
• Keyboard Scrolling in all four directions without deselections.

DB Folders lose their color (if colored) when clicking on the tables inside them.

This is confirmed to happen after you add tables to a PackFile, then open them. Something wrong is going on in the function that detects the folder state in the UI.

Implement multiple simultaneous `Live Filters` for DB Tables and Loc PackedFiles.

There are two ways of doing this:

• By subclassing the filter widget to allow it.
• By stacking multiple filter widgets on top of each other (chaos).

make the communication fail-safe

If I understand #14 correctly, the communication between ui and bg thread may fail, if the user triggers multiple commands at once. Taken from #14:

Imagine we have the commands A0, A1 and B. All three return data, A1 will only be sent after A0 finishes, and A0 and B are sent by user interaction.
Now

• the user initiates A0
• the A0 handler is in a tryrecv-loop, so the user can trigger new commands
• the user initiates B
  The command channel contains: B -> A0
  The bg->ui data channel contains: nothing

Now

• the bg thread handles A0, puts a0-data in the bg->ui channel
• the ui thread handles a0-data, and puts A1 into the ui->bg command channel
  The command channel contains: A1 -> B
  The bg->ui data channel contains: nothing

Now

• the bg thread handles B and puts b-data into the bg->ui channel
• the ui thread expects a1-data but gets b-data 💣 💥

To address that, we must prevent overlapping access to the ui->bg (command) channel. How to archive that depends on what behaviour you want:

1. No command queue

Dead simple: Track channel-usage in a bool, only issue a new command when it is false, set it to true before issuing, set it to false after completion.

Pros:

• simple
• ui still feels responsive because resizing and menu traversing handling still works
• you can upgrade it to 2. later if you feel like it

Cons:

• users have to wait until a command completes before they can issue a new one

2. (Intelligent) command queue

Basically 1. with a dedicated queue attached. UI handlers enqueue commands into the queue, the UI thread takes a new command out of the queue once its predecessor has finished.

Pros:

• if a user performs an action n times, the backend will try to execute it n times

Cons:

• complicated
• if a user performs an action n times, the backend will try to execute it n times

Napoleon Support.

Stuff needed to set this as "Done".

• PackFile Logic.
• SupportedGame data.
• GameSelected code.
• Schema Definitions.

use enums for communication

Let's continue the discussion in a separate issue, #8 is already taking long to load on my phone.

Quite the opposite. The situation is the data is sent sometimes without a command. For example, there are commands that you send and the UI thread waits for a response. Then, depending on the response you send more data or just do nothing.

But why wouldn't that work with two enums? Call the enums command, response, message or whatever, the receiver can match the result and send whatever it wants in reponse (or nothing at all).

There are more responses that commands. There are currently 55 commands, and between 50-100 data types that are send one way or another.

That is a lot indeed, but not too much imo. Which commands have different responses?

Also, there's another problem related to merging the channels. When I started the 0.9 version and the Qt port, I set a goal: the UI MUST NOT HANG. To do it, RPFM currently makes the UI thread to enter on a loop that keeps processing events (like resizing the window) until it gets a response from the background thread.

That is completely reasonable! A few cents from what I noticed while browsing your channel code:

• shouldn't you phase out check_message_validity_recv for the non-blocking variant?
• why don't you loop within check_message_validity_tryrecv? Right now rpfm contains the same loop 22 times
• check_message_validity_tryrecv and check_message_validity_recv_background do exactly the same (?), can't the former be a wrapper around the latter?
• you might want to move the ui and bg code into separate files. It helps matching IDE search results to functionality (at the moment I have to check whether a usage of foo in main.rs is in the bg thread or ui thread), and main.rs is almost 6k lines of code long - that is really a lot

That means you can potentially send a command while there is another one executing itself, making the entire program crash.

Yes and no. Yes, at the moment you can, but you could change that at any time by not sending a new command when there is a pending one on the "wire".

In earlier 0.9 versions, you could crash the program by keeping pressed the "Del" button because the deletion process got messed by a checking process, and message were received in the wrong place. To "fix it", I left a channel for commands only (so commands can be chained and don't interfere with one another) and another for data, and every time data received, RPFM checks that's the type it needs using generics and crashes if it's not .
Merging them will mean the commands can potentially crash in many more places that before.

Are you sure that was what fixed the issue? If process_events allows your event handlers to execute, even with a dedicated data channel you should not send more than one commands at once. Imagine we have the commands A0, A1 and B. All three return data, A1 will only be sent after A0 finishes, and A0 and B are sent by user interaction.

Now

• the user initiates A0
• the A0 handler is in a tryrecv-loop, so the user can trigger new commands
• the user initiates B

The command channel contains: B -> A0
The bg->ui data channel contains: nothing

Now

• the bg thread handles A0, puts a0-data in the bg->ui channel
• the ui thread handles a0-data, and puts A1 into the ui->bg command channel

The command channel contains: A1 -> B
The bg->ui data channel contains: nothing

Now

• the bg thread handles B and puts b-data into the bg->ui channel
• the ui thread expects a1-data but gets b-data 💣 💥

Mmmm.... I'm not sure about the second and third one. The way I see it, the same problem with wrong messages being sent can also happen with enums, because you can receive the wrong variant and the program will have to crash, to avoid weird behavior.

You surely still can send the wrong variant, but it does stop you from putting a wrong data type into the correct variant! Makes reviewing code much easier, since you can be sure that a code change which does not touch the variant handling/sending panics because the channel is disrupted.

And the fifth one.... if you want data to be shared in an enum without deleting the original data (because the threads are isolated), you have to copy it anyways isn't?

Yes and no! Threads are not isolated, rust encourages and enables you to share data in completely safe ways. You don't have (at least without unsafe) the freedom of other programming languages, but if you prove the compiler that what you are doing is safe it lets you do it.

So one the one hand yes, to send or share a type between threads it must implement Send or Send and Sync, and those types in the stdlib internally copy themselves when you move them into a closure.
On the other hand no, you do not have to copy the data. To share a Vec<u8> between threads without copying the vector, you can move it into an Arc and move a/multiple clone(s) to the/some other thread(s). An arc is just a few bytes in size which can be easily cloned, the contained type will exist only once in memory and be dropped as soon as all arcs which point to it are dropped.

Rewrite the entire RigidModel module.

This module is the only one still using the old code for decoding/encoding, and Phazer has done quite an extensive work decoding the files so... we're going to rewrite the module to be up-to-date with the other modules, and to use Phazer work to get more usable stuff. This means we need:

• Port Phazer logic code to Rust (or use his lib, which will mean harder maintainability).
• Get the module up-to-date with the other modules (revised error checking and corrupted file detection).
• Rewrite the UI to show far more data than it's shown now.

Game Paths used internally only work on Windows.

Linux paths are... diferrent. Feral ports have their data in multiple separate subfolders instead in a common "data/" folder. So we need to expand the path system to admit multiple ones, and use X or Y paths depending on what's in their base folder. So far we got:

• Windows: all games use the path game_folder/data.
• Linux: depends on the game:
  • Warhammer 2: Feral Port. It uses game_folder/share/data/data for the base game's data.
  • Warhammer: No idea.
  • Thrones of Britannia: No idea.
  • Attila: In-House Port. No idea.
  • Rome 2: No idea.
  • Shogun 2: Feral Port. It uses game_folder/share/data/data for the base game, and game_folder/share/data/fots for FOTS-Specific PackFiles.
  • Napoleon: No idea.
  • Empire: No idea.
• MacOS: I don't have a Mac to test, so no idea, but in theory if this gets fixed there are not many problems to build RPFM for MacOS and release a Mac version.

`Are you sure?` Dialog not showing up when closing the main window.

The Are you sure? Dialog for avoiding unwanted data loss doesn't pop up when we close the main window. This is because we need to subclass certain window event to show up this dialog in that specific moment, as the events we have access without subclassing cannot stop the Close Window operation.

AutoFill new rows using default values/references.

This is a bit complex because there are a lot of fields that may or may not have values, so we can only apply it to those we're sure always have a value (StringU8/U16, not OptionalStringU8/U16). For numbers is already done.

Make RPFM remember the scrolled position of a previously opened DB Table/Loc.

As the title says. This requires #40, as it's a change people may not want.

Some questions about how you made this!

First of all, this is an amazing work! I wish I could write something like this myself someday.

I'm curious about how you could create the schema file though.

I've also been looking through the game data using PFM recently, but it was rather difficult to navigate between different tables using that tool. So I started researching how I could migrate the raw xml data files provided by the Assembly Kit to a relational database for easier reference among tables. Unfortunately, I couldn't find any file that described the schema. And based on a post on twcenter about how to manually decode the binary data to create the schema file for PFM, I guessed that there was no readily available schema file. I thought of automatically parsing at least the xml files to build a schema of tables, but then I noticed that column names of xml files did not explicitly refer to table names. For example, the armour column in land_units_tables referred to unit_armour_types_tables without telling so. That meant that completely automatic parsing was impossible and that I had to look through the result to personally verify the schema. All in all, I was starting to think that this was more than I could do in my free time.

Then I saw this project and the schema file - you could guess how amazed I was! How did you create it? Could you find some kind of schema file among the binary data? Or did you personally create the entire schema by ironing out each column? Based on the git history, I guess it's more likely the latter but I'm really curious about how you did it!

Empire Support.

Stuff needed to set this as "Done".

• PackFile Logic.
• SupportedGame data.
• GameSelected code.
• Napoleon Definitions, to reduce decoding time. Depends on #32.
• Schema Definitions.

RPFM TreeView gets confused when there is a file and a folder with the same name in the same folder.

Easily reproducible in Shogun 2 data.pack and in some Napoleon and Empire Packs. For some reason sometimes the function that decides if something is a file or a folder gets confused and marks both as files, despite one of them being a folder.

Save funktion not working with edited Script

Hi there,

here my step by step order to reproduce:

1. Open mod
2. Edit script
3. Save mod: get save confirmation from RPFM
4. close mod
5. reopen mod: previously made changes are not saved!

Thanks for you hard work!

Improve behavior for dealing with compressed/encrypted PackedFiles.

Currently, data is decompressed/decoded on-the-fly while read from disk, and it's always stored like that in memory. Currently, saving a PackFile with compressed PackedFiles will cause RPFM to decompress them, then save, which takes a while. The idea is to read the compressed/encrypted data to memory, then decompress/decrypt it when we need it. That way it doesn't have to be decompressed on saving (or recompressed when that feature land).

Issues when you try to extract a folder to a folder where a folder with the same name already exists.

This looks like a Qt/Windows issue, as I just call the default function from Qt, and this happens. Needs some more investigation though.

The install function doesn't work on windows due to filepath issues

Describe your problem
I don't want to annoy you or anything but the install function is still buggy. :) I get the following error when trying to use it on windows:

Error while trying to copy one or more files to the following folder:
"Z:/SteamGames/SteamApps/common/Total War Rome II\data\wolphyx_medals.pack"

Looks like a filepath issue, as the slashes and backslashes are inconsistent. I tried changing them to backslashes in the settings.json, but that just caused RPFM to don't read any settings at all anymore.

Extra info?
Occurs on RPFM 1.3.3 when using the install function

Support for older titles?

Sorry if I have in any way overlooked things, but will this support older titles like Rome 2 or Atilla?
I mostly work on these.

Splendid work btw!

spec for .pack files

Is there a specification somewhere for the .pack files that I can look at? So as to better understand this program.

I know that loading pack files takes up a lot of memory and was wondering if there's a way to interpret the data in the pack files lazily, like if a section regarding character traits has an identifiable byte range that could be used.

I haven't been able to get the application compiled on my windows machine yet, but I aim to help out soon! Awesome project, by the way - excellent way to learn rust just by the sheer breadth of it.

RPFM crashes on installing mods

Describe your problem
Since the newest update, whenever I try to "Install" one of my mods using the install button in the MyMod menu, RPFM crashes completely. This kills the modding workflow significantly, as I have to move files manually to the /data folder to test and/or upload them. I'd appreciate if this could be looked into.
Apart from that - I love the dark theme!

Extra info?
Version 1.3.2
For me this is reproducible with all of my MyMods.

Implement shortcut to select from the selection to the begining/end of a table.

We need to map a key press for this. Not sure how to do it with the current Qt Bindings.

How do I install this?

Really sorry if this seems noobish or anything, but how do I install this tool?

I´m a small time modder and I´d love to try out the new Attila compability,
but I haven´t been able to run this at all. Can´t seem to find any executable.

Support for Total War Arena

Could you also support Total War Arena?

Their .pack header says PFH5 (I assume pfh5 does not equal pfh5 if we are talking about different games?), but rpfm crashes if I drag and drop a pack file into it.