frodo45127 / rpfm Goto Github PK

There are two ways of doing this:

• By subclassing the filter widget to allow it.
• By stacking multiple filter widgets on top of each other (chaos).

If I understand #14 correctly, the communication between ui and bg thread may fail, if the user triggers multiple commands at once. Taken from #14:

Imagine we have the commands A0, A1 and B. All three return data, A1 will only be sent after A0 finishes, and A0 and B are sent by user interaction.
Now

• the user initiates A0
• the A0 handler is in a tryrecv-loop, so the user can trigger new commands
• the user initiates B
  The command channel contains: B -> A0
  The bg->ui data channel contains: nothing

Now

• the bg thread handles A0, puts a0-data in the bg->ui channel
• the ui thread handles a0-data, and puts A1 into the ui->bg command channel
  The command channel contains: A1 -> B
  The bg->ui data channel contains: nothing

Now

• the bg thread handles B and puts b-data into the bg->ui channel
• the ui thread expects a1-data but gets b-data 💣 💥

To address that, we must prevent overlapping access to the ui->bg (command) channel. How to archive that depends on what behaviour you want:

1. No command queue

Dead simple: Track channel-usage in a bool, only issue a new command when it is false, set it to true before issuing, set it to false after completion.

Pros:

• simple
• ui still feels responsive because resizing and menu traversing handling still works
• you can upgrade it to 2. later if you feel like it

Cons:

• users have to wait until a command completes before they can issue a new one

2. (Intelligent) command queue

Basically 1. with a dedicated queue attached. UI handlers enqueue commands into the queue, the UI thread takes a new command out of the queue once its predecessor has finished.

Pros:

• if a user performs an action n times, the backend will try to execute it n times

Cons:

• complicated
• if a user performs an action n times, the backend will try to execute it n times

The Are you sure? Dialog for avoiding unwanted data loss doesn't pop up when we close the main window. This is because we need to subclass certain window event to show up this dialog in that specific moment, as the events we have access without subclassing cannot stop the Close Window operation.

So the user can hide/show multiple columns in one go. So either we find a way to not hide the submenu, or we move it to another place that doesn't dissapear when clicking on it.

Currently, data is decompressed/decoded on-the-fly while read from disk, and it's always stored like that in memory. Currently, saving a PackFile with compressed PackedFiles will cause RPFM to decompress them, then save, which takes a while. The idea is to read the compressed/encrypted data to memory, then decompress/decrypt it when we need it. That way it doesn't have to be decompressed on saving (or recompressed when that feature land).

First of all, this is an amazing work! I wish I could write something like this myself someday.

I'm curious about how you could create the schema file though.

I've also been looking through the game data using PFM recently, but it was rather difficult to navigate between different tables using that tool. So I started researching how I could migrate the raw xml data files provided by the Assembly Kit to a relational database for easier reference among tables. Unfortunately, I couldn't find any file that described the schema. And based on a post on twcenter about how to manually decode the binary data to create the schema file for PFM, I guessed that there was no readily available schema file. I thought of automatically parsing at least the xml files to build a schema of tables, but then I noticed that column names of xml files did not explicitly refer to table names. For example, the armour column in land_units_tables referred to unit_armour_types_tables without telling so. That meant that completely automatic parsing was impossible and that I had to look through the result to personally verify the schema. All in all, I was starting to think that this was more than I could do in my free time.

Then I saw this project and the schema file - you could guess how amazed I was! How did you create it? Could you find some kind of schema file among the binary data? Or did you personally create the entire schema by ironing out each column? Based on the git history, I guess it's more likely the latter but I'm really curious about how you did it!

This setting should make RPFM open the folder in the explorer/file manager where a file/folder has been extracted just after the extraction finished. We can try to add it to the "Extraction successful" dialog as an "Open Folder" button instead. Now that I think of it, that would be a better way to do it.

TableState has grown from 1-2 mildly annoying settings to 3-4, and it's expected to receive more in the future. The idea behind this is to make a setting in the settings window per setting, so the user can enable/disable only what they want.

Could you also support Total War Arena?

Their .pack header says PFH5 (I assume pfh5 does not equal pfh5 if we are talking about different games?), but rpfm crashes if I drag and drop a pack file into it.

Easily reproducible in Shogun 2 data.pack and in some Napoleon and Empire Packs. For some reason sometimes the function that decides if something is a file or a folder gets confused and marks both as files, despite one of them being a folder.

Recently I found out that, dividing a file into smaller vectors, then adding them to a large one, can hurt performance by a lot. The idea is to find if there is any place other than crypto where this can be changed so we can win some extra performance.

Let's continue the discussion in a separate issue, #8 is already taking long to load on my phone.

Quite the opposite. The situation is the data is sent sometimes without a command. For example, there are commands that you send and the UI thread waits for a response. Then, depending on the response you send more data or just do nothing.

But why wouldn't that work with two enums? Call the enums command, response, message or whatever, the receiver can match the result and send whatever it wants in reponse (or nothing at all).

There are more responses that commands. There are currently 55 commands, and between 50-100 data types that are send one way or another.

That is a lot indeed, but not too much imo. Which commands have different responses?

Also, there's another problem related to merging the channels. When I started the 0.9 version and the Qt port, I set a goal: the UI MUST NOT HANG. To do it, RPFM currently makes the UI thread to enter on a loop that keeps processing events (like resizing the window) until it gets a response from the background thread.

That is completely reasonable! A few cents from what I noticed while browsing your channel code:

• shouldn't you phase out check_message_validity_recv for the non-blocking variant?
• why don't you loop within check_message_validity_tryrecv? Right now rpfm contains the same loop 22 times
• check_message_validity_tryrecv and check_message_validity_recv_background do exactly the same (?), can't the former be a wrapper around the latter?
• you might want to move the ui and bg code into separate files. It helps matching IDE search results to functionality (at the moment I have to check whether a usage of foo in main.rs is in the bg thread or ui thread), and main.rs is almost 6k lines of code long - that is really a lot

That means you can potentially send a command while there is another one executing itself, making the entire program crash.

Yes and no. Yes, at the moment you can, but you could change that at any time by not sending a new command when there is a pending one on the "wire".

In earlier 0.9 versions, you could crash the program by keeping pressed the "Del" button because the deletion process got messed by a checking process, and message were received in the wrong place. To "fix it", I left a channel for commands only (so commands can be chained and don't interfere with one another) and another for data, and every time data received, RPFM checks that's the type it needs using generics and crashes if it's not .
Merging them will mean the commands can potentially crash in many more places that before.

Are you sure that was what fixed the issue? If process_events allows your event handlers to execute, even with a dedicated data channel you should not send more than one commands at once. Imagine we have the commands A0, A1 and B. All three return data, A1 will only be sent after A0 finishes, and A0 and B are sent by user interaction.

Now

• the user initiates A0
• the A0 handler is in a tryrecv-loop, so the user can trigger new commands
• the user initiates B

The command channel contains: B -> A0
The bg->ui data channel contains: nothing

Now

• the bg thread handles A0, puts a0-data in the bg->ui channel
• the ui thread handles a0-data, and puts A1 into the ui->bg command channel

The command channel contains: A1 -> B
The bg->ui data channel contains: nothing

Now

• the bg thread handles B and puts b-data into the bg->ui channel
• the ui thread expects a1-data but gets b-data 💣 💥

Mmmm.... I'm not sure about the second and third one. The way I see it, the same problem with wrong messages being sent can also happen with enums, because you can receive the wrong variant and the program will have to crash, to avoid weird behavior.

You surely still can send the wrong variant, but it does stop you from putting a wrong data type into the correct variant! Makes reviewing code much easier, since you can be sure that a code change which does not touch the variant handling/sending panics because the channel is disrupted.

And the fifth one.... if you want data to be shared in an enum without deleting the original data (because the threads are isolated), you have to copy it anyways isn't?

Yes and no! Threads are not isolated, rust encourages and enables you to share data in completely safe ways. You don't have (at least without unsafe) the freedom of other programming languages, but if you prove the compiler that what you are doing is safe it lets you do it.

So one the one hand yes, to send or share a type between threads it must implement Send or Send and Sync, and those types in the stdlib internally copy themselves when you move them into a closure.
On the other hand no, you do not have to copy the data. To share a Vec<u8> between threads without copying the vector, you can move it into an Arc and move a/multiple clone(s) to the/some other thread(s). An arc is just a few bytes in size which can be easily cloned, the contained type will exist only once in memory and be dropped as soon as all arcs which point to it are dropped.

Hi there,

here my step by step order to reproduce:

1. Open mod
2. Edit script
3. Save mod: get save confirmation from RPFM
4. close mod
5. reopen mod: previously made changes are not saved!

Thanks for you hard work!

And I have no idea why. Needs investigation.

Stuff needed to set this as "Done".

• PackFile Logic.
• SupportedGame data.
• GameSelected code.
• Napoleon Definitions, to reduce decoding time. Depends on #32.
• Schema Definitions.

Stuff needed to set this as "Done".

• PackFile Logic.
• SupportedGame data.
• GameSelected code.
• Schema Definitions.

Otherwise a yanked (or non-backwards compatibly updated) (recursive) dependency will break builds on new git checkouts

This looks like a Qt/Windows issue, as I just call the default function from Qt, and this happens. Needs some more investigation though.

This is a bit complex because there are a lot of fields that may or may not have values, so we can only apply it to those we're sure always have a value (StringU8/U16, not OptionalStringU8/U16). For numbers is already done.

Some tables (very few, like sea_surfaces) actually use more than 3 decimals on some cells, and RPFM defaults them to 0.000.

The problem of removing the limitation comes with rounding. Currently RPFM rounds to the third decimal to remove weird precision issues like getting 0.4999999995 instead of 0.5. Removing the limitation means we need to find another way to get rid of the precision problems.

There is probably a bug in the .tsv or .loc parsing logic somewhere. The following bug can be reproduced with the NordoGerman mod for Rome II, which contains a .loc file with a multitude of translations.

I can open the pack file with RPFM and export the .loc as .tsv. But when I reimport the same exported .tsv file immediately afterwards, the parser complains with the following error:

This TSV file has an error in the row 4747, field 2 (both starting at 1). Please, check it and make sure the value in that field is a valid value for that column.

Which is strange because it had no problem opening and exporting the same file before.
The respective row contains the following string: (The [TAB]s represents a Tab whitespace)

event_log_descriptions_event_content_diplo_war[TAB]%S - %S[TAB]true

Maybe there is a problem with the leading % symbol in the second column? Pack File Manager 5.0 can handle this exported .tsv without a problem and reimports it successfully (Apart from stripping all leading " like it always does -.-)

This occurs on the newest version 1.2.2. of RPFM. I hope you can look into it, as I much prefer to edit .loc files with RPFM if it weren't for this bug. Thanks in advance and keep up the good work!

Really sorry if this seems noobish or anything, but how do I install this tool?

I´m a small time modder and I´d love to try out the new Attila compability,
but I haven´t been able to run this at all. Can´t seem to find any executable.

Is there a specification somewhere for the .pack files that I can look at? So as to better understand this program.

I know that loading pack files takes up a lot of memory and was wondering if there's a way to interpret the data in the pack files lazily, like if a section regarding character traits has an identifiable byte range that could be used.

I haven't been able to get the application compiled on my windows machine yet, but I aim to help out soon! Awesome project, by the way - excellent way to learn rust just by the sheer breadth of it.

If you want to fix that within rpfm's pack module you have to

• stop trusting the sizes you read in the header
• ensure the payload-position variable does not overflow (twa_pack_lib uses checked_add)
• enforce an opened file you lazily read from is not modified
• enforce open_dependency_packfile is using a valid file, or stop using unwrap() all the time

For the sake of simplicity, I'd also recommend you move all pack-related filesystem usage into the same module.

We need to map a key press for this. Not sure how to do it with the current Qt Bindings.

Linux paths are... diferrent. Feral ports have their data in multiple separate subfolders instead in a common "data/" folder. So we need to expand the path system to admit multiple ones, and use X or Y paths depending on what's in their base folder. So far we got:

• Windows: all games use the path game_folder/data.
• Linux: depends on the game:
  • Warhammer 2: Feral Port. It uses game_folder/share/data/data for the base game's data.
  • Warhammer: No idea.
  • Thrones of Britannia: No idea.
  • Attila: In-House Port. No idea.
  • Rome 2: No idea.
  • Shogun 2: Feral Port. It uses game_folder/share/data/data for the base game, and game_folder/share/data/fots for FOTS-Specific PackFiles.
  • Napoleon: No idea.
  • Empire: No idea.
• MacOS: I don't have a Mac to test, so no idea, but in theory if this gets fixed there are not many problems to build RPFM for MacOS and release a Mac version.

The tables are encrypted indeed, but I think I managed to get the key. You can use twa_table_decrypt or twa_table_decrypt_lib to decrypt them, but it could be that the ends of the decrypted tables are a bit off because I didn't understand the paddings correctly.

Do you think the decrypted tables look ok? Naturally it would be great if rpfm would display the tables, so I assume we have to create a schema?

Someone asked for this, and it's not a bad idea to make it work better with scripts and all that stuff, but needs some planning and investigation on how to do it.

The sorting/filtering code sorts and filters by text. These columns have no text. So we need to take a look at the code and change it for these columns.

This require us to use a trick in the FileDialog that I don't know if it'll work on Linux and Windows, as for those RPFM defaults to the native FileChooser.

This requires subclassing the filter widget to get it filter only by the last item in the parent-child chain. Because currently it filters by any item in that chain, so if one of the parents doesn't match the filter, it hides him, and everyone of his children, no matter if they matched the filter or not.

Basically, we can use this to filter parts of a table, then wipe out everything not in screen from that table, so what we see is what's left. Useful for reducing tables.

The idea is to make it into a more complete editor. Features to add:

• 4 spaces tabs.
• Custom font selector (optional).
• Syntax Highlighting (very needed).
• Auto completion (at least for LUA).

using carg run in cmd works fine, but not in powershell:

PS D:\repositories\rpfm_vanilla> cargo run
   Compiling rpfm-code v0.9.4 (file:///D:/repositories/rpfm_vanilla)
warning: unused `std::result::Result` which must be used
    --> src\main.rs:4867:25
     |
4867 | /                         packfile::update_packed_file_data_rigid(
4868 | |                             &packed_file_rigid_model,
4869 | |                             &mut pack_file_decoded,
4870 | |                             data.1
4871 | |                         );
     | |__________________________^
     |
     = note: #[warn(unused_must_use)] on by default
     = note: this `Result` may be an `Err` variant, which should be handled

warning: unused `std::result::Result` which must be used
    --> src\main.rs:4890:33
     |
4890 | /                                 packfile::update_packed_file_data_rigid(
4891 | |                                     &packed_file_rigid_model,
4892 | |                                     &mut pack_file_decoded,
4893 | |                                     index
4894 | |                                 );
     | |__________________________________^
     |
     = note: this `Result` may be an `Err` variant, which should be handled

    Finished dev [unoptimized + debuginfo] target(s) in 54.12s
     Running `target\debug\rpfm-code.exe`
error: process didn't exit successfully: `target\debug\rpfm-code.exe` (exit code: 3221225595)

Sorry if I have in any way overlooked things, but will this support older titles like Rome 2 or Atilla?
I mostly work on these.

Splendid work btw!

PFH5 PackFiles use LZMA for PackedFile compression, but they change the header with a non-standard one. So we need:

• Test different compression levels and see if they work ingame.
• Allow compression per PackFile.
• Allow compression per PackedFile (optional).
• Make the UI show somewhere if a PackedFile is compressed or not.

This is a list of possible suggested improvements to make editing stuff faster.

• Ctrl+Delete/BackSpace for deleting everything up to the last/next space or underscore.
• Keyboard Scrolling in all four directions without deselections.

What do I need to support warhammer I?
is it a matter of new json schema?

RPFM sorts them alphabetically, not using the numbers. For example, RPFM will consider 10 to go before 2. This needs some investigation, as I have never touched the sorting code.

While investigating TWA's tables I noticed that some packs were not properly unpacked. Luckily I managed to track the problem down: If 1 << 4 is set in the bitmask, every item in the payload starts at a multiple of 8.

twa_pack_lib fix: TotalWar-Modding/tw_pack_lib@c549376 (unfortunately rather big because of major simplifications)

There is a little boolean there whose use we don't know. This calls for an investigation. Not really affecting how the tables work currently, but I don't like "unknown" parts in the code.

This is confirmed to happen after you add tables to a PackFile, then open them. Something wrong is going on in the function that detects the folder state in the UI.

Describe your problem
Since the newest update, whenever I try to "Install" one of my mods using the install button in the MyMod menu, RPFM crashes completely. This kills the modding workflow significantly, as I have to move files manually to the /data folder to test and/or upload them. I'd appreciate if this could be looked into.
Apart from that - I love the dark theme!

Extra info?
Version 1.3.2
For me this is reproducible with all of my MyMods.

Some Text PackedFiles currently have BOM, shown as ÿþ, or FF FE at the begining of the file. We need to detect that and hide it, so the user can't see it, but RPFM reads it and conserve it when saving.

NOTE: This is specially visible in older games, not so much in newer ones.

Schema definition is missing for campaign_maps_tables in warhammer schema

RPFM v1.2.0 with latest schema updates and I also verified that it's missing in the master branch.

edit: realized I forgot to mention which schema though I'm unsure if older games implemented the table

Describe your problem
I don't want to annoy you or anything but the install function is still buggy. :) I get the following error when trying to use it on windows:

Error while trying to copy one or more files to the following folder:
"Z:/SteamGames/SteamApps/common/Total War Rome II\data\wolphyx_medals.pack"

Looks like a filepath issue, as the slashes and backslashes are inconsistent. I tried changing them to backslashes in the settings.json, but that just caused RPFM to don't read any settings at all anymore.

Extra info?
Occurs on RPFM 1.3.3 when using the install function

The Dark Lord is not pleased with:

• Checkboxes are almost invisible if they're not checked.
• Blue text it's hard to read.
• Disabled text is hard to read.

As the title says. This requires #40, as it's a change people may not want.

Just a simple option in the Right-Click menu of the PackFile.

This module is the only one still using the old code for decoding/encoding, and Phazer has done quite an extensive work decoding the files so... we're going to rewrite the module to be up-to-date with the other modules, and to use Phazer work to get more usable stuff. This means we need:

• Port Phazer logic code to Rust (or use his lib, which will mean harder maintainability).
• Get the module up-to-date with the other modules (revised error checking and corrupted file detection).
• Rewrite the UI to show far more data than it's shown now.