textools / xivmoddingframework Goto Github PK

The presence of such a temp file prevents textools from being able to recover via the help menu.

Recovery could first purge the temp file and perform the cleanup again, and offer to start over if during the cleanup phase any errors are encountered when offsets are validated (assuming they are).

Should use Path.GetDirectoryName instead of substring in https://github.com/TexTools/xivModdingFramework/blob/1235f17/xivModdingFramework/SqPack/DataContainers/IndexFile.cs#L525

Doc: https://docs.microsoft.com/en-us/dotnet/api/system.io.path.getdirectoryname?view=netframework-4.7

The new Fat Cat Loungewear is the games first Body+Hands gear, and is not being handled correctly, the item is listed under Category "Unknown" and displays no materials or model as it defaults to dwn.

In the following code, shouldn't

uncompLength = dealing with last part ? mipLength % 16000 : 16000

have been

uncompLength = dealing with last part && mipLength % 16000 != 0 ? mipLength % 16000 : 16000

instead? Otherwise, the last part of each mipmap would end up being omitted.

This mod has consecutive sqblocks where (size=16b, unk=0, compressed=0 decompressed=16000), and while I'm not sure if the above is the cause (can't think of a reason why would these empty blocks be placed consecutive), thought that it might worth sharing this issue.

Expected behavior?

Enter the name of the partition and load the model and textures.

Actual behavior?

Loading either the model or textures gets you an error window.
"There was an error reading the MDL file."
"Source array was not long enough. Check srcIndex and length, and the array's lower bounds."
*Indexes are fine and confirmed with two different users the item will not load correctly.

Steps to reproduce the problem

1. Open Textools 2.3.8.2
2. Search: Blank Hingan Partition
3. Click on the item.
4. Crash.

...

Error messages?

What aspect of FFXIV TexTools were you using, and what Item were you interacting with?

General preview of item.

Has this has been confirmed as an actual bug?

• This has been confirmed to be a legitimate issue and not caused by bad game data.

Racial Model export (for non existent) racial models I suspect is simply a function of matrix math - as we've not actually done that yet, as I'm aware of.

The general function should be:

• For each bone, starting from root, working down the tree (order may not actually matter)
  • Apply the inverse of the current race bone transformation matrix.
  • Apply the normal target race bone transformation matrix
    • In some fashion apply the appropriate weighting based on bone weights for the vertices.
      • I'm pretty bad at matrix math, so I'm not sure if this is correct, but I'm pretty sure this should just be a multiplier based on bone weight and final position of the bone compared to original position -- As far as vertices are concerned, bone manipulation in terms of individual bones is rotation/etc. agnostic - only the final position matters.

These transformations need only be applied to the position coordinates of each vertex, indices and other values are by nature unaffected.

The current shared model check function only scans the known item list for material variants. It should also check the IMC file for unused IMC variants and add those into the list as dummy NPC items.

With the introduction of the TTModel class and SQLite DB output, we now have more options to manipulate and output Shape data. A question or two needs to be answered first though, then some extensive work done for sorting out the I/O of the data.

1. Does SE actually use the Shape Data for anything other than Position data?

   • If not, we can likely simply read the Shape Data in/out as Morpher channels after #3 is completed.
   • If they do use the other channels, we have to output the data as totally separate Meshes.

2. Reading Custom Data back in:

   1. Vertices and Triangle Indices for the incoming data must match 1:1
   2. Rip through the Vertex list, and copy out any Vertexes that changed in any way.
   3. Write those modified Vertices to a new ShapePart
   4. Find all the Indices that refer to those modified Vertices and create a list of [Index ID => New Vertex ID]

This is however complicated by the fact that Shape Parts are stored at the Mesh Group level, not at a Per-Part level. This means we need to further...

1. Slice up Shape Data on export/import to/from the SQLite DB format, into references based on the constituent parts, rather than Mesh-Group level references.
2. Generate the Morphers/Sub-meshes on a partwise basis.

End 3DS max Scene would likely look like...

• [Group 0]
  • Mesh_0
  • Mesh_0.1
  • Mesh_0.2
  • [Shapes]
    • Shape_0.1
• [Group 1]
  • ...

Hi! I'm your friendly neighbourhood developer who works on tools that can use TTMPs (namely Heliosphere)!

I am curious as to what led to the adoption of using sqpack internally for modpacks and if it can be removed in a new TTMP version. In terms of pros and cons, I can only think of cons, so I'd love to learn more about why it's in use.

However, as far as I understand, even for modifying the game files, TexTools can take tex files, models, etc. and directly modify the game files without going through an intermediary sqpack first. This makes it appear to me that there is actually no reason to continue storing modpacks in sqpack! Again, if I'm wrong, please let me know.

However, for anyone who makes tools working with modpacks, sqpack is an absolute curse for them. There's no single point of truth, no spec, and every implementation of the format will be slightly different and cause incompatibilities. This creates a high barrier to entry for any tools wanting to work on modpacks, as they must first have/create a sqpack decoder! In addition, TexTools does several things wrong when writing sqpacks (which I can't blame whoever implemented - it's a tricky and annoying format), which leads to even a "correct" reader having to make itself work differently to accommodate.

For users, sqpacks generated by TexTools have poor compression (DEFLATE on 16kb discrete chunks - also bad (de)compression time, which leads to poorer UX) and exact duplicates of files (making the size problem worse). They're also completely opaque; the user can't see (or edit) what's actually in the modpack without TexTools.

It would be nice if TTMP3 could be a normal ZIP file or like tar.zst or anything standard. Take a look at something like what Penumbra uses (PMP). It just has the files in folders and has a JSON file pointing to them and which game path they affect. This is much more approachable for developers and users. It should also be easier for TexTools, since it can completely drop the use of sqpack, have fewer bugs with modpacks, and get better compression. It could even deduplicate files (although TexTools could be doing that with sqpack anyway) and get a better compression ratio. If TTMP3 used something like a ZStandard-compressed tar file, it could achieve great compression with really fast decompression, improving UX!

To someone like me who didn't make TexTools, I can't see any reason that it uses sqpack, considering that alternatives like Penumbra don't need it at all. However, there could be things I don't know, so I'd love to have a conversation if that's the case. Either way, working to remove sqpack from modpacks could help everyone in the modding scene.

We need to introduce binary file injection/partial file support for files. The current thought on how to best do this is as follows.

1. Bump TTMP Version
2. Enable <file_path>:: as the file path for a given mod entry to signify it's only changing partial data.
3. Mod List entries need to keep a copy of the original binary injected data inside them for enable/disable purposes.
4. A single unified modified file should be kept for the binary file that is being (partly) overwritten.
   • This entry in the modlist should use a pre-specified hard coded source, ex "FRAMEWORK_INTERNAL"
5. When FFXIV is patched, clone the original file again, then re-play the binary edits onto it.

Binary Files on consideration for user editing:

• EQP - 64 bytes per item set, 8-24 contiguous bits per item slot.
• EQDP - 16 Bytes per item set, 2 contiguous bits per item slot.
• IMC - 6 bytes (non-set) or 30 (set) bytes per item set, 6 contiguous bytes per item slot.

Things to consider:

• If a binary file injection is requested that overwrites some part of another entry, that entry should be disabled, and the original data restored, before writing the new injection.
  • When disabling the injection after, do not re-enable the old one, just go back to vanilla to avoid potential complications?

Now that we have an SQL Cache to store the data in, and users creating custom file paths in FFXIV's file system, we need to create a proper file dependency tree to make mod and modlist exporting and validation sane.

In primary, two functions are needed:

• List GetChildFiles(string internalPath
• List GetParentFiles(string internalPath)
• Maybe - List GetSiblingFiles(string internalPath)

These functions need to handle all the resolution necessary to identify all the children/parents of a file. To this end, the Cache should store the downward facing/child files of each file, so that upward scans can become possible via checking the cache.

Top down at each level dependency search for items should look like:
Real files are in [], folder structure helpers are in italics

Item Set =>Slot => [EQP]/[EQDP] Binary Entries => [MDL Files]/[IMC Binary Entries] => [MTRL Files] => [TEX Files]

Each entry at each level is guaranteed capable of resolving both its own downward facing and equivalent level entries easily given no data other than only its path::binary_offset, as follows

• Item Set - Defines 5 static slots, based on if it's an Equipment or Accessory Set. Used as Offset multiplier for EQP/EQDP entries.
• Slot -> Defines which sub-offset to use for EQP entries
• [EQDP Entry] -> Defines what Racial MDLs exist for the item, Can resolve EQP from offset information.
• [EQP Entry] -> Defines part hiding for the item; from its own offset can extract what Set and Slot it is to find the EQDP entry and go down from there.
• [IMC Entry] -> Defines MTRL folders, can access the MDL file by studying its own file path to determine EQDP offset.
• [MDL File] -> Defines MTRL names, can access the IMC file by studying its own file path.
• [MTRL Files] -> Defines explicit internal paths to TEX files.
• [TEX Files] -> End of chain.

Upward facing entries are more complex, ergo why we cache things.

• [TEX File]
  • A default TEX file can always resolve its parent Equipment Set, and Slot, but must chain-walk down from there to determine which MTRL files actually reference it.
  • A custom path TEX file will only ever be referenced by Modlist Entries.
  • Both Default and Custom path TEX files must be checked against all modified MTRLs to see if they reference it.
  • TEX files are the ONLY files (besides skin MTRLs) that can be referred to/accessed by an item that is not in the same Item Set.
• [MTRL File] - Owning MDL (race+slot) can be identified as part of the file name.
  • Though SE never does it, it's possible for an MDL to reference MTRLs of other slots in the same set & Variant. Worst case search goes up and re-crawls down through all MDLs in the set to check, but we can cache this information for modded MDLs instead.
  • Skin MTRLs are used extremely extensively and building reference lists for them is not feasible/useful.
• [MDL File/IMC Entry] -> Filename/Offset contains Item Set/Slot information, which can be used to resolve the EQP/EQDP entries.
• [EQP/EQDP Entry] -> Set and Slot can be resolved based on their binary offsets.
• Set/Slot -> By definition set is always included with slot. Set on its own is top of the chain.

Not entirely related, but an in-game FFXIV "Item" is defined as a [Set + Slot + IMC Variant] (+ player racial information)
Ergo the game process of resolving an item is
-> Resolve Set
-> Resolve Slot
-> Check EQDP Entry for Racial MDL to Use
-> Check IMC Entry for MTRL Folder
-> Check MDL for MTRL File Name
-> Check MTRL for TEX to use.
-> Check EQP & IMC for part hiding information.
-> Render Model

The [Set + Slot + IMC Variant] information for each equippable item in game is defined in Items.exd - Changing this data however does NOT change the visible model/etc. in game because the data is actually supplied directly via the server in the packets to the client when a new model appears nearby. The client side Items.EXD version is only checked when using the Preview/Try on windows (which do not interact with the server)

Now that MDL writing is slowly getting more sane, it likely makes sense at some point for us to just completely remove the higher LoD entries from the MDL file on import. This has a few benefits.

1. Save file space.
2. Renders the default LoD0 at all ranges, even if the LoD option is turned on, rather than rendering the current broken messes/missing meshes.
3. Renders Shadows with LoD0 even if the Render low LoD shadows option is turned on.

while using xivModdigFramework to export some furniture to .obj I got lots of exceptions saying "Source array was not long enough. Check srcIndex and length, and the array's lower bounds". Problem seems originated from bad offset configuration. Some models can't be read normally, such as "Grade 3 Picture Frame" or “Tier 3 Aquarium”. And a lot of models have texture linking problems such as "Indoor Oriental Garden". Only those has only one texture or meshgroup things seems to be correct and normal.

Textools also impacted by this issue. These things I mentioned above could not be opened in textools either, and those have texture linking problems were displayed as black squares.

Actions list was previously sorted into multiple blocks so that as many actions were listed as possible,

after 6.4 Target to Attack 1 has moved from nearer to the bottom to right up top, and Target to Bind 1 is offset to the right within the list, suspected change to offsets and datalength increase required. Resolved in 6.4 fixes branch so far.

Online Status has a similar issue, the list no longer in the arrangement it was previously, and "Playing triple triad" offset center, also suspecting offset and datalength change required. Resolved in 6.4 fixes branch so far.

New raid related earrings (anabaseios/ascension) cannot be loaded, due to the material being beyond the readable stream.

This is because the material set appears to either be set wrong in the game data, or SE has intentionally started using Diffuse/Specular for some types of Earrings.

This is a game bug, the game data includes Normal/Multi for these earings, but the material set is configured incorrectly for Diffuse/Specular.

Status list doesn't populate at all, no fix found so far.

This leaves the Dark Meteion body unviewable, and results in Problem Checker aborting and unable to check the entire modlist for issues.

Introduction of FBX Support

• Importer is stable now.
• Need to update Export pipeline to use TTModel format.
• Need to introduce [Save to File] function for TTModels
• Need to create FBX Exporter to take DB files and generate FBX files.

The current JSON modlist files should be converted to SQLite or other, actually efficient datastores. With users sometimes reaching over 20,000 active mod entries, reading and writing a potentially 10mb+ json file becomes exceedingly slow.

Additionally, we should take the opportunity to stop storing disabled mods in the DAT files, and create a proper Modpack Ordering system potentially.

Some analysis will be necessary to determine appropriate structures and UIs to make the system function smoothly and match user expectations.