ortham / esplugin Goto Github PK

Source. Even if the file doesn't have them set, the master flag will always be set at runtime for .esm and .esl files, and the light master flag will always be set at runtime for .esl flags. esplugin should be updated to reflect this.

The valid range was previously 0x800 to 0xFFF. HashedFormId::valid_in_light_master() should be updated to use the new range. The version field in plugin HEDR subrecords is now 1.00, so for accuracy it's probably a good idea to only use the newer range when checking subrecords in a plugin with a version >= 1.00. FormIDs don't currently store the version, but there are three padding bytes spare that could be used for this.

Reported by Sharlikran, Infernio and Arthmoor on LOOT's discord server.

The FormID struct is really inefficient, as each FormID in a plugin copies a plugin filename, and there can be hundreds of thousands of them. FormID could instead keep a reference to a string, though plugin filename and master strings don't currently live long enough for that to happen.

It might be better to store FormIDs as u32s, then construct FormID structs only when calculating the intersection with another plugin's FormIDs: references to the plugin filename and master strings could be obtained when this function is called. However, in LOOT a plugin's FormIDs will be compared against many others, and while constructing a FormID would only involve a bitshift, a bitwise AND and a vector index lookup, inserting into a BTreeMap involves a case-insensitive string comparison (for conflicting FormIDs only) so creating the collection may be relatively expensive.

If so, it might be better to cache the FormID objects, but then that will also need the plugin filename and master strings to be stored, which is messier but not significant overhead (especially compared to the current situation).

It currently takes 1 ms to parse Hearthfires.esm's FormIDs and another 4 ms to create a BTreeSet of FormID structs. Checking for overlap with itself takes another 0.3 ms.

esplugin doesn't currently do this, and there's a crash in LOOT that I can't replicate but which is probably due to a panic in libloadorder or esplugin.

I'd like to automate some tedious morrowind modding.

Namely i want to do a mod to turn every spell scroll into a (actual) new spell.

pseudo code:

for every esp or omwaddon:
     for scroll in 'sc_name' (added or override):
          if(scroll has enchantment and doesn't have a script) (it's a normal spell scroll)
               name_suffix = scroll.getName()[10..] ( 'Scroll of [name_suffix]')
               create new spell with name [Scribed [scroll_record]] and effects and costs the same as enchantment
               create script to learn the spell on equipping of the scroll
               add script as override for the scroll
               (scroll override, new spell) to overrides_list

      if (overrides_list not empty)
          create new compatibility omwaddon with overrides:
      

Anyway, this needs both a write capability and at least being able to get the names/enchantment/effects/costs of records. Is the project planning write capability? And if not, you know of a way i can program it? And reading these properties?

Starfield plugins encode their record FormIDs differently to previous games.

In previous games the first byte of the FormID is equal to the index of the master that the record came from, with masters indexed in the order they appear in the plugin header. If the first byte's value is larger than the index of the last master, that indicates that the record is newly added by the current plugin.

In Starfield, the masters' medium and light/small flags are taken into account when the record FormIDs are written, so full plugins follow the same behaviour as in earlier games, but medium and light plugins are represented like they would be in-game. For example, for a plugin with the following masters:

Blank.full.esm (a full master)
Blank.medium.esm (a mediummaster)
Blank.small.esm (a smallmaster)

that overrides a record with object index 0x806 from each of those masters, and adds another record with the same object index, the FormIDs in the plugin will be:

00000806 -> Blank.full.esm's record
01000806 -> this plugin's record
FD000806 -> Blank.medium.esm's record
FE000806 -> Blank.small.esm's record

If there are more full, medium or small plugins then their respective index is incremented:

• if there was a second full master, its overridden records would start 01 and this plugin's would start with 02
• if there was a second medium master, its overridden records would start FD01
• if there was a second small master, its overridden records would start FE001

If the plugin is converted to a small or medium master, the records it adds do not start with FE or FD, they start with an index that is unused by any of its masters, same as if it was a full master. This is partially wrong: that behaviour is only observed until the plugin is reloaded in CK, after which new records in a medium plugin are loaded (and then saved) with FormIDs starting FD, and in a medium plugin they're loaded and saved with FormIDs starting FE. This seems like a bug in the CK.

The CK will compact any records outside the valid range for a medium or small master if you try to convert a plugin to one of them, and that compaction includes any overridden record FormIDs, breaking them (after compaction they look like new records added by the current plugin). That means it's seemingly not possible to have a small/medium master that overrides records added by another small/medium master.

This means that it's not possible to tell which plugin a record in a plugin comes from if it has one or more masters without loading those masters to check what flags they have set, which affects checking if plugins overlap and also checking if a record is a new record or not (which impacts checking if a medium/small/override plugin is valid).

Sorry, I created a rust project, added the esplugin 3.4.0 dependency. But I don't understand how I can take any values from esp by FormID

My main.rs file

extern crate esplugin;

use esplugin::{GameId, Plugin};
use std::path::Path;

const PLUGIN_TO_PARSE: &str = "plugins/HIVE.esp";
const GAME_ID: GameId = GameId::SkyrimSE;

fn main() {
    let mut plugin = Plugin::new(GAME_ID, Path::new(PLUGIN_TO_PARSE));
    plugin.parse_file(false);
    let c = plugin.count_override_records();

    println!("Hello, world!");
    println!("{}", c);
    println!("{}", plugin.path().display());
}

Can you help me in discord or show examples.
Or am I just initially using your library incorrectly

A new Skyrim SE update has expanded the range of supported light plugin FormIDs to 4096.

The new plugins _ResourcePack.esl and unockatja.esl use the increased range and so currently appear invalid. _ResourcePack.esl has a HEDR version of 1.71, while older ESL plugins have a version of 1.70, which may be related (I don't have unockatja.esl to check it).

libloadorder takes about 800x longer to fully parse a load order consisting of Skyrim.esm and 20 copies of Update.esm than it does when just parsing headers. It would be great if full plugin parsing was (much) faster, and in libloadorder's case it just cares if the file format is self-consistent (i.e. structure sizes are correct, actual data could be nonsense), so it might be worth offering a parse-and-discard mode.

The first step is adding some benchmarks though.

Hello! Thanks for the work on this library.

I can't run the tests locally because they rely on a bunch of plugin files which are not in the repo. Have you considered adding them here (it looks like most / all of them are not actual game files, so they should be fine to include)? Or providing the files in another place where they can be downloaded, so it's possible to test the library locally?

As #5 indicates, memory-mapped parsing of headers is slower than just opening the file normally, and I've verified this is true even for very large headers. It seems that there's a data length at which mmap becomes a better choice, and it would be better for esplugin to pick when to use mmap based on that, rather than exposing that detail as part of the API.

The mmap API is labelled as unsafe, but that's due to a very conservative reading of Rust's safety guarantees, because the underlying file could change while it's memory-mapped. However, attempting to open a writable File when a MMap exists or vice versa causes I/O errors (at least on Windows), so I don't think exposing the unsafety of the API is helpful, as the Rust code can't be unsafe itself.

esplugin doesn't currently differentiate between Skyrim and Skyrim SE, but only the latter supports ESL files, so add a game ID for it and extend ESL support to cover it.

In a FormID 0xXXYYYZZZ stored in an ESL file only the ZZZ value is the object index, the rest is used for load order indexing. This will need to be taken into account by esplugin when creating FormID objects and checking for overlap between plugins, because FormIDs normally use 0xYYYZZZ as the object index.

Needed for loot/loot#816.

I've got no idea if this is a bug or if the plugin is invalid, but I'm sure I used to use this plugin back in the day. It's available for download here.

If the plugin isn't actually invalid, this issue is about fixing whatever is causing esplugin to think it is. If the plugin is invalid, then this issue is about improving esplugin's error reporting to include detail about what's invalid where.

esplugin slices vectors at a few points, but doesn't check that the bounds are valid, which risks panicking if the subrecords aren't big enough. The affected functions are:

• Subrecord::decompress_data
• Plugin::has_extension
• Plugin::description
• Plugin::header_version
• Plugin::record_and_group_count