ortham / libloadorder

A cross-platform library for manipulating the load order and active status of plugins for the Elder Scrolls and Fallout games.

Libloadorder

Libloadorder is a cross-platform library for manipulating the load order and active status of plugins for the following games:

• TES III: Morrowind
• TES IV: Oblivion
• TES V: Skyrim
• TES V: Skyrim Special Edition
• TES V: Skyrim VR
• Fallout 3
• Fallout: New Vegas
• Fallout 4
• Fallout 4 VR

This repository hosts two Rust crates: libloadorder is the Rust library, and libloadorder-ffi is the C FFI that wraps it. The doc directory also hosts an mdbook that provides a general introduction to load orders.

To build libloadorder and its C FFI and generate C/C++ headers for it, install Rust and run cargo build --release --all --all-features.

Tests

The tests require testing-plugins, put them in testing-plugins in the repo root.

Run cargo test and cargo bench to run the Rust tests and benchmarks respectively.

To run the FFI tests, make sure you have CMake and C and C++ toolchains installed (e.g. MSVC on Windows, GCC on Linux), then create a directory at ffi/build, then from that directory run:

cmake ..
cmake --build .
ctest

libloadorder's Issues

Use not equal comparsion for timestamps

Currently timetamp comparisons check if the file timetamp is newer than the cached timestamp, but it is possible for the timestamp of a file to be changed so that it is older, as identified by @Utumno in wrye-bash/wrye-bash#195.

The simplest fix for this is to use a not equals comparison operator rather than a greater than operator.

set_load_order should not add additional plugins to the load order

Changing this is a breaking change, but it's a potentially unexpected side-effect, which should be avoided. It's done so that the user doesn't end up with a partial (and so possibly invalid) load order, but a better way of handling this would be to just error if passed an incomplete load order.

Thread safety

Would be good to have. Need to read up on what this entails.

fix_plugin_lists misses some fixes

The stack overflow I reported: wrye-bash/wrye-bash@b79a4c2

is still present in your latest iteration: ddb53c7

Reason:

Calling the method again in the except block is asking for trouble anyway

EDIT: second SO:

In this case it is a misordering of mods and masters - especially in this case the mod was inactive

EDIT: edited the title to reflect the liblo rather than the Bash issue. Real fix should be to add to get_** methods an output parameter which would return the fixed list - the list being fixed while being checked in checkvalid() rather than repeating the ifs in a different method. Moreover the invalid lists are of interest to the client (warn the user, make decisions on how to fix etc) - so the model of the library binding should be return valid, invalid and let the user decide (or even return invalid, warning - where invalid should mean actual - more on this later) - rather than 'throw on error then call fix'. Especially the masters test should be dropped from LO as it was dropped from active - it even fails on inactive mods and that's not what the library should worry about anyway - it is the utilities role to display such inconsistencies to the user and let him decide

Add support for ESL files

Needed for loot/loot#816.

ESLs have been patched into Fallout 4 and Skyrim Special Edition, so they're an update to the Asterisk load order method for the purposes of libloadorder's implementation.

Libloadorder needs to:

• Recognise .esl files as plugins
• When inserting .esl files, treat them like master-flagged plugins.
• Recognise the new implicitly-active ESL plugins. They are loaded in alphabetical order after the existing hardcoded plugins and before masters and other ESL plugins.
• When checking the active plugin limit, exclude ESL plugins.
• Allow up to 4096 ESL plugins to be active at any one time. The limit apparently varies with the number of records in the plugin, but the reason/relationship is unknown, so maybe libloadorder should just treat 4096 as the limit in all cases.

As of Fallout 4 v1.10.26.0.0, the implicitly active ESLs are:

ccBGSFO4001-PipBoy(Black).esl
ccBGSFO4002-PipBoy(Blue).esl
ccBGSFO4003-PipBoy(Camo01).esl
ccBGSFO4004-PipBoy(Camo02).esl
ccBGSFO4006-PipBoy(Chrome).esl
ccBGSFO4012-PipBoy(Red).esl
ccBGSFO4014-PipBoy(White).esl
ccBGSFO4016-Prey.esl
ccBGSFO4017-Mauler.esl
ccBGSFO4018-GaussRiflePrototype.esl
ccBGSFO4019-ChineseStealthArmor.esl
ccBGSFO4020-PowerArmorSkin(Black).esl
ccBGSFO4022-PowerArmorSkin(Camo01).esl
ccBGSFO4023-PowerArmorSkin(Camo02).esl
ccBGSFO4025-PowerArmorSkin(Chrome).esl
ccBGSFO4038-HorseArmor.esl
ccBGSFO4039-TunnelSnakes.esl
ccBGSFO4041-DoomMarineArmor.esl
ccBGSFO4042-BFG.esl
ccBGSFO4043-DoomChainsaw.esl
ccBGSFO4044-HellfirePowerArmor.esl
ccFSVFO4001-ModularMilitaryBackpack.esl
ccFSVFO4002-MidCenturyModern.esl
ccFRSFO4001-HandmadeShotgun.esl
ccEEJFO4001-DecorationPack.esl

As of Skyrim SE v1.5.3.0.8, the implicitly active ESLs are::

ccBGSSSE002-ExoticArrows.esl
ccBGSSSE003-Zombies.esl
ccBGSSSE004-RuinsEdge.esl
ccBGSSSE006-StendarsHammer.esl
ccBGSSSE007-Chrysamere.esl
ccBGSSSE010-PetDwarvenArmoredMudcrab.esl
ccBGSSSE014-SpellPack01.esl
ccBGSSSE019-StaffofSheogorath.esl
ccMTYSSE001-KnightsoftheNine.esl
ccQDRSSE001-SurvivalMode.esl

Remove Boost.Locale dependency

Boost.Locale is used to convert between Windows-1252 and UTF-8, but UTF-16 uses the same small bytes for its first 255 codepoints as Windows-1252, so it's possible to convert between UTF-8 and Windows-1252 using the standard library's support for UTF-8 and UTF-16.

It's also used by libespm, so resolution of WrinklyNinja/libespm#4 is a precondition for this.

Game handle creation fails when empty .esp or .esm files are present

Reported by pStyl3 here.

To replicate, download ASIS and SUM, and run SUM through Mod Organiser, choosing to sort with LOOT. During the process, LOOT will launch with the error:

Error: Game-specific settings could not be initialised. libloadorder failed to create a game handle. Details: ASIS.esp : File is empty.

This is because a zero-byte ASIS.esp file is created before SUM runs LOOT. Ideally, the ASIS.esp created would have a TES4 header record, but libloadorder shouldn't fail when that isn't the case, it should just ignore the file, just like all other non-plugin files. In fact, it usually doesn't fail, but for some reason Mod Organiser's involvement breaks something.

It's worth noting though that this behaviour probably breaks why the ASIS.esp file is being created in the first place, so I should discuss this with whoever is responsible.

Backend rewrite / refactor

To complement my rewrite of libespm2, I want to rewrite libloadorder's backend using TDD:

• Combine the LoadOrder and ActivePlugins classes to eliminate the need to synchronise the two.
• Encapsulate the data members of the combined class, so the rest of libloadorder doesn't know how the load order / active plugin info is stored, for greater maintainability.
• Make error recovery more consistent across the API: in some cases changes are undone, in others they are not, what would be better would be for no changes to be made until they have been validated.
• Use libespm2 for plugin file reading.
• Refactor the _lo_game_handle_int struct so that all game settings are inherited from a GameSettings class. The struct will then only add the C string/array memory storage members.
• Have the LoadOrder constructor store a GameSettings reference to be used instead of having to pass a _lo_game_handle_int object for most of its member functions.
• Read plugin data when loading the load order, and cache it until next load. This will affect change detection though, as it means that changes to master flags will go unnoticed, as the plugins folder modification time only changes if files are added or removed, not if they're edited. As such, the modification time for each plugin would have to be compared against its cached time, but this still involves less filesystem interaction than not caching plugin data or removing the load order cache.

The end result should be that libloadorder has a C++ API that gets wrapped by a C API, which only handles the memory management and exceptions, and contains no logic otherwise.

I would also like to make a few behavioural changes:

• When loading data, treat invalid data like the game would, ie. enforce the master/non-master partition, the 255 active plugins limit and later duplicates "winning", rather than just warning about it.
• Don't check that a plugin's masters load before it. Utumno has pointed out that while doing otherwise causes the game to crash, it causes headaches when uninstalling plugins and in other cases.
• Don't force a timestamp-based game to load it's main master file first, as this isn't a condition the game enforces.
• Deprecate lo_set_game_master() and its C++ equivalent, making them do nothing, as the game master will hold no significance for timestamp-based games, and the filename for the game master is hardcoded in textfile-based games.

While API behaviour change is a breaking change, it's breaking safely, because it means that clients will no longer have to expect weirdness when reading data, and the API will be more permissive. Otherwise, maintaining API compatibility is also a goal.

Tidy up and refactor API code and tests

Fresh off the back of the backend code overhaul in #20, some improvements can also be made to the "frontend" API code:

• Tidying up the API functions' code, refactoring some common memory management operations into _lo_game_handle_int.
• Tidying up the API functions' tests. They're very messy compared to my newer tests, and not nearly as comprehensive, though the actual number of tests can probably be cut as most functionality is contained in the already-tested LoadOrder class.

Error when trying to open empty .esp file during handle creation

Reported here.

The error message is from libespm, but libloadorder shouldn't be attempting to open an empty .esm or .esp file unless it's to check if it's a valid file in the first place, and that should have exceptions caught for evaluation, rather than propagating the error.

Build fails if LaTeX and Doxygen are installed

afaict there is no docs/latex/make.bat and none is ever created.

Implicitly active plugins aren't reported as such if the plugins.txt doesn't exist

I just noticed that if eg. Skyrim/plugins.txt is missing, libloadorder reports that no plugins are active. This is because LoadOrder::loadActivePlugins() is only called if the active plugins file exists, and it also adds the implicitly active plugins.

Rewrite it in Rust

Primarily as a learning exercise (more FFI, docs, libespm's ergonomics, a step up in complexity, etc.), but the library is a bit messy and could do with a refactor anyway. I'd like to avoid API breakage.

---

TODO:

• Reimplement all get/set functionality
• Reimplement thread-safe C FFI
• Reimplement path caching, but only keep it if profiling shows it's beneficial
• Reimplement desync detection
• Reimplement on-disk load order validity detection
• Convert docs

C API Breakage:

• String and array output needs to be explicitly freed in the Rust implementation, as their lifetimes are different. This could be avoided by having an internal static cache, but is that desirable?
• LIBLO_ERROR_TIMESTAMP_READ_FAIL is unused as read failures are reported as I/O errors. This change could potentially be avoided.
• If attempting to write an invalid filename, the Rust implementation errors, while the C++ implementation skips it and returns LIBLO_WARN_BAD_FILENAME. This could be avoided somehow, but it is due to invalid input, so an error is probably more correct.
• Memory allocation failure now causes a panic instead of a returned error code, so LIBLO_ERROR_NO_MEM is unused.
• The constness of some function arguments has changed, though this did not affect client code in the LOOT API.
• Attempting to deactivate an implicitly active plugin that isnot installed would not error in the C++ implementation, despite deactivating an implicitly active plugin and deactivating a plugin that is not installed both being error conditions. The Rust implementation fixes this bug.
• Attempting to set the active plugins giving an array including an implicitly active plugin that is not installed would not error in the C++ implementation, despite inclusion of a normal plugin that is not installed being an error condition. The Rust implementation fixes this bug.

If API breakage is embraced, then replacing the error codes with equivalents that map more cleanly to the Rust error types would be good.

MSVS build files broke

MSVS can't no longer find libespm due to recent changes on master.

set_active_plugins counts light masters and non-light-masters by file extension

In more recent versions of Skyrim / Fallout 4, the .esl file extension just forces the ESL flag at runtime, but a plugin that has it set with a different file extension still (for the purposes of the function) acts like a light master.

Therefore libloadorder needs to load all the plugins before it can count the different types.

Handle Fallout 4's DLC loading correctly

If DLC .esm files are not listed in plugins.txt, FO4 will load them anyway. It seems that which .esm files are DLC plugins are hardcoded into the game, as the game doesn't seem to use DLClist.txt or any other file to determine this.

It's worth noting that the DLC plugins don't appear alongside other plugins in the in-game load order settings UI, so modifying the load order there will remove them from plugins.txt if they were previously added.

If they are not listed in plugins.txt, the DLC plugins will load after all other master plugins, in the order of their release. In this case, the DLC do not load before any master files that have them as masters.

It seems like the DLC loading behaviour is either badly designed (requiring game executable / load order utility updates to support each new release, breaking master files having them as masters) or heavily bugged, so it's worth keeping an eye on this as the behaviour will hopefully change.

Update for Skyrim SE changes

Skyrim SE doesn't include the official plugins in its plugins.txt, so libloadorder shouldn't either. It also currently (as of 1.2.39) loads the DLC plugins in timestamp order, which makes no sense whatsoever, so I'm in favour of just changing the hardcoded order to use the same as NMM as it's also the order that is most commonly assumed for the DLC in the original Skyrim. That order is:

Dawnguard.esm
Hearthfires.esm
Dragonborn.esm

@DuskDweller, will NMM be changing how it handles the DLC plugins' load order for Skyrim SE?

File read exception when getting Skyrim load order and a plugin is missing.

See here.

Catch panics at FFI boundary

Libloadorder doesn't currently do this, and there's a crash in LOOT that I can't replicate but which is probably due to a panic in libloadorder or esplugin.

Active light master limit may be too generous

The Creation Kit wiki page on the file formats says that

A maximum of 4096 .esl files can be loaded by the engine at once, but more Form IDs in an .esl file means that less .esl files can be loaded.
Using .esl files that each have 2048 Form IDs, the limit drops to about 300 loaded .esl files.

libloadorder currently just uses a static 4096 as the max, because I don't understand the relationship between number of FormIDs and the maximum active light masters, but filing this issue in case it gets figured out or more is revealed in future.

Masters that depend on non-masters force the latter to load as masters

It's possible to create a master that depends on a non-master, and the game will load it, but in doing so the non-master will be moved so that it loads before the master (unverified). Libloadorder doesn't currently handle this case, as it doesn't check plugin masters at all.

I'm hoping this behaviour gets classified as a bug and fixed by Bethesda, since (IIRC) the analogous behaviour with a master depending on a non-master is a crash on load, but filing this issue to ensure I don't forget about it.

Edit: Removed mentions of light masters, it's actually about ESMs, which all light masters were when this issue was originally filed.

Setting load order accepts Creation Club plugins in invalid positions

Reported here. The load order should be:

Fallout4.esm
DLCRobot.esm
DLCworkshop01.esm
DLCCoast.esm
DLCworkshop02.esm
DLCworkshop03.esm
DLCNukaWorld.esm
ccbgsfo4024-pacamo03.esl
DLCUltraHighResolution.esm
Unofficial Fallout 4 Patch.esp

but libloadorder will accept:

Fallout4.esm
DLCRobot.esm
DLCworkshop01.esm
DLCCoast.esm
DLCworkshop02.esm
DLCworkshop03.esm
DLCNukaWorld.esm
DLCUltraHighResolution.esm
Unofficial Fallout 4 Patch.esp
ccbgsfo4024-pacamo03.esl

though ccbgsfo4024-pacamo03.esl is a Creation Club plugin and so always loads after the DLC plugins (excluding the ultra high resolution DLC plugin) and before all other plugins.

While libloadorder does limited validation (it doesn't check that a plugin loads after all its masters, for example), it would be good to improve it wherever possible without introducing too much complexity, and this might be one of those cases.

---

Implementing this is almost as simple as checking that the start of the list matches the list of implicitly active plugins, ignoring any of the latter that are missing, up to the end of the former if it is shorter than the latter. However, the original Skyrim has Update.esm as an implicitly active plugin, but it can load after other master files, so that will have to be handled as a special case.

Return a warning code when the API operates on an invalid load order

There should be some method by which a client can be told they need to use lo_fix_plugin_lists before they try try setting the load order themselves. See wrye-bash/wrye-bash#162 for an example use-case.

E.g. when the client loads, if the existing load order is invalid, it should be fixed straight away. If libloadorder makes a change, then another utility makes a change that results in an invalid change, when libloadorder updates its cache, it should also warn then.

A call to CheckValidity after each Load should be enough, and to catch exceptions thrown in it and treat them as a warning. A new warning return code should be added.

LoadOrder tests' active plugins setup is broken

In LoadOrderTest::SetUp(), an initial active plugins file is written, but in input plugin names are supplied as an unordered set, so duplicate entries is not tested (the set is initialised with them, but only one is stored because it's a set), and the order of entries is unknown, which affects synchronisation tests.

Replace Error::GameMasterMustLoadFirst with a more generic hardcoded plugin position error variant

Until v11.2.2, Error::GameMasterMustLoadFirst was only used for indicating when the game's main master file was in the wrong position. It then got used to also indicate invalid positions for other hardcoded plugins, to avoid the breaking change of adding another error variant, but it would be useful to be able to provide the name of the plugin causing an issue, so next time breaking changes are made, also replace the variant with something like Error::InvalidHardcodedPluginPosition(String).

Timestamp load order method should not check for master status

The pluginComparator class should drop the esm tests. The way it is now a 'corrected' load order is returned to the client but the actual load order (the modification times) stay as they were

EDIT: the reason I want this is twofold - first Bash and liblo do double job in reading the timestamps - if liblo would just return me the timestamps I could easily check for master status. But I see now this will get very tricky - essentially a _cache_and_return_timestamps should be needed. I guess this is too much.
The second reason would be to have more control when setting load order of masters between them - for instance when I flip esm status.

So this probably should be renamed to "expose timestamps cache" cause that is what really is about. Will edit as I get more insight into the API.

Refactor common test code so that it's reusable

There's a few copy/pasted functions in the testing code. It might be possible to have a module that can be imported by unit tests and benchmarks alike, but it's more likely that I'll need to create a testing crate.

'Fix Load Order and Active Plugins' Function

ATM the getters for load order and active plugins don't do any checking to see if what they're getting makes sense, so if the plugin list contains files that aren't installed any more, they don't get removed.

The workaround is to have a function that gets the load order, removes any missing plugins, and sets it again. Since this is likely a fairly often occurance though, a function in libloadorder would be useful. Same for the active plugins list. In fact, might as well fix them both at the same time...

Update Fallout 4 load order implementation

As of the availability of the Survival Mode beta for Fallout 4, the load order mechanism has changed. plugins.txt now lists all installed plugin filenames in load order. Active plugins have their filenames prefixed by an asterisk *, for example:

*Fallout4.esm
ArmorKeywords.esm
Armorsmith Extended.esp
Field Scribe Army - AE Comp.esp
*EasyHacking.esp
*Test_WEAP.esp
*Simple Intersection.esp

In the above, Fallout4.esm, EasyHacking.esp and Test_WEAP.esp are all active, and the other plugins are not active.

This new mechanism obsoletes loadorder.txt, so libloadorder should no longer write that file for Fallout 4. I'll call the new mechanism the "asterisk" load order method (so the constant will be LIBLO_METHOD_ASTERISK).

I'll need to update the test suite to check that data is written correctly for the new method, then update the corresponding code accordingly.

Bunch of failed tests (Arch Linux, 64-bit build)

See output at https://gist.github.com/Freso/a7f1c2fee5445dec881a - I don't know whether it is anything to be concerned about, but figured I'd report back in case it's an easy fix. (Maybe it's a Boost 1.59 vs. 1.55 thing, if Travis doesn't complain about it?)

Return string slices from ReadableLoadOrder methods

Now that Plugin::name() no longer allocates, it's possible to avoid copying strings in plugin_names(), plugin_at() and active_plugin_names().

Errors opening file paths with non-ANSI characters

See http://forums.bethsoft.com/topic/1459682-relz-wrye-bash-thread-97/?p=22860364 and onwards.

At the moment, I think the issue is to do with the path being returned by SHGetFolderPath being encoded in Windows/ANSI and not being converted to UTF-8, or something like that. I need to investigate more, really.

Undefined references to several functions

After following the workaround in #32 (Build fails if LaTeX and Doxygen are installed)
The build continues through to the linking phase and spits out the message below.
This might also be related to #31 (Remove Boost.Locale dependency)

Link to file on dropbox -- because github can't properly display the file:
https://dl.dropboxusercontent.com/u/104054522/skyrim_mods/libloadorder/libloadorder_undefined_references_error.txt

Use new Fallout4.ccc and Skyrim.ccc files to determine Creation Club plugins

The .ccc files are text files with \r\n line separators and one CC plugin per line, in their load order. These should be read and cached as implicitly active plugins instead of hardcoding the CC plugins, as the game executables don't hardcode them any more.

Passing invalid load order to lo_set_active_plugins empties plugins.txt

For Skyrim. Reported by DuskDweller. This should be fixed so that the function returns an error code without making any changes.

Compile the dlls

Wrote a wiki article that you may want to incllude in your readme/wiki:

https://github.com/Utumno/libloadorder/wiki/Build-on-windows

For people new to CMake, boost etc this can be handy.

Still I am using MSVC 2013 and when building I get a libloadorder.lib - when I flip the BUILD_SHARED_LIBS flag build fails with:

 1>LINK : fatal error LNK1104: cannot open file 'libboost_filesystem-vc120-mt-sgd-1_56.lib'

So:

1. What is the procedure for generating MSVC solution to build the dlls
2. Is it only possible to build on MSVC 2012 ?

Thanks

File encoding error

See here.

tknikita's plugins.txt and loadorder.txt.

lo_set_active_plugins requires load order to be set for all plugins

If lo_set_active_plugins() is called for Skyrim and passed an array that contains a plugin that hasn't previously had its load order position set, it will not be activated. This behaviour should be changed.

Probably the best fix would be to silently append any such plugins to the load order, but another possibility would be to return an error code if such a plugin was encountered.

DuskDweller also suggested a addIfNotPresent flag input to control the behaviour, but this would break the ABI, and I'd rather avoid that.

Leverage libespm for master flag checking and master files reading

Using libespm to get whether a plugin is a master or not would be preferred to the current method of libloadorder reading the master flag itself, as libloadorder shouldn't concern itself with the file format of plugins.

It would also be good for libloadorder to enforce the requirement that a plugin's masters all load before the plugin, which would require the reading of each plugin's master list, another job for libespm.

Extend lo_fix_plugin_lists to ensure full compliance with validity checks

Right now all that lo_fix_plugin_lists does is make sure all the plugins listed are present. Arthmoor has identified (wrye-bash/wrye-bash#162) that having more than 255 plugins active also causes problems in Bash, and being able to call this function to fix it would be neat.

The checks are given at the top of activeplugins.h and loadorder.h.

Add some kind of automated benchmarking to help catch regressions

I've currently got a stashed changeset in my local repo that provides a test that runs load() on a load order with 1000 plugins and 300 of them listed in plugins.txt/loadorder.txt, doing so a few times, taking an average and printing to stdout. This works for one-off testing, and roughly replicates what I could be doing with nightly Rust's #[bench]. However, it's not ideal because it's not run regularly (it's messy so I don't commit it).

Recently I found that somewhere between 10.0.0 and 10.1.0, performance of loading a load order halved. I tracked most of it down to upgrading WalkDir (which it turns out libloadorder doesn't need anyway...) but there's still a ~ 20% drop in performance even if I go back and run 10.0.0 code, so my PC isn't good for measuring performance trends (could it be due to meltdown/spectre mitigation, or just something happening in the background?). It would be good to have a history of performance results available on CI.

Criterion offers benchmarking on stable, and bencher is a stable port of #[bench], either could potentially be used.

rbegin and rend are not declared (using boost 1.55) and codecvt not found

Debian GNU/Linux 8 (jessie) / Kernel 4.7.0-0.bpo.1-amd64
Boost: v1.55

Compiling libloadorder says that rbegin and rend were not decaled in this scope and
that they should be boost::rbegin / boost::rend.

This is the commit I was building against:
ae2512d

after perfixing all occurrences of rbegin / rend with boost:: the above error went away;
however, now it is complaining about codecvt not being found.

This is the output from make:

[ 46%] Building CXX object CMakeFiles/loadorder.dir/src/backend/LoadOrder.cpp.o
/home/william/temp/libloadorder/src/backend/LoadOrder.cpp: In member function ‘void liblo::LoadOrder::loadActivePlugins()’:
/home/william/temp/libloadorder/src/backend/LoadOrder.cpp:462:40: error: ‘rbegin’ was not declared in this scope
         for (auto it = rbegin(loadOrder); numActivePlugins > maxActivePlugins && it != rend(loadOrder); ++it) {
                                        ^
/home/william/temp/libloadorder/src/backend/LoadOrder.cpp:462:40: note: suggested alternative:
In file included from /usr/include/boost/range/functions.hpp:23:0,
                 from /usr/include/boost/range/iterator_range_core.hpp:28,
                 from /usr/include/boost/range/iterator_range.hpp:13,
                 from /usr/include/boost/iostreams/traits.hpp:39,
                 from /usr/include/boost/iostreams/detail/dispatch.hpp:17,
                 from /usr/include/boost/iostreams/flush.hpp:17,
                 from /usr/include/boost/iostreams/close.hpp:18,
                 from /usr/include/boost/iostreams/device/mapped_file.hpp:20,
                 from /home/william/temp/libloadorder/build/external/src/libespm/include/libespm/Plugin.h:30,
                 from /home/william/temp/libloadorder/src/backend/Plugin.h:32,
                 from /home/william/temp/libloadorder/src/backend/LoadOrder.h:29,
                 from /home/william/temp/libloadorder/src/backend/LoadOrder.cpp:26:
/usr/include/boost/range/rbegin.hpp:46:1: note:   ‘boost::rbegin’
 rbegin( const C& c )
 ^
/home/william/temp/libloadorder/src/backend/LoadOrder.cpp:462:102: error: ‘rend’ was not declared in this scope
         for (auto it = rbegin(loadOrder); numActivePlugins > maxActivePlugins && it != rend(loadOrder); ++it) {
                                                                                                      ^
/home/william/temp/libloadorder/src/backend/LoadOrder.cpp:462:102: note: suggested alternative:
In file included from /usr/include/boost/range/functions.hpp:24:0,
                 from /usr/include/boost/range/iterator_range_core.hpp:28,
                 from /usr/include/boost/range/iterator_range.hpp:13,
                 from /usr/include/boost/iostreams/traits.hpp:39,
                 from /usr/include/boost/iostreams/detail/dispatch.hpp:17,
                 from /usr/include/boost/iostreams/flush.hpp:17,
                 from /usr/include/boost/iostreams/close.hpp:18,
                 from /usr/include/boost/iostreams/device/mapped_file.hpp:20,
                 from /home/william/temp/libloadorder/build/external/src/libespm/include/libespm/Plugin.h:30,
                 from /home/william/temp/libloadorder/src/backend/Plugin.h:32,
                 from /home/william/temp/libloadorder/src/backend/LoadOrder.h:29,
                 from /home/william/temp/libloadorder/src/backend/LoadOrder.cpp:26:
/usr/include/boost/range/rend.hpp:46:1: note:   ‘boost::rend’
 rend( const C& c )
 ^
/home/william/temp/libloadorder/src/backend/LoadOrder.cpp: In lambda function:
/home/william/temp/libloadorder/src/backend/LoadOrder.cpp:519:38: error: ‘rbegin’ was not declared in this scope
             return *rbegin(timestamps) + 60;
                                      ^
/home/william/temp/libloadorder/src/backend/LoadOrder.cpp:519:38: note: suggested alternative:
In file included from /usr/include/boost/range/functions.hpp:23:0,
                 from /usr/include/boost/range/iterator_range_core.hpp:28,
                 from /usr/include/boost/range/iterator_range.hpp:13,
                 from /usr/include/boost/iostreams/traits.hpp:39,
                 from /usr/include/boost/iostreams/detail/dispatch.hpp:17,
                 from /usr/include/boost/iostreams/flush.hpp:17,
                 from /usr/include/boost/iostreams/close.hpp:18,
                 from /usr/include/boost/iostreams/device/mapped_file.hpp:20,
                 from /home/william/temp/libloadorder/build/external/src/libespm/include/libespm/Plugin.h:30,
                 from /home/william/temp/libloadorder/src/backend/Plugin.h:32,
                 from /home/william/temp/libloadorder/src/backend/LoadOrder.h:29,
                 from /home/william/temp/libloadorder/src/backend/LoadOrder.cpp:26:
/usr/include/boost/range/rbegin.hpp:46:1: note:   ‘boost::rbegin’
 rbegin( const C& c )
 ^
In file included from /usr/include/c++/4.9/algorithm:62:0,
                 from /usr/include/boost/smart_ptr/shared_ptr.hpp:42,
                 from /usr/include/boost/shared_ptr.hpp:17,
                 from /usr/include/boost/filesystem/path.hpp:29,
                 from /usr/include/boost/filesystem.hpp:16,
                 from /home/william/temp/libloadorder/build/external/src/libespm/include/libespm/Plugin.h:27,
                 from /home/william/temp/libloadorder/src/backend/Plugin.h:32,
                 from /home/william/temp/libloadorder/src/backend/LoadOrder.h:29,
                 from /home/william/temp/libloadorder/src/backend/LoadOrder.cpp:26:
/usr/include/c++/4.9/bits/stl_algo.h: In instantiation of ‘_OIter std::generate_n(_OIter, _Size, _Generator) [with _OIter = std::insert_iterator<std::set<long int> >; _Size = long unsigned int; _Generator = liblo::LoadOrder::saveTimestampLoadOrder()::<lambda()>]’:
/home/william/temp/libloadorder/src/backend/LoadOrder.cpp:520:10:   required from here
/usr/include/c++/4.9/bits/stl_algo.h:4325:11: error: no match for ‘operator=’ (operand types are ‘std::insert_iterator<std::set<long int> >’ and ‘void’)
  *__first = __gen();
           ^
/usr/include/c++/4.9/bits/stl_algo.h:4325:11: note: candidates are:
In file included from /usr/include/c++/4.9/bits/stl_algobase.h:67:0,
                 from /usr/include/c++/4.9/bits/char_traits.h:39,
                 from /usr/include/c++/4.9/string:40,
                 from /home/william/temp/libloadorder/src/backend/Plugin.h:29,
                 from /home/william/temp/libloadorder/src/backend/LoadOrder.h:29,
                 from /home/william/temp/libloadorder/src/backend/LoadOrder.cpp:26:
/usr/include/c++/4.9/bits/stl_iterator.h:639:7: note: std::insert_iterator<_Container>& std::insert_iterator<_Container>::operator=(const typename _Container::value_type&) [with _Container = std::set<long int>; typename _Container::value_type = long int]
       operator=(const typename _Container::value_type& __value)
       ^
/usr/include/c++/4.9/bits/stl_iterator.h:639:7: note:   no known conversion for argument 1 from ‘void’ to ‘const value_type& {aka const long int&}’
/usr/include/c++/4.9/bits/stl_iterator.h:647:7: note: std::insert_iterator<_Container>& std::insert_iterator<_Container>::operator=(typename _Container::value_type&&) [with _Container = std::set<long int>; typename _Container::value_type = long int]
       operator=(typename _Container::value_type&& __value)
       ^
/usr/include/c++/4.9/bits/stl_iterator.h:647:7: note:   no known conversion for argument 1 from ‘void’ to ‘std::set<long int>::value_type&& {aka long int&&}’
/usr/include/c++/4.9/bits/stl_iterator.h:588:11: note: std::insert_iterator<std::set<long int> >& std::insert_iterator<std::set<long int> >::operator=(const std::insert_iterator<std::set<long int> >&)
     class insert_iterator
           ^
/usr/include/c++/4.9/bits/stl_iterator.h:588:11: note:   no known conversion for argument 1 from ‘void’ to ‘const std::insert_iterator<std::set<long int> >&’
/usr/include/c++/4.9/bits/stl_iterator.h:588:11: note: std::insert_iterator<std::set<long int> >& std::insert_iterator<std::set<long int> >::operator=(std::insert_iterator<std::set<long int> >&&)
/usr/include/c++/4.9/bits/stl_iterator.h:588:11: note:   no known conversion for argument 1 from ‘void’ to ‘std::insert_iterator<std::set<long int> >&&’
CMakeFiles/loadorder.dir/build.make:123: recipe for target 'CMakeFiles/loadorder.dir/src/backend/LoadOrder.cpp.o' failed
make[2]: *** [CMakeFiles/loadorder.dir/src/backend/LoadOrder.cpp.o] Error 1
CMakeFiles/Makefile2:130: recipe for target 'CMakeFiles/loadorder.dir/all' failed
make[1]: *** [CMakeFiles/loadorder.dir/all] Error 2
Makefile:137: recipe for target 'all' failed
make: *** [all] Error 2

[ 51%] Building CXX object CMakeFiles/loadorder.dir/src/api/libloadorder.cpp.o
/home/william/temp/libloadorder/src/api/libloadorder.cpp:31:19: fatal error: codecvt: No such file or directory
 #include <codecvt>
                   ^
compilation terminated.
CMakeFiles/loadorder.dir/build.make:238: recipe for target 'CMakeFiles/loadorder.dir/src/api/libloadorder.cpp.o' failed
make[2]: *** [CMakeFiles/loadorder.dir/src/api/libloadorder.cpp.o] Error 1
CMakeFiles/Makefile2:130: recipe for target 'CMakeFiles/loadorder.dir/all' failed
make[1]: *** [CMakeFiles/loadorder.dir/all] Error 2
Makefile:137: recipe for target 'all' failed
make: *** [all] Error 2

Unit testing

Unit testing of the API and internals would be good. I'm not sure how to accomplish it though, since libloadorder has a lot of filesystem interactions...

I'm going to try implementing the tests using Google Test, since I've seen examples of it being used in CMake + Travis environments, and it's obviously going to be of good quality.

When saving, only write file if necessary

A file write is necessary if plugins.txt/loadorder.txt does not contain entries for all the plugins they need to, in the order they need to be. This adds some overhead, but gets around write permissions issues that can happen if the read-only flag is temporarily set for the file (as has been happening for FO4).

This is for loot/loot#562.

Setting incomplete load order for timestamp-based game can cause other plugins to load in unspecified positions

When setting an incomplete load order for timestamp-based games, the timestamps of plugins not included in the load order are unchanged, and the earliest timestamp used is the earliest timestamp of the specified plugins, so it's possible for those other plugins to end up loading before, amongst or after the specified plugins.

In timestamp-based games, any plugins that aren't specified get loaded in an undefined load order at the end. This is the correct behaviour, and in v6 timestamp-based games would do the same as unspecified plugins would be appended to the given load order. It looks like that needs to be re-implemented.

Improve performance when loading load order

Plugin validity checks are pretty much the only expensive operation in libloadorder, as they involve reading the whole plugin file, which can be quite large and so reading can be slow. The performance of actual plugin reading is up to libespm, but there are a couple of things libloadorder can do to improve its performance:

• At the moment, validity is checked then the plugin is loaded, which involves two file reads (though the second only reads the file header). Given that libloadorder only deals with file headers, this could be combined into a single step, where a Plugin object is constructed, and if an exception is thrown then it is handled as an invalid plugin, and if not that constructed object is stored. Profiling shows this doesn't bring any performance benefit, and it has less clear semantics, so this idea is dropped.
• When the load order is reloaded because filesystem changes have been detected, all data is discarded and re-loaded. Depending on the changes though, it could be that most or all of the existing plugin objects can be reused. Plugin modification times would need to be compared with their cached times to see if the objects need re-loading or not.

Continous Integration

Now that libloadorder should be support being built as a native Linux library, it would be good if its unit tests (to be added as in #8) could be run automatically for each commit. Travis integrates well with GitHub, and I have experience with it.

Check performance impact of parallel iterators not involving filesystem interaction

There are several cases where .par_iter() and other rayon-provided methods are used when sorting or searching, and these may no longer provide any performance boost since the recent optimisations of things like string comparisons were made. It would be good to re-test each of these and make any that don't provide any boost serial again.

Light masters sharing the FE index with another plugin

At least for Fallout 4, it is possible to load a non-master (or presumably a master) in the FE index while light masters are also present, so they end up sharing FE at runtime. In this case, if two records have the same runtime FormID, then the non-ESL record replaces the ESL record.

This behaviour is undesirable, and may lead to strange issues due to records being silently replaced by records that may be completely unrelated. libloadorder should probably disallow activating a normal plugin in the FE slot when light masters are also present.

Document plugin validity condition and caching behaviour

As @Utumno pointed out, libloadorder only deals with valid plugins, but what it considers to be a valid plugin isn't documented.