stephen-bunn / bethesda-structs

A wrapper for Bethesda's popular plugin/archive file formats

This project is archived as I can no longer maintain it!
Sorry for any inconvenience <3.

About

Modding Bethesda's games can be a fine-art.

This package intends to provide clean and accessible methods for parsing and understanding Bethesda's filetypes.

For example:

There are so many "unarchiver" tools for Bethesda's archives (.bsa and .ba2), but no good programmatic way to read these filetypes. Using this package, understanding every little detail about an archive is simple and straight-forward (see BSA Usage and BA2 Usage).

All contributions welcome!

For more advanced usage and information, check out the documentation.

The supported filetypes are parsers not writers.

We do not currently support the writing of archives or plugins.

Installation

Because this is glorious Python, installing bethesda-structs should be super-duper simple.

Using PyPi

The fastest and quickest way to install this packages is by simply using pipenv (or if you're oldschool pip).

$ pipenv install bethesda-structs

Using Git

You can install this package using Git by simply cloning the repo and building the package yourself!

$ git clone https://github.com/stephen-bunn/bethesda-structs.git
$ pipenv install --dev
$ pipenv run python setup.py install

Usage

Using bethesda-structs is designed to be straight-forward and intuitive.

Below are some short examples of parsing various filetypes.

ESP

The ability to parse plugin files is super helpful for understanding the additions and changes that are made to the game.

Currently the only other real tool that can expose this information to you is TESEdit and its sibling applications.

This package aims to provide simple, programmatic access to the in-depth details of a plugin!

Because of how long it takes to build complete subrecord parers for a given plugin version, the only currently supported plugins are:

• FNVPlugin - Fallout: New Vegas (partial)
• F03Plugin - Fallout 3 (partial and experimental)

>>> from bethesda_structs.plugin.fnv import FNVPlugin
>>> plugin = FNVPlugin.parse_file('/media/sf_VMShared/esp/fnv/NVWillow.esp')
>>> print(plugin)
FNVPlugin(filepath='/media/sf_VMShared/esp/fnv/NVWillow.esp')
>>>
>>> # print plugin header (is a record)
...
>>> print(plugin.container.header)
Container:
    type = u'TES4' (total 4)
    data_size = 163
    flags = Container:
        master = True
    id = 0
    revision = 0
    version = 15
    data = b'HEDR\x0c\x00\x1f\x85\xab?\x97\x12\x00\x00#\xad'... (truncated, total 163)
    subrecords = ListContainer:
        Container:
            type = u'HEDR' (total 4)
            data_size = 12
            data = b'\x1f\x85\xab?\x97\x12\x00\x00#\xad\r\x00' (total 12)
            parsed = Container:
                value = Container:
                    version = 1.340000033378601
                    num_records = 4759
                    next_object_id = 896291
                description = u'Header' (total 6)
        Container:
            type = u'CNAM' (total 4)
            data_size = 9
            data = b'llamaRCA\x00' (total 9)
            parsed = Container:
                value = u'llamaRCA' (total 8)
                description = u'Author' (total 6)
        Container:
            type = u'SNAM' (total 4)
            data_size = 16
            data = b'NVWillow v.1.10\x00' (total 16)
            parsed = Container:
                value = u'NVWillow v.1.10' (total 15)
                description = u'Description' (total 11)
        Container:
            type = u'MAST' (total 4)
            data_size = 14
            data = b'FalloutNV.esm\x00' (total 14)
            parsed = Container:
                value = u'FalloutNV.esm' (total 13)
                description = u'Master Plugin' (total 13)
        Container:
            type = u'DATA' (total 4)
            data_size = 8
            data = b'\x00\x00\x00\x00\x00\x00\x00\x00' (total 8)
            parsed = Container:
                value = 0
                description = u'File Size' (total 9)
        Container:
            type = u'ONAM' (total 4)
            data_size = 68
            data = b'V\xe3\x0c\x00\xc3\xe3\x0c\x00\xc4\xe3\x0c\x00\xc5\xe3\x0c\x00'... (truncated, total 68)
            parsed = Container:
                value = ListContainer:
                    844630
                    844739
                    844740
                    844741
                    1372461
                    1372463
                    1383111
                    1385321
                    1387301
                    1387302
                    1387303
                    1387304
                    1387906
                    1457771
                    1479505
                    1520201
                    1544392
                description = u'Overridden Records' (total 18)
>>>
>>> # iterate over KEYM records (only 1 in this plugin)
...
>>> for record in plugin.iter_records('KEYM'):
...     print(record)
...
Container:
    type = u'KEYM' (total 4)
    data_size = 279
    flags = Container:
    id = 17415634
    revision = 0
    version = 15
    data = b'EDID\x17\x00WillowNova'... (truncated, total 279)
    subrecords = ListContainer:
        Container:
            type = u'EDID' (total 4)
            data_size = 23
            data = b'WillowNovacBunga'... (truncated, total 23)
            parsed = Container:
                value = u'WillowNovacBungalowKey' (total 22)
                description = u'Editor ID' (total 9)
        Container:
            type = u'OBND' (total 4)
            data_size = 12
            data = b'\xff\xff\xfc\xff\x00\x00\x01\x00\x04\x00\x00\x00' (total 12)
            parsed = Container:
                value = Container:
                    X1 = -1
                    Y1 = -4
                    Z1 = 0
                    X2 = 1
                    Y2 = 4
                    Z2 = 0
                description = u'Object Bounds' (total 13)
        Container:
            type = u'FULL' (total 4)
            data_size = 27
            data = b'Dino Dee-lite Bu'... (truncated, total 27)
            parsed = Container:
                value = u'Dino Dee-lite Bungalow Key' (total 26)
                description = u'Name' (total 4)
        Container:
            type = u'MODL' (total 4)
            data_size = 23
            data = b'Clutter\\Key01Dir'... (truncated, total 23)
            parsed = Container:
                value = u'Clutter\\Key01Dirty.NIF' (total 22)
                description = u'Model Filename' (total 14)
        Container:
            type = u'ICON' (total 4)
            data_size = 48
            data = b'Interface\\Icons\\'... (truncated, total 48)
            parsed = Container:
                value = u'Interface\\Icons\\PipboyImages\\Ite'... (truncated, total 47)
                description = u'Large Icon Filename' (total 19)
        Container:
            type = u'MICO' (total 4)
            data_size = 66
            data = b'Interface\\Icons\\'... (truncated, total 66)
            parsed = Container:
                value = u'Interface\\Icons\\PipboyImages_sma'... (truncated, total 65)
                description = u'Small Icon Filename' (total 19)
        Container:
            type = u'SCRI' (total 4)
            data_size = 4
            data = b'T.\n\x01' (total 4)
            parsed = Container:
                value = FormID(form_id=17444436, forms=['SCPT'])
                description = u'Script' (total 6)
        Container:
            type = u'YNAM' (total 4)
            data_size = 4
            data = b'\xbb\x10\x07\x00' (total 4)
            parsed = Container:
                value = FormID(form_id=463035, forms=['SOUN'])
                description = u'Sound - Pick Up' (total 15)
        Container:
            type = u'ZNAM' (total 4)
            data_size = 4
            data = b'\xbc\x10\x07\x00' (total 4)
            parsed = Container:
                value = FormID(form_id=463036, forms=['SOUN'])
                description = u'Sound - Drop' (total 12)
        Container:
            type = u'DATA' (total 4)
            data_size = 8
            data = b'\x00\x00\x00\x00\x00\x00\x00\x00' (total 8)
            parsed = Container:
                value = Container:
                    value = 0
                    weight = 0.0
                description = u'Data' (total 4)

BSA

Bethesda's default archive structure.

>>> from bethesda_structs.archive.bsa import BSAArchive
>>> archive = BSAArchive.parse_file('/media/sf_VMShared/bsa/Campfire.bsa')
>>> print(archive)
BSAArchive(filepath=PosixPath('/media/sf_VMShared/bsa/Campfire.bsa'))
>>>
>>> # print archive header
...
>>> print(archive.container.header)
Container:
    magic = b'BSA\x00' (total 4)
    version = 105
    directory_offset = 36
    archive_flags = Container:
        directories_named = True
        files_named = True
    directory_count = 4
    file_count = 493
    directory_names_length = 50
    file_names_length = 14839
    file_flags = Container:
>>>
>>> # print last directory block, containing 1 file record
...
>>> print(archive.container.directory_blocks[-1])
Container:
    name = u'meshes\\mps\x00' (total 11)
    file_records = ListContainer:
        Container:
            hash = 16183754957220078963
            size = 2384
            offset = 25094933
>>>
>>> # print archived filenames (only first 5, 488 more)
...
>>> print(archive.container.file_names)
ListContainer:
    _camp_objectplacementindicatorthread01.psc
    _camp_objectplacementindicatorthread02.psc
    _camp_objectplacementindicatorthread03.psc
    _camp_tentsitlayscript.psc
    campcampfire.psc
    ...
>>>
>>> # extract archive to directory
...
>>> archive.extract('/home/USER/Downloads')

BA2

Bethesda's second archive structure (used in Fallout 4).

BTDX archives (BA2) are harder to extract than their previous version BA2.

The two available archive subtypes are both supported.

General (GNRL)

Used to store generic files in a compressed/bundled file.

>>> from bethesda_structs.archive.btdx import BTDXArchive
>>> archive = BTDXArchive.parse_file('/media/sf_VMShared/ba2/CheatTerminal - Main.ba2')
>>> print(archive)
BTDXArchive(filepath=PosixPath('/media/sf_VMShared/ba2/CheatTerminal - Main.ba2'))
>>>
>>> # print archive header
...
>>> print(archive.container.header)
Container:
    magic = b'BTDX' (total 4)
    version = 1
    type = u'GNRL' (total 4)
    file_count = 982
    names_offset = 3600179
>>>
>>> # print first archive file entry
...
>>> print(archive.container.files[0])
Container:
    hash = 153050373
    ext = u'pex' (total 3)
    directory_hash = 1081231424
    offset = 35376
    packed_size = 0
    unpacked_size = 887
>>>
>>> # extract archive to directory
...
>>> archive.extract('/home/USER/Downloads')

Direct Draw (DX10)

Used to store (specifically) Microsoft Direct Draw textures.

>>> from bethesda_structs.archive.btdx import BTDXArchive
>>> archive = BTDXArchive.parse_file('/media/sf_VMShared/ba2/AK74m - Textures.ba2')
>>> print(archive)
BTDXArchive(filepath=PosixPath('/media/sf_VMShared/ba2/AK74m - Textures.ba2'))
>>>
>>> # print archive header
...
>>> print(archive.container.header)
Container:
    magic = b'BTDX' (total 4)
    version = 1
    type = u'DX10' (total 4)
    file_count = 116
    names_offset = 329069673
>>>
>>> # print first archive file entry
...
>>> print(archive.container.files[0])
Container:
    header = Container:
        hash = 362144756
        ext = u'dds' (total 3)
        directory_hash = 1416395408
        chunks_count = 4
        chunk_header_size = 24
        height = 2048
        width = 2048
        mips_count = 12
        format = 99
    chunks = ListContainer:
        Container:
            offset = 11136
            packed_size = 2714729
            unpacked_size = 4194304
            start_mip = 0
            end_mip = 0
        Container:
            offset = 2725865
            packed_size = 840614
            unpacked_size = 1048576
            start_mip = 1
            end_mip = 1
        Container:
            offset = 3566479
            packed_size = 217598
            unpacked_size = 262144
            start_mip = 2
            end_mip = 2
        Container:
            offset = 3784077
            packed_size = 71579
            unpacked_size = 87408
            start_mip = 3
            end_mip = 11
>>>
>>> # extract archive to directory
...
>>> archive.extract('/home/USER/Downloads')

bethesda-structs's Issues

F04 Plugin Parsing

TES3 Plugin Parsing

F03 Plugin Parsing

fails to extract Cathedral Weathers - Textures.bsa

Hi,

Just wanted to report a problem. I tried to use bethesda-structs to extract Cathedral Weathers - Textures.bsa from the Cathedral Weathers and Seasons mod, and ran into an lz4-related crash.

My python:

(venv) PS D:\git\skyrim-builder> python
Python 3.7.2 (tags/v3.7.2:9a3ffc0492, Dec 23 2018, 23:09:28) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>

My virtual environment:

(venv) PS D:\git\skyrim-builder> pip list
Package                Version    Location
---------------------- ---------- -----------------------------
atomicwrites           1.3.0
attrs                  19.1.0
bethesda-structs       0.1.4
cached-property        1.5.1
certifi                2020.4.5.1
chardet                3.0.4
Click                  7.0
colorama               0.4.1
construct              2.9.45     
flake8                 3.6.0
gitdb2                 2.0.5
GitPython              2.1.11
idna                   2.9
Jinja2                 2.10
lxml                   4.3.0
lz4                    3.0.2
MarkupSafe             1.1.1
mccabe                 0.6.1
more-itertools         7.0.0
multidict              4.5.2
pip                    18.1
pluggy                 0.6.0
py                     1.8.0
pycodestyle            2.4.0
pyflakes               2.0.0
pyfomod                1.2.1      d:\git\pyfomod\src
pynxm                  0.1.0
pytest                 3.6.1
pywin32                227
PyYAML                 5.1.1
requests               2.23.0
setuptools             40.6.2
six                    1.12.0
skyrim-builder         0.1        d:\git\skyrim-builder
skyrim-package-manager 0.1        d:\git\skyrim-package-manager
skyrimbuilder          0.1
smmap2                 2.0.5
spm                    0.1
tabulate               0.8.6
tqdm                   4.46.0
urllib3                1.25.9
websocket-client       0.57.0

My sample repro script:

import click
from pathlib import Path
from bethesda_structs.archive import BSAArchive


@click.command()
@click.argument('bsa')
@click.argument('dest')
def main(bsa, dest):
    bsa = Path(bsa)
    dest = Path(dest)
    assert bsa.exists()
    assert dest.exists()

    print(bsa)
    print(dest)

    archive = BSAArchive.parse_file(str(bsa))
    print(archive.container.header)
    print(archive.container.directory_records)
    archive.extract(str(dest))


if __name__ == '__main__':
    main()

The output:

(venv) PS D:\git\skyrim-builder> python .\test.py 'C:\users\{user redacted}\desktop\Cathedral Weathers - Textures.bsa' C:\users\{user redacted}\desktop\dest       
C:\users\{user redacted}\desktop\Cathedral Weathers - Textures.bsa
C:\users\{user redacted}\desktop\dest
Container: 
    magic = b'BSA\x00' (total 4)
    version = 105
    directory_offset = 36
    archive_flags = Container:
        directories_named = True
        files_named = True
        files_compressed = True
        files_prefixed = True
    directory_count = 2
    file_count = 24
    directory_names_length = 30
    file_names_length = 368
    file_flags = Container:
        dds = True
ListContainer:
    Container:
        hash = 7911102381315617907
        file_count = 8
        name_offset = 452
    Container:
        hash = 13447612162718329721
        file_count = 16
        name_offset = 598
Traceback (most recent call last):
  File ".\test.py", line 25, in <module>
    main()
  File "D:\git\skyrim-builder\venv\lib\site-packages\click\core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "D:\git\skyrim-builder\venv\lib\site-packages\click\core.py", line 717, in main
    rv = self.invoke(ctx)
  File "D:\git\skyrim-builder\venv\lib\site-packages\click\core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "D:\git\skyrim-builder\venv\lib\site-packages\click\core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File ".\test.py", line 21, in main
    archive.extract(str(dest))
  File "D:\git\skyrim-builder\venv\lib\site-packages\bethesda_structs\archive\_common.py", line 170, in extract
    archive_files = list(self.iter_files())
  File "D:\git\skyrim-builder\venv\lib\site-packages\bethesda_structs\archive\bsa.py", line 256, in iter_files
    file_record.offset + (file_record.size & self.SIZE_MASK)
  File "D:\git\skyrim-builder\venv\lib\site-packages\construct\core.py", line 304, in parse
    return self.parse_stream(io.BytesIO(data), **contextkw)
  File "D:\git\skyrim-builder\venv\lib\site-packages\construct\core.py", line 316, in parse_stream
    return self._parsereport(stream, context, "(parsing)")
  File "D:\git\skyrim-builder\venv\lib\site-packages\construct\core.py", line 328, in _parsereport
    obj = self._parse(stream, context, path)
  File "D:\git\skyrim-builder\venv\lib\site-packages\construct\core.py", line 1979, in _parse
    subobj = sc._parsereport(stream, context, path)
  File "D:\git\skyrim-builder\venv\lib\site-packages\construct\core.py", line 328, in _parsereport
    obj = self._parse(stream, context, path)
  File "D:\git\skyrim-builder\venv\lib\site-packages\construct\core.py", line 2468, in _parse
    return self.subcon._parsereport(stream, context, path)
  File "D:\git\skyrim-builder\venv\lib\site-packages\construct\core.py", line 328, in _parsereport
    obj = self._parse(stream, context, path)
  File "D:\git\skyrim-builder\venv\lib\site-packages\construct\core.py", line 3593, in _parse
    return sc._parsereport(stream, context, path)
  File "D:\git\skyrim-builder\venv\lib\site-packages\construct\core.py", line 328, in _parsereport
    obj = self._parse(stream, context, path)
  File "D:\git\skyrim-builder\venv\lib\site-packages\construct\core.py", line 715, in _parse
    return self._decode(obj, context, path)
  File "D:\git\skyrim-builder\venv\lib\site-packages\bethesda_structs\archive\bsa.py", line 48, in _decode
    return lz4.frame.decompress(obj)
RuntimeError: LZ4F_getFrameInfo failed with code: ERROR_frameType_unknown

The lz4 stuff are beyond me to understand. Looking at other BSA extractors, the Bethesda Archive Extractor page has in its version history a line that says "0.07 - Corrected the size sent to LZ4 for decompression, which affected only a very small number of files." Maybe this is related?

Confirmed that the same bsa file can be successfully extracted through the Bethesda Archive Extractor (above mentioned GUI tool) and the BSA Browser, which comes with a cli utility that I can use for the time being. Still though, it would be nice to be able to code this in native python at some point.

Anyway, thanks for your library!

construct.core.MappingError: building failed, unknown label: {'DDPF_ALPHA': True, 'DDPF_RBG': True}

Not sure if this project has been abandoned or not but I figured I might as well give it a shot.

Expected Behavior

Extract .BA2 archive to destination.

Current Behavior

Possible Solution

Might be my version of python? Not sure because i tried the same code with both 3.7.4 and 3.6.8. Might be windows 10? 64bit python?

Steps to Reproduce (for bugs)

1. I run the following code using my pipenv shell
2. Using fallout 4's "Fallout4 - Textures9.ba2" archive to test

Context

Trying to get all the .ba2 files in src folder and extract them to dst folder.

My Environment

• Bethesda Structs Version: 0.1.3
• Python Version: 3.6.8 and 3.7.4
• Operating System and Version: Windows 10 Version 10.0.18362 Build 18362

Plugin Subrecord Parsing

The most critical and logically difficult problem for parsing plugins is determining the subrecord structure to use for parsing the subrecord. This is difficult due to how nested subrecord collections are allowed including the options to be optional and multiple.

• Handle flat subrecord structures
• Handle "optional" subrecord structures
• Handle "multiple" subrecord structures
• Handle infinitely nested subrecord collections
• Handle "optional" subrecord collections
• Handle "multiple" subrecord collections

A quickie post about the issue: http://stephen.bunn.io/blog/2018/07/02/parsing-subrecords/

FNV Plugin Parsing

Fallout: New Vegas support for parsing subrecord structures for various record types in .esp, .esm, and .esl plugins.

Records can be built mostly from fopdoc.

• Plugin Structure
• Group Structure
• Record Structure
• Subrecord Structure
• Subrecord Definition
• Subrecord Collections

---

• ACHR - Placed NPC
• ACRE - Placed Creature
• ACTI - Activator
• ADDN - Addon Node
• ALCH - Ingestible
• ALOC - Media Relation Controller
• AMEF - Ammo Effect
• AMMO - Ammunition
• ANIO - Animated Object
• ARMO - Armor
• ARMA - Armor Addon
• ASPC - Acoustic Space
• AVIF - Actor Value Information
• BOOK - Book
• BPTD - Body Part Data
• CAMS - Camera Shot
• CCRD - Caravan Card
• CDCK - Caravan Deck
• CELL - Cell
• CHAL - Challenge
• CHIP - Casino Chip
• CLAS - Class
• CLMT - Climate
• CMNY - Caravan Money
• COBJ - Constructable Object
• CONT - Container
• CPTH - Camera Path
• CREA - Creature
• CSNO - Casino
• CSTY - Combat Style
• DEBR - Debris
• DEHY - Dehydration Stage
• DIAL - Dialog Topic
• DOBJ - Default Object Manager
• DOOR - Door
• ECZN - Encounter Zone
• EFSH - Effect Shader
• ENCH - Object Effect
• EXPL - Explosion
• EYES - Eyes
• FACT - Faction
• FLST - FormID List
• FURN - Furniture
• GLOB - Global
• GMST - Game Setting
• GRAS - Grass
• HAIR - Hair
• HDPT - Head Part
• HUNG - Hunger Stage
• IDLE - Idle Animation
• IDLM - Idle Marker
• IMGS - Image Space
• IMAD - Image Space Modifier
• IMOD - Item Mod
• INFO - Dialog Response
• INGR - Ingredient
• IPCT - Impact
• IPDS - Impact Data Set
• KEYM - Key
• LAND - Landscape
• LGMT - Lighting Template
• LIGH - Light
• LSCR - Load Screen
• LSCT - Load Screen Type
• LTEX - Landscape Texture
• LVLC - Leveled Creature
• LVLI - Leveled Item
• LVLN - Leveled NPC
• MESG - Message
• MGEF - Base Effect
• MICN - Menu Icon
• MISC - Misc. Item
• MSET - Media Set
• MSTT - Moveable Static
• MUSC - Music Type
• NAVI - Navigation Mesh Info Map
• NAVM - Navigation Mesh
• NOTE - Note
• NPC_ - Non-Player Character
• PACK - Package
• PERK - Perk
• PGRE - Placed Grenade
• PMIS - Placed Missile
• PROJ - Projectile
• PWAT - Placeable Water
• QUST - Quest
• RACE - Race
• RADS - Radiation Stage
• RCCT - Recipe Category
• RCPE - Recipe
• REFR - Placed Object
• REGN - Region
• REPU - Reputation
• RGDL - Ragdoll
• SCOL - Static Collection
• SCPT - Script
• SLPD - Sleep Deprivation Stage
• SOUN - Sound
• SPEL - Actor Effect
• STAT - Static
• TACT - Talking Activator
• TERM - Terminal
• TES4 - Plugin info
• TREE - Tree
• TXST - Texture Set
• VTYP - Voice Type
• WATR - Water
• WEAP - Weapon
• WRLD - Worldspace
• WTHR - Weather

PEX Decompilation

Support decommpilation of PEX compressed scripts back into the source Papyrus.
The best resource for this structure is from UESP.

• Header struct
• String table
• Debug info
• Support decompilation to Papyrus

Save Parsing

I would like to have the ability to parse Bethesda's save format.
Should be a contrib type parser

• Haven't looked into it yet

BSL Workspaces

Support for Building / Writing

As a hobby project, i'm trying to write a python script to automate some transformations on esm / esp files. Your library looks excellent, however after a bit of reading I figured out that there is no write support (or build support in construct terms). I figured I could do something like plugin.plugin_struct.build_file(plugin.container, "Test.esm"). This does produce a valid file, and if you change something like plugin.container.header.version it does show in Test.esm. However, if something in a subrecord is changed (e.g. the CNAM author name in the header), this does not propagate to Test.esm.
What changes would be necessary to enable writing esm's?
Is this a feature you are planning / would be willing to implement?

TES5 Plugin Parsing

TES4 Plugin Parsing