bl-sdk / pythonsdk Goto Github PK

For some reason, .__setitem__ isn't working in Python for my custom TArray type.

When running obj dump, arrays get truncated at 100 elements. It is occasionally useful to view more than this. Since we already create a console key and enable set commands, it would be nice if we also automatically removed this limit.

From this reference:

Patch (this one also works for TPS):
7E 05 B9 64 00 00 00 3B F9 0F 8D
to
75 05 B9 64 00 00 00 3B F9 0F 8D
If you'd also like to remove the message ... XY more elements
Patch this aswell:
0F 8C 7B 00 00 00 8B 8D 9C EE FF FF 83 C0 9D 50
to
0F 85 7B 00 00 00 8B 8D 9C EE FF FF 83 C0 9D 50
TPS Specific
To remove the message mentioned above - patch:
7C 7B 8B 8D 94 EE FF FF
to
75 7B 8B 8D 94 EE FF FF

The SDK supports calling function with out params, by returning the changed parameters in a list.

ret, vehicle = PC.IsUsingVehicleEx(True)

As the title says, if the parameter is a struct or an array, it returns a garbage pointer, which is likely to cause a crash when you do anything else to it.

unrealsdk.Log(PC.Pawn.InvManager.GetItemList())

This is because these two are complex value types, which rely on memory in the params struct. This struct is zero'd and freed shortly after the function call, leaving us with garbage pointers. Fixing this simply requires working out how to make copies of these types, and how to allocate new memory for them.

Unreal allows for optional arguments to functions, these aren't dumped with the SDK generators. This functionality is needed for avoiding incredibly verbose functions.

The SDK codebase is a mess of code styles, since it's been passed between so many authors without proper standardization. We should define a standard, and format it to fix.

Tracking:

• Decide on convention and create config for editorconfig/clang format/others
• C++ Code formatting
  • Github check for formatting
• One class/module per file
• Document all functions/types
  • Github check for documentation

https://cdn.discordapp.com/attachments/580989187336044554/588502476819529728/unknown.png

TArrays are more complicated than a simple struct. Simply malloc and freeing them causes issues, so we need to hook into the init from UE.

We have a logging module, but it's very simple and kind of confusing. We should use a dedicated library instead. Ideally:

• Logging goes to file at generally low log level (not debug by default though), with standard logging format.
• Logging goes to console at higher log level, and with less verbosity (just something like [<module>] message).
• unrealsdk.Log in Python takes a module and integrates directly with logging system.
• stdout in Python goes to file as in normal logging, but goes directly to console without special formatting.
• Python functions exist to change log level of file + console separately (/ log format?).

Trying to assign any FArray object into an array field crashes the game. This means rather unintuitively, the following example will cause a crash:

obj1 = bl2sdk.FindObject("MissionDefinition", "GD_Anemone_Side_VaughnPart2.M_Anemone_VaughnPart2")
obj2 = bl2sdk.FindObject("WeaponTypeDefinition", "GD_Weap_Pistol.A_Weapons.WeaponType_Jakobs_Pistol")
obj3 = bl2sdk.FindObject("SkillDefinition", "GD_Assassin_Skills.Bloodshed.Execute")
obj4 = bl2sdk.FindObject("ItemPoolDefinition", "GD_CustomItemPools_Lilac.Psycho.BlueBold")

# Each line here causes a crash
obj1.Dependencies = obj1.Dependencies
obj2.WeaponIdleAnimations = obj2.WeaponIdleAnimations
obj3.SkillConstraints = obj3.SkillConstraints
obj4.BalancedItems = obj4.BalancedItems

If you convert the FArray into a list beforehand it will correctly be set, and the game won't crash. Not all fields support being assigned from a list however.

obj1 = bl2sdk.FindObject("MissionDefinition", "GD_Anemone_Side_VaughnPart2.M_Anemone_VaughnPart2")
obj1.Dependencies = list(obj1.Dependencies)

Hi there. Been trying to play around with some BL2 mods to spice it up a bit more. Trying to achieve something simple before going in on other mods: use apple1417's SDK Autorun mod to load up UCP on launch. But I'm hitting the snag as described in the title. If ddraw.dll is there, the game briefly shows the splash screen before crashing completely. If it's not there, the game loads up perfectly fine with no issues. I've followed and checked all the various (now kind of outdated) guides and I've done everything right as far as I can see. The issue is purely down to ddraw. I figured maybe the Valorant anti-cheat thing might be preventing it, so I closed that, and nope it still crashes.

Sitting next to the python files in the Binaries/Win32 folder is this log file, which reads thus:

======= Borderlands 2 Python Loader =======
[AntiDebug] Hook added for NtSetInformationThread
[AntiDebug] Hook added for NtQueryInformationProcess
Found EXE name as 'Borderlands2.exe'
[Internal] GObjects = 0x019CAB20
[Internal] GNames = 0x01988744
[Internal] UObject::ProcessEvent() = 0x004F27F0
[Internal] UObject::CallFunction() = 0x004F2300
[Internal] FFrame::Step() = 0x004E31B0
[Internal] UObject::StaticConstructObject() = 0x00429D20
[Internal] UObject::LoadPackage() = 0x0042C8C0
[Internal] GMalloc = 0x0193A8F4
[Internal] FindOrCreateFName = 0x003F08D0
[Internal] GetDefaultObject = 0x003C1F20
Sigscan failed (Signature not found, Mask = FF 83 C4 0C 85 C0 75 1A 6A 01 8D)WINAPI Error when enabling 'SET' commands: 487
Plugin loaded: d
[AntiDebug] NtSetInformationThread called with ThreadHideFromDebugger

I've searched up and down for some kind of fix or alternative solution to this problem and the most I found was just another random mod that acknowledges ddraw.dll can prevent the game from starting up. Any ideas?

This code causes a consistent crash when the hooked function is run (shortly after loading into a map with vendors).

def Log(caller: unrealsdk.UObject, function: unrealsdk.UFunction, params: unrealsdk.FStruct) -> bool:
    unrealsdk.Log(params)
    
unrealsdk.RunHook("WillowGame.PopulationFactoryVendingMachine.CreateInteractiveObject", "test", Log)

The function signature (and thus what the params struct should contain) is as follows:

protected function WillowInteractiveObject CreateInteractiveObject(PopulationMaster Master, Vector SpawnLocation, Rotator SpawnRotation, int GameStage, int AwesomeLevel, InteractiveObjectDefinition SelectedObjectDefinition) {

Hi, i am a player who use Simplified Chinese Language
There is a problem when adding pythonSDK to borderlands 2

Main Menu
Mod Control Settings
Keyboard Control Settings

Using Proton-7.1-GE-2, with launch option: "WINEDLLOVERRIDES="ddraw=n,b" bash -c 'exec "${@/Launcher.exe/Borderlands2.exe}"' -- %command%".

OS: Ubuntu 20.04.4 LTS

The game crashes when I open Borderlands 2 with the launch options mentioned.

FArray needs +=, .append, and .insert` support

FArray = FArray and FStruct = FStruct also needs to be added

When assigning strings to objects' properties using the SDK, the results are as expected, and the changes appear in game as desired:

namePart = bl2sdk.FindObject(
    "ItemNamePartDefinition",
    "GD_ClassMods.Prefix_Assassin.Prefix_Infiltrator_00_Infiltrator"
)
namePart.PartName = "Infilbooper"

However, when the same property is subsequently altered elsewhere in the engine (specifically via a set console command), it triggers a crash:

bl2sdk.GetEngine().GamePlayers[0].Actor.ConsoleCommand(
    "set GD_ClassMods.Prefix_Assassin.Prefix_Infiltrator_00_Infiltrator PartName Infiltrator",
    0
)

We should setup automated builds so we can tell when errors are introduced. We can then also use it as part of branch protection, to prevent merging errors.

We should also just generally have branch protection requiring PRs/preventing force pushing on bl3 and master.

Similar to dynamic objects, this will completely unlock the SDK to work in any version of BL2, past, present, or future.

System info: Arch Linux x86_64 5.10.13-zen1-2-zen
Proton Version: 5.0-10 Also tested with Proton-GE-5.21
Issue: When trying to edit mod settings and keybinds through in-built menu, clicking/hovering on the links does nothing

Latest Python-SDK is installed 0.7.8

Tested with only default mods installed no extras, enabled Borderlands 2 Map Reloader for keybinds and Backpack Manager for mod options. When hovering or clicking on the button error messages are output to python-sdk.log.

At:
Z:\home\michael.local\share\Steam\steamapps\common\Borderlands 2\Binaries\Win32\Mods\ModMenu\KeybindManager.py(231): _HandleSelectionChangeRollover
SystemError: <built-in method next of PyCapsule object at 0x16CEB320> returned NULL without setting an error

At:
Z:\home\michael.local\share\Steam\steamapps\common\Borderlands 2\Binaries\Win32\Mods\ModMenu\OptionManager.py(101): _WillowScrollingListOnClikEvent
SystemError: <built-in method next of PyCapsule object at 0x16CEB320> returned NULL without setting an error

Full python-sdk.log

We currently pull all our files in the top level of the zip, meaning users must manually navigate to <game>\Win32\Binaries to extract them. We should add these two folders, so you can directly extract the zip over the game folder, similarly to modding a lot of other games.

Doing this is a very simple change of course, this issue exists more a reminder to update all the documentation and tutorials currently telling people to navigate.

I saw streamers using it, how can i make it work?

There's likely a flag that can be passed to the function that will fix this. Alternatively, we may be able to load just the needed objects from the package.

Title

Instead of printing out <UObject at 0x00000>, it would be nice if it would instead call UObject.GetFullName() and show that instead.

If you have a reference to an object in Python and the object gets garbage collected in game, then trying to access the object in Python again will sometimes cause crashes.

Here is some example code to cause the crash.

import bl2sdk

bl2sdk.LoadPackage("GD_Mercenary_Streaming_SF")
GCObj = bl2sdk.FindObject("SkillDefinition", "GD_Mercenary_Skills.Brawn.Incite")

# Use a hook so we can try reference the object later
def HookMainMenuInput(caller: bl2sdk.UObject, function: bl2sdk.UFunction, params: bl2sdk.FStruct) -> bool:
    if params.ukey == "F1":
        bl2sdk.Log(str(GCObj))
        bl2sdk.Log(str(GCObj.SkillName))
    return True

bl2sdk.RemoveHook("WillowGame.FrontendGFxMovie.SharedHandleInputKey", "GCDemo")
bl2sdk.RegisterHook("WillowGame.FrontendGFxMovie.SharedHandleInputKey", "GCDemo", HookMainMenuInput)

To test this code run it on the main menu, then wait for the next GC cycle and press F1. While waiting you can run getall SkillDefinition Name in console to see what objects are loaded. Once the object has been removed pressing F1 might crash the game. The crash is not consistent, but still relatively common. If the game doesn't crash then it will never crash using that reference (so re-execute the file), and the log file will contain the following.

>>> pyexec GCDemo.py <<<
[HookManager] (EngineHooks) ERROR: Failed to remove hook "GCDemo" for "WillowGame.FrontendGFxMovie.SharedHandleInputKey"
SkillDefinition GD_Mercenary_Skills.Brawn.Incite
Incite
SkillDefinition GD_Mercenary_Skills.Brawn.Incite
Incite
SkillDefinition GD_Mercenary_Skills.Brawn.Incite
None
SkillDefinition GD_Mercenary_Skills.Brawn.Incite
None


>>> pyexec GCDemo.py <<<
SkillDefinition GD_Mercenary_Skills.Brawn.Incite
Incite
SkillDefinition GD_Mercenary_Skills.Brawn.Incite
Incite

<Game Crashes>

Note that the hook fires on both key press and release, hence the double logging.

Removing the second log line sometimes gives the following output instead. I have never seen it happen with both log calls, but as the crash is inconsistent anyway it might still happen.

>>> pyexec GCDemo.py <<<
[HookManager] (EngineHooks) ERROR: Failed to remove hook "GCDemo" for "WillowGame.FrontendGFxMovie.SharedHandleInputKey"
SkillDefinition GD_Mercenary_Skills.Brawn.Incite
SkillDefinition GD_Mercenary_Skills.Brawn.Incite
SkillDefinition UnknownName.ByteProperty.Incite
SkillDefinition UnknownName.ByteProperty.Incite

<Game Crashes>

I believe making UObject::GetProperty() and UObject::SetProperty() check if the object has been GCed will probably prevent this, though the inconsistent nature of the crash may mean the issue is hidden deeper somewhere.

On a related note, the SDK doesn't have a good way for mods to check if an object has been GCed. Even if this crash gets fixed, mods that are working with objects that may be GCed will need to know this to prevent themselves from causing exceptions.

The console has a very powerful syntax WeaponPartDefinition'GD_Orchid_BossWeapons.AssaultRifle.AR_Barrel_Bandit_Rapier', that will get said object of said class. Mimicking this functionality will be crucial, and if we can add an ability to load said object if it's unloaded, we'll unlock a ton of potential.

I believe that this will be done via StaticLoadObject: http://api.unrealengine.com/INT/API/Runtime/CoreUObject/UObject/StaticLoadObject/index.html

Other important hooks: http://api.unrealengine.com/INT/API/Runtime/CoreUObject/UObject/StaticDuplicateObjectEx/index.html

http://api.unrealengine.com/INT/API/Runtime/CoreUObject/UObject/StaticFindObject/index.html

DynamicLoadObject doesn't seem to work as I would expect it to based on documentation.
https://wiki.beyondunreal.com/Legacy:DynamicLoadObject

Right now, script hooks get parameters via stack, and rely on the user to correctly pop items off of the stack. This introduces a lot of risk for crashing. There is likely some way of making this safe and doing that work before passing it back to the user by analyzing the UFunction.

In this repo's current state, it only works for current patch. Older and future patches will not work, unless we start dynamically getting the class pointers.

pClassPointer = (UClass*)UObject::GObjObjects()->Data[2];

On init, we should be able to iterate through all objects in GObjObjects, run GetName() on them, and make a mapping of string names to indexes. This would let us dynamically get these class pointers.

We're currently using pybind 2.3.dev0, from mid-2019. We should upgrade. #68 is fixed by this, and there are likely other changes which make development better.

Having both of these adds needless complexity on the modder's end, for no extra functionality. These can be merged after #29, because we'll be able to dynamically pop all of the parameters off of the stack and put them into a struct for Python access via __getattr__.

The function signature should be:

def process_hook(caller: UObject, function: UFunction, params: FStruct) -> bool:

Notice the missing stack and result, as manipulating the stack should not be part of Python functionality. result (i believe) is just a pointer to the ReturnValue inside of the params struct, so that can be ignored as well.

As we remove stack, we'll use the return value from the function in the case of hkCallFunction calls, and automatically call SkipFunction if need be, otherwise we will return the stack to the proper place before continuing execution.

autoexec stuff

This'll be implemented after #31 / #38 as this heavily relies on consistent file directories.
We'll want to save the user's force set keybinds for plugins.
In the same vein, it'll probably be best for us to set a user's option values (#40)

1. Whenever the user leaves the keybind menu, we can transfer all of the enabled mod's keybinds into JSON.
2. Afterwards, save the JSON into either: Mods\Settings.json or we could give each mod their own Settings.json file like: Mods\[mod]\Settings.json.

The same concept applies onto the Slider/Boolean/Spinner option types in the PLUGINS submenu in the options menu.

Currently, the sigscans module must know the exact executable name in order to grab patterns. This makes using it on new games unnecessarily difficult, you have to recompile even if none of the patterns change.

Additionally, even within a single game we cannot guarantee executable name - briefly poking through the support server logs, I've seen Borderlands2.exe, borderlands2.exe and Launcher.exe. We've also previously seen GetModuleFileName returning corrupted names, though never been able to replicate.

Proposed solution is to store a list of patterns, and have each game store the best index. We can then iterate through all patterns if the game's unknown.

Whenever I launch the game with PythonSDK installed, the game will crash upon starting up on Windows 11. It's consistent with both BL2 and Pre-Sequel, without the modified binaries the game launches perfectly fine.

Currently the SDK requires the user to navigate a complex - and therefore error prone - assortment of files and folders for both installation and usage.

ddraw.dll
Plugins\
Plugins\python36.dll
Plugins\python36.zip
Plugins\pythonSDK.dll
Plugins\Python\
Plugins\Python\[mod].py

With even more complexity necessary to use mods that depend on additional files as resources:

Plugins\Python\[mod's library dependency].py
Plugins\Python\Textures\
Plugins\Python\Textures\[Texture file imported by mod].png

...etcetera.

A new approach is proposed in order to both simplify this for the user and allow developers freedom in bundling resources with their mods:

ddraw.dll
Mods\
Mods\[mod]\
Mods\[mod]\[mod resource]

In this proposed structure, ddraw.dll includes the libraries and resources the SDK itself is dependent on. (It may possibly also perform loading of additional DLLs located in Plugins\, if the user has any, to retain compatibility with existing plugins).

Besides installation of the SDK itself, this would also simply installation of mods, as the user would only be responsible for copying a single directory for each mod.

Mods will be adding new items, weapons, etc, and these are not currently persisted throughout saves. For these mods to be viable, we'll need to modify save files a bit.

My current proposal is to add a field that is only accessible in C++, that can make saves dependent on certain mods. Skyrim has some system whereby saves know which mods they were made with, and notifies the user that the save may not function properly without them.

This would need to be shown on the main menu when hovering over "Continue" and in the "Select Character" menu when hovering over a file.

"DLC" menu won't open without internet connection

We need a set of example/template mods showing how to do certain tasks, such as creating textures, creating weapons, creating hooks, modifying functions, and debugging.

As in the title, we should follow PEP8. Due to the difference in code style, this will also make it clearer what bits of code are referencing Python vs UnrealScript objects.

Additionally, we should ship a stubs file showing what we export.

https://answers.unrealengine.com/questions/23440/loading-textures-at-runtime.html

This might not be too hard?
StaticLoadObject(UTexture2D::StaticClass(), NULL, *(Path))

This means any mod trying to use them will not work in coop.

Basically all in the title. Issue comes from the selection change hook, it explicitly checks against the English name OPTIONS.

    if caller.MenuDisplayName == "OPTIONS":
        selectedIndex = params.TheList.GetSelectedIndex()
        Options.isMenuPluginMenu = (params.EventID == 0 and (selectedIndex == 5 or selectedIndex == 4))

https://github.com/bl-sdk/PythonSDK/blob/16a3eaf/Mods/OptionManager.py#L133

Please add Russian language support! And then in the settings for the assignment of keys, incomprehensible characters.

I've got a mod which I'm using to regenerate BLCMM data dumps, and the first step of that is looping through a ton of getall statements to find object names (pythonsdk itself still has some deficiencies in that regard, so I can't do so "directly" via the sdk yet). It runs 500 getall statements at a time, and then uses Tick to wait a bit, so we don't run into the Runaway loop detected (over 1000000 iterations) crash.

Anyway, on my machine, after about 18 of those chunks-of-500-getall-statements, I get a crash-to-desktop, with this on the console:

Fatal Python error: deallocating None

Current thread 0x00000028 (most recent call first):
  File "Z:\usr\local\games\Steam\SteamApps\common\Borderlands 2\Binaries\Win32\Mods\TooMuchConsole\__init__.py", line 71 in doGetall
  File "Z:\usr\local\games\Steam\SteamApps\common\Borderlands 2\Binaries\Win32\Mods\TooMuchConsole\__init__.py", line 37 in staticDoApocTick

abnormal program termination

That's from a pared-down version to reproduce the error, which can be found here: https://gist.github.com/bcd3911f378da7fc90b8bfa54c5826c1

Line 71 is just:

    pc.ConsoleCommand(command)

After loading the mod it'll start doing periodic dumps every 9 seconds or so; you can just remain on the main menu and let those happen. As I say, on my machine I get a crash about at the 18-iteration mark.

On my "real" app I've worked around it by just creating some files which I then execute with exec, instead of running all the getalls by hand, so to speak, so that works well enough for me for now, but it'd be nice to track this down.

Rather than defining python dirs and output dirs directly in the project, we should have a prebuild script setting up all needed defines, so that all user specific changes are in a single place.

Basically all in the title. Demo program:

import unrealsdk
unrealsdk.Log(__file__)

>>> pyexec test.py <<<
NameError: name '__file__' is not defined

At:
  D:\SteamLibrary\steamapps\common\Borderlands 2\Binaries\Win32\Mods\\test.py(3): <module>

>>> py from Mods import test <<<
'D:\\SteamLibrary\\steamapps\\common\\Borderlands 2\\Binaries\\Win32\\Mods\\test.py'

Workaround: cause an exception and extract the file path from it. Put this code near the top of the file:

if __name__ == "__main__":
    import sys
    try:
        raise NotImplementedError
    except NotImplementedError:
        __file__ = sys.exc_info()[-1].tb_frame.f_code.co_filename  # type: ignore

If you do:

py import bl2sdk
py PC = bl2sdk.GetEngine().GamePlayers[0].Actor
py bl2sdk.Log(PC.Location)

It'll log out { }
But after that command you do:

py bl2sdk.Log(PC.Location.X)

It'll return PC.Location's X property which while fine gets tedious when you need to write out all of the properties of a specific/unknown struct.