kwsch / flatcrawler

Data crawling utility tool & library to reverse engineer FlatBuffer binaries

FlatCrawler

Data crawling utility tool & library to reverse engineer FlatBuffer binaries with undocumented schemas, programmed in C#.

Loading a FlatBuffer binary into the console application will allow manual traversal through the serialized objects, and displays a terminal interface for program input.

An example manual parse of a FlatBuffer used by Pokémon Sword/Shield is provided in the sandbox project. For more detailed information on how to use this project, refer to the Wiki.

Screenshots

Building

FlatCrawler is a .NET 8.0 application.

The executable can be built with any compiler that supports C# 11.

The .sln can be opened with IDEs such as Visual Studio.

flatcrawler's Issues

Support FlatBuffer's Inline Structs

Currently this isn't supported because the current implementation for 'structs' assume they are arrays with a reference field offset and VTable.

Inlined structs are more like value types that contain a byte buffer of 'struct' size.
This byte buffer has a wrapper type defining the layout of the byte buffer (aka. the struct), but without any vtables to help figure out the layout.

I have partial class laid out, but didn't have the time to complete it yet.

using System;
using System.Diagnostics;

namespace FlatCrawler.Lib;

/// <summary>
/// Node that contains an inlined serialized fixed size value struct.
/// </summary>
/// <inheritdoc cref="FlatBufferFieldValue&lt;T&gt;"/>
public sealed record FlatBufferInlineStruct : FlatBufferNodeField, ISchemaObserver
{
    private DataRange StructDataMemory => new(Offset..(Offset + Size), $"{TypeName}[{Size}]", true);

    // TODO:
    // Add struct type as command arg
    // Add command to reset struct layout
    // Make struct editable, currently is a problem because the tree node printer expects entry count. Which is unknown for structs
    // - structs start with fixed size
    // - user can fill in types
    // - struct will auto-adjust entries and size
    // - Structs can't hold reference types, but can hold other inline structs
    // Possibly I can just append one ghost entry to the tree.
    // When `rf idex type` is called on this entry, it will create a real entry in place of the ghost one and create a new ghost entry when there's any space left
    // The real entry is used to layout the struct
    // FBStruct can be added to keep track of the members and field sizes.
    

    // TODO: Structs can't contain reference types. The class type should reflect that
    public FBClass StructInfo => (FBClass)FieldInfo.Type;

    public override string TypeName { get => $"{StructInfo.TypeName}[{Size}]"; set { } }

    private FlatBufferInlineStruct(FlatBufferFile file, int offset, int dataTableOffset, FlatBufferNode? parent) :
        base(offset, null, dataTableOffset, parent)
    {
        FieldInfo = new FBFieldInfo { Type = new FBClass(file) };
        RegisterStruct();
    }

    ~FlatBufferInlineStruct()
    {
        UnRegisterStruct();
    }

    public override void TrackChildFieldNode(int fieldIndex, ReadOnlySpan<byte> data, TypeCode code, bool asArray, FlatBufferNode childNode)
    {
        StructInfo.SetMemberType(this, fieldIndex, data, code, asArray);

        ref var member = ref Fields[fieldIndex];
        member?.UnRegisterMemory();

        member = childNode;
        childNode.TrackFieldInfo(StructInfo.Members[fieldIndex]);
        childNode.RegisterMemory();
    }

    public override void RegisterMemory()
    {
        FbFile.SetProtectedMemory(StructDataMemory);
        StructInfo.RegisterMemory();
    }

    public override void UnRegisterMemory()
    {
        FbFile.RemoveProtectedMemory(StructDataMemory);

        if (FieldInfo.Type is FBClass c)
            c.UnRegisterMemory();
    }

    public override void TrackFieldInfo(FBFieldInfo sharedInfo)
    {
        UnRegisterStruct();
        FieldInfo = sharedInfo;
        RegisterStruct();
    }

    public override void TrackType(FBFieldInfo sharedInfo)
    {
        UnRegisterStruct();
        FieldInfo = FieldInfo with { Type = sharedInfo.Type, Size = sharedInfo.Size };
        RegisterStruct();
    }

    /// <summary>
    /// Update all data based on the ObjectClass
    /// </summary>
    private void RegisterStruct()
    {
        StructInfo.AssociateVTable(VTable);

        Fields = new FlatBufferNode[StructInfo.Members.Count];

        StructInfo.Observers.Add(this);
    }

    /// <summary>
    /// Remove event associations
    /// </summary>
    private void UnRegisterStruct()
    {
        StructInfo.Observers.Remove(this);
    }

    public void OnMemberCountChanged(int count)
    {
        if (Fields.Length == count)
            return;
        var tmp = new FlatBufferNode[count];
        Fields.CopyTo(tmp, 0);
        Fields = tmp;
    }

    public void OnMemberTypeChanged(MemberTypeChangedArgs e)
    {
        Debug.WriteLine($"Changing Member Type: {e.MemberIndex} {e.OldType} -> {e.NewType}");
        if (!HasField(e.MemberIndex))
            return;

        ref var member = ref Fields[e.MemberIndex];
        member?.UnRegisterMemory();

        member = ReadNode(e.MemberIndex, e.Data, e.NewType.Type, e.FieldInfo.IsArray);
        member.TrackFieldInfo(e.FieldInfo);
        member.RegisterMemory();
    }

    public static FlatBufferInlineStruct Read(int offset, FlatBufferNodeField parent, int fieldIndex, ReadOnlySpan<byte> data, TypeCode type)
    {
        // TODO: Set correct params for data 
        return new FlatBufferInlineStruct(parent.FbFile, offset, 0, parent);
    }

    /// <summary>
    /// Reads a new table node from the specified data.
    /// </summary>
    /// <param name="parent">The parent node.</param>
    /// <param name="fieldIndex">The index of the field in the parent node.</param>
    /// <param name="data">The data to read from.</param>
    /// <param name="type">The type of the array.</param>
    /// <returns>New child node.</returns>
    public static FlatBufferInlineStruct Read(FlatBufferNodeField parent, int fieldIndex, ReadOnlySpan<byte> data, TypeCode type) => Read(parent.GetFieldOffset(fieldIndex), parent, fieldIndex, data, type);
}

Integrate analyzer with new data protection feature

I added the memory protection feature I talked about before.

I thought it could be nice to integrate it into the analyzer.
For example, bin\pokemon\data\poke_misc.bin has an object member field (field ID 10) that could be a whole range of types according to the analyzer.

By filling in some of the known types that extend the range of the data table we can remove a lot of that uncertainty.

Field 9 is clearly a string so I run rf 9 string.
Now running the new debug feature I added under the command 'v', you can see that we basically have all of the data filled out already:

.

Most unknown data ranges are of size 20, so know we know for sure field 10 can't be a ulong[] (since 4 (entries) * 8 (sizeof(ulong)) = 32).

Running the command again doesn't (yet) take this into account.

.

But the new data protection feature doesn't let you enter data that would be out of bounds any longer:

c# version issues

I was trying to reverse the flatbuffers, and I saw your project, it was so helpful to me that it makes it possible to use it without the source protocol files. Similar to proto-raw_data. But there is one point that the net6 version is too high for me to use. Can you give me a lower version to use, thank you

Problems using the project

Hello. I have attempted to use the project with the following file:

https://www.mediafire.com/file/mt04qhvv17a3ic7/test.bin/file

Though, it does not show any structure at all. Does the project require something to do in order for it to work?

Thanks for your time.

Manually link class types

It should be possible to manually link two objects that might share a class.
For example:

vec3f { float x, y, z }

table Transform {
  vec3f position;
  vec3f rotation;
  vec3f scale;
}

table TransformArray {
  Transform[] transforms;
}

Position, rotation and scale share the exact same class, but currently each field needs to be filled out manually.

I'm thinking a solution could be:
Using t [field_id] <type_name> when <type_name> already exists, could prompt the user if they would like to link the class type to the field.