rojo-rbx / rbx-dom

Roblox DOM and (de)serialization implementation in Rust

rbx-dom

rbx-dom is a collection of cross-platform libraries that enables any software to interact with Roblox instances.

Documentation about the project is hosted at dom.rojo.space.

At this moment, we do not specify a MSRV for any project in rbx-dom. If you need to build one of these libraries with an outdated version of Rust, please open an issue explaining what blockers there are to you updating Cargo, what version you would need us to support, and why.

rbx_dom_weak

Weakly-typed Roblox DOM implementation. Defines types for representing instances and properties on them.

rbx_types

Contains Roblox's value types like Vector3 and NumberSequence. Used by crates like rbx_dom_weak and a future rbx_dom_strong crate to let them share types and conversions.

rbx_xml

Serializer and deserializer for for Roblox's XML model and place formats, rbxmx and rbxlx.

rbx_binary

Serializer and deserializer for for Roblox's binary model and place formats, rbxm and rbxl.

rbx_reflection

Roblox reflection types for working with Instances in external tooling.

rbx_reflection_database

Bundled reflection database using types from rbx_reflection. Intended for users migrating from rbx_reflection 4.x and users who need reflection information statically.

rbx_reflector

Command line utility to generate a reflection database for rbx_dom_lua and rbx_reflection_database.

rbx_util

Command line utility to convert and debug Roblox model files.

rbx_dom_lua

Roblox Lua implementation of DOM APIs, allowing Instance reflection from inside Roblox. Uses a data format that's compatible with rbx_dom_weak to facilitate communication with applications outside Roblox about instances.

Property Type Coverage

Property Type	Example Property	rbx_types	rbx_dom_lua	rbx_xml	rbx_binary
Axes	ArcHandles.Axes	✔	✔	✔	✔
BinaryString	Terrain.MaterialColors	✔	➖	✔	✔
Bool	Part.Anchored	✔	✔	✔	✔
BrickColor	Part.BrickColor	✔	✔	✔	✔
Bytecode	N/A	❌	⛔	❌	❌
CFrame	Camera.CFrame	✔	✔	✔	✔
Color3	Lighting.Ambient	✔	✔	✔	✔
Color3uint8	Part.BrickColor	✔	✔	✔	✔
ColorSequence	Beam.Color	✔	✔	✔	✔
Content	Decal.Texture	✔	✔	✔	✔
Enum	Part.Shape	✔	✔	✔	✔
Faces	Handles.Faces	✔	✔	✔	✔
Float32	Players.RespawnTime	✔	✔	✔	✔
Float64	Sound.PlaybackLoudness	✔	✔	✔	✔
Font	TextLabel.Font	✔	✔	✔	✔
Int32	Frame.ZIndex	✔	✔	✔	✔
Int64	Player.UserId	✔	✔	✔	✔
NumberRange	ParticleEmitter.Lifetime	✔	✔	✔	✔
NumberSequence	Beam.Transparency	✔	✔	✔	✔
OptionalCFrame	Model.WorldPivotData	✔	✔	✔	✔
PhysicalProperties	Part.CustomPhysicalProperties	✔	✔	✔	✔
ProtectedString	ModuleScript.Source	✔	✔	✔	✔
Ray	RayValue.Value	✔	✔	✔	✔
Rect	ImageButton.SliceCenter	✔	✔	✔	✔
Ref	Model.PrimaryPart	✔	✔	✔	✔
Region3	N/A	✔	✔	❌	❌
Region3int16	Terrain.MaxExtents	✔	✔	❌	❌
SecurityCapabilities	Folder.SecurityCapabilities	✔	❌	✔	✔
SharedString	N/A	✔	✔	✔	✔
String	Instance.Name	✔	✔	✔	✔
UDim	UIListLayout.Padding	✔	✔	✔	✔
UDim2	Frame.Size	✔	✔	✔	✔
UniqueId	Instance.UniqueId	✔	❌	✔	✔
Vector2	ImageLabel.ImageRectSize	✔	✔	✔	✔
Vector2int16	N/A	✔	✔	✔	❌
Vector3	Part.Size	✔	✔	✔	✔
Vector3int16	TerrainRegion.ExtentsMax	✔	✔	✔	✔
QDir	Studio.Auto-Save Path	⛔	⛔	⛔	⛔
QFont	Studio.Font	⛔	⛔	⛔	⛔

✔ Implemented | ❌ Unimplemented | ➖ Partially Implemented | ⛔ Never

Outcome

This project has unveiled a handful of interesting bugs and quirks in Roblox!

• GuiMain.DisplayOrder is uninitialized, so its default value isn't stable
• MaxPlayersInternal and PreferredPlayersInternal on Players are scriptable and accessible by the command bar
• Instantiating a NetworkClient will turn your edit session into a game client and stop you from sending HTTP requests
• ContentProvider.RequestQueueSize is mistakenly marked as serializable
• Trying to invoke game:GetService("Studio") causes a unique error: singleton Studio already exists
• Color3 properties not serialized as Color3uint8 would have their colors mistakenly clamped in the XML place format. This was bad for properties on Lighting.
• ColorSequence's XML serialization contains an extra value per keypoint that was intended to be used as an envelope value, but was never implemented.

For Maintainers

Cutting new releases is not currently as optimized as it should be. While we work on improving it, packages need to be published in a specific order to make sense. The order that currently works well is:

1. rbx_types
2. rbx_dom_weak and rbx_reflection
3. rbx_reflection_database
4. rbx_binary and rbx_xml

The process for publishing these is:

1. Decide a new version number, following SemVer
2. Update changelog to list new release under its own heading
3. Adjust versions of local dependencies to be the new release (this is why releases must happen in a specific order)
4. Increment version in Cargo.toml
5. Add a git tag in the format library_name-vMAJOR.MINOR.PATCH at the commit that incremented the Cargo version
6. Publish to Cargo

License

rbx-dom is available under the MIT license. See LICENSE.txt for details.

rbx-dom's Issues

Unknown classes cause obscure error

Sometimes classes aren't in the reflection database, like if they'd been added since the reflection database was generated or you're using a custom Studio build.

You'll get an error here:

rbx-dom/rbx_dom_lua/src/init.lua

Line 11 in f1dfe97

local propertyData = currentClass.properties[propertyName]

Make type conversions fail serialization/deserialization?

Right now, when rbx_xml fails to convert a value to the right type when serializing or deserializing, it silently lets the value continue as-is.

We should consider making it fail, or at least fail by default.

rbx_reflection should describe events

Necessary for Kampfkarren/selene#59.

rbx_dom_strong

Right now, we have a weakly-typed DOM implementation. It'd be cool to have a strongly-typed one, too.

rbx_xml is slow!

Using an rbxlx version of the open source version of Miner's Haven as a benchmark.

$ cat decode.lua
remodel.readPlaceFile("minershaven.rbxlx")

$ time remodel decode.lua
real    0m24.496s
user    0m0.000s
sys     0m0.000s

$ du -sh minershaven.rbxl minershaven.rbxlx
7.6M    minershaven.rbxl
143M    minershaven.rbxlx

We might be able to recover some time switching to quick-xml, according to some profiling done by @Kampfkarren. Further investment in the binary format is the end-all solution to this, however.

Lower generate_reflection plugin permissions to regular plugin

Right now the reflection code runs at a higher security level than most code consuming rbx_dom_lua.

We can fix this by either changing where we put the generated plugin, or porting the tool to use https://github.com/LPGhatguy/run-in-roblox.

There might be value in running the generation code at multiple security levels and merging the results in order to collect security information and paint a more complete picture.

Write metadata to models and places

The only piece of metadata in the engine is ExplicitAutoJoints right now. For the best compatibility and least-surprising behavior, we should always write that value to true in the metadata section of all formats.

Since there might be more metadata added to the engine related to content versioning, rbx_dom_weak and related packages should probably gear up for handling more metadata. This can happen either as a property of a tree or as a parameter to serialization.

One way to approach this would be to introduce a new member to WeakDom named metadata, perhaps of type HashMap<String, String>.

Neither rbx-xml nor rbx-binary properly add children

I'm not sure if I'm using it wrong, but rbx-xml doesn't seem to add anything other than the initial root node you create. Here was the code I was using:

lazy_static! {
    #[rustfmt::skip]
    static ref HATS_TREE: RbxTree = {
        let mut tree = RbxTree::new(RbxInstance {
            name: "Hats".to_string(),
            class_name: "Hats".to_string(),
            properties: HashMap::new(),
        });

        let root_id = tree.get_root_id();

        rbx_xml::decode(
            &mut tree,
            root_id,
            fs::File::open("./test/roblox.rbxmx").expect("Couldn't read roblox.rbxmx"),
        ).expect("rbxmx failed to parse");

        tree
    };
}

Followed by:

println!("{:?}", HATS_TREE.descendants(HATS_TREE.get_root_id()).collect::<Vec<_>>());

When this is used, it only shows the root node and no descendants.
[RootedRbxInstance { instance: RbxInstance { name: "Hats", class_name: "Hats", properties: {} }, id: RbxId(Uuid([15, 19, 159, 200, 130, 135, 75, 151, 140, 168, 23, 144, 220, 6, 153, 57])), children: [], parent: None }]

rbx-binary on the other hand shows everything as expected.

roblox.zip

rbx_xml Faces support

The Faces type is only used in the Handles instance.

rbx_xml Axes support

The Axes type is only used in the ArcHandles instance.

WebAssembly demo

One big advantage of using Rust as the backbone for a DOM crate like this is that it can target WebAssembly without lugging around a runtime or libc like Emscripten.

It should be pretty easy to create a binding library around the rbx-dom ecosystem using wasm-bindgen. This would open the door to having a browser-based model viewer and editor, which would be awesome!

RbxValue try_convert/try_convert_ref API

rbx_dom_weak should have an API to fudge the type of a value over to another value if possible. This is useful for increasing content compatibility in deserialization, as well as enabling backward-compatibility when the types of properties change.

A baseline API was introduced in #32, but it should be refined and documented.

rbx-binary and rbx-xml leave debug prints

https://github.com/LPGhatguy/rbx-tree/blob/e0783c23b58bef071b06119a18f7bfcb9ce052ac/rbx_binary/src/deserializer.rs#L45

Perhaps this could be an optional feature?

INF/NAN doesn't work for Vector2

Despite #33, INF/NAN does not work in Vector2 (and others with the same thought process) likely because they use the direct float parse methods instead of the one used for Float32/Float64.

https://github.com/LPGhatguy/rbx-dom/blob/master/rbx_xml/src/types/vectors.rs#L37

Support for INF/NAN in rbx_xml

BillboardGui's MaxDistance is INF and this presumably causes issues. I assume this happens for nan too.

Players service property mapping

• MaxPlayers serializes as MaxPlayersInternal
• PreferredPlayers serializes as PreferredPlayersInternal

Rework ref serialization in rbx_xml

Right now, refs serialized to XML are transformed to match how Roblox generates its GUID-like values:

https://github.com/LPGhatguy/rbx-dom/blob/e90b897f249c33fa366655e471c4366c7d1bc3cc/rbx_xml/src/types/ref.rs#L17-L31

Instead of doing this, we should probably be keeping a ref map and rewriting them to IDs as instances are deserialized, since we don't treat the instance's ref as meaningful!

Port I/O to accept Read/Write inputs by value

This might not make total sense since it'll result in massive monomorphization with varying depths of &mut reference and might cause lots of indirection.

https://rust-lang-nursery.github.io/api-guidelines/interoperability.html#generic-readerwriter-functions-take-r-read-and-w-write-by-value-c-rw-value

C API

It'd be great to make rbx-dom the go-to public implementation of the Roblox DOM and file formats. This would help cement that idea.

Conversions from BrickColor to Color3

This should be pretty straightforward in the context of RbxValue::try_convert_ref. I'm not sure if we want to bother trying to go the other direction since it's lossy.

SharedString in rbx_dom_weak

We should be able to create a shared string dictionary per tree. When moving instances between trees, it should be possible to add entries from the shared string table.

I'm not super sure how the entries should be cleaned up, especially since it's possible for consumers of rbx_dom_weak to mutate properties arbitrarily. Maybe that'll be left up to consumers? Maybe there could be some ref-counting nonsense?

Descendants iterator returns the given ID

This looks like it was an oversight.

It might be a small breaking change to fix this iterator, but I'm not super worried about it since it's generally only useful for debugging. It may affect Rojo.

rbx_reflection should describe methods

I was going to use rbx_reflection to generate a .luacheckrc-like file, and while methods are described in detail in the API dump, they're not accessible at all in rbx_reflection.

rbx_xml doesn't emit the right type when using serialized aliases

A good example of this is BasePart.Color3, which should serialize to a property called Color3uint8 of type Coloruint8. Instead, rbx_xml currently serializes it as a property named Color3uint8 with the type Color3 -- it missed the conversion!

This happens because we pull the type we want from the canonical property descriptor instead of the serialized name descriptor. If serialized_name() is Some, we should look up that property descriptor and convert to its type instead.

Configurable serializer/deserializer

It should be possible to configure both the serialization and deserialization methods in rbx_xml.

The only big option I think we need right now is the ability to turn off reflection-driven serialization.

This is a good opportunity to break the API to pick better names. I like the idea of picking names inspired by serde_json's API (but with a better name for the options struct):

• rbx_xml::from_str(tree: &mut RbxTree, id: RbxId, source: &str, options: DeOptions)
• rbx_xml::to_writer(output: &mut W, tree: &RbxTree, ids: &[RbxId], options: SeOptions)

Better serializer errors for rbx_xml

The errors given by rbx_xml and rbx_binary are terrible.

rbx_binary fails to deserialize BinaryString values

Roblox's binary format doesn't distinguish between strings meant for humans to read and "strings" that are just binary blobs.

This makes Rust angry, since the current version of rbx_binary tries to jam both into String, which is required to be valid UTF-8. We can fix this with reflection guidance for rbx_binary, but there may be a quicker fix (if the string isn't valid UTF-8, just throw it in as a BinaryString?).

Support inferring 'Content' type from string literals

Strings are already marked as an ambiguous case, since they can resolve to either a string or an enum variant.

We should also allow strings to resolve to Content, and potentially also BinaryString depending on how hip we're feeling.

Wrap BinaryString at 72 characters per line

It looks like Roblox's XML BinaryString decoder chokes on input that isn't wrapped at the 72 byte mark. I don't think it's an email client, but we should give it what it wants anyways.

Better error messages

I am currently getting this error:

Message("don\'t know how to decode this prop type")

This is a pretty bad error, it doesn't even say the property type.

Make rbx_reflection API less public

Currently the reflection database is entirely public fields. We should replace those with getters, since everything is read-only anyways.

This would be a good time to fix up the names of these structs as well to mention words like "reflection" and "descriptor" instead of just prefixing everything with Rbx.

rbx_xml empty BinaryString values should not include CDATA

Right now they serialize like this:

<BinaryString name="Tags"><![CDATA[]]></BinaryString>

but they should serialize like this instead:

<BinaryString name="Tags"></BinaryString>

This is safe because it lines up with how Roblox serializes these values.

Smoke name mapping

Opacity -> opacity_xml
Size -> size_xml
RiseVelocity -> riseVelocity_xml

Make rbx_binary unknown property types fail gracefully

Right now, unimplemented properties waffle between failing the operation in progress to panicking using unimplemented!(). We should migrate all of the cases where rbx-dom libraries panic and turn them into proper errors instead.

SharedString in rbx_xml

Depends on #60.

Generated rbxl files have duplicate services when loaded

This manifests itself mostly in Rojo -- services aren't tagged with the right piece of metadata in the binary format I don't think, which causes this fun:

RbxValueConversion should implement Debug

Got this error while writing a program:

113 |                             match dbg!(default.try_convert_ref(value.get_type())) {
    |                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ `rbx_dom_weak::value::RbxValueConversion` cannot be formatted using `{:?}` because it doesn't implement `std::fmt::Debug`

Expand reflection test suite

There are a few minor heuristics around the reflection database that exist right now. We should expand tests in a couple ways:

• Create a place file with every creatable instance inside of it, deserialize it, and check for properties we don't recognize
• Improve robustness of canonical property heuristics by differentiating between default-canonical and explicitly-canonical

rbx_xml Ray support

The Ray type is only serialized in the RayValue instance, but would still be useful for us to implement.

Add snapshot tests with insta

I think the tests we've got for the serialization libraries are insufficient!

Rojo adopted the 'insta' crate for this and it's worked very well. I think it would help us be more confident with changes like #76.

Better rbx_binary serializer errors

rbx_xml's errors are better, but rbx_binary's are still really bad.

RbxTree API improvements

rbx_dom_weak's RbxTree API and related items aren't very idiomatic.

• Remove Rbx prefix from everything?
  • RbxTree -> Tree, RbxInstanceProperties -> InstanceProperties
• Remove deref for RbxInstance -> RbxInstanceProperties
  • These things should just have getters instead
• Rename RbxInstance methods to remove get_ prefix
  • get_id() -> id()
• Change descendants iterator to return IDs instead of RbxInstance references?
• Change RbxInstanceProperties to be a builder with private fields?

Change WeakDom to use Left-Child Right-Sibling representation

Right now, each instance in WeakDom contains a Vec<Ref> for children. Instead, we can make each node keep the referent of its first child and its next sibling. This will stop us needing to allocate a children table for each node.

This approach is commonplace in DOM implementations, including (I believe) many web browser's HTML DOM implementations.

Give every Roblox type its own struct

This idea formed when we got types like ColorSequence that are useful to reason about on their own.

It'd be nice to have actual types for values like Color3 and Vector2 so that we don't have to continually talk about their representations, [f32; 3] and [f32; 2].

This would break existing consumers of rbx_dom_weak, but would be a pretty nice improvement.

Move away from codegen in favor of serialization for reflection

The fact that rbx_reflection is a generated Rust module is a neat trick, but ultimately problematic for bugs and implementation maintenance. We also have to maintain a parallel generator for rbx_dom_lua, and sometimes feature parity is messed up.

Additionally, it becomes difficult to update the reflection information out-of-band. It'd be nice to distribute new reflection information automatically that applications could update to, but baking it into the executable makes that more difficult than it should be.

I think there are a reasonable set of steps to solve this problem:

• Change generate_reflection to output serialized versions of the reflection database, like MessagePack, JSON, or bincode.
• Split rbx_reflection's bundled reflection database into a separate crate that just exposes an embedded serialized database.
• Add configuration for rbx_xml and rbx_binary to let them accept a reference to a reflection database

There may be a small performance cost to switching from generated code to serialized modules, but if we choose the right format, it should be minimal.

Drive serialization from reflection database

I'm pretty confident that the reflection database generated in rbx_reflection is pretty solid, and in the cases it isn't we can apply temporary fixups.

In order to make rbx-dom more compatible with Roblox, we should be able to refactor both rbx_xml and rbx_binary to use the reflection database instead of blindly deserializing and serializing every property.

I think I'll start with rbx_binary since it needs attention anyways.

rbx_xml name mapping for Fire

For Fire, Size serializes as size_xml, and another property I can't remember has a weird name too. Need to add an XML<->Canonical name mapping for Fire.

Add lifetime to RbxValue and friends

It'd be useful to make RbxValue operate similarly to std::borrow::Cow, where they can contain owned or borrowed data. Adding a single lifetime parameter to RbxValue and seeing where necessary changes fall out should be insightful!

Most stuff can move from what is currently RbxValue to RbxValue<'static>, but some things like Rojo's instance snapshot system could benefit pretty greatly from having borrowed RbxValue objects!

RbxTree::remove_instance panics if ID does not exist in tree

The signature of RbxTree::remove_instance seems like it would just return None if the instance doesn't exist in the tree, but instead it panics because of the way it calls orphan_instance.

Use property read/write errors to change scriptability field

Some properties like MeshPart.MeshId are marked as read-write by our input data, but in practice cannot be written to. We should be able to write back the same value that the property already has in order to detect problems here without needing to manually patch.

Similarly, we currently skip trying to collect default property info for properties that throw errors when accessed. Instead, we can downgrade their scriptability status.