nickbabcock / boxcars Goto Github PK

Error decoding frame: attribute decoding error encountered: Unrecognized remote id of 11 for actor id / actor object id / attribute id: 21 / 230 / 34. found attribute ProjectX.GRI_X:Reservations (Reservation) on GameInfo_Soccar.GameInfo.GameInfo_Soccar:GameReplicationInfoArchetype in network cache data. searching all attributes with the same stream id, unknown attributes: [Engine.Pawn:bSimulateGravity, TAGame.CameraSettingsActor_TA:ServerSetUsingSecondaryCamera, TAGame.GameEvent_Team_TA:bDisableMutingOtherTeam], all attributes with that stream id: [(Archetypes.Ball.Ball_Default: Engine.Pawn:bSimulateGravity), (Archetypes.Car.Car_Default: Engine.Pawn:bSimulateGravity), (Archetypes.GameEvent.GameEvent_Soccar: TAGame.GameEvent_Team_TA:bDisableMutingOtherTeam), (Engine.Pawn: Engine.Pawn:bSimulateGravity), (Engine.PlayerReplicationInfo: Engine.PlayerReplicationInfo:PlayerName), (GameInfo_Soccar.GameInfo.GameInfo_Soccar:GameReplicationInfoArchetype: ProjectX.GRI_X:Reservations), (ProjectX.GRI_X: ProjectX.GRI_X:Reservations), (ProjectX.PRI_X: Engine.PlayerReplicationInfo:PlayerName), (ProjectX.Pawn_X: Engine.Pawn:bSimulateGravity), (TAGame.Ball_TA: Engine.Pawn:bSimulateGravity), (TAGame.CameraSettingsActor_TA: TAGame.CameraSettingsActor_TA:ServerSetUsingSecondaryCamera), (TAGame.Car_TA: Engine.Pawn:bSimulateGravity), (TAGame.Default__CameraSettingsActor_TA: TAGame.CameraSettingsActor_TA:ServerSetUsingSecondaryCamera), (TAGame.Default__PRI_TA: Engine.PlayerReplicationInfo:PlayerName), (TAGame.GRI_TA: ProjectX.GRI_X:Reservations), (TAGame.GameEvent_Soccar_TA: TAGame.GameEvent_Team_TA:bDisableMutingOtherTeam), (TAGame.GameEvent_Team_TA: TAGame.GameEvent_Team_TA:bDisableMutingOtherTeam), (TAGame.PRI_TA: Engine.PlayerReplicationInfo:PlayerName), (TAGame.RBActor_TA: Engine.Pawn:bSimulateGravity), (TAGame.Vehicle_TA: Engine.Pawn:bSimulateGravity)]. Context: on frame: 0, last updated actor: (actor stream id / object id / name: 21 / 230 / GameInfo_Soccar.GameInfo.GameInfo_Soccar:GameReplicationInfoArchetype, attribute stream id / object id / name: 34 / 224 / ProjectX.GRI_X:Reservations, attribute: Reservation(Reservation { number: 4, unique_id: UniqueId { system_id: 1, remote_id: Steam(76561198408617501), local_id: 0 }, name: Some("Twitch - AvocadoZLive"), unknown1: true, unknown2: true, unknown3: Some(0) }))
6bacc4c0-5c98-4653-be7b-804ed422ce7f.zip

Currently the library only compiles on Rust nightly due to the reliance on nightly features on serde.

DamageState:

• u8 - State of the dropshot tile (0 - undamaged, 1 - damaged, 2 - destroyed)
• bit - False when undamaged, True otherwise
• u32 - Player actor id of the player of inflicted the damage
• Vector3f - position of the ball at the time of the damage
• bit - True for the dropshot tile that was hit by the ball (center tile of the damage area)
• bit - unknown, seems to be always 0

AppliedDamage might be just an older version of DamageState, would need further investigation with older dropshot replays

Loadout:
second to last 32 bits (unknown5 in Rattletrap, here it's discarded) is the product id of the player's avatar border.

OS: Windows 10
Command used: rrrocket.exe -n "C3FF7EE89CA5A5A5F04C443B2A57D0F0.replay"
Command error: An error occurred: Unable to parse replay
Caused by:
0: Error decoding frame: attribute unknown or not implemented: actor id / actor object id / attribute id: 5 / 153 / 31. found attribute ProjectX.GRI_X:MatchGuid (unknown to boxcars) on GameInfo_Basketball.GameInfo.GameInfo_Basketball:GameReplicationInfoArchetype in network cache data. This is likely due to a rocket league update or an atypical replay. File a bug report! Context: on frame: 0, last updated actor: (actor stream id / object id / name: 5 / 153 / GameInfo_Basketball.GameInfo.GameInfo_Basketball:GameReplicationInfoArchetype, attribute stream id / object id / name: 21 / 133 / Engine.GameReplicationInfo:ServerName, attribute: String("USE4919862026241070452")) 1: Error decoding frame: attribute unknown or not implemented: actor id / actor object id / attribute id: 5 / 153 / 31. found attribute ProjectX.GRI_X:MatchGuid (unknown to boxcars) on GameInfo_Basketball.GameInfo.GameInfo_Basketball:GameReplicationInfoArchetype in network cache data. This is likely due to a rocket league update or an atypical replay. File a bug report! Context: on frame: 0, last updated actor: (actor stream id / object id / name: 5 / 153 / GameInfo_Basketball.GameInfo.GameInfo_Basketball:GameReplicationInfoArchetype, attribute stream id / object id / name: 21 / 133 / Engine.GameReplicationInfo:ServerName, attribute: String("USE4919862026241070452")) 2: attribute unknown or not implemented: actor id / actor object id / attribute id: 5 / 153 / 31

If you need more informations just tell me :)
Hope this is usefull

Here you're handling a very abnormal behavior of replay files to delegate the value of a ByteProperty to its key instead. However, I noticed in some replay files (of my own matches, actually) that this issue can also happen with "OnlinePlatform_Epic" and surprisingly "OnlinePlatform_Dingo", and god only knows what "Dingo" platform is :D

Due to the amount of "platform types" that usually get read into the key instead of the value of ByteProperties, I thought to myself why not flip the pattern match, and only read a value if the key was "Platform", but then you're never sure if ByteProperty is only used for this :/

Currently the only logic in place is reading the patch / net version and grabbing 32 or 40 bytes. Rattletrap contains the logic to decode a string. It would be nice to see a unit test or a description of how the name is encoded.

Currently AttributeDecoder::decode_* return an unguarded Attribute. This can cause a panic if not enough data is available. A better solution is to change the return type to Result<Attribute, AttributeError> where AttributeError looks something like:

#[derive(PartialEq, Debug, Clone, Fail)]
pub enum AttributeError {
    #[fail(display = "Not enough data to decode attribute {}", _0)]
    NotEnoughDataFor(&'static str),

    #[fail(display = "Unrecognized remote id of {}", _0)]
    UnrecognizedRemoteId(u8),

    // Etc
}

Benchmarks will need to be ran before and after implementation

This is an ongoing list of desired API changes and discrepancies that have been found

• Attribute::NewColor represent 0xAARRGGBB and an i32 isn't intuitive (amusingly #102 reverted from an u32, and I can't figure out why)
• Split ActiveActor into ObjectTarget (or ObjectId as it is more colloquially known in the code) for the following objects:
  • Engine.GameReplicationInfo:GameClass
  • TAGame.CrowdManager_TA:ReplicatedGlobalOneShotSound
  • TAGame.GameEvent_TA:MatchTypeClass
  • TAGame.GameEvent_Soccar_TA:SubRulesArchetype
• Attribute associated with TAGame.Team_TA:LogoData should not be ActiveActor. Instead, it should be a LogoData struct/class that defines the following for deserialization:

  swap_colors: bool
  logo_id: u32
  

• Instead of splitting actor events into new, updated, and deleted, there should just be one list of actor events as it seems theoretically possible for these to be interwoven and for order to potentially matter. Has been investigated in https://github.com/nickbabcock/boxcars/tree/tape
• Being able to parse into an existing replay to amortize actor allocations (only seems useful for stuff like server backends). Has been investigated in https://github.com/nickbabcock/boxcars/tree/tape

There is no timeline for enacting these changes (or a subset of them), as I'm wary of breaking changes if others have come to rely on it.

While initially promising, failure seems to be too heavy-handed of a requirement to force onto the user. There seems to be some sort of consensus that failure should be used only for applications.

References:

• Do I really need failure/error-chain?
• Current state of error handling in Rust?
• Redefining Failure
• Goodbye failure

Handwriting errors or something like snafu may be the best way forward

Hi, I just tested the dropshot mode and tried to parse it but it gave me this error
Any ideas ?
204C425A44122F8D2EEBF4898DFE5381.zip

Using latest version

Caused by:
0: Error decoding frame: time is out of range: 0.00000000000000000000000000000000028925757. Context: on frame: 1, backtracking to frame 0, last new actor: (id: 6, nameId: 6, objId: 254, objName: TAGame.Default__PRI_Breakout_TA, initial trajectory: Trajectory { location: None, rotation: None })
1: Error decoding frame: time is out of range: 0.00000000000000000000000000000000028925757. Context: on frame: 1, backtracking to frame 0, last new actor: (id: 6, nameId: 6, objId: 254, objName: TAGame.Default__PRI_Breakout_TA, initial trajectory: Trajectory { location: None, rotation: None })
2: time is out of range: 0.00000000000000000000000000000000028925757

7354f090-54ef-4bb3-8b80-10d101913b1d.zip

Error decoding frame: time is out of range: 0.000000000000000000000000000000000000071193495. Context: on frame: 1, backtracking to frame 0, last new actor: (id: 56, nameId: 56, objId: 325, objName: TAGame.Default__MaxTimeWarningData_TA, initial trajectory: Trajectory { location: None, rotation: None })

Right now there are a lot of bits.read_*_unchecked(). These methods can panic if no data is left in the bitstream. There are bits.read_*() which do not panic but come at a slight performance penalty (~10% decrease in throughput (cd bitter && cargo bench for comparison)). I believe the best of both worlds would be to check if there is enough bytes before any unchecked() calls.

For instance, if there are more than 23 bytes left in the network stream, then we can decode a NewActor using all unchecked methods. Else fall back to the checked methods. This may be kinda gross and cause a lot of code duplication (see decode and decode_unchecked for Vector and Rotation), but I don't have a better solution at the time.

RocketLeagueReplayParser contains a reference to decoding UTF-16 strings that don't follow the traditional string decoding. Need to confirm with a test case.

Was reading through the source code and saw that you attempted to implement slice-by-8 but didn't manage to get it to work. I've got it working in my parser if you're still interested in implementing it https://github.com/Bakkes/CPPRP/blob/master/CPPRP/CRC.h#L245
I got about a 4x speedup using it when crcing header + network data.

I've tried to implement it in Rust myself and send a PR, but I've never programmed in it before and didn't manage to get very far. Feel free to copy/convert the code or close this issue if it's not something you're interested in anymore, just thought I'd let you know after reading the comments in crc.rs 😛

When I added rayon support to rrrocket, I changed the functionality so that rrrocket is ran over multiple files so that we can see Rust's parallel iterators at work (and see the huge performance boost). rrrocket currently creates a sibling .json file for each replay, but I miss the days when rrrocket would print the json to stdout so I could pipe with jq. Maybe there should be a cli option for this, and if more than one file is given, error out.

Hi, I am wondering if this parser is working. Also, I cannot figure out how to use it. Any help is appreciated.

Error decoding frame: time is out of range: 0.00000000000000000000000000030380158. Context: on frame: 1, backtracking to frame 0, last new actor: (id: 16, nameId: 16, objId: 304, objName: TAGame.Default__RumblePickups_TA, initial trajectory: Trajectory { location: None, rotation: None })
140A56AC4DEDEF693C3375A77AAD3F1D.zip

Just letting you know that the new RL 1.76 update came with some new classes/properties for the heatseeker mode.

Commit with changes needed for my replay parser: Bakkes/CPPRP@16f248e

Based on summary given by CantFly in the ballchasing discord here: https://discordapp.com/channels/577096078843707392/577564876000460821/700580447750717531

Attached 2 replays from new gamemode heatseekerreplays.zip

Hi @nickbabcock, sorry for filing an issue about this -- I do realize that its not exactly the most appropriate medium in which to have this conversation but I wasn't sure how else to contact you.

As I'm sure you're at least somewhat aware, the representation of game state provided by the rocket league replay format is a bit strange. I suspect that the replay representation corresponds very directly to the way that things are represented in unreal engine in the actual game, but for many use cases, this representation is really cumbersome and difficult to work with.

This is (part of) the reason for the existence of carball (https://github.com/SaltieRL/carball) library, as it basically takes the representation spit out by boxcars and strips away some of the nastier details (especially the way things is represented in different actors). I spent some time resuscitating carball (its in a pretty sorry state and it has some protobuf generation stuff that doesn't play well with modern python tooling), and carving out only the parts that are really necessary to transform the replay into something useful, and i did get it working (you can find that https://github.com/CUBTeamRocket/carball-lite).

However, after noticing that it chokes/loses data in certain cases, I started reading its code to try to fix these issues and realized that it is a needlessly complicated tangled mess that represents things in very inefficient ways internally (so as to make it horribly slow), violates the single responsibility principle flagrantly, and introduces completely unnecessary abstractions.

This compelled me to decide to start from scratch and write something that does what carball does (specifically just the conversion of the replay into a format where its easy to look up e.g. each players state at any given time). I've made quite a bit of progress on this (see colonelpanic8@1fba7a4), but a few things still need work.

tl;dr - I'm wondering if this is the kind of thing that you might want included in boxcars itself, or if I should simply make a separate crate that depends on boxcars for this. I can see arguments for both courses of action, and so I thought I'd ask you to see what you think. I obviously understand if you'd rather not include something like this in boxcars, but I do think that almost anyone dealing with the frames data is going to need to have to do something like this to the date, so I thought it might also be nice to have it built in.

Also -- since my guess is that you probably have as good of an understanding of the replay format as any non-psyonix employee out there, I'd totally appreciate thoughts/input you have on what I'm working on, and in particular any of your understanding of how boost is represented. From what I can work out from how carball handles it, you basically have to build up a derived representation where you estimate the boost total by subtracting from the value found in the replay whenever the boost is considered active. This feels super gross and I'm really hoping that the carball folks were just wrong about this, but it seems like there might simply be no other way.

A boxcars::Replay is currently tied to the byte slice which it's parsed from, which has a nice property: reducing allocations when parsing the header. The downside is that there is inconvenience on the client especially if they want the Replay to be long lived as they have to keep track of the byte slice and the Replay, which often poses a problem in Rust. Yes there are workarounds but boxcars should be flexible for all situations.

Solution 1

Implement ToOwned. This would require a another type, something like ReplayOwned (could there be a better name) which has all the same fields as a Replay but all the data is owned.

Pros:

• Not a breaking change
• Clients can pay for only what they use (eg: if they only parse the header they don't have pay for unnecessary allocations)

Cons:

• May not be obvious to newcomers why there are two types or how to get to the correct type.
• Additional code maintenance

Solution 2

Make Replay own all it's data. The bulk of a replay's size is from the network data and the network data owns all it's data. Thus bending over backwards to save a few allocations in the header seems like it shouldn't be a priority. It's not like parsing the header doesn't incur any allocations (as several Vec are allocated)

Pros:

• One type that is user friendly
• Consistency -- make all data owned data

Cons:

• Breaking change
• There may be performance degradation for those who only want to parse the header.

If the benchmarks don't a major slowdown, my preference would be solution 2.

Download the replay file from here:
https://ballchasing.com/replay/6fa49901-0840-4685-8f14-ed6548971460#overview

Upload to your web parser or parse locally.

The Player "EvillRage" has 2 goals in this game, and shows up in the "Goals" array, but is missing from the Player Stats area. There are a few bots in here, wondering if that's causing problems.

Hello again! I've ran some tests for my replay parser to see if the crc check could be sped up even more using slice by 16/32 and some loop unrolling.

It looks like SB8 is faster than SB16 when CRCing up to 20kb (typical replay header size?). SB16 seems to be faster >20kb and even twice as fast at >500kb so probably worth implementing when CRCing the entire replay.

SB16 -> SB32 didn't seem to make much of a difference with GCC and is sometimes even slower, however with clang it is faster (but still not as fast as GCC's SB16/32). I'm not sure what the result would be in the rust compiler.

Test results: https://gist.github.com/Bakkes/e5ad150aa76d63d50c23f9fd76001edd#file-resulsts-txt-L1

Code: https://gist.github.com/Bakkes/e5ad150aa76d63d50c23f9fd76001edd#file-crctest-cpp-L1 (includes table generation at compile time so code doesn't get as cluttered)

Full tables are dumped at the bottom of that gist if you're interested in trying SB16/32 in rust.

If any of you have ever tried making a custom training pack of decent size(or with progression), you'd notice its missing a TON of features. I think incorporating them into Boxcars would make for a ton of possibilities.

This issue on another repo gave some hints to getting started:
jjbott/RocketLeagueReplayParser#13

Hi!

I am working on this project for replay parsing:
https://github.com/SaltieRL/carball

And we have been using rattletrap but are looking for something a bit faster.
So I was wondering if we could fork this into our org to work on it or if there was a way to reach you if we had questions as we worked on moving it over to our new version.
Also was wondering what the state of this was in terms of completeness in response compared to rattletrap.

I came accross are some unsigned ints with value 4294967295 (2^32 - 1). It would make more sense for them to be signed with value -1. It would also slightly reduce json string size (for rrrocket).

• the u32 in FlaggedInt
• the u32 in StatEvent
• All actor_id references
  • instigator_id in Pickup and PickupNew
  • attacker_actor_id and victim_actor_id in Demolish
  • etc.

I will update this list if I come accross any more while integrating boxcars with carball

• Create subproject called boxcars-web
• Use wasm-bindgen (currently not on crates.io) to create the typescript / wasm file
• Create website that demonstrates parsing a replay in the browser

When decoding a new actor, the extracted name_id is unused. The NewActor struct should take it in as a field.

I tested on your Rocket League Replay Parser test site and a WebAssembly compiled version of boxcars and found that Platform is always returning 0. When parsing the replay myself in Node, I'm getting values of OnlinePlatform_Steam or OnlinePlatform_Epic, but with boxcars I get 0.

My parser:

Boxcars Output:

No need to require usage of serde_json library in the main library code, but until cargo can handle splitting binary and library dependencies the quickest and dirtiest way was to combine them

Hello,

I think the last rocket league update broke something.
Replay parsing stopped working after the last update (2.23)

command: rrrocket.exe -n "59D3E66B4D39CFD6AFE32F82CFD00C45.replay"
os: windows 10

Here's the message

An error occurred: Unable to parse replay ./59D3E66B4D39CFD6AFE32F82CFD00C45.replay Caused by: 0: Error decoding frame: new actor object id out of range: 536870913. Context: on frame: 0, last updated actor: (actor stream id / object id / name: 9 / 254 / GameInfo_Soccar.GameInfo.GameInfo_Soccar:GameReplicationInfoArchetype, attribute stream id / object id / name: 27 / 240 / Engine.GameReplicationInfo:bMatchIsOver, attribute: Boolean(false)) 1: Error decoding frame: new actor object id out of range: 536870913. Context: on frame: 0, last updated actor: (actor stream id / object id / name: 9 / 254 / GameInfo_Soccar.GameInfo.GameInfo_Soccar:GameReplicationInfoArchetype, attribute stream id / object id / name: 27 / 240 / Engine.GameReplicationInfo:bMatchIsOver, attribute: Boolean(false)) 2: new actor object id out of range: 536870913

Replay file :
59D3E66B4D39CFD6AFE32F82CFD00C45.zip

I'm here if you need more informations

cargo +nightly fuzz run no-crc-body

Will crash almost instantly due to the abundance of unguarded, unchecked bitter statements. This means that anyone who sends in a slightly corrupted replay will see boxcars panic. Boxcars should never panic.

Boxcars doesn't panic when we don't have to parse the body:

cargo +nightly fuzz run no-crc-no-body

I'll close this issue once boxcars has been successfully fuzzed on no-crc-body for a week.

Bytes 4-8 make up the first cyclic redundancy check (CRC) for the header. The check ensures that the data has not be tampered with or, more likely, corrupted.

I tried utilizing crc-rs with community-calculated parameters, but didn't get anywhere.

Only the happy path has been focused on, but equally important is the sad path to ensure that error messages are user friendly.

is labelled as stiffness in rattletrap and bakkes

This would be 90% of the file, and it's a shame that my enthusiasm for
implementing this section waned. When the developers of the game say the
section isn't easy to parse, the major rocket league libraries dedicate half of
their code to parsing the section, and the with each patch everything breaks,
it's an incredible feat for anyone to retain enthusiasm. Way to go maintainers!

We already extracted most of the interesting bits like player stats and goals
contained in the header, so it's not a tremendous loss if we can't parse the
network data. If we were able to parse the network data, it would allow us to
run benchmark against other implementations.

I don't think this is an issue with carball's invocation. I'm seeing this error on recent replay files:
Exception: Error decoding frame: attribute unknown or not implemented: actor id / actor object id / attribute id: 13 / 314 / 31. found attribute ProjectX.GRI_X:MatchGuid (unknown to boxcars) on GameInfo_Soccar.GameInfo.GameInfo_Soccar:GameReplicationInfoArchetype in network cache data. This is likely due to a rocket league update or an atypical replay. File a bug report! Context: on frame: 0, last updated actor: (actor stream id / object id / name: 13 / 314 / GameInfo_Soccar.GameInfo.GameInfo_Soccar:GameReplicationInfoArchetype, attribute stream id / object id / name: 21 / 294 / Engine.GameReplicationInfo:ServerName, attribute: String("TNA114-Voxel5"))
example.zip

Rocket League seemed to have updated something relating to demos and they no longer appear in the JSON output after parsing a replay. Demos (pre-season 4) were under the actor "TAGame.Car_TA:ReplicatedDemolish", but since season 4, they no longer appear, even if there are demos in the game. Ballchasing had an issue with demos always being 0 and fixed the problem, but no one seems to be aware of what the problem was.

Hello, could you add an argument that saves the content as a file?

Current nom API makes streaming data non-intuitive 😕

Looks like a new API is on the horizon, so it could prove prudent to wait for that.

Currently having boxcars, rrrocket, and boxcapy share the same repo is a bit cumbersome. Ideally rrrocket should have its own repo because:

• Boxcars and rrrocket have different releases. Boxcars, as the main library, should have its own release. rrrocket should have its own release to provide static executable. Since both need their own release (and their own changelog) and intertwining releases in a single repo may be confusing so breaking them out into separate repos will be easier to conceptualize changes + releases
• For a fully reproducible build, rrrocket should have a lockfile, but is unable to when it's a submodule of boxcars
• Boxcars repo should focus on API documentation (and look like a crate) whereas rrrocket should focus on user usage

Since boxcapy interprets the JSON from rrrocket -- they both can be included in the same repo.

The downside is that the replays will need to copied to both repos, and this repo will need to redirect users who want to download rrrocket. But this seems minor in the long run.

Snippets like these:

bail!("Time too small");
bail!("Delta too small");
format_err!("Actor id: {} was not found", actor_id);

Should be replaced with a formal error type. Something along the lines of:

#[derive(PartialEq, Debug, Clone, Fail)]
pub enum NetworkParsingError {
    #[fail(display = "Time is out of range: {}", _0)]
    TimeOutOfRange(f32),

    #[fail(display = "Delta is out of range: {}", _0)]
    DeltaOutOfRange(f32),

    // Etc
}

Which will ease unit tests that stress these (better to test with a compiled type than a string message), but also allow future library users to tweak their behavior based on the errors.

I'm trying to simply list which car the players loaded in with. I know I need to do something with TAGame.PRI_TA:ClientLoadoutOnline but I'm fairly new to Rust and am having a lot of difficulties understanding how to interact with this crate.

I tried applying jjbot's fix to his parser(jjbott/RocketLeagueReplayParser@81c09f3#diff-0c02ca5dbc85c6f2a5d0987bcd952c04) by adding an entry to OBJECT_CLASSES similar to how he did. But this still fails to parse after adding it.

 .entry("ProjectX.Default__NetModeReplicator_X", "\"ProjectX.NetModeReplicator_X\"")

Here is a replay that breaks master.

21A81.zip

let y: Option<i32> = None
let x = y.unwrap()

Is a panic in disguise if y is None. There are several instances in the code that call unwrap. These should be converted into an appropriate error (something along the lines of "channel bits exceeded maximum" or "not enough data")

This issue will be closed when all non-test unwraps are removed, but this can be tackled piecemeal.