HashSet Deserialization loses IEqualityComparer about ceras

[NarryG]

It wasn't really written as anything more than a fun little feature that barely anyone uses. It's also only used when actually going over the network.
It worked well enough with BinaryFormatter so I don't see why it wouldn't work well enough with Ceras. This isn't a core feature of the program and it was more of a fun experiment than anything. The tagline on it is "Yes it Kinda Works TM".

Taking a peek at the code, it's just letting the garbage collector take care of the images. It's literally just snagging the framebuffer from Bizhawk, shoving it into a command, serializing it, sending it across, the other side deserializes it, sticks it into a PictureBox for 1/whatever of a second, then it replaces it with the next one.

[NarryG]

Looking again it's actually a read only property with a private backing field. Looking at the tutorial, I'm getting the impression read only properties aren't happening?

For now I just derived a type which calls the base constructor with my custom comparator, but this isn't a great solution. Is there some way to do this directly through Ceras that I'm not aware of?

[rikimaru0345]

There are a few things to consider here:

1. It should be pretty easy to populate the object instead. When deserializing, create a new instance of your hashset with the equality comparer. Then use the Deserialize(ref T object, ...) overload. Since HashSet implements ICollection Ceras will be able to re-use the instance and just call Clear() + Add() for the items.

2. There's a small difference between computed properties and get-only-properties (but the lines are pretty blurry anyway, it's all about conventions and assuming what the compiler hopefully does at that point) but deserializing "computed properties" is not a thing, there's nothing to serialize. But get-only-properties are possible, just not reliable to do. I'm looking for a way to do it eventually...

3. What you probably actually need is support for the eventually planned "constructor formatter". Which is basically just a formatter that supports constructing objects using a specific constructor. Some might want to call it a serialization-constructor, but that would imply that it's made specifically for serialization (with an attribute that marks it, like for Json.net I think?), but Ceras will probably support every constructor as long as it can find a matching one (matching member names to parameter names of the ctor). But that is still pretty far off as there's lots to do, but I'm on it :)

So, try Deserialize(ref...) and let me know how that goes. I'm pretty sure that will solve your problem.

[rikimaru0345]

There is actually a 4th approach you can take as well.

There is a factory method, where you are given a type and then you can construct an object and give it back to Ceras. I believe it's called config.ObjectFactory.

Obviously this will only help if your graph is small and you only have one sort of HashSet with a known comparer... otherwise things become more complicated (still possible, but more tedious to do).

But maybe there's an idea in there for the future!
We could make it easier to manage stuff through ObjectFactory by making it so the factory-callback also gives you the member that Ceras wants the object for!
That way you would know what Type Ceras wants, and what member it wants it for, so you can customize it based on everything (field-name, class or struct the object will be put in, ...)

Let me know what you think

[NarryG]

So number 1 isn't really possible due to the way this code is structured as of right now. We have a little networking library that wraps basically any object in a generic command format and can send it between two processes. BinaryFormatter worked perfectly for this because it serializes literally everything, and Ceras has worked almost perfectly for this, but it's a super generic call (we have a class which contains an Object(), we shove whatever we need in there, serialize it, send it across, deserialize it, cast it back to what we know it is). Everything goes through a single deserialize call so I can't use a specific version.

Number 3 sounds exactly like what I need.

I can take a look at number 4 as well. As I said, everything is wrapped in this generic command format. In the case of the Hashset, it's RTC_Command -> Object Command = Dictionary<String,<HashSet<byte[]>>. It's an unknown number of a single type of hashset stored in a dictionary, all made with that same comparer.
In this case, it's the only instance of this specific type being thrown across so it looks like it may get the job done, but I may stick with my workaround because I don't want to limit myself and then have to refactor it back. There's other instances where I could see us using a HashSet<byte[]> so I'll just use this custom class which calls the base constructor with the comparer for now.

[rikimaru0345]

So number 1 isn't really possible due to the way this code is structured as of right now. We have a little networking library that wraps basically any object in a generic command format and can send it between two processes. BinaryFormatter worked perfectly for this because it serializes literally everything, and Ceras has worked almost perfectly for this, but it's a super generic call (we have a class which contains an Object(), we shove whatever we need in there, serialize it, send it across, deserialize it, cast it back to what we know it is). Everything goes through a single deserialize call so I can't use a specific version.

Makes sense.
The idea is cool, I'm doing a very similar thing for another project.
But you can improve things a lot by not having this sort of "container"object and instead doing your serialization and deserialization like this:
Serialize<object>(mySpecificCommandPacketThing);
That saves you one allocation of the wrapper object.

There are more ways to make it much more efficient but I'd take to long to explain all of that. Take a look at the tutorial.cs file there are a lot of ideas in there.

Number 3 sounds exactly like what I need.
I see, I'll see if I can reorder my todo list then.
Might be best to actually talk on discord or somewhere else so I can be sure to get the details right since you have an actual use-case already and I don't have one (yet).
I've got the thing planned through pretty much completely already ,so in total I don't think it should take longer than a few days.

Your workaround sounds fine.

Actually that reminds me: if you're concerned about performance, you can get out even more speed by taking care of "useless" allocations. Like pooling/reusing objects, especially network packets.
There's some pooling example in the tutorial as well I think (called Step3_Recycling() or so).

[rikimaru0345]

Also take a look at KnownTypes that's like the number 1 thing you want to look at whenever you are dealing with anything network related (or anything that creates many small individual packets).

It will massively reduce your packet size because every type (if a type is even needed!) can be written in just one byte (a byte that has to be written anyway, so maybe it would be more accurate to say it even comes for free).

Basically what you do:
While creating the serializer you add all the types you ever want to send to config.KnownTypes and then those types will be never be written out as their full name.
The other receiving side has to have the same setup (well, obviously haha).

[NarryG]

While it is a networking library, it's primarily for local IPC. The trick with it is that the same code is running in two processes which lets us work in either attached mode (it goes through the library but it routes locally and skips the networking step) or we can use it in detached where it routes through the network.
The reason for this is because the program we're running inside of (the Bizhawk emulator) tends to crash a lot (since the use-case for the entire tool is injecting garbage into emulated memory to find fun glitches). The old versions were built purely attached to Bizhawk, but with 3.0 we split it so we can run detached and have a persistent version of the UI driving the emulator process.

We could have used a different kind of IPC, but we used networking as we decided it might be fun to allow remote-control for dumb multiplayer stuff. There's only one situation where it's sending a lot of packets and that's if you're doing the video streaming. I haven't actually tested that with Ceras yet and I won't be surprised if it needs some rejiggering (as it uses System.Image and I don't think that has a default constructor).

I'll take a look at KnownTypes and put all the major stuff in there. In terms of re-use, I could probably make some default stuff for the big types. Ceras already dropped the time taken for the longest operation from 12 seconds to 2 seconds, and very little of that is the serialization/deserialization (it's the generation of 65535 objects). I may optimize it more but for how it's being used, it's efficient enough at the moment.

And yeah Discord may make this a bit easier if you want to hit me up there. I'm Narry#2170

And just for reference, here's the container object
https://github.com/ircluzar/RTC3/blob/master/Real-Time%20Corruptor/BizHawk_RTC/BizHawk.Client.EmuHawk/RTC/Forms/RTC_Multiplayer_Form.cs#L473

And here's where ceras is being used.
https://github.com/ircluzar/RTC3/blob/master/Real-Time%20Corruptor/BizHawk_RTC/BizHawk.Client.EmuHawk/RTC/RTC_NetCore.cs#L183

Once I was sure Ceras would get the job done, I was gonna swap away from the current funky header system I have in place and swap it out for the stuff you mentioned in the tutorial.

[rikimaru0345]

System.Image is pretty interesting.
I have some experience with a similar situation as you have though (sending images over IPC) and in the past I found it to be easier to have a shared memory region and put the image there, and then have a mutex to prevent the other process from reading an image that is half-written (like literally no "vsync" haha).

Continously constructing images should work as well, maybe not as fast (as you naturally copy the binary data around so much...). The biggest issue with System.Image itself is that you have to .Dispose it very quickly.

Maybe I can think of something, but adding support for System.Image would be pretty easy.

I may optimize it more but for how it's being used, it's efficient enough at the moment.

Ok cool, if you ever find that performance is a bottleneck somehow let me know, there's a lot that can be done.