Conflation about engine.io

[EugenDueck]

Somehow I must have forgotten to paste in the footnote that I refer to in the text. So I'll try to reproduce it:

[1] This could be implemented by either one of following or some combination, potentially configurable:

• Make the buffer bounded, and when it gets full, kick client
• Use a ring buffer, where older entries get overwritten
• Call buffer = conflate(buffer) when buffer is full - but this depends on conflate(buffer) returning less elements than it receives, but that could be checked, and a fallback to kick client could be implemented

[EugenDueck]

Quoting my last statement:

... depends on conflate(buffer) returning less elements than it receives

This reminds me that calling this feature conflation would probably be too narrow. If I'll come up with a better name, I'll post it here, but conceptually, this is a general feature that lets application code manipulate (conflate, augment, modify) the buffer, just before being actually sent to a client. Somewhat long for a name...

[rauchg]

I want this

[rauchg]

@EugenDueck I wonder if all that we should do is add a callback to send. In the case of websocket, we fire when the ws transport fires the callback. In the case of polling, when the incoming GET poll request consumes the message.

Then it's up to a higher-level impl to decide on the conflation mechanics.

[EugenDueck]

I wonder if all that we should do is add a callback to send.

That's basically what I'm doing, except that - in my socket.io implementation that I just tested yesterday and it seems fine - I wrap the packets in a new Packet class, which holds and caches

• the data variant of the packet (i.e. the JS object)
• the on-the-wire variant of the packet

It offers methods getAsData and getAsOnTheWireOrSomething (i forgot the exact name I used). Those are the ones that are used by socket.io, and those methods will create the representation if it does not exist and cache it for reuse.

Otherwise there'd be a lot of waste in back- and forth-encoding, especially with multicast (rooms) and broadcast.

Will try to prepare a engine.io variant of it as soon as time permits. Question: Do you agree that the callback (provided by the app that is using socket.io/engine.io) should not see the on-the-wire representation but the JS object?

[rauchg]

Agreed completely. In fact, in engine.io Socket hols an array of packet objects, and only the Transport deals with encoding logic.

[EugenDueck]

Alright. Will hopefully get a pull-request done later today.

Question about the rough timeline. When talking engine.io 1.0, are you talking days, or weeks, or months?

[rauchg]

Days

[EugenDueck]

Gotcha. One more question about the big picture: As I don't see them in engine.io: stores will stay in socket.io?

[rauchg]

Yep. In fact, that's why I want the reliable messaging layer directly in socket.io. It'd be really cool if processes could die, come and go, yet messages stay in let's say redis.

[EugenDueck]

Indeed.

I'm sorry this is getting even more off-topic, but in terms of the big picture, which it is good to always have when making non-trivial changes: Does (the current and/or the future) socket.io's RedisStore allow to non-sticky sessioned load balancing? (Just haven't had the time to dig into it yet)

[rauchg]

Per discussion, we're moving it into a different module.