Pong example not properly working about nettyrpc

[beef331]

Ah investigating the issue is everything is traced back to rpcTick, first you read the managed ID, which moves the stream position, then you read the relayed id, then breakafter relaying, so at most 1 procedure can run. If the managed ID isnt there you need to move the pos of the stream back int bytes so that it can then read the appropriate data.

[beef331]

So the list of things that need to be fixed are server needs to relay to all clients, need to stop breaking after managed/relayed events. Looking at it now, we really should put a messageType at the start of the packets so we dont need to have this weird while loop, so then we add only messages that corespond to the buffers to the buffers.

[RattleyCooper]

I will investigate after work. I bet it has something to do with rpcTick. I can't test at work but I have a couple ideas as to what may be causing it. The relay procedure should only send RPCs to Connections that are not the caller.

I did notice this in your pong example:

proc update(dt: float32) =
  client.rpcTick()
  client.tick()

Since rpcTick is making netty tick the additional call to tick might be clearing out messages that should be processed(netty sets the messages queue length to 0 when it ticks)? Not 100% sure, but I'll dig into this more bc it seems like a good place to start.

I'm sorry btw, I should have done better at testing on my end :/ I'll update once I have more info.

[RattleyCooper]

When I add client.tick() after the call to client.rpcTick() in the relayed chat client example I get the same behavior you described. Messages hit the server but the messages queue gets cleared out depending on when the client receives the packets. It appears random when it fails, but it's all based on if the packets come in after client.rpcTick or if they come in after client.tick. The extra call to client.tick shouldn't be there either way so I'm removing it for now and will try to confirm whether or not that is what was causing the issue.

Edit: Yeah, it's more complicated than just that. I got SDL2 and understand the problem better now. I think it's probably related to rpcTick and logic that reads the stream data. There are likely multiple messages that need to be processed but rpcTick is breaking the while loop that reads the stream data and calls procs.

Edit2: Moving while loop into for msg in reactor.messages loop and processing each message's stream separately helps fix some sync issues but there is still some sync issues that need to be worked out. Likely related to the relay proc not updating all the appropriate clients. The paddle on one of the clients stops syncing

[RattleyCooper]

Ah investigating the issue is everything is traced back to rpcTick, first you read the managed ID, which moves the stream position, then you read the relayed id, then breakafter relaying, so at most 1 procedure can run.

Yep, that's what I'm finding out as well.

Moving while loop into for msg in reactor.messages loop and processing each message's stream separately helps fix some sync issues but there is still some sync issues that need to be worked out. Likely related to the relay proc not updating all the appropriate clients.

[beef331]

Ah actually they use two different buffers, so I'm wrong. Does the relayed even need the conection parameter, since it should just send it to all users other than that it recieved it from(if it's the server), otherwise it doesnt care the connection when running the procedure?

[RattleyCooper]

If the managed ID isnt there you need to move the pos of the stream back int bytes so that it can then read the appropriate data.

Gotcha

Ah actually they use two different buffers, so I'm wrong.

No, you're still correct because it still breaks regardless.

[RattleyCooper]

Does the relayed even need the conection parameter, since it should just send it to all users other than that it recieved it from(if it's the server), otherwise it doesnt care the connection when running the procedure?

I pass the connection in so it knows which connection to ignore when sending the relay

[beef331]

I am talking about relayedEvents*: array[uint16, proc(data: var NettyStream, conn: Connection)], this is used just to invoke the internal procedure, why does it even need connection?

[RattleyCooper]

I am talking about relayedEvents*: array[uint16, proc(data: var NettyStream, conn: Connection)], this is used just to invoke the internal procedure, why does it even need connection?

You're totally correct, it does not technically need the connection, especially since the Connection will always refer to the server on the client's end and is useless for the server to have.

[RattleyCooper]

So the list of things that need to be fixed are server needs to relay to all clients, need to stop breaking after managed/relayed events. Looking at it now, we really should put a messageType at the start of the packets so we dont need to have this weird while loop, so then we add only messages that corespond to the buffers to the buffers.

So, even with a messageType being added to the beginning of the message we still have to use the while loop on the buffer, and we have to process the entire buffer. This is because the local procedures write to the sendBuffer and the sendBuffer is sent every rpcTick. The server relays the sendBuffer that contains multiple messages but the clients are not processing the entire data stream.

relayed macro rewrites the procs like so:

proc updatePaddle(p: Paddle; isLocal: bool = true) =
  if (not isLocal):
    for x in paddles.mitems:
      if (not x.local):
        x.y = p.y
  if isLocal:
    write(sendBuffer, MessageType.Relayed)
    write(sendBuffer, 2'u16)
    write(sendBuffer, p)

proc updateBall(inBall: Ball; isLocal: bool = true) =
  ball = inBall
  if isLocal:
    write(sendBuffer, MessageType.Relayed)
    write(sendBuffer, 3'u16)
    write(sendBuffer, inBall)

You can see how the messages add up in the buffer, but rpcTick doesn't process the entire buffer

[beef331]

Well of course the while loop is used, you can have more than a single proc call in a message, by weird while loop I meant one with indecisiveness. This removes this by making it so you know which type of proc you got, so you can just handle it accordingly.

[RattleyCooper]

Well of course the while loop is used, you can have more than a single proc call in a message, by weird while loop I meant one with indecisiveness. This removes this by making it so you know which type of proc you got, so you can just handle it accordingly.

Ok gotcha. I'm making good progress. Have the pong example working with messages being dispatched based on a MessageType. I'm going to work up an authoritative server example based on the pong example and make sure I didn't break anything else.

One thought though. The rpc procedures send the sendBuffer when they're called instead of writing to the sendBuffer and sending the entire buffer in the rpcTick procedure. This was originally done to make it easy for the server to send an RPC to all connected clients, or a single client. I'm not sure if sending when the rpc procedures are called is better or worse than writing to the buffer and sending the entire buffer. Thoughts? I feel like keeping things consistent will make things easier to maintain, but I also can't think of a way to do it without using a sendAllBuffer or something like that.

[beef331]

It's best to send a single large message than a bunch of smaller messages, especially since every frame(atleast in the pong example) there will be a tick. This way we can reduce the network traffic over time and minimize network usage(small packets also contain the netty reliable UDP logic).

[RattleyCooper]

Ok, I will incorporate a separate buffer for sending to all clients and move all the sending logic from the rpc procedures over to rpcTick.

[RattleyCooper]

I may end up just doing separate tick procedures. 1 for relaying and 1 for authoritative servers. Sure, people won't be able to mix the 2 together but tbh that may be better anyway. It's just getting to be a huge pain making them both work with the same tick procedure.

[beef331]

I dont quite follow why it's an issue, could you push the code somewhere so I could take a look? Assuming it's there is no way to figure relay length, what could be done is the rpc adds to a buffer which then on rpc tick is appended to the current buffer so it turns into MessageKind, MessageLen, messages that way relaying can be done by the server since it knows when it hits a MessageKind.Relayed the next int is message length, it can then go back 5 ints and send that to all clients.

As an aside, dont feel pressured to continue this work, if you dont want to i can always take over 😄

[RattleyCooper]

Well I really wanted to support direct sending to a specific client so that an RPC doesn't necessarily have to go out to every single client if it isn't needed. We need access to the Connection that sent the RPC, but with the way things are set up it's a bit tricky. In rpcTick the data stream is written to the buffer and then the while loop reads the entirety of the buffer to dispatch RPCs:

for msg in reactor.messages:
  theBuffer.addToBuffer(msg.data)

while (not theBuffer.atEnd):
  # Dispatch local RPC but networked RPC requires access to the msg.conn Connection object for direct sending.

And there is no way to track the connection that sent the message unless we process the stream as it's being read, or unless we include some kind of connection id. netty doesn't sync connection id's or anything like that so I am having a lot of trouble figuring out how the server is going to handle sending RPCs to specific clients instead of all clients. Including the Connection id in the messages from the client doesn't work because the server-side connection id is unique and the client-side id can't be used to match up the server-side id.

Here is my current rpcTick:

proc relayServerTick() =
  for msg in reactor.messages:
    for connec in reactor.connections:
      if connec != msg.conn:
        reactor.send(connec, msg.data)

proc rpcTick*(sock: Reactor, server: bool = false) =
  ## Parses all packets recieved since last tick.
  ## Invokes procedures internally.  If no procedure is
  ## found it will relay the command.
  sock.tick()
  if not server:
    client.send(sendBuffer)

  var theBuffer = NettyStream()
  var conn: Connection # CLIENT-SIDE IS ALWAYS RECEIVING FROM THE SERVER!
  for msg in reactor.messages:
    theBuffer.addToBuffer(msg.data)
    conn = msg.conn  # Since the connection is always the server this is safe for client.  Used for relay

  while(not theBuffer.atEnd):
    let messageType = block:  # Read the message type
      var res: MessageType
      theBuffer.read res
      res

    case messageType:
    of MessageType.Relayed:
      if server:
        relayServerTick()
        return
      let relayedId = block:
        var res: uint16
        theBuffer.read res
        res
      if(relayedEvents[relayedId] != nil):
        relayedEvents[relayedId](theBuffer)
    of MessageType.Networked:
      # Logic for networked RPC if we have access to Connection
      raise newException(NettyRpcException, "Networked messages mixed with relayed messages.  Don't mix relayed and networked pragmas.")

It works for relays. Haven't figured out how to get the authoritative stuff working since I can't process the buffer in the for msg in reactor.messages: loop. I don't think there is any guarantee all the RPCs contained in a message are from the same client. The conn = msg.conn works for relays because the client will only ever be communicating with the server so it's safe.

I am trying a separate tick procedure that tries to process the stream as it's read in but that isn't really working out so far:

  for msg in reactor.messages:
    echo "got message"
    theBuffer.addToBuffer(msg.data)

    while(not theBuffer.atEnd):
      # theBufferPos = theBuffer.pos
      let messageType = block:  # Read the message type
        var res: MessageType
        try:
          theBuffer.read res  # If we can't read the MessageType maybe we are missing part of the buffer?
        except RangeDefect:
          echo "Range defect"
          # theBuffer.pos = theBufferPos
          break  # So break out of the while loop, add to buffer and try again.
        res

As an aside, dont feel pressured to continue this work, if you dont want to i can always take over 😄

I mean, I wouldn't be against some help. I just can't think of a reliable way to do this the way I want. I feel like I'm spinning my wheels. I'm not sure how to incorporate the direct sending without being able to read the buffer on the fly.

Here is the code I'm testing with just to try and get the authoritative chat client working. I just copied the nettyrpc.nim and the nettystream.nim to the directory.

https://github.com/RattleyCooper/nettyrpc-1/tree/master/example/authoritativeChatClient

[RattleyCooper]

I think I have a working solution for reading a partial data stream for networked procs. It seems kind of hacky but it does work. It tries to read the data stream and if it can't then it breaks out of the while loop and adds to the buffer and tries again.

proc networkTick*(sock: Reactor, server: bool = false) =
  sock.tick()
  if server:
    for k, s in directSends.pairs:
      var directSendStream = NettyStream()
      for stream in s:
        directSendStream.addToBuffer(stream.getBuffer)
      reactor.send(k, directSendStream.getBuffer)
    directSends.clear()
  sendall(sendAllBuffer)

  var theBuffer = NettyStream()
  var theBufferPos = theBuffer.pos
  for msg in reactor.messages:
    # echo "got message"
    theBuffer.addToBuffer(msg.data)
    block blockBuster:
      while(not theBuffer.atEnd):
        let messageType = block:  # Read the message type
          var res: MessageType
          try:
            theBuffer.read res
          except RangeDefect:
            break blockBuster  # break out of while loop.
          res

        case messageType:
        of MessageType.Networked:
          let managedId = block: 
            var res: int
            theBuffer.read res
            res
          if managedEvents.hasKey(managedId):
            managedEvents[managedId](theBuffer, msg.conn)
        of MessageType.Relayed:
          raise newException(NettyRpcException, "Relayed messages mixed with networked messages.  Don't mix relayed and networked pragmas.")

I think I can merge this with the normal rpcTick procedure without much trouble but I'm not sure this is the best approach.

[beef331]

except RangeDefect: This is bad form, defects are not to be caught they're to be avoided. What's the issue with putting RPC inline with relayed? Doesnt the RPC only call on the server then the server reasons to who to send any resultant action?

[RattleyCooper]

What's the issue with putting RPC inline with relayed? Doesnt the RPC only call on the server then the server reasons to who to send any resultant action?

I'm not sure I follow what you mean about inlining with relayed. With the authoritative approach the client can send an RPC to the server which runs a local procedure and the server can also send an RPC directly to a client or to all clients, which run their own procedures.

except RangeDefect: This is bad form, defects are not to be caught they're to be avoided.

I just can't think of any other way to do it. If we know there is a potential for incomplete data why is it bad form to handle that scenario?

[beef331]

The rpc message is like MessageKind.Networked, networkedID, params it doesnt have a target built into the message, if you want to send it to a specific client it's up to the server to reason that, as such it can be stored with the relayed messages assuming your relayed messages are written as such MessageKind.Relayed, lengthOfRelayedMessageinBytes,relayedMessages(all of them together) Defects are not meant to be caught and in some Nim configs cannot be caught, as such you should avoid them instead of catching them.

[beef331]

The following is what i'm suggesting, might make it more clear

proc networkTick*(sock: Reactor, server: bool = false) =
  sock.tick()
  if server:
    for k, s in directSends.pairs:
      var directSendStream = NettyStream()
      for stream in s:
        directSendStream.addToBuffer(stream.getBuffer)
      reactor.send(k, directSendStream.getBuffer)
    directSends.clear()
  sendall(sendAllBuffer)

  var theBuffer = NettyStream()
  var theBufferPos = theBuffer.pos
  for msg in reactor.messages:
    # echo "got message"
    theBuffer.addToBuffer(msg.data)
      while(not theBuffer.atEnd):
        let messageType = block:  # Read the message type
          var iVal: uint8
           theBuffer.read iVal
           if iVal in MessageType.low.ord .. MessageType.high.ord:
             MessageType(iVal)
           else:
              break # We've got a wrong message

        case messageType:
        of MessageType.Networked:
          let managedId = block: 
            var res: int
            theBuffer.read res
            res
          if managedEvents.hasKey(managedId):
            managedEvents[managedId](theBuffer, msg.conn)
        of MessageType.Relayed:
          let 
            messageLength = block:
              var res: int
              theBuffer.read res
             theEnd = theBuffer.getPosition + messageLength
          while theBuffer.getPosition < theEnd:
             # Treat like a relayed message

[RattleyCooper]

How would you getting the message size in bytes? I'm doing it in the macro but not sure if this is a good way to go about it. Ends up with a proc that creates a temporary netty stream every time a relayed proc is called so I can calculate the size of the message:

proc ballScored(leftScr, rightScr: int; isLocal: bool = true) =
  leftScore = leftScr
  rightScore = rightScr
  if isLocal:
    var tempStream = NettyStream()
    write(tempStream, MessageType.Relayed)
    write(tempStream, 3'u16)
    write(tempStream, leftScr)
    write(tempStream, rightScr)
    var msgLen = sizeof tempStream.getBuffer
    msgLen += sizeof msgLen
    write(sendBuffer, MessageType.Relayed)
    write(sendBuffer, msgLen)
    write(sendBuffer, 3'u16)
    write(sendBuffer, leftScr)
    write(sendBuffer, rightScr)

[beef331]

This goes to my suggestion of having a relayed buffer, you write all the messages to that then when you go to send you appened the relay buffer to the message contigiously.

[RattleyCooper]

I think I might be confused or just have information overload or something.

I was thinking about this and it seems pretty trivial to create a relay using the networked pragma:

# Server-side authoritative relay
proc doRelay(procName: string, theArgs: tuple) {.networked.} =
  rpc(procName, theArgs)

and then convert the relayed pragma to simply call the rpc proc that would call the doRelay with the info needed:

proc ballScored(leftScr, rightScr: int; isLocal: bool = true) =
  leftScore = leftScr
  rightScore = rightScr
  if isLocal:
    rpc("doRelay", (procName: "ballScored", theArgs: (leftScr: leftScr, rightScr: rightScr, isLocal: isLocal)))

This would get rid of the requirement for MessageType and the MessageLenth since we are only dealing with 1 type of message, and makes the entire project easier to maintain because we're not having to decipher all this extra stuff related to messages.

Thoughts? lmk if I'm overlooking something. My brain feels like a potato today, but I feel like this might be the route to go to simplify things.

Edit: Ah, nvm the NettyStream doesn't seem to want to read tuples 😡 :

  test "tuple":
    let a = (1, 1.0, -1)
    var ns = NettyStream()
    ns.write(a)
    ns.pos = 0
    var b: tuple   # invalid type: 'tuple' for var
    ns.read(b)
    assert a.b == 1.0

[beef331]

We can go with this assuming you automate the sending of that tuple/message(and also use a static ID), I was planning on swapping to disruptek's frosty anywho(though with an enum bug the nico test fails). That's not an issue with netty stream, that just invalid code, b: tuple is a generic, not a concrete type. The way to do that properly is:

proc thing(t: tuple) =
  echo t
  var a: t.type
  echo a
  
thing((10, 20, 30))
thing((a: 30, b: 20, c: 50))

[RattleyCooper]

We can go with this assuming you automate the sending of that tuple/message(and also use a static ID), I was planning on swapping to disruptek's frosty anywho(though with an enum bug the nico test fails).

Sick, I'll proceed with this route.

That's not an issue with netty stream, that just invalid code, b: tuple is a generic, not a concrete type.

That's really good to hear because I was super stoked to figure out a potential way to simplify this. So I'll need to use the thing.type in the recBody of the macro to set up receiving properly.

[RattleyCooper]

@beef331

Am I missing something else when trying to read tuples?

  test "tuple":
    let a = (a: 1, b: 1.0, c: -1)
    var ns = NettyStream()
    ns.write(a)
    ns.pos = 0
    var b: a.type  # Type mismatch error
    ns.read(b)
    assert a.b == b.b

[beef331]

Ah actually nettystream doesnt support tuples, even more reason for that move to frosty. I can intergrate frosty ontop of your changes if you just get them done and don't worry about the compiler errors.

[RattleyCooper]

I mean, could I dynamically generate read procs that accept a concrete tuple type in the meantime? Seems doable in the macro, even if it's not the best long-term solution.

[beef331]

Eh just use the updated code for nettystreams it was a simple add to generic, cause i guess we dont want to break this for stable Nim presently, since frosty requires devel.

[RattleyCooper]

Oh sweet, that is way easier 🎉 I thought there was some kind of limitation. I was looking at the write proc you had that handled tuples and totally forgot to look for the read proc.

[RattleyCooper]

@beef331 Do you know how to get the concrete tuple type in a macro? I need to evaluate the theArgsType NimNode(a nnkTupleClassTy) so I can use the concrete type in the recBody.

proc relay(procName: string; theArgs: tuple; conn: Connection = Connection()) =
  var theArgsType = type(theArgs)           # Need the value from this in the anonymous proc.
  rpc(procName, theArgs)

managedEvents[4038518758] = proc (data: var NettyStream; conn: Connection) =
  let procName = block:
    var temp`gensym71: string
    data.read(temp`gensym71)
    temp`gensym71
  let theArgs = block:
    var temp`gensym72: tuple         # invalid type `tuple` for var / Need to fill in with type from `theArgsType`, or something similar
    data.read(temp`gensym72)
    temp`gensym72
  relay(procName, theArgs, conn)

Do you know of any way to get the concrete tuple type into the anonymous proc being used so I can read data into that temp var?

[beef331]

You'll need to emit the tuple type constructor ie (bool, string, float,...).

[RattleyCooper]

You'll need to emit the tuple type constructor ie (bool, string, float,...).

I'm not sure how to emit it though. It doesn't look like I have access to that information in the AST. Is it possible to use an ident to get that information from the theArgs tuple?

[beef331]

Isnt it just the ident defs stored in the proc parameter list? So procDef[3][1..^1] would be the types to go inside a tupleTy.

[RattleyCooper]

I'm not using a concrete tuple type in the proc def for the relay

proc relay(procName: string; theArgs: tuple; conn: Connection = Connection()) =
  var theArgsType = type(theArgs)
  rpc(procName, theArgs)

[beef331]

Correct but you are concrete types for the proc you have {.relayed.} on.

[RattleyCooper]

relay is the {.networked.} proc I want to use to support the {.relayed.} proc

[beef331]

Here is an example of what I mean:

import std/macros

macro relayed(a: untyped) =
  result = newStmtList(a)
  result.add newCall(ident"echo", nnkTupleTy.newTree(a.params[1..^1]))
  
proc doStuff(a, b, c: int, d: float) {.relayed.} = discard

You of course will want to set the ^1 node to newEmptyNode() since tuples do not allow (a: int = 30)

[RattleyCooper]

Well the relay proc still needs to read data sent from the client, since it's a server-side RPC. The client needs to send the RPC they want relayed along with a tuple of args that should go with it. relay needs to handle any arbitrary tuple so it can handle multiple scenarios.

Right now the nettyrpc.nim script won't compile because I don't have any concrete tuple types to read data in with. The relayed RPCs don't exist server-side so I don't have any of the proc defs to work with.

[beef331]

This is where i think you need an ID/packet size cause when you send a message to relay the server shouldnt need to do anything but take the message and pass it a long. Which is what I thought that relay proc was for, it'd write the tuple to a buffer, then send the ID + length before.

[RattleyCooper]

Well that is what the relay proc is for. relay is a server-side RPC that would do nothing more than forward the message to all the other clients. It does a wonderful job of writing a tuple and sending it, but there is no way to read an arbitrary tuple into the stream because I can't do var temp: tuple; ns.read temp

[beef331]

Ok push the code and let me take a look at it, so I can actually formulate an idea/plan.

[RattleyCooper]

Ok, headed home and I'll get it pushed here in a few minutes.

[RattleyCooper]

https://github.com/RattleyCooper/nettyrpc-1/blob/master/src/nettyrpc.nim

[beef331]

I've got most of the logic implemented on https://github.com/beef331/nettyrpc/tree/fixings though presently there is an issue with the data, it causes multiple out of range errors. It was not viable to be able to encode the information for a relay any other way in my view, so we have a second buffer for relayed events.

[RattleyCooper]

Does this actually write the length of the buffer?

sendBuffer.write(relayBuffer.getBuffer) # Writes len, message

I updated to:

proc rpcTick*(sock: Reactor, server: bool = false) =
  sock.tick()
  if server:
    for k, s in directSends.pairs:  # Direct sends from the server.
      var directSendStream = NettyStream()
      for stream in s:
        directSendStream.addToBuffer(stream.getBuffer)
      reactor.send(k, directSendStream.getBuffer)
    directSends.clear()
  else:
    if relayBuffer.size > 0:  # This was also relayBuffer.pos, not sure if it should be .size
      let sizeOfBuffer: int64 = sizeof(relayBuffer) + sizeof(int64)
      sendBuffer.write(MessageType.Relayed) # Id
      sendBuffer.write(sizeOfBuffer)
      sendBuffer.write(relayBuffer.getBuffer) # Writes len, message
      relayBuffer.clear()

and get an out of memory error when running pong example...

Edit:

Does the lent string in proc getBuffer*(ns: NettyStream): lent string = ns.buffer prepend the length of the string in bytes? Documentation on lent seems to be missing

[beef331]

Yea the length is written, we dont worry about the size of the header cause on clients we read id, length then the buffer. so the length doesnt count the id/length. lent is just "borrow the memory if possible", it's the read only variant of var T.

[RattleyCooper]

How is the message length written though? Doesn't theBuffer.getBuffer only write a string?

[beef331]

Look at the nettystream write string implementation, all will be revealed. It writes len followed by the string chars.