[VAX-1006] Possible race condition in `process.subscribe()` about electric

[balegas]

We're working on some improvements to the protocol and will look into this after.

#423

[alco]

I'm currently investigating a very similarly looking problem, though I traced it down to different locations in the client code from the original report here. Instead of an undefined exception, I'm getting this:

encountered a subscription error: Received subscribe response for unknown subscription bfca735d-5037-4685-b547-74a4946118f1
connectivity state changed disconnected
client disconnected from server
Uncaught:
i [Error]: Received subscribe response for unknown subscription bfca735d-5037-4685-b547-74a4946118f1
    at q.internalError (file:///app/node_modules/.pnpm/[email protected][email protected]/node_modules/electric-sql/dist/satellite/shapes/cache.js:1:3515)
    at q.subscriptionResponse (file:///app/node_modules/.pnpm/[email protected][email protected]/node_modules/electric-sql/dist/satellite/shapes/cache.js:1:682)
    at Re.handleSubscription (file:///app/node_modules/.pnpm/[email protected][email protected]/node_modules/electric-sql/dist/satellite/client.js:1:11025)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async oe.subscribe (file:///app/node_modules/.pnpm/[email protected][email protected]/node_modules/electric-sql/dist/satellite/process.js:3:2997)
    at async Module.$ (file:///app/dist/client.js:1:1125)
    at async REPL7:1:33 {
  code: 26
}

I can reliably reproduce this when running the E2E test suite in #500.

Here's the sequences of protocol messages that leads to the exception:

[rpc] send: #SatSubsReq{id: bfca735d-5037-4685-b547-74a4946118f1, shapes: [{"$type":"Electric.Satellite.SatShapeReq","requestId":"fe977dcd-c79c-431a-b9da-2aa923c6dfce","shapeDefinition":{"$type":"Electric.Satellite.SatShapeDef","selects":[{"$type":"Electric.Satellite.SatShapeDef.Select","tablename":"items"}]}}]}
[proto] send: #SatRpcRequest{method: subscribe, requestId: 3}
[proto] recv: #SatRpcResponse{method: subscribe, requestId: 3}
[proto] recv: #SatSubsDataBegin{id: bfca735d-5037-4685-b547-74a4946118f1, lsn: MjU1MTEyODA=}
[proto] recv: #SatShapeDataBegin{id: fe977dcd-c79c-431a-b9da-2aa923c6dfce}
[proto] recv: #SatOpLog{ops: []}
[proto] recv: #SatShapeDataEnd{}
[proto] recv: #SatSubsDataEnd{}
actually changed notifier
[rpc] recv: #SatSubsResp{id: bfca735d-5037-4685-b547-74a4946118f1}

For comparison, here's what the working sequence looks like:

[rpc] send: #SatSubsReq{id: b1f3087c-6eac-43c9-92b7-83d622abf678, shapes: [{"$type":"Electric.Satellite.SatShapeReq","requestId":"737a6fa4-1b48-4c01-855b-a4d4c187da3f","shapeDefinition":{"$type":"Electric.Satellite.SatShapeDef","selects":[{"$type":"Electric.Satellite.SatShapeDef.Select","tablename":"items"}]}}]}
[proto] send: #SatRpcRequest{method: subscribe, requestId: 3}
[proto] recv: #SatRpcResponse{method: subscribe, requestId: 3}
[rpc] recv: #SatSubsResp{id: b1f3087c-6eac-43c9-92b7-83d622abf678}
[proto] recv: #SatSubsDataBegin{id: b1f3087c-6eac-43c9-92b7-83d622abf678, lsn: MjU1MTEwNjQ=}
[proto] recv: #SatShapeDataBegin{id: 737a6fa4-1b48-4c01-855b-a4d4c187da3f}
[proto] recv: #SatOpLog{ops: []}
[proto] recv: #SatShapeDataEnd{}
[proto] recv: #SatSubsDataEnd{}
actually changed notifier

The difference between the two being in the ordering of the SatSubsResp message, everything else remains the same.

I was looking into how my server-side changes could cause this when I remembered this issue that had already been reported by @js2702. So it looks like my changes aren't the root cause but have somehow made it more likely for the race to occur.

[alco]

A bit more info on the error I'm seeing: the exception is raised in SubscriptionDataCache.subscriptionResponse(), and the subscription itself gets deleted in SubscriptionDataCache.reset() when the SatSubsDataEnd message is processed, so that there's no cache entry for it by the time SatSubsResp arrives.

[alco]

After studying the server log in the failing E2E tests in #500, I'm inclined to conclude that it's a client-side error. I couldn't see any difference in processing the SatSubsReq between two clients on the server, yet each client logs that fact that it received the response at different points.

cc @balegas @kevin-dp

[icehaunter]

@alco that's definitely a client-side issue. The issue there is the difference where the event loop gets resolved. RPC message decode returns a promise resolution, but that releases the event loop, and sometimes, if the message from the server arrives in the subsequent frames, the event loop enters in message processing instead of in resolving await call - and the message gets processed ahead of handling of RPC response

[alco]

@icehaunter I'm at a loss as to why my changes to how PG connections are pooled on the server could affect this client-side issue such that I see 1 to 3 E2E tests fail with this error every time I run the E2E workflow from #500 on CI. It hasn't succeeded once here on GH, I've only been able to see the full E2E test suite pass on my machine.

[davidmartos96]

@alco When @js2702 and I reported this issue it was only happening on a Linux machine, on macOS it was working correctly.
We stopped seeing this issue when we implemented the new overhauled RPC communication on the Dart client, though. We also managed to avoid the issue temporarily by adding the equivalent of await asyncSleep(0) for Dart, when receiving websocket messages on the client, although that's more a hack than a proper fix, hence why we opened the issue.