I do not really see the point in the mentioned multi-process approach. How would this ensure a websocket is bound to the particular TCP handling task's thread. Or have you just meant it to be an add-on comment on the port-per-thread solution? I need to ask as wll, why do you favor multi-process over multi-threaded? Having multiple processes sharing the same port seems to me equivalent to having multiple threads within a single process listening on the same port.

Yeah, this was more of a general remark, because it usually makes sense to design with that setup in mind. Some advantages:

  • No global GC lock that can otherwise easily destroy the multi-threading advantage
  • Easy to scale across multiple machines
  • More robust against fatal application errors (n-1 out of n processes will continue to run)
  • Avoids error prone thread synchronization and shared memory state, which is otherwise often tempting to do

I am definitively thankful for the last hint, Sönke. Could you please guide me a bit in the direction where should I hook that callback in? I am prepared to contribute the solution in a PR with a little help of yours then :-)

I think I missed that the decision to which thread an incoming WS request belongs needs to be made based on the session ID. The callback based approach would basically just work for a very early decision, for example based on the remote IP address. Everything else would already have created too many thread-specific data dependencies.

Since all other thread transfer solutions would really be very involved (especially if thread safety is supposed to be guaranteed), would it maybe be possible to approach this from the other side and establish the web socket connection first and then the TCP connection afterwards? That would probably mean that the credentials would also need to go over the web socket and that the TCP connection can't outlive the web socket, but maybe that's acceptable?

Spontaneously, I'm a bit out of ideas for a true single-thread solution. As another shot in the dark, would it make sense to try to keep the network code to a single thread, and to just offload computationally expensive tasks to worker threads?

As a final thought, if there is no practical solution on the application side, enabling moving of just the TCP connection between threads would be a lot easier than for web socket connections:

  • Implementing the reverse of EventDriverSockets.adoptStream in eventcore
    • makes sure that there is only a single reference to the stream socket left
    • clears all data from the slot associated with the socket and removes it from the event loop
    • returns the socket FD without closing it
  • Implement a special snapshot type within vibe-core
    • captures the socket FD, as well as the current state of a TCP connection (contents of TCPConnection.m_context)
    • is typed immutable or shared, so that it can be passed between threads
  • A new API function would then copy the state and use the eventcore function above to create such a snapshot
  • Another API function would use adoptStream and would reapply the state to create a new TCPConnection

Just naming these things well will be interesting...