On Wed, 27 Nov 2013 19:19:15 +0100, Sönke Ludwig wrote:

Am 27.11.2013 18:30, schrieb Matthew Fong:

I just now spotted this website while chatting: http://bitcoinity.org/markets/mtgox/USD
This is a realtime charting website for BTC. It seems to use EventSource/Server Sent Events to push new transactions to all connected clients. Now, currently there are roughly 17000 clients connected and I was asking myself two things:

  1. How easily could we integrate Server Sent Events in the current version of Vibe.d? My guess is that one could use it already using the HttpServerResponse, it doesn't offer a nice API AFAIK though.

It looks like not much is needed to implement it - just setting
HTTPServerResponse.contentType = "text/event-stream"; and using
HTTPServerResponse.bodyWriter to write successive events, right?

  1. How much memory does a Task/Fiber (eg. of libev) use and how many fibers are practically usable?

A fiber uses a minimum of a single memory page (typically 4k) of
reserved memory and can then scale up to the size set using
setTaskStackSize (16k by default). It will always consume the full
stack size in virtual address space though, so a 64-bit system should be
used.

I'v just read that nginx seems to use less memory per connection, quotes from here http://www.aosabook.org/en/nginx.html :

on an idle keepalive connection, nginx spends just 550 bytes of memory. A possible optimization for future releases of nginx would be to reuse and share memory buffers for long-lived connections.

So I wonder, is it possible to optimize and further reduce a fiber's memory usage? or use some share memory buffers in thread local?

The 17k connections would thus result in a minimum amount of about 66MB
used for fibers and about the same amount on top for handling the HTTP
request. There is also some additional overhead for libevent and the
message queue that each task has (this should probably rather be created
lazily, but currently isn't). But all in all, everything should fit well
inside sane boundaries.