RejectedSoftware Forums

i made it work more or less, basically i got confused because msgpack-rpc-d has "VibeCustomMain" set, which i didn't notice, so i needed to manually call runEventLoop.

the remaining question is on how to start a runWorkerTask and share its tId with later spawned Tasks.
when i call runWorkerTaskH from main it fails as there is no valid tID in main. as a temporary solution
i wrapped runWorkerTaskH (and runTask for spawning a server) into another runTask but then i get

[warn] eventbaseloop: reentrant invocation. Only one eventbaseloop can run on each event_base at once.

which i guess is not optimal.

=> how can i share a WorkerTask's id with tasks started from main (so those can message the worker)?

On Sat, 06 Dec 2014 16:27:49 GMT, yawniek wrote:

how is it possible to start both, a webserver and listening to a tcp socket and then have those
communicating with a separate worker thread.

essentially what i want to do is have a Rest API and a msgpack-rpc that can put/set values into leveldb.
thus leveldb shall run in another thread and both, the web and rpc interface should be able to communicate with it.

i'm more or less stuck in that when i call listenTCP (via msgpack-rpc-d's TCPServer) in main/this there is no valid Task.getThis() and thus runWorkerTaskH fails.

On Sun, 07 Dec 2014 05:27:50 GMT, Etienne Cimon wrote:

On Sat, 06 Dec 2014 18:18:42 GMT, yawniek wrote:

=> how can i share a WorkerTask's id with tasks started from main (so those can message the worker)?

A worker task still has access to the global scope. You can set the Tid from the worker task in this gobal scope:

__gshared Tid g_worker; 
__gshared TaskMutex g_mtx;

Any regular task can consult the global scope to figure out how to communicate with it. Use something like this:

synchronized(g_mtx)
	g_worker.send( ... )

Is your leveldb wrapper doing blocking TCP or is it just an embedded connector? It seems odd having to put it in a foreign thread, but I've done this already for async files:

https://github.com/etcimon/libasync/blob/master/source/libasync/threads.d

Thanks Etienne, very helpful hints.

i probably want your async branch as soon as i start to shard over multiple leveldb backends.
and yes indeed, leveldb does block. My benchmark inserts around 70M 128bit keys (no values) in 4M batches, when writing the batch it takes a few seconds. (4M was empirically the best batch size on my macbook, https://gist.github.com/yannick/b638cab9f76d845446da).

not sure on how to handle reads though, is there a way to do blocking calls into Workers or do i need to send a struct with the sender and then reply?
maybe i go back to just having a fiber and then buffer writes within a decoupled server that also does the sharding.