RejectedSoftware Forums

Sign up

Threading issues

Hi!

I'm running into some weird threading issue with vibe.

My app basically listens for HTTP requests, and each request pushes a task into the worker-thread queues with runWorkerTask(), saves a file to disk using vibe's non-blocking FileStream.write(), and then waits for the task to finish with receiveOnly().

The worker task basically does some blocking IO through a foreign library, and then sends the results back to the request fiber with send();

This approach should be solid, right?

This all works fine as long as there is only one request at any given time. When I introduce concurrent requests, as little as 2, it immediately crashes, with no callstack or error message on Linux. On Windows I get a "Resuming task in foreign thread." assert.

I was wondering what is the best way to debug these? Workarounds? Ideas? :)

Cheers,
-M

Re: Threading issues

On Wed, 17 Jun 2015 17:52:16 GMT, Márcio Martins wrote:

Hi!

I'm running into some weird threading issue with vibe.

My app basically listens for HTTP requests, and each request pushes a task into the worker-thread queues with runWorkerTask(), saves a file to disk using vibe's non-blocking FileStream.write(), and then waits for the task to finish with receiveOnly().

The worker task basically does some blocking IO through a foreign library, and then sends the results back to the request fiber with send();

This approach should be solid, right?

This all works fine as long as there is only one request at any given time. When I introduce concurrent requests, as little as 2, it immediately crashes, with no callstack or error message on Linux. On Windows I get a "Resuming task in foreign thread." assert.

I was wondering what is the best way to debug these? Workarounds? Ideas? :)

Cheers,
-M

I would try to rule out any culprits. Do blocking file I/O instead, replace the call to the foreign library with a sleep, remove any parameters in runWorkerTask, try using runTask instead of runWorkerTask, etc.

Once you find what resolves the issue, you might discover that you forgot a mutex along the way or that you have an assertion failure somewhere that got eaten up.

Re: Threading issues

Thanks Etienne!

The issue was that I had a shared vibe ConnectionPool handing over TCP sockets to every worker thread. This doesn't seem to work with vibe's ConnectionPool, so I'm now creating a ConnectionPool per thread, which is sort of wasteful especially when running on a machine with many cores, but works.

Cheers,
-M

Re: Threading issues

On Thu, 18 Jun 2015 17:21:57 GMT, Márcio Martins wrote:

Thanks Etienne!

The issue was that I had a shared vibe ConnectionPool handing over TCP sockets to every worker thread. This doesn't seem to work with vibe's ConnectionPool, so I'm now creating a ConnectionPool per thread, which is sort of wasteful especially when running on a machine with many cores, but works.

Cheers,
-M

The ConnectionPool is not thread-safe, were you protecting it with a TaskMutex?

Re: Threading issues

No, for some reason, I assumed it was thread-safe without making sure... That came back to megabyte me.

I switched to TLS instead of locking as it was the quickest patch to get it not crashing. I will revisit this at a later point.

It might be worth-while to have ConnectionPool be thread-safe. Either by default and/or through some locking-policy template parameter. Otherwise we'd have to lock on the user code, which might mean the lock will be held for much more than necessary, though I have not looked at the implementation yet.
At the very least it reduces the risk of end-users getting it wrong, like I did.