RejectedSoftware Forums

Sign up

Running through a queue with multiple workers

Hi guys,

I'm pretty sure there is an elegant way to do this in vibe so maybe somebody can point me in the right direction. We interact with an external API which takes about 10 seconds per request. About once a day we need to request updates for records that are pending. One request per record and we are looking at 200-400 records. While the response is slow, we have been advised by our vendor that it has no issues with concurrent processing as they have a large farm and the API will will take up to 10 concurrent requests per user and we are being encouraged to utilise this.

So I want to take an array of ids numbering 200+ and and then do concurrent requests to the API with a fixed limit. Say no more than 5 requests running at a time. As each request finishes, grab the next id from the queue until there are none left. Although I can see that I can create a bunch of tasks and then wait for them with join, I can't find an elegant way to do the limiting.

Any help please?

David.

Re: Running through a queue with multiple workers

On Tue, 30 Sep 2014 03:42:40 GMT, David Monagle wrote:

Hi guys,

I'm pretty sure there is an elegant way to do this in vibe so maybe somebody can point me in the right direction. We interact with an external API which takes about 10 seconds per request. About once a day we need to request updates for records that are pending. One request per record and we are looking at 200-400 records. While the response is slow, we have been advised by our vendor that it has no issues with concurrent processing as they have a large farm and the API will will take up to 10 concurrent requests per user and we are being encouraged to utilise this.

So I want to take an array of ids numbering 200+ and and then do concurrent requests to the API with a fixed limit. Say no more than 5 requests running at a time. As each request finishes, grab the next id from the queue until there are none left. Although I can see that I can create a bunch of tasks and then wait for them with join, I can't find an elegant way to do the limiting.

Any help please?

David.

I'd do it exactly as you described and start only N tasks (with N being the limit):

Item[] queue;

void worker()
{
    while (!queue.empty) {
        // take the next item from the queue
        auto itm = queue.front;
        queue.popFront();

        // ... perform request for itm ...
    }
}

Task[] tasks;
foreach (i; 0 .. 5) tasks ~= runTask(&worker);
foreach (t; tasks) t.join();

Since all tasks live in the same thread, the code can stay pretty simple.

This sounds like a useful idiom to have directly in vibe.d, maybe it's time to start a parallelism module.

Re: Running through a queue with multiple workers

On Tue, 30 Sep 2014 06:47:23 GMT, Sönke Ludwig wrote:

On Tue, 30 Sep 2014 03:42:40 GMT, David Monagle wrote:

Hi guys,

I'm pretty sure there is an elegant way to do this in vibe so maybe somebody can point me in the right direction. We interact with an external API which takes about 10 seconds per request. About once a day we need to request updates for records that are pending. One request per record and we are looking at 200-400 records. While the response is slow, we have been advised by our vendor that it has no issues with concurrent processing as they have a large farm and the API will will take up to 10 concurrent requests per user and we are being encouraged to utilise this.

So I want to take an array of ids numbering 200+ and and then do concurrent requests to the API with a fixed limit. Say no more than 5 requests running at a time. As each request finishes, grab the next id from the queue until there are none left. Although I can see that I can create a bunch of tasks and then wait for them with join, I can't find an elegant way to do the limiting.

Any help please?

David.

I'd do it exactly as you described and start only N tasks (with N being the limit):

Item[] queue;

void worker()
{
    while (!queue.empty) {
        // take the next item from the queue
        auto itm = queue.front;
        queue.popFront();

        // ... perform request for itm ...
    }
}

Task[] tasks;
foreach (i; 0 .. 5) tasks ~= runTask(&worker);
foreach (t; tasks) t.join();

Since all tasks live in the same thread, the code can stay pretty simple.

This sounds like a useful idiom to have directly in vibe.d, maybe it's time to start a parallelism module.

Time to start a parellelism module? Your TODO list must be massive...

Thanks for the example. Makes perfect sense and really I should have been able to put that together. Just approaching the problem from the wrong way. Thanks again!