requestHTTP performance - RejectedSoftware Forums

Zdeněk

Posted Tue, 09 Jun 2015 20:01:41 GMT

Hello everyone,

I've been having some performance problems in my REST interface and I've tracked it down to the performance of requestHTTP.

I've created a very simple application which demonstrates this problem:

shared static this()
{
    import std.stdio;
    import vibe.http.client;
    import vibe.stream.operations;

    int cnt = 0;
    foreach(i; 0..1_000)
    {
        requestHTTP("an example URL I am working with",
            (scope req) {
                req.method = HTTPMethod.GET;
            },
            (scope res) {
                res.bodyReader.readAll();
                cnt++;
                if (i%50 == 0) i.writeln;
            }
        );
    }
}

This takes about 40 seconds, which is way too much. I am certain that it's not caused by the server, as that can do a few thousands requests per second.

What I've noticed is that the application probably spends a lot of time waiting, because it doesn't use a lot of processing power (it uses about 1% of my CPU). This is expected, as it has to wait for the network (even though I'm running the server on the same machine), but still seems too low.

The situation gets better when I run the request in a task (using runTask), but such an approach requires a lot of memory for all the tasks and is not applicable in my case, where my server has to send a request to another server when it itself receives a request.

Any ideas how to fix this? Thanks.