RejectedSoftware Forums

Sign up

[VibeD] Hand made proxy

Greetings.

I am trying to learn Vibe.D and rewrite using it, my old project I've wrote with my (already rotting) personal framework

Project been written as commercial project for one of my clients so I can't really release source code here, but in tl;dr it is:

Proxy that gets file from remote server (process of getting file itself is rather complicated) and then serves it to the connected client.

In my old project I used threads because there are no more than 15 clients connected same time. There was main thread listening for connections, and 15 worker threads that were making bridge between client and remote server.

Now I tried few different approaches with Vibe.d but all had some issues.
The far I could get was this:
http://dpaste.dzfl.pl/d4dabd0a [1]

But it still has some issues. It works great for small files but for larger
(~150MB) VibeD just closes remote connection and web browsers just keep trying to receive something until it timeouts.

Another issue which will probably come is the fact that Vibe.D won't be able to serve more than 1 client at the same time. At least my attempts with 2MB file resulted with: 1st client downloading file, 2nd client getting 500 from vibe.d

Question is then: How to properly handle it with VibeD? How do you handle more threads with VibeD?

Aha, and please don't get into details like "your buffer is too small" etc. It isn't important at this stage.

Thanks in advance!

PS. I am cross posting it from forum.dlang.org:
http://forum.dlang.org/post/jugkmleyzqmzimyeahoi@forum.dlang.org

[1]

import vibe.d;

import std.stdio;

void handle(HTTPServerRequest, HTTPServerResponse res)
{

string link = "http://yourdomain.com/file_here";

requestHTTP(link, null,
    (scope HTTPClientResponse r)
    {
        enum BUFF_SIZE = 1024;
        ubyte[BUFF_SIZE] buff;
        ulong last_read = 0;

        res.headers = r.headers;
        res.headers["Content-Disposition"] = `attachment; filename="fname.gz"`;
        
        while (r.bodyReader().empty == false)
        {
            last_read = r.bodyReader().leastSize;
            if (last_read > BUFF_SIZE)
                r.bodyReader().read(buff);
            else
                r.bodyReader().read(buff[0..last_read]);

            res.bodyWriter.write(buff[0..(last_read > BUFF_SIZE ? 1024 : last_read)]);

            if (last_read < BUFF_SIZE) 
            {
                break;
            }
        }
    }
);

}

static this()
{

setLogLevel(LogLevel.trace);

auto settings = new HTTPServerSettings;
settings.hostName = "localhost";
settings.port = 8004;
settings.bindAddresses = ["127.0.0.1"];

auto router = new URLRouter;
router.get("/download/*", toDelegate(&handle));

listenHTTP(settings, router);

}

Re: [VibeD] Hand made proxy

On Sun, 11 Aug 2013 15:12:48 GMT, nazriel wrote:

(...)

if (last_read < BUFF_SIZE) 
{
    break;
}

(...)

This check should be removed, as it may very well trigger in the middle of the transfer. It's the number of bytes in the receive buffer of the socket and that may be just a few bytes for example if the client reads faster than the origin server writes.

You can also replace the whole while loop with the following, which will basically do exactly the same thing:

res.bodyWriter.write(r.bodyReader);

The issue with parallel requests is strange though. What is the exact exception/call stack which causes the 500 error? Can you maybe try to use a StopWatch or log messages to see where it hangs in each request handler?

Re: [VibeD] Hand made proxy

On Mon, 12 Aug 2013 17:45:55 GMT, Sönke Ludwig wrote:

On Sun, 11 Aug 2013 15:12:48 GMT, nazriel wrote:

(...)

if (last_read < BUFF_SIZE) 
{
    break;
}

(...)

This check should be removed, as it may very well trigger in the middle of the transfer. It's the number of bytes in the receive buffer of the socket and that may be just a few bytes for example if the client reads faster than the origin server writes.

You can also replace the whole while loop with the following, which will basically do exactly the same thing:

res.bodyWriter.write(r.bodyReader);


I could swear I used it and I had issues with it.
No idea what happened back then but now it just works hah.

Everything seems to work now.
The only issue I have got is the fact, that when I set loglevel to trace, and client disconnects (for example stops download), app seems to still download file from remote server (I maybe wrong here. Got bunch of:

evbuffer_read 10592 bytes (fd 10)
 .. got 1448 bytes
yield
socket 10 read event!
resume
evbuffer_read 9144 bytes (fd 10)
 .. got 1448 bytes

And those appears after client connects/server starts fetching file

Shouldn't it stop downloading remote file after user disconnects in my scenario?
(

res.bodyWriter.write(r.bodyReader);

)

Thanks alot!

Re: [VibeD] Hand made proxy

On Mon, 12 Aug 2013 20:42:15 GMT, nazriel wrote:

On Mon, 12 Aug 2013 17:45:55 GMT, Sönke Ludwig wrote:

On Sun, 11 Aug 2013 15:12:48 GMT, nazriel wrote:

(...)

if (last_read < BUFF_SIZE) 
{
    break;
}

(...)

This check should be removed, as it may very well trigger in the middle of the transfer. It's the number of bytes in the receive buffer of the socket and that may be just a few bytes for example if the client reads faster than the origin server writes.

You can also replace the whole while loop with the following, which will basically do exactly the same thing:

res.bodyWriter.write(r.bodyReader);


I could swear I used it and I had issues with it.
No idea what happened back then but now it just works hah.

Everything seems to work now.
The only issue I have got is the fact, that when I set loglevel to trace, and client disconnects (for example stops download), app seems to still download file from remote server (I maybe wrong here. Got bunch of:

(...)

Shouldn't it stop downloading remote file after user disconnects in my scenario?

I think this may happen because the body writer stream is a ChunkedOutputStream in this case, which will happily buffer without checking the underlying connection. I'll add a manual check and also make the default chunk size limited to avoid using up all memory in case of large files.

Re: [VibeD] Hand made proxy

Am 16.08.2013 11:34, schrieb Sönke Ludwig:

On Mon, 12 Aug 2013 20:42:15 GMT, nazriel wrote:

On Mon, 12 Aug 2013 17:45:55 GMT, Sönke Ludwig wrote:

On Sun, 11 Aug 2013 15:12:48 GMT, nazriel wrote:

(...)

 if (last_read < BUFF_SIZE)
 {
     break;
 }

(...)

This check should be removed, as it may very well trigger in the middle of the transfer. It's the number of bytes in the receive buffer of the socket and that may be just a few bytes for example if the client reads faster than the origin server writes.

You can also replace the whole while loop with the following, which will basically do exactly the same thing:

 res.bodyWriter.write(r.bodyReader);


I could swear I used it and I had issues with it.
No idea what happened back then but now it just works hah.

Everything seems to work now.
The only issue I have got is the fact, that when I set loglevel to trace, and client disconnects (for example stops download), app seems to still download file from remote server (I maybe wrong here. Got bunch of:

(...)

Shouldn't it stop downloading remote file after user disconnects in my scenario?

I think this may happen because the body writer stream is a ChunkedOutputStream in this case, which will happily buffer without checking the underlying connection. I'll add a manual check and also make the default chunk size limited to avoid using up all memory in case of large files.

Hmm actually this is not true in this particular case.
ChunkedOutputStream.write is overloaded for the stream case and does
the right thing. I would normally expect it to exit with an exception as
soon as the client connection gets terminated - I'll look into it when I
get the chance.