RejectedSoftware Forums

Sign up

HTTPClient

There is something wrong with HTTPClient. I can't download any page.
Windows 7 64, dmd 2.063.2, vibe.d last master.
This code

import vibe.d;

shared static this() {
    runTask({
        logInfo("start task");
        requestHTTP("http://vibed.org", (scope req) {}, (scope res) {
            logInfo("downloading");
            string data = res.bodyReader.readAllUTF8();
            logInfo("length = %s", data.length);
        });
        logInfo("end task");
    });
}

prints

d:\tmp\test>test --vvv
[02250F80:00000000 dbg] Create FreeListAlloc 8
[02250F80:00000000 dbg] Create FreeListAlloc 16
[02250F80:00000000 dbg] Create FreeListAlloc 32
[02250F80:00000000 dbg] Create FreeListAlloc 64
[02250F80:00000000 dbg] Create FreeListAlloc 128
[02250F80:00000000 dbg] Create FreeListAlloc 256
[02250F80:00000000 dbg] Create FreeListAlloc 512
[02250F80:00000000 dbg] Create FreeListAlloc 1024
[02250F80:00000000 dbg] Create FreeListAlloc 2048
[02250F80:00000000 dbg] Create FreeListAlloc 65540
[02250F80:00000000 dia] libevent version: 2.0.20-stable
[02250F80:00000000 dia] libevent is using win32 for events.
[02250F80:00000000 dbg] Initializing OpenSSL...
[02250F80:00000000 dbg] ... done.
[02250F80:00000000 dbg] Creating new fiber...
[02250F80:00000000 dia] Running event loop...
start task
[02250F80:02250E80 dbg] creating new HTTPClient connection, all 0 are in use
[02250F80:02250E80 dbg]  ... 2252C00
[02250F80:02250E80 dbg] Now got 1 connections
[02250F80:02250E80 dbg] Disconnected to avoid timeout
[02250F80:02250E80 dbg] dnsresolve
[02250F80:02250E80 dbg] dnsresolve yield
[02250F80:02250E80 dbg] dnsresolve ret 0
[02250F80:00000000 dbg] Socket event on fd 328: 128 (2C88E4 vs 2C88E4)

package.json

{
    "name": "test",
    "dependencies": {
        "vibe-d": "~master"
    }
}

Re: HTTPClient

On Mon, 30 Sep 2013 09:53:54 GMT, Jack Applegame wrote:

There is something wrong with HTTPClient. I can't download any page.
Windows 7 64, dmd 2.063.2, vibe.d last master.
(...)

Turns out the issue was the URL "http://vibed.org". There was no check to see if the part after the host name is empty and the resulting request was "GET HTTP/1.1" instead of "GET / HTTP/1.1". Fixed now on master.

Re: HTTPClient

Another issue. I write OpenID authentication system (Valve Steam community). I found some stack corruption or something similar while downloading some gzipped pages.
In particular, auth checking page of the Steam OpenID system. Test case:

shared static this() {
    runTask({
        logInfo("start task");
        requestHTTP("https://steamcommunity.com/openid/login?openid.mode=check_authentication", (scope req) {
            //req.headers.remove("Accept-Encoding");
        }, (scope res) {
            logInfo("downloading");
            string data = res.bodyReader.readAllUTF8();
            logInfo("length = %s", data.length);
        });
        logInfo("end task");
    });
}

Looks like responder callback never returns. Log misses "end task":

d:\tmp\test>test
start task
downloading
length = 51

If turn off compression by removing Accept-Encoding header, then downloading works fine.

Re: HTTPClient

Am 30.09.2013 19:06, schrieb Jack Applegame:

Another issue. I write OpenID authentication system (Valve Steam community). I found some stack corruption or something similar while downloading some gzipped pages.
In particular, auth checking page of the Steam OpenID system. Test case:
(...)

It's possible that this is an issue of the GzipInputStream - I've
reimplemented it recently using zlib directly and the zlib API is so
awful that I can't guarantee for anything right now :/. But using
std.zlib had numerous issues, too, and also requires a lot of
superfluous GC allocations, so I think this is the better long term
solution.

I'll look into it.

Re: HTTPClient

On Mon, 30 Sep 2013 19:13:28 +0200, Sönke Ludwig wrote:

Am 30.09.2013 19:06, schrieb Jack Applegame:

Another issue. I write OpenID authentication system (Valve Steam community). I found some stack corruption or something similar while downloading some gzipped pages.
In particular, auth checking page of the Steam OpenID system. Test case:
(...)

It's possible that this is an issue of the GzipInputStream - I've
reimplemented it recently using zlib directly and the zlib API is so
awful that I can't guarantee for anything right now :/. But using
std.zlib had numerous issues, too, and also requires a lot of
superfluous GC allocations, so I think this is the better long term
solution.

I'll look into it.

Fixed. Was introduced by the fix for #191.