I try to download the new version of dub and vibe.d from code.dlang.org, as well as vibed.org, all of them behave weird, the download abruptly breaks at a random point EVERY time wget retries to download it. Is it a network problem, or does anyone else have the same situation?
I have the same problem.
I suspect it is because of a bug in vibe.d.
Am 28.11.2013 20:00, schrieb Uplink_Coder:
I have the same problem.
I suspect it is because of a bug in vibe.d.
I see it, too. It seems to be a bug in the reverse proxy. I'm looking
into it now.
On 2013-11-28 14:10, Sönke Ludwig wrote:
Am 28.11.2013 20:00, schrieb Uplink_Coder:
I have the same problem.
I suspect it is because of a bug in vibe.d.I see it, too. It seems to be a bug in the reverse proxy. I'm looking
into it now.
That's probably why I'm unable to download all of the message headers
for vibe.d's newsgroup (NNTP server) using thunderbird. It stops
randomly after a second, between 700 and 1100 messages.
Am 28.11.2013 15:22, schrieb zhaopuming:
I try to download the new version of dub and vibe.d from code.dlang.org, as well as vibed.org, all of them behave weird, the download abruptly breaks at a random point EVERY time wget retries to download it. Is it a network problem, or does anyone else have the same situation?
I was able isolate the cause now and enabled a workaround for the two
sites, so downloads should work again, unless the connection is very
slow (download takes more than an hour).
It's currently still very unclear why that issue occurs now and not
right from the beginning, but what's clear so far is that this happens:
On the server, there is a reverse proxy and a separate process for each
site. When the site process now writes the response containing the
downloaded file, the whole file seems to be buffered somewhere and the
request is handled almost immediately. The site process in turn starts
waiting for the next request and times out after the keep alive timeout
(was 10 seconds). However, the reverse proxy is still forwarding data
from the socket of the site process to the socket of the requesting
client. Once the timeout happens, the site process terminates the
connection and the reverse proxy cannot read the rest of the data in the
buffer anymore.
What's still unclear is where that buffer is and how to wait for it to
be flushed. The workaround now was to increase the keep alive timeout to
an hour.
On 2013-11-28 16:16, Sönke Ludwig wrote:
Am 28.11.2013 15:22, schrieb zhaopuming:
I try to download the new version of dub and vibe.d from code.dlang.org, as well as vibed.org, all of them behave weird, the download abruptly breaks at a random point EVERY time wget retries to download it. Is it a network problem, or does anyone else have the same situation?
I was able isolate the cause now and enabled a workaround for the two
sites, so downloads should work again, unless the connection is very
slow (download takes more than an hour).It's currently still very unclear why that issue occurs now and not
right from the beginning, but what's clear so far is that this happens:On the server, there is a reverse proxy and a separate process for each
site. When the site process now writes the response containing the
downloaded file, the whole file seems to be buffered somewhere and the
request is handled almost immediately. The site process in turn starts
waiting for the next request and times out after the keep alive timeout
(was 10 seconds). However, the reverse proxy is still forwarding data
from the socket of the site process to the socket of the requesting
client. Once the timeout happens, the site process terminates the
connection and the reverse proxy cannot read the rest of the data in the
buffer anymore.What's still unclear is where that buffer is and how to wait for it to
be flushed. The workaround now was to increase the keep alive timeout to
an hour.
This didn't fix the vibe.d newsgroup headers download issue for NNTP /
Thunderbird. It goes up towards 1800 now rather than 700, but it still
drops. Could you check? I don't know if it worked in the past.
On 2013-11-28 18:04, Etienne wrote:
On 2013-11-28 16:16, Sönke Ludwig wrote:
Am 28.11.2013 15:22, schrieb zhaopuming:
I try to download the new version of dub and vibe.d from
code.dlang.org, as well as vibed.org, all of them behave weird, the
download abruptly breaks at a random point EVERY time wget retries to
download it. Is it a network problem, or does anyone else have the
same situation?I was able isolate the cause now and enabled a workaround for the two
sites, so downloads should work again, unless the connection is very
slow (download takes more than an hour).It's currently still very unclear why that issue occurs now and not
right from the beginning, but what's clear so far is that this happens:On the server, there is a reverse proxy and a separate process for each
site. When the site process now writes the response containing the
downloaded file, the whole file seems to be buffered somewhere and the
request is handled almost immediately. The site process in turn starts
waiting for the next request and times out after the keep alive timeout
(was 10 seconds). However, the reverse proxy is still forwarding data
from the socket of the site process to the socket of the requesting
client. Once the timeout happens, the site process terminates the
connection and the reverse proxy cannot read the rest of the data in the
buffer anymore.What's still unclear is where that buffer is and how to wait for it to
be flushed. The workaround now was to increase the keep alive timeout to
an hour.This didn't fix the vibe.d newsgroup headers download issue for NNTP /
Thunderbird. It goes up towards 1800 now rather than 700, but it still
drops. Could you check? I don't know if it worked in the past.
Sorry, your changes didn't change anything it still blocks around 700. I
have a feeling it's because it doesn't use Keep-Alive to keep the
connection open.
Am 29.11.2013 00:09, schrieb Etienne:
On 2013-11-28 18:04, Etienne wrote:
On 2013-11-28 16:16, Sönke Ludwig wrote:
Am 28.11.2013 15:22, schrieb zhaopuming:
I try to download the new version of dub and vibe.d from
code.dlang.org, as well as vibed.org, all of them behave weird, the
download abruptly breaks at a random point EVERY time wget retries to
download it. Is it a network problem, or does anyone else have the
same situation?I was able isolate the cause now and enabled a workaround for the two
sites, so downloads should work again, unless the connection is very
slow (download takes more than an hour).It's currently still very unclear why that issue occurs now and not
right from the beginning, but what's clear so far is that this happens:On the server, there is a reverse proxy and a separate process for each
site. When the site process now writes the response containing the
downloaded file, the whole file seems to be buffered somewhere and the
request is handled almost immediately. The site process in turn starts
waiting for the next request and times out after the keep alive timeout
(was 10 seconds). However, the reverse proxy is still forwarding data
from the socket of the site process to the socket of the requesting
client. Once the timeout happens, the site process terminates the
connection and the reverse proxy cannot read the rest of the data in the
buffer anymore.What's still unclear is where that buffer is and how to wait for it to
be flushed. The workaround now was to increase the keep alive timeout to
an hour.This didn't fix the vibe.d newsgroup headers download issue for NNTP /
Thunderbird. It goes up towards 1800 now rather than 700, but it still
drops. Could you check? I don't know if it worked in the past.Sorry, your changes didn't change anything it still blocks around 700. I
have a feeling it's because it doesn't use Keep-Alive to keep the
connection open.
Okay the original problem should be solved now (instead of just a
workaround): The buffer that was responsible for this was libevent's
input buffer of the reverse proxy for the connection to the server
process. This buffer is now limited to avoid unlimited memory usage. But
while this also fixes the bug, there was a logic error on top of this.
Responses from the server process were only read until the connection
was terminated, and any remaining contents in the input buffer was just
dropped. The behavior has now been corrected and the input buffer gets
always completely processed regardless of the remote hanging up.
I'm not sure if this also fixes the NNTP problem, but I'll investigate
that tomorrow (need to get some sleep first).
On Thu, 28 Nov 2013 16:00:03 -0500, Etienne wrote:
On 2013-11-28 14:10, Sönke Ludwig wrote:
Am 28.11.2013 20:00, schrieb Uplink_Coder:
I have the same problem.
I suspect it is because of a bug in vibe.d.I see it, too. It seems to be a bug in the reverse proxy. I'm looking
into it now.That's probably why I'm unable to download all of the message headers
for vibe.d's newsgroup (NNTP server) using thunderbird. It stops
randomly after a second, between 700 and 1100 messages.
Found it. It was caused by a malformed message header, which caused the client to disconnect before the whole list was received. Headers are now properly sanitized (393d100).
On 2013-11-29 3:11 AM, Sönke Ludwig wrote:
On Thu, 28 Nov 2013 16:00:03 -0500, Etienne wrote:
On 2013-11-28 14:10, Sönke Ludwig wrote:
Am 28.11.2013 20:00, schrieb Uplink_Coder:
I have the same problem.
I suspect it is because of a bug in vibe.d.I see it, too. It seems to be a bug in the reverse proxy. I'm looking
into it now.That's probably why I'm unable to download all of the message headers
for vibe.d's newsgroup (NNTP server) using thunderbird. It stops
randomly after a second, between 700 and 1100 messages.Found it. It was caused by a malformed message header, which caused the client to disconnect before the whole list was received. Headers are now properly sanitized (393d100).
Wow, everything downloaded in microseconds now :)