RejectedSoftware Forums

Sign up

How to write input range in http request?

I have a lazy input range that produce alot of data (e.g. 1GB). I want to send this data via http without creating temporary array. The most intuitive way would be overloaded method "writeBody" that takes any range of bytes or chars. But there is no such overload. I tried to use StreamOutputRange but it looks wierd:

(scope req) {
	auto writer = StreamOutputRange(req.bodyWriter);
	foreach (c; range) {
		writer.put(c);
	}
}

I am sure that it is not effective and requires internal buffer for whole data. Is it possible to move this foreach iteration into vibe.d? Something like

(scope req) {
	auto reader = InputRangeStream(range); // must be template to inline ranges iteration
	req.writeBody(reader);
},

In this case vibe.d may write data directly to output buffer after this delegate completed. No additional memory required (only small buffer for one tcp frame that already exists). I am not sure about Content-Length. In case of RandomAccessRange everything is ok, in case of InputRange the only way is chunked encoding.

Same improvement would be perfect for raw TCP.

Re: How to write input range in http request?

On Fri, 14 Oct 2016 11:49:21 GMT, Grigorii Smorkalov wrote:

I have a lazy input range that produce alot of data (e.g. 1GB). I want to send this data via http without creating temporary array. The most intuitive way would be overloaded method "writeBody" that takes any range of bytes or chars. But there is no such overload. I tried to use StreamOutputRange but it looks wierd:

(scope req) {
	auto writer = StreamOutputRange(req.bodyWriter);
	foreach (c; range) {
		writer.put(c);
	}
}

I am sure that it is not effective and requires internal buffer for whole data. Is it possible to move this foreach iteration into vibe.d? Something like

(scope req) {
	auto reader = InputRangeStream(range); // must be template to inline ranges iteration
	req.writeBody(reader);
},

In this case vibe.d may write data directly to output buffer after this delegate completed. No additional memory required (only small buffer for one tcp frame that already exists). I am not sure about Content-Length. In case of RandomAccessRange everything is ok, in case of InputRange the only way is chunked encoding.

Same improvement would be perfect for raw TCP.

The StreamOutputRange approach has the drawback of having to copy the data once, but other than that is efficient (it copies and writes the data to output stream in chunks). However, due to the nature of ranges, this copy cannot be avoided, except for the corner case of plain arrays. The reverse approach with InputRangeStream would have to do exactly the same, just with a different API.

If you know the length of the stream, you can set the Content-Length header before writing to bodyWriter, otherwise it will default to chunked transfers.

Generally, I think it makes sense to offer a writeBody overload that encapsulates this logic. An OutputStream.write overload taking an input range could also make sense.

I've opened a ticket #1594

Re: How to write input range in http request?

On 10/14/16 7:49 AM, Grigorii Smorkalov wrote:

I have a lazy input range that produce alot of data (e.g. 1GB). I want to send this data via http without creating temporary array. The most intuitive way would be overloaded method "writeBody" that takes any range of bytes or chars. But there is no such overload. I tried to use StreamOutputRange but it looks wierd:

(scope req) {
	auto writer = StreamOutputRange(req.bodyWriter);
	foreach (c; range) {
		writer.put(c);
	}
}

I am sure that it is not effective and requires internal buffer for whole data.

I did something similar for an input range of JSON objects. I think this
is optimal, even if you may disagree. It must buffer the data somewhere,
because i/o requires a buffer. I don't think the copying is avoidable.

However, support for range output/input would be awesome for vibe.d.
Especially in REST interfaces, putting all data into an array can be
impossible and expensive. This is just a stream, we should be able to
process it one item at a time, and D has extensive support for such a
concept!

-Steve