Is it safe to require users to call flush accordingly when implementing a stream?

Permalink: HTTP NNTP

Posted Fri, 31 Mar 2017 07:44:43 GMT

I've implemented a noise crypto stream wrapper for vibe.D: http://code.dlang.org/packages/vibe-noisestream

Currently whenever a user calls write(ubyte[]) I simply encrypt that data and call the downstream write function. However, if the users writes many small packets that increases the protocol overhead as every encrypted packet needs additional 2+16 bytes for crypto. So I was wondering whether it is safe to concat the data from multiple writes and send it only when the internal buffer is full or the user calls flush. Is it reasonable to expect users to properly flush streams?

Another unrelated problem, maybe someone has got a good idea: The upcoming noisesocket protocol will likely allow packets up to 2^16-1 bytes. A packet can only be decrypted at once so the complete packet needs top be cached. This leads to huge buffers which is especially annoying when using many streams in parallel. So has anyone already thought about or implemented some kind of buffer sharing between streams / Fibers in vibe.d?

Re: Is it safe to require users to call flush accordingly when implementing a stream?

Permalink: HTTP NNTP

Sönke Ludwig

Posted Sun, 2 Apr 2017 12:08:37 +0200 in reply to Johannes Pfau

Am 31.03.2017 um 09:44 schrieb Johannes Pfau:

I've implemented a noise crypto stream wrapper for vibe.D: http://code.dlang.org/packages/vibe-noisestream

Currently whenever a user calls write(ubyte[]) I simply encrypt that data and call the downstream write function. However, if the users writes many small packets that increases the protocol overhead as every encrypted packet needs additional 2+16 bytes for crypto. So I was wondering whether it is safe to concat the data from multiple writes and send it only when the internal buffer is full or the user calls flush. Is it reasonable to expect users to properly flush streams?

By spec this is be safe to do and at least all places in vibe.d itself
will call flush properly (including the StreamOutputRange proxy).
.finalize() should also call flush() internally.

Another unrelated problem, maybe someone has got a good idea: The upcoming noisesocket protocol will likely allow packets up to 2^16-1 bytes. A packet can only be decrypted at once so the complete packet needs top be cached. This leads to huge buffers which is especially annoying when using many streams in parallel. So has anyone already thought about or implemented some kind of buffer sharing between streams / Fibers in vibe.d?

I don't know of an existing implementation in this area. The closest
thing would probably be the ConnectionPool, which uses simple
reference counting and a LocalTaskSemaphore to implement a limited
cross-task resource sharing mechanism.

Re: Is it safe to require users to call flush accordingly when implementing a stream?

Permalink: HTTP NNTP

Johannes Pfau

Posted Mon, 03 Apr 2017 18:42:25 GMT in reply to Sönke Ludwig

On Sun, 2 Apr 2017 12:08:37 +0200, Sönke Ludwig wrote:

By spec this is be safe to do and at least all places in vibe.d itself
will call flush properly (including the StreamOutputRange proxy).
.finalize() should also call flush() internally.

OK thanks, so I'll implement this optimization. If finalize is supposed to call flush internally, finalize can't be called on an closed stream (i.e. if the other side closed the connection). Am I supposed to check if the connection is closed in finalize or is this the users responsibility? Also is it valid to free certain C handles in finalize?

Another unrelated problem, maybe someone has got a good idea: The upcoming noisesocket protocol will likely allow packets up to 2^16-1 bytes. A packet can only be decrypted at once so the complete packet needs top be cached. This leads to huge buffers which is especially annoying when using many streams in parallel. So has anyone already thought about or implemented some kind of buffer sharing between streams / Fibers in vibe.d?

I don't know of an existing implementation in this area. The closest
thing would probably be the ConnectionPool, which uses simple
reference counting and a LocalTaskSemaphore to implement a limited
cross-task resource sharing mechanism.

Hmm OK. Looks like we'll get buffer size negotiation in the noise socket protocol so this might be less important problem than I initially thought.