RejectedSoftware Forums

Sign up

Pages: 1 2

readAll() blocks, what is the alternative?

Hi,

I am evaluating Vibe.d for my next project. It will be used as a TCP and UDP server.

I just compiled basic TCP example but the momnet I wanted to do simple change things started to work unexpectedly.

In my simple example client app just writes bytes to socket and then starts reading.

Things work ok when handler looks like this:

void onConnection(TCPConnection conn)
{
  conn.write(conn);
}

but when I change it to:

void onConnection(TCPConnection conn)
{
  auto data = conn.readAll;
  conn.write(data);
}

handler never writes data back because readAll() blocks. In other thread I found the answer: realAll() blocks until client closes connection. Answer also suggests readLine() but in my case it wont work as data is binary.

My question: how to read InputStream the same way conn.write(conn) does in the first version?

Re: readAll() blocks, what is the alternative?

On Fri, 07 Mar 2014 22:48:49 GMT, hardcoder wrote:

Hi,

I am evaluating Vibe.d for my next project. It will be used as a TCP and UDP server.

I just compiled basic TCP example but the momnet I wanted to do simple change things started to work unexpectedly.

In my simple example client app just writes bytes to socket and then starts reading.

Things work ok when handler looks like this:

void onConnection(TCPConnection conn)
{
  conn.write(conn);
}

but when I change it to:

void onConnection(TCPConnection conn)
{
  auto data = conn.readAll;
  conn.write(data);
}

handler never writes data back because readAll() blocks. In other thread I found the answer: realAll() blocks until client closes connection. Answer also suggests readLine() but in my case it wont work as data is binary.

My question: how to read InputStream the same way conn.write(conn) does in the first version?

You can get really creative interacting with the TCPConnection with just conn.read(dst) and conn.write(src). There's a lot of helpers if you look around in the library but here's a few ways: 1) you can use the primitives, 2) you can use readLine() by making sure you write a \n after every statement or 3) you could also push some objects directly down the stream using a serialization library.

1) The most common way is to allocate the destination bytes and simply read to them:

while (conn.waitForData(5.seconds)){
	ubyte[] len = new ubyte[1];
	conn.read(len);
	ubyte[] data = new ubyte[len];
	conn.read(data);
}

(2) You can send the data followed by a line separation
conn.write(cast(ubyte[])"hello there\n")
and on the other end:
auto recv = conn.readLine();

(3) Using a serialization library like msgpack-d you can send your objects and rebuild them at the other end of your tcp connection. In vibe you have json serialization or you can also use msgpack-d
https://github.com/msgpack/msgpack-d

import std.file;
import msgpack;

struct S { int x; float y; string z; }

void onConnection(TCPConnection conn)
{
	S input = S(10, 25.5, "message");

	// serialize data
	ubyte[] inData = pack(input);
	
	// write data to the stream
	conn.write(inData);
}

on the other end...

void onConnection(TCPConnection conn)
{
	S input = S(10, 25.5, "message");
	// read data from a file
	ubyte[] outData = new ubyte[150];
	
	conn.read(outData);
	
	// unserialize the data
	S target = outData.unpack!S();
	
	// verify data is the same
	assert(target.x == input.x);
	assert(target.y == input.y);
	assert(target.z == input.z);
}

Note that you could use another type of buffering mechanism to read the data to such as a FixedRingBuffer I gave it 150 ubytes for simplification.

Hope it helps!

Re: readAll() blocks, what is the alternative?

On Sat, 08 Mar 2014 01:42:13 GMT, Etienne Cimon wrote:

Hope it helps!

1) I don't know the length of the data upfront. Normally I would go with this solution but it is broken too as reading in while loop never leaves conn.read()
as it tries to read even if there is nothing. This is broken:

  while (!conn.empty)
  {
    conn.read(buf);  // never leaves
    data ~= buf;
  }

read() should return number of bytes actually read, then say readBuffer() should block until buffer is filled, possibly taking timeout argument.

2) As I mentioned, readLine is a no-go.

3) This is high level logic already, I just want to read the data as client wrote it. And it will not work if client sends less than 150 bytes.

Why is this so difficult? IMHO readAll() is broken. Why would one ever possibly want a connection to close to signalize eof? It is super easy to send conn InputStream back, why is buffering this data so difficult?

Thanks a lot for your answer and suggestions but I think my evaluation is done :(

Re: readAll() blocks, what is the alternative?

read() should return number of bytes actually read,

You can call conn.leastSize() to get the length and then conn.read(ubyte[]) to receive the data.

then say readBuffer() should block until buffer is filled, possibly taking timeout argument.

I would use conn.readTimeout(Duration) here to stop waiting after a certain timeout. You can have very small timeouts in the std.datetime.Duration type

3) This is high level logic already, I just want to read the data as client wrote it. And it will not work if client sends less than 150 bytes.

That's one way to do it, but if you look carefully at the libraries based on TCP/IP you'll see they all attempt to offer other, more creative ways to read/write data.

Why is this so difficult? IMHO readAll() is broken. Why would one ever possibly want a connection to close to signalize eof? It is super easy to send conn InputStream back, why is buffering this data so difficult?

I agree that readAll() doesn't do what it says. I think it was mostly meant to get the data remaining in the buffers when a connection is closed.

It seems like you're looking to make a REST service. You should take a look at the examples available in the examples directory.

Good luck finding what you need :)

Re: readAll() blocks, what is the alternative?

On Fri, 07 Mar 2014 22:48:49 GMT, hardcoder wrote:

Hi,

I am evaluating Vibe.d for my next project. It will be used as a TCP and UDP server.

I just compiled basic TCP example but the momnet I wanted to do simple change things started to work unexpectedly.

In my simple example client app just writes bytes to socket and then starts reading.

Things work ok when handler looks like this:

void onConnection(TCPConnection conn)
{
  conn.write(conn);
}

but when I change it to:

void onConnection(TCPConnection conn)
{
  auto data = conn.readAll;
  conn.write(data);
}

handler never writes data back because readAll() blocks. In other thread I found the answer: realAll() blocks until client closes connection. Answer also suggests readLine() but in my case it wont work as data is binary.

My question: how to read InputStream the same way conn.write(conn) does in the first version?

Try Reading and writing in separate tasks - a bit like in this sample:

https://github.com/rejectedsoftware/vibe.d/blob/master/examples/tcp_separate/source/app.d

That way you don't need to wait until all data came in. Reading can be done in chunks then

Re: readAll() blocks, what is the alternative?

On Sun, 09 Mar 2014 02:20:08 GMT, Stephan wrote:

That way you don't need to wait until all data came in. Reading can be done in chunks then

I think it was for a war game with a C++ background. He was probably trying to rule out D from the start, didn't even look at the interface of the connection stream. Whenever a big improvement over the old comes up, resistance to change is in its way smiling gracefully; until the tide comes. :P

Re: readAll() blocks, what is the alternative?

On Sun, 09 Mar 2014 04:13:33 GMT, Etienne Cimon wrote:

On Sun, 09 Mar 2014 02:20:08 GMT, Stephan wrote:

That way you don't need to wait until all data came in. Reading can be done in chunks then

I think it was for a war game with a C++ background. He was probably trying to rule out D from the start, didn't even look at the interface of the connection stream. Whenever a big improvement over the old comes up, resistance to change is in its way smiling gracefully; until the tide comes. :P

You obviously did read the docs yet still can't answer my question. I never ask questions until I've read the docs and looked at examples. I asked because things don't work as they should, not because I don't know how to approach them. It is TCP there are not rocket science.

I also wrote test app using leastSize() property from the start (as this is used in conn.write(InputStream)) but this solution desn't work either because conn.will still block on empty(). Yes, I come from C++ so I expect things to be at least as good in newer languages. My client app is in C# and as a D enthusiast I wanted to use Vibe.d for backend.

Re: readAll() blocks, what is the alternative?

On Sun, 09 Mar 2014 02:20:08 GMT, Stephan wrote:

On Fri, 07 Mar 2014 22:48:49 GMT, hardcoder wrote:

Hi,

I am evaluating Vibe.d for my next project. It will be used as a TCP and UDP server.

I just compiled basic TCP example but the momnet I wanted to do simple change things started to work unexpectedly.

In my simple example client app just writes bytes to socket and then starts reading.

Things work ok when handler looks like this:

void onConnection(TCPConnection conn)
{
  conn.write(conn);
}

but when I change it to:

void onConnection(TCPConnection conn)
{
  auto data = conn.readAll;
  conn.write(data);
}

handler never writes data back because readAll() blocks. In other thread I found the answer: realAll() blocks until client closes connection. Answer also suggests readLine() but in my case it wont work as data is binary.

My question: how to read InputStream the same way conn.write(conn) does in the first version?

Try Reading and writing in separate tasks - a bit like in this sample:

https://github.com/rejectedsoftware/vibe.d/blob/master/examples/tcp_separate/source/app.d

That way you don't need to wait until all data came in. Reading can be done in chunks then

Sorry but I don't see how reading in different task changes semantics or readAll().

Re: readAll() blocks, what is the alternative?

On Sun, 09 Mar 2014 09:56:10 GMT, hardcoder wrote:

On Sun, 09 Mar 2014 02:20:08 GMT, Stephan wrote:

On Fri, 07 Mar 2014 22:48:49 GMT, hardcoder wrote:

Hi,

I am evaluating Vibe.d for my next project. It will be used as a TCP and UDP server.

I just compiled basic TCP example but the momnet I wanted to do simple change things started to work unexpectedly.

In my simple example client app just writes bytes to socket and then starts reading.

Things work ok when handler looks like this:

void onConnection(TCPConnection conn)
{
  conn.write(conn);
}

but when I change it to:

void onConnection(TCPConnection conn)
{
  auto data = conn.readAll;
  conn.write(data);
}

handler never writes data back because readAll() blocks. In other thread I found the answer: realAll() blocks until client closes connection. Answer also suggests readLine() but in my case it wont work as data is binary.

My question: how to read InputStream the same way conn.write(conn) does in the first version?

Try Reading and writing in separate tasks - a bit like in this sample:

https://github.com/rejectedsoftware/vibe.d/blob/master/examples/tcp_separate/source/app.d

That way you don't need to wait until all data came in. Reading can be done in chunks then

Sorry but I don't see how reading in different task changes semantics or readAll().

Well you simply do not want to use readAll when you want to pipe chunks of incoming data to another stream.
use read() for reading (and waiting) for chunks of data until the stream end is reached. yes read will block in this case too, but that does not matter, because of the concept of tasks. it is just that one task that gets blocked.
if you even don't want to chunk it and send every byte as it arrives, you can use waitForData which blocks until any data arrives and ask how much of a chunk actually did arrive using leastSize.

Re: readAll() blocks, what is the alternative?

On Sun, 09 Mar 2014 09:56:10 GMT, hardcoder wrote:

Sorry but I don't see how reading in different task changes semantics or readAll().

I couldn't find any tcp interface other than Qt with the readAll() method.

http://qt-project.org/doc/qt-4.8/qiodevice.html#readAll

This interface is designed without tasks in mind. The i/o methods are all non-blocking by design to avoid freezing a thread, because they're built on a fiber-less scheme.

On the other hand, vibe.d craves blocking because it makes the code linear, although with fibers behind the scene the thread never blocks!

See the following example:

string data;
runTask({ data = cast(string) conn.readUntil(cast(ubyte[])"|"); });
some_long_routine(data);

Without runTask, data wll be available to somelongroutine(). But with it, the data will be fetched while somelongroutine executes with empty data. Note: All of this happens within the same thread! You'd need a complex set of threading tricks or slots/signals in Qt/C++ to achieve this unless you're a boost::asio/co-routine guru.

All in all, you can't compare the semantics behind vibe.d's ConnectionStream interface with Qt's QIODevice, it's like trying to compare qt's signals/slots with c# events/delegates.

Pages: 1 2