RejectedSoftware Forums

Sign up

Pages: 1 2 3

Re: Status of Win32 version (listenTcp, etc)

But a simpler alternative to a message queue is to use a vibe.core.mutex.Mutex, which interacts gracefully with the event loop and then lock and aquire the connection only during a send:

So a vibe Mutex will yield the fiber when blocked? That's indeed an interesting solution.

Exactly. (In other words, it behaves like a usual mutex, but lets other fibers run while it's blocked.)

Btw. do you need to guarantee the same order of packets for all clients? If so, I think a message queue or a kind of mutex is needed anyway. Of course, it's possible to handle out-of-order packets on the receiver side, just wondering.

Yeah as I mentioned I don't really have this constraint, but I'm starting to warm up to the notion of just storing a short packet queue for each client. If I use a "Signal" like in the broadcast example, I'm assuming I lose a lot of the niceness of blocking "deep" within TcpConnection routines (send/receive) though, right? i.e. because I need to be able to pop all the way back up to the top level and handle something unrelated (i.e. broadcast), and then find out how to go back down to where I was? Seems to negate a lot of the Fibering model if I'm forced to do it that way though. I think I'd rather it just work that if the fiber is blocked on "something else" (send/receive say) that it stays blocked, and only when it finishes all that stuff will it get back to the message queue.

I guess I'm not totally clear how Signal's interact with other events, including ones internal to Vibe. Is it basically a "wait on multiple objects" sort of scenario? If so, will the Vibe send/receive routines wake up, check that they still are blocked and yield again if the event being emitted is not relevant to what the Fiber is currently doing?

The Signal is usable in two ways:

  1. The normal way, wait()ing for the signal to be emitted. Multiple fibers can wait (block) for the signal and all will continue when the singal is emitted. In this case, there is no visible concurrency between the signal and any other asynchronous object such as the TCP connection - they are just running in sequence from the view of the fiber.

  2. The way that is used in the WebSocket broadcast example. Each fiber first acquires the signal to receive events from it (a signal can be owned by multiple fibers at once). By using rawYield(), the fibers are then waiting for any event concerning them, which in this example can either be the signal that has been emitted, or the TCP connection which has new data available for read. Events always only affect those fibers that own the corresponding object in the in the acquire()/release() sense. So this scenario is indeed basically "wait for multilpe objects", where the multiple objects are all objects that have been acquired by the fiber.

    After rawYield() continues, the event that caused this can be indirectly determined by asking the TCP connection if data is available, or by asking the Signal if the emit count was increased. So there is no direct access to the underlying event that caused rawYield to continue. The following read() and write() operations will again block as usual, though, even if the Signal is emitted again in the meantime - they ignore all other events that may occur while they are waiting for "their" events.

    Since this broadcast construct in fact uses implementation details/internals, I really want to wrap this into a high-level primitive instead. I consider this just a temporary compromise to make this kind of bidirectional braodcasting work at all, with the danger that it relies on the way events are handled internally, which might (although probably not) change in the future or between different back ends.

I'm afraid I was too lazy to return some kind of handle from listenTcp that would make this possible ;) Will do as soon as possible.

No issues, I can live without it in the short term as well :) Was just curious if I had missed some bit of API.

Thanks for all the information! Getting pretty close to switching the mainline over to the vibe version :)

Sound good! Btw., it's still at least half a year off until I'll get to this, but I also plan to use vibe.d in a game/simulation engine... currently I'm held up by a lot of other stuff, but it would be a dream to be able to work on this kind off stuff again :)

Re: Status of Win32 version (listenTcp, etc)

The Signal is usable in two ways:

  1. The normal way, wait()ing for the signal to be emitted. Multiple fibers can wait (block) for the signal and all will continue when the singal is emitted.

Ok so this is effectively what a TcpConnection will do when I do a "receive" for N bytes until that many bytes are available? So no other signals than the one it is waiting on will wake the fiber at that point, correct? That's actually the desirable behaviour I think.

After `rawYield()` continues, the event that caused this can be indirectly determined by asking the TCP connection if data is available, or by asking the `Signal` if the emit count was increased.

Alright, that makes sense. I implemented this model with a simple queue of outgoing packets. The main loop basically checks for incoming data, and consumes it all if any is available (it'll block waiting for a full packet to come in, etc). After that if there is anything pending in the send queue it will flush that all out (again, presumably blocking if the socket send buffer fills) and then do a rawYield. Basically the same model as the broadcast example, except I use one signal per client which fires when anything is added to that client's queue, as not all cross-client messages are broadcasts.

Seems to work properly, so modulo figuring out a policy for maximum send queue size, etc. I'm happy :)

I consider this just a temporary compromise to make this kind of bidirectional braodcasting work at all, with the danger that it relies on the way events are handled internally, which might (although probably not) change in the future or between different back ends.

Right, I'll put a big warning in my code that it might break in the future :) Presumably if it does break, the broadcast example will be updated with the new "best known method"?

Sound good! Btw., it's still at least half a year off until I'll get to this, but I also plan to use vibe.d in a game/simulation engine... currently I'm held up by a lot of other stuff, but it would be a dream to be able to work on this kind off stuff again :)

Cool, yeah it's a lot of fun :) This project is a volunteer "spare time" type of thing for an old game, but doing it in D with fibers and lots of clever mixins (for stuff like swapping endianness of structures) has made the code extremely concise and fun to play around with!

Thanks for all your help. I think everything is pretty much up and running now in terms of what was already working. Next is to tackle the HTTP stuff and make it interact nicely with a web server (as a client, for now).

Re: Status of Win32 version (listenTcp, etc)

On Fri, 21 Dec 2012 00:21:25 GMT, punkUser wrote:

The Signal is usable in two ways:

  1. The normal way, wait()ing for the signal to be emitted. Multiple fibers can wait (block) for the signal and all will continue when the singal is emitted.

Ok so this is effectively what a TcpConnection will do when I do a "receive" for N bytes until that many bytes are available? So no other signals than the one it is waiting on will wake the fiber at that point, correct? That's actually the desirable behaviour I think.

Yes, correct

After rawYield() continues, the event that caused this can be indirectly determined by asking the TCP connection if data is available, or by asking the Signal if the emit count was increased.

Alright, that makes sense. I implemented this model with a simple queue of outgoing packets. The main loop basically checks for incoming data, and consumes it all if any is available (it'll block waiting for a full packet to come in, etc). After that if there is anything pending in the send queue it will flush that all out (again, presumably blocking if the socket send buffer fills) and then do a rawYield. Basically the same model as the broadcast example, except I use one signal per client which fires when anything is added to that client's queue, as not all cross-client messages are broadcasts.

Seems to work properly, so modulo figuring out a policy for maximum send queue size, etc. I'm happy :)

I consider this just a temporary compromise to make this kind of bidirectional braodcasting work at all, with the danger that it relies on the way events are handled internally, which might (although probably not) change in the future or between different back ends.

Right, I'll put a big warning in my code that it might break in the future :) Presumably if it does break, the broadcast example will be updated with the new "best known method"?

One thing I thought about was writing a messaging API, compatible to std.concurrency, based on the fiber aware Signal/Mutex classes. That would cover such use cases in a nice and safe way and would also allow for painless migration if std.concurrency should ever support such fiber-aware thread primitives.

Re: Status of Win32 version (listenTcp, etc)

Just updated to the latest released version (0.7.10) and it seems like listen sockets aren't working for me now. The function appears to succeed and everything, but there doesn't appear to be a socket listening... any attempts to connect to it from another client just immediately fail.

I'm still using the libevent implementation. Any ideas?

Re: Status of Win32 version (listenTcp, etc)

Am 06.01.2013 05:11, schrieb punkUser:

Just updated to the latest released version (0.7.10) and it seems like listen sockets aren't working
for me now. The function appears to succeed and everything, but there doesn't appear to be a socket
listening... any attempts to connect to it from another client just immediately fail.

I'm still using the libevent implementation. Any ideas?

That's unexpected. Do you get a "connection refused" error or something else?

Can you try to setLogLevel(LogLevel.Trace) and put the log on pastebin or similar? I'll take a look.

Re: Status of Win32 version (listenTcp, etc)

That's unexpected. Do you get a "connection refused" error or something else?

No, it's as if the socket is not even listening to the client it appears. Same behavior as if the server isn't running at all.

Can you try to setLogLevel(LogLevel.Trace) and put the log on pastebin or similar? I'll take a look.

Did that but vibe doesn't seem to write anything at all to the log... if I add a logDebug() message it goes there, but my listenTcp/runEventLoop calls don't appear to log anything. I stepped through them and indeed they don't seem to have any logging calls in the libevent backend?

Re: Status of Win32 version (listenTcp, etc)

Hmm... the plot thickens: the firewall seems to see that vibe is hosting a listen socket (asks if it should be allowed), but still nothing is able to connect to it.

Even this relatively simple code doesn't work...

import vibe.vibe;
import std.stdio;

int main(string[] argv)
{
setLogFile("vibe_log.txt", LogLevel.Trace);

listenTcp(80, (conn) {

writeln(conn.peerAddress());

} );

runEventLoop();
return 0;
}

Chrome will say "couldn't connect", nothing printed in console or log, etc.

I realize I'm using vibe in a sort of weird way, but I'd like to control the entry points and link it just as a library rather than using the surrounding build infrastructure. It seemed to be working fine before, but not anymore :(

So to be clear what I'm doing is...

1) using provided visual D project
2) adding "vibe.d" and a few missing files in templ to it
3) removing stream/stream from it to eliminate warning
4) removing VibeWin32Driver version from compile options
5) linking resulting library in with my code (for example, the above)

Is this supposed to work or am I bypassing too much of the infastructure? Is there a preferred way for me to accomplish what I'm trying to do? Am I just missing some sort of initialization function or similar maybe?

Re: Status of Win32 version (listenTcp, etc)

So I've verified that the socket seems to be getting created okay, but the onConnect event that is supposed to be linked to it via libevent is never getting called. Not sure how to debug it further as libevent is somewhat opaque.

Re: Status of Win32 version (listenTcp, etc)

Aha I think I found the issue... I'm calling the default "address-less" version of listenTcp then connecting to localhost. Before in vibe this would forward to "listenTcp(port, connection_callback, "0.0.0.0");", but as of the latest version it uses "::" as the default address instead. For some reason this is disrupting my ability to connect via IPv4.

Adding "0.0.0.0" as an explicit address to my listenTcp calls seems to have resolved the issue. I'm not sure if I fully understand the new logic in the address-less listenTcp to be honest... I'm guess the issue is that multiple listeners are created but the TcpListener object that was added doesn't support that so the idea is to return null instead? Except that return null only happens if the first listenTcp succeeds, and if that is the case it never calls the second one now. Perhaps a bug?

Re: Status of Win32 version (listenTcp, etc)

On Sun, 06 Jan 2013 20:52:25 GMT, punkUser wrote:

1) using provided visual D project
2) adding "vibe.d" and a few missing files in templ to it

I'll fix that in the default project.

3) removing stream/stream from it to eliminate warning

This, too.

4) removing VibeWin32Driver version from compile options

I just implemented listenTcp in the latest version on git master, so you could try to skip this step now.

5) linking resulting library in with my code (for example, the above)

Is this supposed to work or am I bypassing too much of the infastructure? Is there a preferred way for me to accomplish what I'm trying to do? Am I just missing some sort of initialization function or similar maybe?

This should definitely be fine. There is nothing special going on regarding the build otherwise, only that 'vibe' automatically downloads dependencies, if specified.

Pages: 1 2 3