On Sat, 02 Jun 2018 08:21:31 GMT, Sönke Ludwig wrote:

On Fri, 01 Jun 2018 18:55:26 GMT, Carl Sturtivant wrote:

On Fri, 01 Jun 2018 13:45:41 GMT, Carl Sturtivant wrote:

From a vibe.d application on linux: I don't know if this is vibe.d related, am investigating.

A computationally intensive process run from the command line works fine, runs to completion after several minutes, writing a few hundred lines of text to standard output and creating, writing to and closing around 200 files of size around 20KB.

Now run from std.process.pipeProcess and periodically tested for completion with tryWait interleaved with sleep everything seems fine for a while. htop reveals that one core is running this at 100% CPU, and the first 76 files appear one after another, but the 77th file is opened and nothing is written to it, and htop reveals that the CPU usage has dropped to zero, and yet the process is still running according to ps, and this continues indefinitely, no error message, no indication from tryWait that it is done. htop does not reveal at any point that memory use is even half of what is available.

Previously with similar processes that are a somewhat scaled back version of the one that fails as above, there's been no difference between what happened at the command line and what's happening here.

Any ideas or suggestions?

Well, it looks like I simply overflowed the stdout pipe buffer for the process. I am reading output from that buffer when the process is done so as not to block. And I have a loop that periodically sleeps and non-blocking-tests whether the process is done so as to keep other fibers alive while the process runs.

Now I need to read from the processes stdout while not blocking in that loop. Any advice about the right way to go about this in a vibe.d application?

There is a thread based wrapper that can be used to drain any file descriptor (std.stdio.File) without blocking the calling thread: StdFileStream

When handling many file descriptors, this is currently quite inefficient due to the required thread(s), but this particular situation it should fit perfectly. Setting one up works as like this:

import vibe.stream.stdio : StdFileStream;

    auto s = new StdFileStream(true, false); // read=yes, write=no
    while (!s.empty) {
        ubyte[256] buf;
        auto len = min(s.leastSize, buf.len);
        s.read(buf[0 .. len]);
        // write buf[0 .. len] somewhere

Thanks. Nice, and just what I need.