RejectedSoftware Forums

Sign up

CPU spikes / busy loop with FileDescriptorEvent?

I'm seeing some weird behaviour with FileDescriptorEvents. Consider this reduced example for vibed 0.7.20:

module main;

import core.sys.posix.sys.socket, core.sys.posix.fcntl;
import std.datetime, std.stdio;
import vibe.d;

void main()
{
    auto c = new C();
    runTask(&c.test);
    setIdleHandler(&c.idle);
    runEventLoop();
}

class C
{
    bool idle()
    {
        static SysTime time;
        writeln(Clock.currTime - time);
        time = Clock.currTime;
        return false;
    }

    void test()
    {

        auto address = resolveHost("google.com", AF_INET, true);
        address.port = 80;
        auto fd = socket(address.family, SOCK_STREAM, 0);
        auto flags = fcntl(fd, F_GETFL, 0);
        flags = flags | O_NONBLOCK;
        assert(fcntl(fd, F_SETFL, flags) == 0);
        connect(fd, address.sockAddr, address.sockAddrLen);
        auto _watcher = createFileDescriptorEvent(fd,FileDescriptorEvent.Trigger.any);
        _watcher.wait(dur!"seconds"(30));
    }
}

When I run this the idle handler is called with very small delays (~3 μs) for some time and the OS task manager shows 100% CPU consumption. If I use FileDescriptorEvent for real I/O operations I see 100% CPU load all the time.

With this simple example the CPU load normalizes after a few seconds but with more complicated examples I also managed to keep the CPU busy even after all tasks have finished. I can't seem to reproduce that now though.

OS is linux x86_64, compilers dmd 2065 or gdc 2064, libevent backend, vibed 0.7.20.

Is there anything wrong with the way I'm using FileDescriptorEvent or is this some kind of bug?

Re: CPU spikes / busy loop with FileDescriptorEvent?

So I debugged this a little and two things caught my attention:

Problem 1

https://github.com/rejectedsoftware/vibe.d/blob/master/source/vibe/core/drivers/libevent2.d#L747

while ((m_activeEvents & which) == Trigger.none)
    getThreadLibeventDriverCore().yieldForEvent();

What if I created the FileDescriptorEvent with Trigger.any, then wait(Trigger.read)? A connection will almost always be signalled as writeable, and as long as I don't write data it will stay writeable. So the event will trigger all the time and set m_activeEvents to Trigger.write causing a busy loop?

I think vibed should rearm the event instead of using a repeating event, at least if the trigger changed.

Problem 2

But the real problem here is different: The event is not unarmed fast enough:
After wait yields here onFileTriggered is called once. The first time evt.m_waiter is set, the task is continued and wait returns as expected. However, the event is still alive and onFileTriggered will be called many more times this time doing effectively nothing. evt.m_waiter is null, so it only sets the triggers but nobody ever reads/writes data on the file descriptor and onFileTriggered fires again and again. I'm not sure why this even stops eventually is this only because of the GC collecting the FileDescriptorEvent? This is obviously also a problem even if data is still being read.

The naive solution to both problems it to create the event in wait, then delete it as soon as wait completes. I guess you ruled that out for performance reasons?

The other solution for problem two is to event_add the event at the start of wait and event_del before wait returns. Problem 1 still needs to be solved then, though. Probably by creating a new event when the trigger type changes.

Re: CPU spikes / busy loop with FileDescriptorEvent?

On Sat, 07 Jun 2014 08:12:10 GMT, Johannes Pfau wrote:

So I debugged this a little and two things caught my attention:

Problem 1

https://github.com/rejectedsoftware/vibe.d/blob/master/source/vibe/core/drivers/libevent2.d#L747

while ((m_activeEvents & which) == Trigger.none)
    getThreadLibeventDriverCore().yieldForEvent();

What if I created the FileDescriptorEvent with Trigger.any, then wait(Trigger.read)? A connection will almost always be signalled as writeable, and as long as I don't write data it will stay writeable. So the event will trigger all the time and set m_activeEvents to Trigger.write causing a busy loop?

I think vibed should rearm the event instead of using a repeating event, at least if the trigger changed.

Problem 2

But the real problem here is different: The event is not unarmed fast enough:
After wait yields here onFileTriggered is called once. The first time evt.m_waiter is set, the task is continued and wait returns as expected. However, the event is still alive and onFileTriggered will be called many more times this time doing effectively nothing. evt.m_waiter is null, so it only sets the triggers but nobody ever reads/writes data on the file descriptor and onFileTriggered fires again and again. I'm not sure why this even stops eventually is this only because of the GC collecting the FileDescriptorEvent? This is obviously also a problem even if data is still being read.

The naive solution to both problems it to create the event in wait, then delete it as soon as wait completes. I guess you ruled that out for performance reasons?

The other solution for problem two is to event_add the event at the start of wait and event_del before wait returns. Problem 1 still needs to be solved then, though. Probably by creating a new event when the trigger type changes.

There is a third possible cause for the CPU spikes - the wait overload that takes a Duration uses a timer to handle the timeout, but there is currently an issue in rearmTimer where it doesn't always remove the timer from the internal timer heap. This can cause the heap to grow to huge sizes, if rearmTimer is used repeatedly. Eventually, once time progresses, the heap will be cleared again.

I've created a ticket (#695) to remember this. I'll try to perform some testing in the coming days.