On 2013-12-02 08:35:17 +0000, Sönke Ludwig said:

On Sun, 01 Dec 2013 15:25:27 -0500, Etienne wrote:

On 2013-12-01 02:34, Sönke Ludwig wrote:

Am 01.12.2013 04:14, schrieb Etienne:

I'm writing a Lua D adaptation for Vibe.d to add scripting support. When

requesting data from HDD using loadFrom(path), would there be a function

that yields during a HDD request?

Do you mean loadfile as used for loading a script? In that case I'd

simply ignore HDD I/O as the file size should be small enough for I/O

not to really matter. If it's about loading data from disk from inside a

LUA script, you'd probably have to register your own function for

loading the contents and skip loading the one from the LUA standard

library (or unregister/overwrite it)*.

I saw posix has aioread(&cbAio) and windows has FILEFLAG_OVERLAPPED

for async HDD file reading. It might be quite easy to overload the

std.file.read functions for yielding in the future, I know my

application will be receiving lots of huge files that need to be parsed

into a database with vibe.d, so anything above the likely 200MB/s of HDD

load could easily block the application entirely, it'll have to be

delegated to the kernel until it's inside a buffer. Using vibe.d,

aioread and FILEFLAG_OVERLAPPED seems like the easy way out for me.

I'm wondering though, do you see any complications coming up with this idea?

So is this related to LUA or HD access in general?

The win32 driver already uses FILE_FLAG_OVERLAPPED (currently
requires "subConfigurations": {"vibe-d": "win32"} but will be the
default on Windows soon), but since libevent doesn't support async.
file I/O, the libevent driver still uses standard blocking access,
albeit with intermittent yield() calls, so it shouldn't usually block
event processing. It is planned to offload disk I/O to a thread pool to
improve that.

I'm not really sure how well aio_read and the likes can be mixed with
libevent from the outside. But at some point I'd like to make a native
wrapper around epoll/kqueue and the like instead of libevent. Then
async. file I/O will also be implemented natively instead of relying on
threading.

What kinds of things does libevent do that would be needed to be implemented?