I'm writing a Lua D adaptation for Vibe.d to add scripting support. When
requesting data from HDD using loadFrom(path), would there be a function
that yields during a HDD request?
Am 01.12.2013 04:14, schrieb Etienne:
I'm writing a Lua D adaptation for Vibe.d to add scripting support. When
requesting data from HDD using loadFrom(path), would there be a function
that yields during a HDD request?
Do you mean loadfile as used for loading a script? In that case I'd
simply ignore HDD I/O as the file size should be small enough for I/O
not to really matter. If it's about loading data from disk from inside a
LUA script, you'd probably have to register your own function for
loading the contents and skip loading the one from the LUA standard
library (or unregister/overwrite it)*.
On 2013-12-01 02:34, Sönke Ludwig wrote:
Am 01.12.2013 04:14, schrieb Etienne:
I'm writing a Lua D adaptation for Vibe.d to add scripting support. When
requesting data from HDD using loadFrom(path), would there be a function
that yields during a HDD request?Do you mean loadfile as used for loading a script? In that case I'd
simply ignore HDD I/O as the file size should be small enough for I/O
not to really matter. If it's about loading data from disk from inside a
LUA script, you'd probably have to register your own function for
loading the contents and skip loading the one from the LUA standard
library (or unregister/overwrite it)*.
I saw posix has aioread(&cbAio) and windows has FILEFLAGOVERLAPPED
for async HDD file reading. It might be quite easy to overload the
std.file.read functions for yielding in the future, I know my
application will be receiving lots of huge files that need to be parsed
into a database with vibe.d, so anything above the likely 200MB/s of HDD
load could easily block the application entirely, it'll have to be
delegated to the kernel until it's inside a buffer. Using vibe.d,
aioread and FILEFLAGOVERLAPPED seems like the easy way out for me.
I'm wondering though, do you see any complications coming up with this idea?
On Sun, 01 Dec 2013 15:25:27 -0500, Etienne wrote:
On 2013-12-01 02:34, Sönke Ludwig wrote:
Am 01.12.2013 04:14, schrieb Etienne:
I'm writing a Lua D adaptation for Vibe.d to add scripting support. When
requesting data from HDD using loadFrom(path), would there be a function
that yields during a HDD request?Do you mean loadfile as used for loading a script? In that case I'd
simply ignore HDD I/O as the file size should be small enough for I/O
not to really matter. If it's about loading data from disk from inside a
LUA script, you'd probably have to register your own function for
loading the contents and skip loading the one from the LUA standard
library (or unregister/overwrite it)*.I saw posix has aioread(&cbAio) and windows has FILEFLAGOVERLAPPED
for async HDD file reading. It might be quite easy to overload the
std.file.read functions for yielding in the future, I know my
application will be receiving lots of huge files that need to be parsed
into a database with vibe.d, so anything above the likely 200MB/s of HDD
load could easily block the application entirely, it'll have to be
delegated to the kernel until it's inside a buffer. Using vibe.d,
aioread and FILEFLAGOVERLAPPED seems like the easy way out for me.I'm wondering though, do you see any complications coming up with this idea?
So is this related to LUA or HD access in general?
The win32 driver already uses FILE_FLAG_OVERLAPPED
(currently requires "subConfigurations": {"vibe-d": "win32"}
but will be the default on Windows soon), but since libevent doesn't support async. file I/O, the libevent driver still uses standard blocking access, albeit with intermittent yield()
calls, so it shouldn't usually block event processing. It is planned to offload disk I/O to a thread pool to improve that.
I'm not really sure how well aio_read
and the likes can be mixed with libevent from the outside. But at some point I'd like to make a native wrapper around epoll/kqueue and the like instead of libevent. Then async. file I/O will also be implemented natively instead of relying on threading.
On 2013-12-02 08:35:17 +0000, Sönke Ludwig said:
On Sun, 01 Dec 2013 15:25:27 -0500, Etienne wrote:
On 2013-12-01 02:34, Sönke Ludwig wrote:
Am 01.12.2013 04:14, schrieb Etienne:
I'm writing a Lua D adaptation for Vibe.d to add scripting support. When
requesting data from HDD using loadFrom(path), would there be a function
that yields during a HDD request?
Do you mean loadfile as used for loading a script? In that case I'd
simply ignore HDD I/O as the file size should be small enough for I/O
not to really matter. If it's about loading data from disk from inside a
LUA script, you'd probably have to register your own function for
loading the contents and skip loading the one from the LUA standard
library (or unregister/overwrite it)*.
I saw posix has aioread(&cbAio) and windows has FILEFLAG_OVERLAPPED
for async HDD file reading. It might be quite easy to overload the
std.file.read functions for yielding in the future, I know my
application will be receiving lots of huge files that need to be parsed
into a database with vibe.d, so anything above the likely 200MB/s of HDD
load could easily block the application entirely, it'll have to be
delegated to the kernel until it's inside a buffer. Using vibe.d,
aioread and FILEFLAG_OVERLAPPED seems like the easy way out for me.
I'm wondering though, do you see any complications coming up with this idea?
So is this related to LUA or HD access in general?
The win32 driver already uses
FILE_FLAG_OVERLAPPED
(currently
requires"subConfigurations": {"vibe-d": "win32"}
but will be the
default on Windows soon), but since libevent doesn't support async.
file I/O, the libevent driver still uses standard blocking access,
albeit with intermittentyield()
calls, so it shouldn't usually block
event processing. It is planned to offload disk I/O to a thread pool to
improve that.I'm not really sure how well
aio_read
and the likes can be mixed with
libevent from the outside. But at some point I'd like to make a native
wrapper around epoll/kqueue and the like instead of libevent. Then
async. file I/O will also be implemented natively instead of relying on
threading.
What kinds of things does libevent do that would be needed to be implemented?
Am 02.12.2013 13:35, schrieb Shammah Chancellor:
On 2013-12-02 08:35:17 +0000, Sönke Ludwig said:
On Sun, 01 Dec 2013 15:25:27 -0500, Etienne wrote:
On 2013-12-01 02:34, Sönke Ludwig wrote:
Am 01.12.2013 04:14, schrieb Etienne:
I'm writing a Lua D adaptation for Vibe.d to add scripting
support. Whenrequesting data from HDD using loadFrom(path), would there be a
functionthat yields during a HDD request?
Do you mean loadfile as used for loading a script? In that case I'd
simply ignore HDD I/O as the file size should be small enough for I/O
not to really matter. If it's about loading data from disk from
inside aLUA script, you'd probably have to register your own function for
loading the contents and skip loading the one from the LUA standard
library (or unregister/overwrite it)*.
I saw posix has aioread(&cbAio) and windows has FILEFLAGOVERLAPPED
for async HDD file reading. It might be quite easy to overload the
std.file.read functions for yielding in the future, I know my
application will be receiving lots of huge files that need to be parsed
into a database with vibe.d, so anything above the likely 200MB/s of
HDD
load could easily block the application entirely, it'll have to be
delegated to the kernel until it's inside a buffer. Using vibe.d,
aioread and FILEFLAGOVERLAPPED seems like the easy way out for me.I'm wondering though, do you see any complications coming up with
this idea?So is this related to LUA or HD access in general?
The win32 driver already uses
FILE_FLAG_OVERLAPPED
(currently
requires"subConfigurations": {"vibe-d": "win32"}
but will be the
default on Windows soon), but since libevent doesn't support async.
file I/O, the libevent driver still uses standard blocking access,
albeit with intermittentyield()
calls, so it shouldn't usually
block event processing. It is planned to offload disk I/O to a thread
pool to improve that.I'm not really sure how well
aio_read
and the likes can be mixed
with libevent from the outside. But at some point I'd like to make a
native wrapper around epoll/kqueue and the like instead of libevent.
Then async. file I/O will also be implemented natively instead of
relying on threading.What kinds of things does libevent do that would be needed to be
implemented?
It's not that much really, basically it would look like the
Win32EventDriver structure-wise, just using epoll/kqueue/poll etc. as
the means to implement the event loop. All the buffering, filtering and
all that ther stuff that libevent provides is not really necessary.
On 2013-12-01 2:34 AM, Sönke Ludwig wrote:
Am 01.12.2013 04:14, schrieb Etienne:
I'm writing a Lua D adaptation for Vibe.d to add scripting support. When
requesting data from HDD using loadFrom(path), would there be a function
that yields during a HDD request?Do you mean loadfile as used for loading a script? In that case I'd
simply ignore HDD I/O as the file size should be small enough for I/O
not to really matter. If it's about loading data from disk from inside a
LUA script, you'd probably have to register your own function for
loading the contents and skip loading the one from the LUA standard
library (or unregister/overwrite it)*.
I just finished adapting this Lua wrapper for D - very useful for quick
debugging huge apps or just doing any runtime routines, it runs in a
vibe.d instance very well. It implements all functions and members of D
Classes in a lua interpreter like this:
Lua["varname"] = new SomeDClass();
https://github.com/JakobOvrum/LuaD/pull/60
e.g
import luad.all;
static void panic(LuaState lua, in char[] error)
{
throw new Exception(error);
}
struct Contact
{
string forename, surname;
string email;
uint number;
}
Contact[] fromFile(in char[] path)
{
auto lua = new LuaState;
Contact[] contacts;
lua["Contact"] = (Contact c)
{
contacts ~= c;
};
lua.doFile(path);
lua.setPanicHandler(&panic);
lua.doString(`function sum(n) return (n*n - n)/2; end
local s = 0
for i=1,100 do
s = s + i
assert(sum(i) == s, "Case "..i.." failed: expected "..s.." but got
"..sum(i).." instead.")
end`);
return contacts;
}
void index(HTTPServerRequest req, HTTPServerResponse res)
{
Contact[] phonebook = fromFile("contacts.lua");
res.render!("index.dt", req, phonebook);
}
contacts.lua :
Contact
{
forename = "Foo";
surname = "Bar";
email = "foobar@example.com";
number = 123456789;
}
Contact
{
forename = "John";
surname = "Smith";
email = "johnsmith@elpmaxe.moc";
number = 987654321;
}
On Sat, 30 Nov 2013 22:14:12 -0500, Etienne wrote:
I'm writing a Lua D adaptation for Vibe.d to add scripting support. When
requesting data from HDD using loadFrom(path), would there be a function
that yields during a HDD request?
Using LuaD with VibeD is a very good idea! Where can I get the code from and give it a try? :D
On 2013-12-03 4:08 AM, Dejan Lekic wrote:
On Sat, 30 Nov 2013 22:14:12 -0500, Etienne wrote:
I'm writing a Lua D adaptation for Vibe.d to add scripting support. When
requesting data from HDD using loadFrom(path), would there be a function
that yields during a HDD request?Using LuaD with VibeD is a very good idea! Where can I get the code from and give it a try? :D
I have a fork on github right here:
https://github.com/etcimon/LuaD
Just copy the folder locally and add this to your package.json
"dependencies": {
"luad": {"version": "~master", "path": "../../include/LuaD/"},
...
}
and then import lua.all in your vibe.d project
Having a scripting language on a compiled application is always a good
idea, especially if you're going to avoid rebuilding a live server every
time something changes ;) And you can always move CPU heavy operations
in D once in a while on every version iteration of your webserver.