How to Handle Long Running Processes on Server Side

Pages: 1 2 3

Re: How to Handle Long Running Processes on Server Side

Permalink: HTTP NNTP

Posted Wed, 12 Mar 2014 21:22:09 GMT in reply to Sönke Ludwig

On Tue, 11 Mar 2014 20:23:53 GMT, Sönke Ludwig wrote:

On Tue, 11 Mar 2014 20:03:22 GMT, Craig Dillabaugh wrote:

(...)
Since, I figure someone may ask what classifyTask does here it is:

clip

Is it possible that something I am doing in my classifyTask() function is blocking the server response?

The std.process.wait is the evil call here. There are two possibilities to solve this:

use runWorkerTask + proper synchronization instead of runTask

use these gists: PipedProcess, StdFileStream

So I've attempted to try runWorkerTask, I have the following code to run my classifyTask:

runWorkerTask( &classifyTask, bands.idup, "cluster", "isoclus.out.json", image_dir, image);  //source/app.d(361)

But this fails with

Error: static assert  "Argument type ulong[] is not safe to pass between threads."
source/app.d(361): instantiated from here: runWorkerTask!(void, ulong[], string, string, string, ImageDB)

Is there any way to pass an array to a function run in runWorkerTask?

Re: How to Handle Long Running Processes on Server Side

Permalink: HTTP NNTP

Sönke Ludwig

Posted Wed, 12 Mar 2014 22:50:03 +0100 in reply to Craig Dillabaugh

Am 12.03.2014 22:22, schrieb Craig Dillabaugh:

(...)
So I've attempted to try runWorkerTask, I have the following code to run my classifyTask:
runWorkerTask( &classifyTask, bands.idup, "cluster", "isoclus.out.json", image_dir, image);  //source/app.d(361)
But this fails with
Error: static assert  "Argument type ulong[] is not safe to pass between threads."
source/app.d(361): instantiated from here: runWorkerTask!(void, ulong[], string, string, string, ImageDB)
Is there any way to pass an array to a function run in runWorkerTask?

You can either use immutable(ulong)[] or shared(ulong)[] for this
purpose, where immutable here seems to be the most likely candidate
(either directly build the array as immutable, or use .idup or
assumeUnique).

If you ever have a situation where you are certain that it is safe to
pass some data to another thread, you can also use cast(immutable) or
cast(shared) to talk the compiler into accepting it, but of course
this is unsafe and not recommended practice.

Re: How to Handle Long Running Processes on Server Side

Permalink: HTTP NNTP

Etienne Cimon

Posted Thu, 13 Mar 2014 00:34:16 GMT in reply to Sönke Ludwig

On Wed, 12 Mar 2014 22:50:03 +0100, Sönke Ludwig wrote:

If you ever have a situation where you are certain that it is safe to
pass some data to another thread, you can also use cast(immutable) or
cast(shared) to talk the compiler into accepting it, but of course
this is unsafe and not recommended practice.

Do you think casting a struct as immutable would perform better than using makeIsolated on a class however? (I never managed to make a struct isolated)

Re: How to Handle Long Running Processes on Server Side

Permalink: HTTP NNTP

Craig Dillabaugh

Posted Thu, 13 Mar 2014 01:23:16 GMT in reply to Sönke Ludwig

On Wed, 12 Mar 2014 22:50:03 +0100, Sönke Ludwig wrote:

Am 12.03.2014 22:22, schrieb Craig Dillabaugh:
(...)
So I've attempted to try runWorkerTask, I have the following code to run my classifyTask:
runWorkerTask( &classifyTask, bands.idup, "cluster", "isoclus.out.json", image_dir, image);  //source/app.d(361)
But this fails with
Error: static assert  "Argument type ulong[] is not safe to pass between threads."
source/app.d(361): instantiated from here: runWorkerTask!(void, ulong[], string, string, string, ImageDB)
Is there any way to pass an array to a function run in runWorkerTask?
You can either use immutable(ulong)[] or shared(ulong)[] for this
purpose, where immutable here seems to be the most likely candidate
(either directly build the array as immutable, or use .idup or
assumeUnique).

If you ever have a situation where you are certain that it is safe to
pass some data to another thread, you can also use cast(immutable) or
cast(shared) to talk the compiler into accepting it, but of course
this is unsafe and not recommended practice.

Sorry, I shouldn't have even posted that, it dawned on me as I was writing the post that I needed to add immutable to the function signature, but I had a mental breakdown and went ahead and posted after I had fixed the code.

Thanks though for you always helpful and informative responses.

Re: How to Handle Long Running Processes on Server Side

Permalink: HTTP NNTP

Sönke Ludwig

Posted Thu, 13 Mar 2014 09:02:14 +0100 in reply to Etienne Cimon

Am 13.03.2014 01:34, schrieb Etienne Cimon:

On Wed, 12 Mar 2014 22:50:03 +0100, Sönke Ludwig wrote:

If you ever have a situation where you are certain that it is safe to
pass some data to another thread, you can also use cast(immutable) or
cast(shared) to talk the compiler into accepting it, but of course
this is unsafe and not recommended practice.

Do you think casting a struct as immutable would perform better than using makeIsolated on a class however? (I never managed to make a struct isolated)

I forgot about isolated, so yes that may be a better choice than
shared, if applicable. For pointers to struct, you can use
assumeIsolated for now, but I'll also adjust makeIsolated to work
for structs.

Re: How to Handle Long Running Processes on Server Side

Permalink: HTTP NNTP

Etienne Cimon

Posted Thu, 13 Mar 2014 15:00:20 GMT in reply to Sönke Ludwig

On Thu, 13 Mar 2014 09:02:14 +0100, Sönke Ludwig wrote:

Am 13.03.2014 01:34, schrieb Etienne Cimon:

On Wed, 12 Mar 2014 22:50:03 +0100, Sönke Ludwig wrote:

If you ever have a situation where you are certain that it is safe to
pass some data to another thread, you can also use cast(immutable) or
cast(shared) to talk the compiler into accepting it, but of course
this is unsafe and not recommended practice.

Do you think casting a struct as immutable would perform better than using makeIsolated on a class however? (I never managed to make a struct isolated)

I forgot about isolated, so yes that may be a better choice than
shared, if applicable. For pointers to struct, you can use
assumeIsolated for now, but I'll also adjust makeIsolated to work
for structs.

So I tried assumeIsolated on a struct pointer as a immutable(Struct)*, after a lot of trial and error I guess I can report on the general behavior.

First, it fails at private string isolatedAggregateMethodsString(T)() complaining that there's no members to loop from in foreach( mname; __traits(allMembers, T) ), I had to add static if( __traits(compiles, __traits(allMembers, T)) ) before the loop to get past that.

Then, it returns an IsolatedRef!(immutable(Struct)**) which doesn't pass the static assert(isWeaklyIsolated!(FieldTypeTuple!T)) in private struct IsolatedRef(T). So I removed that assert.

Finally, when I try to send it, it'll take another pointer from it and my types become conflictual with unsafeGet.

I'll just keep on moving isolated classes for now :-P

Re: How to Handle Long Running Processes on Server Side

Permalink: HTTP NNTP

Sönke Ludwig

Posted Thu, 13 Mar 2014 18:58:23 GMT in reply to Etienne Cimon

On Thu, 13 Mar 2014 15:00:20 GMT, Etienne Cimon wrote:

On Thu, 13 Mar 2014 09:02:14 +0100, Sönke Ludwig wrote:

Am 13.03.2014 01:34, schrieb Etienne Cimon:

On Wed, 12 Mar 2014 22:50:03 +0100, Sönke Ludwig wrote:

If you ever have a situation where you are certain that it is safe to
pass some data to another thread, you can also use cast(immutable) or
cast(shared) to talk the compiler into accepting it, but of course
this is unsafe and not recommended practice.

Do you think casting a struct as immutable would perform better than using makeIsolated on a class however? (I never managed to make a struct isolated)

I forgot about isolated, so yes that may be a better choice than
shared, if applicable. For pointers to struct, you can use
assumeIsolated for now, but I'll also adjust makeIsolated to work
for structs.

So I tried assumeIsolated on a struct pointer as a immutable(Struct)*, after a lot of trial and error I guess I can report on the general behavior.

First, it fails at private string isolatedAggregateMethodsString(T)() complaining that there's no members to loop from in foreach( mname; __traits(allMembers, T) ), I had to add static if( __traits(compiles, __traits(allMembers, T)) ) before the loop to get past that.

Then, it returns an IsolatedRef!(immutable(Struct)**) which doesn't pass the static assert(isWeaklyIsolated!(FieldTypeTuple!T)) in private struct IsolatedRef(T). So I removed that assert.

Finally, when I try to send it, it'll take another pointer from it and my types become conflictual with unsafeGet.

I'll just keep on moving isolated classes for now :-P

Should work now with 3d8bbb1. But always beware of the issue that destructors are never called for GC allocated structs.

Re: How to Handle Long Running Processes on Server Side

Permalink: HTTP NNTP

Etienne Cimon

Posted Thu, 13 Mar 2014 19:48:57 GMT in reply to Sönke Ludwig

On Thu, 13 Mar 2014 18:58:23 GMT, Sönke Ludwig wrote:

On Thu, 13 Mar 2014 15:00:20 GMT, Etienne Cimon wrote:

On Thu, 13 Mar 2014 09:02:14 +0100, Sönke Ludwig wrote:

Am 13.03.2014 01:34, schrieb Etienne Cimon:

On Wed, 12 Mar 2014 22:50:03 +0100, Sönke Ludwig wrote:

If you ever have a situation where you are certain that it is safe to
pass some data to another thread, you can also use cast(immutable) or
cast(shared) to talk the compiler into accepting it, but of course
this is unsafe and not recommended practice.

Do you think casting a struct as immutable would perform better than using makeIsolated on a class however? (I never managed to make a struct isolated)

I forgot about isolated, so yes that may be a better choice than
shared, if applicable. For pointers to struct, you can use
assumeIsolated for now, but I'll also adjust makeIsolated to work
for structs.

So I tried assumeIsolated on a struct pointer as a immutable(Struct)*, after a lot of trial and error I guess I can report on the general behavior.

First, it fails at private string isolatedAggregateMethodsString(T)() complaining that there's no members to loop from in foreach( mname; __traits(allMembers, T) ), I had to add static if( __traits(compiles, __traits(allMembers, T)) ) before the loop to get past that.

Then, it returns an IsolatedRef!(immutable(Struct)**) which doesn't pass the static assert(isWeaklyIsolated!(FieldTypeTuple!T)) in private struct IsolatedRef(T). So I removed that assert.

Finally, when I try to send it, it'll take another pointer from it and my types become conflictual with unsafeGet.

I'll just keep on moving isolated classes for now :-P

Should work now with 3d8bbb1. But always beware of the issue that destructors are never called for GC allocated structs.

For some reasons I was hoping for the structs to be able to stay on the stack =)

There's still an error when trying to send: Error: no property 'move' for type 'Message'

Here's what I'm doing:

auto isol = makeIsolated!Message(msg[0], msg[1], msg[2], cast(string)msg[3]);
ptask.task.send(isol.move());

it works fine if Message is a class though

Re: How to Handle Long Running Processes on Server Side

Permalink: HTTP NNTP

Sönke Ludwig

Posted Thu, 13 Mar 2014 21:05:22 +0100 in reply to Etienne Cimon

Am 13.03.2014 20:48, schrieb Etienne Cimon:

On Thu, 13 Mar 2014 18:58:23 GMT, Sönke Ludwig wrote:

On Thu, 13 Mar 2014 15:00:20 GMT, Etienne Cimon wrote:

On Thu, 13 Mar 2014 09:02:14 +0100, Sönke Ludwig wrote:

Am 13.03.2014 01:34, schrieb Etienne Cimon:

On Wed, 12 Mar 2014 22:50:03 +0100, Sönke Ludwig wrote:

If you ever have a situation where you are certain that it is safe to
pass some data to another thread, you can also use cast(immutable) or
cast(shared) to talk the compiler into accepting it, but of course
this is unsafe and not recommended practice.

Do you think casting a struct as immutable would perform better than using makeIsolated on a class however? (I never managed to make a struct isolated)

I forgot about isolated, so yes that may be a better choice than
shared, if applicable. For pointers to struct, you can use
assumeIsolated for now, but I'll also adjust makeIsolated to work
for structs.

So I tried assumeIsolated on a struct pointer as a immutable(Struct)*, after a lot of trial and error I guess I can report on the general behavior.

First, it fails at private string isolatedAggregateMethodsString(T)() complaining that there's no members to loop from in foreach( mname; __traits(allMembers, T) ), I had to add static if( __traits(compiles, __traits(allMembers, T)) ) before the loop to get past that.

Then, it returns an IsolatedRef!(immutable(Struct)**) which doesn't pass the static assert(isWeaklyIsolated!(FieldTypeTuple!T)) in private struct IsolatedRef(T). So I removed that assert.

Finally, when I try to send it, it'll take another pointer from it and my types become conflictual with unsafeGet.

I'll just keep on moving isolated classes for now :-P

Should work now with 3d8bbb1. But always beware of the issue that destructors are never called for GC allocated structs.

For some reasons I was hoping for the structs to be able to stay on the stack =)

They can! There is actually no need to make a struct explicitly
isolated, because it's passed by value anyway. Just all members of the
struct need to be either POD or immutable/shared/isolated. Isolated is
really just interesting for reference types.

There's still an error when trying to send: Error: no property 'move' for type 'Message'

Here's what I'm doing:
auto isol = makeIsolated!Message(msg[0], msg[1], msg[2], cast(string)msg[3]);
ptask.task.send(isol.move());
it works fine if Message is a class though

This is actually a side effect of the above. I'm not sure if the current
solution is optimal, though. Maybe it should throw an error instead.
What happens now is that makeIsolated!Message() verifies
isWeaklyIsolated!Message and then simply returns a value of type
Message. To create an isolated reference, makeIsolated!(Message*)()
is necessary.

Re: How to Handle Long Running Processes on Server Side

Permalink: HTTP NNTP

Etienne Cimon

Posted Thu, 13 Mar 2014 20:22:02 GMT in reply to Sönke Ludwig

On Thu, 13 Mar 2014 21:05:22 +0100, Sönke Ludwig wrote:

Am 13.03.2014 20:48, schrieb Etienne Cimon:

On Thu, 13 Mar 2014 18:58:23 GMT, Sönke Ludwig wrote:

On Thu, 13 Mar 2014 15:00:20 GMT, Etienne Cimon wrote:

On Thu, 13 Mar 2014 09:02:14 +0100, Sönke Ludwig wrote:

Am 13.03.2014 01:34, schrieb Etienne Cimon:

On Wed, 12 Mar 2014 22:50:03 +0100, Sönke Ludwig wrote:

If you ever have a situation where you are certain that it is safe to
pass some data to another thread, you can also use cast(immutable) or
cast(shared) to talk the compiler into accepting it, but of course
this is unsafe and not recommended practice.

Do you think casting a struct as immutable would perform better than using makeIsolated on a class however? (I never managed to make a struct isolated)

I forgot about isolated, so yes that may be a better choice than
shared, if applicable. For pointers to struct, you can use
assumeIsolated for now, but I'll also adjust makeIsolated to work
for structs.

So I tried assumeIsolated on a struct pointer as a immutable(Struct)*, after a lot of trial and error I guess I can report on the general behavior.

First, it fails at private string isolatedAggregateMethodsString(T)() complaining that there's no members to loop from in foreach( mname; __traits(allMembers, T) ), I had to add static if( __traits(compiles, __traits(allMembers, T)) ) before the loop to get past that.

Then, it returns an IsolatedRef!(immutable(Struct)**) which doesn't pass the static assert(isWeaklyIsolated!(FieldTypeTuple!T)) in private struct IsolatedRef(T). So I removed that assert.

Finally, when I try to send it, it'll take another pointer from it and my types become conflictual with unsafeGet.

I'll just keep on moving isolated classes for now :-P

Should work now with 3d8bbb1. But always beware of the issue that destructors are never called for GC allocated structs.

For some reasons I was hoping for the structs to be able to stay on the stack =)

They can! There is actually no need to make a struct explicitly
isolated, because it's passed by value anyway. Just all members of the
struct need to be either POD or immutable/shared/isolated. Isolated is
really just interesting for reference types.
There's still an error when trying to send: Error: no property 'move' for type 'Message'

Here's what I'm doing:
auto isol = makeIsolated!Message(msg[0], msg[1], msg[2], cast(string)msg[3]);
ptask.task.send(isol.move());
it works fine if Message is a class though
This is actually a side effect of the above. I'm not sure if the current
solution is optimal, though. Maybe it should throw an error instead.
What happens now is that makeIsolated!Message() verifies
isWeaklyIsolated!Message and then simply returns a value of type
Message. To create an isolated reference, makeIsolated!(Message*)()
is necessary.

I think I used the proper type and passing algorithm, but I still get this error:

E:\installdir\dmd2\windows\bin....\src\phobos\std\variant.d(608): Error: cannot modify struct p IsolatedValueProxyTuple!(Message) with immutable members
E:\installdir\dmd2\windows\bin\..\..\src\phobos\std\variant.d(294): Error: cannot modify struct zat IsolatedValueProxyTuple!(Message) with immutable members
E:\installdir\dmd2\windows\bin....\src\phobos\std\variant.d(612): Error: template instance std.variant.VariantN!(20u).VariantN.handler!(IsolatedValueProxyTuple!(Message)) error instantiating
E:\installdir\dmd2\windows\bin....\src\phobos\std\variant.d(562): instantiated from here: opAssign!(IsolatedValueProxyTuple!(Message))
source/vibe/concurrency.d(1088): instantiated from here: ctor!(IsolatedValueProxyTuple!(Message))
chd\localcache.d(108): instantiated from here: send!(Message)
chd\localcache.d(98): ... (1 instantiations, -v to show) ...
chd\localcache.d(66): instantiated from here: fetch!(Tuple!string)
chd\connection.d(427): instantiated from here: get!(A)
E:\installdir\dmd2\windows\bin\..\..\src\phobos\std\variant.d(562): Error: template instance std.variant.VariantN!(20u).VariantN.opAssign!(IsolatedValueProxyTuple!(Message)) error instantiating
source/vibe/concurrency.d(1088): instantiated from here: ctor!(IsolatedValueProxyTuple!(Message))
chd\localcache.d(108): instantiated from here: send!(Message)
chd\localcache.d(98): instantiated from here: send!(Tuple!(uint, string, const(Command), ubyte[]))
chd\localcache.d(66): instantiated from here: fetch!(Tuple!string)
chd\connection.d(427): instantiated from here: get!(A)
source/vibe/concurrency.d(1088): Error: template instance std.variant.VariantN!(20u).VariantN._ctor!(IsolatedValueProxyTuple!(Message)) error instantiating
chd\localcache.d(108): instantiated from here: send!(Message)
chd\localcache.d(98): instantiated from here: send!(Tuple!(uint, string, const(Command), ubyte[]))
chd\localcache.d(66): instantiated from here: fetch!(Tuple!string)
chd\connection.d(427): instantiated from here: get!(A)
chd\localcache.d(108): Error: template instance vibe.core.concurrency.send!(Message) error instantiating
chd\localcache.d(98): instantiated from here: send!(Tuple!(uint, string, const(Command), ubyte[]))
chd\localcache.d(66): instantiated from here: fetch!(Tuple!string)
chd\connection.d(427): instantiated from here: get!(A)
chd\localcache.d(98): Error: template instance chd.localcache.LocalCache.send!(Tuple!(uint, string, const(Command), ubyte[])) error instantiating
chd\localcache.d(66): instantiated from here: fetch!(Tuple!string)
chd\connection.d(427): instantiated from here: get!(A)
chd\localcache.d(66): Error: template instance chd.localcache.LocalCache.fetch!(Tuple!string) error instantiating
chd\connection.d(427): instantiated from here: get!(A)
chd\connection.d(427): Error: template instance chd.localcache.LocalCache.get!(A) error instantiating
source/vibe/concurrency.d(1111): Error: cannot modify struct retfield0 Message with immutable members

Pages: 1 2 3