RejectedSoftware Forums

Sign up

Pages: 1 2 3

Automatic GC collection runs?

Just a thought I had: Would it be reasonable for vibe.d to tell
the GC to run a collection cycle whenever vibe.d detects it's a
good time to do so? (For example, when there's no traffic for a
certain amount of time.) Or does vibe.d already do something like
this?

Re: Automatic GC collection runs?

Am 12.06.2012 09:01, schrieb Nick Sabalausky:

Just a thought I had: Would it be reasonable for vibe.d to tell the GC
to run a collection cycle whenever vibe.d detects it's a good time to do
so? (For example, when there's no traffic for a certain amount of time.)
Or does vibe.d already do something like this?

It doesn't do this right now but that would be an idea and should be
relatively easy to do using a timer once there is support for an
onIdle() callback, which I have lying around somwhere already (I'll
check this in when I get some time). Would be interesting to test how
efficient this is for reducing delays in practice.

Re: Automatic GC collection runs?

Am 12.06.2012 19:38, schrieb Sönke Ludwig:

Am 12.06.2012 09:01, schrieb Nick Sabalausky:

Just a thought I had: Would it be reasonable for vibe.d to tell the GC
to run a collection cycle whenever vibe.d detects it's a good time to do
so? (For example, when there's no traffic for a certain amount of time.)
Or does vibe.d already do something like this?

It doesn't do this right now but that would be an idea and should be
relatively easy to do using a timer once there is support for an
onIdle() callback, which I have lying around somwhere already (I'll
check this in when I get some time). Would be interesting to test how
efficient this is for reducing delays in practice.

I've just added this as an optional feature. With
DFLAGS="-version=VibeIdleCollect", it should now run a GC collection
whenever there was no activity for 2 seconds.

Re: Automatic GC collection runs?

On Friday, 13 July 2012 at 10:51:11 UTC, Sönke Ludwig wrote:

Am 12.06.2012 19:38, schrieb Sönke Ludwig:

Am 12.06.2012 09:01, schrieb Nick Sabalausky:

Just a thought I had: Would it be reasonable for vibe.d to
tell the GC
to run a collection cycle whenever vibe.d detects it's a good
time to do
so? (For example, when there's no traffic for a certain
amount of time.)
Or does vibe.d already do something like this?

It doesn't do this right now but that would be an idea and
should be
relatively easy to do using a timer once there is support for
an
onIdle() callback, which I have lying around somwhere already
(I'll
check this in when I get some time). Would be interesting to
test how
efficient this is for reducing delays in practice.

I've just added this as an optional feature. With
DFLAGS="-version=VibeIdleCollect", it should now run a GC
collection whenever there was no activity for 2 seconds.

I've done a some simple Pong benchmark(serving "Hello world") of
the Vibe server with 1 million connections and 100 simultaneous:

 ab -n 1000000 -c 100

And while the performance was very good, the memory usage grew to
numbers of few hundred megabytes(the highest was 600Mb) without
any attempt to recycle memory. NodeJS for example kept it around
30Mb under the same load. This case is probably not realistic for
low load scenarios, but if we want to sell the project for high
load we better have a good GC. But I understand that this is a
long standing problem in D...

Re: Automatic GC collection runs?

On Wednesday, 5 September 2012 at 14:22:50 UTC, Eldar
Insafutdinov wrote:

On Friday, 13 July 2012 at 10:51:11 UTC, Sönke Ludwig wrote:

Am 12.06.2012 19:38, schrieb Sönke Ludwig:

Am 12.06.2012 09:01, schrieb Nick Sabalausky:

Just a thought I had: Would it be reasonable for vibe.d to
tell the GC
to run a collection cycle whenever vibe.d detects it's a
good time to do
so? (For example, when there's no traffic for a certain
amount of time.)
Or does vibe.d already do something like this?

It doesn't do this right now but that would be an idea and
should be
relatively easy to do using a timer once there is support for
an
onIdle() callback, which I have lying around somwhere already
(I'll
check this in when I get some time). Would be interesting to
test how
efficient this is for reducing delays in practice.

I've just added this as an optional feature. With
DFLAGS="-version=VibeIdleCollect", it should now run a GC
collection whenever there was no activity for 2 seconds.

I've done a some simple Pong benchmark(serving "Hello world")
of the Vibe server with 1 million connections and 100
simultaneous:

ab -n 1000000 -c 100

And while the performance was very good, the memory usage grew
to numbers of few hundred megabytes(the highest was 600Mb)
without any attempt to recycle memory. NodeJS for example kept
it around 30Mb under the same load. This case is probably not
realistic for low load scenarios, but if we want to sell the
project for high load we better have a good GC. But I
understand that this is a long standing problem in D...

Ok, I've done some more tests. This time I was stress testing the
server built with DFLAGS="-version=VibeIdleCollect" . There was
one invocation of GC.collect() before the benchmark started(the
same one as in my previous post). During the stress test
GC.collect() wasn't called from that handler which is expected as
the server wasn't idling. The memory usage grew to 380Mb. However
when the benchmark finished there was GC collection invoked, but
the memory usage stayed at the same level. This suggests there is
a leak somewhere. The server was pretty simple:
http://pastie.org/4670356

Re: Automatic GC collection runs?

On Wed, 05 Sep 2012 23:22:03 +0200
"Eldar Insafutdinov" e.insafutdinov@gmail.com wrote:

Ok, I've done some more tests. This time I was stress testing the
server built with DFLAGS="-version=VibeIdleCollect" . There was
one invocation of GC.collect() before the benchmark started(the
same one as in my previous post). During the stress test
GC.collect() wasn't called from that handler which is expected as
the server wasn't idling. The memory usage grew to 380Mb. However
when the benchmark finished there was GC collection invoked, but
the memory usage stayed at the same level. This suggests there is
a leak somewhere.

Or D's curse of false pointers. Did you compile it as 32-bit or 64-bit?

Re: Automatic GC collection runs?

On Wednesday, 5 September 2012 at 21:34:21 UTC, Nick Sabalausky
wrote:

On Wed, 05 Sep 2012 23:22:03 +0200
"Eldar Insafutdinov" e.insafutdinov@gmail.com wrote:

Ok, I've done some more tests. This time I was stress testing
the server built with DFLAGS="-version=VibeIdleCollect" .
There was one invocation of GC.collect() before the benchmark
started(the same one as in my previous post). During the
stress test GC.collect() wasn't called from that handler which
is expected as the server wasn't idling. The memory usage grew
to 380Mb. However when the benchmark finished there was GC
collection invoked, but the memory usage stayed at the same
level. This suggests there is a leak somewhere.

Or D's curse of false pointers. Did you compile it as 32-bit or
64-bit?

32bit. Is it worth trying 64 bit?

Re: Automatic GC collection runs?

On Thu, 06 Sep 2012 00:10:36 +0200
"Eldar Insafutdinov" e.insafutdinov@gmail.com wrote:

On Wednesday, 5 September 2012 at 21:34:21 UTC, Nick Sabalausky
wrote:

On Wed, 05 Sep 2012 23:22:03 +0200
"Eldar Insafutdinov" e.insafutdinov@gmail.com wrote:

Ok, I've done some more tests. This time I was stress testing
the server built with DFLAGS="-version=VibeIdleCollect" .
There was one invocation of GC.collect() before the benchmark
started(the same one as in my previous post). During the
stress test GC.collect() wasn't called from that handler which
is expected as the server wasn't idling. The memory usage grew
to 380Mb. However when the benchmark finished there was GC
collection invoked, but the memory usage stayed at the same
level. This suggests there is a leak somewhere.

Or D's curse of false pointers. Did you compile it as 32-bit or
64-bit?

32bit. Is it worth trying 64 bit?

Yes, particularly to help narrow down the cause:

If the problem persists on 64-bit then it sounds like a possible
memory leak as you suggested. But if the problem is false pointers,
then it has a high likelihood of going away on 64-bit (since
non-pointer values will be much less likely to match 64-bit memory
addresses).

Re: Automatic GC collection runs?

On Wednesday, 5 September 2012 at 21:22:03 UTC, Eldar
Insafutdinov wrote:

On Wednesday, 5 September 2012 at 14:22:50 UTC, Eldar
Insafutdinov wrote:

On Friday, 13 July 2012 at 10:51:11 UTC, Sönke Ludwig wrote:

Am 12.06.2012 19:38, schrieb Sönke Ludwig:

Am 12.06.2012 09:01, schrieb Nick Sabalausky:

Just a thought I had: Would it be reasonable for vibe.d to
tell the GC
to run a collection cycle whenever vibe.d detects it's a
good time to do
so? (For example, when there's no traffic for a certain
amount of time.)
Or does vibe.d already do something like this?

It doesn't do this right now but that would be an idea and
should be
relatively easy to do using a timer once there is support
for an
onIdle() callback, which I have lying around somwhere
already (I'll
check this in when I get some time). Would be interesting to
test how
efficient this is for reducing delays in practice.

I've just added this as an optional feature. With
DFLAGS="-version=VibeIdleCollect", it should now run a GC
collection whenever there was no activity for 2 seconds.

I've done a some simple Pong benchmark(serving "Hello world")
of the Vibe server with 1 million connections and 100
simultaneous:

ab -n 1000000 -c 100

And while the performance was very good, the memory usage grew
to numbers of few hundred megabytes(the highest was 600Mb)
without any attempt to recycle memory. NodeJS for example kept
it around 30Mb under the same load. This case is probably not
realistic for low load scenarios, but if we want to sell the
project for high load we better have a good GC. But I
understand that this is a long standing problem in D...

Ok, I've done some more tests. This time I was stress testing
the server built with DFLAGS="-version=VibeIdleCollect" . There
was one invocation of GC.collect() before the benchmark
started(the same one as in my previous post). During the stress
test GC.collect() wasn't called from that handler which is
expected as the server wasn't idling. The memory usage grew to
380Mb. However when the benchmark finished there was GC
collection invoked, but the memory usage stayed at the same
level. This suggests there is a leak somewhere. The server was
pretty simple: http://pastie.org/4670356

I don't think the gc will give memory back to there os unless the
os requires it or you explicitly tell it to do so.

Re: Automatic GC collection runs?

Ok, I've done some more tests. This time I was stress testing the server
built with DFLAGS="-version=VibeIdleCollect" . There was one invocation
of GC.collect() before the benchmark started(the same one as in my
previous post). During the stress test GC.collect() wasn't called from
that handler which is expected as the server wasn't idling. The memory
usage grew to 380Mb. However when the benchmark finished there was GC
collection invoked, but the memory usage stayed at the same level. This
suggests there is a leak somewhere. The server was pretty simple:
http://pastie.org/4670356

Does the GC automatically release unused memory to the system by now? I
remember the times when this wasn't implemented at all.

In earlier tests memory usage stayed very low for me (in the 20-30 MB
range), but now I can confirm that one of my server processes just was
at over 1 GB. I'll investigate this, maybe there is actually a leak in
the manual memory management parts.

Pages: 1 2 3