RejectedSoftware Forums

Sign up

Pages: 1 2

Correct setup of redis?

avatar
simendsjo
Posted Fri, 20 Sep 2013 19:31:21 GMT
Reply

What is the correct way of using redis?
I tried having a single __gshared instance, but that didn't turn out too well: 

Exception in lev_lock_mutex: Assertion failure
Exception in lev_unlock_mutex: Unable to unlock mutex
Got exception when resuming task onSocketRead: Resuming task in foreign thread.

Using one RedisClient per thread didn't work either: 

Error after page has been written: core.exception.UnicodeException@src/rt/util/utf.d(290): invalid UTF-8 sequence
----------------
./simendsjo-homepage(dchar rt.util.utf.decode(const(char[]), ref ulong)+0x1be) [0x6cd562]
./simendsjo-homepage(_aApplycd1+0x55) [0x6c55a1]
./simendsjo-homepage(@safe void vibe.textfilter.html.filterHTMLEscape!(std.array.Appender!(immutable(char)[]).Appender).filterHTMLEscape(ref std.array.Appender!(immutable(char)[]).Appender, immutable(char)[])+0x3c) [0x5eefb4]
./simendsjo-homepage(immutable(char)[] vibe.textfilter.html.htmlEscape(immutable(char)[])+0x50) [0x5ed3a0]
./simendsjo-homepage(void vibe.templ.utils.__T11reqInjectorS143app9pgGetPostZ.reqInjector(vibe.http.server.HTTPServerRequest, vibe.http.server.HTTPServerResponse).void pgGetPost!(_D4vibe5templ5utils34__T11reqInjectorS143app9pgGetPostZ11reqInjectorFC4vibe4http6server17HTTPServerRequestC4vibe4http6server18HTTPServerResponseZv3reqC4vibe4http6server17HTTPServerRequest).pgGetPost(vibe.http.server.HTTPServerRequest, vibe.http.server.HTTPServerResponse).void compileDietFile!("post.list.dt", _D4vibe5templ5utils34__T11reqInjectorS143app9pgGetPostZ11reqInjectorFC4vibe4http6server17HTTPServerRequestC4vibe4http6server18HTTPServerResponseZv205__T9pgGetPostS187_D4vibe5templ5utils34__T11reqInjectorS143app9pgGetPostZ11reqInjectorFC4vibe4http6server17HTTPServerRequestC4vibe4http6server18HTTPServerResponseZv3reqC4vibe4http6server17HTTPServerRequestZ9pgGetPostMFC4vibe4http6server17HTTPServerRequestC4vibe4http6server18HTTPServerResponseZv5postsAC3app4Post).compileDietFile(vibe.core.stream.OutputStream)+0x120) [0x62b738]
./simendsjo-homepage(void vibe.templ.utils.__T11reqInjectorS143app9pgGetPostZ.reqInjector(vibe.http.server.HTTPServerRequest, vibe.http.server.HTTPServerResponse).void pgGetPost!(_D4vibe5templ5utils34__T11reqInjectorS143app9pgGetPostZ11reqInjectorFC4vibe4http6server17HTTPServerRequestC4vibe4http6server18HTTPServerResponseZv3reqC4vibe4http6server17HTTPServerRequest).pgGetPost(vibe.http.server.HTTPServerRequest, vibe.http.server.HTTPServerResponse).@property void render!("post.list.dt", _D4vibe5templ5utils34__T11reqInjectorS143app9pgGetPostZ11reqInjectorFC4vibe4http6server17HTTPServerRequestC4vibe4http6server18HTTPServerResponseZv205__T9pgGetPostS187_D4vibe5templ5utils34__T11reqInjectorS143app9pgGetPostZ11reqInjectorFC4vibe4http6server17HTTPServerRequestC4vibe4http6server18HTTPServerResponseZv3reqC4vibe4http6server17HTTPServerRequestZ9pgGetPostMFC4vibe4http6server17HTTPServerRequestC4vibe4http6server18HTTPServerResponseZv5postsAC3app4Post).render(vibe.http.server.HTTPServerResponse)+0x1f5) [0x58deed]
./simendsjo-homepage(void vibe.templ.utils.__T11reqInjectorS143app9pgGetPostZ.reqInjector(vibe.http.server.HTTPServerRequest, vibe.http.server.HTTPServerResponse).void pgGetPost!(_D4vibe5templ5utils34__T11reqInjectorS143app9pgGetPostZ11reqInjectorFC4vibe4http6server17HTTPServerRequestC4vibe4http6server18HTTPServerResponseZv3reqC4vibe4http6server17HTTPServerRequest).pgGetPost(vibe.http.server.HTTPServerRequest, vibe.http.server.HTTPServerResponse)+0x5b) [0x58dceb]
./simendsjo-homepage(void vibe.templ.utils.__T11reqInjectorS143app9pgGetPostZ.reqInjector(vibe.http.server.HTTPServerRequest, vibe.http.server.HTTPServerResponse)+0x16) [0x58dc86]
./simendsjo-homepage(void std.functional.DelegateFaker!(void function(vibe.http.server.HTTPServerRequest, vibe.http.server.HTTPServerResponse)*).DelegateFaker.doIt(vibe.http.server.HTTPServerRequest, vibe.http.server.HTTPServerResponse)+0x1b) [0x68a35b]
./simendsjo-homepage(void vibe.http.router.URLRouter.handleRequest(vibe.http.server.HTTPServerRequest, vibe.http.server.HTTPServerResponse)+0xc7) [0x62e06f]
./simendsjo-homepage(bool vibe.http.server.handleRequest(vibe.core.stream.Stream, immutable(char)[], vibe.http.server.HTTPServerListener, ref vibe.http.server.HTTPServerSettings, ref bool)+0x1b9a) [0x62938a]
./simendsjo-homepage(void vibe.http.server.handleHTTPConnection(vibe.core.net.TCPConnection, vibe.http.server.HTTPServerListener)+0x156) [0x627696]
./simendsjo-homepage(void vibe.http.server.listenHTTPPlain(vibe.http.server.HTTPServerSettings, void delegate(vibe.http.server.HTTPServerRequest, vibe.http.server.HTTPServerResponse)).void doListen(vibe.http.server.HTTPServerSettings, vibe.http.server.HTTPServerListener, immutable(char)[]).void __lambda93(vibe.core.net.TCPConnection)+0x28) [0x6234b0]
./simendsjo-homepage(extern (C) nothrow void vibe.core.drivers.libevent2_tcp.onConnect(int, short, void*).void ClientTask.execute()+0x18b) [0x5bc993]
./simendsjo-homepage(void vibe.core.core.CoreTask.run()+0x115) [0x5c5a55]
./simendsjo-homepage(void core.thread.Fiber.run()+0x2a) [0x6f4aa2]
./simendsjo-homepage(fiber_entryPoint+0x61) [0x6f49ad]
[(nil)]
Error after page has been written: core.exception.UnicodeException@src/rt/util/utf.d(290): invalid UTF-8 sequence
----------------
./simendsjo-homepage(dchar rt.util.utf.decode(const(char[]), ref ulong)+0x1be) [0x6cd562]
./simendsjo-homepage(_aApplycd1+0x55) [0x6c55a1]
./simendsjo-homepage(@safe void vibe.textfilter.html.filterHTMLEscape!(std.array.Appender!(immutable(char)[]).Appender).filterHTMLEscape(ref std.array.Appender!(immutable(char)[]).Appender, immutable(char)[])+0x3c) [0x5eefb4]
./simendsjo-homepage(immutable(char)[] vibe.textfilter.html.htmlEscape(immutable(char)[])+0x50) [0x5ed3a0]
./simendsjo-homepage(void vibe.templ.utils.__T11reqInjectorS143app9pgGetPostZ.reqInjector(vibe.http.server.HTTPServerRequest, vibe.http.server.HTTPServerResponse).void pgGetPost!(_D4vibe5templ5utils34__T11reqInjectorS143app9pgGetPostZ11reqInjectorFC4vibe4http6server17HTTPServerRequestC4vibe4http6server18HTTPServerResponseZv3reqC4vibe4http6server17HTTPServerRequest).pgGetPost(vibe.http.server.HTTPServerRequest, vibe.http.server.HTTPServerResponse).void compileDietFile!("post.list.dt", _D4vibe5templ5utils34__T11reqInjectorS143app9pgGetPostZ11reqInjectorFC4vibe4http6server17HTTPServerRequestC4vibe4http6server18HTTPServerResponseZv205__T9pgGetPostS187_D4vibe5templ5utils34__T11reqInjectorS143app9pgGetPostZ11reqInjectorFC4vibe4http6server17HTTPServerRequestC4vibe4http6server18HTTPServerResponseZv3reqC4vibe4http6server17HTTPServerRequestZ9pgGetPostMFC4vibe4http6server17HTTPServerRequestC4vibe4http6server18HTTPServerResponseZv5postsAC3app4Post).compileDietFile(vibe.core.stream.OutputStream)+0x120) [0x62b738]
./simendsjo-homepage(void vibe.templ.utils.__T11reqInjectorS143app9pgGetPostZ.reqInjector(vibe.http.server.HTTPServerRequest, vibe.http.server.HTTPServerResponse).void pgGetPost!(_D4vibe5templ5utils34__T11reqInjectorS143app9pgGetPostZ11reqInjectorFC4vibe4http6server17HTTPServerRequestC4vibe4http6server18HTTPServerResponseZv3reqC4vibe4http6server17HTTPServerRequest).pgGetPost(vibe.http.server.HTTPServerRequest, vibe.http.server.HTTPServerResponse).@property void render!("post.list.dt", _D4vibe5templ5utils34__T11reqInjectorS143app9pgGetPostZ11reqInjectorFC4vibe4http6server17HTTPServerRequestC4vibe4http6server18HTTPServerResponseZv205__T9pgGetPostS187_D4vibe5templ5utils34__T11reqInjectorS143app9pgGetPostZ11reqInjectorFC4vibe4http6server17HTTPServerRequestC4vibe4http6server18HTTPServerResponseZv3reqC4vibe4http6server17HTTPServerRequestZ9pgGetPostMFC4vibe4http6server17HTTPServerRequestC4vibe4http6server18HTTPServerResponseZv5postsAC3app4Post).render(vibe.http.server.HTTPServerResponse)+0x1f5) [0x58deed]
./simendsjo-homepage(void vibe.templ.utils.__T11reqInjectorS143app9pgGetPostZ.reqInjector(vibe.http.server.HTTPServerRequest, vibe.http.server.HTTPServerResponse).void pgGetPost!(_D4vibe5templ5utils34__T11reqInjectorS143app9pgGetPostZ11reqInjectorFC4vibe4http6server17HTTPServerRequestC4vibe4http6server18HTTPServerResponseZv3reqC4vibe4http6server17HTTPServerRequest).pgGetPost(vibe.http.server.HTTPServerRequest, vibe.http.server.HTTPServerResponse)+0x5b) [0x58dceb]
./simendsjo-homepage(void vibe.templ.utils.__T11reqInjectorS143app9pgGetPostZ.reqInjector(vibe.http.server.HTTPServerRequest, vibe.http.server.HTTPServerResponse)+0x16) [0x58dc86]
./simendsjo-homepage(void std.functional.DelegateFaker!(void function(vibe.http.server.HTTPServerRequest, vibe.http.server.HTTPServerResponse)*).DelegateFaker.doIt(vibe.http.server.HTTPServerRequest, vibe.http.server.HTTPServerResponse)+0x1b) [0x68a35b]
./simendsjo-homepage(void vibe.http.router.URLRouter.handleRequest(vibe.http.server.HTTPServerRequest, vibe.http.server.HTTPServerResponse)+0xc7) [0x62e06f]
./simendsjo-homepage(bool vibe.http.server.handleRequest(vibe.core.stream.Stream, immutable(char)[], vibe.http.server.HTTPServerListener, ref vibe.http.server.HTTPServerSettings, ref bool)+0x1b9a) [0x62938a]
./simendsjo-homepage(void vibe.http.server.handleHTTPConnection(vibe.core.net.TCPConnection, vibe.http.server.HTTPServerListener)+0x156) [0x627696]
./simendsjo-homepage(void vibe.http.server.listenHTTPPlain(vibe.http.server.HTTPServerSettings, void delegate(vibe.http.server.HTTPServerRequest, vibe.http.server.HTTPServerResponse)).void doListen(vibe.http.server.HTTPServerSettings, vibe.http.server.HTTPServerListener, immutable(char)[]).void __lambda93(vibe.core.net.TCPConnection)+0x28) [0x6234b0]
./simendsjo-homepage(extern (C) nothrow void vibe.core.drivers.libevent2_tcp.onConnect(int, short, void*).void ClientTask.execute()+0x18b) [0x5bc993]
./simendsjo-homepage(void vibe.core.core.CoreTask.run()+0x115) [0x5c5a55]
./simendsjo-homepage(void core.thread.Fiber.run()+0x2a) [0x6f4aa2]
./simendsjo-homepage(fiber_entryPoint+0x61) [0x6f49ad]
[```

Re: Correct setup of redis?

avatar
simendsjo
Posted Fri, 20 Sep 2013 19:36:19 GMT in reply to simendsjo
Reply

On Fri, 20 Sep 2013 19:31:21 GMT, simendsjo wrote:

What is the correct way of using redis?

Oh.. And fetching a value from redis is actually slower than mariadb - why is that?
From mariadb takes ~40ms, while redis takes 40-100. All I do is get/set.

Re: Correct setup of redis?

avatar
simendsjo
Posted Fri, 20 Sep 2013 20:25:25 GMT in reply to simendsjo
Reply

On Fri, 20 Sep 2013 19:36:19 GMT, simendsjo wrote:

On Fri, 20 Sep 2013 19:31:21 GMT, simendsjo wrote:

What is the correct way of using redis?

Oh.. And fetching a value from redis is actually slower than mariadb - why is that?
From mariadb takes ~40ms, while redis takes 40-100. All I do is get/set.

Building with -profile doesn't work, so I'm unable to find out why it's slow.
Could it be the calls to format() is RedisConnection.request?

Re: Correct setup of redis?

avatar
Sönke Ludwig
Posted Sat, 21 Sep 2013 12:07:58 +0200 in reply to simendsjo
Reply

Am 20.09.2013 22:25, schrieb simendsjo:

On Fri, 20 Sep 2013 19:36:19 GMT, simendsjo wrote:

On Fri, 20 Sep 2013 19:31:21 GMT, simendsjo wrote:

What is the correct way of using redis?

Oh.. And fetching a value from redis is actually slower than mariadb - why is that?
From mariadb takes ~40ms, while redis takes 40-100. All I do is get/set.

Building with -profile doesn't work, so I'm unable to find out why it's slow.
Could it be the calls to format() is RedisConnection.request?

Without looking at the source at all a wild guess would be that there
is a flush() missing after sending each command and even with that
enabling tcpNoDelay may be necessary.

Re: Correct setup of redis?

avatar
simendsjo
Posted Sat, 21 Sep 2013 11:45:28 GMT in reply to Sönke Ludwig
Reply

On Sat, 21 Sep 2013 12:07:58 +0200, Sönke Ludwig wrote:

Am 20.09.2013 22:25, schrieb simendsjo:

On Fri, 20 Sep 2013 19:36:19 GMT, simendsjo wrote:

On Fri, 20 Sep 2013 19:31:21 GMT, simendsjo wrote:

What is the correct way of using redis?

Oh.. And fetching a value from redis is actually slower than mariadb - why is that?
From mariadb takes ~40ms, while redis takes 40-100. All I do is get/set.

Building with -profile doesn't work, so I'm unable to find out why it's slow.
Could it be the calls to format() is RedisConnection.request?

Without looking at the source at all a wild guess would be that there
is a flush() missing after sending each command and even with that
enabling tcpNoDelay may be necessary.

At least there's nothing wrong with my redis install: 

====== SET ======
  10000 requests completed in 0.06 seconds
  50 parallel clients
  3 bytes payload
  keep alive: 1

100.00% <= 0 milliseconds
161290.33 requests per second

====== GET ======
  10000 requests completed in 0.06 seconds
  50 parallel clients
  3 bytes payload
  keep alive: 1

100.00% <= 0 milliseconds
161290.33 requests per second

Re: Correct setup of redis?

avatar
simendsjo
Posted Sat, 21 Sep 2013 11:52:12 GMT in reply to Sönke Ludwig
Reply

On Sat, 21 Sep 2013 12:07:58 +0200, Sönke Ludwig wrote:

Am 20.09.2013 22:25, schrieb simendsjo:

On Fri, 20 Sep 2013 19:36:19 GMT, simendsjo wrote:

On Fri, 20 Sep 2013 19:31:21 GMT, simendsjo wrote:

What is the correct way of using redis?

Oh.. And fetching a value from redis is actually slower than mariadb - why is that?
From mariadb takes ~40ms, while redis takes 40-100. All I do is get/set.

Building with -profile doesn't work, so I'm unable to find out why it's slow.
Could it be the calls to format() is RedisConnection.request?

Without looking at the source at all a wild guess would be that there
is a flush() missing after sending each command and even with that
enabling tcpNoDelay may be necessary.

Just tried both, but there is no difference.

Re: Correct setup of redis?

avatar
simendsjo
Posted Sat, 21 Sep 2013 12:10:42 GMT in reply to simendsjo
Reply

On Sat, 21 Sep 2013 11:52:12 GMT, simendsjo wrote:

On Sat, 21 Sep 2013 12:07:58 +0200, Sönke Ludwig wrote:

Am 20.09.2013 22:25, schrieb simendsjo:

On Fri, 20 Sep 2013 19:36:19 GMT, simendsjo wrote:

On Fri, 20 Sep 2013 19:31:21 GMT, simendsjo wrote:

What is the correct way of using redis?

Oh.. And fetching a value from redis is actually slower than mariadb - why is that?
From mariadb takes ~40ms, while redis takes 40-100. All I do is get/set.

Building with -profile doesn't work, so I'm unable to find out why it's slow.
Could it be the calls to format() is RedisConnection.request?

Without looking at the source at all a wild guess would be that there
is a flush() missing after sending each command and even with that
enabling tcpNoDelay may be necessary.

Just tried both, but there is no difference.

I found the slow line: 

auto ln = cast(string)m_conn.readLine();

in RedisReply.this.

Seems this is calling ufcs readLine, which in turn calls readUntil(InputStream..

Re: Correct setup of redis?

avatar
simendsjo
Posted Sat, 21 Sep 2013 12:51:53 GMT in reply to simendsjo
Reply

On Sat, 21 Sep 2013 12:10:42 GMT, simendsjo wrote:

On Sat, 21 Sep 2013 11:52:12 GMT, simendsjo wrote:

On Sat, 21 Sep 2013 12:07:58 +0200, Sönke Ludwig wrote:

Am 20.09.2013 22:25, schrieb simendsjo:

On Fri, 20 Sep 2013 19:36:19 GMT, simendsjo wrote:

On Fri, 20 Sep 2013 19:31:21 GMT, simendsjo wrote:

What is the correct way of using redis?

Oh.. And fetching a value from redis is actually slower than mariadb - why is that?
From mariadb takes ~40ms, while redis takes 40-100. All I do is get/set.

Building with -profile doesn't work, so I'm unable to find out why it's slow.
Could it be the calls to format() is RedisConnection.request?

Without looking at the source at all a wild guess would be that there
is a flush() missing after sending each command and even with that
enabling tcpNoDelay may be necessary.

Just tried both, but there is no difference.

I found the slow line: 

auto ln = cast(string)m_conn.readLine();

in RedisReply.this.

Seems this is calling ufcs readLine, which in turn calls readUntil(InputStream..

Some more details. It's all in readUntil `while(!stream.empty)`.
steam.empty is whats causing all the problems.

Re: Correct setup of redis?

avatar
simendsjo
Posted Sat, 21 Sep 2013 13:38:34 GMT in reply to simendsjo
Reply

On Sat, 21 Sep 2013 12:51:53 GMT, simendsjo wrote:

On Sat, 21 Sep 2013 12:10:42 GMT, simendsjo wrote:

On Sat, 21 Sep 2013 11:52:12 GMT, simendsjo wrote:

On Sat, 21 Sep 2013 12:07:58 +0200, Sönke Ludwig wrote:

Am 20.09.2013 22:25, schrieb simendsjo:

On Fri, 20 Sep 2013 19:36:19 GMT, simendsjo wrote:

On Fri, 20 Sep 2013 19:31:21 GMT, simendsjo wrote:

What is the correct way of using redis?

Oh.. And fetching a value from redis is actually slower than mariadb - why is that?
From mariadb takes ~40ms, while redis takes 40-100. All I do is get/set.

Building with -profile doesn't work, so I'm unable to find out why it's slow.
Could it be the calls to format() is RedisConnection.request?

Without looking at the source at all a wild guess would be that there
is a flush() missing after sending each command and even with that
enabling tcpNoDelay may be necessary.

Just tried both, but there is no difference.

I found the slow line: 

auto ln = cast(string)m_conn.readLine();

in RedisReply.this.

Seems this is calling ufcs readLine, which in turn calls readUntil(InputStream..

Some more details. It's all in readUntil `while(!stream.empty)`.
steam.empty is whats causing all the problems.

Ok, some further investigations reveal libevent2tcp.d -> leastSize -> mctx.core.yieldForEvent()

Re: Correct setup of redis?

avatar
simendsjo
Posted Sat, 21 Sep 2013 13:47:05 GMT in reply to simendsjo
Reply

On Sat, 21 Sep 2013 13:38:34 GMT, simendsjo wrote:

On Sat, 21 Sep 2013 12:51:53 GMT, simendsjo wrote:

On Sat, 21 Sep 2013 12:10:42 GMT, simendsjo wrote:

On Sat, 21 Sep 2013 11:52:12 GMT, simendsjo wrote:

On Sat, 21 Sep 2013 12:07:58 +0200, Sönke Ludwig wrote:

Am 20.09.2013 22:25, schrieb simendsjo:

On Fri, 20 Sep 2013 19:36:19 GMT, simendsjo wrote:

On Fri, 20 Sep 2013 19:31:21 GMT, simendsjo wrote:

What is the correct way of using redis?

Oh.. And fetching a value from redis is actually slower than mariadb - why is that?
From mariadb takes ~40ms, while redis takes 40-100. All I do is get/set.

Building with -profile doesn't work, so I'm unable to find out why it's slow.
Could it be the calls to format() is RedisConnection.request?

Without looking at the source at all a wild guess would be that there
is a flush() missing after sending each command and even with that
enabling tcpNoDelay may be necessary.

Just tried both, but there is no difference.

I found the slow line: 

auto ln = cast(string)m_conn.readLine();

in RedisReply.this.

Seems this is calling ufcs readLine, which in turn calls readUntil(InputStream..

Some more details. It's all in readUntil `while(!stream.empty)`.
steam.empty is whats causing all the problems.

Ok, some further investigations reveal libevent2tcp.d -> leastSize -> mctx.core.yieldForEvent()

Well.. Seems all the work is happening in Fiber.yield() - this is druntime specific, right?

Pages: 1 2