Pages: 1 2 3

Problem with vibe.d + ddb

Permalink: HTTP NNTP

Posted Fri, 02 May 2014 00:14:28 GMT

Hello,

I'm having a problem that I did not use to have, using vibe.d + ddb (postgres driver with vibe.d support). I'm not sure with which version of vibe.d or ddb it started with, so I provide details for several vibe.d releases. I did not test with several ddb releases because there is only one release tag, which I actually did help create :-) (it used to use a ~master dependency).

The bug can be exercised with the following program. It should be compiled and run as part of vibe.d, but it does not even require that one does a vibe.d's listenHTTP for the problem to appear (I think):

import ddb.postgres;

PGConnection pgdb;

shared static this()
{
	auto pdb = new PostgresDB([
		"host":     "localhost",
		"database": "foo",
		"user":     "bar",
		"password": "baz",
	]);
    
	pgdb = pdb.lockConnection();
}

shared static ~this()
{
    pgdb.close();
}

The program works correctly except when closing (with ctrl-C), after which it complains and often crashes. Here is the output for several vibe.d versions:

0.7.19

OS X:
Received signal 2. Shutting down.
Exception in lev_lock_mutex: Unknown lock handle
Exception in lev_unlock_mutex: Unknown lock handle
Exception in lev_lock_mutex: Unknown lock handle
Exception in lev_unlock_mutex: Unknown lock handle
Exception in lev_lock_mutex: Unknown lock handle
Exception in lev_unlock_mutex: Unknown lock handle
Exception in lev_lock_mutex: Unknown lock handle
Exception in lev_unlock_mutex: Unknown lock handle
Exception in lev_lock_mutex: Unknown lock handle

Linux:
Received signal 2. Shutting down.
Exception in lev_lock_mutex: Unknown lock handle
Exception in lev_unlock_mutex: Unknown lock handle
Exception in lev_lock_mutex: Unknown lock handle
[warn] Epoll ADD(4) on fd 8 failed.  Old events were 0; read change was 0 (none); write change was 1 (add): Invalid argument
Exception in lev_unlock_mutex: Unknown lock handle
Exception in lev_lock_mutex: Unknown lock handle
Exception in lev_unlock_mutex: Unknown lock handle
Exception in lev_lock_mutex: Unknown lock handle
[warn] Epoll ADD(4) on fd 8 failed.  Old events were 0; read change was 0 (none); write change was 1 (add): Invalid argument
Exception in lev_unlock_mutex: Unknown lock handle
Exception in lev_lock_mutex: Unknown lock handle
Exception in lev_unlock_mutex: Unknown lock handle
Exception in lev_lock_mutex: Unknown lock handle
Exception in lev_unlock_mutex: Unknown lock handle
core.exception.AssertError@../.dub/packages/vibe-d-0.7.19/source/vibe/core/driver.d(27): No event driver loaded. Did the vibe.core.core module constructor run?
----------------
./appname(_d_assert_msg+0x45) [0x739d7d]
./appname(vibe.core.driver.EventDriver vibe.core.driver.getEventDriver(bool)+0x5f) [0x630ae3]
./appname(void vibe.core.core.VibeDriverCore.yieldForEvent()+0x1c1) [0x62c99d]
./appname(void vibe.core.drivers.libevent2_tcp.Libevent2TCPConnection.close()+0x14b) [0x63f86f]
./appname(void ddb.postgres.PGConnection.close()+0x7d) [0x6124c9]
./appname(void app._sharedStaticDtor2()+0x1f) [0x5f5b67]
./appname(void app.__modshareddtor()+0x9) [0x60f0b1]
./appname(pure void rt.minfo.__T17runModuleFuncsRevS452rt5minfo11ModuleGroup8runDtorsMFZv9__lambda1Z.runModuleFuncsRev(object.ModuleInfo*[])+0x61) [0x77869d]
./appname(void rt.minfo.ModuleGroup.runDtors()+0x1d) [0x778319]
./appname(int rt.minfo.rt_moduleDtor().__foreachbody1(ref rt.sections_linux.DSO)+0x1c) [0x744c58]
./appname(int rt.sections_linux.DSO.opApplyReverse(scope int delegate(ref rt.sections_linux.DSO))+0x46) [0x744eca]
./appname(rt_moduleDtor+0x19) [0x744c35]
./appname(rt_term+0x4a) [0x74003a]
./appname(void rt.dmain2._d_run_main(int, char**, extern (C) int function(char[][])*).runAll()+0x3e) [0x74034e]
./appname(void rt.dmain2._d_run_main(int, char**, extern (C) int function(char[][])*).tryExec(scope void delegate())+0x2a) [0x7402da]
./appname(_d_run_main+0x1a3) [0x74025b]
./appname(main+0x25) [0x60f185]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f804f51fde5]

0.7.18

OS X:
Exception in lev_unlock_mutex: Unknown lock handle
Exception in lev_lock_mutex: Unknown lock handle
Exception in lev_unlock_mutex: Unknown lock handle
Exception in lev_lock_mutex: Unknown lock handle
Exception in lev_unlock_mutex: Unknown lock handle
core.exception.AssertError@../../../../.dub/packages/vibe-d-0.7.18/source/vibe/core/driver.d(28): No event driver loaded. Did the vibe.core.core module constructor run?
----------------
5   www.luismarques.eu                  0x000000010663ad8d _d_assert_msg + 69
6   www.luismarques.eu                  0x00000001065af336 vibe.core.driver.EventDriver vibe.core.driver.getEventDriver(bool) + 90
7   www.luismarques.eu                  0x00000001065aa49d void vibe.core.core.VibeDriverCore.yieldForEvent() + 329
8   www.luismarques.eu                  0x00000001065b79cd void vibe.core.drivers.libevent2_tcp.Libevent2TCPConnection.close() + 381
9   www.luismarques.eu                  0x000000010655f8d5 void ddb.postgres.PGConnection.close() + 125
10  www.luismarques.eu                  0x000000010654ae4f void db._sharedStaticDtor3() + 55
11  www.luismarques.eu                  0x0000000106511d09 void db.__modshareddtor() + 9
12  www.luismarques.eu                  0x000000010664dc49 pure void rt.minfo.__T17runModuleFuncsRevS452rt5minfo11ModuleGroup8runDtorsMFZv9__lambda1Z.runModuleFuncsRev(object.ModuleInfo*[]) + 97
13  www.luismarques.eu                  0x000000010664d685 void rt.minfo.ModuleGroup.runDtors() + 29
14  www.luismarques.eu                  0x000000010664da0c int rt.minfo.rt_moduleDtor().__foreachbody1(ref rt.sections_osx.SectionGroup) + 28
15  www.luismarques.eu                  0x000000010664d9e9 rt_moduleDtor + 41
16  www.luismarques.eu                  0x0000000106647ec5 rt_term + 73
17  www.luismarques.eu                  0x0000000106648222 void rt.dmain2._d_run_main(int, char**, extern (C) int function(char[][])*).runAll() + 62
18  www.luismarques.eu                  0x00000001066481b1 void rt.dmain2._d_run_main(int, char**, extern (C) int function(char[][])*).tryExec(scope void delegate()) + 45
19  www.luismarques.eu                  0x000000010664812d _d_run_main + 449
20  www.luismarques.eu                  0x000000010656a1ea main + 34
21  libdyld.dylib                       0x00007fff8d5275fd start + 1
22  ???                                 0x0000000000000001 0x0 + 1

0.7.17

OS X:
Received signal 2. Shutting down.
Exception in lev_lock_mutex: Assertion failure

0.7.16

No longer compiles:

Undefined symbols for architecture x86_64:
  "_D4vibe4core3log10ss_loggersAOC4vibe4core3log6Logger", referenced from:
      _D4vibe4core3log186__T3logVE4vibe4core3log8LogLevel2VAyaa67_2e2e2f2e2e2f2e6475622f7061636b616765732f766962652d642d302e372e31392f736f757263652f766962652f636f72652f636f6e6e656374696f6e706f6f6c2e64Vi60TAyaTmZ3logFNbAyaKmZv in libddb.a(log_d80_2457.o)
      _D4vibe4core3log187__T3logVE4vibe4core3log8LogLevel2VAyaa67_2e2e2f2e2e2f2e6475622f7061636b616765732f766962652d642d302e372e31392f736f757263652f766962652f636f72652f636f6e6e656374696f6e706f6f6c2e64Vi55TAyaTPvZ3logFNbAyaKPvZv in libddb.a(log_d7e_2412.o)
      _D4vibe4core3log190__T3logVE4vibe4core3log8LogLevel2VAyaa67_2e2e2f2e2e2f2e6475622f7061636b616765732f766962652d642d302e372e31392f736f757263652f766962652f636f72652f636f6e6e656374696f6e706f6f6c2e64Vi53TAyaTAyaTmZ3logFNbAyaKAyaKmZv in libddb.a(log_d7c_273c.o)
      _D4vibe4core3log192__T3logVE4vibe4core3log8LogLevel0VAyaa67_2e2e2f2e2e2f2e6475622f7061636b616765732f766962652d642d302e372e31392f736f757263652f766962652f636f72652f636f6e6e656374696f6e706f6f6c2e64Vi50TAyaTAyaTmTmZ3logFNbAyaKAyaKmKmZv in libddb.a(log_d78_29a8.o)
ld: symbol(s) not found for architecture x86_64

This behavior does not happen if one skips the module destructor:

shared static ~this()
{
    //pgdb.close();
}

Yet, the close operation of ddb does not have anything very obvious to fix:

void close()
{
    sendTerminateMessage();
    stream.socket.close();
}

void sendTerminateMessage()
{
    stream.write('X');
    stream.write(cast(int)4);
}

...so I don't know how to proceed. Since the crash seems vibe.d-related, I'm asking for help here, for a start. Any ideas?

Re: Problem with vibe.d + ddb

Permalink: HTTP NNTP

Sönke Ludwig

Posted Fri, 02 May 2014 12:59:07 +0200 in reply to Luís Marques

Am 02.05.2014 02:14, schrieb Luís Marques:

Hello,

I'm having a problem that I did not use to have, using vibe.d + ddb (postgres driver with vibe.d support). I'm not sure with which version of vibe.d or ddb it started with, so I provide details for several vibe.d releases. I did not test with several ddb releases because there is only one release tag, which I actually did help create :-) (it used to use a ~master dependency).

The bug can be exercised with the following program. It should be compiled and run as part of vibe.d, but it does not even require that one does a vibe.d's listenHTTP for the problem to appear (I think):
(...)

I've installed PostgreSQL 9.3.4 on Windows and set the password "admin"
for the "postgres" user and updated the parameters for the PostgresDB
constructor. However, I'm always getting a "FATAL 28P01:" error stating
that the password based authentication has failed. Is there anything I'm
missing?

Re: Problem with vibe.d + ddb

Permalink: HTTP NNTP

Luís Marques

Posted Fri, 02 May 2014 11:13:28 GMT in reply to Sönke Ludwig

On Fri, 02 May 2014 12:59:07 +0200, Sönke Ludwig wrote:

I've installed PostgreSQL 9.3.4 on Windows and set the password "admin"
for the "postgres" user and updated the parameters for the PostgresDB
constructor. However, I'm always getting a "FATAL 28P01:" error stating
that the password based authentication has failed. Is there anything I'm
missing?

I also had difficulties with postgresql authentication. As a temporary workaround, I ended up choosing the following authentication method:

(file /etc/postgresql/9.1/main/pg_hba.conf)
host    all             all             127.0.0.1/32            trust

Which means:

When trust authentication is specified, PostgreSQL assumes that anyone who can connect to the server is authorized to access the database with whatever database user name they specify (even superuser names). Of course, restrictions made in the database and user columns still apply. This method should only be used when there is adequate operating-system-level protection on connections to the server.

trust authentication is appropriate and very convenient for local connections on a single-user workstation. It is usually not appropriate by itself on a multiuser machine. However, you might be able to use trust even on a multiuser machine, if you restrict access to the server's Unix-domain socket file using file-system permissions.

I had assumed it was just my ignorance of Postgresql that was causing my authentication issues, but now I'm starting to reconsider that position :-) (If you manage to make Postgresql + ddb work with the md5-based password authentication please say so).

Re: Problem with vibe.d + ddb

Permalink: HTTP NNTP

Sönke Ludwig

Posted Fri, 02 May 2014 12:11:23 GMT in reply to Luís Marques

On Fri, 02 May 2014 11:13:28 GMT, Luís Marques wrote:

On Fri, 02 May 2014 12:59:07 +0200, Sönke Ludwig wrote:

I've installed PostgreSQL 9.3.4 on Windows and set the password "admin"
for the "postgres" user and updated the parameters for the PostgresDB
constructor. However, I'm always getting a "FATAL 28P01:" error stating
that the password based authentication has failed. Is there anything I'm
missing?

I also had difficulties with postgresql authentication. As a temporary workaround, I ended up choosing the following authentication method:
(file /etc/postgresql/9.1/main/pg_hba.conf)
host    all             all             127.0.0.1/32            trust
Which means:
When trust authentication is specified, PostgreSQL assumes that anyone who can connect to the server is authorized to access the database with whatever database user name they specify (even superuser names). Of course, restrictions made in the database and user columns still apply. This method should only be used when there is adequate operating-system-level protection on connections to the server.

trust authentication is appropriate and very convenient for local connections on a single-user workstation. It is usually not appropriate by itself on a multiuser machine. However, you might be able to use trust even on a multiuser machine, if you restrict access to the server's Unix-domain socket file using file-system permissions.
I had assumed it was just my ignorance of Postgresql that was causing my authentication issues, but now I'm starting to reconsider that position :-) (If you manage to make Postgresql + ddb work with the md5-based password authentication please say so).

Okay, using "password" instead of "trust" or "md5" also did the trick for me.

The issue is caused by the order or module destructors:

vibe.core.core's ~static this is called, cleaning up the event driver
the application's ~shared static this is called, failing to close the connection, because the driver is already destroyed
vibe.core.core's ~shared static this would be called, performing the final clean up of vibe.d

I've implemented a workaround that delays the deletion of the main event driver to step 3, which fixes the issue (and hopefully doesn't introduce new ones ;)

Commit: 2be4cce

Re: Problem with vibe.d + ddb

Permalink: HTTP NNTP

Luís Marques

Posted Fri, 02 May 2014 13:53:51 GMT in reply to Sönke Ludwig

On Fri, 02 May 2014 12:11:23 GMT, Sönke Ludwig wrote:

Commit: 2be4cce

I'm confused with dub's version selection mechanism. In the previous days, I successfully did an add-local for ddb, so that I could use my fork. But not I'm having trouble trying to select that commit (or just ~master) for vibe.d. I have:

dub.json:
"dependencies": {
    "vibe-d": ">=0.1.19",
    "ddb": ">=0.1.0"
},

dub.selections.json:
"versions": {
    "vibe-d": "0.7.19",
    "libevent": "~master",
    "openssl": "~master",
    "libev": "~master",
    "ddb": "0.1.0"
}

If I change vibe-d to "~master" in dub.selections.json, dub build overrides it with 0.7.19 again. If I do an add-local of my local ~master of the vibe-d repo, then dub build keeps using 0.7.19. If I then change dub.json to...

"dependencies": {
    "vibe-d": "~master",
    "ddb": ">=0.1.0"
},

...it complains that it "Could not find a valid dependency tree configuration", even though ddb has a dub.json of...

"dependencies": {
    "vibe-d": {"version": ">=0.7.19", "optional": true}
}

...and I would think that >=0.7.19 is compatible with ~master, even more so with an "optional" dependency.

It's hard to understand all of these concepts and how they interact because there doesn't seem to be documentation for them, and for the CLI commands.

Also, how about adding support so that I can specify directly in dub.json/dub.selections.json/wherever that I want to use "rev:2be4cce", instead of having to do an add-local checked-out to that particular commit? (assuming that's how I should do it)

Re: Problem with vibe.d + ddb

Permalink: HTTP NNTP

Luís Marques

Posted Fri, 02 May 2014 14:09:26 GMT in reply to Sönke Ludwig

On Fri, 02 May 2014 12:11:23 GMT, Sönke Ludwig wrote:

The issue is caused by the order or module destructors:

vibe.core.core's ~static this is called, cleaning up the event driver

the application's ~shared static this is called, failing to close the connection, because the driver is already destroyed

vibe.core.core's ~shared static this would be called, performing the final clean up of vibe.d

I've implemented a workaround that delays the deletion of the main event driver to step 3, which fixes the issue (and hopefully doesn't introduce new ones ;)

Anyway, how would a sound / long-term design solve that? Should apps register their destructors with vibe.d? Should D implement some kind of module destructor dependency mechanism? Ideas?

Re: Problem with vibe.d + ddb

Permalink: HTTP NNTP

Sönke Ludwig

Posted Fri, 02 May 2014 16:22:04 +0200 in reply to Luís Marques

Am 02.05.2014 16:09, schrieb Luís Marques:

On Fri, 02 May 2014 12:11:23 GMT, Sönke Ludwig wrote:

The issue is caused by the order or module destructors:

vibe.core.core's ~static this is called, cleaning up the event driver

the application's ~shared static this is called, failing to close the connection, because the driver is already destroyed

vibe.core.core's ~shared static this would be called, performing the final clean up of vibe.d

I've implemented a workaround that delays the deletion of the main event driver to step 3, which fixes the issue (and hopefully doesn't introduce new ones ;)

Anyway, how would a sound / long-term design solve that? Should apps register their destructors with vibe.d? Should D implement some kind of module destructor dependency mechanism? Ideas?

D already implements such a mechanism, it's just that all per-thread
destructors are run first, but the individual module destructors of each
type are always called in dependee-first order (and the constuctors in
dependecy-first order). So I'd say that this fix is all that is needed.
It should completely hide the destruction of the driver from the
application now.

Re: Problem with vibe.d + ddb

Permalink: HTTP NNTP

Sönke Ludwig

Posted Fri, 02 May 2014 14:30:31 GMT in reply to Luís Marques

On Fri, 02 May 2014 13:53:51 GMT, Luís Marques wrote:

On Fri, 02 May 2014 12:11:23 GMT, Sönke Ludwig wrote:

Commit: 2be4cce

I'm confused with dub's version selection mechanism. In the previous days, I successfully did an add-local for ddb, so that I could use my fork. But not I'm having trouble trying to select that commit (or just ~master) for vibe.d. I have:
(...)

I think you may have an older preview version of DUB 0.9.22. There have been a number of fixes since the last beta (will get out another one shortly).

How it should work now is that you can GIT checkout a repository and "add-local" (without an explicit version) or "add-path" register it to DUB. The version of that local working copy will then be determined using the latest version tag instead of simply using the branch name, if possible.

In vibe.d's example, this would be something like vibe-d 0.7.20-alpha.1+commit.72.g2be4cce. Running dub upgrade --prerelease will then automatically update dub.selections.json properly if the dependency has been specified as a proper range, such as "~>0.7.19" (equivalent to ">=0.7.19 <0.8.0"). Manually editing dub.selections.json works now properly, too (it only overwrites entries now if they don't match any existing version).

Re: Problem with vibe.d + ddb

Permalink: HTTP NNTP

Luís Marques

Posted Fri, 02 May 2014 14:48:29 GMT in reply to Sönke Ludwig

On Fri, 02 May 2014 14:30:31 GMT, Sönke Ludwig wrote:

I think you may have an older preview version of DUB 0.9.22. There have been a number of fixes since the last beta (will get out another one shortly).

How it should work now is that you can GIT checkout a repository and "add-local" (without an explicit version) or "add-path" register it to DUB. The version of that local working copy will then be determined using the latest version tag instead of simply using the branch name, if possible.

In vibe.d's example, this would be something like vibe-d 0.7.20-alpha.1+commit.72.g2be4cce. Running dub upgrade --prerelease will then automatically update dub.selections.json properly if the dependency has been specified as a proper range, such as "~>0.7.19" (equivalent to ">=0.7.19 <0.8.0"). Manually editing dub.selections.json works now properly, too (it only overwrites entries now if they don't match any existing version).

dub's ~master works for the vibe.d version selection (yay!), but it breaks the build:

$ dub build               
Target vibe-d 0.7.20-alpha.1.72.g2be4cce is up to date. Use --force to rebuild.
Target ddb 0.1.0 is up to date. Use --force to rebuild.
Building www.luismarques.eu ~master configuration "dev-osx", build type debug.
Compiling...
Error: cannot read file app.d

BTW, I keep forgetting which dub version I have installed at a given time because there's an issue with the dub package for my package manager, homebrew. I tried to fix it, but did not succeed. I have asked for help; I was going to put here a link for the help email, but I don't see it yet in the archives (http://librelist.com/browser/homebrew/), so let me quote it:

I haven't found a way yet to fix the following issue. The software 'dub' has a build.sh shell script which writes the git version to a source file before compiling the software:

(...)
echo Generating version file...
GITVER=$(git describe) || GITVER=unknown
echo "module dub.version; enum dubVersion = \"$GITVER\";" > source/dub/version.d
(...)

Because homebrew copies the files outside of the git repo before doing a 'system "./build.sh"', that mechanism fails to generate the version, so the command 'dub help' says 'DUB version unknown'.

I'm not sure how to fix this. I thought about doing a custom/derived GitDownloadStrategy, although that seems a bit hackish, but I wasn't able to make that work anyway (if I recall correctly I was having problems with the homebrew paths; not knowing Ruby doesn't help).

Any suggestions on a good way to address this issue?

Just so you are aware of this issue, or in case anyone here knows how to solve this :-)

Re: Problem with vibe.d + ddb

Permalink: HTTP NNTP

Luís Marques

Posted Fri, 02 May 2014 16:16:05 GMT in reply to Luís Marques

On Fri, 02 May 2014 14:48:29 GMT, Luís Marques wrote:

Just so you are aware of this issue, or in case anyone here knows how to solve this :-)

Fixed: https://github.com/Homebrew/homebrew/pull/28920

Pages: 1 2 3