RejectedSoftware Forums

Sign up

Pages: 1 2

Template-heavy code and DUB

My Dgraph project substantially consists of generic, template-heavy code -- the kind of stuff which in C++ would probably be provided as Boost-style headers with no "source" files. From a DUB point of view, what that means is that if DUB builds a libdgraph.a static library, which is then compiled into other executables, there's a speed hit by comparison with those files just compiling in the dgraph source.

Is there a recommended way to specify in package.json that a package should (or could) be treated as a "header library", where dependent code should just build stuff in via module imports, rather than via linking in a prebuilt library?

Re: Template-heavy code and DUB

On Mon, 21 Oct 2013 04:08:02 GMT, Joseph Rushton Wakeling wrote:

Is there a recommended way to specify in package.json that a package should (or could) be treated as a "header library", where dependent code should just build stuff in via module imports, rather than via linking in a prebuilt library?

I noticed the "sourceLibrary" target option listed among the available target types, but when I try using it, issuing a "dub build" command results in "Error: Assertion failure"

Re: Template-heavy code and DUB

Perhaps it's best to summarize what I have done so far so that people can advise how best to tweak it.

Dgraph consists of a library (which ideally should be considered a sourceLibrary) plus 3 executables that are simple test demos of the library's performance. I've got the library part building with the following package.json:

{
    "name": "dgraph",
    "description": "A library for creating, analysing and manipulating graphs (networks).  It aims to be fast and memory-efficient while also being easy to use and extend.",
    "authors": ["Joseph Rushton Wakeling"],
    "copyright": "Copyright © 2013 Joseph Rushton Wakeling",
    "homepage": "https://github.com/WebDrake/Dgraph",
    "license": "GPL-3.0 or later",
    "targetType": "library",
    "excludedSourceFiles": ["source/graphtest.d", "source/betweenness50.d", "source/betweenness10k.d"],
    "subPackages": [
        {
            "name": "dgraph-test",
            "targetType": "executable",
            "sourceFiles": ["source/graphtest.d"]
        },
        {
            "name": "dgraph-betweenness50",
            "targetType": "executable",
            "sourceFiles": ["source/betweenness50.d"]
        },
        {
            "name": "dgraph-betweenness10k",
            "targetType": "executable",
            "sourceFiles": ["source/betweenness10k.d"]
        }
    ]
}

Replacing "library" with "sourceLibrary", as previously mentioned, results in "Error: Assertion failure".

Issuing dub build with the above package.json results in the static library part being built without apparent problem as libdgraph.a. However, the executables don't build, and trying to force their being built manually by e.g. dub build dgraph:dgraph-test results in: "Error: Unknown build configuration: library".

I've tried adding a dependency on dgraph to the individual executables, which results in the following error message:

Failed to retrieve metadata for package dgraph: Could not find package candidate for dgraph ~master
Could not resolve dependencies
The dependency graph could not be filled, there are unresolved dependencies.
The following changes will be performed:
Failure dgraph ~master, userWide
Issued by: 
 dgraph:dgraph-test: ~master
Error: Unknown build configuration: library

I have tried adding "version": "~master" to package.json to resolve this, but the same error results. I've also tried the converse -- instead of adding a dgraph dependency to the executables, adding a dependency on the executables to dgraph, but the executables are still not built.

So, I think I'm about at the limits of faffing around without really understanding what I'm doing, and would really appreciate some advice on how to get this working. :-)

Re: Template-heavy code and DUB

Am 21.10.2013 06:20, schrieb Joseph Rushton Wakeling:

On Mon, 21 Oct 2013 04:08:02 GMT, Joseph Rushton Wakeling wrote:

Is there a recommended way to specify in package.json that a package should (or could) be treated as a "header library", where dependent code should just build stuff in via module imports, rather than via linking in a prebuilt library?

I noticed the "sourceLibrary" target option listed among the available target types, but when I try using it, issuing a "dub build" command results in "Error: Assertion failure"

I'll look into the assertion failure, it's probably a missing error
message along the lines of "The target type is 'sourceLibrary', which
has no target. Stopping build.", or did you invoke the "dub build " on a
dependent project?

But what you can do, assuming that you only have files that need not
be compiled, is to set "targetType": "sourceLibrary" and then put all
files in an "import" folder instead of "source", or manually set
"sourcePaths": [] to avoid compiling them.

If you have a mixture of files, just put those into an "import" folder
which don't need to be compiled up front. The target type in this case
can be the default "library".

Re: Template-heavy code and DUB

On Mon, 21 Oct 2013 08:55:53 +0200, Sönke Ludwig wrote:

I'll look into the assertion failure, it's probably a missing error
message along the lines of "The target type is 'sourceLibrary', which
has no target. Stopping build.", or did you invoke the "dub build " on a
dependent project?

No, I invoked the build on dgraph itself. I did imagine there might be such a "no target" issue for a sourceLibrary, but I had assumed it would not get in the way of building executable subpackages.

But what you can do, assuming that you only have files that need not
be compiled, is to set "targetType": "sourceLibrary" and then put all
files in an "import" folder instead of "source", or manually set
"sourcePaths": [] to avoid compiling them.

I'll give that a go.

If you have a mixture of files, just put those into an "import" folder
which don't need to be compiled up front. The target type in this case
can be the default "library".

Well, leaving aside the stuff that would go in the "import" folder, the only things needing to be built are executables, not a library. The executables shouldn't be built in the case of someone just wanting dgraph as a library, though -- they're test demos, not essential tools. It seemed natural to make them subpackages in that case.

Re: Template-heavy code and DUB

On Mon, 21 Oct 2013 08:55:53 +0200, Sönke Ludwig wrote:

Am 21.10.2013 06:20, schrieb Joseph Rushton Wakeling:

On Mon, 21 Oct 2013 04:08:02 GMT, Joseph Rushton Wakeling wrote:

Is there a recommended way to specify in package.json that a package should (or could) be treated as a "header library", where dependent code should just build stuff in via module imports, rather than via linking in a prebuilt library?

I noticed the "sourceLibrary" target option listed among the available target types, but when I try using it, issuing a "dub build" command results in "Error: Assertion failure"

I'll look into the assertion failure, it's probably a missing error
message along the lines of "The target type is 'sourceLibrary', which
has no target. Stopping build.", or did you invoke the "dub build " on a
dependent project?

Error message fixed by 1201e4c.

Selecting the wrong configuration for sub packages ("library") is fixed by 2b29072.

And finally, putting the root package as a dependency into the sub packages is fixed by 9700a53.

Everything should work now. However, I would recommend to put the test packages in their own part of the directory structure, for example:

/
  source/
    dgraph/
      graph.d
      metric.d
  tests/
    graphtest/
      source/
        graphtest.d
      package.json

(Single-file packages would help a lot to keep the number of files/folders down in the future)

The main reason is that those test modules will pollute the D package namespace if they are inside of the same folder as the library modules. It's also obvious from the directory structure where the tests are and how they can be built, without having to look at the package.json file.

Re: Template-heavy code and DUB

On Mon, 21 Oct 2013 08:55:53 +0200, Sönke Ludwig wrote:

But what you can do, assuming that you only have files that need not
be compiled, is to set "targetType": "sourceLibrary" and then put all
files in an "import" folder instead of "source", or manually set
"sourcePaths": [] to avoid compiling them.

If you have a mixture of files, just put those into an "import" folder
which don't need to be compiled up front. The target type in this case
can be the default "library".

OK, all the library files are now in import/dgraph and I've tweaked package.json to the following:

{
    "name": "dgraph",
    "version": "~master",
    "description": "A library for creating, analysing and manipulating graphs (networks).  It aims to be fast and memory-efficient while also being easy to use and extend.",
    "authors": ["Joseph Rushton Wakeling"],
    "copyright": "Copyright © 2013 Joseph Rushton Wakeling",
    "homepage": "https://github.com/WebDrake/Dgraph",
    "license": "GPL-3.0 or later",
    "targetType": "sourceLibrary",
}

I symlinked the package dir into .dub/packages and dub list-installed recognizes its presence. I then created a separate project containing just source/app.d and the following package.json:

{
    "name": "dgraph-test",
    "version": "~master",
    "targetType": "executable",
    "dependencies": { "dgraph": "~master" }
}

Issuing dub build in that project results in the following error:

$ dub build
Checking dependencies in '/home/joseph/code/D/dgtest'
Building configuration "application", build type debug
Compiling...
source/app.d(24): Error: module graph is in file 'dgraph/graph.d' which cannot be read
import path[0] = source
import path[1] = ../../../.dub/packages/dgraph/source
import path[2] = /opt/dmd/include/d2
Error: DMD compile run failed with exit code 1

Run 'dub help' for usage information.

I see you've since replied in greater depth (and updated DUB, thank you!) as I was writing this, so I'm just posting this for information, will follow up if your later suggestions don't work.

Re: Template-heavy code and DUB

On Mon, 21 Oct 2013 08:05:51 GMT, Joseph Rushton Wakeling wrote:

I symlinked the package dir into .dub/packages and dub list-installed recognizes its presence. I then created a separate project containing just source/app.d and the following package.json:

{
    "name": "dgraph-test",
    "version": "~master",
    "targetType": "executable",
    "dependencies": { "dgraph": "~master" }
}

Issuing dub build in that project results in the following error

... which persists even with the latest updates to dub. I think it's simply because the library source is in import/ instead of source/.

I moved the library source back to source/dgraph and rewrote the dgraph package.json to be:

{
    "name": "dgraph",
    "version": "~master",
    "description": "A library for creating, analysing and manipulating graphs (networks).  It aims to be fast and memory-efficient while also being easy to use and extend.",
    "authors": ["Joseph Rushton Wakeling"],
    "copyright": "Copyright © 2013 Joseph Rushton Wakeling",
    "homepage": "https://github.com/WebDrake/Dgraph",
    "license": "GPL-3.0 or later",
    "targetType": "sourceLibrary",
    "excludedSourceFiles": ["source/graphtest.d", "source/betweenness50.d", "source/betweenness10k.d"]
}

... and now my downstream separate dgtest project builds successfully. Good progress! So I'll see if I can now get things working using separate directory structures as you suggest -- much preferable to the "excludedSourceFiles" approach I've had to take here (which is necessary, otherwise the downstream project starts complaining about more than one main()).

Re: Template-heavy code and DUB

Am 21.10.2013 10:22, schrieb Joseph Rushton Wakeling:

On Mon, 21 Oct 2013 08:05:51 GMT, Joseph Rushton Wakeling wrote:

I symlinked the package dir into .dub/packages and dub list-installed recognizes its presence. I then created a separate project containing just source/app.d and the following package.json:

{
     "name": "dgraph-test",
     "version": "~master",
     "targetType": "executable",
     "dependencies": { "dgraph": "~master" }
}

Issuing dub build in that project results in the following error

... which persists even with the latest updates to dub. I think it's simply because the library source is in import/ instead of source/.

I'll try to reproduce that later on a Linux machine.

I moved the library source back to source/dgraph and rewrote the dgraph package.json to be:

{
     "name": "dgraph",
     "version": "~master",
     "description": "A library for creating, analysing and manipulating graphs (networks).  It aims to be fast and memory-efficient while also being easy to use and extend.",
     "authors": ["Joseph Rushton Wakeling"],
     "copyright": "Copyright © 2013 Joseph Rushton Wakeling",
     "homepage": "https://github.com/WebDrake/Dgraph",
     "license": "GPL-3.0 or later",
     "targetType": "sourceLibrary",
     "excludedSourceFiles": ["source/graphtest.d", "source/betweenness50.d", "source/betweenness10k.d"]
}

... and now my downstream separate dgtest project builds successfully. Good progress! So I'll see if I can now get things working using separate directory structures as you suggest -- much preferable to the "excludedSourceFiles" approach I've had to take here (which is necessary, otherwise the downstream project starts complaining about more than one main()).

For the separate package.json approach, instead of the symlink
workaround, you can explicitly specify the path to the root package like
this:

{
	...
	"dependencies": {
		"dgraph": {"version": "~master", "path": "../../"}
	}
}

Hope this works out.

Re: Template-heavy code and DUB

On Mon, 21 Oct 2013 12:18:18 +0200, Sönke Ludwig wrote:

Hope this works out.

Well, here's the solution I wound up with:

{
    "name": "dgraph",
    "version": "~master",
    "description": "A library for creating, analysing and manipulating graphs (networks).  It aims to be fast and memory-efficient while also being easy to use and extend.",
    "authors": ["Joseph Rushton Wakeling"],
    "copyright": "Copyright © 2013 Joseph Rushton Wakeling",
    "homepage": "https://github.com/WebDrake/Dgraph",
    "license": "GPL-3.0 or later",
    "targetType": "sourceLibrary",

    "subPackages": [
        {
            "name": "graphtest",
            "targetType": "executable",
            "sourcePaths": ["util/graphtest/source/"],
            "dependencies": { "dgraph": "~master" }
        },

        {
            "name": "betweenness50",
            "targetType": "executable",
            "sourcePaths": ["util/betweenness50/source/"],
            "dependencies": { "dgraph": "~master" }
        },

        {
            "name": "betweenness10k",
            "targetType": "executable",
            "sourcePaths": ["util/betweenness10k/source/"],
            "dependencies": { "dgraph": "~master" }
        }

    ]
}

dub build dgraph:whatever works to build any of the executables, and downstream projects seem to interact fine with dgraph as a sourceLibrary. The one thing I'd like to have is some way to build all of the executables at once -- dub build dgraph:util or something like that; I tried creating a subpackage with "targetType": "none" with dependencies on all of them, but dub rejected that as "none" implied, nothing to build.

I didn't quite follow your proposal to have a separate package.json in each of the different executables' project directories -- how would this work from a dub build perspective?

Pages: 1 2