RejectedSoftware Forums

On Wed, 04 Sep 2013 08:48:16 GMT, Sönke Ludwig wrote:

There are the two directories /var/lib/dub/ and ~/.dub/ where it installs source packages (more or less just a verbatim copy of the git repository contents, except that the package.json gets annotated with an additional "version" field). Installing binary files was/is on the agenda, but mostly just for application packages, as it is too easy to break binary compatibility when using precompiled libraries (e.g. by using different -version= identifiers) for it to make much sense in the current ecosystem of libraries. But this is something that mostly Robert Klotzner was driving forward and now he is still busy with his studies, so this may take a while until it is picked up again.

So a package manager that doesn't install packages, hmm...

Yes. It has gained some additional support here since the early days:

• "dub add-local <PATH>" will register a single package folder to dub
• "dub add-path <PATH>" will register a directory containing multiple packages to search
• a DUBPATH environment variable can be defined along the lines of go's GOPATH
• package versions in all those cases are inferred using "git describe", checking a "version" field in package.json, or falling back to ~master (in this order)
• private registries can be specified using --registry=URL or by putting a "registryUrls" field in the user or system configuration file (e.g. ~/.dub/settings.json)
• being able to specify a git URL for a dependency is planned, but still needs some details to be worked out

I see.

Currently, whenever a version based dependency is given (e.g. ">=1.0.0"), only tagged versions are considered. Similarly, when a branch dependency is given (e.g. "~master") only the HEAD of that branch is tracked. Two extensions to this have been considered up to now:

• Make it possible to also specify exact commit SHA's
• Let a version based dependency using ">=" fall back to "~master" in case the same package is also referenced as "~master" by some other involved package (thus assuming that version tags are always on master or at least that master is source compatible with tagged versions and not something completely different)

I've been looking at the registry, http://code.dlang.org/, it seems I can't tell the registry what's available?

On Wed, 04 Sep 2013 08:48:16 GMT, Sönke Ludwig wrote:

On Wed, 04 Sep 2013 06:53:31 GMT, Jacob Carlborg wrote:

I've been thinking about giving DUB a try. I know we have talked about some of this before but I have some questions about how DUB works.

• Where does dub installs everything? Does it even install things?

There are the two directories /var/lib/dub/ and ~/.dub/ where it installs source packages (more or less just a verbatim copy of the git repository contents, except that the package.json gets annotated with an additional "version" field).

Is installation target chosen on per-package basis or set up once? Can path for global one be configured?

Sorry for the late reply, I forgot about this. E-mail notification would be nice :)

On Wed, 04 Sep 2013 18:11:03 GMT, Sönke Ludwig wrote:

It does install packages, just that they are source packages and not binary packages. It's no different from all those package managers for the various interpreted languages if you want. But if you have a good idea how to sanely install binary packages considering that there is almost no guarantee for binary compatibility unless everything is exactly the same, I'm all ears.

What needs to be exactly the same? I guessing here:

• Compiler
• Architecture
• version flags

Compiler:

The compiler needs to be exactly the same since it doesn't have any binary or source compatibility. A field for specifying the supported compiler would probably be a good idea. An idea would be to hook dub up with DVM, if you're interested. Then dub could automatically install a switch compilers for different packages.

Architecture:

When installing a package it would be only for that given operating system. So what's useful here is having 32bit and 64bit packages. On Mac OS X it's already solved with universal binaries, we don't need to bother with that. On Linux, FreeBSD and Windows it should compile one 32bit package and one 64bit package. Don't know if it should do that by default or compile the native architecture only. Flags for controlling this would be needed. A field for specifying the supported architecture is needed as well.

When the package is installed it will install into a directory structure, like this:

~/.dub
|
|__ packages
  |
  |__ foo
    |
    |__ lib
    |
    |__ lib64
    |
    |__ src
    |
    |__ ... (and so on)
        

Possibly put the lib folders in an architecture directory to support corss-compiling in the future.

Version flags:

This might be a bit tricky. Macports on Mac OS X uses something called "variants". A package would have a list of variants, basically corresponding to version flag. Macports uses the plus sign (+) to indicate a variant. When depending on a specific variant of a package it would look something like this:

"dependencies": {
    "ddbc +mysql +sqlite": ">=0.2.9"
}

The above would install the "ddbc" package with the variants "mysql" and "sqlite". The directory structure could look something like this:

~/.dub
|
|__ packages
  |
  |__ ddbc+mysql+sqlite
    |
    |__ lib
    |
    |__ lib64
    |
    |__ src
    |
    |__ ... (and so on)

Or put the variants in a sub directory:

~/.dub
|
|__ packages
  |
  |__ ddbc
    |
    |__ variants
    | |
    | |__ +mysql+sqlite
    |   |
    |   |__ lib
    |   |
    |   |__ lib64
    |
    |__ src
    |
    |__ ... (and so on)

I think I don't fully get the question. You can register any package to make it available. The registry currently only supports GitHub and BitBucket hosted repositories, though. If you need something else like sourcing form a locally hosted .zip file or from a git URL, that's quite simple to add, I'm sure we can work something out there.

Looking at the registry, http://code.dlang.org/, I see that some packages have "~master" as the latest version, some have "2.2.2" and some have "~issue/2". I would most likely never want "master" to be available. I'm certainly never, ever want "~issue/2" to be available.

Am 11.09.2013 09:10, Jacob Carlborg wrote:

Sorry for the late reply, I forgot about this. E-mail notification would be nice :)

Use the NNTP interface! (Just saying that because I started adding email notifications almost a year ago, but never got the guts to finish them ;))

On Wed, 04 Sep 2013 18:11:03 GMT, Sönke Ludwig wrote:

It does install packages, just that they are source packages and not binary packages. It's no different from all those package managers for the various interpreted languages if you want. But if you have a good idea how to sanely install binary packages considering that there is almost no guarantee for binary compatibility unless everything is exactly the same, I'm all ears.

What needs to be exactly the same? I guessing here:

• Compiler
• Architecture
• version flags

That and possibly other compiler flags, depending on the desired use case (e.g. debug vs. release version or the variant of an external link dependency that is used). But already all combinations of compiler (version), architecture and version flags can be far too many to install them all by default -- but if we accept that, we are back to caching compiled binaries lazily (and maybe compiling some kind of default configuration eagerly if desired), which is exactly what is planned.

Compiler:

The compiler needs to be exactly the same since it doesn't have any binary or source compatibility. A field for specifying the supported compiler would probably be a good idea. An idea would be to hook dub up with DVM, if you're interested. Then dub could automatically install a switch compilers for different packages.

Architecture:

When installing a package it would be only for that given operating system. So what's useful here is having 32bit and 64bit packages. On Mac OS X it's already solved with universal binaries, we don't need to bother with that. On Linux, FreeBSD and Windows it should compile one 32bit package and one 64bit package. Don't know if it should do that by default or compile the native architecture only. Flags for controlling this would be needed. A field for specifying the supported architecture is needed as well.

When the package is installed it will install into a directory structure, like this:

 ~/.dub
 |
 |__ packages
   |
   |__ foo
     |
     |__ lib
     |
     |__ lib64
     |
     |__ src
     |
     |__ ... (and so on)
         

Possibly put the lib folders in an architecture directory to support corss-compiling in the future.

This is more or less what I have in mind, except that binaries would be put into a directory that is unique for all aspects that affect binary generation (i.e. the above) instead of only the architecture. Or rather into multiple directories, each for one combination. The unique part could be a hash value of all related factors or we could try to get something human readable, such as "(compiler)-(compiler version)-(os)-(arch)-(configuration)-..."

Version flags:

This might be a bit tricky. Macports on Mac OS X uses something called "variants". A package would have a list of variants, basically corresponding to version flag. Macports uses the plus sign (+) to indicate a variant. When depending on a specific variant of a package it would look something like this:

 "dependencies": {
     "ddbc +mysql +sqlite": ">=0.2.9"
 }

The above would install the "ddbc" package with the variants "mysql" and "sqlite". The directory structure could look something like this:

 ~/.dub
 |
 |__ packages
   |
   |__ ddbc+mysql+sqlite
     |
     |__ lib
     |
     |__ lib64
     |
     |__ src
     |
     |__ ... (and so on)

Or put the variants in a sub directory:

 ~/.dub
 |
 |__ packages
   |
   |__ ddbc
     |
     |__ variants
     | |
     | |__ +mysql+sqlite
     |   |
     |   |__ lib
     |   |
     |   |__ lib64
     |
     |__ src
     |
     |__ ... (and so on)

DUB has "configurations", but only one configuration can be active at a time for each package. But the latter variant using a sub directory is what I'm also imagining (just using a scheme that includes all aspects as mentioned above).

I think I don't fully get the question. You can register any package to make it available. The registry currently only supports GitHub and BitBucket hosted repositories, though. If you need something else like sourcing form a locally hosted .zip file or from a git URL, that's quite simple to add, I'm sure we can work something out there.

Looking at the registry, http://code.dlang.org/, I see that some packages have "~master" as the latest version, some have "2.2.2" and some have "~issue/2". I would most likely never want "master" to be available. I'm certainly never, ever want "~issue/2" to be available.

I agree that "~issue/2" should at least never be the default - this needs to be changed so that "~master" comes first. However, it's difficult to say what should be available. For example, there may be a "~2.x-devel" or a "~stable" branch that may very well make sense to use in some situations.

In general I agree of course that versioned releases are by far preferable, but during early development, or when having tightly coupled packages, it can be necessary to work on branches. I just don't want to make that impossible or force more structure/process on developers than necessary.