RejectedSoftware Forums

Sign up

Pages: 1 2

External dependencies

So I downloaded vibe.d, created a "hello world" project with a dub template, and compiled it. This was the result on my OSX box:

Building configuration "application", build type debug
Compiling...
Linking...
ld: library not found for -levent
collect2: ld returned 1 exit status
--- errorlevel 1
Error: Link command failed with exit code 1

libevent2 isn't a dub package so this error makes sense, but I'm wondering how the situation could be handled better. Should there be some way for a project to enumerate external dependencies?

Re: External dependencies

On Tue, 08 Oct 2013 16:55:46 GMT, Sean Kelly wrote:

So I downloaded vibe.d, created a "hello world" project with a dub template, and compiled it. This was the result on my OSX box:

Building configuration "application", build type debug
Compiling...
Linking...
ld: library not found for -levent
collect2: ld returned 1 exit status
--- errorlevel 1
Error: Link command failed with exit code 1

libevent2 isn't a dub package so this error makes sense, but I'm wondering how the situation could be handled better. Should there be some way for a project to enumerate external dependencies?

There have been ideas in that direction in #52 and #34 and also in some earlier discussions (need to search for those). However, there have been strong objections from package maintainers regarding the feasibility or hygiene of this approach in general.

I agree it's a very ambitious side project. However maybe there is some middle ground solution where at least a customized error message can be output in case of a missing library. In our case, external libraries of this kind ideally should only occur for Deimos(-like) modules, so it's not unrealistic that there will be good broad support for this solution, even if it requires manual work.

Crude syntax idea:

{
  "libs": "event", "ssl",
  "libNames": {
    "event": "Libevent 2.0.x",
    "ssl": "OpenSSL"
  }
}

The automatically generated message could then look like "Libevent 2.0.x development files have not been found. Please install them using the system's package manager." - maybe with some platform specific additions: " Homebrew is recommended to manage packages on OS X - see http://brew.sh/");

Having said that, at some point I personally would really like to have a full solution, so that DUB (or a utility added on top) can also automatically generate system packages for various systems.

Re: External dependencies

On 2013-10-09 08:00, Sönke Ludwig wrote:

There have been ideas in that direction in #52 and #34 and also in some earlier discussions (need to search for those). However, there have been strong objections from package maintainers regarding the feasibility or hygiene of this approach in general.

I agree it's a very ambitious side project. However maybe there is some middle ground solution where at least a customized error message can be output in case of a missing library. In our case, external libraries of this kind ideally should only occur for Deimos(-like) modules, so it's not unrealistic that there will be good broad support for this solution, even if it requires manual work.

I'm wondering how difficulty it would be to support external
dependencies. Something like:

{
"external" : {

 "curl": "1.2.3"

}
}

The tool would then just run "apt-get install curl", "port install curl"
or whatever the native package manager is for the given platform.

The problems I see with this is that Windows doesn't really have a
package manager. Mac OS X doesn't have one installed by default but I
think it's ok rely on Homebrew and MacPorts.

The other problem is package names. Different package manager might use
slightly different names for the same packages. Also MacPorts support
variants where other package managers might require a separate package.

/Jacob Carlborg

Re: External dependencies

Am 09.10.2013 20:25, schrieb Jacob Carlborg:

On 2013-10-09 08:00, Sönke Ludwig wrote:

There have been ideas in that direction in
#52 and
#34 and also in
some earlier discussions (need to search for those). However, there
have been strong objections from package maintainers regarding the
feasibility or hygiene of this approach in general.

I agree it's a very ambitious side project. However maybe there is
some middle ground solution where at least a customized error message
can be output in case of a missing library. In our case, external
libraries of this kind ideally should only occur for Deimos(-like)
modules, so it's not unrealistic that there will be good broad support
for this solution, even if it requires manual work.

I'm wondering how difficulty it would be to support external
dependencies. Something like:

{
"external" : {

 "curl": "1.2.3"

}
}

The tool would then just run "apt-get install curl", "port install curl"
or whatever the native package manager is for the given platform.

The problems I see with this is that Windows doesn't really have a
package manager. Mac OS X doesn't have one installed by default but I
think it's ok rely on Homebrew and MacPorts.

The other problem is package names. Different package manager might use
slightly different names for the same packages. Also MacPorts support
variants where other package managers might require a separate package.

This and things like different alternatives for the same package in some
package systems but only one in others, certain versions only available
on some systems, conflicts with other installed packages, etc.

So this would require at least a comprehensive database to intelligently
map to each system and some careful thought how to handle each source of
errors. While this may work reasonably well in the end for common
packages, there is indeed a high chance for error is such a system,
and the detached nature of the central database vs. the package
description also slightly obfuscates the source. It will also need some
community based, moderated edit scheme to work.

On the other hand it has the advantage of taking the burden of platform
specific configuration off of the developers shoulders and puts it on
the community, thus making the process more agile and less tied to
releases of a package.

Re: External dependencies

It is quite interesting and promising approach and lot of work is already done by PackageKit for most Linux distros. But it is still rather complex and fragile matter, I don't think it is worth trying to address it at such early stages. It is better to focus a bit on general infrastructure right now.

Re: External dependencies

On 09/10/2013 19:25, Jacob Carlborg wrote:

On 2013-10-09 08:00, Sönke Ludwig wrote:

There have been ideas in that direction in
#52 and
#34 and also in
some earlier discussions (need to search for those). However, there
have been strong objections from package maintainers regarding the
feasibility or hygiene of this approach in general.

I agree it's a very ambitious side project. However maybe there is
some middle ground solution where at least a customized error message
can be output in case of a missing library. In our case, external
libraries of this kind ideally should only occur for Deimos(-like)
modules, so it's not unrealistic that there will be good broad support
for this solution, even if it requires manual work.

I'm wondering how difficulty it would be to support external
dependencies. Something like:

{
"external" : {

 "curl": "1.2.3"

}
}

The tool would then just run "apt-get install curl", "port install curl"
or whatever the native package manager is for the given platform.

The problems I see with this is that Windows doesn't really have a
package manager. Mac OS X doesn't have one installed by default but I
think it's ok rely on Homebrew and MacPorts.

The other problem is package names. Different package manager might use
slightly different names for the same packages. Also MacPorts support
variants where other package managers might require a separate package.

These solutions involving dub using external/system package managers
don't feel right at all... I think this is a wrong approach.

Doesn't using external libraries imply you need a D binding/import
anyways? As such doesn't it follow that it is best that an external
library be wrapped by a dub-package? Here's the idea:
The dub-package wrapper would include the library binaries (for all
available platforms) and the respective D imports/bindings. This would
make the library available for dub usage in all platforms, with no
reliance on system package managers (thus, cross-platform).
If a newer version of the library is made available (say, you have a
dub-package for curl-1.2.3 and now 1.2.4 is out, then the dub-package
will have to be manually updated. Some auxilary tool could be used to
help with this process. This tool could potentially interact with a
system package manager, but it would be separate from dub (or sitting at
a higher level in the toolchain).

This approach requires more work to update libraries when new versions
arrive, but it's easier to implement (dub-wise) and provides a clean
platform-independent abstraction, because dub remains independent of
system package managers.
I think it's a worth tradeoff. In any case, any library updates other
than service-level version updates will (or should) require an update to
the D bindings since it means the underlying API has changed.

Bruno Medeiros

Re: External dependencies

On Thu, 21 Nov 2013 13:11:27 +0000, Bruno Medeiros wrote:

These solutions involving dub using external/system package managers
don't feel right at all... I think this is a wrong approach.

It is only possible correct approach for external/system dependencies that does not introduce own duplicate ecosystem.

Doesn't using external libraries imply you need a D binding/import
anyways?

Not necessarily, you can add needed extern(C) forward declaration at any point of program.

As such doesn't it follow that it is best that an external
library be wrapped by a dub-package?

It makes sense for wrapper package to depend on OS package but it is pretty much all.

Here's the idea:
The dub-package wrapper would include the library binaries (for all
available platforms)

I will throw it away from Arch repos the very moment it will happen. Not really threatening but should show how serious am I about it.

Really, just pick some random free package manager implementation for Windows / Mac and push it as default for that platform. It will do much more good in long term.

Re: External dependencies

On 21/11/2013 18:58, Dicebot wrote:

On Thu, 21 Nov 2013 13:11:27 +0000, Bruno Medeiros wrote:

These solutions involving dub using external/system package managers
don't feel right at all... I think this is a wrong approach.

It is only possible correct approach for external/system dependencies that does not introduce own duplicate ecosystem.

Disk space is incredibly cheap. I don't mind duplicated files.
As for duplication of effort (maintenance effort etc.), yes there is
some extra effort involved, but I think it's worthwhile to pay that, in
order to have 100% cross-platform support.

Here's the idea:
The dub-package wrapper would include the library binaries (for all
available platforms)

I will throw it away from Arch repos the very moment it will happen. Not really threatening but should show how serious am I about it.

What would you throw away from Arch repos? I'm not sure I'm
understanding you correctly. In this approach, the wrapper is a
dub-package, and therefore would live in the Dub registry (plus some git
repo for the actual contents), not in the Arch Linux repo.

Really, just pick some random free package manager implementation for Windows / Mac and push it as default for that platform. It will do much more good in long term.

"just pick some random free package manager implementation for Windows /
Mac" ??
Are you not thinking you are massively understimating the complexity of
the problem? Even if there was lots of manpower to develop this approach
in Dub, it is still a brittle mechanism. What about incompatibilities in
design between the various package managers? (I would bet there would be
lots of issues like that) What about incompatibilities between the
packages in the various PMs, relating to the same library. Issues, like,
one version is present in one PM, but not the other, or the version is
incorrectly specified in one PM but not the other, or even the package
is not present at all in one or some of the PMs... How can these issues
be adequately addressed if there is not one single entity per library
responsible for maitaining the various package formats of that library.

The only way to fully address this problem, taking care of both of our
concerns, would be to develop our own cross-platform system package
manager (or extend an existing one). But that is a massive undertaking.

Re: External dependencies

Can't really discuss that in details right now but here is an old argument with Jacob, it is on similar topic : http://forum.rejectedsoftware.com/groups/rejectedsoftware.dub/thread/274

Re: External dependencies

On 2013-11-21 14:11, Bruno Medeiros wrote:

These solutions involving dub using external/system package managers
don't feel right at all... I think this is a wrong approach.

Doesn't using external libraries imply you need a D binding/import
anyways? As such doesn't it follow that it is best that an external
library be wrapped by a dub-package? Here's the idea:
The dub-package wrapper would include the library binaries (for all
available platforms) and the respective D imports/bindings. This would
make the library available for dub usage in all platforms, with no
reliance on system package managers (thus, cross-platform).
If a newer version of the library is made available (say, you have a
dub-package for curl-1.2.3 and now 1.2.4 is out, then the dub-package
will have to be manually updated. Some auxilary tool could be used to
help with this process. This tool could potentially interact with a
system package manager, but it would be separate from dub (or sitting at
a higher level in the toolchain).

This approach requires more work to update libraries when new versions
arrive, but it's easier to implement (dub-wise) and provides a clean
platform-independent abstraction, because dub remains independent of
system package managers.
I think it's a worth tradeoff. In any case, any library updates other
than service-level version updates will (or should) require an update to
the D bindings since it means the underlying API has changed.

Just take a look at the current issues with Curl for DMD. It's not very
easy to make a library work across all different flavors of Linux.

/Jacob Carlborg

Pages: 1 2