RejectedSoftware Forums

Sign up

Pages: 1 2

Re: Handling of a main() containing file.

Am 18.03.2013 21:39, schrieb Jacob Carlborg:

On Mon, 18 Mar 2013 06:39:22 GMT, Sönke Ludwig wrote:

On Sun, 17 Mar 2013 21:01:15 GMT, Robert wrote:

My thinking was if it is a source library, dub should not provide the main-containing file of vibe to the compiler if the application provides its own, but it should do that if the application does not provide its own main. But you are right, if we simply build static/dynamic libraries and use them, the linker takes care of that automatically.

I think this is a misunderstanding of what we both thought what a "main file" is (containing main() vs. being the root of the dependency hierarchy). So in particular the difficult problem is not to know if an application has a "main file", but if that file actually contains main() or not.

You cannot rely on a library having a "main file" or having a file containing "main()". It's possible and a perfectly valid use case to have a library containing only two files that don't import each other. Doesn't Dub just builds everything in the source directory? Actually, what Dub should do is pass all files in the source directory to Rdmd (or similar) to get all dependencies of all files, then compile all those files. Of course, in my opinion, that should be handled by a proper build tool and not the package manager.

I always thought that rdmd only accepts a single source file as the last
argument. Anyway, this is specifically about using rdmd as the build tool.

But remember, separate package/build tool or not, the goal for dub is to
have a build description that can be used to source any build tool. It's
on of the main features to have support for generating foreign
build/make/project files, so that a project can easily be integrated in
foreign build chains and can be used properly with IDEs.

Re: Handling of a main() containing file.

Yes, so if a custom "configurations" section or an own "sourcePaths" definition are specified, they will override the default choices. See package_.d line 101 ff. It first searches for all standard folder names and adds them, then parses the package.json file to let the user override the settings, and finally generates default configurations if none have been specified.

Ah, yeah now I see, the parsing of the package.json overwrites the values and does not append. Perfect :-)

I don't see how the static/dynamic libraries would solve that. If vibe.d would be compiled as a dynamic library, the linker should not consider it's main function at all and if compiled as a static library it should produce conflicting definition errors as in the source library case. It would be great if the linker could somehow be employed for this, but I don't see how yet.

As the linker only looks up symbols in libraries when they can not be found in the provided .o files, it will only use the main in the library if you provided none. You only get a conflict if you depend on some symbol that is within the same module as main. I just tested it on Linux, it works: If I don't provide my own main the one of the dynamic/static library is used, if I provide one, mine is used. I get a conflict if I define main and depend on a symbol that is in the same file as the library's main, but only then. To be honest, I would not even have thought of putting main in a library, but your statement about no-conflict, made me realize that this in fact works this way.

So in summary: If we use libraries/rdmd then dub does not need to know if the application provides its own main or not, nor in which file it would be. For rdmd it still needs to know the "app file", whether it contains main or not is irrelevant. The main/no-main problem is gone.

I think the approach with handling modules and HTML documentation and also the version statements is a good idea. But I think we shouldn't rely on that information for the build, if just because we may not be able to continually generate this information for branches if there are many active projects. Avoiding dependencies to rebuild also sounds like a low-priority issue, so as long as we don't introduce a road block for that somewhere, I think we can get to that once everything else is settled. Let's first get the build semantics right (...and some important stuff that has nothing to do with semantics at all).

dub would not need to rely on this information, if the registry can not provide it, dub can fall back to either:

  1. dumb behaviour (every time version statements are changed a new version of the library is built)
  2. Parse the code itself to gather the version/debug statements. (Bad at least because it would add another dependency to dub)
  3. Use rdmd. (We could for example simply require rdmd for branch builds)
  4. Do as C/C++ does, hope for the best (the version/debug command line changed do not affect the build). I don't really like that, but if dub would simply print a warning that it does not have this information and you have to take care, this might be fine for branch builds as the user always can fall back to rdmd.

Actually I started to think about this, because I had some problems building software with dub (dub seemed to not build the dependency at all, when using the default build method). In my quest for the reason, I started to think how I would do it, the last post was what I came up with ;-) -> So in essence this proposal was my concept of getting build semantics right, but to be honest I don't even understand exactly what your current approach is, maybe you could briefly explain what it looks like? And what the problems are?

  • How do you decide what to build? Simply all modules of all dependencies? Track them somehow via imports?
  • If you decided what to build, how do you build it?
    • All source files passed at once to dmd? Does this scale? If not, how do you split up?
    • One module at a time?
    • Some combination?

Also, when implementing this, I would make the whole thing a small separate build server that is controlled by a simple REST interface from the start, so that it can be a separate process with different user rights and on a different server (or multiple servers).

Sounds good. Btw, do you have the resources to host such a server? Or do you know someone who does?

The question I have in mind is how official it will become to build with rdmd. With voices that IDEs should also use the external build tool to build a project and other voices that always want to build with --rdmd it sounds like we should possibly actually provide rdmd support for all kinds of projects and we could maybe even go back and base everything on it...

I finally need to look up how rdmd works in detail, I am just guessing at the moment.

I think building libraries and rdmd are enough options. I

Re: Handling of a main() containing file.

I think building libraries and rdmd are enough options. I

-> Hmm, my reply got somehow truncated. I don't remember what I was going to say, so just ignore the last sentence. (Wasn't that important.)

Re: Handling of a main() containing file.

On Tue, 19 Mar 2013 07:25:17 +0100, Sönke Ludwig wrote:

I always thought that rdmd only accepts a single source file as the last
argument. Anyway, this is specifically about using rdmd as the build tool.

I don't know. If it can't it should be fixed or rdmd is not good enough.

Re: Handling of a main() containing file.

On Tue, 19 Mar 2013 19:07:17 GMT, Jacob Carlborg wrote:

On Tue, 19 Mar 2013 07:25:17 +0100, Sönke Ludwig wrote:

I always thought that rdmd only accepts a single source file as the last
argument. Anyway, this is specifically about using rdmd as the build tool.

I don't know. If it can't it should be fixed or rdmd is not good enough.

It's not good enough also for a number of other reasons, and the whole concept will also be difficult to get right. For example, suppose we have some dependencies build as dynamic libraries. How should rdmd or any dependency based build tool know where it has to stop when including D modules (i.e. at the border to the dynamic library)? It also needs to support separate compile/link and single file compilation modes for larger projects as a fallback in case of linker issues. Then there is the issue of the additional compiler run to initially determine all dependencies or recompile whenever compilation failed because new dependencies were added and the issue with missing local and string imports ind dmd's .dep files.

These are all reasons why I chose to not rely on rdmd/dependency tracking (and merely included it as a fallback option), but still there are a lot of opposing voices.

Re: Handling of a main() containing file.

On Tue, 19 Mar 2013 16:57:03 GMT, Robert wrote:

Yes, so if a custom "configurations" section or an own "sourcePaths" definition are specified, they will override the default choices. See package_.d line 101 ff. It first searches for all standard folder names and adds them, then parses the package.json file to let the user override the settings, and finally generates default configurations if none have been specified.

Ah, yeah now I see, the parsing of the package.json overwrites the values and does not append. Perfect :-)

I don't see how the static/dynamic libraries would solve that. If vibe.d would be compiled as a dynamic library, the linker should not consider it's main function at all and if compiled as a static library it should produce conflicting definition errors as in the source library case. It would be great if the linker could somehow be employed for this, but I don't see how yet.

As the linker only looks up symbols in libraries when they can not be found in the provided .o files, it will only use the main in the library if you provided none. You only get a conflict if you depend on some symbol that is within the same module as main. I just tested it on Linux, it works: If I don't provide my own main the one of the dynamic/static library is used, if I provide one, mine is used. I get a conflict if I define main and depend on a symbol that is in the same file as the library's main, but only then. To be honest, I would not even have thought of putting main in a library, but your statement about no-conflict, made me realize that this in fact works this way.

So in summary: If we use libraries/rdmd then dub does not need to know if the application provides its own main or not, nor in which file it would be. For rdmd it still needs to know the "app file", whether it contains main or not is irrelevant. The main/no-main problem is gone.

Interesting, I didn't know that. Is that standard behavior (i.e. does it work with Optlink, VisualStudio link.exe, the LLVM linker and maybe unilink)?

I think the approach with handling modules and HTML documentation and also the version statements is a good idea. But I think we shouldn't rely on that information for the build, if just because we may not be able to continually generate this information for branches if there are many active projects. Avoiding dependencies to rebuild also sounds like a low-priority issue, so as long as we don't introduce a road block for that somewhere, I think we can get to that once everything else is settled. Let's first get the build semantics right (...and some important stuff that has nothing to do with semantics at all).

dub would not need to rely on this information, if the registry can not provide it, dub can fall back to either:

  1. dumb behaviour (every time version statements are changed a new version of the library is built)

I actually think that this is sufficient. Usually you don't change versions all of the time (in isolation), but rather switch configurations or build types. If there is one binary per configuration/build type there is no need to be more clever. Even for tagged versions.

  1. Parse the code itself to gather the version/debug statements. (Bad at least because it would add another dependency to dub)
  2. Use rdmd. (We could for example simply require rdmd for branch builds)
  3. Do as C/C++ does, hope for the best (the version/debug command line changed do not affect the build). I don't really like that, but if dub would simply print a warning that it does not have this information and you have to take care, this might be fine for branch builds as the user always can fall back to rdmd.

Actually I started to think about this, because I had some problems building software with dub (dub seemed to not build the dependency at all, when using the default build method).

Can you go into more detail? I'd like to understand the problem.

In my quest for the reason, I started to think how I would do it, the last post was what I came up with ;-) -> So in essence this proposal was my concept of getting build semantics right, but to be honest I don't even understand exactly what your current approach is, maybe you could briefly explain what it looks like? And what the problems are?

  • How do you decide what to build? Simply all modules of all dependencies? Track them somehow via imports?

The former. With incomplete support to build dependencies as separate libraries.

  • If you decided what to build, how do you build it?
    • All source files passed at once to dmd? Does this scale? If not, how do you split up?

This is the case right now. How good it scales depends a lot on how CTFE/template heavy the code is, but you can definitely hit walls.

- One module at a time?

This

- Some combination?

and the aforementioned per-dependency builds are planned.

Also, when implementing this, I would make the whole thing a small separate build server that is controlled by a simple REST interface from the start, so that it can be a separate process with different user rights and on a different server (or multiple servers).

Sounds good. Btw, do you have the resources to host such a server? Or do you know someone who does?

I have one machine that would be suited, but that's in the internal network and I don't really feel comfortable using it for external stuff security wise. Running builds on the virtual server that runs the registry would probably hamper the execution of the other services considerably (esp. considering its mere 2GB of RAM). Maybe someone has a few spare bucks for an EC2 instance? I guess that could stay turned off most of the time and should come out quite cheap.

The question I have in mind is how official it will become to build with rdmd. With voices that IDEs should also use the external build tool to build a project and other voices that always want to build with --rdmd it sounds like we should possibly actually provide rdmd support for all kinds of projects and we could maybe even go back and base everything on it...

I finally need to look up how rdmd works in detail, I am just guessing at the moment.

The other issue with rdmd of course is that it currently is not actually sufficient as a general build tool, so all this would mean a partially dysfunctional package system until all issues and missing features of rdmd (or dmd) are sorted out.

Re: Handling of a main() containing file.

It's not good enough also for a number of other reasons, and the whole concept will also be difficult to get right. For example, suppose we have some dependencies build as dynamic libraries. How should rdmd or any dependency based build tool know where it has to stop when including D modules (i.e. at the border to the dynamic library)? It also needs to support separate compile/link and single file compilation modes for larger projects as a fallback in case of linker issues. Then there is the issue of the additional compiler run to initially determine all dependencies or recompile whenever compilation failed because new dependencies were added and the issue with missing local and string imports ind dmd's .dep files.

These are all reasons why I chose to not rely on rdmd/dependency tracking (and merely included it as a fallback option), but still there are a lot of opposing voices.

Hmm, it seems my assumption about the inner workings of rdmd were right? It just tracks down all needed modules via the imports, locates them and puts them all at the compiler command line? So basically any libraries are ignored? I think I will take the time and study the code a bit more.

Re: Handling of a main() containing file.

Am 20.03.2013 15:16, schrieb Robert:

It's not good enough also for a number of other reasons, and the whole concept will also be difficult to get right. For example, suppose we have some dependencies build as dynamic libraries. How should rdmd or any dependency based build tool know where it has to stop when including D modules (i.e. at the border to the dynamic library)? It also needs to support separate compile/link and single file compilation modes for larger projects as a fallback in case of linker issues. Then there is the issue of the additional compiler run to initially determine all dependencies or recompile whenever compilation failed because new dependencies were added and the issue with missing local and string imports ind dmd's .dep files.

These are all reasons why I chose to not rely on rdmd/dependency tracking (and merely included it as a fallback option), but still there are a lot of opposing voices.

Hmm, it seems my assumption about the inner workings of rdmd were right? It just tracks down all needed modules via the imports, locates them and puts them all at the compiler command line? So basically any libraries are ignored? I think I will take the time and study the code a bit more.

AFAIK that's correct.

Re: Handling of a main() containing file.

Interesting, I didn't know that. Is that standard behavior (i.e. does it work with Optlink, VisualStudio link.exe, the LLVM linker and maybe unilink)?

Well my knowledge about such things is highly Linux specific ( I never really used anything else ), wikipedia suggests in the overview section of:

http://en.wikipedia.org/wiki/Linker_%28computing%29

that some linkers do include the complete library in the output and not only the referenced modules. In this case two mains would probably conflict, but only if the linker does not consider the library symbols as weak symbols. I don't know whether there is some standard behaviour for linkers or where I would find such a specification. The easiest way would be to try out all major linkers/platforms and see whether it works everywhere. I believe Jacob has a Mac, you seem to have a Windows installation, maybe you could give it a try?

The test case I used: Provide two source files, one with some function, one with main, link them together in a static and in a dynamic library.
Write another two source files, one with main one without (does not matter really what it contains). Put some printf/cout/writeln statement in each function. Try to link the no-main file with the library, then the main-file. If both succeed, run them and check the output to see what main gets called. In the external main file you can reference the function in the non-main library file, but no symbol in the main defining file.

dub would not need to rely on this information, if the registry can not provide it, dub can fall back to either:

  1. dumb behaviour (every time version statements are changed a new version of the library is built)

I actually think that this is sufficient. Usually you don't change versions all of the time (in isolation), but rather switch configurations or build types. If there is one binary per configuration/build type there is no need to be more clever. Even for tagged versions.

We should definitely start this way, if it works well for most/all users no need to implement something more sophisticated, if something more sophisticated seems necessary it can be added any time.

Can you go into more detail? I'd like to understand the problem.

When I simply ran dub on the package (almost empty with dlibgit as its only dependency), it did not build the dependency, it only added its import paths, no source files. Using --rdmd fixed the problem. I need to track this down a bit more, I very soon drifted off to thinking about building in general and how to do it. It could easily have been my fault.

The former. With incomplete support to build dependencies as separate libraries.

-> Great, may I jump in and complete it, at least for Linux. Maybe some pointers where you started? generators/build.d? What is already done, what needs to be done next?

This is the case right now. How good it scales depends a lot on how CTFE/template heavy the code is, but you can definitely hit walls.

- One module at a time?

This

- Some combination?

and the aforementioned per-dependency builds are planned.

Could per dependency-library builds be /the/ way to do it, with dropping all others? Or are there reasons to support alternate methods? Libraries are of course no silver bullet, even a single package can be too much to compile at once, so the "one-module-at-a-time" option or some deterministic way of splitting things up would be needed anyway.

I have one machine that would be suited, but that's in the internal network and I don't really feel comfortable using it for external stuff security wise. Running builds on the virtual server that runs the registry would probably hamper the execution of the other services considerably (esp. considering its mere 2GB of RAM). Maybe someone has a few spare bucks for an EC2 instance? I guess that could stay turned off most of the time and should come out quite cheap.

I see, I believe we will find a solution, first I should have something implemented ;-)

The question I have in mind is how official it will become to build with rdmd. With voices that IDEs should also use the external build tool to build a project and other voices that always want to build with --rdmd it sounds like we should possibly actually provide rdmd support for all kinds of projects and we could maybe even go back and base everything on it...

Why do people care about --rdmd or not? How does it affect them, as long as it works correctly?

The other issue with rdmd of course is that it currently is not actually sufficient as a general build tool, so all this would mean a partially dysfunctional package system until all issues and missing features of rdmd (or dmd) are sorted out.

Wasn't the purpose of rdmd scripting? I myself, would never had thought of using it for building whole projects+dependencies.

Best regards,
Robert

Pages: 1 2