RejectedSoftware Forums

Sign up

Pages: 1 2

Handling of a main() containing file.

Discussion moved from a pull request:

Regarding the app.d problematic, I feel we need to start a more thorough discussion first. As you have probably seen, that "appFile" field would basically be equivalent to adding that file as an "excludedFiles" setting in all non-application configurations, except when --rdmd is used. And --rdmd is still not working when bulding a static or dynamic lib, so that's a related problem ("mainFile"/"rootFile"?). And then there is still the thing with vibe.d's -version=VibeCustomMain that I would like to improve if possible and that may also interact with the descision how to handle the app file. Let's open a forum topic, so it doesn't get lost here.

Exactly any library target would exclude it, the executable target would use it.

I intended to change the look up for the default locations "src/app.d" and "source/app.d" to search it in any specified sourcePath, but I just realized that this is pretty bad, for the not uncommon configuration that there is no source/src folder but instead immediately the D package tree. This is especially common for packages that adhere to the preferred behaviour of just exporting a single package. An example can be found here:

https://github.com/jacob-carlborg/orange

Putting orange in src, would just add another folder into the hierarchy. The current dub design offers two possibilities to handle this case:

  1. Add the project directory itself as a source directory ("."). The problem is that dub traverses the whole tree it finds in the search for source files, including the .dub folder with all the dependencies. So this approach does not work.
  2. Second add the orange directory as source path and the current directory as an import path. Resulting in an invalid import path: orange/ to be used, but this should not do any real harm.

So as only 2. really works, searching for the app.d file in all source paths, would be bad, because there might be an orange.app file which really is not /the/ app.d we are looking for.

1 could be improved, by having dub ignore files/directories starting with a dot, but I am not sure whether this is enough. Another possibility would be to offer an "includePackages" option. If specified only the listed directories would be copied/linked over to an import directory (as explained in my proposal "D package = package"), which would then be added as import path instead. This would solve both problems of cluttered source directories and packages put in the project's root.

For libraries like vibe, which provide their own main() function. I would suggest adding a libraryMainFile configuration to the package.json. dub will then only link it in, if the application being built has no explicitly defined mainFile.

So we would have:

  • mainFile
  • appFile
  • libraryMainFile

appFile being like mainFile (only link it in when building an application) but it does not specify its own main and dub would scream out loud if no library defines libraryMainFile if an executable is being built (or alternatively rely on the compiler to do this). If mainFile is specified any libraryMainFile would not be linked in.

For an orange-like configuration appFile/mainFile would be mandatory as you would not be able to rely on auto detection of the app/main file.

I also thought of making it somehow possible to disable the auto detection of the app.d file, but as top level modules should really be discouraged I don't see any real reason to do this.

Best regards,
Robert

Re: Handling of a main() containing file.

If multiple libraries you depend on define libraryMainFile then you would also have to provide your own mainFile to resolve the conflict.

Re: Handling of a main() containing file.

Am 16.03.2013 10:48, schrieb Robert:> Discussion moved from a pull request:

Regarding the app.d problematic, I feel we need to start a more thorough discussion first. As you have probably seen, that "appFile" field would basically be equivalent to adding that file as an "excludedFiles" setting in all non-application configurations, except when --rdmd is used. And --rdmd is still not working when bulding a static or dynamic lib, so that's a related problem ("mainFile"/"rootFile"?). And then there is still the thing with vibe.d's -version=VibeCustomMain that I would like to improve if possible and that may also interact with the descision how to handle the app file. Let's open a forum topic, so it doesn't get lost here.

Exactly any library target would exclude it, the executable target would use it.

I intended to change the look up for the default locations "src/app.d" and "source/app.d" to search it in any specified sourcePath, but I just realized that this is pretty bad, for the not uncommon configuration that there is no source/src folder but instead immediately the D package tree. This is especially common for packages that adhere to the preferred behaviour of just exporting a single package. An example can be found here:

https://github.com/jacob-carlborg/orange

Putting orange in src, would just add another folder into the hierarchy. The current dub design offers two possibilities to handle this case:

  1. Add the project directory itself as a source directory ("."). The problem is that dub traverses the whole tree it finds in the search for source files, including the .dub folder with all the dependencies. So this approach does not work.
  2. Second add the orange directory as source path and the current directory as an import path. Resulting in an invalid import path: orange/ to be used, but this should not do any real harm.

Number 2 should in fact not add an invalid import path. The import path is supposed to be only added if "source"/"src" was auto-detected as the main source folder. But manually specifying "sourcePaths" really just specifies source paths and doesn't make them import paths.

So as only 2. really works, searching for the app.d file in all source paths, would be bad, because there might be an orange.app file which really is not /the/ app.d we are looking for.

The whole app.d/<package name>.d file feature is really just supposed to be a short-cut for apps that want to use that application/library split convention. See ddox for an example that doesn't rely on the automatic detection.

1 could be improved, by having dub ignore files/directories starting with a dot, but I am not sure whether this is enough. Another possibility would be to offer an "includePackages" option. If specified only the listed directories would be copied/linked over to an import directory (as explained in my proposal "D package = package"), which would then be added as import path instead. This would solve both problems of cluttered source directories and packages put in the project's root.

Copying around source files sounds like it may create problems, especially with IDE based builds. I think having a few straw import folders is OK, as this kind of structure is not the recommended way and also without DUB the library has the same issue (except if it has a make file that first copies all files to somewhere else). No need to additionally support that kind of structure IMHO.

For libraries like vibe, which provide their own main() function. I would suggest adding a libraryMainFile configuration to the package.json. dub will then only link it in, if the application being built has no explicitly defined mainFile.

The problem is that it may need the vibe.d main() even if the application also has an app.d (that only contains a static this). In the old days, rdmd used to resolve that problem purely by tracking the actual dependencies. Now that needs to be done manually, as this is no option for IDE builds. (If all IDEs would agree to use rdmd for building then it would of course be a totally different thing)

So we would have:

  • mainFile
  • appFile
  • libraryMainFile

appFile being like mainFile (only link it in when building an application) but it does not specify its own main and dub would scream out loud if no library defines libraryMainFile if an executable is being built (or alternatively rely on the compiler to do this). If mainFile is specified any libraryMainFile would not be linked in.

For an orange-like configuration appFile/mainFile would be mandatory as you would not be able to rely on auto detection of the app/main file.

I also thought of making it somehow possible to disable the auto detection of the app.d file, but as top level modules should really be discouraged I don't see any real reason to do this.

I think only one "mainFile" field is needed both, for applications and libraries, and only for "dub --rdmd" so that rdmd has something to start. The appFile/libraryMainFile fields would just be another shortcut for functionality that is already there (using configurations and "excludeFiles") and I don't think this is necessary.

Note that it is one of the new features in 0.9.11 that the auto-detection really just pre-fills one or two configurations ("application" and "library") automatically, depending on which files are found.

Re: Handling of a main() containing file.

Am 16.03.2013 11:38, schrieb Robert:

If multiple libraries you depend on define libraryMainFile then you would also have to provide your own mainFile to resolve the conflict.

If the libraries are built as source libraries, their main file is
irrelevant, only the one of the main project is needed to have something
for rdmd. If on the other hand they are static/dynamic libraries, their
main files will be used individually to build each of them - no
conflicts there.

Re: Handling of a main() containing file.

Let me quickly explain what really is my problem with VibeCustomMain. There are two conflicting goals which were not conflicting when only rdmd was used:

  1. The whole static this {} thing for vibe.d is meant foremost for small projects and also especially for developers coming from scripting languages. They frequently seem to think that not having an explicit main loop or main function is some serious advantage and this basically was the answer to that. It also helps to hide a small design problem that still exists WRT vibedist, but that's a different topic.

    So in this scenario, using static this instead of main() should mean that absolutely no configuration is necessary, because it is especially intended for small projects and beginners.

  2. Now that vibe.d is just another library in the DUB context, point 1 suddenly forces everyone who wants to use it merely as a library to put a -version=CustomMain in the project. Care must be taken to not put that version in a "library" configuration, as it is applied to all projects that depend on this project, too, and if those actually want to use only static this it would fail to link with no main() defined. (That's also the reason why the ddox projects manually defines its configurations).

In essence, it would be nice if both cases would work without configuration, like they used to in the pure rdmd days by importing either vibe.d or vibe.vibe. Preferably this should work without looking at the source file, especially without compiling it to look for dependencies, and without introducing something more or less hard-coded for vibe.d.

Re: Handling of a main() containing file.

Number 2 should in fact not add an invalid import path. The import path is supposed to be only added if "source"/"src" was auto-detected as the main source folder. But manually specifying "sourcePaths" really just specifies source paths and doesn't make them import paths.

Ah ok, great! Sorry I missed that.

Copying around source files sounds like it may create problems, especially with IDE based builds. I think having a few straw import folders is OK, as this kind of structure is not the recommended way and also without DUB the library has the same issue (except if it has a make file that first copies all files to somewhere else). No need to additionally support that kind of structure IMHO.

Makes sense. What if I have my source files in src/source is it possible to use them as source directory but not as import directory? Is there a way to override the default? Should we probably drop a default setting if it is configured explicitly?

The problem is that it may need the vibe.d main() even if the application also has an app.d (that only contains a static this). In the old days, rdmd used to resolve that problem purely by tracking the actual dependencies. Now that needs to be done manually, as this is no option for IDE builds. (If all IDEs would agree to use rdmd for building then it would of course be a totally different thing)

I see (also thanks for explaining how vibe used to work). This means if we handle dependencies by simply creating libraries static/dynamic and linking to them (so the linker will only use vibe's main if the user has not defined one) or alternatively relying on rdmd, the vibe's main problem is solved? rdmd builds would simply only work if there is an app.d in src/source, so dub could simply provide an error message if --rdmd is used but no app.d is found, sounds ok to me.

I am currently thinking about handling dependencies by simply building a static/dynamic library for each package and link to it. I think this would be the most straight forward and efficient approach, but as you already might have noticed in the "conflict" discussion, I am very much in favour for making things as robust as possible. I want things to just work for the user and not having to handle subtle errors or unexpected behaviour. That is why I would put some work in having dub and the registry do what it can to make sure that what you are linking to, matches the definitions the application sees at compile time.

This means, at the compilation of the application the same -version and -debug arguments should be used as they were used when building the library. (Is there something else to be considered?) Of course not every -version/-debug argument really affects the library, this is why I would like to implement the following in the registry:

  • When a new version of a package is released, the registry will download it and extract it.
  • The modules in the package will be recorded in the database as belonging to the package
  • HTML documentation is generated
  • The code gets searched for any version() and debug() statements, those will be listed online so the user knows what debugging/version options the library has. In addition dub can use this information to know what version/debug statements are actually going to affect the build, so it only needs to provide a new build of a library on actually relevant changed settings.

For retrieving the version/debug information I intend using one of the D parsers, that people have already written. It might not be 100% reliable, because of string mixins, but as the use of version() is seldom enough, its use within string mixins should be even more rare. A simple statement in a dub/registry tutorial/documentation/guideline that those would not be handled correctly should be enough.

What do you think? What are your plans?

For the problem of this thread (and vibe's main): Simply rely on linker or rdmd -> No need to specify the main/app file. For the rdmd build we simply assume it is app.d, if not, you don't use the --rdmd build?

Best regards,

Robert

Re: Handling of a main() containing file.

If the libraries are built as source libraries, their main file is
irrelevant, only the one of the main project is needed to have something
for rdmd. If on the other hand they are static/dynamic libraries, their
main files will be used individually to build each of them - no
conflicts there.

My thinking was if it is a source library, dub should not provide the main-containing file of vibe to the compiler if the application provides its own, but it should do that if the application does not provide its own main. But you are right, if we simply build static/dynamic libraries and use them, the linker takes care of that automatically.

Re: Handling of a main() containing file.

On Sun, 17 Mar 2013 21:01:15 GMT, Robert wrote:

If the libraries are built as source libraries, their main file is
irrelevant, only the one of the main project is needed to have something
for rdmd. If on the other hand they are static/dynamic libraries, their
main files will be used individually to build each of them - no
conflicts there.

My thinking was if it is a source library, dub should not provide the main-containing file of vibe to the compiler if the application provides its own, but it should do that if the application does not provide its own main. But you are right, if we simply build static/dynamic libraries and use them, the linker takes care of that automatically.

I think this is a misunderstanding of what we both thought what a "main file" is (containing main() vs. being the root of the dependency hierarchy). So in particular the difficult problem is not to know if an application has a "main file", but if that file actually contains main() or not.

Re: Handling of a main() containing file.

Am 17.03.2013 21:56, schrieb Robert:>

Number 2 should in fact not add an invalid import path. The import path is supposed to be only added if "source"/"src" was auto-detected as the main source folder. But manually specifying "sourcePaths" really just specifies source paths and doesn't make them import paths.

Ah ok, great! Sorry I missed that.

Copying around source files sounds like it may create problems, especially with IDE based builds. I think having a few straw import folders is OK, as this kind of structure is not the recommended way and also without DUB the library has the same issue (except if it has a make file that first copies all files to somewhere else). No need to additionally support that kind of structure IMHO.

Makes sense. What if I have my source files in src/source is it possible to use them as source directory but not as import directory? Is there a way to override the default? Should we probably drop a default setting if it is configured explicitly?

Yes, so if a custom "configurations" section or an own "sourcePaths" definition are specified, they will override the default choices. See package_.d line 101 ff. It first searches for all standard folder names and adds them, then parses the package.json file to let the user override the settings, and finally generates default configurations if none have been specified.

The problem is that it may need the vibe.d main() even if the application also has an app.d (that only contains a static this). In the old days, rdmd used to resolve that problem purely by tracking the actual dependencies. Now that needs to be done manually, as this is no option for IDE builds. (If all IDEs would agree to use rdmd for building then it would of course be a totally different thing)

I see (also thanks for explaining how vibe used to work). This means if we handle dependencies by simply creating libraries static/dynamic and linking to them (so the linker will only use vibe's main if the user has not defined one) or alternatively relying on rdmd, the vibe's main problem is solved? rdmd builds would simply only work if there is an app.d in src/source, so dub could simply provide an error message if --rdmd is used but no app.d is found, sounds ok to me.

I don't see how the static/dynamic libraries would solve that. If vibe.d would be compiled as a dynamic library, the linker should not consider it's main function at all and if compiled as a static library it should produce conflicting definition errors as in the source library case. It would be great if the linker could somehow be employed for this, but I don't see how yet.

I am currently thinking about handling dependencies by simply building a static/dynamic library for each package and link to it. I think this would be the most straight forward and efficient approach, but as you already might have noticed in the "conflict" discussion, I am very much in favour for making things as robust as possible. I want things to just work for the user and not having to handle subtle errors or unexpected behaviour. That is why I would put some work in having dub and the registry do what it can to make sure that what you are linking to, matches the definitions the application sees at compile time.

This means, at the compilation of the application the same -version and -debug arguments should be used as they were used when building the library. (Is there something else to be considered?) Of course not every -version/-debug argument really affects the library, this is why I would like to implement the following in the registry:

  • When a new version of a package is released, the registry will download it and extract it.
  • The modules in the package will be recorded in the database as belonging to the package
  • HTML documentation is generated
  • The code gets searched for any version() and debug() statements, those will be listed online so the user knows what debugging/version options the library has. In addition dub can use this information to know what version/debug statements are actually going to affect the build, so it only needs to provide a new build of a library on actually relevant changed settings.

For retrieving the version/debug information I intend using one of the D parsers, that people have already written. It might not be 100% reliable, because of string mixins, but as the use of version() is seldom enough, its use within string mixins should be even more rare. A simple statement in a dub/registry tutorial/documentation/guideline that those would not be handled correctly should be enough.

What do you think? What are your plans?

I think the approach with handling modules and HTML documentation and also the version statements is a good idea. But I think we shouldn't rely on that information for the build, if just because we may not be able to continually generate this information for branches if there are many active projects. Avoiding dependencies to rebuild also sounds like a low-priority issue, so as long as we don't introduce a road block for that somewhere, I think we can get to that once everything else is settled. Let's first get the build semantics right (...and some important stuff that has nothing to do with semantics at all).

Also, when implementing this, I would make the whole thing a small separate build server that is controlled by a simple REST interface from the start, so that it can be a separate process with different user rights and on a different server (or multiple servers).

For the problem of this thread (and vibe's main): Simply rely on linker or rdmd -> No need to specify the main/app file. For the rdmd build we simply assume it is app.d, if not, you don't use the --rdmd build?

The question I have in mind is how official it will become to build with rdmd. With voices that IDEs should also use the external build tool to build a project and other voices that always want to build with --rdmd it sounds like we should possibly actually provide rdmd support for all kinds of projects and we could maybe even go back and base everything on it...

Then of course it still has some severe bugs because of dmd's lacking dependency output (string imports and local imports are not tracked).

...actually I think the whole rdmd functionality should be part of DMD itself, as it could collect the needed dependencies during compilation instead of doing a separate compilation pass just for that, but even if DMD already does everything else (docs, coverage, profiling) this idea seems to produce no consent.

Re: Handling of a main() containing file.

On Mon, 18 Mar 2013 06:39:22 GMT, Sönke Ludwig wrote:

On Sun, 17 Mar 2013 21:01:15 GMT, Robert wrote:

My thinking was if it is a source library, dub should not provide the main-containing file of vibe to the compiler if the application provides its own, but it should do that if the application does not provide its own main. But you are right, if we simply build static/dynamic libraries and use them, the linker takes care of that automatically.

I think this is a misunderstanding of what we both thought what a "main file" is (containing main() vs. being the root of the dependency hierarchy). So in particular the difficult problem is not to know if an application has a "main file", but if that file actually contains main() or not.

You cannot rely on a library having a "main file" or having a file containing "main()". It's possible and a perfectly valid use case to have a library containing only two files that don't import each other. Doesn't Dub just builds everything in the source directory? Actually, what Dub should do is pass all files in the source directory to Rdmd (or similar) to get all dependencies of all files, then compile all those files. Of course, in my opinion, that should be handled by a proper build tool and not the package manager.

Pages: 1 2