Am 13.12.2013 18:37, schrieb Dicebot:> On Thu, 12 Dec 2013 17:32:53 +0100, Sönke Ludwig wrote:

What exactly were the issues where it doesn't scale?

There were some issues related to file layout - it didn't like our default which implies several different "main" files in same subfolder with rest of source mostly shared, tried to compile all despite different configuration with different target files used. Most likely a bug I will have a look at if try this again at some point.

You need to either explicitly exclude the other main files using "excludedSourceFiles" in each configuration, or selectively include only the right files for each configuration/sub package. It's of course possible that there is a bug, but at least there are some existing projects using that structure.

BTW a lot of such issues result from fitting legacy structures into DUB packages (which is supposed to work, of course!). But once projects get properly separated into packages with a clean application/library distinction, things work much more smoothly -- in this particular case with the added benefit that you can do stupid things like import tool2main; from tool1.

But actual scalability issues pretty much come from simplicity of format used. Being unable to define meta-variables resulted in lot of added duplication over original makefiles (there were several lists used in different contexts).

That's a good point and once a better format is supported, that can be easily addressed. (even if the need for this is relatively rare compared to most other systems because of the orthogonal build types, platform settings, and configurations)

Typical makefile dependency chain (use dstep to generate module from c file which is generated by external tool from spec file) simply did not map at all - only option was to use a custom pre-build script but than generated files did not get into dub describe and were not seen in Mono-D project.

The generated files need to be explicitly specified using "sourceFiles". I have an open ticket for that: https://github.com/rejectedsoftware/dub/issues/144 (but I think that there was actually some kind of problem with supporting that, not sure)

I also don't like added amount of "includeXXX"/"excludeYYY"-style directives which makes it rather hard to follow project description with many configurations.

You mean "sourceFiles" and "excludedSourceFiles"? "excludedSourceFiles" could surely be dropped, but then (additively) specifying the list of files may be much more verbose in some situations. But supporting glob expressions for "sourceFiles" is also planned.

Remember that the build system is still in its infancy and hasn't
received much work. But everything else should work fine in projects of
any size.

Well, its infancy is exactly the chance to figure out what is missing here and transition to more powerful system :)

The build system (or "build generator") is in its infancy, not the concept ;)

The top missing things are separate compilation of separate packages, partial (re)compilation and supporting a lot of little customizations that might be necessary to get around various bugs/limitations of the current compiler implementation.

My initial proposal to separate package.json was because package management looks mostly fine right now and are unlikely to undergo any major changes (well, apart from dub CLI itself, I'd love to rewrite it completely some years later :P) - build part does not look that smooth.

Do you mean the infant "build generator" which doesn't look nice?

If you really just mean the CLI, you should take a look at the current git master (source/dub/commandline.d). It has been rewritten to be much more modular and extensible.

The problem is that the more generalized you make the system the more
specialized will the "build descriptions" become - like Heisenberg's
uncertainty principle ;). At some point you lose the "just works" and
have to adjust everything for every dependency and every
platform/whatever. Also you'll frequently discover packages which don't
support this or that kind of build (debug build, or building as shared
library, or 64-bit). My goal is to find a sweet spot that fits and has
advantages for >90% of the projects, but still allows to handle the rest
of the projects one way or another.

I don't think that fitting everyone into overly simple base is a good way to address it. Much better way is to actually abuse dub registry with some CI on top to get the information about platforms and configurations supported, probably try to motivate authors by assigning some kind of "rating" to submitted projects that conform desired look&feel well. There will always be some project that won't fit well into constraints and still be potentially useful to someone (and of course, I don't think I am the only one who had an idea of using local dub registry to handle internal company environment).

"overly simple" is highly subjective. I'd argue that it is very sufficient for the vast majority of cases. For anything else it allows to invoke external tools. Things can still be improved there, but I don't see anything limiting in general.

It is a very difficult task to define system that is both flexible for actual usage and constrained enough to allow dub to infer necessary metadata from it, sure. But unless it is proven otherwise, I dont' think it is impossible. Now there are quite lot of projects submitted to gather the information about and use at as input to brainstorm better system. If requirements can be somewhat formally defined, I am pretty sure thread asking for ideas in that domain will have a huge success in D newsgroup :)

That it's not "flexible for actual usage" might be a slight exaggeration, or do you mean that it currently cannot infer metadata? ;)

For one, changing anything to be more like "make" will instantly lose the ability to generate VisualD projects.

Brainstorming improvements is always nice*, but they should stay backwards compatible. Changing the whole philosophy of the project at this point would be a disaster (and I still think it's exactly the right choice).

* but I really can't participate in a larger discussion now - it's really bad, I barely have time to breathe

If a project has a structure that doesn't really fit, on one hand you
could argue that it likely could be simplified by splitting it up into
smaller units. On the other hand you can also always do things like
making a wrapper DUB package that just defines import paths and import
libraries, but invokes "make" or another build system from the pre-build
or pre-generate commands to actually build the package.

Yep, but I consider dub describe and its nice integration with IDE's a very important feature to lose.

You only have to use it for the parts that don't fit (e.g. running DStep). And even if you completely switch to a separate build tool, you can still have the IDE display all source files and so on, if the "importPaths" (and possibly other fields) are properly set up.