RejectedSoftware Forums

Sign up

Pages: 1 2 3 4

Should packages be restricted to a single name space?

So over the last days we have had a long discussion initiated by Robert Klotzner about how a package manager for D should look like. One of the controversial topics was if each package should be restricted to have all source files inside of a single D package. The idea is to allow no conflicting packages to be created, so that all public packages are guaranteed to not cause conflicts during compilation.

I think this approach has some obvious merits to it and would facilitate a very clean way of laying things out as packages. I would also happily adjust my packages to such a system. However, I still struggle with the idea because of some issues:

  1. Existing projects would often violate this constraint and would have to be restructured in one way or another. I think that this might drive some people away and some important libraries may in turn not be available as packages.

  2. It hampers forking projects or crafting alternative implementations since all modules need to be renamed first before publishing as a package. It would also mean that all packages that now depend on the forked package instead of the original need to be adjusted accordingly. This could be mitigated to some degree by meta packages that switch to different implementations conditionally, but those would also need to be maintained by someone and, if they should have a pretty name, it would imply that their existence was planned already before the first original library appeared. Furthermore it would mean that it's author would be the one who solely decides which implementations get supported.

  3. One idea was to force the name of a package to be equal to the root name space for the package (e.g. a "derelict.gl" package would only contain modules below derelict.gl). There are some precedents where the name differs, so this would be another case of 1. It also would require to rename a number of already existing packages. (As a technical detail, dots are not allowed in package names, but that can be resolved in one way or another)

  4. It would considerably increase the implementation complexity of the registry if it is supposed to be enforced reliably. It would also require more education for people who want to create packages and thus also increase the perceived complexity of the system.

  5. Orphaned packages would clutter up the available name space and some kind of manual intervention would be required to remove those packages. This may be a difficult judgement call as people might still be using those packages (except, if the replacement is supposed to be 100% compatible). Similarly, the first who gets a name space, owns it. Should someone register "derelict" or "deimos" and use it for something else, who decides what happens?

  6. The strict enforcement breaks down with multiple registries as they have no way to know which name spaces are already in use on other registries. This may not be much of an issue though, as long as there is only one main public registry.

  7. There may still be conflicts, for example for string imports or runtime resources. However, these conflicts might even be desirable (e.g. overriding of Diet templates).

How much weight each point has compared to the benefits of this approach varies a lot with personal preferences and available work force. My main concern is number 1. It would be great to get some more opionions on this, especially from people affected by 1 and also to look at how many popular projects would actually have to be restructured.

Re: Should packages be restricted to a single name space?

On Sat, 09 Mar 2013 12:11:07 GMT, Sönke Ludwig wrote:

So over the last days we have had a long discussion initiated by Robert Klotzner about how a package manager for D should look like. One of the controversial topics was if each package should be restricted to have all source files inside of a single D package. The idea is to allow no conflicting packages to be created, so that all public packages are guaranteed to not cause conflicts during compilation.

It would be great to get some more opionions on this, especially from people affected by 1 and also to look at how many popular projects would actually have to be restructured.

While I definitely think that it would be good practice for projects to use a single package hierarchy, I don't think that it's a good idea to require it. D generally takes the approach of encouraging the safe path while not stopping you from going the unsafe one if you feel that you need to. Granted, this isn't a discussin on @safe, but I think that the same, basic principle applies. Good pratice should be encouraged but not enforced, because sometimes it's not the best way to go.

And think about this: Phobos currently has both the std and etc packages. If it were ever to switch to dub (which I wouldn't expect it to, but if dub were ever to become D's official package manager, maybe it would), it would be screwed by this suggestion. And while it may be better in general to put all of the code for a library in a package named after that library, why should I have to create two separate projects to be able to have two separate package hierarchies? Maybe I want to put two related package hierarchies in the same project but have them be separate (and on that count, it should be possible to create multiple libraries from the same dub package as some projects will need that). As long as the names are unique enough, it shouldn't be a problem at all. And if two projects end up with conflicting packages, then that can be sorted out when it happens. It's unlikely to be common anyway, particularly as most projects should and probably will have a base package named after the project.

Another thought. What if I had two libraries that had the same API but just had different implementations? I could then have two projects with exactly the same package hierarchy and swap one out for the other simply by changing the dub configuration file for my project (or even base it on the configuration - though depending on how dub installs stuff, that may not work). There are other ways to handle that which may be cleaner (e.g. using different version identifiers), but that might be something that someone would want to do. In particular, if a project got forked, the fork may want to keep exactly the same package layout in order to make it easy for other projects to switch over to it, whereas if they were forced to have a base package named after the project, that wouldn't work.

But really, I just think that it's a bad idea to create artificial restrictions on package layout. Sure, there are certain things that would be best practice and should be done normally (e.g. having a single node for a library's package hierarchy, and have that base package be named after the library), but I think that enforcing that is going too far.

Re: Should packages be restricted to a single name space?

I've just discovered that the "package name" equals "name space" convention would actually be quite painful for what I intended to do with vibe.d. That is, I want to split it up several sub packages (the actual split is not yet decided). I really wouldn't want to have to split them up into one package per vibe.* sub package, not even if it is possible to define multiple sub packages within the same repository and directory hierarchy. That's just not the right level to split things up. And the alternative would be to break all dependent code and restructure everything (e.g. for vibe-core: vibe.core, vibe.core.inet, vibe.core.utils), which is also not the intention.

Example split:

vibe-d -> vibe-http, vibe-mail, vibe-db
 - vibe.vibe
 - vibe.d

vibe-core
 - vibe.core.*
 - vibe.inet.*
 - vibe.utils.*

vibe-ext -> vibe-core
 - vibe.data.*
 - vibe.stream.*
 - vibe.templ.*
 - vibe.textfilter.*

vibe-http -> vibe-ext
 - vibe-http.*

vibe-db -> vibe-ext
 - vibe.db.*

vibe-mail -> vibe-core
 - vibe.mail.*

Re: Should packages be restricted to a single name space?

On Sat, 09 Mar 2013 20:10:12 GMT, Jonathan M Davis wrote:

While I definitely think that it would be good practice for projects to use a single package hierarchy, I don't think that it's a good idea to require it. D generally takes the approach of encouraging the safe path while not stopping you from going the unsafe one if you feel that you need to. Granted, this isn't a discussin on @safe, but I think that the same, basic principle applies. Good pratice should be encouraged but not enforced, because sometimes it's not the best way to go.

In some cases I think it can be good to have some kind of enforcement, but I tend to agree here - it has a lot of implications and I'm still not sure of the actual practical benefit (maybe a conflict indeed never occurs because everyone is careful enough, or libraries that conflict are never actually used together, or only one of the conflicting packages is used at all).

And think about this: Phobos currently has both the std and etc packages. If it were ever to switch to dub (which I wouldn't expect it to, but if dub were ever to become D's official package manager, maybe it would), it would be screwed by this suggestion. And while it may be better in general to put all of the code for a library in a package named after that library, why should I have to create two separate projects to be able to have two separate package hierarchies? Maybe I want to put two related package hierarchies in the same project but have them be separate (and on that count, it should be possible to create multiple libraries from the same dub package as some projects will need that). As long as the names are unique enough, it shouldn't be a problem at all. And if two projects end up with conflicting packages, then that can be sorted out when it happens. It's unlikely to be common anyway, particularly as most projects should and probably will have a base package named after the project.

I think this case could be handled by Robert's proposal of multiple sub packages inside of a single package description. He wanted to get something up on a wiki shortly.

Another thought. What if I had two libraries that had the same API but just had different implementations? I could then have two projects with exactly the same package hierarchy and swap one out for the other simply by changing the dub configuration file for my project (or even base it on the configuration - though depending on how dub installs stuff, that may not work). There are other ways to handle that which may be cleaner (e.g. using different version identifiers), but that might be something that someone would want to do. In particular, if a project got forked, the fork may want to keep exactly the same package layout in order to make it easy for other projects to switch over to it, whereas if they were forced to have a base package named after the project, that wouldn't work.

Agree, that was also one of my concerns (2.).

But really, I just think that it's a bad idea to create artificial restrictions on package layout. Sure, there are certain things that would be best practice and should be done normally (e.g. having a single node for a library's package hierarchy, and have that base package be named after the library), but I think that enforcing that is going too far.

Re: Should packages be restricted to a single name space?

On Sun, 10 Mar 2013 08:16:44 GMT, Sönke Ludwig wrote:

I've just discovered that the "package name" equals "name space" convention would actually be quite painful for what I intended to do with vibe.d. That is, I want to split it up several sub packages (the actual split is not yet decided). I really wouldn't want to have to split them up into one package per vibe.* sub package, not even if it is possible to define multiple sub packages within the same repository and directory hierarchy. That's just not the right level to split things up. And the alternative would be to break all dependent code and restructure everything (e.g. for vibe-core: vibe.core, vibe.core.inet, vibe.core.utils), which is also not the intention.

Example split:

vibe-d -> vibe-http, vibe-mail, vibe-db
 - vibe.vibe
 - vibe.d

vibe-core
 - vibe.core.*
 - vibe.inet.*
 - vibe.utils.*

vibe-ext -> vibe-core
 - vibe.data.*
 - vibe.stream.*
 - vibe.templ.*
 - vibe.textfilter.*

vibe-http -> vibe-ext
 - vibe-http.*

vibe-db -> vibe-ext
 - vibe.db.*

vibe-mail -> vibe-core
 - vibe.mail.*

Hmm, first of all you don't have to create a subpackage for each sub-D-package in vibe, you can have some be included in the vibe package and some being extensions. Second, to take vibe-ext as an example. You have an awesome JSON framework in vibe.data, much better than the one in phobos. If I wanted to use it, I would have to pull in also vibe.stream, vibe.templ? Even if I don't do any vibe based website? To me it seems that the split up per D-package is also in this example a good idea. On the other hand having something like a taskset "vibe-ext" which pulls in a number of sub-modules might of course be good idea and could easily be achieved without even a single extension to my design (which still needs to be published), just create an empty vibe.ext package, which just depends on the corresponding sub packages. Simply put a sub-package in your package.json named "vibe.ext" without a vibe/ext folder actually existing so the package would be empty but you can specify arbitrary dependencies to be pulled in. This way you get the desired grouping, but with the additional benefit that the user can actually also just use a subset you might not even have thought of to be useful.

So no restructuring needed, it would just work.

Best regards,

Robert

Re: Should packages be restricted to a single name space?

On Sun, 10 Mar 2013 10:03:21 GMT, Robert wrote:

Hmm, first of all you don't have to create a subpackage for each sub-D-package in vibe, you can have some be included in the vibe package and some being extensions.

In that case I dont' see how anything is to be enforced here? The only thing would be that e.g. vibe-core and vibe-ext both share a common prefix with their D packages, but technically the name now tells nothing about conflicts anymore.

Second, to take vibe-ext as an example. You have an awesome JSON framework in vibe.data, much better than the one in phobos. If I wanted to use it, I would have to pull in also vibe.stream, vibe.templ? Even if I don't do any vibe based website? To me it seems that the split up per D-package is also in this example a good idea.

As said, this was just an example that I quickly put together. However, the thing with all the vibe-ext packages is that I would like to have them completely separated out at some point (the JSON/BSON stuff would actually great to have in Phobos), but don't want to break backwards compatibility now. So at some point they would not even live in the vibe.* namespace - to me it doesn't make much sense to split this up now and create the impression that those are good to use outside of the vibe context - they are, but only after they have been properly extracted.

Note that this kind of fine-grained splitting of packages would have been a PITA back when vibe started and no package manager was available, which is why all the common functionality today is concentrated in vibe.d itself. If one had been available, I would have separated them out right from the start.

On the other hand having something like a taskset "vibe-ext" which pulls in a number of sub-modules might of course be good idea and could easily be achieved without even a single extension to my design (which still needs to be published), just create an empty vibe.ext package, which just depends on the corresponding sub packages. Simply put a sub-package in your package.json named "vibe.ext" without a vibe/ext folder actually existing so the package would be empty but you can specify arbitrary dependencies to be pulled in. This way you get the desired grouping, but with the additional benefit that the user can actually also just use a subset you might not even have thought of to be useful.

As said in my mail, this public sub-package concept adds some complexity and I'm really not sure about the value it actually gains. It's quite possible that I'm wrong on that, but up to now I still can't see the compelling argument. I also fear that the registry might end up with a plethora (exaggerated) of small sub-packages that have no actual use by themselves (e.g. I may be wrong, but I think nobody would use those org.* DWT sub-packages in isolation, but always as a whole, no point in making them all so prominent in the registry).

So no restructuring needed, it would just work.

I'm looking forward to the new proposal, as I obviously got a bit behind with your latest ideas. I'll have time in the evening to read through it properly.

Re: Should packages be restricted to a single name space?

In that case I dont' see how anything is to be enforced here? The only thing would be that e.g. vibe-core and vibe-ext both share a common prefix with their D packages, but technically the name now tells nothing about conflicts anymore.

Let me rephrase that: I don't see what of the discussed conventions is actually left. As far as I see I would violate them, except for vibe-http, vibe-mail and vibe-db.

Re: Should packages be restricted to a single name space?

On Sun, 10 Mar 2013 10:39:39 GMT, Sönke Ludwig wrote:

On Sun, 10 Mar 2013 10:03:21 GMT, Robert wrote:

Hmm, first of all you don't have to create a subpackage for each sub-D-package in vibe, you can have some be included in the vibe package and some being extensions.

In that case I dont' see how anything is to be enforced here? The only thing would be that e.g. vibe-core and vibe-ext both share a common prefix with their D packages, but technically the name now tells nothing about conflicts anymore.

It does, first of all you are only allowed to used vibe.core because it is specified in the vibe package to be a sub-package, so the vibe package explicitly gave away this part of its namespace (if you created a vibe/core directory later it would be part of the vibe.core package not vibe). Second any package may pull in dependencies, regardless whether it is empty or not, this does not prevent you from looking up in the registry whether a particular name is already taken. The point is, that it suffices to look at available packages and you know what name spaces are already claimed. Also a user won't get surprised because some package he uses, exports an completely unrelated (even undocumented) D-package. There is no problem if a package claims a name space that is effectively not used.

Second, to take vibe-ext as an example. You have an awesome JSON framework in vibe.data, much better than the one in phobos. If I wanted to use it, I would have to pull in also vibe.stream, vibe.templ? Even if I don't do any vibe based website? To me it seems that the split up per D-package is also in this example a good idea.

As said, this was just an example that I quickly put together. However, the thing with all the vibe-ext packages is that I would like to have them completely separated out at some point (the JSON/BSON stuff would actually great to have in Phobos), but don't want to break backwards compatibility now. So at some point they would not even live in the vibe.* namespace - to me it doesn't make much sense to split this up now and create the impression that those are good to use outside of the vibe context - they are, but only after they have been properly extracted.

If vibe.data depends on std.stream you would just add it to its dependency list. Sub-packages which are not really standalone or are just tiny, would remain part of the vibe package.

Note that this kind of fine-grained splitting of packages would have been a PITA back when vibe started and no package manager was available, which is why all the common functionality today is concentrated in vibe.d itself. If one had been available, I would have separated them out right from the start.

You don't have to put stuff in a separate repository to make it a separate package. In the easiest case, a single line added to package.json would suffice to create a sub-package.

On the other hand having something like a taskset "vibe-ext" which pulls in a number of sub-modules might of course be good idea and could easily be achieved without even a single extension to my design (which still needs to be published), just create an empty vibe.ext package, which just depends on the corresponding sub packages. Simply put a sub-package in your package.json named "vibe.ext" without a vibe/ext folder actually existing so the package would be empty but you can specify arbitrary dependencies to be pulled in. This way you get the desired grouping, but with the additional benefit that the user can actually also just use a subset you might not even have thought of to be useful.

As said in my mail, this public sub-package concept adds some complexity and I'm really not sure about the value it actually gains. It's quite possible that I'm wrong on that, but up to now I still can't see the compelling argument. I also fear that the registry might end up with a plethora (exaggerated) of small sub-packages that have no actual use by themselves (e.g. I may be wrong, but I think nobody would use those org.* DWT sub-packages in isolation, but always as a whole, no point in making them all so prominent in the registry).

The only reason where you would get "unwanted" sub-packages would be that you have more than one group of sub-packages to be published separately and even in this case I don't really see a problem with small packages if you have automated dependency resolution, also the creation of sub-packages is trivial for the developer. You only have to specify the name and dependencies.

Having said this, you are right that this is an actual drawback of the proposal: If you want to split up a package in smaller ones and those sub-packages don't match your D-packages, you end up in having some packages in the registry which might never get used standalone.

So no restructuring needed, it would just work.

I'm looking forward to the new proposal, as I obviously got a bit behind with your latest ideas. I'll have time in the evening to read through it properly.

I hope to have an updated/improved version online by then.

Re: Should packages be restricted to a single name space?

Ok, I reworked my proposal and improved it based on recent feedback, you can find it here:

https://vhios.dyndns.org/drepoDraft.pdf

I haven't found the time yet to make it a wiki page, so I just uploaded it on my home server.

I removed most of the sections not applying to the topic discussed here, for the few things that remained (implementation details) please consider opening a separate thread if you want to discuss those.

Best regards,

Robert

Re: Should packages be restricted to a single name space?

https://vhios.dyndns.org/drepoDraft.pdf

Soenke, I hope the things I wrote about the current implementation are entirely correct, if not please correct me, I will fix it ASAP.

Pages: 1 2 3 4