RejectedSoftware Forums

Sign up

Pages: 1 2 3

Alternative package description formats

There has been some discussion about using a different format than JSON for the package description files. While this topic is techinically not particularily pressing, it would be good to know where to go with this early. The two candidates so far are YAML and SDL. JSON will still be supported for backwards compatibility and as the default format that is transferred over the network (package registry). The following sections list the pros/cons of each format and an example of how it could look like. My favorite so far is SDL, which was originally proposed by Nick Sabalausky, who also made a D implementation. So now I think it would be good to get some more input to get a collective opinion on this before implementing anything.

JSON

While I think the current JSON based format for the package description is OK, it does have a few drawbacks:

  • No support for comments
  • Key names must be quoted, although they are just plain identifiers
  • Commas between individual fields -> danger of syntax errors for the last field
  • The very generic structure makes it easy to learn, but sometimes a bit verbose
  • Relatively deep nesting

Example:

{
	"name": "my-package",
	"description": "A package for demonstration purposes",
	"dependencies": {
		"vibe-d": ">=0.7.13",
		"sub-package": {"version": "~master", "path": "./sub-package"}
	},
	"configurations": [
		{
			"name": "console",
			"targetType": "executable",
			"versions": ["ConsoleApp"]
		},
		{
			"name": "gui",
			"targetType": "executable",
			"versions": ["ConsoleApp"],
			"libs-windows": ["gdi32", "user32"]
		}
	]
}

YAML

YAML basically has the same structure as JSON, but allows an alternative syntax using indentation instead of clamps to group items together. It also removes the quotes around strings. Furthermore, tt allows a multitude of ways to format the file (including JSON, which is a subset of YAML), which can be handy, but can also make it harder to learn or read, depending on how it is actually written. It follows the same structure as JSON, but supports comments.

Example:

---
name: my-package
description: A package for demonstration purposes
dependencies:
	vibe-d: ">=0.7.13"
	sub-package:
		version: ~master
		path: ./sub-package

configurations:
	# command line version
	- name: console
	  targetType: executable
	  versions:
	    - ConsoleApp

	# Win32 based GUI version
	- name: gui
	  targetType: executable
	  versions:
	    - UseWinMain
	  libs-windows:
	    - gdi32
	    - user32		

SDL

Finally, SDL has a tag+attribute based structure that differs from the other two and allows to vary how certain data fields are specified. It uses curly braces as the means to create a tag hierarchy and otherwise goes without punctuation (except for quotation marks) and thus results in clean files. Syntax-wise it comes closest to D when comparing the three contenders. Finally, the tag based nature facilitates a design with less nesting and less syntactic ambiguity (e.g. fields that may only occur once can be attributes).

Example:

name "my-package"
description "A package for demonstration purposes"

dependency "vibe-d" version=">=0.7.13"
dependency "sub-package" version="~master" path="./sub-package"

# command line version
configuration "console" {
	targetType "executable"
	versions "ConsoleApp"
	libs-windows "gdi32" "user32"
}

# Win32 based GUI version
configuration "gui" {
	targetType "executable"
	versions "UseWinMain"
	libs-windows "gdi32" "user32"
}

Re: Alternative package description formats

On Sat, 09 Mar 2013 12:11:09 GMT, Sönke Ludwig wrote:

There has been some discussion about using a different format than JSON for the package description files. While this topic is techinically not particularily pressing, it would be good to know where to go with this early. The two candidates so far are YAML and SDL. JSON will still be supported for backwards compatibility and as the default format that is transferred over the network (package registry). The following sections list the pros/cons of each format and an example of how it could look like. My favorite so far is SDL, which was originally proposed by Nick Sabalausky, who also made a D implementation. So now I think it would be good to get some more input to get a collective opinion on this before implementing anything.

I think SDL would be the best.

Re: Alternative package description formats

I agree, SDL seems to be the best. I'd never heard of it before this so it's completely foreign to me but seems to be clean and intuitive enough that I don't think anyone would have trouble picking it up.

I've always really liked the way YAML looks but have found it deceptively difficult to write YAML by hand.

Re: Alternative package description formats

On Sat, 09 Mar 2013 12:11:09 GMT, Sönke Ludwig wrote:

There has been some discussion about using a different format than JSON for the package description files. While this topic is techinically not particularily pressing, it would be good to know where to go with this early. The two candidates so far are YAML and SDL. JSON will still be supported for backwards compatibility and as the default format that is transferred over the network (package registry). The following sections list the pros/cons of each format and an example of how it could look like. My favorite so far is SDL, which was originally proposed by Nick Sabalausky, who also made a D implementation. So now I think it would be good to get some more input to get a collective opinion on this before implementing anything.

I confess that I detest yaml due to the fact that it's whitespace sensitive. json is far superior IMHO. As for SDL, the format seems a bit off to me due to the complete lack of separators or delimiters on most lines, though at least it requires fewer quotes than json does. Overall, I'm perfectly okay with json except that the lack of comments is a serious downside (is that a json spec issue or an issue with the current implementation?), but I'm not against SDL if the consensus is that it's better. But the change needs to be made soon if you want to minimize breakage when people update, and the fact that dub has been getting some attention it the main D newsgroup will likely lead to it being used more soon if it hasn't already (that's how I found out about it).

I think that one key thing to remember is that if dub is trying to become D's official package manager, it can't rely on external libraries for anything (except for the standard library). So, any package format that it supports either needs to be parseable with Phobos or with dub's own code. json is the only one of the three that's currently in Phobos (though std.json does need an overhaul). So, if dub goes with SDL (or yaml), it either needs to an SDL parser internally, or an SDL parser needs to be added to Phobos (though it's quite possible that one could be used internally and then added to Phobos later).

Re: Alternative package description formats

I think that one key thing to remember is that if dub is trying to become D's official package manager, it can't rely on external libraries for anything (except for the standard library). So, any package format that it supports either needs to be parseable with Phobos or with dub's own code.

That is something I don't really understand. If we go for SDL for example, why would it be better to redo Nick's work in dub or copy his code in, than having a build script download his library into the build directory? It is essentially the same thing, except that you end up in having to maintain a copy of already existing code.

I do understand that we should probably not depend on a library offering functionality that is also present in phobos, because why would we do that? Isn't phobos good enough? ...
But I don't really see the point in duplicating code. Soenke already did this for some functionality in order to make bootstrapping easy, but at least to me it seems to be less work to maintain a build script downloading some source code, than to maintain copies of libraries.

Of course in order to keep things simple, it might make sense to put more advanced, useful but not strictly required functionality into a separate package, like 'dub-extensions' in order to minimize the needed libraries of the core.

Best regards,

Robert

Re: Alternative package description formats

But the change needs to be made soon if you want to minimize breakage when people update, and the fact that dub has been getting some attention it the main D newsgroup will likely lead to it being used more soon if it hasn't already (that's how I found out about it).

We thought an automatic conversion from JSON to other formats should be easy because it is a data driven format. Meaning it should not be a problem to replace the format at some day, maybe having dub accepting both for a while.

Best regards,
Robert

Re: Alternative package description formats

On Thu, 14 Mar 2013 08:35:34 GMT, Robert wrote:

I think that one key thing to remember is that if dub is trying to become D's official package manager, it can't rely on external libraries for anything (except for the standard library). So, any package format that it supports either needs to be parseable with Phobos or with dub's own code.

That is something I don't really understand. If we go for SDL for example, why would it be better to redo Nick's work in dub or copy his code in, than having a build script download his library into the build directory? It is essentially the same thing, except that you end up in having to maintain a copy of already existing code.

I do understand that we should probably not depend on a library offering functionality that is also present in phobos, because why would we do that? Isn't phobos good enough? ...
But I don't really see the point in duplicating code. Soenke already did this for some functionality in order to make bootstrapping easy, but at least to me it seems to be less work to maintain a build script downloading some source code, than to maintain copies of libraries.

It has been mandated by Walter and Andrei that anything official to the d-programming-language org must not rely on any D code outside of it. In particular, whatever program becomes D's official package manager cannot rely on 3rd party libraries. So, anything it needs either needs to be in Phobos or in its own repo. This was brought up with regards to Orbit (another package manager for D), which currently relies on Tango and some other 3rd party stuff.

So, if dub is going to be D's official package manager, it can't be using 3rd party code. Any 3rd party code that it currently relies on or will need to rely on for new functionality (e.g. reading SDL files) needs to either be integrated into dub's repo, or it needs to get through the Phobos review process so that it can be in Phobos. If dub isn't trying to be D's official package manager, then it doesn't matter. 3rd party apps can rely on whatever 3rd party code that they want to rely on, but that doesn't fly for official stuff.

Re: Alternative package description formats

On the other hand it would make perfect sense to get any library dub depends on into phobos eventually, as it is shipped with the distribution anyway in one way or the other.

Re: Alternative package description formats

It has been mandated by Walter and Andrei that anything official to the d-programming-language org must not rely on any D code outside of it. In particular, whatever program becomes D's official package manager cannot rely on 3rd party libraries. So, anything it needs either needs to be in Phobos or in its own repo. This was brought up with regards to Orbit (another package manager for D), which currently relies on Tango and some other 3rd party stuff.

So, if dub is going to be D's official package manager, it can't be using 3rd party code. Any 3rd party code that it currently relies on or will need to rely on for new functionality (e.g. reading SDL files) needs to either be integrated into dub's repo, or it needs to get through the Phobos review process so that it can be in Phobos. If dub isn't trying to be D's official package manager, then it doesn't matter. 3rd party apps can rely on whatever 3rd party code that they want to rely on, but that doesn't fly for official stuff.

I know the discussion, I also see how depending on Tango is problematic. But it is practically no difference whether code is in the same repository as dub or not, there is no additional library shipped with the core distribution or anything, as dub would link the code in statically.

But it is probably better, to discuss these things at actual inclusion time.

Best regards,
Robert

Re: Alternative package description formats

I think what is important that we don't happen to depend on libraries that would not be accepted in Phobos, like Tango or some port of the Java standard library, ...

Then when striving for an inclusion in the core distribution, all non dub specific functionality should become part of phobos. In the long run, even the dub library itself could be part of phobos. But I think it is a good idea not to rush into things, because real world use before inclusion in phobos is always a good thing.

If non dub specific functionality would be accepted only if it was part of dub's source code, I think this would just be cheating without taking into account the original motivation for the restriction that we should only depend on phobos.

Do you know any guidelines/restrictions on what libraries can become part of phobos or not?

Best regards,
Robert

Pages: 1 2 3