RejectedSoftware Forums

Sign up

REST: ContentType and custom [de]serialization

Hello all,
I started using vibe.d to build a REST API. I really love it !

I want to build a REST API.
I've seen vibe.http.rest and it's really good, but it seems to be lacking / forbidding a feature which I absolutely need: a way to handle [de]serialization.
(At least that's what https://github.com/rejectedsoftware/vibe.d/blob/master/examples/rest/source/app.d#L90 let me to believe)

1) The main reason is that I would like to provide different media (ContentType) to people. So far, XML, JSON, HTML.
2) I'll also use ContentType over the URL-way for versioning (I'm aware of it's drawbacks).
3) And of course, I'd like to take full advantages of it by using objects / structs in the client, not return a string that I have to pass to another function.

What I tried so far was mostly based on @before / @after attributes:

  • @before: handles the authen (and set others parameters based on the header, like the version), works like a charm;
  • @after: I tried to use it to set the format, but I need to return something (for 3), and it WILL be serialized. If I just write to HTTPServerResponse, my data get duplicated, so I needed to tell the framework how to do it. I first override toString(), which in turns called a delegate (toJson, toHtml, toXML), but no luck with that (doesn't get called?). I also thought of implementing a custom Json object, but it's a struct, so no inheritance, and that's really hacky.

So the only solution I see so far is to implement the interface in the client only (I'll use JSON only for D), and implement the server without vibe.http.rest, only using plain URLRouter and stuff.

Did I missed a point ? Is there any plan to support custom [de]serialization sooner or later ?

Re: REST: ContentType and custom [de]serialization

On Tue, 19 Nov 2013 23:22:30 GMT, Mathias LANG wrote:

Hello all,
I started using vibe.d to build a REST API. I really love it !

I want to build a REST API.
I've seen vibe.http.rest and it's really good, but it seems to be lacking / forbidding a feature which I absolutely need: a way to handle [de]serialization.
(At least that's what https://github.com/rejectedsoftware/vibe.d/blob/master/examples/rest/source/app.d#L90 let me to believe)

The comment is not entirely correct anymore. The latest 0.7.18 betas support @name("somename"), @ignore, @optional and @byName attributes to customize how fields are (de-)serialized. However, it's still not possible to customize the serializer type that is used for the REST interface.

1) The main reason is that I would like to provide different media (ContentType) to people. So far, XML, JSON, HTML.
2) I'll also use ContentType over the URL-way for versioning (I'm aware of it's drawbacks).
3) And of course, I'd like to take full advantages of it by using objects / structs in the client, not return a string that I have to pass to another function.

So it looks like two additions to the REST system would be necessary: Allow to specify the serializer (JSON/XML/...) that is used, and allow to specify a custom suffix for each URL (".json"/".xml"/...). Then I'd imagine this to work in that scenario:

router.registerRestInterface!JsonStringSerializer(..., ".json");
router.registerRestInterface!XMLStringSerializer(..., ".xml");
router.registerRestInterface!HTMLStringSerializer(..., ".html");

Of course, the XML and HTML serializers would also have to be written, but that's trivial.

One thing to think about is how query parameters will be handled in this case. Currently all primitive types are directly converted to string (e.g. 2.5 gets param=2.5) and complex parameters get serialized to JSON (e.g. SomeStruct(2.5, "hello") might get param={"field1":2.5,"field2":"hello"}). XML or HTML might be less practical here. But JSON isn't optimal either, so maybe there is something entirely different that could be used here instead.

What I tried so far was mostly based on @before / @after attributes:

  • @before: handles the authen (and set others parameters based on the header, like the version), works like a charm;
  • @after: I tried to use it to set the format, but I need to return something (for 3), and it WILL be serialized. If I just write to HTTPServerResponse, my data get duplicated, so I needed to tell the framework how to do it. I first override toString(), which in turns called a delegate (toJson, toHtml, toXML), but no luck with that (doesn't get called?). I also thought of implementing a custom Json object, but it's a struct, so no inheritance, and that's really hacky.

Since the @before/@after feature is quite new, I didn't yet have the chance to actually try it out, so I'll defer an answer to later or to @Dicebot (who implemented it).

So the only solution I see so far is to implement the interface in the client only (I'll use JSON only for D), and implement the server without vibe.http.rest, only using plain URLRouter and stuff.

Did I missed a point ? Is there any plan to support custom [de]serialization sooner or later ?

I'm planning to rework the REST interface code a bit so that the MethodStyle is passed as a template parameter instead of as a runtime parameter (this would allow to shift a great amount of work from runtime to compile time and thus improve performance). I guess in that process it will be easy to also make the serializer a template argument. But I can't give an exact time schedule for this, yet.

Re: REST: ContentType and custom [de]serialization

On Wed, 20 Nov 2013 09:13:14 GMT, Sönke Ludwig wrote:

1) The main reason is that I would like to provide different media (ContentType) to people. So far, XML, JSON, HTML.
2) I'll also use ContentType over the URL-way for versioning (I'm aware of it's drawbacks).
3) And of course, I'd like to take full advantages of it by using objects / structs in the client, not return a string that I have to pass to another function.

So it looks like two additions to the REST system would be necessary: Allow to specify the serializer (JSON/XML/...) that is used, and allow to specify a custom suffix for each URL (".json"/".xml"/...). Then I'd imagine this to work in that scenario:

router.registerRestInterface!JsonStringSerializer(..., ".json");
router.registerRestInterface!XMLStringSerializer(..., ".xml");
router.registerRestInterface!HTMLStringSerializer(..., ".html");

Of course, the XML and HTML serializers would also have to be written, but that's trivial.

An alternative option may be to expose some struct/class that controls serialization and writing data to response via @after so that it can be tweaked in any way on per-method basis. I have never thought much about this development direction when working on REST part so hard to make any good proposals at once but sounds like a reasonable feature set to support.

What I tried so far was mostly based on @before / @after attributes:

  • @before: handles the authen (and set others parameters based on the header, like the version), works like a charm;
  • @after: I tried to use it to set the format, but I need to return something (for 3), and it WILL be serialized. If I just write to HTTPServerResponse, my data get duplicated, so I needed to tell the framework how to do it. I first override toString(), which in turns called a delegate (toJson, toHtml, toXML), but no luck with that (doesn't get called?). I also thought of implementing a custom Json object, but it's a struct, so no inheritance, and that's really hacky.

Since the @before/@after feature is quite new, I didn't yet have the chance to actually try it out, so I'll defer an answer to later or to @Dicebot (who implemented it).

I don't think @after can be used here. It was designed to make any adjustments to returned value and/or additions to response HTTP. Use case of completely overriding default data output mechanism with own one simply did not come to my mind. It can probably be added relatively simply, hard part is design, not implementation.

It is awesome to hear this stuff actually works though! Because I had no time to actual use it in live project personally since have added it :)

I'm planning to rework the REST interface code a bit so that the MethodStyle is passed as a template parameter instead of as a runtime parameter (this would allow to shift a great amount of work from runtime to compile time and thus improve performance). I guess in that process it will be easy to also make the serializer a template argument. But I can't give an exact time schedule for this, yet.

Damn, I have kept being tempted to move it to template parameter since first changes to this module and resisted all the time because it seemed like intended choice :D

Re: REST: ContentType and custom [de]serialization

On 2013-11-20 10:13, Sönke Ludwig wrote:

So it looks like two additions to the REST system would be necessary: Allow to specify the serializer (JSON/XML/...) that is used, and allow to specify a custom suffix for each URL (".json"/".xml"/...). Then I'd imagine this to work in that scenario:

router.registerRestInterface!JsonStringSerializer(..., ".json");
router.registerRestInterface!XMLStringSerializer(..., ".xml");
router.registerRestInterface!HTMLStringSerializer(..., ".html");

Of course, the XML and HTML serializers would also have to be written, but that's trivial.

One thing to think about is how query parameters will be handled in this case. Currently all primitive types are directly converted to string (e.g. 2.5 gets param=2.5) and complex parameters get serialized to JSON (e.g. SomeStruct(2.5, "hello") might get param={"field1":2.5,"field2":"hello"}). XML or HTML might be less practical here. But JSON isn't optimal either, so maybe there is something entirely different that could be used here instead.

I have no idea how this stuff works in vibe.d but why would you want
JSON in a query parameter?

/Jacob Carlborg

Re: REST: ContentType and custom [de]serialization

Am 20.11.2013 18:42, schrieb Jacob Carlborg:

On 2013-11-20 10:13, Sönke Ludwig wrote:

So it looks like two additions to the REST system would be necessary:
Allow to specify the serializer (JSON/XML/...) that is used, and allow
to specify a custom suffix for each URL (".json"/".xml"/...). Then I'd
imagine this to work in that scenario:

router.registerRestInterface!JsonStringSerializer(..., ".json");
router.registerRestInterface!XMLStringSerializer(..., ".xml");
router.registerRestInterface!HTMLStringSerializer(..., ".html");

Of course, the XML and HTML serializers would also have to be written,
but that's trivial.

One thing to think about is how query parameters will be handled in
this case. Currently all primitive types are directly converted to
string (e.g. 2.5 gets param=2.5) and complex parameters get
serialized to JSON (e.g. SomeStruct(2.5, "hello") might get
param={"field1":2.5,"field2":"hello"}). XML or HTML might be less
practical here. But JSON isn't optimal either, so maybe there is
something entirely different that could be used here instead.

I have no idea how this stuff works in vibe.d but why would you want
JSON in a query parameter?

Usually you wouldn't want that, but it was the most natural choice for
the case when someone declares a complex parameter for a GET method.
Another possibility would be to break up the struct into multiple
parameters (e.g. param_field1=2.5&param_field2=hello), but then there
are possible ambiguities and the implementation complexity grows
considerably.

Re: REST: ContentType and custom [de]serialization

Am 20.11.2013 14:59, schrieb Dicebot:

On Wed, 20 Nov 2013 09:13:14 GMT, Sönke Ludwig wrote:

1) The main reason is that I would like to provide different media (ContentType) to people. So far, XML, JSON, HTML.
2) I'll also use ContentType over the URL-way for versioning (I'm aware of it's drawbacks).
3) And of course, I'd like to take full advantages of it by using objects / structs in the client, not return a string that I have to pass to another function.

So it looks like two additions to the REST system would be necessary: Allow to specify the serializer (JSON/XML/...) that is used, and allow to specify a custom suffix for each URL (".json"/".xml"/...). Then I'd imagine this to work in that scenario:

router.registerRestInterface!JsonStringSerializer(..., ".json");
router.registerRestInterface!XMLStringSerializer(..., ".xml");
router.registerRestInterface!HTMLStringSerializer(..., ".html");

Of course, the XML and HTML serializers would also have to be written, but that's trivial.

An alternative option may be to expose some struct/class that controls serialization and writing data to response via @after so that it can be tweaked in any way on per-method basis. I have never thought much about this development direction when working on REST part so hard to make any good proposals at once but sounds like a reasonable feature set to support.

But that would mean that each method has to be implemented for each
content type (unless I'm missing something).

I'm planning to rework the REST interface code a bit so that the MethodStyle is passed as a template parameter instead of as a runtime parameter (this would allow to shift a great amount of work from runtime to compile time and thus improve performance). I guess in that process it will be easy to also make the serializer a template argument. But I can't give an exact time schedule for this, yet.

Damn, I have kept being tempted to move it to template parameter since first changes to this module and resisted all the time because it seemed like intended choice :D

Yeah, let's say it was some kind of historic accident that I've never
really noticed because I've always left the parameter to its default ;)

Re: REST: ContentType and custom [de]serialization

On Wed, 20 Nov 2013 09:13:14 GMT, Sönke Ludwig wrote:

So it looks like two additions to the REST system would be necessary: Allow to specify the serializer (JSON/XML/...) that is used, and allow to specify a custom suffix for each URL (".json"/".xml"/...). Then I'd imagine this to work in that scenario:

router.registerRestInterface!JsonStringSerializer(..., ".json");
router.registerRestInterface!XMLStringSerializer(..., ".xml");
router.registerRestInterface!HTMLStringSerializer(..., ".html");

Of course, the XML and HTML serializers would also have to be written, but that's trivial.

One thing to think about is how query parameters will be handled in this case. Currently all primitive types are directly converted to string (e.g. 2.5 gets param=2.5) and complex parameters get serialized to JSON (e.g. SomeStruct(2.5, "hello") might get param={"field1":2.5,"field2":"hello"}). XML or HTML might be less practical here. But JSON isn't optimal either, so maybe there is something entirely different that could be used here instead.

Will it then be possible to use all the "normal" server features then (I'm thinking about render here) ?
I also quite don't get the custom suffix. The server will use the same resources, represented in different way. So I would rather go for a predicate that get the HTTPServerRequest and tells if the registered interface + serializer couple can handle the request.
But then the order or registration should matter, or there should be a way to set the default event handler (because if I'm asked for a wrong version / ContentType, I want to answer with a 415, not a 404). For this part I don't know what's already buit in Vibe, but I guess there's something ?

I'm planning to rework the REST interface code a bit so that the MethodStyle is passed as a template parameter instead of as a runtime parameter (this would allow to shift a great amount of work from runtime to compile time and thus improve performance). I guess in that process it will be easy to also make the serializer a template argument. But I can't give an exact time schedule for this, yet.

Good to know! Well after experimenting a bit, I think I will stay out of vibe.http.rest for the server, even if it gets strong foundations, it will get in my way for my use case. I'll go back at it when I switch to the client part. I'll try to give a feedback here when the job is done. Feel free to email me if I can be of any help.

Usually you wouldn't want that, but it was the most natural choice for
the case when someone declares a complex parameter for a GET method.
Another possibility would be to break up the struct into multiple
parameters (e.g. param_field1=2.5&param_field2=hello), but then there
are possible ambiguities and the implementation complexity grows
considerably.

It seemed odd to me too, but then I thought how funny it would be to have associative array with classes as key and values. Maybe provide a way to have a simple parameter schema is there is no struct / AA / classes involved (only 'scalars' and arrays) ?

On Wed, 20 Nov 2013 13:59:33 GMT, Dicebot wrote:

An alternative option may be to expose some struct/class that controls serialization and writing data to response via @after so that it can be tweaked in any way on per-method basis. I have never thought much about this development direction when working on REST part so hard to make any good proposals at once but sounds like a reasonable feature set to support.

[...]

I don't think @after can be used here. It was designed to make any adjustments to returned value and/or additions to response HTTP. Use case of completely overriding default data output mechanism with own one simply did not come to my mind. It can probably be added relatively simply, hard part is design, not implementation.

It is awesome to hear this stuff actually works though! Because I had no time to actual use it in live project personally since have added it :)

Yup I know, I was just trying to see if I can hack my way to make it do what I want it to do. Turns out I couldn't. The design seems quite neat to me, I was able to use @before to provide an UserContext to my function (so far, only authnz), and @after's goal was quite clear. The only issue was it's not on the API doc on the site.

Re: REST: ContentType and custom [de]serialization

On Thu, 21 Nov 2013 14:11:49 GMT, Mathias LANG wrote:

I'm planning to rework the REST interface code a bit so that the MethodStyle is passed as a template parameter instead of as a runtime parameter (this would allow to shift a great amount of work from runtime to compile time and thus improve performance). I guess in that process it will be easy to also make the serializer a template argument. But I can't give an exact time schedule for this, yet.

Good to know! Well after experimenting a bit, I think I will stay out of vibe.http.rest for the server, even if it gets strong foundations, it will get in my way for my use case. I'll go back at it when I switch to the client part. I'll try to give a feedback here when the job is done. Feel free to email me if I can be of any help.

One of best helpful things to do it to simply provide feedback upon use cases where it was unsuitable for implementing target REST functionality together with any API suggestions that can possibly make it work. I like tinkering with that module but I am doing zero actual web dev which is one of main blockers for further improvements :)

Yup I know, I was just trying to see if I can hack my way to make it do what I want it to do. Turns out I couldn't. The design seems quite neat to me, I was able to use @before to provide an UserContext to my function (so far, only authnz), and @after's goal was quite clear. The only issue was it's not on the API doc on the site.

Website has docs for last released version and this functionality has been only part of git master so far. You are experimenting with the bleeding edge ;) At some point there will be separate page in vibed.org devoted exclusively to dealing with RESt features but I'd like to have a chance to actually use this stuff personally before writing it. Sorry, tiny bit of unlucky timing!

Re: REST: ContentType and custom [de]serialization

On 2013-11-20 22:39, Sönke Ludwig wrote:

Usually you wouldn't want that, but it was the most natural choice for
the case when someone declares a complex parameter for a GET method.
Another possibility would be to break up the struct into multiple
parameters (e.g. param_field1=2.5&param_field2=hello), but then there
are possible ambiguities and the implementation complexity grows
considerably.

We're using Ruby on Rails with quite many fairly complex forms. These
are all POST/PUT requests but they would work for GET as well. Just use
nested query parameters:

foo[x][y]

Shouldn't be an ambiguities since all the fields in a class/struct need
to have unique names anyway.

/Jacob Carlborg