RejectedSoftware Forums

Sign up

JSON parser and complex structures

Hi!

Lets say I have this structure:

{
     "event": "handshake",
     "code" : 666,
     "data" : {
          "some" : 1,
          "other": "foobar"
     }
}

To get event or code is pretty simple

auto json = parseJson(input);
string event = json.event.get!string;

but I don't get how to convert data to any useable type, Variant or associative array, or class instance.

I can do

auto data = json["data"]["other"].get!string;

But what to do, for example, with array of objects of 10 properties?

auto data = json["data"].to!array;

throws

Error: undefined identifier array, did you mean function asArray?
Serializing composite type VariantN!32LU which has no serializable fields

auto data = json["data"].to!object;

throws

Error: template instance to!(object) does not match template declaration to(T)()

class Handshake
{
    int some;
    string other;
}

auto data = json["data"].to!Handshake;

throws

Error: function vibe.data.json.Json.to!(Handshake).to has no return statement, but is expected to return a value of type inout(Handshake)
Error: template instance vibe.data.json.Json.to!(Handshake) error instantiating

Re: JSON parser and complex structures

On Sun, 12 Apr 2015 23:48:58 GMT, Konstantin Ilchenko wrote:

Hi!

Lets say I have this structure:

{
     "event": "handshake",
     "code" : 666,
     "data" : {
          "some" : 1,
          "other": "foobar"
     }
}

if you know the schema a-priori you could use orvids json library:

https://github.com/Orvid/JSONSerialization/tree/master/JSONSerialization

@serializable @ignoreUndefined struct Event { 

   string event;
   ulong code;

  @serializable @ignoreUndefined struct Data { 
    ulongs some; 
    string other; 
   };
   
   Data data; 
  
}


Event ev = fromJSON!Event('{"event": "handshake","code" : 666,"data" : { "some" : 1, "other": "foobar"}}');

Re: JSON parser and complex structures

On 13.04.2015 um 01:48 Konstantin Ilchenko wrote:

(...)

but I don't get how to convert data to any useable type, Variant or associative array, or class instance.

I can do

auto data = json["data"]["other"].get!string;

But what to do, for example, with array of objects of 10 properties?

auto data = json["data"].to!array;

throws

Error: undefined identifier array, did you mean function asArray?
Serializing composite type VariantN!32LU which has no serializable fields

That should be json["data"].get!(Json[]) instead (to!(Json[]) works as well). Maybe adding support for json["data"].get!(Json.Type.object) would be an interesting alternative, though.

auto data = json["data"].to!object;

throws

Error: template instance to!(object) does not match template declaration to(T)()

Similarly, should be json["data"].get!(Json[string]).

class Handshake
{
     int some;
     string other;
}

auto data = json["data"].to!Handshake;

throws

Error: function vibe.data.json.Json.to!(Handshake).to has no return statement, but is expected to return a value of type inout(Handshake)
Error: template instance vibe.data.json.Json.to!(Handshake) error instantiating

This is done with deserializeJson: json["data"].deserializeJson!Handshake
You can process nested structures, arrays, AAs etc. with it. If performance is a concern, this also allows to process the JSON sting directly, without parsing it into a Json value first, so this is the generally recommended way to work with specific JSON structures.

Re: JSON parser and complex structures

Got it, thanks!

Re: JSON parser and complex structures

On Mon, 13 Apr 2015 19:43:02 GMT, Sönke Ludwig wrote:

class Handshake
{
     int some;
     string other;
}

This is done with deserializeJson: json["data"].deserializeJson!Handshake
You can process nested structures, arrays, AAs etc. with it. If performance is a concern, this also allows to process the JSON sting directly, without parsing it into a Json value first, so this is the generally recommended way to work with specific JSON structures.

nice, i somehow missed that.

  • can/will it ignore additional properties within the doc that are not defined in the class?

  • whats actually the current status of https://github.com/s-ludwig/stddatajson/tree/master/source/stdx/data/json versus vibe.d's parser?

i used the stddatajson for reading specific fields from a stream of docs where i needed only the header and it was way faster than anything else in d (but still half the speed i got with crystal).
i hope stddatajson makes into std, the lexer/parser are very nice to read!

Re: JSON parser and complex structures

Am 13.04.2015 um 22:52 schrieb yawniek:

On Mon, 13 Apr 2015 19:43:02 GMT, Sönke Ludwig wrote:

class Handshake
{
      int some;
      string other;
}

This is done with deserializeJson: json["data"].deserializeJson!Handshake
You can process nested structures, arrays, AAs etc. with it. If performance is a concern, this also allows to process the JSON sting directly, without parsing it into a Json value first, so this is the generally recommended way to work with specific JSON structures.

nice, i somehow missed that.

  • can/will it ignore additional properties within the doc that are not defined in the class?

Yes, it currently always does. There should probably also be a strict
mode, though...

  • whats actually the current status of https://github.com/s-ludwig/stddatajson/tree/master/source/stdx/data/json versus vibe.d's parser?

It's planned to deprecate the one in vibe.d as soon as it lands in
Phobos. It will first be layered on top of stdx.data.json (when it will
hopefully be std.experimental.data.json) with implicit convertibility
using alias this. Then all of vibe.d's own functions will be switched
to use stdx.data.json. Finally, after (hopefully) it got renamed to
std.data.json, vibe.data.json.Json will get deprecated and
eventually be removed.

Apart from that, it will not affect vibe.data.serialization or
serializeToJson.

i used the stddatajson for reading specific fields from a stream of docs where i needed only the header and it was way faster than anything else in d (but still half the speed i got with crystal).
i hope stddatajson makes into std, the lexer/parser are very nice to read!

Did you try the latest version? At least with LDC and using full
optimizations we finally got some numbers that were faster than Crystal
or RapidJSON. DMD is still behind RapidJSON, but I think it was on par
with Crystal. There is still potential for optimization, so I'm pretty
sure that we'll have a very compelling package in the end.

Re: JSON parser and complex structures

thanks for the outlook, good to know.

Did you try the latest version? At least with LDC and using full
optimizations we finally got some numbers that were faster than Crystal
or RapidJSON. DMD is still behind RapidJSON, but I think it was on par
with Crystal. There is still potential for optimization, so I'm pretty
sure that we'll have a very compelling package in the end.

yes, after some profiling (and trying out the byLineFast implementation that is floating in the dlang forum) i realized that most of my lost time was due to inefficient by line parsing in std.stdio. bummer, i totally did not expect this.

so yep, its actually now on par/faster than crystal. i think its pretty awesome that you can get C speed with very clean and dense code.