RejectedSoftware Forums

Sign up

Reading nested Json Objects

Hello,

I have a JSON document read in from MongoDB and want to access some field that contains another Json object (as a child) and an array of Json objects as another child. The key for the field is called 'gtlayer' in my particular document. I can access this object in the Mongo client using the following:

db.projects.find( { }, { gtlayer: 1 } )

Inside my gtlayer json object there is an object called shape the value of which is an array describing various shapes. My ultimate goal is to read all the shape objects into an array of D classes representing the various shapes. Anyway I am stumped on how to properly get at this data. I currently do the following:

auto cur_proj = projects.find( Bson.EmptyObject, ["gtlayer": 1] ); \\Get a cursor to the database.
   
if( cur_proj.empty() ) {
    //write some error message.
} else {
    Json gtlayer = Json.EmptyObject;

    foreach( Bson bs; cur_proj ) {
        Json doc = bs.toJson();
	/* How do I check if current doc is "_id" or "gtlayer" ???
         * Alternately can I find the correct object/document without using the
         * foreach loop at all, as that would seem preferable. 
         */
         if( / *found it */ ) {
            gtlayer = doc;
            break;
          }
      }

      //Skips some sanity checks.
      auto shapes = gtlayer["shapes"].get!Json; //Is this how I get the 'Shapes' object?

Thanks for any help.

Craig

Re: Reading nested Json Objects

On Thu, 19 Sep 2013 21:07:37 GMT, Craig Dillabaugh wrote:

Hello,

I have a JSON document read in from MongoDB and want to access some field that contains another Json object (as a child) and an array of Json objects as another child. The key for the field is called 'gtlayer' in my particular document. I can access this object in the Mongo client using the following:

db.projects.find( { }, { gtlayer: 1 } )

Inside my gtlayer json object there is an object called shape the value of which is an array describing various shapes. My ultimate goal is to read all the shape objects into an array of D classes representing the various shapes. Anyway I am stumped on how to properly get at this data. I currently do the following:

auto cur_proj = projects.find( Bson.EmptyObject, ["gtlayer": 1] ); \\Get a cursor to the database.
   
if( cur_proj.empty() ) {
    //write some error message.
} else {
    Json gtlayer = Json.EmptyObject;

    foreach( Bson bs; cur_proj ) {
        Json doc = bs.toJson();
	/* How do I check if current doc is "_id" or "gtlayer" ???
         * Alternately can I find the correct object/document without using the
         * foreach loop at all, as that would seem preferable. 
         */
         if( / *found it */ ) {
            gtlayer = doc;
            break;
          }
      }

      //Skips some sanity checks.
      auto shapes = gtlayer["shapes"].get!Json; //Is this how I get the 'Shapes' object?

Thanks for any help.

Craig

Okay, so cur_proj will iterate over all documents in the "projects" collection and each returned Bson document will have the form {"_id": ObjectID(...), "gtlayer": ...}. So to get the gtlayer.shapes field, you could just do the following:

auto cur_proj = projects.find( Bson.EmptyObject, ["gtlayer": 1] ); \\Get a cursor to the database.
foreach (bs; cur_proj) {
    assert(!bs.gtlayer.isNull());
    auto shapes = bs.gtlayer.shapes.get!(Bson[]);
    Shape[] ret = shapes.map!(deserializeBson!Shape).array;
    // ...
}

If you only ever have a single document in the collection, or intend to find just a single one, findOne would be the right approach:

auto bs = projects.findOne(Bson.EmptyObject, ["gtlayer": 1]);
enforce(!bs.isNull(), "Couldn't find project");
assert(!bs.gtlayer.isNull());
auto shapes = bs.gtlayer.shapes.get!(Bson[]);
Shape[] ret = shapes.map!(deserializeBson!Shape).array;
// ...

That is assuming that Shape can be Bson (de)serialized properly (i.e. it either has all fields publicly accessible or has read/write properties for each of them or has fromBson/toBson methods).

...it's very possible that I got something totally wrong about the problem, though. I'll try again if this doesn't make sense ;)

Re: Reading nested Json Objects

On Fri, 20 Sep 2013 06:16:03 GMT, Sönke Ludwig wrote:

On Thu, 19 Sep 2013 21:07:37 GMT, Craig Dillabaugh wrote:

Hello,

I have a JSON document read in from MongoDB and want to access some field that contains another Json object (as a child) and an array of Json objects as another child. The key for the field is called 'gtlayer' in my particular document. I can access this object in the Mongo client using the following:

clip

Craig

Okay, so cur_proj will iterate over all documents in the "projects" collection and each returned Bson document will have the form {"_id": ObjectID(...), "gtlayer": ...}. So to get the gtlayer.shapes field, you could just do the following:

auto cur_proj = projects.find( Bson.EmptyObject, ["gtlayer": 1] ); \\Get a cursor to the database.
foreach (bs; cur_proj) {
    assert(!bs.gtlayer.isNull());
    auto shapes = bs.gtlayer.shapes.get!(Bson[]);
    Shape[] ret = shapes.map!(deserializeBson!Shape).array;
    // ...
}

If you only ever have a single document in the collection, or intend to find just a single one, findOne would be the right approach:

auto bs = projects.findOne(Bson.EmptyObject, ["gtlayer": 1]);
enforce(!bs.isNull(), "Couldn't find project");
assert(!bs.gtlayer.isNull());
auto shapes = bs.gtlayer.shapes.get!(Bson[]);
Shape[] ret = shapes.map!(deserializeBson!Shape).array; 
// ...

That is assuming that Shape can be Bson (de)serialized properly (i.e. it either has all fields publicly accessible or has read/write properties for each of them or has fromBson/toBson methods).

...it's very possible that I got something totally wrong about the problem, though. I'll try again if this doesn't make sense ;)

Thanks. Everything works up to the last step where I get the following:

source/app.d(137): Error: template instance shapes!(map(deserializeBson!(Shape))) shapes is not a template declaration, it is a variable
source/app.d(137): Error: template instance shapes!(map(deserializeBson!(Shape))) shapes is not a template declaration, it is a variable

Where line 137 is the line Shape[] ret = shapes.map!(deserializeBson!Shape).array, which I figure I should be able to fix, but I have no idea where the map!(...).array bit comes from.

Cheers,
Craig

Re: Reading nested Json Objects

Am 20.09.2013 19:57, schrieb Craig Dillabaugh:

On Fri, 20 Sep 2013 06:16:03 GMT, Sönke Ludwig wrote:

Thanks. Everything works up to the last step where I get the following:

source/app.d(137): Error: template instance shapes!(map(deserializeBson!(Shape))) shapes is not a template declaration, it is a variable
source/app.d(137): Error: template instance shapes!(map(deserializeBson!(Shape))) shapes is not a template declaration, it is a variable

Where line 137 is the line Shape[] ret = shapes.map!(deserializeBson!Shape).array, which I figure I should be able to fix, but I have no idea where the map!(...).array bit comes from.

Cheers,
Craig

The error message looks strange, but map() is in std.algorithm and
array() in std.array, so if any of those are not imported, maybe
that somehow causes them.

Otherwise this is the equivalent using a loop:

Shape[] ret;
foreach (bs; shapes) ret ~= deserializeBson!Shape(bs);

Re: Reading nested Json Objects

On Fri, 20 Sep 2013 20:39:00 +0200, Sönke Ludwig wrote:

Am 20.09.2013 19:57, schrieb Craig Dillabaugh:

On Fri, 20 Sep 2013 06:16:03 GMT, Sönke Ludwig wrote:

Thanks. Everything works up to the last step where I get the following:

source/app.d(137): Error: template instance shapes!(map(deserializeBson!(Shape))) shapes is not a template declaration, it is a variable
source/app.d(137): Error: template instance shapes!(map(deserializeBson!(Shape))) shapes is not a template declaration, it is a variable

Where line 137 is the line Shape[] ret = shapes.map!(deserializeBson!Shape).array, which I figure I should be able to fix, but I have no idea where the map!(...).array bit comes from.

Cheers,
Craig

The error message looks strange, but map() is in std.algorithm and
array() in std.array, so if any of those are not imported, maybe
that somehow causes them.

Otherwise this is the equivalent using a loop:

Shape[] ret;
foreach (bs; shapes) ret ~= deserializeBson!Shape(bs);

OK. The loop version worked so I am a happy camper.

However, since the map version looked so elegant I wanted to try and get it to work, but it did not, even with the proper imports.

I tried the following and while I got a different error, I still couldn't get it to compile.

auto bson_shapes = map!(deserializeBson!Shape)(shapes);

In this case I received the error (there were more, but usually the first error reported is the cause of the rest when templates are concerned):

source/app.d(138): Error: template vibe.data.bson.deserializeBson matches more than one template declaration, ../../.dub/packages/vibe-d-0.7.17/source/vibe/data/bson.d(946):deserializeBson(T)(ref T dst, Bson src) and ../../.dub/packages/vibe-d-0.7.17/source/vibe/data/bson.d(951):deserializeBson(T)(Bson src)

It seems it cannot decide which prototype for deserializeBson to choose from, I assume I want the second.

Now, back to the foreach loop version. My structs to read a shape look like follows:

struct Circle {
  int x;
  int y;
  int r;
}

//OK, so every shape is a circle for now .... baby steps ...
struct Shape {
  int 		class_;
  string 	type_;
  Circle 	coords;
  int		id;
}

My Json records for a shape look like follows:

{"class":1,"type":"circle","coords":{"x":3921,"y":2607,"r":4},"id":2}

Originally the Circle struct coords were of type double, but this causes a runtime exception of the form:

object.Exception@../../.dub/packages/vibe-d-0.7.17/source/vibe/data/bson.d(597): BSON value is type 'Int', expected to be one of [Double]

Of course for "x":3921 the value of x is interpreted as an integer and the Bson.Type is set to Int. In this case this isn't a big problem, since having integer coordinates is fine for my needs. But what if, for whatever reason, I could not assume x,y and r were ints, but they could be any value (including an int). Is there any way to tell deserializeBson to interpret all numeric values as doubles (if there is no easy answer no worries, I am just curious)?

Cheers,
Craig

Re: Reading nested Json Objects

On Fri, 20 Sep 2013 20:07:19 GMT, Craig Dillabaugh wrote:

My Json records for a shape look like follows:

{"class":1,"type":"circle","coords":{"x":3921,"y":2607,"r":4},"id":2}

Originally the Circle struct coords were of type double, but this causes a runtime exception of the form:

object.Exception@../../.dub/packages/vibe-d-0.7.17/source/vibe/data/bson.d(597): BSON value is type 'Int', expected to be one of [Double]

Of course for "x":3921 the value of x is interpreted as an integer and the Bson.Type is set to Int. In this case this isn't a big problem, since having integer coordinates is fine for my needs. But what if, for whatever reason, I could not assume x,y and r were ints, but they could be any value (including an int). Is there any way to tell deserializeBson to interpret all numeric values as doubles (if there is no easy answer no worries, I am just curious)?

One thing that I think would help a lot in many cases would be a less strict deserialiation mode where fields can be missnig or where some conversions such as long->double are allowed. It would be trivial enough to add, but I never did something like that because I was hoping for std.serialiazion to step into that void. But AFAICS it seems like another round of review/voting is necessary before it can be included. So when there is some time in the coming days, I'll add it.

Re: Reading nested Json Objects

On Sun, 22 Sep 2013 12:59:03 GMT, Sönke Ludwig wrote:

On Fri, 20 Sep 2013 20:07:19 GMT, Craig Dillabaugh wrote:

My Json records for a shape look like follows:

{"class":1,"type":"circle","coords":{"x":3921,"y":2607,"r":4},"id":2}

Originally the Circle struct coords were of type double, but this causes a runtime exception of the form:

object.Exception@../../.dub/packages/vibe-d-0.7.17/source/vibe/data/bson.d(597): BSON value is type 'Int', expected to be one of [Double]

Of course for "x":3921 the value of x is interpreted as an integer and the Bson.Type is set to Int. In this case this isn't a big problem, since having integer coordinates is fine for my needs. But what if, for whatever reason, I could not assume x,y and r were ints, but they could be any value (including an int). Is there any way to tell deserializeBson to interpret all numeric values as doubles (if there is no easy answer no worries, I am just curious)?

One thing that I think would help a lot in many cases would be a less strict deserialiation mode where fields can be missnig or where some conversions such as long->double are allowed. It would be trivial enough to add, but I never did something like that because I was hoping for std.serialiazion to step into that void. But AFAICS it seems like another round of review/voting is necessary before it can be included. So when there is some time in the coming days, I'll add it.

Just bumping this thread to ask if this feature has been added? I've actually run into this issue now. I have an array of doubles that I am reading in, but if the fractional part any number is left off by the generating program (which can happen) then I crash while trying to deserialize.

Cheers,

Craig

Re: Reading nested Json Objects

Am 01.11.2013 21:47, schrieb Craig Dillabaugh:

On Sun, 22 Sep 2013 12:59:03 GMT, Sönke Ludwig wrote:

On Fri, 20 Sep 2013 20:07:19 GMT, Craig Dillabaugh wrote:

My Json records for a shape look like follows:

{"class":1,"type":"circle","coords":{"x":3921,"y":2607,"r":4},"id":2}

Originally the Circle struct coords were of type double, but this causes a runtime exception of the form:

object.Exception@../../.dub/packages/vibe-d-0.7.17/source/vibe/data/bson.d(597): BSON value is type 'Int', expected to be one of [Double]

Of course for "x":3921 the value of x is interpreted as an integer and the Bson.Type is set to Int. In this case this isn't a big problem, since having integer coordinates is fine for my needs. But what if, for whatever reason, I could not assume x,y and r were ints, but they could be any value (including an int). Is there any way to tell deserializeBson to interpret all numeric values as doubles (if there is no easy answer no worries, I am just curious)?

One thing that I think would help a lot in many cases would be a less strict deserialiation mode where fields can be missnig or where some conversions such as long->double are allowed. It would be trivial enough to add, but I never did something like that because I was hoping for std.serialiazion to step into that void. But AFAICS it seems like another round of review/voting is necessary before it can be included. So when there is some time in the coming days, I'll add it.

Just bumping this thread to ask if this feature has been added? I've actually run into this issue now. I have an array of doubles that I am reading in, but if the fractional part any number is left off by the generating program (which can happen) then I crash while trying to deserialize.

Cheers,

Craig

On git master, if you define version "VibeNewSerialization", the
implicit integer->float conversion should work. It also supports tagging
individual fields with @optional now (a mode making all fields
optional is not there, but would be very simple to add if necessary).
But it's still opt-in, because the code is very new and may have some
bugs left.

Re: Reading nested Json Objects

On Sat, 02 Nov 2013 09:47:55 +0100, Sönke Ludwig wrote:

Am 01.11.2013 21:47, schrieb Craig Dillabaugh:

On Sun, 22 Sep 2013 12:59:03 GMT, Sönke Ludwig wrote:

On Fri, 20 Sep 2013 20:07:19 GMT, Craig Dillabaugh wrote:

My Json records for a shape look like follows:

{"class":1,"type":"circle","coords":{"x":3921,"y":2607,"r":4},"id":2}

Originally the Circle struct coords were of type double, but this causes a runtime exception of the form:

object.Exception@../../.dub/packages/vibe-d-0.7.17/source/vibe/data/bson.d(597): BSON value is type 'Int', expected to be one of [Double]

Of course for "x":3921 the value of x is interpreted as an integer and the Bson.Type is set to Int. In this case this isn't a big problem, since having integer coordinates is fine for my needs. But what if, for whatever reason, I could not assume x,y and r were ints, but they could be any value (including an int). Is there any way to tell deserializeBson to interpret all numeric values as doubles (if there is no easy answer no worries, I am just curious)?

One thing that I think would help a lot in many cases would be a less strict deserialiation mode where fields can be missnig or where some conversions such as long->double are allowed. It would be trivial enough to add, but I never did something like that because I was hoping for std.serialiazion to step into that void. But AFAICS it seems like another round of review/voting is necessary before it can be included. So when there is some time in the coming days, I'll add it.

Just bumping this thread to ask if this feature has been added? I've actually run into this issue now. I have an array of doubles that I am reading in, but if the fractional part any number is left off by the generating program (which can happen) then I crash while trying to deserialize.

Cheers,

Craig

On git master, if you define version "VibeNewSerialization", the
implicit integer->float conversion should work. It also supports tagging
individual fields with @optional now (a mode making all fields
optional is not there, but would be very simple to add if necessary).
But it's still opt-in, because the code is very new and may have some
bugs left.

Thanks, sounds great.

When you say define version "VibeNewSerialization", is that something you add to your package.json file, or do you mean using the D version(..) statement.

Cheers,
Craig

Re: Reading nested Json Objects

Am 04.11.2013 18:28, schrieb Craig Dillabaugh:

When you say define version "VibeNewSerialization", is that something you add to your package.json file, or do you mean using the D version(..) statement.

Cheers,
Craig

The former, in the form of the "versions" field:

{
	...
	"versions": ["VibeNewSerialization"]
}

But if no issues crop up during the next two weeks, I'll make the new
implementation the default one.

Regards,
Sönke