Deserialization of partial data

Permalink: HTTP NNTP

Posted Wed, 01 Jul 2015 00:19:31 GMT

Reply

Hi guys,

I might be missing the flowers for the trees here as I've been working with vibe.d for a long time. However, I cannot find a simple vibe.d way to do something that is extremely common when building API backends.

Say we have a User object that looks like this:

struct User {
    string name;
    string reference;
    int salary;
}

So in my my API I have a putUpdate method. This does the following:

Takes in a JSON structure with the updates
Prunes out anything that that user should not be able to update
Loads and deserialises the record from the database
Applies the JSON update structure to the record

So the first step and the third step are rather easy with vibe.d. The second step, pruning, I have written in a separate library, but there are plenty of tools to do this easily with vibe.d (Though the standard Json and Bson types could use the ability to merge and prune conveniently)

The final step is my biggest issue. There does not appear to be a way to deserialise partial data over the top of an existing record. The deserialize method replaces it's target completely.

The only solution I have at the moment is:

Load the original record in and serialise it to Json
Merge the Json updates (not part of vibe standard library as far as I can tell)
Deserialise the update back to the model

Although this works, it is a lot of effort to go to and something that has to be done in each API in the application. While it is wrapped in a class at the moment, the solution feels inefficient our models also contain an internal state (not serialised) which has to be saved and restored each time we do this.

I'm really hoping that I've either missed something in the vibe library, or somebody has a much cleaner way of doing this already?

Thanks for your help guys,

David.

Re: Deserialization of partial data

Permalink: HTTP NNTP

Etienne Cimon

Posted Wed, 01 Jul 2015 03:19:46 GMT in reply to David Monagle

Reply

On Wed, 01 Jul 2015 00:19:31 GMT, David Monagle wrote:

Say we have a User object that looks like this:
struct User {
    string name;
    string reference;
    int salary;
}
Applies the JSON update structure to the record

You can serialize the update to a Json object, and deserialize the record to a Json object, and strip the Json object to a partial of the record, and do the merge using a recursive algorithm.

This is how I understood your problem: (untested!)

Json apply(Json update, Json record) {

   if (update.type == Json.Type.object)
   {
       foreach (ref string key, ref Json val; update)
       {
            if (record[key].type == Json.Type.undefined)
                 record[key] = val;
            else apply(key, record[key]);
       }
   } else record = update;
}

Re: Deserialization of partial data

Permalink: HTTP NNTP

David Monagle

Posted Wed, 01 Jul 2015 04:05:53 GMT in reply to Etienne Cimon

Reply

On Wed, 01 Jul 2015 03:19:46 GMT, Etienne Cimon wrote:

You can serialize the update to a Json object, and deserialize the record to a Json object, and strip the Json object to a partial of the record, and do the merge using a recursive algorithm.

Hi Etienne,

Yes this is exactly how I stated my current solution. I already have a json merge function that works satisfactorily. My big issue is that in deserialising the resulting Json, my class is overwritten, losing program state that is not serialized or written to the database.

Some theoretical code below:

// Take note, that this code wasn't written to compile. Just to illustrate a point.
Graph graph;

// The returned user is now a member of the graph
auto user = graph.find!User("email", "testuser@test.com");

Json updates = ...;

auto serializedUser = user.serializeToJson;
serializedUser.mergeInPlace(updates);

user.deserializeJson(serializedUser); // At this point "user" is now a new instance of User and the graph holds the original unmodified user;

graph.sync; // Would usually write the changed object to the database, but the object in the graph has not been changed

What I would like to see in the real world:

auto user = graph.find!User("email", "testuser@test.com");
user.mergeWithJson(updates);
graph.sync;

I point this out, not because I am looking for somebody to code me a solution, as I can do that. But vibe.d becomes a bit of a hard sell when updating a model from an API call is such a headache.