RejectedSoftware Forums

On 7/21/16 5:53 PM, Steven Schveighoffer wrote:

Let's say I have a request to get all items in a database.

I want to serialize the data to JSON to the client, but I don't want to
read the entire database into an array/Json object, and then write it out.

OK, so here is some code I wrote to try and do this with existing
features (the orgs range is something I wrote that fetches data into a
struct one database row at a time):

         // write each org as json data
         res.headers["Content-type"] = "application/json";
         auto rng = StreamOutputRange(res.bodyWriter);
         rng.put("[");
         bool first = true;
         foreach(o; orgs)
         {
             if(first)
                 first = false;
             else
                 rng.put(",");
             // serializeToJson(&rng, o); // option 1
             rng.put(serializeToJsonString(o)); // option 2
         }
         rng.put("]");

So there are some performance issues here. If I use option 1, it's very
slow. For my dataset which has about 20,000 rows, it takes 4 seconds to
transmit the result.

For option 2, it takes 2 seconds. This doesn't make a lot of sense, I
would have expected option 1 to be faster.

I changed the output to just print the id of the row object, and the
time reduced to 400ms. So this pretty much measures the time to fetch
data from the DB.

If I save the outputted json to a file, and serve that, it's 150ms.

This shouldn't take that long to serialize json. Has anyone examined
json performance in vibe? I'd like to help...

Oh, and yes I'm using -b release.

-Steve

On 7/22/16 9:40 AM, Steven Schveighoffer wrote:

On 7/21/16 8:17 PM, Steven Schveighoffer wrote:

So there are some performance issues here. If I use option 1, it's very
slow. For my dataset which has about 20,000 rows, it takes 4 seconds to
transmit the result.

For option 2, it takes 2 seconds. This doesn't make a lot of sense, I
would have expected option 1 to be faster.

I didn't record this properly. Option 2 is slower, 2.7 seconds. Option 1
is faster at 2.1 seconds.

But there is definitely no way this should be so slow. I'd expect
something near the 400ms. Investigating some more.

More info: It's not the json parser.

I parse all the objects to an Appender instead of the stream, and it's
still slow (~2s) when I write the Appender data to the stream. However,
if I just throw away the data from the appender, and then just send
"[]", the time taken is 500ms.

Something in writing the data using the bodyWriter is making this really
really slow.

If I put all the data into the appender, and then write that, the speed
is around 500-600ms. But obviously, I don't want to do that.

-Steve