RejectedSoftware Forums

Sign up

Elastic Search Vibe.d bulk insert

avatar
mcd
Posted Fri, 26 Jun 2015 04:24:33 GMT
Reply

I am looking at ElasticSearch and Vibe.d. It appears that to do a bulk insert in elastic search the json has to look like: 

{ "index" : { "_index" : "test", "_type" : "type1", "_id" : "1" } }
{ "field1" : "value1" }
{ "delete" : { "_index" : "test", "_type" : "type1", "_id" : "2" } }
{ "create" : { "_index" : "test", "_type" : "type1", "_id" : "3" } }
{ "field1" : "value3" }
{ "update" : {"_id" : "1", "_type" : "type1", "_index" : "index1"} }
{ "doc" : {"field2" : "value2"} }

ref: https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html

Because there is no brackets enclosing the object as a single statement, there are some JSON errors 

Error: Expected end of string after JSON value.

I believe this error is something with vibe transport. Is there any way to accomplish this with vibe.d?

Re: Elastic Search Vibe.d bulk insert

avatar
Márcio Martins
Posted Fri, 26 Jun 2015 15:31:42 GMT in reply to mcd
Reply

I'm not sure I understand your issue, but it seems like you want to construct a json string by hand without the root object braces.

Here's a quick example: 

struct Row {
  string value1;
  string value2;
}

Row[] rows;

rows ~= Row("foo", "bar");
rows ~= Row("moo", "car");

Appender!string jsonString;

foreach (row; rows) {
  jsonString.put(serializeToJsonString(row));
}

req.writeBody(cast(ubyte[])jsonString, "application/json");

Re: Elastic Search Vibe.d bulk insert

avatar
mcd
Posted Fri, 26 Jun 2015 19:10:17 GMT in reply to Márcio Martins
Reply

This would still have everything in a single object correct? For elastic search bulk it appears to have to be newline delimitted json passed to the request. 

{"index": {"_index": "bulk_test", "_type" : "test"}\n
{"col1" : "val1"}\n
{"index": {"_index": "bulk_test", "_type" : "test"}\n
{"col1" : "val2"}\n

How would you build and pass something like this in vibe?

Re: Elastic Search Vibe.d bulk insert

avatar
mcd
Posted Fri, 26 Jun 2015 19:35:22 GMT in reply to mcd
Reply

I got it working. I saw I was using writeJsonBody instead of writeBody.