RejectedSoftware Forums

On Sat, 10 Mar 2018 02:50:42 GMT, Carl Sturtivant wrote:

The following expression to use mongodb to produce a list of small documents is now broken, though it compiled and worked correctly before. It doesn't compile with any of dmd 2.077.1 , 2.078.3 or 2.079.0 :

db[collection].aggregate( 
    ["$group":["_id":"$"~variable]] 
)

and produces the error message

../../../.dub/packages/vibe-d-0.8.3/vibe-d/data/vibe/data/serialization.d(487,16): Error: functions cannot return a tuple
../../../.dub/packages/vibe-d-0.8.3/vibe-d/data/vibe/data/serialization.d(526,17): Error: template instance `vibe.data.serialization.serializeValueImpl!(BsonSerializer, DefaultPolicy).serializeValueDeduced!(Pipeline).serializeValueDeduced.safeGetMember!"pipeline"` error instantiating
../../../.dub/packages/vibe-d-0.8.3/vibe-d/data/vibe/data/serialization.d(354,115):        instantiated from here: serializeValueDeduced!(Pipeline)
../../../.dub/packages/vibe-d-0.8.3/vibe-d/data/vibe/data/serialization.d(205,58):        instantiated from here: serializeValue!(Pipeline)
../../../.dub/packages/vibe-d-0.8.3/vibe-d/data/vibe/data/serialization.d(152,49):        instantiated from here: serializeWithPolicy!(BsonSerializer, DefaultPolicy, Pipeline)
../../../.dub/packages/vibe-d-0.8.3/vibe-d/data/vibe/data/serialization.d(146,11):        ... (3 instantiations, -v to show) ...
source/database.d(114,49):        instantiated from here: aggregate!(string[string][string])

Any help would be greatly appreciated.

I've opened a ticket and will look into a fix: #2110

To work around the issue, the aggregate expression can be embedded into an array literal:

db[collection].aggregate( 
    [["$group":["_id":"$"~variable]]]
)

On Sat, 10 Mar 2018 11:13:52 +0100, Sönke Ludwig wrote:

Incoming fix: #2111

Many thanks.

Am 10.03.2018 um 16:26 schrieb Carl Sturtivant:

On Sat, 10 Mar 2018 09:41:11 GMT, Sönke Ludwig wrote:

To work around the issue, the aggregate expression can be embedded into an array literal:

db[collection].aggregate(
     [["$group":["_id":"$"~variable]]]
)

The documentation
https://vibed.org/api/vibe.db.mongo.collection/MongoCollection.aggregate
presents two templates.

Bson aggregate(ARGS...) (
   ARGS pipeline
);

MongoCursor!R aggregate(R, S) (
   S[] pipeline,
   AggregateOptions options
);

and yet says further down the page it says,

Returns

An array of documents returned by the pipeline

This seems to contradict the second template.

This was an oversight during the aggregate overhaul that happened for
this release, I'll rephrase that appropriately.

Presumably the original code that wouldn't compile used the first template, and this fix that does compile would seem to use the second which returns a cursor, not a Bson list of documents, correct?

With the workaround, it will still match the first overload, but it hits
a different path inside, which doesn't invoke the serialization code
that failed to compile. The failing code path was also added/modified
during the aggregate overhaul.

More generally than my example and its workaround fix, is it true that use of the first template is limited by the 16MB hard limit on query result size in mongodb, so the returned Bson list must be inside this limit?

Whereas presumably use of the second template may produce (via the returned cursor) a sequence of documents whose summed size is not limited to 16MB, as long as each individual document is inside that limit, correct?

The first overload is based on the second MongoCursor based one, so,
despite the return value being a single Bson, the limit in both cases
only applies to each individual document returned, not to the whole
aggregate result.

However, because assembling this single Bson value is usually
unnecessarily memory intensive, it still makes sense to use the second
overload instead (by passing AggregateOptions.init as a second argument).

Am 11.03.2018 um 20:42 schrieb Carl Sturtivant:

On Sat, 10 Mar 2018 18:02:19 +0100, Sönke Ludwig wrote:

(...)

This is nice!

Was aggregate limited to 16MB back with 0.8.2, unlike now?

As far as I can tell that should have been the case indeed, as the old
non-cursor based query mechanism was used and the result was actually
sent as a single document by the server.