Performance deterorates with memory heavy apps.

We have been running into severe performance issues with a particular server that initializes 3 GB of memory and keeps it for the life of the server. The performance drop happens when running ab test just on ping.

• First 10k requests (ping 6 ms) on same machine
• Second 10k requests (ping 11 ms) on same machine
• Third 10k requests (ping 20ms) on same machine

It then starts deteriorating. Sometimes hanging for 500ms for just pings.

I wrote a skeleton server to try and invistigate the issue. The app does not but ping back on request. I reserver large memory (2GB) and the issue is visible. if I do not reserver any memory, it works just fine with consistent ping time of 1ms.

void startServer() {
	version(linux) {
		import etc.linux.memoryerror;
		static if (is(typeof(registerMemoryErrorHandler)))
			registerMemoryErrorHandler();
	}

	auto serverSettings = new HTTPServerSettings;
	serverSettings.hostName = "searcher.ebookingservices.com";
	serverSettings.bindAddresses = options.bindip;
	serverSettings.port = options.port;

	serverSettings.errorPageHandler = toDelegate(&errorPage);
	serverSettings.options = HTTPServerOption.defaults | HTTPServerOption.errorStackTraces;

	serverSettings.accessLogFormat = "%h - %u %t \"%r\" %s %b \"%{Referer}i\" \"%{User-Agent}i\" %D";
	serverSettings.accessLogFile = options.accessLog;

	serverSettings.disableDistHost = true;

	serverSettings.useCompressionIfPossible = false;

	auto router = (new URLRouter)
		.get("/ping", &pingHandler); // used only for monitoring

	listenHTTP(serverSettings, router);
}

 void pingHandler(HTTPServerRequest req, HTTPServerResponse res) {res.writeBody("pong");}



extern(C) __gshared string[] rt_options = [ "gcopt=profile:0 minPoolSize:512 initReserve:2048 heapSizeFactor:1 help" ];

version(VibeCustomMain) {
	int main(string[] args) {
		import vibe.core.args : finalizeCommandLineOptions;
		import vibe.core.core : runEventLoop, lowerPrivileges;
		import vibe.core.log;
		import std.encoding : sanitize;

		try {
			if (!finalizeCommandLineOptions())
				return 0;
		} catch (Exception e) {
			return 1;
		}

		startServer();

		lowerPrivileges();

		try {
			return runEventLoop();
		} catch (Throwable e) {
			//logError("Unhandled exception in event loop: %s", e.msg);
			//logDiagnostic("Full exception: %s", e.toString().sanitize());
			return 1;
		}
	}
} else {
	shared static this() {
		startServer();
	}
}

if you comment this line

extern(C) __gshared string[] rt_options = [ "gcopt=profile:0 minPoolSize:512 initReserve:2048 heapSizeFactor:1 help" ];

Then it wont allocate memory and the difference in performance is quite huge.

From running a profiler on the app, it looks like with large memorry allocated, vibe array utils is called on a lot inside libevent.

Any help or insights would be appreciated.