Testing/Benchmarking vibe.d

Permalink: HTTP NNTP

Posted Thu, 26 Dec 2013 12:01:24 GMT

Hi all,

I'm currently working on getting vibe.d to run on LDC.

I've fixed (or rather worked around, as it's actually the frontend messing things up) that linking problem and also fixed two other bugs I stumbled across when building vibe.d with a debug version of LDC. Right now, the merge-2.064 branch with my open pull requests builds a few of the examples fine on Linux x86_64, and all the basics seem to be working fine (I couldn't observe any of the crashes mentioned in the various vibe.d bug reports). The HTTP benchmark examples seem to perform a bit better than when built with DMD, but that was just a very cursory test.

Now, two questions:
1) Is there a comprehensive test suite I can run to ensure that there are no more miscompilations/… left in the basic structure? How can I easily run the unit test using dub (vibe.d being a library project)?

2) I recall seeing some promising performance numbers for vibe.d quite a while back, although I don't remember where exactly that was. Are there any benchmark implementations I could try to get some "interesting" numbers?

Thanks,
David

Re: Testing/Benchmarking vibe.d

Permalink: HTTP NNTP

Dicebot

Posted Thu, 26 Dec 2013 15:03:31 GMT in reply to David Nadlinger

On Thu, 26 Dec 2013 12:01:24 GMT, David Nadlinger wrote:

Now, two questions:
1) Is there a comprehensive test suite I can run to ensure that there are no more miscompilations/… left in the basic structure? How can I easily run the unit test using dub (vibe.d being a library project)?

Pretty good coverage can be achieved by doing 2 steps:

1) Going throgh https://github.com/rejectedsoftware/vibe.d/tree/master/examples and running both dub run and dub run --build=unittest for each example to verify that common use patterns work as expected

2) Building whole vibe.d with dub --build=unittest to run all general unittests

There are also some high-level functional tests in https://github.com/rejectedsoftware/vibe.d/tree/master/tests but those don't get much attention right now

2) I recall seeing some promising performance numbers for vibe.d quite a while back, although I don't remember where exactly that was. Are there any benchmark implementations I could try to get some "interesting" numbers?

Most recent published benchmark of vibe.d usage was https://atilanevesoncode.wordpress.com/2013/12/05/go-vs-d-vs-erlang-vs-c-in-real-life-mqtt-broker-implementation-shootout/

AFAIK https://github.com/rejectedsoftware/vibe.d/tree/master/examples/bench-http-server and https://github.com/rejectedsoftware/vibe.d/tree/master/examples/bench-http-request are what Sonke himself has been using for quick tests.

My old stuff is here https://github.com/Dicebot/web-performance-tests but it was focused on comparing different implementations in one specific scenario, not comprehensive coverage of different load/processing patterns

Re: Testing/Benchmarking vibe.d

Permalink: HTTP NNTP

Sönke Ludwig

Posted Thu, 26 Dec 2013 15:21:29 GMT in reply to David Nadlinger

Disclaimer: this was written well before Dicebots reply, but not submitted, so some have already been said.

Am 26.12.2013 13:01, schrieb David Nadlinger:

Hi all,

I'm currently working on getting vibe.d to run on LDC.

I've fixed (or rather worked around, as it's actually the frontend messing things up) that linking problem and also fixed two other bugs I stumbled across when building vibe.d with a debug version of LDC. Right now, the merge-2.064 branch with my open pull requests builds a few of the examples fine on Linux x86_64, and all the basics seem to be working fine (I couldn't observe any of the crashes mentioned in the various vibe.d bug reports). The HTTP benchmark examples seem to perform a bit better than when built with DMD, but that was just a very cursory test.

That's great news! I'll add LDC builds to my CI server when it's ready.

Now, two questions:
1) Is there a comprehensive test suite I can run to ensure that there are no more miscompilations/… left in the basic structure? How can I easily run the unit test using dub (vibe.d being a library project)?

With GIT master of DUB, "dub test" should work. For older versions there is a special "unittest" configuration ("dub --config=unittest --build=unittest").

The unit tests are not really suited for testing I/O and fiber related things, but the benchmark examples should normally be sufficiently complex to catch most issues there. For template/CTFE/linker stress testing, the vibed.org and code.dlang.org websites are also living quite on the cutting edge (at least for DMD).

2) I recall seeing some promising performance numbers for vibe.d quite a while back, although I don't remember where exactly that was. Are there any benchmark implementations I could try to get some "interesting" numbers?

What I used was "weighttp" on two separate linux machines connected by GbE. The client machine had to have a powerful CPU or "weighttp" wasn't able to saturate the "bench-http-server". Currently it's a little unclear how well the performance actually is, as it hasn't been benchmarked for a while and there were some changes in critical parts that may have had an influence. There is also a recent thread about this.

Re: Testing/Benchmarking vibe.d

Permalink: HTTP NNTP

David Nadlinger

Posted Fri, 27 Dec 2013 23:25:16 GMT in reply to Sönke Ludwig

Hi,

Thanks a lot to both of you for the quick responses.

On Thu, 26 Dec 2013 15:21:29 GMT, Sönke Ludwig wrote:

With GIT master of DUB, "dub test" should work. For older versions there is a special "unittest" configuration ("dub --config=unittest --build=unittest").

The unit tests are not really suited for testing I/O and fiber related things, but the benchmark examples should normally be sufficiently complex to catch most issues there. For template/CTFE/linker stress testing, the vibed.org and code.dlang.org websites are also living quite on the cutting edge (at least for DMD).

I had a look at the vibe.d unit tests, the examples and the vibed.org/code.dlang.org repos. With LDC from the merge-2.064 branch they should now all build, and the few I actually tested seem to work.

What I used was "weighttp" on two separate linux machines connected by GbE. The client machine had to have a powerful CPU or "weighttp" wasn't able to saturate the "bench-http-server". Currently it's a little unclear how well the performance actually is, as it hasn't been benchmarked for a while and there were some changes in critical parts that may have had an influence. There is also a recent thread about this.

Hm, I don't think I'll be able to set up two locally connected, beefy enough machines until the holidays are over. Just using the benchmark server/client examples, it seems that the CPU overhead is a little lower than with the DMD-compiled executables, but that's obviously not a very good way of measuring things.

David

Re: Testing/Benchmarking vibe.d

Permalink: HTTP NNTP

Dicebot

Posted Sat, 28 Dec 2013 22:25:45 GMT in reply to David Nadlinger

On Fri, 27 Dec 2013 23:25:16 GMT, David Nadlinger wrote:

What I used was "weighttp" on two separate linux machines connected by GbE. The client machine had to have a powerful CPU or "weighttp" wasn't able to saturate the "bench-http-server". Currently it's a little unclear how well the performance actually is, as it hasn't been benchmarked for a while and there were some changes in critical parts that may have had an influence. There is also a recent thread about this.

Hm, I don't think I'll be able to set up two locally connected, beefy enough machines until the holidays are over. Just using the benchmark server/client examples, it seems that the CPU overhead is a little lower than with the DMD-compiled executables, but that's obviously not a very good way of measuring things.

This is probably most difficult part when trying to do trustworthy benchmark in this domain. Modern server solutions are so often so capable that for a typical consumer computer it is easier to hit the bottleneck of network card h/w / network driver / I/O routines than CPU limit. Of course one can always add expensive computations into request handlers but that won't be pure benchmark of network server anymore then.