Today I have done a test to see how vibe-d reacts to segmentation faults and I sadly (but not unexpectedly) found that a segmentation fault brings the whole server down. In other words, if a coding mistake was present in a production environment (for example, dereferencing a null pointer) and a segfault occurs, all requests being served at that moment will be dropped. I realise that these cases should be rare but it is still not acceptable in a production environment at least.
I realise that there is no easy solution for this, at least not to my knowledge. Maybe a segfault signal handler can help, but I don't really know much about this.
Do you have any ideas/suggestions for a solution?
On Tue, 25 Jun 2013 20:34:26 GMT
"Yazan Dabain" yazan.dabain@gmail.com wrote:
Today I have done a test to see how vibe-d reacts to segmentation
faults and I sadly (but not unexpectedly) found that a segmentation
fault brings the whole server down. In other words, if a coding
mistake was present in a production environment (for example,
dereferencing a null pointer) and a segfault occurs, all requests
being served at that moment will be dropped. I realise that these
cases should be rare but it is still not acceptable in a production
environment at least. I realise that there is no easy solution for
this, at least not to my knowledge. Maybe a segfault signal handler
can help, but I don't really know much about this. Do you have any
ideas/suggestions for a solution?
You could omit passing the -release and -noboundscheck flags to DMD.
It's not exactly a real solution to your problem since segfaults can
still occur, but will insert some checks which will catch at least
some problems before they actually turn into segfaults.
Although really, in a live production environment, you should probably
be running a watchdog service on the server anyway. (Although I'm
currently a hypocrite for saying that...)
Am 26.06.2013 18:59, schrieb Nick Sabalausky:
On Tue, 25 Jun 2013 20:34:26 GMT
"Yazan Dabain" yazan.dabain@gmail.com wrote:Today I have done a test to see how vibe-d reacts to segmentation
faults and I sadly (but not unexpectedly) found that a segmentation
fault brings the whole server down. In other words, if a coding
mistake was present in a production environment (for example,
dereferencing a null pointer) and a segfault occurs, all requests
being served at that moment will be dropped. I realise that these
cases should be rare but it is still not acceptable in a production
environment at least. I realise that there is no easy solution for
this, at least not to my knowledge. Maybe a segfault signal handler
can help, but I don't really know much about this. Do you have any
ideas/suggestions for a solution?You could omit passing the -release and -noboundscheck flags to DMD.
It's not exactly a real solution to your problem since segfaults can
still occur, but will insert some checks which will catch at least
some problems before they actually turn into segfaults.Although really, in a live production environment, you should probably
be running a watchdog service on the server anyway. (Although I'm
currently a hypocrite for saying that...)
I remember there were some discussions about using libsegfault on Linux
to convert segfaults into exceptions. But I don't remember why this was
rejected in the end.
Anyway, this may be a possibility if it allows to throw an exception
from the fault handler.
On Wed, 26 Jun 2013 19:09:49 +0200, Sönke Ludwig wrote:
Am 26.06.2013 18:59, schrieb Nick Sabalausky:
On Tue, 25 Jun 2013 20:34:26 GMT
"Yazan Dabain" yazan.dabain@gmail.com wrote:Today I have done a test to see how vibe-d reacts to segmentation
faults and I sadly (but not unexpectedly) found that a segmentation
fault brings the whole server down. In other words, if a coding
mistake was present in a production environment (for example,
dereferencing a null pointer) and a segfault occurs, all requests
being served at that moment will be dropped. I realise that these
cases should be rare but it is still not acceptable in a production
environment at least. I realise that there is no easy solution for
this, at least not to my knowledge. Maybe a segfault signal handler
can help, but I don't really know much about this. Do you have any
ideas/suggestions for a solution?You could omit passing the -release and -noboundscheck flags to DMD.
It's not exactly a real solution to your problem since segfaults can
still occur, but will insert some checks which will catch at least
some problems before they actually turn into segfaults.Although really, in a live production environment, you should probably
be running a watchdog service on the server anyway. (Although I'm
currently a hypocrite for saying that...)I remember there were some discussions about using libsegfault on Linux
to convert segfaults into exceptions. But I don't remember why this was
rejected in the end.Anyway, this may be a possibility if it allows to throw an exception
from the fault handler.
A watchdog service is indeed necessary. Maybe vibe's load balancer can do that too (just a thought). Anyway, I think that maybe this is a penalty that I'm willing to pay for considering the performance and efficiency that vibe and D provide.
Thanks Nick Sabalausky and Sönke Ludwig.
Am 28.06.2013 18:57, schrieb Yazan Dabain:
On Wed, 26 Jun 2013 19:09:49 +0200, Sönke Ludwig wrote:
Am 26.06.2013 18:59, schrieb Nick Sabalausky:
On Tue, 25 Jun 2013 20:34:26 GMT
"Yazan Dabain" yazan.dabain@gmail.com wrote:Today I have done a test to see how vibe-d reacts to segmentation
faults and I sadly (but not unexpectedly) found that a segmentation
fault brings the whole server down. In other words, if a coding
mistake was present in a production environment (for example,
dereferencing a null pointer) and a segfault occurs, all requests
being served at that moment will be dropped. I realise that these
cases should be rare but it is still not acceptable in a production
environment at least. I realise that there is no easy solution for
this, at least not to my knowledge. Maybe a segfault signal handler
can help, but I don't really know much about this. Do you have any
ideas/suggestions for a solution?You could omit passing the -release and -noboundscheck flags to DMD.
It's not exactly a real solution to your problem since segfaults can
still occur, but will insert some checks which will catch at least
some problems before they actually turn into segfaults.Although really, in a live production environment, you should probably
be running a watchdog service on the server anyway. (Although I'm
currently a hypocrite for saying that...)I remember there were some discussions about using libsegfault on Linux
to convert segfaults into exceptions. But I don't remember why this was
rejected in the end.Anyway, this may be a possibility if it allows to throw an exception
from the fault handler.A watchdog service is indeed necessary. Maybe vibe's load balancer can do that too (just a thought). Anyway, I think that maybe this is a penalty that I'm willing to pay for considering the performance and efficiency that vibe and D provide.
Thanks Nick Sabalausky and Sönke Ludwig.
Yes that's indeed one of the planned features of the load balancer (I'll
continue work on it in about one or two months and then it should be
usable).
But generally I mostly think about segfaults the same way as any other
bug - they have to be fixed, but with core dumps enabled that is usually
easy (since most of the time segfault == null pointer dereference in D).
Interrupting unrelated requests that are handled at the same time surely
makes them worse than some other bugs, but at least the site would have
to be fairly high volume/serve large files for that to become a real issue.
On Sun, 30 Jun 2013 20:17:43 +0200, Sönke Ludwig wrote:
Am 28.06.2013 18:57, schrieb Yazan Dabain:
On Wed, 26 Jun 2013 19:09:49 +0200, Sönke Ludwig wrote:
Am 26.06.2013 18:59, schrieb Nick Sabalausky:
On Tue, 25 Jun 2013 20:34:26 GMT
"Yazan Dabain" yazan.dabain@gmail.com wrote:Today I have done a test to see how vibe-d reacts to segmentation
faults and I sadly (but not unexpectedly) found that a segmentation
fault brings the whole server down. In other words, if a coding
mistake was present in a production environment (for example,
dereferencing a null pointer) and a segfault occurs, all requests
being served at that moment will be dropped. I realise that these
cases should be rare but it is still not acceptable in a production
environment at least. I realise that there is no easy solution for
this, at least not to my knowledge. Maybe a segfault signal handler
can help, but I don't really know much about this. Do you have any
ideas/suggestions for a solution?You could omit passing the -release and -noboundscheck flags to DMD.
It's not exactly a real solution to your problem since segfaults can
still occur, but will insert some checks which will catch at least
some problems before they actually turn into segfaults.Although really, in a live production environment, you should probably
be running a watchdog service on the server anyway. (Although I'm
currently a hypocrite for saying that...)I remember there were some discussions about using libsegfault on Linux
to convert segfaults into exceptions. But I don't remember why this was
rejected in the end.Anyway, this may be a possibility if it allows to throw an exception
from the fault handler.A watchdog service is indeed necessary. Maybe vibe's load balancer can do that too (just a thought). Anyway, I think that maybe this is a penalty that I'm willing to pay for considering the performance and efficiency that vibe and D provide.
Thanks Nick Sabalausky and Sönke Ludwig.Yes that's indeed one of the planned features of the load balancer (I'll
continue work on it in about one or two months and then it should be
usable).But generally I mostly think about segfaults the same way as any other
bug - they have to be fixed, but with core dumps enabled that is usually
easy (since most of the time segfault == null pointer dereference in D).
Interrupting unrelated requests that are handled at the same time surely
makes them worse than some other bugs, but at least the site would have
to be fairly high volume/serve large files for that to become a real issue.
Is there some changes in this subject? Cause I discovered today this comportment. That is so brutal.
I think that with a large code base it so easy to forget a pointer test, even with unit-tests.
How much we have to trust third party code?