Hi,
I tried to serve a file via HTTP (but serveStaticFiles does not match my requirements). My first try was to use vibe.stream.stdio.StdFileStream, but this was very slow with speed like 350kB/s.
Then I looked into the implementation of sendFile and switched to vibe.core.file.FileStream, which serves the same 100MB file in an instant.
Everything from and to localhost.
Where does the difference come from and what is StdFileStream good for?
Thanks,
Tobias
Am 03.11.2019 um 16:04 schrieb StdFileStream really slow:
Hi,
I tried to serve a file via HTTP (but serveStaticFiles does not match my requirements). My first try was to use vibe.stream.stdio.StdFileStream, but this was very slow with speed like 350kB/s.
Then I looked into the implementation of sendFile and switched to vibe.core.file.FileStream, which serves the same 100MB file in an instant.
Everything from and to localhost.
Where does the difference come from and what is StdFileStream good for?
Thanks,
Tobias
StdFileStream is a compromise implementation using threads to support
blocking file descriptors when nothing else is available. If possible, FileStream is always the preferred implementation.
However, 350 kB/s is definitely far slower than it should be. The first
thing that I noticed was that the default buffer size is rather small
with just 2 kB, it should be at least 64 kB, ideally 256 kB or more.
Another thing is that the data is copied up to three times - once into a
temporary read buffer, then into the TaskPipe buffer, and finally into
the user target buffer.
Finally, reading from a StdFileStream is implemented by reading a
single character at a time to ensure that data is read in real-time for
interactive pipes (terminal input). For actual disk files this is
obviously a performance hazard and there should probably a flag in the
constructor to control this behavior.
On 2019-11-04 21:05, Sönke Ludwig wrote:
StdFileStreamis a compromise implementation using threads to support
blocking file descriptors when nothing else is available. If possible,FileStreamis always the preferred implementation.However, 350 kB/s is definitely far slower than it should be. The first
thing that I noticed was that the default buffer size is rather small
with just 2 kB, it should be at least 64 kB, ideally 256 kB or more.Another thing is that the data is copied up to three times - once into a
temporary read buffer, then into theTaskPipebuffer, and finally into
the user target buffer.Finally, reading from a
StdFileStreamis implemented by reading a
single character at a time to ensure that data is read in real-time for
interactive pipes (terminal input). For actual disk files this is
obviously a performance hazard and there should probably a flag in the
constructor to control this behavior.
There's the "sendfile" function as well [1]. It sends a file over a
socket directly in the kernel without copying and data to user space.
That should be the fastest option. But it's only for Posix.
[1] https://linux.die.net/man/2/sendfile
/Jacob Carlborg
On Mon, 4 Nov 2019 21:05:52 +0100, Sönke Ludwig wrote:
StdFileStreamis a compromise implementation using threads to support
blocking file descriptors when nothing else is available. If possible,FileStreamis always the preferred implementation.However, 350 kB/s is definitely far slower than it should be. The first
thing that I noticed was that the default buffer size is rather small
with just 2 kB, it should be at least 64 kB, ideally 256 kB or more.Another thing is that the data is copied up to three times - once into a
temporary read buffer, then into theTaskPipebuffer, and finally into
the user target buffer.Finally, reading from a
StdFileStreamis implemented by reading a
single character at a time to ensure that data is read in real-time for
interactive pipes (terminal input). For actual disk files this is
obviously a performance hazard and there should probably a flag in the
constructor to control this behavior.
Although quite late: Thanks for your reply and for vibe.d. It is a good framework.
