RejectedSoftware Forums

Sign up

Automatically generated ddoc on code.dlang.org

How realistic would it be to have code.dlang.org automatically generate documentation for all Dub packages? That includes be able to view the generated documentation on code.dlang.org.

Re: Automatically generated ddoc on code.dlang.org

Am 24.10.2014 08:38, schrieb Jacob Carlborg:

How realistic would it be to have code.dlang.org automatically generate documentation for all Dub packages? That includes be able to view the generated documentation on code.dlang.org.

Technically it would be very simple if DDOX is used as a library.
Contrary to what I initially thought, it also wouldn't be necessary to
play around with virtual environments for security, because instead of
actually building the project with JSON output it should be sufficient
to just run DDOX with its dparser (the one used for Dscanner) based back
end.

The only thing is that for efficiency reasons, it would probably be a
good idea to also implement serialization functionality for the internal
AST, so that it can be stored in the database instead of constantly
reparsing or storing everything in memory (or maybe a fixed size cache
is enough for now).

Re: Automatically generated ddoc on code.dlang.org

On 2014-10-27 07:58, Sönke Ludwig wrote:

Technically it would be very simple if DDOX is used as a library.
Contrary to what I initially thought, it also wouldn't be necessary to
play around with virtual environments for security, because instead of
actually building the project with JSON output it should be sufficient
to just run DDOX with its dparser (the one used for Dscanner) based back
end.

Ok, cool.

The only thing is that for efficiency reasons, it would probably be a
good idea to also implement serialization functionality for the internal
AST, so that it can be stored in the database instead of constantly
reparsing or storing everything in memory (or maybe a fixed size cache
is enough for now).

Hmm, I was just thinking it would generate the HTML once for each
package and version and store it on disk. Then just serve the static
HTML pages. You had something else in mind?

/Jacob Carlborg

Re: Automatically generated ddoc on code.dlang.org

On Tue, 28 Oct 2014 21:24:06 +0100, Jacob Carlborg wrote:

On 2014-10-27 07:58, Sönke Ludwig wrote:

Technically it would be very simple if DDOX is used as a library.
Contrary to what I initially thought, it also wouldn't be necessary to
play around with virtual environments for security, because instead of
actually building the project with JSON output it should be sufficient
to just run DDOX with its dparser (the one used for Dscanner) based back
end.

Ok, cool.

The only thing is that for efficiency reasons, it would probably be a
good idea to also implement serialization functionality for the internal
AST, so that it can be stored in the database instead of constantly
reparsing or storing everything in memory (or maybe a fixed size cache
is enough for now).

Hmm, I was just thinking it would generate the HTML once for each
package and version and store it on disk. Then just serve the static
HTML pages. You had something else in mind?

My idea was to use the same integrated serving as for http://vibed.org/api/. One advantage this would have is that it is much less expensive in terms of disk space and I/O, as well as in general preprocessing time. The full API documentation for vibe.d for example takes something around 2500 files in 30MB, so this could get costly pretty quick.

Re: Automatically generated ddoc on code.dlang.org

On Tue, 18 Nov 2014 18:25:20 GMT, Sönke Ludwig wrote:

My idea was to use the same integrated serving as for http://vibed.org/api/. One advantage this would have is that it is much less expensive in terms of disk space and I/O, as well as in general preprocessing time. The full API documentation for vibe.d for example takes something around 2500 files in 30MB, so this could get costly pretty quick.

Ok, I see. It was the disk space I was most worried about, hence the "realistic" part in my original post.

If we compare code.dlang.org with rubygems.org, rubugems has 5822 packages and Dub has 369 packages. You say vibe.d is around 2500 files, I would guess that mosts projects is noway near that amount of files. But if we take that as an example, 5822 packages multiplied with 30 MB gives under 180 GB. In my opinion that's nothing in todays hard drives which have capacity of several terabytes. Although, of course I'm not expecting you to pay for this, you're already doing a great work with vibe.d and Dub :). BTW, if you serialize the AST to disk, won't that have the same problem as the pre-generated documentation in regards of disk space.

About the I/O, in my experience serving static pages are almost always faster than dynamically generated pages.

I'm not I understand the issue with preprocessing time. Won't your solution need to be processed basically on all requests compared to my solution which is once per package and version?

/Jacob Carlborg

Re: Automatically generated ddoc on code.dlang.org

Am 19.11.2014 10:36, schrieb Jacob Carlborg:

On Tue, 18 Nov 2014 18:25:20 GMT, Sönke Ludwig wrote:

My idea was to use the same integrated serving as for http://vibed.org/api/. One advantage this would have is that it is much less expensive in terms of disk space and I/O, as well as in general preprocessing time. The full API documentation for vibe.d for example takes something around 2500 files in 30MB, so this could get costly pretty quick.

Ok, I see. It was the disk space I was most worried about, hence the "realistic" part in my original post.

If we compare code.dlang.org with rubygems.org, rubugems has 5822 packages and Dub has 369 packages. You say vibe.d is around 2500 files, I would guess that mosts projects is noway near that amount of files. But if we take that as an example, 5822 packages multiplied with 30 MB gives under 180 GB. In my opinion that's nothing in todays hard drives which have capacity of several terabytes. Although, of course I'm not expecting you to pay for this, you're already doing a great work with vibe.d and Dub :). BTW, if you serialize the AST to disk, won't that have the same problem as the pre-generated documentation in regards of disk space.

It's just that the disk space of the VM that is currently in use is
already quite limited. If I'd have a dedicated server (or someone else
would donate one), that would definitely change the significance.

About the I/O, in my experience serving static pages are almost always faster than dynamically generated pages.

As soon as they are cached in memory, they are hard to beat, of course,
but in many cases dynamic pages can also be very fast. I'd need to
perform an actual benchmark, though, to say anything substantial about DDOX.

I'm not I understand the issue with preprocessing time. Won't your solution need to be processed basically on all requests compared to my solution which is once per package and version?

For the dynamic solution, all that is needed per page request is a quick
iteration over the relevant part of the AST (stored in RAM), generating
HTML on the fly for the particular item requested. In then case of
offline HTML, all items would need to be processed in advance. So for
many project updates and few documentation users the dynamic solution is
likely to be much more resource friendly - of course for large numbers
of concurrent documentation users, the numbers might as well turn.

But another possible benefit of dynamically generated documentation is
that any improvements/fixes to DDOX would immediately be available for
all projects and versions, without having to regenerate everything.
Perhaps this is a more compelling argument than resource usage, I didn't
think of that initially.

Re: Automatically generated ddoc on code.dlang.org

On 2014-11-28 16:21, Sönke Ludwig wrote:

It's just that the disk space of the VM that is currently in use is
already quite limited. If I'd have a dedicated server (or someone else
would donate one), that would definitely change the significance.

I see.

But another possible benefit of dynamically generated documentation is
that any improvements/fixes to DDOX would immediately be available for
all projects and versions, without having to regenerate everything.
Perhaps this is a more compelling argument than resource usage, I didn't
think of that initially.

I would be satisfied either way, as long as it's fast enough ;)

/Jacob Carlborg