RejectedSoftware Forums

Sign up

Pages: 1 2

Expose and consume Thrift services

As I understand, right now vibe.d can host REST based services (JSON). But it would be awesome to be able to expose and consume Thrift Services too:

  • 'non-blocking' (fiber based)
  • support SSL (so you can use it for client-server RPC over internet)
  • multiplexing (configured through routing)

Do you think would be feasible?
Is it something like this planed or in the works?

Thanks

Re: Expose and consume Thrift services

On Fri, 30 May 2014 19:07:52 GMT, Tavi wrote:

As I understand, right now vibe.d can host REST based services (JSON). But it would be awesome to be able to expose and consume Thrift Services too:

  • 'non-blocking' (fiber based)
  • support SSL (so you can use it for client-server RPC over internet)
  • multiplexing (configured through routing)

Do you think would be feasible?
Is it something like this planed or in the works?

Thanks

What I know is that David Nadlinger made a D port of Thrift some years ago (https://github.com/apache/thrift/tree/master/lib/d). There is also a mirror that provides a DUB package. I'm unfortunately not familiar with the Thrift API, but since it's (also) based on libevent, I suspect that it should be relatively easy to use it together with vibe.d seamlessly.

Re: Expose and consume Thrift services

On Fri, 30 May 2014 19:38:35 GMT, Sönke Ludwig wrote:

There is also a mirror that provides a DUB package.

Last time I checked, that package unfortunately was not up to date with the latest fixes in the official repository. If/when the Dub registry gets support for adding packages with the root (dub.json) in a subdirectory, then I'd like to take over the package and make it point to the upstream (Apache) Git repository.

I'm unfortunately not familiar with the Thrift API, but since it's (also) based on libevent, I suspect that it should be relatively easy to use it together with vibe.d seamlessly.

libevent is actually not integrated into the Thrift transport model all that well. You can only use a special server implementation with it, and since vibe.d was not really around back when I wrote the thing, I also had to write my own async I/O implementation (based on libevent/fibers) for the client side.

However, it is possible to adapt from a vibe.d Stream to a Thrift Transport quite easily both on the client and the server side. I've successfully used the following implementation in a small prototype that unfortunately hasn't made it to production yet:

https://gist.github.com/klickverbot/539deb163c6b7cf73843

There is some impedance mismatch between the two interfaces, but it's not too bad. There is also a listenThrift method that allows you to quickly create a Thrift server, very similar to listenTCP or listenHTTP.

Hope this helps,
David

Re: Expose and consume Thrift services

Oh, what I forgot to mention in my last post:

On Fri, 30 May 2014 19:07:52 GMT, Tavi wrote:

  • 'non-blocking' (fiber based)
  • support SSL (so you can use it for client-server RPC over internet)

When using the simple thrift_vibe adapter, these are taken care of by using the vibe.d facilities. In fact, I implemented the initial support for using SSL client/server certificates in vibe.d a while back specifically to be able to tunnel Thrift connections.

  • multiplexing (configured through routing)

Multiplexing multiple services onto one port/connection was only added very recently to Thrift, i.e. after I wrote the D implementation. Nobody has updated the D library to support it since, though it should not be too hard to add (see https://issues.apache.org/jira/browse/THRIFT-1915). How do you envision this to interact with the routing mechanism?

Best,
David

Re: Expose and consume Thrift services

I am not sure if RPC is Thrift's greatest strength.

It excels as a data storage/interchange format, it is phenomenal at that.

Cassandra was one of the biggest known users of Thrift RPC and they have replaced it with an alternative RPC mechanism in their latest versions.

Why not just return Thrift byte array from a REST endpoint using a different content type, e.g.

application/thrift

or something like that?

That would give you access to the nice semantics of REST services but with a more compact data format than JSON.

Re: Expose and consume Thrift services

On Wed, 04 Jun 2014 17:06:41 GMT, Jacek Furmankiewicz wrote:

I am not sure if RPC is Thrift's greatest strength. […] That would give you access to the nice semantics of REST services but with a more compact data format than JSON.

What are "the nice semantics of REST services" when it comes to RPC?

I'd be quick to agree that Thrift's RPC mechanism is rather basic – for example, it lacks a way to combine together multiple calls, and many other more advanced features. But I don't see how a REST interface – I assume you are referring to a standard restful HTTP API – would be more semantically more expressive.

David

Re: Expose and consume Thrift services

Am 04.06.2014 15:06, schrieb David Nadlinger:

On Fri, 30 May 2014 19:38:35 GMT, Sönke Ludwig wrote:

There is also a mirror that provides a DUB package.

Last time I checked, that package unfortunately was not up to date with the latest fixes in the official repository. If/when the Dub registry gets support for adding packages with the root (dub.json) in a subdirectory, then I'd like to take over the package and make it point to the upstream (Apache) Git repository.

I'm unfortunately not familiar with the Thrift API, but since it's (also) based on libevent, I suspect that it should be relatively easy to use it together with vibe.d seamlessly.

libevent is actually not integrated into the Thrift transport model all that well. You can only use a special server implementation with it, and since vibe.d was not really around back when I wrote the thing, I also had to write my own async I/O implementation (based on libevent/fibers) for the client side.

However, it is possible to adapt from a vibe.d Stream to a Thrift Transport quite easily both on the client and the server side. I've successfully used the following implementation in a small prototype that unfortunately hasn't made it to production yet:

https://gist.github.com/klickverbot/539deb163c6b7cf73843

There is some impedance mismatch between the two interfaces, but it's not too bad. There is also a listenThrift method that allows you to quickly create a Thrift server, very similar to listenTCP or listenHTTP.

Hope this helps,
David

I've added a link to the Gist in the wiki:
https://github.com/rejectedsoftware/vibe.d/wiki

Re: Expose and consume Thrift services

Depends what you want to do. You could expose HTTP REST endpoints with GET and query params and that is as basic as it gets.

Re: Expose and consume Thrift services

On Wed, 04 Jun 2014 21:34:10 +0200, Sönke Ludwig wrote:

I've added a link to the Gist in the wiki:
https://github.com/rejectedsoftware/vibe.d/wiki

I should probably just create a Dub package for it, since it neither really belongs into Thrift nor into Vibe.d…

David

Re: Expose and consume Thrift services

On Wed, 04 Jun 2014 13:06:00 GMT, David Nadlinger wrote:

On Fri, 30 May 2014 19:38:35 GMT, Sönke Ludwig wrote:

There is also a mirror that provides a DUB package.

Last time I checked, that package unfortunately was not up to date with the latest fixes in the official repository. If/when the Dub registry gets support for adding packages with the root (dub.json) in a subdirectory, then I'd like to take over the package and make it point to the upstream (Apache) Git repository.

That would be great. The currently registered one indeed does not compile.

However, it is possible to adapt from a vibe.d Stream to a Thrift Transport quite easily both on the client and the server side. I've successfully used the following implementation in a small prototype that unfortunately hasn't made it to production yet:

https://gist.github.com/klickverbot/539deb163c6b7cf73843

There is some impedance mismatch between the two interfaces, but it's not too bad. There is also a listenThrift method that allows you to quickly create a Thrift server, very similar to listenTCP or listenHTTP.

Nice. I've started to implement it too and got something similar :-) The main difference is that I took the vibe HTTP Server code and adjusted so that a service processor object would be created per connection. I think this would be practical, if you expose an API over the public network and want to add some sort of authentication:

interface Foo { void foo(); }
class FooImpl { override void foo {} }
	
shared static this {
  auto thriftServerSettings = new VibedThriftServerSettings;
  thriftServerSettings.port = 9090;
  thriftServerSettings.bindAddresses = ["::1", "127.0.0.1"];
 
  // use default TBinaryProtocol, TBufferedTransport
  alias FooService = VibedThriftService!(Foo, FooImpl); // use default TBinaryProtocol, TBufferedTransport
 
  listenThrift!FooService(thriftServerSettings);
}

For each connection a new FooImpl processor object will be created. Maybe using a free list (or typed custom allocator) would be better.

https://gist.github.com/tavi-cacina/f9d79bbfb7897e91e9a9

I'm not experienced with D, Vibe or Thrift, so my code and templates are not so clean (yet), but I would like to help make this combo production ready

Regards,
Tavi

Pages: 1 2