Forums  > Software  > Apache Pulsar  
     
Page 1 of 1
Display using:  

Maggette


Total Posts: 1139
Joined: Jun 2007
 
Posted: 2019-07-22 05:38
Hi,

I am living in the Kafka hell a lot. Stuff works, with competitive latency, throughput and it is quite robust and resilient. Still there are a couple of things that make me think using it for my current use case (some relevant features that make Kafka easy for DevOps teams are proprietary, I will also need some queue mechanism...)

Pulsar seems to check all the boxes. Anybody any insights?
Thx
pulsar project here

Ich kam hierher und sah dich und deine Leute lächeln, und sagte mir: Maggette, scheiss auf den small talk, lass lieber deine Fäuste sprechen...

victorin


Total Posts: 7
Joined: Oct 2010
 
Posted: 2019-07-23 08:43
I've been using rabbitmq for years and very happy with it

Maggette


Total Posts: 1139
Joined: Jun 2007
 
Posted: 2019-07-23 15:10
Thx victorin,

there are a couple of reasons that make rabbit rather inconvinient for my specific use case..amongst others:
- I need multiple users per message. My understanding of rabbitmq I need different qeues here
- no message encryption
- no producer message ack
- I like message ordering...


Ich kam hierher und sah dich und deine Leute lächeln, und sagte mir: Maggette, scheiss auf den small talk, lass lieber deine Fäuste sprechen...

EspressoLover


Total Posts: 377
Joined: Jan 2015
 
Posted: 2019-08-02 17:34
All I can say about Apache Pulsar, is I've heard only good things about it vis-a-vis Kafka. Using Bookeeper to abstract out the state from the broker layer seems like an unmitigated win. It's just a fundamentally better architecture. The biggest downside I see is that Pulsar has about two orders of magnitude fewer contributors than Kafka. I'm always hesitant to put a software project that has a high risk of dying at the core of my infrastructure.

To hijack this topic on a bit of a tangent (sorry maggette)... I've increasingly found myself reaching for a service mesh in places where my knee-jerk impulse is a message bus architecture. There's a lot of hard-earned distributed systems wisdom in the end-to-end principle. To the extent that state and logic can be moved from the core of the network to the endpoints of the network, that almost always reduces the brittleness of the system.

The modern tooling around service meshes makes it easy to replace a lot of the functionality of message brokers in a more peer-to-peer way. To start, it seems like at least half the message buses in the wild pretty much are just used as a form of service discovery. Instead of finding each other, producers/consumers just subscribe to a pre-determined topic. But Istio, or even just vanilla Kubernetes makes this a trivially easy problem to solve. Some other reasons to use a message broker:

* Load balancing. Envoy and/or HAProxy now does a much better job of this than Kafka/RabbitMQ
* Fault tolerance: Kubernetes+Istio can easily spin up new consumers/producers if one dies and handle failover.
* Encryption/Authentication: Handled by Envoy automagically
* Monitoring/Tracing: Istio definitely excels beyond any of the message brokers here.
* Buffering on bursts of data: Client/server requests with exponential backoff (which you get for free with Envoy) often achieves the same goal as middleware buffering for many workloads.
* Asynchronous requests: Most languages have great async libraries nowadays. Plus gRPC and/or Istio can help here.
* Streaming: gRPC/HTTP2 is now quite good at high-performant streaming
* Fanout: Probably the weakest pillar of the service mesh approach. There are some solutions, but definitely nothing as smooth or elegant as message brokers, I'll admit.

Anyway, I don't want to oversell the point I'm trying to make. There's still many many cases where a message bus is the right architecture. But I do think that people tend to have a cognitive bias to reach for a message broker before considering other solutions. It's very tempting to abstract everything into a single core system where all the state and logic can be centrally administered. But most times you tend to pay for the simplicity in the design phase, with a lot more headaches in the maintenance phase.

Good questions outrank easy answers. -Paul Samuelson

Maggette


Total Posts: 1139
Joined: Jun 2007
 
Posted: 2019-08-02 21:25
Hi,

yeah, there are several architecture choices that IMHO make Pulsar vastly superior to Kafka. We will start with Pulsar and hope that it meshes as well with Flink as Kafka does.

The contributor thing is a good point I didn't really think about. SO thanks for that. My hope is, if a tool is battle tested and is part of some big company infrastructure I will be fine. We will see.

And thanks for the Hijack. I am biased here. I am a simple person when it comes to that and tend to do things that worked well for me and I have experience with. This makes me fast and a high performer in the eyes of many, but I guess it probably leads to poor design and architecture choices at times. So I do reach out to a classical message based system broker thing without considering alternatives (if it is not super simple and it is obvious that it would be an overkill)

Ich kam hierher und sah dich und deine Leute lächeln, und sagte mir: Maggette, scheiss auf den small talk, lass lieber deine Fäuste sprechen...

EspressoLover


Total Posts: 377
Joined: Jan 2015
 
Posted: 2019-08-02 22:16
Well, I'd say being hard-nosed practical and gravitating to proven and familiar technologies is rarely a fault in our line of work. Smiley

Good questions outrank easy answers. -Paul Samuelson

TonyC
Nuclear Energy Trader

Total Posts: 1298
Joined: May 2004
 
Posted: 2019-08-14 06:41
> yeah, there are several architecture choices that
> IMHO make Pulsar vastly superior to Kafka.

" Kafka on Kubernetes - Just Because You can Doesn't Mean You Should "
http://meetu.ps/e/H33Ff/tdKDv/d

flaneur/boulevardier/remittance man/energy trader
Previous Thread :: Next Thread 
Page 1 of 1