查看: 140|回复: 0

Kafka as a Messaging System

[复制链接]
论坛徽章:
21
投资理财徽章
日期:2015-04-02 14:37:21Hadoop研习者初级
日期:2017-01-18 16:14:39机器学习徽章
日期:2017-07-27 17:25:09机器学习徽章
日期:2017-08-10 15:34:18python徽章
日期:2018-06-07 16:21:43python徽章
日期:2018-06-21 17:15:38人工智能徽章
日期:2018-07-19 15:00:17OpenAI课程徽章
日期:2018-08-16 15:34:25Kaggle徽章
日期:2019-05-16 15:54:26Hadoop研习者中级
日期:2017-01-06 10:48:29Hadoop研习者初级
日期:2016-12-29 16:44:02Openstack徽章
日期:2016-11-17 15:59:19
发表于 2019-4-13 19:27 | 显示全部楼层 |阅读模式

Kafka

How does Kafka's notion of Streams compare to a traditional enterprise messaging system?
Messaging traditionally has two models: queuing and publish-subscribe. In a queue, a pool of consumers may read from a server and each record goes to one of them; in publish-subscribe the record is broadcast to all consumers. Each of these two models has a strength and a weakness. The strength of queuing is that it allows you to divide up the processing of data over multiple consumer instances, which lets you scale your processing. Unfortunately, queues aren't multi-subscriber—once one process reads the data it's gone. Publish-subscribe allows you broadcast data to multiple processes, but has no way of scaling processing since every message goes to every subscriber.
The consumer group concept in Kafka generalizes these two concepts. As with a queue the consumer group allows you to divide up processing over a collection of processes (the members of the consumer group). As with publish-subscribe, Kafka allows you to broadcast messages to multiple consumer groups.
The advantage of Kafka's model is that every topic has both these properties—it can scale processing and is also multi-subscriber—there is no need to choose one or the other.
Kafka has stronger ordering guarantees than a traditional messaging system, too.
A traditional queue retains records in-order on the server, and if multiple consumers consume from the queue then the server hands out records in the order they are stored. However, although the server hands out records in order, the records are delivered asynchronously to consumers, so they may arrive out of order on different consumers. This effectively means the ordering of the records is lost in the presence of parallel consumption. Messaging systems often work around this by having a notion of "exclusive consumer" that allows only one process to consume from a queue, but of course this means that there is no parallelism in processing.
Kafka does it better. By having a notion of parallelism—the partition—within the topics, Kafka is able to provide both ordering guarantees and load balancing over a pool of consumer processes. This is achieved by assigning the partitions in the topic to the consumers in the consumer group so that each partition is consumed by exactly one consumer in the group. By doing this we ensure that the consumer is the only reader of that partition and consumes the data in order. Since there are many partitions this still balances the load over many consumer instances. Note however that there cannot be more consumer instances in a consumer group than partitions.

回复

使用道具 举报

您需要登录后才可以回帖 登录 | 立即注册 新浪微博登陆

本版积分规则

 

GMT+8, 2019-6-17 11:32 , Processed in 0.104725 second(s), 30 queries .

关闭

扫一扫加入
本版微信群