And kafka was originally designed for the initial intention of the log statistical analysis, and now based on the background of the big data can also be done to analyze the operation of the data analysis statistics, and redis is the main scenario is a memory database, as a message queue for the reliability is too poor, and the speed is too dependent on network IO, the speed of the server on the local machine, and easy to appear in the data pile up problem, in a relatively lightweight occasions can be applied.
RabbitMQ, which follows the AMQP protocol, is developed in the inherently highly concurrent erlanng language, and is used for real-time messaging with high reliability requirements.
kafka is Linkedin in December 2010 open source message publishing subscription system , it is mainly used to deal with active streaming data , large data volume data processing .
RabbitMQ follows the AMQP protocol , RabbitMQ broker consists of Exchange, Binding, queue , which exchange and binding form the message routing key ; client Producer by connecting the channel and server to communicate . Consumer from the queue to get the message to consume (long connection , queue has a message will be pushed to the consumer side , consumer cycle from the input stream to read data ). rabbitMQ to broker as the center ; there is a message acknowledgement mechanism .
kafka follows the general MQ structure, producer, broker, consumer, consumer-centric, message consumption information is stored on the client consumer, the consumer according to the point of consumption, from the broker batch pull data; no message confirmation mechanism.
rabbitMQ is slightly inferior to kafka in terms of throughput, their starting point is not the same, rabbitMQ support for reliable delivery of messages, support for transactions, does not support batch operations; storage-based reliability requirements storage can be used in memory or hard disk.
kafka has a high throughput, internal message batch processing, zero-copy mechanism, data storage and acquisition is a local disk sequential batch operation, with O(1) complexity, message processing efficiency is very high.
rabbitMQ supports miror queues, where the main queue fails and the miror queue takes over.
kafka's broker supports a master/standby mode.
RabbitMQ's load balancing requires a separate loadbalancer for support.
kafka uses zookeeper to manage brokers and consumers in the cluster. You can register a topic with zookeeper; through the zookeeper's coordination mechanism, the producer saves information about the brokers that correspond to the topic, and sends it to the brokers either randomly or polled; and the producer can send information to the brokers randomly or polled. The producer can specify a slice based on semantics, and messages are sent to a slice of the broker.