1, batch

Batch processing is a common demand in the evening of big data processing, batch processing is mainly operated on large-volume static datasets, and return the results after the completion of the accounting process. In view of such a processing model, batch processing has an obvious drawback, that is, in the face of large-scale data, in the power of accounting and processing, not as satisfactory.

Nowadays, batch processing is extremely good at dealing with a lot of persistent data, and is therefore often used to analyze historical data.

2, stream processing

After the batch processing of another common need, is the flow of processing for real-time data into the system for accounting operations, the results of the process is immediately available, and will continue to be updated with the arrival of new data.

In real-time, stream processing is excellent, but stream processing at the same time can only handle a (true stream processing) or a very small number of (micro-batch processing, Micro-batch Processing) data, different records only to maintain the status of the fewest number of different records, the hardware requirements are also higher.

3, batch processing + stream processing

In the practice of the use of evening, batch processing and stream processing together with the existence of many scenarios, hybrid processing framework is designed to deal with such issues. Provide a generalized processing solution for data processing , not only can supply the methods needed to process data , together with their own integration items , libraries , things , can meet the graphical analysis , machine learning , interactive query and other scenarios .

About what are the common processing frameworks for big data, Green Vine editor will share with you here. If you have a strong interest in big data engineering, I hope this article can help you. If you still want to learn more about data analysts, big data engineers tips and materials, you can click on other articles on this site to learn.