1. Data collection, build a data warehouse, data collection is to bury the data through the front-end, the interface log call flow data, database capture, the customer uploads their own data, the information base data to a variety of dimensions to save up, the feeling that some of the data is not useful (at first do only think about the function of some of the data was not collected, and then was the boss of a lecture).
2. Data cleansing/preprocessing: that is, the received data is simply processed, such as the ip into an address, filter out dirty data.
3. After the data can be processed on the data processing, data processing, data processing in many ways, generally divided into offline processing, real-time processing, offline processing is a daily timed processing, commonly used in Ali's maxComputer, hive, MapReduce, offline processing is mainly storm, spark, hadoop, through a number of data processing frameworks, you can calculate the data. Data processing framework, you can calculate the data into a variety of KPIs, here need to pay attention to, do not just think about the function, the main thing is to build up a variety of data dimensions, the basic data to do all, but also reusable, later on, you can put a variety of KPIs to show a combination of random.
4. Data display, data out of no use, to visualize, do MVP, is to quickly make an effect, not suitable for timely adjustment, this point is somewhat similar to the Scrum Agile development, data display can be used datav, God's policy, etc., the front-end of the good can be ignored, draw their own page.
The penetration of big data processing in various industries is getting deeper and deeper, for example, the financial industry needs to use big data systems combined with VaR (value at risk) or machine learning programs for credit risk control, retail, catering industry needs big data systems to achieve auxiliary sales decision-making, a variety of IOT scenarios need to be big data systems to continue to aggregate and analyze the time-series of data, major technology companies need to The big data analytics center is a big place to be.