The key technologies of big data processing generally include: big data acquisition, big data preprocessing, big data storage and management, big data analysis and mining, big data presentation and application (big data retrieval, big data visualization, big data application, big data security, etc.).
First, big data collection technology
Data refers to various types of structured, semi-structured (or called weakly structured) and unstructured massive data obtained through RFID radio frequency data, sensor data, social network interaction data and mobile Internet data, etc., and is the root of the big data knowledge service model. The focus should be on breakthroughs in distributed high-speed and highly reliable data crawling or collection, high-speed data full image and other big data collection technologies; breakthroughs in high-speed data parsing, conversion and loading and other big data integration technologies; and the design of quality assessment models and the development of data quality technologies.
The Internet is a magical big network, big data development and software customization is also a model, here to provide the most detailed offer, if you really want to do, you can come here, the beginning of the number of this cell phone is one eight seven in the middle of the three children
Zero last is one four two five zero, in order to combine them can be found, I would like to say that unless you want to do or understand this aspect of the content, don't come if you're just scraping by.
Big data collection is generally divided into big data intelligent perception layer: mainly including data sensing system, network communication system, sensing adaptation system, intelligent identification system and hardware and software resources access system, to achieve the structured, semi-structured, unstructured massive data intelligent identification, positioning, tracking, access, transmission, signal conversion, monitoring, preliminary processing and management. It must focus on overcoming intelligent identification, sensing, adaptation, transmission, access and other technologies for big data sources. Basic Support Layer: Provide the basic support environment such as virtual servers, databases of structured, semi-structured and unstructured data, and IOT network resources required by the big data service platform. It focuses on tackling distributed virtual storage technology, visualization interface technology for big data acquisition, storage, organization, analysis and decision-making operations, network transmission and compression technology for big data, and privacy protection technology for big data.
Two, big data preprocessing technology
Mainly complete the analysis of the received data, extraction, cleaning and other operations.1) Extraction: Because the acquired data may have a variety of structures and types, the data extraction process can help us to transform these complex data into a single or easy to deal with the configuration, in order to achieve the purpose of rapid analysis and processing.2) Cleaning: For big data, not all of them are valuable. Not all valuable, some data is not what we care about, while some other data is completely wrong interference, so the data should be filtered through the "denoising" to extract valid data.
Three, big data storage and management technology
Big data storage and management to use the memory to store the collected data, the establishment of the corresponding database, and management and call. Focus on solving complex structured, semi-structured and unstructured big data management and processing technology. It mainly solves several key problems such as storable, representable, processable, reliable and effective transmission of big data. Develop reliable distributed file system (DFS), energy-efficiency optimized storage, computation into storage, big data de-redundancy, and efficient and low-cost big data storage technology; breakthrough in distributed non-relational big data management and processing technology, heterogeneous data data fusion technology, data organization technology, and research on big data modeling technology; breakthrough in big data indexing technology; breakthrough in big data movement, backup, replication, and other technologies; develop big data visualization technology. ; develop big data visualization technology.
Development of new database technology, database is divided into relational database, non-relational database and database caching system. Among them, non-relational databases mainly refer to NoSQL databases, which are divided into: key-value databases, column-store databases, graph-store databases and document databases and other types. Relational databases include traditional relational database systems and NewSQL databases.
Development of big data security technology. Improve data destruction, transparent encryption and decryption, distributed access control, data auditing and other technologies; breakthroughs in privacy protection and inference control, data authenticity identification and forensics, data holding integrity verification and other technologies.