The three methods of data processing are: data cleaning, data conversion, and data analysis.

I. Data Cleaning

Data cleaning refers to screening, filtering, and correcting the raw data in order to make it conform to the requirements of analysis. There may be errors, missing, duplicates, outliers and other problems in the original data, which will affect the quality of the data and the results of the analysis. Therefore, data cleaning is the first and most critical step in data analysis.

The specific methods of data cleaning include the following:

1. Delete duplicate data: If there is duplicate data in the data set, it needs to be deleted to avoid the impact on the analysis results.

2, fill the missing values: If there are missing values in the data set, they need to be filled to ensure the completeness and accuracy of the data. Filling method can be mean filling, median filling, plurality filling and so on.

3, remove outliers: If there are outliers in the data set, they need to be removed to avoid interference with the analysis results.

4, check the data format: the format of the data should meet the requirements, such as date format, digital format. If the format does not meet the requirements, need to be adjusted.

5, standardized data: If there are inconsistent units in the data set, it needs to be standardized to facilitate analysis and comparison.

Two, data conversion

Data conversion refers to the original data into a form suitable for analysis. Raw data may exist in different forms and structures and need to be converted so that they can be analyzed.

The specific methods of data conversion include the following:

1. Data type conversion: the type of data will be converted, such as the type of string into a digital type, the date type into a timestamp type.

2, data structure conversion: the structure of the data to be converted, such as the wide table is converted to a long table, multi-dimensional array is converted to a one-dimensional array.

3, data merger: multiple data sets and into a data set for analysis.

4, data splitting: split a data set into multiple data sets to facilitate analysis.

5, pivot table: the data will be pivoted to facilitate data analysis and comparison.

Three, data analysis

Data analysis refers to the statistics, analysis and modeling of data in order to mine the information and laws in the data. Data analysis is the ultimate purpose of data processing, but also the most valuable part of data processing.

The specific methods of data analysis include the following:

1. Descriptive statistical analysis: descriptive statistical analysis of data, such as calculating the mean, median, variance, etc., in order to understand the distribution and characteristics of data.

2, exploratory data analysis: exploratory data analysis of data, such as plotting histograms, scatter plots, box-and-line plots, etc., in order to facilitate the discovery of patterns and relationships in the data.

3, hypothesis testing: hypothesis testing of data to verify the correctness and reliability of the research hypothesis.

4, data modeling: modeling the data in order to mine the information and laws in the data and make predictions and decisions.

5, data visualization: visualize data to facilitate the presentation of the results and conclusions of data analysis to others.

Data Processing and Data Management:

Data processing is the process of extracting valuable information from a large amount of raw data, i.e., the process of converting data into information. Mainly on the input of various forms of data processing and organizing, the process contains the collection of data, storage, processing, classification, grouping, calculation, sorting, conversion, retrieval and dissemination of the evolution and derivation of the whole process.

Data management refers to the data collection, organization, storage, maintenance, retrieval, transmission and other operations, is the basic link of the data processing business, and is a must for all data processing process **** the same part.

Data processing, usually simple calculations, and data processing operations in the processing of calculations due to different businesses, the need to write applications to address the needs of the business.

While data management is more complex, due to the explosive growth of available data, and the variety of data, from the data management point of view, not only to use the data, but also to effectively manage the data. Therefore, there is a need for a common, easy-to-use and efficient management software to manage data effectively.

Data processing and data management are linked, the advantages and disadvantages of data management technology will have a direct impact on the efficiency of data processing. The database technology is a branch of computer applications that has been researched and developed and perfected in response to this demand goal. Big data processing data era concept of the three major changes: to all not sampling, to efficiency not absolute precision, to correlation not causation.

There are actually many specific big data processing methods, but according to a long time of practice, Tianhou data summarized a basic big data processing process, and this process should be able to help you rationalize the processing of big data. The whole process can be summarized in four steps, which are collection, import and preprocessing, statistics and analysis, and mining.