Traditional data analysis tools
1, Excel as a spreadsheet software, suitable for simple statistics (grouping / summing, etc.) needs, because of its convenience and functionality to meet the needs of many scenarios, so the actual become the most commonly used software tools for researchers. The disadvantage is that it is a single function, and can handle small data size. These two years Excel in big data (such as geographic visualization and network relationship analysis) also made some enhancements, but the application of limited capacity.
2, SPSS (SPSS Statistics) and SAS as a commercial statistical software, to provide research commonly used in classical statistical analysis (such as regression, variance, factor, multivariate analysis, etc.) processing. SAS is rich and powerful (including the ability to plot), and support programming to expand its analytical capabilities, suitable for complex and demanding statistical analysis.
Tools for data storage and management
Hadoop is now almost equivalent to Big Data. It is an open source distributed infrastructure for storing very large data sets in clusters of computers. You can increase or decrease the amount of data you have at will without worrying about hardware failure.Hadoop provides storage for any kind of massive data, powerful processing capabilities, and the ability to work in almost unlimited parallel.
Hadoop is not for data beginners. To take full advantage of Hadoop's capabilities, you need to understand Java.Learning Java can be time-consuming, but Hadoop is definitely worth the effort, as a large number of companies and technologies rely on it and even become integrated with it.
Tools used for data cleansing
Before you can data mine, you should cleanse your data.OpenRefine is now an open source tool used to specifically cleanse messy data. Thereby enabling you to easily and quickly explore large data sets that have some degree of unstructuredness.
Tools used for data mining
Data mining as an important area of big data applications, based on traditional statistical analysis, more emphasis on the provision of machine learning methods, focusing on high-dimensional space under the complexity of the data association relationship and deduction capabilities. The representative is SPSS Modeler, SPSS Modeler's statistical functions are relatively limited, mainly to provide the implementation of machine learning algorithms (decision trees, neuron networks, classification, clustering and prediction, etc.) for commercial mining. At the same time, its data preprocessing and results to assist in the analysis is quite convenient, which is particularly suitable for rapid mining in a commercial environment. However, as far as processing power is concerned, it actually feels difficult to cope with data scales of more than hundreds of millions of dollars.
Programming languages commonly used for big data
1, R language is a language used for statistical analysis and mapping. If the above mentioned data mining and statistical software doesn't fulfill your needs then R language will surely help. In fact if you are going to be a data scientist, understanding the R language is a must-have skill.2. Python language - the biggest advantage is in text processing as well as large data volume processing scenarios and is easy to develop. In the related analytics field, Python is replacing R with more and more momentum.
Learning a single tool in your data career is hardly a one-trick pony. The tools available today are getting easier to use and more powerful, but there are times when it's better to program yourself. Even if you're not a professional programmer, understanding the basics of how these languages work can be beneficial to understanding how many of the tools work and how to use them.