My team and I have been doing some big data related work lately, and let me answer this question.

First the first question, what is a big data platform?

When we talk about a platform, we tend to know that there must be more than one thing in it, it's a collection of many things, and the same is true for the big data platform, first of all, if you use a few words to describe it then it is "it's a data solution", and then further analyze it is: Big Data Platform is a distributed data solution that is based on the idea of "big data", and it's a big data solution. Data platform is a distributed storage-based, integrated data acquisition, data cleaning, data flow, data analysis, data output and other tools set of a data solution. Its core mission is to provide data storage and data analysis services to target customers.

So what are its core components? There are various ways to realize it, so I'll cite one of the most typical big data platform structures as an illustration.

At present, whether domestic or foreign, the most widely used and the most typical big data platform is Hadoop as the core of the functional extension of the ecosystem, the industry called it Hadoop ecosystem, it is open source and free to use, what does it look like? Its face is basically this:

From the above figure we know that it is a set of Hadoop distributed file system as the core of the data processing tool set, the purpose is to provide users with data analysis services for an integrated solution.

When do you need a big data platform?

Simply put, when the total amount of data is so large that there is no way to store, analyze, and compute traditional stand-alone data solutions, a big data platform should be used.

For example, home computers are generally configured with 2TB size hard disk (storage capacity is equivalent to 18 128G iPhone), the general tens of thousands of dollars of commercial server capacity of about 32TB capacity, high-end stand-alone memory can reach more than 100TB, but the amount of data, such as if it is even larger, such as on the jump on an order of magnitude of 1,000TB, that is, 1PB! Around, the stand-alone system can not help, not only the storage capacity can not help, computing power can not cope with, because we know that the performance of a single computer has a limit, too much data disk retrieval reading speed will slow down, CPU and memory pressure will become large, this time the need to complete a data analysis task is going to take a very long time, then this time the big data platform comes in handy, a characteristic of the big data platform is that multiple data analyzers can be used to analyze the data, but also to analyze the data. A feature of the big data platform is that multiple computers form a cluster to fight collectively and in parallel, and theoretically can be expanded indefinitely.

I hope my answer can help you, there are any questions please leave a message in the comments section, but also welcome to online counseling