In today's world, with the rapid development and progress of IT Internet information technology.

At present, the big data industry is also getting hotter and hotter, which leads to the domestic big data talent is also extremely lack of the following IT training/introduction about the Hadoop environment to manage big data storage skills.

1, distributed storage traditionalized centralized storage has existed for some time.

But big data is not really suitable for centralized storage architecture.

Hadoop is designed to bring computation closer to the data nodes while employing the massive horizontal scaling capabilities of the HDFS file system.

Though, the usual solution to Hadoop's inefficiency in managing its own data is to store Hadoop data on a SAN.

But this also creates its own performance and scale bottlenecks.

Now, if you process all your data through a centralized SAN processor, it goes against the distributed and parallelized nature of Hadoop.

You either have to manage multiple SANs for different data nodes or centralize all of your data nodes into one SAN.

But Hadoop is a distributed application, and as such it should be running on distributed storage, so that the storage retains the same flexibility as Hadoop itself, though it also requires embracing a software-defined storage solution and running it on commercial servers, which is naturally more efficient compared to bottlenecked Hadoop.

2. Hyper-converged vs. distributed Note that you should not confuse hyper-converged with distributed.

Some hyper-converged solutions are distributed storage, but often the term means that your applications and storage are kept on the same compute node.

This is trying to solve the problem of data localization, but it creates too much resource contention.

This Hadoop application and storage platform will compete for the same memory and CPU.

Hadoop runs on the proprietary application tier and distributed storage runs on the proprietary storage tier which is better.

After that, caching and tiering are used to address data localization and compensate for network performance loss.

3, avoid controller bottlenecks (ControllerChokePoint) to achieve an important aspect of the goal is - to avoid processing data through a single point such as a traditional controller.

Instead, make sure that the storage platform parallelization, performance can be significantly improved.

In addition, this solution offers incremental scalability.

Adding functionality to a data lake is as easy as throwing an x86 server in there.

A distributed storage platform will automatically add functionality and realign data if needed.

4. The key to mastering big data is deletion and compression techniques.

Usually, 70% to 90% of the data within a big data set is simplified.

In terms of petabytes of capacity, it can save tens of thousands of dollars in disk costs.

Modern platforms offer inline (vs. post-processing) de-duplication and compression, greatly reducing the capacity required to store data.

5. Consolidating Hadoop distributions Many large organizations have multiple Hadoop distributions.

Maybe it's because developers need it or the enterprise department has adapted to different versions.

Either way it often ends up being the maintenance and operation of these clusters.

Multiple Hadoop distribution storage can lead to inefficiencies once the massive amounts of data really start to impact an organization.

We can gain data efficiency by creating a single, de-duplicated and compressed data lake.6. VirtualizationHadoop virtualization has swept the enterprise market.

More than 80% of physical servers in many regions are now virtualized.

But there are also still many enterprises that shy away from virtualized Hadoop because of performance and data localization issues.

7. Creating a resilient data lakeCreating a data lake is not easy, but big data storage may be in demand.

We have many ways to do this, but which is the right one? This right architecture should be a dynamic, elastic data lake that can store data from all resources in multiple formats (architected, unstructured, semi-structured).

More importantly, it must support applications executing not on remote resources but on local data resources.