Big data challenges and opportunities exist side by side, the development of big data in the next few years will be shifted from the stage of inflated expectations and hype in the previous few years to the stage of rational development and application on the ground, and big data will gradually step into the period of rational development in the next few years. The future development of big data still has many challenges, but the outlook is still very optimistic.
Challenges to the development of big data
There are still many challenges to the development of big data, including seven major challenges: the business sector does not have a clear demand for big data leading to the gradual loss of data assets; the seriousness of data silos within the enterprise, resulting in the value of the data can not be fully tapped; the low availability of data, data quality is poor, resulting in the inability to utilize the data; the data-related management techniques and architecture is backward, resulting in the lack of big data processing capabilities; poor data security capabilities and awareness of prevention, resulting in data leakage; the lack of big data talent makes it difficult to carry out big data work; the more open big data is, the more valuable it is, but the lack of big data-related policies and regulations makes it difficult to strike a balance between data openness and privacy, and also makes it difficult to open it up in a better way.
>>>> Challenge 1: Business departments do not have clear big data needs
Many enterprise business departments do not understand big data, or the application scenarios and value of big data, so it is difficult to put forward the accurate needs of big data. Due to the lack of clarity in the needs of the business sector, the big data department is a non-profit department, and the enterprise decision makers are worried about investing a relatively large amount of cost, which has led to a lot of hesitation in building a big data department, or a lot of enterprises are in a wait-and-see attitude to try it out, which fundamentally affects the development of the enterprise in the direction of big data, and also prevents the enterprise from accumulating and tapping into its own data assets, and even due to the lack of application scenarios for the data, the Delete a lot of valuable historical data, resulting in the loss of enterprise data assets. Therefore, this aspect needs big data practitioners and experts together to promote and share big data application scenarios, so that more business people understand the value of big data.
>>>> Challenge 2: Serious data silos within the enterprise
The most important challenge for enterprises to start big data is the fragmentation of data. In many enterprises, especially large enterprises, data are often scattered in different departments, and these data exist in different data warehouses, different departments of the data technology may not be the same, which leads to the enterprise's own internal data can not be opened. If the data is not connected, the value of big data is very difficult to explore. Big data requires the correlation and integration of different data in order to better utilize the advantages of understanding customers and business. How to connect the data of different departments, and realize the technology and tools **** share, in order to better utilize the value of enterprise big data.
>>>> > Challenge 3: Low data availability and poor data quality
Many medium-sized as well as large enterprises are also generating large amounts of data every moment, but many enterprises pay little attention to the pre-processing stage of big data, resulting in very irregular data processing. Big data preprocessing stage requires the extraction of data to convert data into data types that are easy to process, cleaning and denoising of data to extract valid data and other operations. Even many enterprises have a lot of irregularities and irrationalities in the reporting of data. All of the above reasons, resulting in poor availability of enterprise data, poor data quality, inaccurate data. The significance of big data is not only to collect a huge scale of data information, there is also a good pre-processing of the collected data processing, it is possible for data analysis and data mining personnel from the availability of big data to extract valuable information.Sybase data show that high-quality data data application can significantly improve the business performance of the enterprise, the availability of data to improve the 10%, the enterprise's performance improves by at least 10 percent.
>>>> Challenge 4: Data-related management technology and architecture
The challenges of technical architecture include the following: (1) Traditional database deployments can't handle terabytes of data, and the rapidly growing volume of data exceeds the management capabilities of traditional databases. How to build a distributed data warehouse, and can easily expand a large number of servers has become a challenge for many traditional enterprises; (2) many enterprises use traditional database technology, in the beginning of the design did not consider the diversity of data categories, especially for structured data, semi-structured and unstructured data compatibility; (3) traditional enterprise database, the data processing time requirements of the data is not very high, these data statistical results often lag one or two days to be counted. However, big data requires real-time data processing for minute or even second-level calculations. Traditional database architects lack the ability to process data in real time; (4) Massive data requires a good network architecture and a strong data center to support it, and the operation and maintenance of the data center will also become a challenge. How to ensure data stability, support high concurrency at the same time, reduce the low load situation of the server, become a key work of the massive data center operation and maintenance.
>>>> Challenge 5: Data Security
Networked life makes it easier for criminals to get information about people, and there are more criminal means that are not easy to be tracked and prevented, and there may be more sophisticated scams. How to ensure the security of users' information has become a very important topic in the era of big data. With more and more online data, the motivation of hackers to commit crimes is stronger than ever. The leakage of sensitive personal information such as passwords of some well-known websites and the theft of users' data due to system vulnerabilities have already awakened us to the need to strengthen the construction of big data network security. In addition, the continuous increase of big data, the physical security requirements for data storage will be higher and higher, thus putting forward higher requirements for multiple copies of data and disaster recovery mechanisms. At present, the data security of many traditional enterprises is worrying.
>>>> > Challenge 6: the lack of big data talent
Every aspect of big data construction needs to rely on professionals to complete, so it is necessary to cultivate and create a team of big data construction professionals who have mastered the big data technology, know how to manage, and have experience in the application of big data. The current lack of big data-related talent will hinder the development of the big data market. According to Gartner's forecast, by 2015, there will be 4.4 million new big data-related jobs worldwide, and 25% of organizations will have a chief data officer position. Big data-related positions require a combination of people with a comprehensive knowledge of math, statistics, data analytics, machine learning and natural language processing. In the future, big data will appear about 1 million talent gap, in various industries big data in high-end talents will become the hottest talent, covering big data data development engineers, big data analysts, data architects, big data backend development engineers, algorithm engineers and other directions. Therefore, it is necessary for colleges and universities and enterprises*** to work together to cultivate and excavate. The biggest problem at present is that many colleges and universities lack big data, so enterprises with big data should jointly train talents with schools.
>>>> > Challenge 7: The trade-off between data openness and privacy
Today, with the growing importance of big data applications, the openness*** of data resources has become the key to maintaining an advantage in the data war. The application of ****enjoyment of commercial and personal data can not only promote the development of related industries, but also bring great convenience to our lives. Due to the lack of unified planning in the construction of government, enterprise and industry informatization systems, and the lack of unified standards between systems, many "information islands" have been formed, and the degree of data openness is low due to administrative monopolies and commercial interests, which creates a great obstacle to the use of data. Another important factor that restricts China's data resources openness and **** enjoyment is the imperfect policies and regulations, and the lack of corresponding legislation on big data mining. It is impossible to both guarantee *** enjoyment and prevent abuse. Therefore, the establishment of a benign development of the data *** enjoyment ecosystem is a chop that China's big data development needs to move past. At the same time, how to balance openness and privacy is also the biggest problem faced in the process of opening up big data. How to effectively protect the privacy of citizens and enterprises while promoting the full openness, application and *** enjoyment of data, and gradually strengthen privacy legislation, will be a major challenge in the era of big data.
The development trend of big data
While big data is still in its infancy and there are many challenges, the future development is still very optimistic. The development of big data presents eight trends: data resourcefulness, will become the most valuable asset; big data in more traditional industries in the enterprise management landing; big data and traditional business intelligence fusion, industry customized solutions will emerge; data will be more and more open, data *** enjoyment of the alliance will emerge; big data security is more and more attention to the big data security market will become more and more important; big data to promote the smart city development, for the engine of the smart city; big data will give rise to a number of new jobs and corresponding professions; big data in many ways to improve our lives.
>>>> Trend 1: data resourcing, will become the most valuable asset
With the development of big data applications, the value of big data can be fully embodied, big data in the enterprise and social level to become an important strategic resource, the data has become the new strategic high ground, is the new focus of the everyone to grab. The Wall Street Journal touted in a report titled "Big Data, Big Impact" that data has become a new asset class, like currency or gold.Companies such as Google, Facebook, Amazon, Tencent, Baidu, Alibaba, and 360 are using the power of big data to achieve greater commercial success, and financial and telecommunication firms are also utilizing big data to enhance their own competitiveness. We have reason to believe that big data will continue to be an asset for organizations and businesses, and a powerful weapon for improving their competitiveness.
>>>>Trend 2: Big data in more traditional industries in the enterprise management landing
A new technology is often applied in a few industries to achieve good results, there is a strong demonstration effect on other industries. At present, big data has been better applied in large Internet enterprises, big data in other industries, especially telecommunications and finance are also gradually achieved results in a variety of application scenarios. Therefore, we have reason to believe that big data, as a tool to create new value from data, will be applied in enterprises in many industries, bringing extensive social value. Big data will be useful in helping enterprises better understand and meet customer needs and potential needs, and better applied in business operation intelligent monitoring, refined enterprise operation, customer lifecycle management, refined marketing, operation analysis and strategic analysis. Enterprise management has both art and science, I believe that big data in the scientific management of enterprises have more significant promotion, so that more enterprises embracing big data to realize the wisdom of enterprise management.
>>>> Trend 3: The integration of big data and traditional business intelligence, industry customized solutions will emerge
From the field of traditional business intelligence will be big data as an additional source of data, while big data practitioners believe that the traditional business intelligence is only a method to deal with a small amount of data in their field. Big data users prefer a holistic solution that not only collects, processes and analyzes business data within the organization, but also brings in unstructured data from the Internet such as web browsing, microblogging and messaging. In addition to this, it is also hoped that the location information of mobile devices can be combined, so that the enterprise can form a comprehensive and complete platform for the development of data value. After all, whether it is big data or business intelligence, the purpose is to serve for analysis, and the data is fully integrated, which is more conducive to the discovery of new business opportunities, which is big data business intelligence. At the same time, due to the differences in the industry, it is difficult to develop a set of big data business intelligence analysis system applicable to all industries, therefore, in some larger industry markets, big data service providers will provide big data services with more customized business intelligence solutions. We believe that more customized big data BI solutions will emerge in telecom, finance, retail and other industries.
>>>> Trend 4: Data will become more and more open, and data **** enjoyment alliance will appear
The more connected big data is, the more valuable it is, and the more open it is, the more valuable it is. In particular, the public **** business and Internet companies will be more and more open data. We see that the governments of the United States, the United Kingdom, Australia and other countries are making efforts to make data on government and public **** undertakings. And some cities and departments in China are also gradually working on data openness. For example, Beijing started trial operation of the government data resource network in 2012, and officially opened it at the end of 2013; Shanghai launched a pilot project to open up government data resources in 2012, with data related to geographic location, transportation, economic statistics, and qualifications; and in 2014, Guizhou Province also joined in data opening, with Guizhou on the Cloud officially going online in October. For different industries, the more ****ed up the data is, the more valuable it is too. If each hospital wants to obtain more disease characteristics library as well as drug efficacy information, then it needs to ****share medical information across the country, or even the world, so that it can be analyzed through the platform to obtain greater value. We believe that there will be a trend of ****sharing of data, and data alliances in different fields will emerge.
>>>>Trend 5: Big data security is getting more and more attention, and the big data security market will become more and more important
With the value of data becoming more and more important, the security and stability of big data will also be gradually emphasized. The network and digital life also make it easier for criminals to obtain information about others, and there are also more scams and crimes, so in the era of big data, whether it is for the protection of the data itself, or for the security of some of the information evolved from the data, the high demand for big data analytics will be crucial to the enterprise. Big data security is corresponding to the big data business, compared with traditional security, the biggest difference between big data security is that security vendors think about security issues when the first thing to do is to carry out business analysis, and to find out the threats against the business of big data, and then put forward targeted solutions. For example, for the data storage scenario, many companies currently use open source software such as Hadoop technology to solve big data problems, due to its open source nature, but its security problems are also prominent. Therefore, the market needs more professional security vendors to provide professional services for different big data security issues.
>>>>>Trend 6: Big data promotes the development of smart cities for the engine of smart cities
With the development of big data, big data will play an increasingly important role in smart cities. As the gathering of population brings pressure to the city in various aspects such as transportation, medical care, construction, etc., it is necessary for the city to be able to carry out resource layout and deployment in a more reasonable way, and the smart city is the optimal solution for the transformation of urban governance. Smart city is to realize efficient government management, convenient livelihood services and sustainable industrial development of the city through the interconnection ability, comprehensive perception ability and information utilization ability of things to things, things to people and people to people, and through the new generation of information technology such as Internet of Things, mobile Internet and cloud computing. The biggest difference between the smart city and the previous concept of digital city is the intelligent processing of the information obtained from the perception layer. From urban digitalization to urban intelligence, the key is to achieve intelligent processing of digital information, the core of which is the introduction of big data processing technology. Big data is the core wisdom engine of the smart city. Intelligent security, intelligent transportation, intelligent medical care, intelligent city management, etc., are all smart city application areas based on big data.
>>>> Trend 7: Big Data will give rise to a number of new jobs and corresponding professional
The emergence of a new industry, there will be a new demand for jobs, and the emergence of big data will also introduce a number of new jobs, such as big data analysts, data management specialists, big data algorithm engineers, data product manager etc. Data analytics talents with rich experience will become a scarce resource, and data-driven jobs will show explosive growth. And due to the strong market demand, colleges and universities will gradually open big data-related majors to train corresponding professionals. Enterprises will also work closely with colleges and universities to assist them in jointly cultivating big data talents. For example, in 2014, IBM comprehensively promoted cooperation with universities in the field of big data, introduced a strong R&D team and business partners, and promoted industry-oriented industry-university-research innovation cooperation in "big data platform" and "big data analysis" as well as the construction of a systematic knowledge system. It also promotes industry-university-research cooperation and systematic knowledge system construction of "Big Data Platform" and "Big Data Analytics" as well as the cultivation of high-value talents, and builds credit courses related to Big Data in line with China's teaching characteristics and talents' needs in order to prepare for the construction of specialties.
>>>> Trend 8: Big data is improving our lives in many ways
Big data is not only used in business and government, but also applied to our lives. In terms of health: we can use smart bracelet monitoring, tracking our sleep patterns to understand the quality of sleep; we can use smart blood pressure monitors, smart heart rate meter remote monitoring in the remote home of the elderly health, so that far away from the other side of the field workers more assured; in terms of travel: we can use smart navigation travel GPS data to understand the traffic situation, and according to the congestion situation Real-time route optimization. In terms of home life: big data will become the core of the smart home, smart home appliances to achieve anthropomorphic intelligence, the product through the sensors and control chips to capture and process information, can be based on the residential space environment and user needs to automatically set the control, and even put forward proposals to optimize the quality of life, such as our refrigerator may early in the morning every day to suggest us the day's recipes.