Big data challenges and opportunities coexist. In the next few years, the development of big data will shift from the expected expansion stage and hype stage in previous years to the rational development stage and the landing application stage. In the next few years, big data will gradually enter a period of rational development. There are still many challenges in the future development of big data, but the prospects are still very optimistic. ?
Challenges of Big Data Development
At present, the development of big data still faces many challenges, including seven challenges: business departments have no clear demand for big data, resulting in the gradual loss of data assets; The serious data island within the enterprise leads to insufficient data value mining; Low data availability and poor data quality make the data unusable; Data-related management technology and architecture are backward, resulting in insufficient big data processing capacity; Poor data security ability and awareness of prevention, leading to data leakage; The lack of big data talents makes it difficult to carry out big data work; The more open big data is, the more valuable it is. The lack of policies and regulations related to big data makes it difficult to balance data openness and privacy, and it is also difficult to open it better.
Challenge 1: The business unit has no clear big data requirements.
Many enterprise business departments don't understand big data, and they don't understand the application scenarios and values of big data, so it is difficult to put forward the precise requirements of big data. Because the demand of the business department is not clear, and the big data department is a non-profit department, enterprise decision makers are worried about the relatively high cost, which leads many enterprises to hesitate when building the big data department, or many enterprises are in a wait-and-see state, which fundamentally affects the development of enterprises in the direction of big data, and also hinders enterprises from accumulating and mining their own data assets. Even because the data has no application scenarios, many valuable historical data are deleted, resulting in the loss of enterprise data assets. Therefore, big data practitioners and experts need to work together to promote and share big data application scenarios, so that more business people can understand the value of big data.
Challenge 2: Serious data silos within the enterprise
The biggest challenge for enterprises to start big data is the fragmentation of data. In many enterprises, especially large enterprises, data are often scattered in different departments, and these data exist in different data warehouses. The data technologies of different departments may be different, which makes it impossible to get through their own data. If you don't get through these data, the value of big data is difficult to tap. Big data needs the correlation and integration of different data in order to give full play to the advantages of understanding customers and business. How to get through the data of different departments and realize the sharing of technology and tools can better play the value of enterprise big data.
Challenge 3: Low data availability and poor data quality.
Many large and medium-sized enterprises are also generating a large amount of data all the time, but many enterprises do not pay enough attention to the preprocessing stage of big data, resulting in irregular data processing. In the preprocessing stage of big data, it is necessary to extract data, convert the data into data types that are easy to process, and clean and denoise the data to extract effective data. Even many enterprises have many irregular and unreasonable situations when reporting data. The above reasons lead to poor usability, poor data quality and inaccurate enterprise data. The significance of big data is not only to collect large-scale data information, but also to preprocess the collected data so that data analysts and data miners can extract valuable information from high-availability big data. Sybase data shows that the application of high-quality data can significantly improve the business performance of enterprises, the data availability can be improved by 10%, and the performance of enterprises can be improved by at least 10%.
Challenge 4: Data-related management technology and architecture
The challenges of technical architecture include the following aspects: (1) Traditional database deployment can't handle TB-level data, and the rapidly growing data exceeds the management capacity of traditional databases. How to build a distributed data warehouse and easily expand a large number of servers has become a challenge for many traditional enterprises. (2) Many enterprises adopt the traditional database technology, without considering the diversity of data categories at the beginning of design, especially the compatibility of structured data, semi-structured data and unstructured data; (3) The database of traditional enterprises does not require high data processing time, and the statistical results of these data are often counted after a day or two. But big data needs to process data in real time, and calculate in minutes or even seconds. Traditional database architects lack real-time data processing ability; (4) Massive data needs a good network architecture and a powerful data center to support it, and the operation and maintenance of the data center will also become a challenge. How to reduce the low load of the server while ensuring data stability and supporting high concurrency has become the key work for the operation and maintenance of massive data centers.
Challenge 5: Data security
Network life makes it easier for criminals to obtain human information, and there are more criminal means that are not easy to be tracked and prevented, and there may be more ingenious scams. How to ensure the information security of users has become a very important issue in the era of big data. There are more and more data on the Internet, and the motive of hacker crime is stronger than ever. Password leakage and system vulnerabilities of some well-known websites lead to the leakage of personal sensitive information such as user data theft, which reminds us to strengthen the security construction of big data networks. In addition, with the increase of big data, the physical security requirements for data storage will be higher and higher, which also puts forward higher requirements for multiple copies of data and disaster tolerance mechanism. At present, the data security of many traditional enterprises is worrying.
Challenge 6: Lack of big data talents
Every link of big data construction needs professionals to complete. Therefore, it is necessary to cultivate and bring up a professional team of big data construction with experience in big data technology, management and big data application. At present, the lack of talents related to big data will hinder the development of the big data market. According to Gartner's forecast, by 20 15, 4.4 million jobs related to big data will be created worldwide, and 25% of organizations will set up chief data officer jobs. Jobs related to big data need compound talents who can comprehensively control mathematics, statistics, data analysis, machine learning and natural language processing. In the future, there will be a talent gap of about 6.5438+million in big data, and middle and high-end talents in big data will become the hottest talents in various industries, covering data development engineers, big data analysts, data architects, big data background development engineers, algorithm engineer and other directions. Therefore, universities and enterprises need to work together to cultivate and excavate. At present, the biggest problem is that many colleges and universities lack big data, and enterprises with big data should jointly train talents with schools.
Challenge 7: Trade-off between data disclosure and privacy
Today, the application of big data is becoming more and more important, and the opening and sharing of data resources has become the key to maintaining the advantage in the data war. The application of business data and personal data can not only promote the development of related industries, but also bring great convenience to our lives. Due to the lack of unified planning and standards for government, enterprise and industry information system construction, many "information islands" have been formed, and limited by administrative monopoly and commercial interests, the degree of data openness is low, which has caused great obstacles to data utilization. Another important factor that restricts the opening and sharing of data resources in China is the imperfect policies and regulations, and the lack of corresponding legislation for big data mining. It is impossible to guarantee * * * enjoyment and prevent abuse. Therefore, the establishment of a benign data sharing ecosystem is a great progress in China's national data development. At the same time, how to balance openness and privacy is also the biggest problem in the process of big data opening. How to effectively protect the privacy of citizens and enterprises while promoting the full openness, application and enjoyment of data, and gradually strengthen privacy legislation will be a major challenge in the era of big data.
The development trend of big data
Although big data is still in its infancy and there are many challenges, the future development is still very optimistic. The development of big data presents eight major trends: data resources will become the most valuable assets; Big data has landed in the enterprise management of more traditional industries; With the integration of big data and traditional business intelligence, industry customized solutions will emerge; Data will become more and more open, and data sharing alliances will emerge; Big data security is getting more and more attention, and the big data security market will become more and more important; Big data promotes the development of smart cities and is the engine of smart cities; Big data will give birth to a number of new jobs and corresponding majors; Big data is improving our lives in many ways.
Trend 1: Data resources will become the most valuable assets.
With the development of big data applications, the value of big data can be fully reflected. Big data has become an important strategic resource at the enterprise and social levels. Data has become a new strategic commanding height and a new focus for everyone to grab. In a report entitled "Big Data, Big Impact", The Wall Street Journal claimed that data has become a new asset class, just like money or gold. Companies such as Google, Facebook, Amazon, Tencent, Baidu, Alibaba and 360 are using big data to achieve greater commercial success, and financial and telecommunications companies are also using big data to enhance their competitiveness. We have reason to believe that big data will continue to be an asset of institutions and enterprises and a powerful weapon to enhance competitiveness.
Trend 2: Big data is managed in more traditional industries.
A new technology is often applied in a few industries and has achieved good results, which has a strong demonstration role for other industries. At present, big data has been well applied in large Internet companies, and big data in other industries, especially telecommunications and finance, has gradually achieved results in various application scenarios. Therefore, we have reason to believe that big data, as a tool to create new value from data, will be applied in enterprises in many industries and bring extensive social value. Big data will help enterprises better understand and meet customer needs and potential needs, and be better applied to intelligent monitoring of business operations, refined operation of enterprises, customer life cycle management, refined marketing, business analysis and strategic analysis. Enterprise management has both art and science. I believe that big data has a more significant role in promoting the scientific management of enterprises, so that more enterprises that embrace big data can realize smart enterprise management.
Trend 3: Big data is integrated with traditional business intelligence, and industry customized solutions will emerge.
People in the field of traditional business intelligence regard big data as a new data source, while big data practitioners think that traditional business intelligence is only a method when dealing with a small amount of data in their field. Big data users hope to get a whole solution, that is, not only to collect, process and analyze business data within the enterprise, but also to introduce unstructured data such as web browsing, Weibo and WeChat on the Internet. In addition, we hope to combine the location information of mobile devices to form a comprehensive and complete data value development platform for enterprises. After all, whether it is big data or business intelligence, the purpose is to serve analysis, and the comprehensive integration of data is more conducive to discovering new business opportunities. This is big data business intelligence. At the same time, due to the differences in industries, it is difficult to develop a set of big data business intelligence analysis systems suitable for various industries. Therefore, in some large-scale industry markets, big data service providers will provide more customized business intelligence solutions for big data services. We believe that there will be more big data business intelligence customized solutions in telecom, finance, retail and other industries.
Trend 4: Data will become more and more open, and data sharing alliances will emerge.
The more relevant big data is, the more valuable and open it is. In particular, public enterprises and Internet companies will have more and more open data. We can see that the governments of the United States, Britain, Australia and other countries have made great efforts in the data of government and public utilities. And some cities and departments in China are gradually opening up data. For example, Beijing started the trial operation of the government data resource network on 20 12 and officially opened it at the end of 20 13; Shanghai started the pilot project of opening government data resources on 20 12, and the data involved geographical location, traffic, economic statistics, qualifications, etc. In 20 14, Guizhou province also joined the data opening, and in 10, Guizhou officially went to the cloud. For different industries, the more data you enjoy, the more valuable it is. If every hospital wants to get more information about disease characteristics and curative effect, it needs to enjoy the national and even global medical information, so that it can be analyzed through the platform and get more value. We believe that data will show a trend of sharing, and data alliances in different fields will appear.
Trend 5: Big data security is getting more and more attention, and the big data security market will become more and more important.
As the value of data becomes more and more important, the security and stability of big data will be gradually valued. Internet and digital life also make it easier for criminals to obtain other people's information and have more tricks and criminal means. Therefore, in the era of big data, whether it is the protection of data itself or the security of some information evolved from data, it will be very important for enterprises with high requirements for big data analysis. Big data security corresponds to big data business. Compared with traditional security, the biggest difference of big data security is that when security vendors think about security issues, they must first analyze the business, find out the threats faced by big data business, and then propose targeted solutions. For example, for data storage scenarios, many enterprises currently use open source software such as Hadoop technology to solve big data problems. Because of its open source, its security problem is also very prominent. Therefore, the market needs more professional security vendors to provide professional services for different big data security issues.
Trend 6: Big data promotes the development of smart cities and is the engine of smart cities.
With the development of big data, big data will play an increasingly important role in smart cities. Due to the pressure brought by population gathering on transportation, medical care and architecture, cities need to arrange and allocate resources more reasonably. Smart cities are the best solution for governance transition. Smart city is to realize efficient government management, convenient people's livelihood services and sustainable industrial development through the interconnection ability, comprehensive perception ability and information utilization ability between things and people, and through the new generation of information technologies such as Internet of Things, mobile Internet and cloud computing. Compared with the previous concept of digital city, the biggest difference of smart city lies in the intelligent processing of information obtained by the perception layer. From city digitalization to city intelligence, the key is to realize the intelligent processing of digital information, and its core is the introduction of big data processing technology. Big data is the core wisdom engine of smart cities. Smart security, smart transportation, smart medical care and smart city management are all smart city application fields based on big data.
Trend 7: Big data will give birth to a number of new jobs and corresponding majors.
The emergence of a new industry will inevitably have new job requirements, and the emergence of big data will also create a number of new jobs, such as big data analysts, data management experts, big data algorithm engineer, data product managers and so on. Experienced data analysts will become scarce resources, and data-driven work will show explosive growth. Due to the strong market demand, colleges and universities will gradually open big data-related majors and train corresponding professionals. Enterprises will also work closely with universities to help them jointly cultivate big data talents. For example, in 20 14, IBM comprehensively promoted the cooperation with universities in the field of big data, introduced powerful R&D teams and business partners, promoted the innovative cooperation between "Big Data Platform" and "Big Data Analysis" in Industry-University-Research, built a systematic knowledge system and trained high-value talents, and built credit courses related to big data that meet the teaching characteristics and talent needs of China, so as to prepare for the construction of characteristic majors in the future.
Trend 8: Big data is improving our lives in many ways.
Big data is not only used in enterprises and governments, but also in our lives. In terms of health: we can use smart bracelets to monitor and track our sleep patterns and understand the quality of sleep; We can use intelligent sphygmomanometer and intelligent heart rate meter to remotely monitor the health status of the elderly at home in different places, so that migrant workers far away can feel more at ease; In terms of travel, you can use the GPS data of intelligent navigation to understand the traffic situation and adjust the route in real time according to the congestion situation. In terms of home life, big data will become the core of smart home, and smart home appliances will realize anthropomorphic intelligence. Products can capture and process information through sensors and control chips, and can automatically set controls according to the living space environment and user needs, and even put forward suggestions to optimize the quality of life. For example, our refrigerator will remind us of the recipes of the day every morning.
These are the seven challenges and eight trends in the era of big data that Bian Xiao shared for you. For more information, you can pay attention to the global ivy and share more dry goods.