Read "The Age of Big Data" experience (I)
After reading "The Age of Big Data", I feel that an era of great change is coming. Although it is not quite clear exactly what thinking and operating methods are going to be revolutionized, it is clear that the author wants to ? end? or subvert some of the theories, methods and approaches that have traditionally served as the basics of our thinking and existence. My mind was so strongly shaken by such ideas that I couldn't help but tremble.
? In the age of small data, we hypothesize about how the world works and then validate that hypothesis by collecting and analyzing data.? With the transition from the age of hypotheses to the age of data, we're also likely to think that we don't need theories anymore.? The book almost certainly seeks to subvert the theories and methods of statistics, and also attempts to do so by quoting Anderson, the editor-in-chief of Wired magazine? The theories of quantum physics are out of touch with reality? to? put an end to? quantum mechanics. I'm happy about this, because both statistics and quantum mechanics were classes I studied in college until I couldn't pass them with cramps. But these two theories are just too big, too authoritative, and too fundamental, and I don't think I can get rid of both of them with one book that will give me headaches for the rest of my life. The author doesn't really dare to flag up the argument for subverting them, after all, but still prefaces it with ? likely to think? such an umbrella.
In recent decades, we are always encountering all kinds of new thinking. The first thing we should do in front of the new thinking is to break and stand, to change their own traditions, to keep up with the times. Even if the brain can not yet keep up, the mouth must keep up, otherwise it may be labeled as a rigid thinking or even hinder the development of the world. Since big data is ? the inevitable change that leads to the future? , then I must? not be limited by traditional mindsets and inherent biases implicit in a particular field? , join the authors in dismissing statistics and quantum mechanics first. I don't like and can't learn them anyway.
Once we humans have the ability to collect and process data to the tune of tens of bytes or more, and we can turn samples into totals, plus the ability to face up to confounding at the expense of precision, it seems like it's really possible to throw out statistics based on sampling. But by statistics and quantum mechanics and many others? that we are also likely to think we no longer need? Theoretically traced, they are almost all based on the one **** same foundation? Logic. If you accidentally give logic or logical thinking or logical reasoning along with ? no longer needed? then that would worry me!
Page 16 of The Age of Big Data? At the heart of Big Data is prediction? Logic is? describing spatio-temporal information? class? with? class? rules of sequential change relationships that are valid and unchanging over long periods of time. Both seem to do the same thing. But what Big Data wants? not causation, but correlation? It's not about cause and effect, it's about correlation. It is enough to know what is, not why? and the Law of Sufficient Reason, one of the four fundamental laws of logic (the Law of Identity, the Law of Contradiction, the Law of Exclusion, and the Law of Sufficient Reason)? It is clearly stated that Everything has a sufficient reason for its existence. And the three parts of logical reasoning? Inductive logic, retrospective logic and deductive logic are all based on cause and effect. The two seem to be in opposition to each other. With two methods opposing each other on the same matter, there should only be one result, and that is to disprove one or the other. That's what worries me so much.
But I can't just wait and see, like a bystander, which one? stands out? , because I am in the middle of it. I can't think or work and naturally I can't live unless the problem is solved! Not to mention two more scary things.
One: after more than a hundred years of quantum mechanics, in order to deal with the problem of mixing, combining mass and speed to energy, in order to reconcile the contradictions between quantum mechanics and relativity, another quantum field theory, and then seven to mess around with wormholes and Rosen bridges, and finally bending four-dimensional spacetime to allow for time travel, hating to cause that horrible time-traveling machine right away. The only thing stopping those ? Einstein? Einsteins? fooling around? is causality, because a father is a father and a son is a son. So will big data by confronting the mixed nature, give up causality and finally, instead of a time machine, so that the father is no longer the father, the son is no longer the son? Second: the fundamental difference between humans and machines is that humans have logical thinking while machines do not. The Age of Big Data is also worried? The final decision-making will be machines rather than people? If it is true that one day because of the abandonment of logical thinking and the sci-fi movie described on the machine domination of the world to eliminate the results of human beings, then I might as well jump off a building now before it is too late.
Fortunately, I know I'm a layman in terms of what statistics, quantum mechanics, logic, and big data are all about, so maybe a big part of the above is nonsense, and the so-called worries don't exist. But when a problem arises, it's better to solve it, otherwise you won't be able to sleep. If you can't solve it yourself, you can only rely on experts to show you the way.
So I would like to propose a rationalization to the author of "The Age of Big Data": continue to write this book, at least add a fourth part? Logical Thinking in the Age of Big Data.
Read the "Age of Big Data" experience (2)
The arrival of the information age, we feel is the rapid changes in technology, and with it the transformation of lifestyle? The information age that we commented on in this way has changed into a once upon a time. Today, the era of big data has become a hot topic. The author here illustrates the information and data, just trying to firstly illustrate the relationship and difference between information and data, and also trying to illustrate why the information age is transformed into the big data age? What has the era of big data brought us?
Definition of information and data. Wikipedia explains: information, also known as information, is a highly generalized abstract concept, is a dynamic category in the development of the content and the name of the mutual exchange of information, information is defined without a unified definition, but the information has an objective, dynamic, transmission, **** enjoyment, economic and other characteristics is everyone's **** knowledge. Data: or information, refers to the description of the symbolic record of things, can be defined as the meaning of the entity, it involves the existence of things in the form. It is a discrete and objective set of factual descriptions of events that constitute the raw material of information and knowledge. Data can be divided into two categories: analog data and digital data. Data refers to computer processing? Raw material? such as graphics, sound, text, numbers, characters and symbols. From the definition, it seems that data is raw virgin land that needs to be plowed. Information, on the other hand, is information that has been processed and can be disseminated. The Information Age relies on an explosion of data, only when the data explodes into an unmanageable state, the Big Data Age is born. Is this background material that was not articulated in the book The Age of Big Data?
In the book The Age of Big Data, the difference between the age of big data and the age of small data: 1. Thinking conventions. The big data era difference and shift is that the desire for causality is abandoned and replaced by a focus on correlation. That is to say, as long as you know? What? and do not need to know? Why? The author speaks in absolutes, but reflects on the essential difference. More and more mixed data, resulting in the application of the idea can only try to observe, rather than pouring everything into the reasoning? It is also wise to 2. use uses. Small data stays to illustrate the past, big data with driving the past to predict the future. The author believes that the use of data intended for what, with the data itself has nothing to do with the interpretation of the data reader, and the correlation is more conducive to predicting the future.3, structure. Big data is more reflected in the massive unstructured data itself and the integration of processing methods. Big data is more like theory and reality go hand in hand, the theory to create a method to deal with unstructured data, the results of the processing and the future to verify.4, the basis of analysis. Big data is in the context of the Internet data from quantitative to qualitative change. The author believes that the era of small data, that is, the information age, is the premise of the era of big data, the era of big data is sublimation and evolution, the essence is complementary, rather than separate from each other.
The story of the future of data. What are the expectations and revelations from the development of data? The banking industry has a natural potential for big data. Customer data, transaction data, management data and other massive data continues to grow, massive opportunities and challenges come with it, adapt to change, the survival of the fittest. We can have a broader space for business development, more accurate decision-making and judgment, and better operational management capabilities? can all of these are based on the ability to collect, organize, harness and analyze data, based on innovative thinking and execution that stands out. Therefore, the construction of ? Data Warehouse and cultivate the ability of data thinking. Data thinking and develop data governance. Data Governance Create a data warehouse. Data Integration and realize the power of data. Data applications to embrace Big Data. Big Data era, from the data to seize the value, laugh at the change of the wind and clouds, and steadily win the future.
Read "The Age of Big Data" experience (3)
This book mainly introduces the application of big data in modern business operations, as well as its impact on modern business operations.
The structural framework of the book "The Age of Big Data" follows the common approach of academic books. In other words, it starts with the phenomenon and then dissects it to offer an explanation of the phenomenon. Then through the explanation in the future prediction, and the future of the problem may appear to put forward their own views and countermeasures.
The following highlights the main content of the book The Age of Big Data.
The Age of Big Data opens with a story about Google's success in predicting the 20XX outbreak of H1N1 in the United States, the direction of its spread, and the potential number of patients, using data left behind by people who searched for keywords on search engines. Google's prediction was nearly a month ahead of the government's, compared to the government's ability to get data only a week or two after the outbreak. At the same time, the correlation between Google's predictions and the government's data was 97%, which means that the confidence interval for Google's predictions was 3%, a figure much smaller than the conventional statistical confidence interval of 5%! And this number is the best proof of the relative accuracy of prediction results and predictability of events in the age of big data! With this and other cases, Victor suggests that in the age of Big Data? Sample = Total? idea. We all know that when the sample converges infinitely to the totality, the descriptive data obtained through calculation will converge infinitely to the nature of the event itself. And before taking the ? sample<overall? The practice to a large extent can not do further description of things, because the previous era of data acquisition and storage and processing itself is very difficult only to lead to people to take a sample of the way to measure things. The emergence of Internet terminals and computers has greatly reduced the difficulty of acquiring, storing and processing data, and thus the relatively higher accuracy of ? Sample = total?
Next, Victor went through the failure of IBM's computer translation program to pursue high accuracy, and Google simply scanned and stored all the corresponding text statements in the thesaurus, so no matter what needs to be translated, as long as there is a link to the Google thesaurus will appear to be translated, and although sometimes the translation is very inane, but most of the time is still correct, so the Google's computer translation program. The success of Google's computer translation program shows that the pursuit of accuracy in the era of big data is not particularly obvious, but on the contrary, the era of big data is built on the basis of big data, so the era of big data is the pursuit of a full range of digital measurements regardless of its accuracy in the end how high, because a large amount of data will bury a few problematic data brought about by the impact. At the same time, a large amount of data will infinitely approximate the way things are.
After that, Victor predicted an important profession spawned in the age of big data? Data scientists, an amalgam of mathematicians, statisticians, and programmers, a group of people who will be able to get whatever results they want from the data they acquire. In other words, as long as there is enough data, everything about us, both external and internal, that we don't want others to know will be revealed to the fullest extent in front of this group of guys. So in order to prevent privacy from being exploited by these guys in the era of big data, Victor suggests dividing them into two groups, one that uses the data to serve the business sector, and another that examines whether they are legally obtaining and using the data, and whether they are infringing on personal privacy.
In any case, the age of big data will come, whether we accept it or not!
I think the book The Age of Big Data is very well written and worth reading. Because it will give us a lot of inspiration, for example, your comments or photos on relevant social networking sites are likely to be ? Data scientists? They can use the data to sell it to online stores. However, the truth is that we will be the ones who are predicted to be seduced. So be careful what you leave behind online.
I liked this book because it showed me a new world.
Reading The Age of Big Data (IV)
Taking advantage of the weekend, I read Tu Zi Pei's masterpiece Big Data in one sitting. It's a great book, the lines flow like water and it's fascinating. In the book, you read less about big data technology and more about the evolution of American politics, economy, society and culture related to big data. As an information technology practitioner, after reading the whole book, I y felt the respective characteristics of China and the United States in terms of information technology, but also saw the gap between us and the United States. There are several aspects of the experience, but a glimpse of the basic can see the whole picture.
First, the breadth and depth of government business database disclosure. In recent years, with the advancement of China's information disclosure work, governments at all levels are actively promoting the disclosure of online government information through the construction of government portals, but the disclosure of our information, at this stage, is still mainly the government's policies, laws and regulations, standards, official notices, job duties, business guidelines, work dynamics, personnel appointments and dismissals, and other administrative affairs information disclosure. Of course, real-time disclosure of government business databases has also made great progress. On the Chinese government portal, you can check some public welfare databases, such as the economic statistics of the National Bureau of Statistics, the national air and hydrological data provided by the data center of the Ministry of Environmental Protection, the national meteorological data provided by the General Administration of Meteorology, the national flight information provided by the General Administration of Civil Aviation, etc.; visiting the websites of various ministries and commissions, you can also check a lot of operational data, such as the project establishment database of the Development and Reform Commission, the enterprise credit database of the Bureau of Industry and Commerce, Ministry of Land and Resources of the land license library, the State Administration of Safety Supervision of the coal mine safety early warning information base, various types of project bidding information base and so on. This is a very big progress, but also so many years of e-government construction achieved results and value! However, a lot of data in the government business database has not yet realized the public, a lot of data because of departmental interests and ? Confidentiality? and other factors, but also limited to departmental internal staff use, not open to the public; has been open to the public data is also limited to a part of the basic information and statistical information, more data has not been made public. From the point of view of the U.S. practice of data disclosure recorded in the book "Big Data", the U.S. has a greater breadth and depth of data disclosure. Americans believe that ? data collected with taxpayer money should be freely available for taxpayer use? , despite the fact that the U.S. government is also resistant to the openness of data, but the people's will can not be defeated, the U.S. government's business data is more and more open, especially in the Obama administration signed the "Transparent and Open Government" document, the openness is even greater.DATA.GOV is the U.S. Coalition Government's new construction of a unified open data portal, the site in accordance with the original data, geographic data and data application tools to organize the GOV is a new unified open data portal built by the U.S. Coalition Government, which organizes raw data, geographic data and data application tools according to the various types of open data, and has opened 378,529 raw and geographic data sets. There is no such open data portal in China. In addition, due to the different systems, the depth of openness of U.S. business information is also great, for example, the online publication of the U.S. President? White House visitor records? Published or even visit the White House of all kinds of people related information; U.S. FedSpending website, able to track, record, and analyze the federal government, article by article, every financial expenditure. This in China, at present, should not be realized.
Second, the analysis of government-to-business data. At present, the business data provided by Chinese government websites at all levels is still basically a data table, and some websites can provide some statistical charts, but very little can be done to achieve cross-departmental on-line analysis of data, data correlation analysis. This is mainly due to the fact that in the past, the construction of China's government informatization was still at the stage of departmental construction. The United States in this regard, the pace to be faster, the United States DATA.GOV website, not only to provide raw data and geographic data, but also provides a lot of data tools, many of these tools are the public, public interest organizations and some commercial institutions to provide, these applications for data processing, online analysis, social network-based correlation analysis and other aspects of the provision of means. For example, the White House Visitor Search tool available on DATA.GOV searches for visitor information and correlates White House visitors with other microblogs, social networking sites, and so on to improve visitor transparency.
The third is about the privacy of personal data. In the United States, the privacy of citizens and own inviolable, the United States does not have a personal identity card, and can not be established based on the personal identity card number of personal information associated with the establishment of ? Central Data Bank? The proposal of establishing a centralized data bank has also been repeatedly rejected. This, in China is not a problem, each citizen has a unique identity information, through the ID card information, you can access the basic information of citizens. In the future, with the construction of the national population-based database and other basic resource base, citizens' social security, medical and other relevant information can also be easily accessed, of course, the information is still limited to the use of government departments, but it is difficult to fully ensure that the integration of these personal information is not leaked or utilized.
Data is the foundation of information construction, and the two big countries will learn from each other in the field of big data to complement each other's strengths and weaknesses, which will propel the world into the information age. I am pleased to see that the U.S. government launched in 20XX the ? Big Data Research and Development Program? , investing $200 million to promote research in the fields of big data extraction, storage, analysis, ****enjoyment, visualization and other areas, and compared it with supercomputing and Internet investment. In the same year, the Chinese government 20XX also approved the ? Twelfth Five-Year Plan for National Government Informatization Construction Project? , with total investment estimated at tens of billions of dollars, specializing in five major construction projects with five major repositories of population, legal person, space, macroeconomics and culture. The era of open, *** enjoy and intelligent big data has come!
I carefully recommend