Generally speaking, situational awareness can acquire, understand, display and predict the future development trend of security elements that can cause system state changes in large-scale system environment. The proposal of joint operations and network-centric warfare has promoted the emergence and continuous development of situation awareness. As an important platform and material basis to realize situation awareness, the complex demand and characteristics of situation map for data and information constitute a prominent big data problem. Thinking from the height of big data and solving the information processing problems faced by situational awareness are important methods to study situational awareness in joint operations. By analyzing the data type, structure and characteristics of situational awareness in joint operations, it is concluded that situational awareness faces the challenge of big data. The problems that may need to be solved and the application requirements of cutting-edge information technology are discussed preliminarily. Finally, the key data and information processing technology are studied. This research has important exploration value for the research of "big data" in the field of military information processing and digital decision-making.
Related references (online excerpts):
1 Introduction
With the rapid development of computer and communication technology, the application of computer network is more and more extensive and the scale is getting bigger and bigger. Multi-level network security threats and security risks are also increasing. The threats and losses caused by network viruses and Dos/DDos attacks are increasing, and network attacks are developing towards distribution, scale and complexity. Only relying on a single network security protection technology, such as firewall, intrusion detection, antivirus and access control, can no longer meet the needs of network security. There is an urgent need for new technology to find abnormal events in the network in time, grasp the network security situation in real time, and turn what has been remedied beforehand and afterwards into automatic evaluation and prediction beforehand, thus reducing the network security risks and improving the network security protection ability.
Network security situation awareness technology can integrate all aspects of security factors, dynamically reflect the overall network security situation, and predict and warn the development trend of network security. The unique characteristics of big data technology, such as mass storage, parallel computing and efficient query, have created opportunities for the breakthrough of large-scale network security situational awareness technology. With the help of big data analysis, thousands of network logs and other information are automatically analyzed and deeply mined, and the security state of the network is analyzed and evaluated, so as to perceive abnormal events and overall security situation in the network.
2 Network security situation related concepts
2. 1 Network Situation Awareness
The concept of situational awareness was put forward by Endsley in 1988. SA is the acquisition, understanding and short-term prediction of environmental factors in a certain time and space. The whole situation awareness process can be intuitively represented by the three-level model shown in figure 1.
The so-called network situation refers to the current state and changing trend of the whole network, which is composed of various network equipment operating conditions, network behaviors and user behaviors.
Network Situation Awareness (CSA) was first proposed by Tim Bass in 1999. Network situation awareness is the recent development trend of acquiring, understanding, displaying and predicting security factors that can cause network situation changes in a large-scale network environment.
Situation is a state, a trend, a whole and overall concept, and no single situation or state can be called situation. Therefore, the understanding of the situation emphasizes the environment, dynamics and integrity. Environment means that the application environment of situational awareness is a network with a certain scale in a wider range; Dynamism means that the situation changes with time, and the situation information not only includes the past and present state, but also predicts the future trend; Holiness is the embodiment of the relationship between situational entities. The state change of some network entities will affect the state of other network entities, and then affect the situation of the whole network.
2.2 Network security situation awareness
Network security situation awareness is to use data fusion, data mining, intelligent analysis and visualization technologies to directly display the real-time security status of the network environment and provide guarantee for network security. With the help of network security situation awareness, network supervisors can know the state of the network, the attacked situation, the attack source and which services are vulnerable to attack in time, and take measures against the attacked network; Network users can clearly grasp the security situation and trend of their own networks, and make corresponding preventive measures to avoid and reduce the losses caused by viruses and malicious attacks in the network; Emergency organizations can also understand the security status and development trend of the network they serve from the network security status, and provide a basis for making predictable emergency plans.
3 Network security situation awareness related technologies
For large-scale networks, on the one hand, there are many network nodes, complex branches, large data flow, heterogeneous network environment and many application platforms; On the other hand, the technology and means of network attack are developing towards platformization, integration and automation. Network attack has stronger concealment and longer delay, and the network threat is getting bigger and bigger, resulting in more and more losses. On the basis of collecting network resources, network security situation awareness should be completed through data preprocessing, network security situation feature extraction, situation evaluation, situation prediction and situation display, so as to show the whole network security situation in real time and accurately and detect potential and malicious attacks, which involves many related technical problems, mainly including data fusion technology, data mining technology, feature extraction technology, situation prediction technology and visualization technology.
3. 1 data fusion technology
Because the data of cyberspace situational awareness comes from many network devices, its data format, data content and data quality vary widely, and its storage forms and semantics are different. If we can preprocess these data with different formats from different network locations and using different channels, and then carry out normalized fusion operation on this basis, we can provide a more comprehensive and accurate data source for network security situation awareness, so as to get a more accurate network situation. Data fusion technology is a multi-level and multi-level data processing process, which mainly completes the complementary fusion of multi-source information with similar or different characteristic patterns from the network, and completes the automatic monitoring, correlation, correlation, estimation and combination of data, thus obtaining more accurate and reliable conclusions. According to the abstract degree of information, data fusion can be divided into three levels: data level fusion, feature level fusion and decision level fusion, among which feature level fusion and decision level fusion are widely used in situation awareness.
3.2 Data mining technology
Network security situational awareness converts the collected data of a large number of network devices into data units in a unified format after data fusion. These data units are huge and carry a lot of information, and useful information and useless information are mixed together, which is difficult to identify. In order to grasp the relatively accurate and real-time network security situation, it is necessary to eliminate interference information. Data mining refers to the extraordinary process of mining useful information from a large number of data, that is, discovering hidden, regular, unknown in advance, but potentially useful and ultimately understandable information and knowledge from a large number of incomplete, noisy, fuzzy and random practical application data [1]. Data mining can be divided into descriptive mining and predictive mining. Descriptive mining is used to describe the general characteristics of data in a database. Predictive mining infers from the current data and predicts it. Data mining methods mainly include correlation analysis, sequence pattern analysis, classification analysis and cluster analysis. Correlation analysis method is used to mine the relationship between data; Sequence pattern analysis focuses on the causal relationship between data; Classification analysis classifies data by establishing an analysis model for predefined classes. The commonly used models are decision tree model, Bayesian classification model and neural network model. Cluster analysis does not depend on predefined classes, and its division is unknown. Commonly used methods include fuzzy clustering, dynamic clustering and density-based methods.
3.3 Feature Extraction Technology
Network security situation feature extraction technology is to combine large-scale network security information into one or more groups of values in a certain range through a series of mathematical methods. These values have a series of characteristics, showing the real-time operation of the network to reflect the security status and threat degree of the network. Feature extraction of network security situation is the basis of network security situation assessment and prediction, which has an important influence on the whole situation assessment and prediction. The main methods for extracting the characteristics of network security situation are analytic hierarchy process, fuzzy analytic hierarchy process, Delphi method and comprehensive analysis method.
3.4 Situation Prediction Technology
Network security situation prediction is an important part of network security situation awareness, which uses scientific theories, methods and various experiences, judgments and knowledge to speculate, estimate and analyze its possible changes in a certain period of time in the future. The security situation of the network in different periods is interrelated, and the change of security situation has certain inherent laws, which can predict the future security situation of the network, thus predictably configure security policies, realize dynamic network security management, and prevent the occurrence of large-scale network security incidents. Network security situation prediction methods mainly include neural network prediction method, time series prediction method and prediction method based on grey theory.
3.5 Visualization technology
Network security situation generation is to show the current state and future trend according to the analysis results of a large number of data, but it is difficult to find useful and key information through traditional text or simple graphic representation. Visualization technology is a theory, method and technology that uses computer graphics and image processing technology to convert data into graphics or images and display them on the screen for interactive processing. It involves many fields such as computer graphics, image processing, computer vision, computer aided design and so on. At present, many studies have applied visualization technology and visualization tools to the field of situation awareness. At each stage of network security situation awareness, we make full use of visualization methods to integrate network security situation into a coherent network security situation map, quickly discover network security threats and intuitively grasp the network security situation.
4. Network security situation awareness based on multi-source logs.
With the expansion of network scale and the increase of network attack complexity, security devices such as intrusion detection, firewall, antivirus and security audit have been widely used in the network. Although these security devices have played a certain role in network security, they have great limitations, mainly in the following aspects: First, the massive alarms and logs of various security devices have low semantic level, high redundancy and large storage space, and there are a large number of false alarms, which leads to the real alarm information being submerged. Second, most safety devices have a single function, resulting in different alarm information formats, which are difficult to comprehensively analyze and sort out, and information sharing and data interaction cannot be realized, resulting in the overall protection efficiency of safety devices not being fully exerted. Thirdly, the processing results of each security device can only reflect the operation status of a certain aspect of the network, and it is difficult to provide comprehensive and intuitive information on the overall security status and trend of the network. In order to effectively overcome these limitations of network security management, we propose network security situation awareness based on multi-source logs.
4. 1 Acquisition of network security situational awareness elements based on multi-source logs
Network security situation awareness based on multi-source logs is to extract, analyze and process the log information provided by various security devices deployed in the network, realize real-time monitoring of network situation, identify and warn potential and malicious network attacks, give full play to the overall efficiency of each security device, and improve the network security management ability.
Network security situational awareness based on multi-source logs mainly collects firewall logs, intrusion detection logs, key host logs and host vulnerability information at the network entrance. By fusing and analyzing these log information from different devices, real and effective information about the network security situation can be comprehensively and profoundly mined, which is different from analyzing the network security status only based on a single log source.
Compared with situation, it can improve the comprehensiveness and accuracy of network security situation.
4.2 Use big data for multi-source log analysis and processing
Network security situational awareness based on multi-source logs collects a large amount of data generated by various detection methods and event reporting mechanisms on various security devices. However, these original daily information are massive, redundant and wrong, which can not be used as the direct information source of situation awareness, and must be processed by correlation analysis and data fusion. What kind of technology can be used to quickly analyze and process these massive and diverse data?
The emergence of big data has expanded computing and storage resources. The diversity of big data itself supports three characteristics: multi-type data format, large-capacity data storage and fast processing, which is exactly what is needed for network security situation awareness analysis and processing based on multi-source logs. The multi-type data format of big data can enable network security situational awareness to obtain more types of log data, including logs of networks and security devices, logs of network operation information, services and applications, etc. Massive data storage of big data is exactly what massive log storage and processing need. The rapid processing of big data provides technical support for the deep security analysis of high-speed network traffic, and provides computing resources for high-intelligence model algorithms. Therefore, we use the basic platform provided by big data and the technical support of big data processing to analyze and respond to the network security situation.
Correlation analysis. Firewall logs and intrusion detection logs in the network are descriptions of the security event traffic entering the network. For a possible attack event, a large number of logs and related alarm records will be generated, and there are many redundancies and associations in these records. Therefore, first of all, it is necessary to make a single-source correlation analysis of the obtained original logs, so as to transform the massive original logs into intuitive and understandable security events that may be harmful to the network. Network security situation awareness based on multi-source logs adopts alarm correlation based on similarity, which can better control the number of alarms after correlation and help reduce complexity. The processing process is as follows: firstly, the main attributes in the alarm log are extracted to form the original alarm; Then, through repeated alarm aggregation, an aggregated alarm is generated; Define the calculation method of similarity for each attribute of aggregation alarm and assign the weight; Calculate the similarity of two aggregated alarms, and decide whether to exceed the alarm by comparing with the similarity threshold; Finally, the address range and alarm information belonging to the same alarm are output to generate security events.
Fusion analysis. Multi-source logs are redundant and complementary. With the help of data fusion technology, situational awareness can make multiple data sources learn from each other, thus providing guarantee for the sensing process and generating the security situation more accurately. Through the correlation process of single source log alarm, the respective security events are obtained respectively. For multi-source security incidents from firewalls and intrusion detection logs, D-S evidence theory (put forward by Dempster in 1967 and later popularized by Shafer in 1976) is adopted to judge the reliability of security incidents, which further improves the accuracy and reduces false positives. The basic idea of applying D-S evidence theory to security event fusion is as follows: firstly, the feasible initial trust distribution method is studied, and the information degree function is assigned to firewall and intrusion detection; Then, through the D-S composition rules, the credibility of the fused security events is obtained.
Analysis of situational elements. Through the security analysis of daily records of network portal security equipment, only the possible attack information entering the target network can be obtained, while the security events that really have a decisive impact on the network security situation need to be finally confirmed through the comprehensive analysis of the attack knowledge base and the specific network environment. It is mainly divided into three steps: the first step, through the study of a large number of network attack examples, the available attack knowledge base is obtained, which mainly includes the principles, characteristics and action environment of various network attacks; The second is to analyze the system vulnerabilities and possible vulnerabilities of hosted services on key hosts, establish a vulnerability knowledge base of the current network environment, analyze the topological structure and performance indicators of the current network environment, and obtain a network environment knowledge base; The third is to confirm the validity of security events through vulnerability knowledge base, that is, network attack events that have an impact on the current network. In the process of network security event generation and attack event confirmation, the situation factors used to evaluate the whole network security situation are extracted, which mainly include the security threats faced by the whole network, the security threats faced by the branch network, the security threats suffered by the host and the degree of these threats.
5 conclusion
In order to solve the increasingly serious network security threats and challenges, applying situational awareness technology to network security can not only fully grasp the current network security situation, but also predict the future network security trend. On the basis of introducing the related concepts and technologies of network security situation, this paper discusses the network security situation awareness based on multi-source logs, focusing on the acquisition of network security situation awareness elements based on multi-source logs, and the correlation analysis, fusion analysis and situation element analysis of multi-source logs using big data. Other related contents need further discussion and research.