System Introduction
The overall architecture of the China Development Bank's basic database system consists of three parts: the data management layer, the application control layer and the user interface layer. The data management layer is responsible for managing data at all levels of the China Development Bank; the application control layer is responsible for handling the business control logic of the application system based on the basic database system; the user interface layer handles the user-human-computer interaction interface, separates the user interface from the complex business control logic, and is responsible for providing the business information to the users in a user-friendly and consistent way.
1. Data management layer
In the basic database system of the National Development Bank (NDB), different levels of data need to be managed:
Real-time volatile data:
Created and managed by the operational application system of the NDB's daily business.
High-quality consistent data:
Comprehensive data for a unified business view of the CDB is obtained by performing basic code conversion and inconsistency issues on data stored in different operational application systems of the CDB.
Derived Data:
It is the data generated by varying degrees of aggregation on the basis of consistent data.
Metadata:
Metadata is descriptive data about the above categories of data and serves as a catalog of information at the enterprise level for CDB. Metadata describes and locates the ins and outs of data elements: where the data comes from, how it is transformed, how often it is extracted, and where it goes. The data warehouse facilitates data workers to find, understand, and utilize the above categories of data precisely through the effective management of metadata.
The data management system uses a DB-ODS-DW three-tier architecture to manage the above types of data. DB refers to real-time volatile data and external data, ODS (Operational Data Store) includes high-quality consistent data and derived data, and DW (Data Warehouse) includes historical high-quality consistent data and derived data.
ODS as an intermediate level, on the one hand, it contains enterprise global consistent and detailed data, which can be processed globally operationally; on the other hand, it is a subject-oriented, integrated data environment suitable for completing daily reports and decision-making data processing analysis. It can be seen that ODS supports business-oriented operations on the one hand and is theme-oriented on the other. The so-called theme refers to the business objects concerned in the business development of China Development Bank, such as project development, credit management and fund management, which is to categorize the data at a higher level, and make a shift from application-oriented to theme-oriented for the raw data from various departments, i.e., the whole system will be designed according to the business objects, not according to the administrative framework. Under the theme to place a variety of basic data related to the theme, combined together is the basic data source. The basic data source is the core of the whole ODS, storing the most basic non-derived data. From the above analysis, it can be seen that the first step in building a data warehouse is to build the base data source. This requires analyzing the business processes and needs of the relevant departments of China Development Bank, and solving the problems of data inconsistency, dispersion, integrity and heterogeneity by cleaning, extracting and converting the data from the accounting information system and the external data entry.
The subject-oriented and integrated nature of ODS makes the data in ODS very close to the data in DW in terms of static characteristics. However, there are still many basic and important differences between ODS and DW. First, ODS stores mainly recent data, whereas DW is heavily characterized by historical data that is stored for a long time and can be queried repeatedly. Secondly, ODS supports record-oriented online refresh to meet the needs of NDB's global applications, including enterprise-level OLTP; whereas the underlying data in DW is not modifiable. Third, it provides a consistent data environment for extraction to the ODS data warehouse DW, which is mainly used for long-term trend analysis or strategic decision-making.
1) Data Sources
China Development Bank (CDB) Business System Data
CDB's business processing systems include those that are already in operation (the accounting system), those that are under construction (credit management and off-site auditing), and those that are being prepared for the construction of various business processing systems. The data from these systems periodically form incremental files that are extracted into the head office operational database (ODS) by the database extraction agent program (Agent).
External data
External data, depending on business requirements, can be loaded into the head office operational database (ODS) or directly into the data warehouse.
Supplementary data
Supplementary data, entered manually or poured in by the receiving program.
2) Basic data collection
In order to improve the efficiency and quality of basic data collection, it is necessary to take into account the business needs, data volume, data loading cycle and technical infrastructure of a variety of factors, the development of practical data extraction, purification, conversion and loading strategy, and select the appropriate tools to assist in the collection of basic data.
For the data managed by the existing business application system of China Development Bank, every effort should be made to differentiate between stock data, incremental data, and changed data (for example, changed data can be obtained by adding triggers), because in the wide area network environment, the extraction, transmission, and loading of stock data, which increases the pressure on the network, is not desirable. And no matter which database you choose, the database management system has limited speed for loading large amounts of data, and large amounts of data loading generally affects the operation of the database by other users.
As network bandwidth permits, the ODS at the head office collects and stores detailed business data from each branch, and the detailed business data from each branch is automatically extracted to the head office through a data collection agent (Agent). The strategy for data extraction, transmission and loading is to perform batch loading of stock data for the first data initialization, and incremental and change data loading in the future. The loading cycle is hourly, daily, monthly, or quarterly and yearly, depending on business requirements.
As business grows and the volume of detailed business data increases beyond the load of network bandwidth, it is recommended that each branch set up an ODS to collect and store its own detailed business data, and the head office ODS to collect and store the aggregated business data from each branch to minimize the volume of data to be extracted, transmitted, and loaded.
Visual Warehouse Manager (IBM Visual Warehouse) is an integrated tool for creating and maintaining data warehouses from IBM, which can define, create, manage, monitor, and maintain data warehouses, as well as automate the extraction of heterogeneous data sources into a centralized, integrated data warehouse management environment, which utilizes a distributed client/server (Client/Server) architecture. Client/Server) architecture, including the following parts:
Data Warehouse Server
Data Warehouse Administrative Clients
Data Warehouse Agents (Visual Warehouse Agents)
Control Database
Data Warehouse (Target Database)
Data Warehouse Server runs on top of the Windows NT operating system and monitors and manages data warehouse processing, provides time-based and event-based scheduling mechanisms, and also controls the activities of data warehouse agents.
The Data Warehouse Agent, under the control of the Data Warehouse Server, handles accessing, filtering, transferring, and loading of source data into the target data warehouse. The data warehouse agent can run on different system platforms such as NT, AIX, OS/400, OS/2, and SUN. In order to improve processing efficiency and scalability, generally in the data source and the target data warehouse where the machine are installed data warehouse agent.
The control database is generated by the Data Warehouse Manager and utilized by the Data Warehouse Agent. The Visual Data Warehouse Manager stores all metadata in the control database, which can also be managed by a metadata management tool integration (the tool is called Dataguide and is one of the components of the Visual Data Warehouse Manager).
While the mechanisms for automating data extraction, transfer and loading can be implemented by choosing the appropriate tools, data extraction, transformation and purification for the actual data environment require self-designed procedures, because of the non-standardization of the actual data and the complexity of the data transformation, the commoditized tools for data extraction, transformation and purification do not achieve the expected results in practical applications.
2. Head Office ODS
Head Office ODS consists of two layers of data, one layer is the basic data source, which is the most basic and non-derivative data generated by the business of China Development Bank; the other layer is the secondary summary data. The secondary summary data is placed in the three modules of project acceptance, loan management and fund management, which directly provide data support for the three business subsystems of project acceptance, loan management and fund management. The data in the basic data source is mainly converted from the accounting information system, and at the same time, part of the basic data comes from external data entry.