Current location - Loan Platform Complete Network - Big data management - What are the collection techniques for big data in the power grid
What are the collection techniques for big data in the power grid
Database collection system, system log collection system, network data collection system.

1, database collection system. Through the database collection system directly with the enterprise business background server combination, the enterprise business background every moment in the generation of a large number of business records written to the database, and finally by a specific processing sub-licensing system for system analysis.

2. System log collection system. Log data information on the log collection, collection, and then data analysis, mining the company's business platform log data in the potential value. In short, the collection of log data to provide offline and online real-time analysis use.

3, network data collection system. Obtain data from websites by means of web crawlers and public **** APIs provided by some website platforms (e.g. Twitter and Sina Weibo APIs). Web data of unstructured data and semi-structured data can be extracted from web pages, and they can be extracted, cleaned, converted into structured data and stored as unified local file data. The above is the collection technology of big data in the power grid.