The person who knows you best is not a boyfriend or girlfriend, but a reptile. Fully explain the positioning of crawlers in the Internet industry.

What is a crawler?

Popular understanding: A crawler is a program that simulates the behavior of humans requesting websites. It can automatically request web pages, capture the data, and then use certain rules to extract valuable data.

It is said that more than 50% of the traffic on the Internet is created by crawlers. Maybe you see that a lot of popular data is created by crawlers, so it can be said that without crawlers, there would be no prosperity of the Internet.

But behind the prosperity, the survival status of an industry becomes more and more obvious, and light and darkness will also usher in a real explosion at this time.

The light and darkness of crawlers

Reptile technology is not mysterious. It is nothing more than three steps: "crawling" the page, "shoveling" the data, and processing and cleaning.

1. The bright side of crawlers

The core logic of search engines like Baidu and Google is also crawlers - crawl to the keywords that users want, and then display the search results.

"Making valuable things better presented is the biggest contribution of crawlers."

2. The dark side of crawlers

In the era of big data , crawlers have become a shortcut to obtain data at low cost, and are often reduced to "dark weapons".

In 2014, during the climax of Internet entrepreneurship, crawler technology ushered in a small explosion.

"There is a type of company in the crawler ecological chain that specializes in crawling for people and increasing virtual visits to the APP." Netizens said that for 10,000 views, the price is 10 yuan.

In 2013, when Internet finance was still very popular, a company called "**LI" suddenly appeared in the industry, and crawler services began to be well-known by people in the industry.

At that time, P2P and consumer loans were emerging. Users needed to fill in a lot of personal information on their mobile phones, such as name, ID number, home address, bank card number, etc. In the early days, the entire industry was not fully developed, and many companies’ apps were of that type. It's stuck as soon as you use it, and it's dead as soon as it gets stuck.

Filling in so much information is troublesome, and post-processing is also a problem. At this time, the crawler service came into being.

You can simply understand the crawler service in the mutual fund industry as invoicing. It’s too laborious to type out a meal invoice every time. Nowadays, most invoicing services only need to scan WeChat or enter the first one or two words and they will be automatically associated with you, saving time and effort. This was probably the purpose at that time.

It should be pointed out in particular that the crawler service at this time is limited to operator data.

In 2016, the cash loan industry became the largest buyer of crawler products.

Crawler companies provide them with a variety of services: such as crawling Taobao, social networks, online banking, etc.

"Whether it is a website or an APP, as long as you have an account and password, you can crawl it. You can even crawl the central bank's credit report," a data person in charge claimed, "With good technology, there is no data that cannot be crawled. ".

The most popular product among them is call record (operator data) crawling.

The crawling method is also very simple. As long as the user provides a mobile phone number and service code, they can log in to the systems of major operators and crawl.

Operator data is most useful for post-loan collection. Taobao, Alipay data, credit card details, and email bills all represent your spending power.

Insurance policies are rarely used in actual business operations in the mutual financial industry. Data on social security and provident funds also help lending institutions measure your loan qualifications and repayment ability.

Different parties have different needs for this data. For example, let’s take a guess, who is the 714 anti-aircraft gun’s favorite?

In fact, 714 anti-aircraft gun does not need so much crawler data. It has only one true love, and that is operator data. As for the reason, it is very simple. It is convenient for collection and collection of high overdue fines. Just take interest.

Who are the remaining crawler services provided to? There are consumer finance companies, banks, and consumer loans. But relatively speaking, even slightly formal companies will not use the crawler business as abnormally as 714 Anti-aircraft Gun.

Why crawl a person’s carrier data? There's a lot of use here.

For the mutual finance industry, the most direct and effective thing is: post-loan collection.

When you know the contact information of a person's parents, spouse, colleagues and other social relationships, once the person does not pay back the money, the so-called violent collection and phone bombing "call you to death" software will come. Because the operator's data is in hand, violent collection should not be too easy.

Many companies that provide crawler services did not expect that they would eventually die from violent debt collection. This makes many users take the wrong approach.

The craziest, most fascinating and most dangerous thing about the financial industry is not that it is too close to money, but that it is too close to making big money.

3. The future path of crawlers

On June 1 this year, the "Cybersecurity Law" came into effect, which is extremely strict:

Unauthorized crawling If a user's mobile phone address book contains more than 50 records, a legal person may be imprisoned for up to 3 years;

There are media reports that the industry is planning a "Trial Measures for the Protection of Personal Financial Information". The introduction of the "Measures" may mean that Further clarify the qualifications of credit reporting agencies. The promulgation of the Personal Information Protection Law in the future reflects the trend of increasingly strict legislation in the field of personal privacy information protection.

Daily sharing:

There are no ifs in life, only consequences and results; maturity means facing all the little things with a smile; having an indifferent heart, not being happy with things, Don't be sad for yourself, know how to accept the regrets in life, cherish the touches in life, learn to let nature take its course, be content with the situation, and look at the passing of the world with a smile.