We all know that in the country if you buy things to go to Taobao to buy it, for the ordinary netizens they will not be in the browser to search for the domain name to enter the Taobao site, more in the search engine search "Taobao", we can see that Taobao blocking search engine spiders crawl, Taobao in the root directory of the site to set the relevant commands in the The robots.txt file in the root directory of the site to set the relevant commands to prohibit search engine spiders to obtain web page information.

What is this "robots.txt"?

robots.txt (in lowercase letters) is an ASCII-encoded text file that is stored in the root directory of some websites' servers. Its role is to tell the search engine crawler robot (also known as web spiders, roamers) which content in the site is not allowed to be crawled by the search engine crawlers, which are allowed to be crawled. Because URLs are case-sensitive in some systems, robots.txt filenames are standardized to lowercase. robots.txt is placed in the root directory of the website.

Why does Taobao do this?

There are a variety of online statements, in my opinion there are 2 important reasons:

1, to protect the user's information

The search engine on the dynamic page of the crawl technology is more and more mature, in the case of not blocking the crawler, the crawler can be captured to the user login after the information, after the capture is possible in the search engine search time to show up based on the premise that the crawler can be captured after the user login information. This premise Taobao consider shielding.

2, grab the traffic entrance

Imagine if in the Baidu search, search Taobao store name can show the Taobao store, how many people will directly use the Baidu search Taobao store to buy what they want, and will not first enter the Taobao network and then search Taobao store? The first part of the traffic belongs to the Taobao Baidu to take away, Taobao can do what?

Then imagine, these flows are robbed by Baidu after the Taobao home page of the big picture recommended how much money can be sold? The first thing you need to do is to get your hands dirty. The first thing you need to do is to get your hands on some of the most popular products in the world, and you'll be able to do it all on your own. These flows are taken away by Baidu is a huge loss for Taobao.

Comprehensive these 2 important factors we understand that Taobao by banning the search engine crawler crawl, grab a large number of traffic, while Taobao is also in the search engine to buy traffic, as follows:

Tmall itself is shielded from the spider crawling, but they are in the Baidu paid to promote the ads, which is tantamount to paying to buy the Baidu search engine traffic.

Taobao has a large flow of big data in the case of the development of a variety of products for merchants, so that merchants on this part of their own needs to pay for the traffic to buy, so as to realize the flow of cash.

/