The site: command is used to know that there is something you need to find in a certain site, so you can limit your search to that site and improve the efficiency of your query.

The way to use it is to add "site:site domain" to the end of the query. For example, you can query the site: site: so-and-so.com.

Site command syntax format has two kinds:

1, site: domain name keyword

2, keyword site: domain name

site: with or without www results may be different, because some domain names also include second-level domain names, such as : site:www.某某.com和site:某某.com,搜索结果就不一样, between site: and site name, do not take a space.

Web crawlers download web pages from the World Wide Web for search engines and are an important component of search engines. Traditional crawlers start with the URL of one or a number of initial web pages, get the URL on the initial web page, and in the process of crawling the web page, continuously extract new URLs from the current page into the queue, until the system meets certain stopping conditions.

The workflow of a focused crawler is more complex, and it needs to filter out links that are irrelevant to the topic according to certain web analyzing algorithms, and keep the useful links and put them into a queue of URLs waiting to be crawled.

Relative to general-purpose web crawlers, focused crawlers also need to solve three main problems:

(1) description or definition of the crawling target;

(2) analysis and filtering of web pages or data;

(3) search strategy for URLs.

Baidu Encyclopedia-Site Command

Baidu Encyclopedia-Web Crawler

Baidu Encyclopedia-SITE

Baidu Encyclopedia-Search Engine Inclusion