How to share through robot flexible control spiders crawl


Shanghai dragon no one does not know spider, spider and robot.txt file is the exchange of documents, a lot of people would say the introduction how to let the spider to crawl, but few people know that using robot.txt to manipulate robot.txt spiders, spiders can make good use of the words for you. Below I introduce how to manipulate spider.

. The same page


use robots file can protect website security, some people may ask how to protect the site, if no robots files shield the spiders to crawl admin background page or background the following documents, the spider can include the default page, so that hackers can take advantage of organic.


six. No two level domain name capture


site map

Robots also supports the site map file submission function, submit site map can better let the spider crawling website, but most people do not have to submit web site map, although there is no great influence recommendations.

Four. Submit the

search engine is a new thing, not to repeat or have included pages, even included the post will be deleted, so the beginning we can open the robots file, write to read this page banned Spider code, give a good impression to the spider, and greatly reduce the effect of impression branch. The future spider crawling website.

two website security

points above are enough to let you control the spider a lot, need to communicate as people, we also need to communicate and spiders. In this paper, by Schindler released in Fuzhou Shanghai dragon printing network 贵族宝贝fzywzx贵族宝贝, reproduced please keep connection!  


sites will provide some special services for VIP members, but do not think this is the search engine to search service.

also hotlink several people, but once the search engine "hotlink" the guy you 100M broadband is also too much, if you do not picture website, and do not want to be the search engine "hotlink" your website.

Leave a Reply

Your email address will not be published. Required fields are marked *