Robot.txt file
The robot text is a file that restricts the Search Engine spiders from indexing certain pages of the website.You may prevent your personal/incomplete pages as well as guestbook pages of your site from indexing through this file. Many webmasters use it to avoid spamming. Robot txt codes are listed below.
HTML meta tags for robots.
< meta name="robots" content="noindex,nofollow" />
To allows all robots
User-agent: *
Disallow:
To all robots out
User-agent: *
Disallow: /
To prevent pages from all crawlers
User-agent: *
Disallow: /page name/
To prevent pages from specific crawler
User-agent: GoogleBot
Disallow: /page name/
|