Robot.txt file - DevHunters.com l Webmaster Forum - Web Advertising - Web Design

weblaunchphxx · 12-13-2007, 02:54 AM

The robot text is a file that restricts the Search Engine spiders from indexing certain pages of the website.You may prevent your personal/incomplete pages as well as guestbook pages of your site from indexing through this file. Many webmasters use it to avoid spamming. Robot txt codes are listed below.

HTML meta tags for robots.
< meta name="robots" content="noindex,nofollow" />

To allows all robots
User-agent: *
Disallow:

To all robots out
User-agent: *
Disallow: /

To prevent pages from all crawlers
User-agent: *
Disallow: /page name/

To prevent pages from specific crawler
User-agent: GoogleBot
Disallow: /page name/

weblord · 12-13-2007, 03:00 AM

thanks for the info.

bizwiz · 12-13-2007, 09:35 AM

Nice info to let the spiders crawl my sites faster :)

karthikeyan · 12-13-2007, 11:18 AM

Cool and nice info !

Simonas · 12-13-2007, 02:02 PM

Thank you for information ;)

Captain Tycoon · 12-13-2007, 03:15 PM

Thanks for sharing the information, Its really useful if you have a large website!

mountlaurel · 12-13-2007, 09:49 PM

i really like the explanation really thanks..

Have A Look At Some Of Our Webmaster Related Sponsors!
Script Directory	Advertise Here	Market Leverage	Free Templates	3D Guide-characters	Advertise Here

12-13-2007, 02:54 AM	#1 (permalink)
weblaunchphxx Field Master Join Date: Aug 2007 Posts: 122 iTrader: 0 / 0% Hunter Bux: 0	Robot.txt file The robot text is a file that restricts the Search Engine spiders from indexing certain pages of the website.You may prevent your personal/incomplete pages as well as guestbook pages of your site from indexing through this file. Many webmasters use it to avoid spamming. Robot txt codes are listed below. HTML meta tags for robots. < meta name="robots" content="noindex,nofollow" /> To allows all robots User-agent: * Disallow: To all robots out User-agent: * Disallow: / To prevent pages from all crawlers User-agent: * Disallow: /page name/ To prevent pages from specific crawler User-agent: GoogleBot Disallow: /page name/ __________________ Graphics Design \|\| Looking for Search Engine Marketing

12-13-2007, 03:00 AM	#2 (permalink)
weblord Sharp Shooter Join Date: Mar 2007 Location: Philippines Posts: 1,325 iTrader: 1 / 100% Hunter Bux: 611.00	thanks for the info. __________________ Software \| William R. Nabaza Blogs \| U.S.$5 Domains \| FriendTEK \| Arcontica \| Maypajo\| www.sendload.net \| Forums\| Unfolds Videos \| WebTor\| Url Shortener\| 25%discount\| Free Hosting Blogs\| \| Web Directory\| Weblord's Community\| Freebies\| Cash Surf \| Funny Jokes \| www.bloghaven.org \| Videos \| Videos

12-13-2007, 09:35 AM	#3 (permalink)
bizwiz More Than Meets The Eye Join Date: Jun 2007 Posts: 1,783 iTrader: 0 / 0% Hunter Bux: 715.00	Nice info to let the spiders crawl my sites faster :) __________________ DropShip Business \| DropShip Business Link Building Service \| Coffee Shop \| 2 Girls 1 Finger \| Pókersuli * Free file hosting * \| $1 Bid Directory \| Dogs for Sale

12-13-2007, 11:18 AM	#4 (permalink)
karthikeyan Field Master Join Date: Jun 2007 Posts: 604 iTrader: 0 / 0% Hunter Bux: 509.00	Cool and nice info ! __________________ My Tech Blog

12-13-2007, 02:02 PM	#5 (permalink)
Simonas Field Master Castle Defend Champion! Join Date: Oct 2007 Posts: 117 iTrader: 0 / 0% Hunter Bux: 54.00	Thank you for information ;)

12-13-2007, 03:15 PM	#6 (permalink)
Captain Tycoon Field Master Join Date: Oct 2007 Location: United Kingdom Posts: 469 iTrader: 0 / 0% Hunter Bux: 177.00	Thanks for sharing the information, Its really useful if you have a large website! __________________ Polar Byte - Your Free Hosting Services :eek: ☆ Qoni ☆ Voogi ☆ Aoni ☆ Amut ☆ :eek: ☆ Captain Tycoon ☆ Insurance Document ☆ Great Mesothelioma Guide ☆ Gamer ☆ Loans, Mortgage & Banks

12-13-2007, 09:49 PM	#7 (permalink)
mountlaurel Field Master Join Date: Oct 2007 Posts: 284 iTrader: 1 / 100% Hunter Bux: 260.00	i really like the explanation really thanks.. __________________ Barbie GAMES Myspace Proxy LinkSys Router Youtube Myspace Proxy Myspace Proxies Myspace Proxy Proxy List

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode