Comments on: How to create a “robots.txt” file? http://www.invesp.com/blog/blogging/how-to-create-a-robotstxt-file.html Conversion Rate Optimization Blog Mon, 30 Jan 2012 16:07:43 +0000 hourly 1 http://wordpress.org/?v=3.1 By: xHTML Coding http://www.invesp.com/blog/blogging/how-to-create-a-robotstxt-file.html#comment-3429 xHTML Coding Mon, 15 Sep 2008 15:23:15 +0000 http://www.invesp.com/blog/blogging/how-to-create-a-robotstxt-file.html#comment-3429 With Wordpress, Google seems to like to dump the dupe content penalty in the tags and the archive pages. The robots.txt is the best possible way to sort this major problem which could be a real pain in the... With WordPress, Google seems to like to dump the dupe content penalty in the tags and the archive pages.

The robots.txt is the best possible way to sort this major problem which could be a real pain in the…

]]> By: Afef http://www.invesp.com/blog/blogging/how-to-create-a-robotstxt-file.html#comment-2711 Afef Thu, 29 May 2008 10:57:16 +0000 http://www.invesp.com/blog/blogging/how-to-create-a-robotstxt-file.html#comment-2711 I just realized i didn't have a robots text file - good thing i came across your blog post! Thanks for the info! Those are some really helpful tips for beginers! :) I just realized i didn’t have a robots text file – good thing i came across your blog post! Thanks for the info! Those are some really helpful tips for beginers! :)

]]> By: Jason http://www.invesp.com/blog/blogging/how-to-create-a-robotstxt-file.html#comment-2367 Jason Fri, 11 Apr 2008 05:58:34 +0000 http://www.invesp.com/blog/blogging/how-to-create-a-robotstxt-file.html#comment-2367 Your blog is very informative, I have learned so much from it. It is like daily newspaper :). Added to fav Your blog is very informative, I have learned so much from it. It is like daily newspaper :) . Added to fav

]]> By: Andy Bolton http://www.invesp.com/blog/blogging/how-to-create-a-robotstxt-file.html#comment-2296 Andy Bolton Fri, 28 Mar 2008 18:05:09 +0000 http://www.invesp.com/blog/blogging/how-to-create-a-robotstxt-file.html#comment-2296 Thanks mate... Makes things a little clearer. Keep these blogs coming I may want more. Thanks mate… Makes things a little clearer. Keep these blogs coming I may want more.

]]> By: Quick SEO fix for Wordpress blogs http://www.invesp.com/blog/blogging/how-to-create-a-robotstxt-file.html#comment-2263 Quick SEO fix for Wordpress blogs Sun, 23 Mar 2008 00:27:43 +0000 http://www.invesp.com/blog/blogging/how-to-create-a-robotstxt-file.html#comment-2263 [...] change the link structure. Recommended Reading: I’ve found another quick fix at invesp.com with a quick fix to your robots.txt which also stops crawlers from double-indexing pages on your [...] [...] change the link structure. Recommended Reading: I’ve found another quick fix at invesp.com with a quick fix to your robots.txt which also stops crawlers from double-indexing pages on your [...]

]]> By: 152 Dos & Don’ts of Business Blogging | The Invesp Blog http://www.invesp.com/blog/blogging/how-to-create-a-robotstxt-file.html#comment-2180 152 Dos & Don’ts of Business Blogging | The Invesp Blog Thu, 13 Mar 2008 20:20:37 +0000 http://www.invesp.com/blog/blogging/how-to-create-a-robotstxt-file.html#comment-2180 [...] Do create a robot.txt file [...] [...] Do create a robot.txt file [...]

]]> By: Quick SEO fix for Wordpress blogs http://www.invesp.com/blog/blogging/how-to-create-a-robotstxt-file.html#comment-2075 Quick SEO fix for Wordpress blogs Tue, 26 Feb 2008 16:10:24 +0000 http://www.invesp.com/blog/blogging/how-to-create-a-robotstxt-file.html#comment-2075 [...] Reading: I’ve found another quick fix at invesp.com with a quick fix to your robots.txt which also stops crawlers from double-indexing pages on your [...] [...] Reading: I’ve found another quick fix at invesp.com with a quick fix to your robots.txt which also stops crawlers from double-indexing pages on your [...]

]]> By: Quick Optimization Fix for WordPress Links http://www.invesp.com/blog/blogging/how-to-create-a-robotstxt-file.html#comment-2060 Quick Optimization Fix for WordPress Links Sun, 24 Feb 2008 18:39:37 +0000 http://www.invesp.com/blog/blogging/how-to-create-a-robotstxt-file.html#comment-2060 [...] should automatically change the link structure. *Update:* I’ve found another quick fix at invesp.com with a quick fix to your robots.txt which also stops crawlers from double-indexing pages on your [...] [...] should automatically change the link structure. *Update:* I’ve found another quick fix at invesp.com with a quick fix to your robots.txt which also stops crawlers from double-indexing pages on your [...]

]]> By: Phil Booker http://www.invesp.com/blog/blogging/how-to-create-a-robotstxt-file.html#comment-1639 Phil Booker Thu, 27 Dec 2007 15:08:16 +0000 http://www.invesp.com/blog/blogging/how-to-create-a-robotstxt-file.html#comment-1639 Ahh... Don't crawl on certain pages is fine. but how do you stop all random crawls that try and recover emails and stuff from your website? Please... I want Google, Yahoo and MSN etc but not the spamming crawlers. What to do? Phil Ahh… Don’t crawl on certain pages is fine. but how do you stop all random crawls that try and recover emails and stuff from your website?
Please… I want Google, Yahoo and MSN etc but not the spamming crawlers.
What to do?
Phil

]]> By: Leslie http://www.invesp.com/blog/blogging/how-to-create-a-robotstxt-file.html#comment-1478 Leslie Sat, 01 Dec 2007 18:34:23 +0000 http://www.invesp.com/blog/blogging/how-to-create-a-robotstxt-file.html#comment-1478 I need to know how to create the robot.text file in the first place, can you help? I need to know how to create the robot.text file in the first place, can you help?

]]>