Comment about:  Webmaster Tips: Blocking Selected User-Agents
.
September 21, 2008

Great Post!

I agree. The bad guys are always one step ahead so security should always be a concern. In my article, I detailed how to write your robots.txt, and I agree that this is should not really be used for security purposes.

How to write robots.txt:
http://sin8.com/tip-3-its-important-to-hide-parts-of-your-site-from-search-engines
Carl


Syndication OptionsRSS (Rich Site Summary) Feed Atom Feed OPML (Outline Processor Language) Feed MYST-ML (MyST Markup Language) Content Feed MS-Office Smart Tag Subscription