robots-wizard.com

Overview

A robots.txt file allows you to keep material out of search engines. For a full description of robots.txt, please go to http://www.robotstxt.org.

robots.wizard™ allows you to browse your file system and indicate which files and directories should not be indexed. This tool is intended to make it easier to create and maintain the robots.txt file.

Examples of pages and files that you may want to keep out of search engines are:

• Administration consoles that have links on site pages. Suppressing the links will provide even better security. This may not be possible if the content is delivered by an application.
  
• Supporting files that are pulled in through frames or iframes. These pages may be indexed out of context.
  
• Stylesheets and scripts that are not intended to stand alone
  
• Client access pages that require authentication.
  
• Pages that have content that should not be indexed for some reason.
  
• Multimedia content such as audio, video, or images.
  
• Data directories.
  

robots.wizard(tm)

robots.wizard provides three free tools to help manage robots.txt files. robots.txt files help search engines decide which pages of your site to index.

• robots.wizard allows you to create and edit a robots.txt file using a nice file system navigation inteface.
• robots.checker checks a robots.txt file for syntax. An invalid robots.txt file may be disregarded or misinterpreted by search engine spiders.
• robots.spider spiders a site and checks the robots.txt file for references. This allows you to see the types of content the search engines might index, so you can decide if you want them to be indexed or not.

robots.design(tm)

Uses the colors in an image to generate sed commands suitable for modifying site files. Allows extremely cost-efficient translation of a comp, site snapshot, or existing CSS color attributes into an application skin. This can save a tremendous amount of time.

Run/Demo/Download

These tools are distributed under GPL and the source code is well documented

_{Contact bgamrat at the domain for information.}

Copyright © 2008 Betsy Gamrat. All Rights Reserved. robots.wizard, robots.checker, robots.spider are trademarks of Betsy Gamrat