Robots.txt is a visual editor for Robot Exclusion Files and a log analyzer
software. It allows a user to quickly and easily create the Robots.txt files
required to instruct search engine spiders, which parts of a Web site are not
to be indexed and made searchable by the general Web public and then to identify
spiders, which do not keep to those instructions. The program provides a user
with a way to log onto his FTP or local network server and then select the documents
and directories which are not to be made searchable.
Robots.txt Editor program will help you optimize and fine tune your
Web site for search engines. No more wasting time creating your Robots.txt file
by hand and wondering if it is formatted properly.
By means of this program you will be able to:
- Visually generate industry standard Robots.txt files
- Identify malicious and unwanted spiders and ban them from your site
- Direct search engine crawlers to the appropriate pages for multilingual
sites
- Keep spiders out of sensitive and private areas of your Web site
- Upload the correctly formatted robots.txt file directly to your FTP server
not switching from Robots.txt Editor
- Track spider visits
- Create spider visits reports in HTML, Microsoft Excel CSV and XML formats
Controlling Spiders with Robots.txt Files
How can you take control over robot intruders and make them useful for your
site?
Before indexing your site, the spider downloads Robots.txt file that contains
instructions on what should and what should not be indexed. Therefore a key
to controlling spiders is Robots.txt file. If you have a large Web site or update
it frequently, creating and editing it will be a hard and dull work.
Robots.txt Editor is an easy-to-navigate visual editor that will enable you
to specify different directives for selected spiders in specific areas of the
site and generate the Robots.txt file quickly and easily. You will not have
to waste your time creating Robots.txt files by hand and wondering if it is
formatted correctly! Used along with other Web site promotion software of Net
Promoter, Robots.txt Editor enhances the efficiency and effectiveness of your
optimization strategy.
- Select which search engine spiders to work with
- Add, edit or delete your own spiders
- Visually edit your Robots.txt file
- Import existing Robots.txt files
- Read your Web site structure either locally or from the Web server
- Group spiders by primary language or country and set rules for the group
- Program contains database of more than 150 US and international spider user-agent
identifiers
Robots.txt Editor along with FTP Uploader and a number of other useful features,
make up a powerful and handy tool for controlling spiders on your Web site.
Analyzing Spiders' Visits To Your Web Site
All spider visits are recorded in a log file stored at your server. Again,
you may review all log files manually and compose your database.
But why dig yourself into this dull and time-consuming task? You may take advantage
of our Log Analyzer module that allows you to download and save log files with
information on spider visits, and to export it into different file formats.
Besides, the information will be structured. You'll be able to track the visits
of spiders by time period or by specific pages, and to generate reports by various
criteria.
This information is extremely useful when promoting your Web site for major
search engines around the world. You know exactly when the spider or robot visits,
and what pages on your Web site it indexes. You don't have to guess and try
to decipher if a new page was indexed by a search engine spider or not. Your
log file contains all this information.
Spider Log Analyzer
- Uses log file analysis to detect spider visits and pages indexed
- Provides log file analysis for unlimited number of log files at the same
time
- Mines log files either locally or via FTP
- Allows browsing of your Web server via FTP to find your log file
- Supports compressed log files
- Supports log files from Windows, Linux and MAC Web servers
- Can add any new spiders found on your Web site to Robots.txt Editor
- Auto detects your log file format, and in a rare case where it can't, you
can specifiy the format profile manually in Profile Manager
- Indicates visually the pages requested by spiders by mistake (i.e., 404
Not Found) and redirected requests
- Indicates pages and images that were accessed against the rules of project's
Robots.txt file
- Exports spider and page visits into Microsoft Excel CSV (Comma Separated
Values) file or XML format file
- Exports raw log entries of spider visits to W3C Extended Log File
- Exports all spider IP addresses so they can be used by cloaking software
- Provides printable customizable reports in HTML format
