Track Search Engine Spiders / Robots

You have just placed your new site online.
How can you find out that a search engine is crawling your site ?
Some tool is needed to track search engine spiders or robots.
Or consult your logfiles. Boring and time consuming isn't it ?

Use TrackBot to track search engines, it sends you an e-mail when a search engine visits your site. This way you are informed, just seconds later.

What is TrackBot ?
TrackBot is a PHP script based on the script called RobotVisit from Steve Dawson.
I found his script difficult to update with new search engine spiders. So I modified it to my needs and gave it a new name.

This information will be mailed to you :

• Name of the search engine (user Agent)
• The page that is crawled
• IP and hostname of the search engine
• Exact date and time of the visit

The mail looks like this :

Tracked Search Engines
It will track these search engines by default:

• Google
• Yahoo (Slurp)
• MSN Search
• Alexa
• Alta Vista
• GigaBlast
• Ask Jeeves

You can add as many search engines as you wish. Add the correct botname in the script and it will also be tracked.
More information can be found on the Web Robots Database.

Installation
Installation is simple. Upload the script to your site and add the following code on the first line in the page that have to be tracked. Do this for every page.

I assume that your pages are written in PHP and not static HTML.

Testing 1 2 3
Add 'Mozilla' to the array SearchBot$ or any other word that could identify your browser. The script should then be triggered the moment you or someone else visits the page.
Remove it after testing or you get a mail for every visitor.
And those visitors are not search engines :-)

Warning
If your site contains many pages and you include the code on every page, you may end up with tons of mails.

Download
Download TrackBot

Help
This script is free to use, but donations via PayPal are allways welcome.
I'll use them to keep the site running.

or place a link on your page to this page using the following HTML code :