Web Analytics Tutorial

Lesson 3 – Search Engines

Updating My Listing

Figure 8. The Known Robots report tells you when your site was last crawled by each search engine.

As mentioned previously, search engines will crawl your site with a robot. The paths of these robots and their traffic can be analyzed. Summary provides the Known Robots report, Figure 8, showing details about the visits of robots that it is aware of. Not all robots are from search engines, but you can usually identify them in the report because they have the phrase “Search Indexer” in them. Using the Known Robots reports, you can determine how recently your site was last indexed by each engine. For example, in Figure 8, Google last indexed the site on 02/10/02.

When all other factors are equal (which is rare) search engines often list most recently crawled sites first, so you want your site to be crawled on a regular basis. However, because of the volume of information they have to process, search engines will punish you if you submit your site too often, especially if there has been no change to the content of the site. Sites that have recent Last Hit Dates in the Known Robots report you can safely ignore for submission purposes – they are already crawling your site regularly, so you do not need to help them find it. If there are important engines that have not recently indexed your site, then you will probably want to resubmit to their indexing queue.

There are a plethora of services available on the web to assist you in submitting your site to search engines. Some will claim they submit you to 400 search engines, some 4,000! A quick look at your Search Engine report will tell you that beyond the first ten engines, the rest are insignificant. When you filter out the engines that have recently crawled your site, you are left with a very manageable number of submissions that should not take too long to do by hand. For each search engine, go to the web site and look for a submission link (usually near the bottom). You can probably even reduce your workload one more step by starting with Inktomi, which powers most of the top ten search sites.

Increasing Crawl Frequency

Most robots crawl sites by following links. These links, however, are often sorted by popularity. If a site is referenced from many other pages, then the robot will promote it in the queue for crawling. So, in addition to increasing your ranking (on search engines such as Google), getting other sites to link to yours will increase your index frequency. This reduces the number of manual submissions you need to do and makes your work easier.

Web Directories

At this point we should also mention web directories. The two major ones are Yahoo! and Open Directory. While these are not strictly search engines, they are used by a huge number of people to locate web content. Directories never crawl sites – you submit pages and a human looks at them and decided whether to include it or not. You will need to submit to these services as well and may have to wait a while before your site gets listed. But you can use the Referrer Report to see which directory pages are sending you traffic.