Web Analytics Tutorial

Lesson 11 – Site and Server Diagnostics

Broken Links

Figure 1. The Bad Links report lets you see where broken links are on your site.

Despite the best efforts of designers, even simple sites can end up with broken links in them. This looks bad and frustrates customers, so you want to fix them as soon as possible. To help you maintain your site, Summary has the Bad Links report , Figure 1, that show which pages on your site failed and the local referrer that was associated with it. You can use this report to quickly find and fix the links. Go to the referring page and find the link to the request that failed.

If your site was well built and is well maintained, this report will be empty – your clue that you have done a good job. The Bad Links report shows broken links on your web site over all time. So when you fix the broken links, they will still show up in the reports because, historically, they were broken at one point.

Bad Referrers

Figure 2. The Failed Referrers report tells you where other sites have broken links to yours.

When links on your site are broken, it is easy to fix them. However, often other sites will have links to your site that do not work, either because the link has a typo in it, or because it links to a page that existed at one point, but no longer does. Summary’s Failed Referrers report, Figure 2, lists all requests to your server from external referrers that failed. You can go through this list and find the pages on other web sites with broken links. Following the referrers, you may be able to find contact information for the manager of the site and request that he or she fix the error. However, it is often easier to create a page on your site matching the failed request than to get someone to change another site. As with the Bad Links report, the data in the Failed Referrers is historical, so it may include references to broken links that have been fixed.

Figure 3. The Failed Requests report shows all requests that failed, for any reason.

Other Failed Requests

Figure 3 shows the Failed Requests report. This show all requests to your server that were not delivered. This failure could be because the requested file does not exist or because it is a CGI program or dynamic web page that produced an error when it was trying to run. The failed requests that are not in the Bad Links or Failed Referrers reports are generally created by robots. This is because those reports are dependent on referrer information and most robots don not provide that. If you make changes to your site and remove pages, then robots may try to request the old pages that were previously in their indexes just to see if the pages are still there (and to see if the content has changed.) You will also likely see some typos from users who tried to type a URL to your site into their browser but accidentally typed it incorrectly. The Failed Request report includes the Last Hit Date column which can help you determine if a failure has been fixed on your site.

Two entries you will often see near the top of your Failed Requests report are /robots.txt and /favicon.ico. The robots.txt file is something that well-behaved robots or spiders check to see what parts of your site they are allowed to index. In Lesson 12 - Investigating Troublemakers we will discuss the robots.txt file in more detail and provide a basic file you can use to allow full access by robots without getting failed requests.

Microsoft Internet Explorer 5 introduced a new technique to allow you to customize the icon that your visitors see when they save links to your site. IE 5 and later (and now Konqueror and perhaps other browsers) look for a file called favicon.ico on your web server when the user bookmarks a page on your site or saves a link to it on her computer. IE looks first at the root of your site (/favicon.ico) then in the directory where the page is located. If you create an icon in the root of your site it will be applied to all pages. If you have one in a particular directory, then each directory can have its own icon. You can find complete details on how to set up your site with favicon.ico in this Web Developer’s Virtual Library article.

If you have always had a /robots.txt or /favicon.ico file on your web site, then these items will not appear in your Failed Requests report. If you added one at one point, the Failed Requests report will always list the historical failures that are listed in your log files, but the Last Hit Date column will be old enough to indicate that the error no longer exists.