Summary.Net Archives
 
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Summary-Talk] Using Summary with logs rotated by logrotate - updated instructions

I was informed of some errors in my instructions already. Attached is an 
updated copy.

Nick

-- 
Nick Davis 
Associate Systems Administrator 
ndavis@iexposure.com 
Internet Exposure, Inc. 
http://www.iexposure.com  

(612)676-1946 
Web Development-Web Marketing-ISP Services
Title: Webstats Administration

Webstats Administration

by Nick

v1.0 July, 2003

  1. Add a new website to webstats
  2. Remove a website from webstats
  3. Rerun a fresh set of webstats
  4. Remove all of the *.gz log files

Setup summary stats

Logon to summary.server.tld as root, and su to user "summary". Go to /home/summary/logs/ and make a directory for that domain's log files to be downloaded into. The directory name should be <domainname.tld>. If the domain name is abcincorporated.com, the directory would be abcincoporated.com. You are done with this part and can log off of the summary server.

Next, you will need to configure Summary to download the log files and run stats for the site. The Summary configuration should be the same for every site and must be done via the web interface. Open up a web browser and go to http://summary.server.tld:9000/config. This will prompt you for a user/password. Look in PMS for the summary admin username and password. You should be at the Global Configuration screen after you login. Scroll all the way to the bottom and click on "Add New sub-report". This will bring up a new window with several boxes for you to fill in to create the first sub-report.

For "Site Descriptive Name", enter the name of the web site (ABC, Inc.)
For "Preferred Domain Name", enter www.domain.tld
For "Name required to access reports", enter the domain nameminus the tld (so abcincorporated.com would be abcincorporated)
For "Password required to access reports", enter the password as follows: a number, a common word, and then another number, such as 4cat73
For "Identifier for use in report URLs", enter the domain name minus the tld
For "Send report to e-mail address", enter summaryadmin@mydomain.tld
For "Other local domain names", enter at least domain.tld. If this site has multiple domain names, you should enter all domain names here. For all but the domain you used in Preferred Domain Name", you need to enter both www.domain.tld and domain.tld.
For "Domains to include in this sub-report", enter everything from the previous box (Other local domain names) and the "Preferred Domain Name".
For "Files to include in this sub-report", enter "*". Just thestar charactor.
Now click on the "Update Information" button. This will take you back to the Global Configuration page.

Next, you need to tell Summary where to download the log files from. Scroll down to the sub-report you just created and click on it. You should be at the sub-report configuration page, scroll down to the section below the boxes called "Configure" and click on "Log Downloading".
For "URL to download log from", enter "ftp://< server name that website is on> /< directory for that website's logs>/<name of log file>. The directory is relative to /var/log/apache/
For "Name required to access download", enter "summary"
For "Password required to access download", enter the password for user summary. It is in pms.
For "Store downloads in sub-folder named", enter domain.tld. This is the same as the directory you created in the first step.
Now click on "Update Information". You are now back to the Global Configuration page and you are done creating the first sub-report.

To create the second sub-report, scroll to the bottom and click on"Add New sub-report".

For "Site Descriptive Name", enter "domain.tld extra log file"
For "Preferred Domain Name", leave this field empty
For "Name required to access reports", enter "extra"
For "Password required to access reports", enter "extra"
For "Identifier for use in report URLs", enter "domain.tld.extra"
For "Send report to e-mail address", leave this field blank
For "Other local domain names", leave this field is blank! Ifthere is a "*" star in this box you must delete the star!
For "Domains to include in this sub-report", leave this field blank! If there is a "*" star in this box you must delete the star!
For "Files to include in this sub-report", leave this field blank! If there is a "*" star in this box you must delete the star!
Now click on the "Update Information" button. This will take you back to the Global Configuration page.

Next, you need to tell Summary where to download the log files from. Scroll down to the sub-report you just created and click on it. You should be at the sub-report configuration page, scroll down to the section below the boxes called "Configure" and click on "Log Downloading".
For "URL to download log from", enter "ftp://< server name that website is on> /< directory for that website's logs>/<name of logfile.0>. The directory is relative to /var/log/apache/ The ".0" is to ensure that we also get the main logfile after it has been rotated so we do not miss any logs due to rotation!
For "Name required to access download", enter "summary"
For "Password required to access download", enter the password for user summary. It is in pms.
For "Store downloads in sub-folder named", enter domain.tld. This is the same as the directory you created in the first step.
Now click on "Update Information". You are now back to the Global Configuration page and you are done creating the second sub-report.

Extra helpful configs!

  • This step is not necessary, but can be helpful! If you want the log reports to show that they started on the date the website went live instead of the datethe website was added to the server you can do so. Click on the first sub-reportfor that site. At the bottom is the "Configure" section, click on "Time Units".The only thing you need to change is the first box "Start Processing on". You can enter the date that the site went live in there, and Summary will ignore all log entries before that date:) If you are wondering what log entries there arebefore a site goes live, here are a few: web robots, developers, and testers forthe site.

  • Another helpful config! You can setup filters That will allow you to ignore IP addresses and/or domains names for this sub report. I would recommend entering both the IP block and domain names. You can use the wild card "*" where needed to ignore a block of IP's or all subdomains of a domain.

The reason we need two sub-reports: Log downloading is complicated by log rotation, so we get the main log file and the first rotated file to ensure that we get all logs. Log rotation is set to run every Sunday morning at 0625.Summary is set to run at 0600 every day, so we can get most of the logs before they are rotated. Summary is running in "incremental mode" meaning it only downloads the new parts of the log files. Once a log file has changed from log.0to log.1.gz, summary cannot detect that it is the same file that used to be log.0. This is why we don't just use the .log* wildcard to grab all log files for download.

Remove a website from summary

  • Open up a web browser and go to http://summary.server.tld:9000/config. This will prompt you for a user/password. (look in PMS for the summary admin username and password.) Once you enter the username and password, you should be at the Global Configuration screen.
  • Scroll down to the sub-report you want to remove and click on it. This will bring up the configuration page for the sub-report.
  • Near the bottom of the page is a button called "Delete sub-report". Click that button and say bye bye to the sub-report. Now do the same for the second sub-report for the site you want to remove.
  • ssh to summary.server.tld as su to user "root", then su to user "summary".
  • Go to /home/summary/logs/ and find the directory for that domain's log files to be downloaded into. The directory name should be <domainname.tld>. If the domain name is abcincorporated.com, the directory would be abcincoporated.com.
  • Delete the directory for that website. You are done with this part and can log off of the summary server.

Rerun a fresh set of webstats

To rerun a complete set of webstats there are 4 important steps you need to do.

  1. Modify summary.conf to download all log files instead of just .log and .log.0.
    • ssh to summary.server.tld and su to user "root", then su to user "summary".
    • Go to /home/summary/config/ and run vi summary.conf.
    • In vi type this: ":g/log\.0/s//log\.\*/g" and it will convert all of the log.0's to log.*.
    • Save the file. Now all log files will be downloaded when summary does the next stats run.
  2. Reset the webstats database.
    • Open up a web browser and go to http://summary.server.tld:9000/config. This will prompt you for a user/password. (look in PMS for the summary admin username and password.) Once you enter the username and password, you should be at the Global Configuration screen.
    • Uncheck the check box next to "Enable Incremental Processing".
    • Click on the "Update Information" button to save your changes.
    • Check the check box next to "Enable Incremental Processing".
    • Click on the "Update Information" button to save your changes. Now the webstats database has been reset.
  3. Run the stats.
    • You can either run the stats manually or wait for summary to run overnight by itself.
    • To run the stats manually, click on "Process Logs Now" on the Global Configuration page.
    • DO NOT DO THE NEXT STEPS UNTIL THE STATS HAVE COMPLETED RUNNING
  4. Modify summary.conf to download only .log and .log.0 instead of all log files.
    • ssh to summary.server.tld and su to user "root", then su to user "summary".
    • Go to /home/summary/config/ and run vi summary.conf.
    • In vi type this: ":g/log\.\*/s//log\.0/g" and it will convert all of the log.*'s to log.0.
    • Save the file. Now only the log and log.0 log files will be downloaded when summary does the next stats run.
  5. Remove all of the *.gz log files.
    • ssh to summary.server.tld and su to user "root", then su to user "summary".
    • Go to /home/summary/logs/ and run this: "find . -name "*.gz" -exec rm -f {} \;".
    • That command is very touchy, so enter it exactly as written! It will find all files in all directorys sub to logs that end in .gz and delete them.
    • Once that is done, you are done:)
    • We want to remove the *.gz files because summary will get confused when log.0 is rotated into log.1.gz. It cannot tell that they used to be the same file!