Summary.Net Archives
 
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Summary-Talk] Using Summary with logs rotated by logrotate

Attached is an html file that contains all of the instructions I have written 
for our use of summary. It has instructions on how to get around the log 
rotation problem where the rotated files are gzip'd. 

We use summary on linux and have incremental processing enabled. Most of the 
theory should work across the board.

Our log files rotate like this:

site.log
site.log.0
site.log.1.gz
site.log.2.gz
...

Maybe someone has better methods of doing things. In that case I'd be 
interested to hear those methods.

Hope that helps!

Nick

On Thursday 31 July 2003 04:49, nvarney@edt.fr wrote:
> > If you have logrotate setup to gzip the logs, you need to setup two
> > subreports
> > for each set of logs for summary to work properly.
>
> I don't see what do you mean.
>
> > If you are not zipping the logs, you can just use a wild card in the
> > logfile
> > name in the summary config to get all of your logs. Everytime you logs
> > get
> > rotated, summary will figure out which are the new files and adjust
> > accordingly.
> >
> > DownURL: ftp://webserver.domain.com/logsfolder/logfile.log*
> >
> > This will get logfile.log, logfile.log.1, logfile.log.2 etc.. And will
> > work
> > fine as long as you do not delete the logfiles off of your summary
> > server.
>
> That's how I configured Summary.
> But Summary don't keep all downloaded logs, file are overwriten due to
> checksum fail
>
> I will surely find another way to rotate logs on the server with date
> related names.
>
> Nicolas
> -------------
> Go to <http://summary.net/list.html> to update subscription info.

-- 
Nick Davis 
Associate Systems Administrator 
ndavis@iexposure.com 
Internet Exposure, Inc. 
http://www.iexposure.com  

(612)676-1946 
Web Development-Web Marketing-ISP Services
Title: Webstats Administration

Webstats Administration

by Nick

v1.0 July, 2003

  1. Add a new website to webstats
  2. Remove a website from webstats
  3. Rerun a fresh set of webstats
  4. Remove all of the *.gz logfiles

Setup summary stats

Logon to the summary server and su to root, and su to user "summary". Go to /home/summary/logs/ and make a directory for that domain's logfiles to be downloaded into. The directory name should be <domainname.tld>. If the domain name is abcincorporated.com, the directory would be abcincoporated.com. You are done with this part and can log off of the summary server.

Next, you will need to configure Summary to download the logfiles and run stats for the site. The Summary configuration should be the same for every site and must be done via the web interface. Open up a web browser and go to http://<summary.server.tld>:9000/config. This will prompt you for a user/password. You should be at the Global Configuration screen after you login. Scroll all the way to the bottom and click on "Add New Sub-report". This will bring up a new window with several boxes for you to fill in to create the first sub-report.

For "Site Descriptive Name", enter the name of the web site (abc, Inc.)
For "Preferred Domain Name", enter www.domain.tld
For "Name required to access reports", enter the domain nameminus the tld (so abcincorporated.com would be abcincorporated)
For "Password required to access reports", enter the password as follows: a number, a common word, and then another number, such as 4cat73
For "Identifier for use in report URLs", enter the domain name minus the tld
For "Send report to e-mail address", enter tjust@iexposure.com
For "Other local domain names", enter at least domain.tld. If this site has multiple domain names, you should enter all domain names here. For all but the domain you used in Preferred Domain Name", you need to enter both www.domain.tld and domain.tld.
For "Domains to include in this Sub-report", enter everything from the previous box (Other local domain names) and the "Preferred Domain Name".
For "Files to include in this Sub-report", enter "*". Just thestar charactor.
Now click on the "Update Information" button. This will take you back to the Global Configuration page.

Next, you need to tell Summary where to download the logfiles from. Scroll down to the sub-report you just created and click on it. You should be at the sub-report configuration page, scroll down to the section below the boxes called "Configure" and click on "Log Downloading".
For "URL to download log from", enter "ftp://< server name that website is on> /< directory for that website's logs; relative to /var/log/apache/>/< logfile name>
For "Name required to access download", enter "summary"
For "Password required to access download", enter the password for user summary.
For "Store downloads in sub-folder named", enter domain.tld. This is the same as the directory you created in the first step.>
Now click on "Update Information". You are now back to the Global Configuration page and you are done creating the first sub-report.

To create the second sub-report, scroll to the bottom and click on"Add New Sub-report".

For "Site Descriptive Name", enter "domain.tld extra logfile"
For "Preferred Domain Name", leave this field empty
For "Name required to access reports", enter "extra"
For "Password required to access reports", enter "extra"
For "Identifier for use in report URLs", enter "domain.tld.extra"
For "Send report to e-mail address", leave this field blank
For "Other local domain names", leave this field is blank! Ifthere is a "*" star in this box you must delete the star!
For "Domains to include in this Sub-report", leave this field blank! If there is a "*" star in this box you must delete the star!
For "Files to include in this Sub-report", leave this field blank! If there is a "*" star in this box you must delete the star!
Now click on the "Update Information" button. This will take you back to the Global Configuration page.

Next, you need to tell Summary where to download the logfiles from. Scroll down to the sub-report you just created and click on it. You should be at the sub-report configuration page, scroll down to the section below the boxes called "Configure" and click on "Log Downloading".
For "URL to download log from", enter "ftp://< server name that website is on> /< directory for that website's logs; relative to /var/log/apache/>/< logfile name.0> The ".0" is to ensure that we also get the main logfile after it has been rotated so we do not miss any logs due to rotation!
For "Name required to access download", enter "summary"
For "Password required to access download", enter the password for user summary.
For "Store downloads in sub-folder named", enter domain.tld. This is the same as the directory you created in the first step.>
Now click on "Update Information". You are now back to the Global Configuration page and you are done creating the second sub-report.

Extra helpful configs!

  • This step is not necessary, but can be helpful! If you want the log reports to show that they started on the date the website went live instead of the date the website was added to the server. Click on the first sub-report for that site. At the bottom is the "Configure" section, click on "Time Units".The only thing you need to change is the first box "Start Processing on". You can enter the date that the site went live in there, and Summary will ignore all log entries before that date:) If you are wondering what log entries there are before a site goes live, here are a few: web robots, developers, and testers for the site.

  • Another helpful config! You can setup filters That will allow you to ignore IP addresses and/or domains names for this subreport. I would recommend entering both the IP block and domain names. You can use the wild card "*" where needed to ignore a block of IP's or all subdomains of a domain.

The reason we need two sub-reports: Log downloading is complicated by log rotation, so we get the main logfile and the first rotated file to ensure thatwe get all logs. Log rotation is set to run every Sunday morning at 0625. Summary is set to run at 0600 every day, so we can get most of the logs beforethey are rotated. Summary is running in "incremental mode" meaning it only downloads the new parts of the log files. Once a log file has changed from log.0 to log.1.gz, summary cannot detect that it is the same file that used to be log.0. This is why we don't just use the .log* wildcard to grab all logfiles for download.

Remove a website from summary

  • Open up a web browser and go to http://<summary.server.tld>:9000/config. This will prompt you for a user/password. Once you enter the username and password, you should be at the Global Configuration screen.
  • Scroll down to the sub-report you want to remove and click on it. This will bring up the configuration page for the subreport.
  • Near the bottom of the page is a button called "Delete Sub-report". Click that button and say bye bye to the sub-report. Now do the same for the second Sub-report for the site you want to remove.
  • ssh to <summary.server.tld> as su to user "root", then su to user "summary".
  • Go to /home/summary/logs/ and find the directory for that domain's logfiles to be downloaded into. The directory name should be <domainname.tld>. If the domain name is abcincorporated.com, the directory would be abcincoporated.com.
  • Delete the directory for that website. You are done with this part and can log off of the summary server.

Rerun a fresh set of webstats

To rerun a complete set of webstats there are 5 important steps you need to do.

  1. Modify summary.conf to download all log files instead of just .log and .log.0.
    • ssh to <summary.server.tld> and su to user "root", then su to user "summary".
    • Go to /home/summary/config/ and run vi summary.conf.
    • In vi type this: ":g/log\.0/s//log\.\*/g" and it will convert all of the log.0's to log.*.
    • Save the file. Now all log files will be downloaded when summary does the next stats run.
  2. Reset the webstats database.
    • Open up a web browser and go to http://<summary.server.tld>:9000/config. This will prompt you for a user/password. Once you enter the username and password, you should be at the Global Configuration screen.
    • Uncheck the check box next to "Enable Incremental Processing".
    • Click on the "Update Information" button to save your changes.
    • Check the check box next to "Enable Incremental Processing".
    • Click on the "Update Information" button to save your changes. Now the webstats database has been reset.
  3. Run the stats.
    • You can either run the stats manually or wait for summary to run overnight by itself.
    • To run the stats manually, click on "Process Logs Now" on the Global Configuration page.
    • DO NOT DO THE NEXT STEPS UNTIL THE STATS HAVE COMPLETED RUNNING
  4. Modify summary.conf to download only .log and .log.0 instead of all log files.
    • ssh to <summary.server.tld> and su to user "root", then su to user "summary".
    • Go to /home/summary/config/ and run vi summary.conf.
    • In vi type this: ":g/log\.\*/s//log\.0/g" and it will convert all of the log.*'s to log.0.
    • Save the file. Now only the log and log.0 log files will be downloaded when summary does the next stats run.
  5. Remove all of the *.gz log files.
    • ssh to <summary.server.tld> and su to user "root", then su to user "summary".
    • Go to /home/summary/logs/ and run this: "find . -name "*.gz" -exec rm -f {} \;".
    • That command is very touchy, so enter it exactly as written! It will find all files in all directorys sub to logs that end in .gz and delete them.
    • Once that is done, you are done:)
    • We want to remove the *.gz files because summary will get confused when log.0 is rotated into log.1.gz. It cannot tell that they used to be the same file!