Re: [Summary-Talk] SQL?

[Jason Linhart <jason@xxxxxxxxxxx>]

On 7/12/04 10:44 AM Greg Swallow (gswallow@www.IN.gov) wrote:

>However, I personally love this software.  Are there any plans in the 
>works to transfer Summary to Oracle/MySQL/MSSQL?  Would this be a 
>reasonable feature request?

Switching Summary over to an SQL database would slow things down by a 
factor of about 10,000 (or more). That isn't a typo. Summary's internal 
database is very very optimized for speed in the particular kinds of 
situations that Summary encounters, while an SQL database is optimized 
for other things. Some log analysis programs allow exporting their 
internal database to SQL, but no commercial package actually uses an SQL 
database when processing logs.

>Lately our database file 
>has grown to 1.3GB in size, and it's causing serious problems for us -- 
>stats take 4+ days to process log files and write static reports, 
>crashes regularly, etc.  We have 1 GB of RAM

You should look at the memory FAQ, 
<http://summary.net/manual/faq.html#memorylack>. If you haven't already 
done the things it recommends, then you are in for some major memory 
savings. If you have already done most of the things there, then you may 
want to upgrade to Summary 2.5, which has some new memory saving options 
that are sure to take care of things.

If you get memory usage down below 1 Gig Summary will speed up 
*dramatically*. On a fairly modern machine Summary should be able to 
process your logs in one to three hours. We have customers that process 
far larger collections of logs. Once you get into that log file size 
range you do need to watch the memory usage occasionally. Having made any 
necessary adjustments, things should be fine.

>I'm also beginning to think that a SQL-based 
>statistics package will be a lot more efficient.  Therefore, we're 
>probably shopping around for a larger stats package very soon.

There are very few log analysis programs that won't have the exact same 
kinds of issues with memory usage that Summary has, and they tend to be 
*very* expensive and require clusters of several machines to run.

Jason

-----------------
Jason@Summary.Net
-----------------
Dr. Seuss books . . . can be read and enjoyed on several levels. For
example, 'One Fish Two Fish, Red Fish Blue Fish' can be deconstructed
as a searing indictment of the narrow-minded binary counting system.
  -- Peter van der Linden, Expert C Programming, Deep C Secrets
-------------
Go to <http://summary.net/list.html> to update subscription info.