[Summary-Talk] processes exiting abnormally

[Dale LaFountain <dalel@xxxxxxxxxxxxx>]

Yesterday I found that my Summary server was stuck in the dns lookup
phase of it's daily incremental processing for several hours.  Looking
at summary.log, I found this error:

One of our processes exited abnormally!
One of our processes exited abnormally!

I quit and restarted summary.app, and the database came up as
corrupted.  Re-process.  After processing a few days worth of logs I
received the same same error.

I removed the last file on the list before the error and repeated.
Same error (no additional files processed).

There were lots of older logfiles in the logs directory that weren't
being processed due to time limits from my config, so I moved all but
the last 4 months worth of logs to another directory.  Same error.

Next I removed all logfiles except for 2 vhosts and repeated.  Same
error.

I upgraded from 2.5.2 CLI  to 2.5.3 CLI, hoping that would fix the
problem.  Same error.

OK, so something is wrong with my config or dnscache.  I renamed both
of those, leaving the subset of logfiles I whittled down to in the last
step.  I processed the last 5 days of logs with no config at all
(besides my serial), and Summary finally made it all the way through
processing without crashing.

So, I spent the next couple hours rebuilding my config file (I should
have tried putting the huge dnscache file back to see if it failed, but
I forgot).  After I rebuilt all my subreports and cleaned them up a
little, I processed the last 90 days of logs, did about 800,000 dns
lookups to rebuild my dns cache, and all seemed to be well.

Fast forward 24 hours to today.  The 24 hour script kicks and
incremental processing starts.  Failure again, this time with a new
error before the ones I saw yesterday:

Visit.cpp:674: failed assertion
One of our processes exited abnormally!
One of our processes exited abnormally!

OK, quit, restart summary, turn off incremental processing, and run
again.  It just finished a couple minutes ago without errors.  I'm
afraid that it will crash again tonight, but I wanted to write to the
list to get some advice or see if this has come up for anyone else.

My hardware is a dual 2Ghz G5 xServe with 10.3.5 Server, 2GB RAM, 2 x
SoftRaid-mirrored 250GB drives, 1GB ram unused according to top.

After trimming the fat, I have about 2200 logfiles, totaling 785MB of
gzipped apache logs, which cover 4 months (and my reports are just the
last 90 days).  I have 15 subreports configured.

Everything was running fine with this same configuration for several
months, up until yesterday.  Nothing was changed in the config that I
am aware of (and I'm the only admin), other than new logfiles being
added.  However, crashing occurs before these new files are being
parsed. Processing "only" new logfiles works without problems.

I think I've covered most of the standard diagnostics at this point.
Please advise.

Thanks,

Dale
-------------
Go to <http://summary.net/list.html> to update subscription info.