No Outage (not at 0650Z anyway!)

[Glen <glen at amsl.com>]

All -

Good morning.

At 06:55 GMT - five minutes past the time of our two outages this past 
week - the server is running quite smoothly and unencumbered, and has 
not (at least so far) gone down or shown any signs of doing so.

Progress was definitely made today.  Things we did include such elements 
as:  (Suppressing the desire to work "Fear" and "Surprise" into this 
message)...

* Did a bunch of digging through the kernel.  Found some strangeness in 
the way memory was being handled generally, found a possible way of 
adding some sanity to the oom-killer process, re-tuned the kernel to 
implement and fix that.

* In concert with Henrik, installed a modified version of the TMDA 
handler to fine-tune how TMDA is dispatched, thus reducing significantly 
the added load from TMDA (with great success, thank you Henrik!)

* Installed a logging system to capture snapshots of all kind of server 
data and log it; in the event that the server -does- go down this will 
hopefully give us something more to look at.

* Found a way to exempt certain processes from the oom-killer, applied 
those exemptions to the syslog and cron and sshd processes.  I -hope- I 
never get to test this, but I applied it nonetheless.

And of course,

* Sat up with the machine until now, to watch firsthand what is happening.

I did see a lot of python activity, and a lot of datatracker activity, 
and that activity increased dramatically after 06:30Z.  TMDA activity 
also increased dramatically; the new handler performed very well and our 
overall load remained low.  The usual other things all ran correctly and 
as they should, and the server performed perfectly, averaging less than 
2% cpu, peaking at 17% at one point, with memory staying well below the 
50% mark throughout the night.

So did these changes solve the problem?  The re-tuning and the TMDA 
stuff -may- have.  I won't stay up every night, but I *will* stay up 
Wednesday-into-Thursday again, since that's when the last crash 
occurred, just to see what happens.... and we'll go from there.

I also found a way to have the oom-killer reboot the server if it gets 
into that much trouble.  I haven't turned that on yet; however, when 
this little episode is all done, I will turn it on, so that any failures 
of this type in the future are hopefully limited to a few minutes of 
automatic recovery, rather than outages like this.

Have a good night/day as appropriate, and, if nothing happens between 
now and then, I'll post another email Thursday.

Glen Barney