[Mageia-sysadm] Http server problem

[Michael scherer]

Hi,
as some of you may have seen, the http server was down this
morning on mageia.org since 00:18 ( CEST ). The server is still hosted on zarb.org,
so I suspect no one could restart it except the admin team.
I did restarted on 09h35.

>From what I have seen, the problem was caused by a http 
process that was not killed on midnight. The restart was caused by 
automated cfengine upgrade, after installing new php ( [3] MDVSA-2010:239 ).

Munin[1] show us a peak in IO on /var, /home and swap that somehow impacted
mysql, postfix and nagios. I suspect the cause was the swap activity due to some
problem in the apache process.

My own analysis is that the apache restart script was unable to cope with the 
fact that one process was not killed. As systemd coder remind us [2], this
is quite hard to do properly on linux with current tools.

I also suspect that php upgrade somehow messed with apache in some wrong
way, causing weird behavior. This is likely unreproductible and undebuggable,
maybe a race condition due to the high activity of the server combined
to the slowness, I do not know. Anyway, I think we are safe until next php
upgrade.


[1] https://www.zarb.org/munin/servers/ryu/index.html
[2] http://0pointer.de/blog/projects/systemd-for-admins-4.html
[3] http://www.mandriva.com/en/security/advisories?name=MDVSA-2010:239

-- 
Michael Scherer