[Mageia-sysadm] planning for sysadmin task

[Romain d'Alverny]

On Sat, Oct 30, 2010 at 10:55, Michael Scherer <misc at zarb.org> wrote:
> Le mardi 26 octobre 2010 à 16:39 +0200, Romain d'Alverny a écrit :
>> At some extent, RPM dependencies would be a useful thing for setting
>> up the application but this mostly happens once (first install) and
>> can be easily hosted within the web application itself (and then
>> handle the error) - WordPress and Drupal do it for instance.
>
> It also prevent the removal of used dependencies.
> This can happen either when we are cleaning the server, or when we
> upgrade the server, or another application.

As a web developer, I don't expect a production server to change,
apart from security updates (which should not remove some package,
unless that's a specific case that is handled offline at first).

If that's for a server upgrade/reinstall, then the webapp should:
 - have this documented (in some preinstall validation script);
 - handle this (warning in case of missing dependencies and going
offlinle if really needed).

> If tomorrow, we discover a huge security hole in php-hugesecurityhole
> rpm, we need to know who use it to assess the security of the
> infrastructure. And without knowing what other packages use the rpm,
> this is gonna be slightly complicated to know if we are affected or not.

Indeed.

However, in the past 6 years both at Mandriva and elsewhere, I do not
remember a situation where this could not be managed without packaging
(we did not package web apps):
 - only used library packages were installed;
 - if an issue was raised for an installed package, web team was
notified and a check on both sides (server and app); some times we had
to code around a missing feature;
 - when it happened that a package was missing (removed, or
undocumented when doing a new fresh install), that meant at most a few
minutes of (managed) service unavailability.

>> So we can discuss this further with other future webteam members but I
>> will seriously not manage a production environment that goes through
>> packaging for app updates.
>
> Well, if creating a package is just a single command ( as would be a
> upgrade to the production server ), I do not think it will be much of a
> problem. The only issue is to find someone skilled enough to create a
> shell script for that and I do not really think that it will be a big
> problem. We have a team of 8 admins and there is several volunteers
> eager to help, it would be quite weird to have no one able to do it in
> time.

It's not about "in time". It's about "instantly". Or in a matter of a
single click; as soon as a feature is in, working, reviewed and
approved by webmasters, it should go online asap (unless there's a
specific milestone constraint).

Yet I believe this is not an obstacle to RPM packaging, unless there
are big changes in the app process/layout.

Provided we speak about the same type of web app, of course.

 - I am ok to use packages to deploy some web apps, like
infrastructure, libraries, slow pace deployment/updates web apps
(still, with care);

 - I am not confortable at all to use a packaging system to deploy web
apps that will get frequent updates (main website and related web
services, blogs, forums, ideas, etc.) or even specific
rollback/offmode procedures. And I would prefer the web app to take
care itself of missing dependencies (stopping the service and
providing a feedback about the fix to be done). Relying on a SCM +
scripts deploy/check mechanism seems a lot more flexible, fast and
platform agnostic to me. But I may be wrong.

If we can have a specific build platform for webapps into packages and
have these easily deploy with pre/post install scripts doing their job
on a test server, then a prod server, why not. But I really see this
as a huge payload at this point for a not so obvious gain.

Or we could have a dual approach with only core components of a webapp
being packaged, and more moving ones being deployed by versioning.
Still...

Not related to packaging, but that's a complement to the discussion,
we may need that a privileged access is available to lead webmasters
(logs read access, shell access to webapps root and config).


>> That does not mean I don't care about security - that means that
>> there's a balance to find and that web developers have to be in charge
>> of their apps security as well. So if that means we need to have
>> separate servers to isolate risks, so be it. If that means we need to
>> go for a different type of hosting, so be it.
>
> Separating server do not really help much, if there is a security
> problem, it will be there wherever you are.

Depends where the exploit comes from: if it comes from the application
or a specific library, separating servers does help; you kill the
compromised instance and reload a fresh new one. If it comes from
system level components, that's another issue.

> We will have work to do to be sure the server is clean after being audited,

I'd be more radical. If a server hosting a web app is compromised, we
audit it, but we reinstall a full clean one and redeploy the
applications.

> the reputation will be affected none the less, and if the server is used for
> spam/attack/whatever, we have to take care of this.

Sure. But things do break. Nothing to be ashamed of. We have to live
with this and balance security requirements with flexibility as well.

Security is not only a server-level issue, it's an app-level issue as
well. Leaving webapps developers with no clue/hand over the
server-level conf/knowledge is not going to help. At least, lead
webmasters are not supposed to be illiterate regarding system
administration.

Romain