[Mageia-dev] Regression testing for generated RPM

[Jochen Breuer]

Hi everyone,

I've already posted this to the Mageia forum, but doktor5000 suggested to
also post this to the mailing list.

I'd like to ask if it would make sense for Mageia to automatically test
generated RPM packages. The idea isn't new. Ubuntu is using regression
tests with python-unit for a lot of packages in their distribution:
https://wiki.ubuntu.com/MeetingLogs/devweek0909/RegressionTests

I think that using automated regession tests would make the job of QA a lot
easier. Failures within packages would surface much faster and a lot of
manual checking could be avoided. Don't get me wrong. Every package that
would pass the regression tests should still be checked by a human. ;)

The following scenario could be used for this. After the package hast been
generated, the RPM (or job) is passed to the regession test server. This
server spawns a new virtual machine with the Mageia version the package
should be checked on. When the machine is booted, the package is installed
with all the needed dependencies and the tests are executed. If everyting
went smooth, the package might be checked again by QA manually. If one or
more tests failed, there is something wrong with package, although the
package generation worked, and the maintainer/developer might want to take
a look at it again.

To setup this up a combination of hudson (hudson-ci.org) and vagrant (
vagrantup.com) could be used. vagrant allows to use VirtualBox to boot
serveral VM instances from base images that are thrown away after
everything is done. This way multiple versions of Mageia could be tested on
one well equipped server. One drawback with VirtualBox would be the missing
ARM support.

Let's use something like a JSON lib for Python as an example. The package
generation was successful. But Python is a highly dynamic language, so some
submodule is missing due to a wrong install path, although "import json-xy"
of the main module works flawlessly. Without further testing this will only
surface when a poor developer uses the submodule or when a second package
that depends on that submodule is tested by QA and shows errors or
misbehaviours.
Second example, same lib. The JSON lib usually ships with a handy
executable to lint check JSON files, but upstream decided to change that.
Due to an error in the upstream repo, the executable script file is still
there but it's empty or there is gibberish in it. There is no more lint
checking possible with this executable. Automatic testing would show this
very fast and without someone sitting there, pasting some JSON into a file
to feed the executable with it.

Testing UI would be an other story, but I guess you get the idea. ;-)

That do you think about that? I think it would enhance the quality on
Mageia in the long run and take some work from QA to find the really tricky
bugs.

Regards,

Jochen
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/mageia-dev/attachments/20130104/182edc02/attachment.html>