[Mageia-sysadm] [sysadmin-reports] Hobbit [727252] valstar.mageia.org:disk CRITICAL (RED)

[nicolas vigier]

On Sat, 15 Sep 2012, Thomas Backlund wrote:

> nicolas vigier skrev 15.9.2012 17:51:
>> On Sat, 15 Sep 2012, root at mageia.org wrote:
>>
>>> red Sat Sep 15 16:26:57 CEST 2012 - Filesystems NOT ok
>>> &red /tmp (100% used) has reached the PANIC level (95%)
>>>
>>> Filesystem         1024-blocks      Used Available Capacity Mounted on
>>> /dev/md0              20152044  10375432   8752928      55% /
>>> /dev/sda1              1004024     92956    860064      10% /boot
>>> /dev/sdb1              1004024     92792    860228      10% /boot2
>>> /dev/mapper/vg0-tmp   32640904  30979996      2868     100% /tmp
>>
>> It seems /tmp was full because of rpmlint temporary files. Maybe because
>> we had a few kernel packages finishing build at the same time ?
>>
>
> Probably yes....
>
> As for pushing all kernels at the ~same time was intentional this time...
>
> I wanted to see how the BS would handle max load now that new
> ecosse is working too (as kernel builds do manage to max out
>  buildnodes due to good support for parallel builds)
>
> But I didn't even think about valstar getting into trouble :/
>
> core kernel is the worst one for rpmlint as all rpms +
> their unpacked contents need ~20+ GB diskspace (every
> kernel-*-debug needs some 2+ GB) ....
>
>
> But otoh I think it was nice side-effect to identify a SPOF
> now (instead of release time) so we can get it fixed...

Yes.

>
> question is... do we somehow need to limit how many rpmlint
> processes is started (depending on cpu load or free disk space
> in /rmp), or should we just hope the extra disk space is enough ?

Maybe youri could check if there is enough space on /tmp before
attempting to run rpmlint on the package.

>
>  Because load on the server was too high, I killed all rpmlint processes,
>> and removed all rpmlint temporary files in /tmp.
>>
>
> So whah happend to the packages being rpmlinted ? did they get uploaded 
> anyway ?

Yes. Actually they were already listed as uploaded on pkgsubmit when I
killed the rpmlint processes. I don't know why there still was rpmlint
process running when the package was already uploaded.