[Mageia-sysadm] [sysadmin-reports] Hobbit [727252] valstar.mageia.org:disk CRITICAL (RED)

[Thomas Backlund]

nicolas vigier skrev 15.9.2012 17:51:
> On Sat, 15 Sep 2012, root at mageia.org wrote:
>
>> red Sat Sep 15 16:26:57 CEST 2012 - Filesystems NOT ok
>> &red /tmp (100% used) has reached the PANIC level (95%)
>>
>> Filesystem         1024-blocks      Used Available Capacity Mounted on
>> /dev/md0              20152044  10375432   8752928      55% /
>> /dev/sda1              1004024     92956    860064      10% /boot
>> /dev/sdb1              1004024     92792    860228      10% /boot2
>> /dev/mapper/vg0-tmp   32640904  30979996      2868     100% /tmp
>
> It seems /tmp was full because of rpmlint temporary files. Maybe because
> we had a few kernel packages finishing build at the same time ?
>

Probably yes....

As for pushing all kernels at the ~same time was intentional this time...

I wanted to see how the BS would handle max load now that new
ecosse is working too (as kernel builds do manage to max out
  buildnodes due to good support for parallel builds)

But I didn't even think about valstar getting into trouble :/

core kernel is the worst one for rpmlint as all rpms +
their unpacked contents need ~20+ GB diskspace (every
kernel-*-debug needs some 2+ GB) ....


But otoh I think it was nice side-effect to identify a SPOF
now (instead of release time) so we can get it fixed...

question is... do we somehow need to limit how many rpmlint
processes is started (depending on cpu load or free disk space
in /rmp), or should we just hope the extra disk space is enough ?

  Because load on the server was too high, I killed all rpmlint processes,
> and removed all rpmlint temporary files in /tmp.
>

So whah happend to the packages being rpmlinted ? did they get uploaded 
anyway ?

> I also added 30GB to /tmp.

That will help not hitting this issue again...

--
Thomas