Kernel Upgrades Tonight 11PM-Midnight

     I am planning on doing kernel upgrades tonight starting at 11pm.  If things go well they may be concluded by 11:30, if they go like last time might be 2:30, hopefully not.

     There has been problems with grub configuration being properly updated by the automatic kernel installer lately and when it fails it does so silently so I don’t know it’s failed until I try to boot and it doesn’t.  Most of the time this only affects the virtual machines but last time it affected one of the physical hosts requiring a drive to the co-location facility.

     So, probably facilities will be mostly usable by 11:30PM, there are generally one or two servers that don’t properly bind NIS or mount NFS and it takes me about another half hour to go through and resolve those but there is the potential for a physical host to have grub issues like last time so please don’t plan anything critical during this interval.

The Denial of Service Attack Concluded

     The denial of service attack seems to have concluded.

     It was not of sufficient intensity to cause lost packets but it did cause delays of up to two seconds from time to time which would be noticeable if you were typing something interactively like using an ssh terminal or composing e-mail.

     There was a brief interruption near the end of about two minutes but that was me rebooting the router to apply a software fix.

 

Everything is Back Online

     For some reason one physical server and two virtual servers would not boot after the new kernel was installed.  Something went wrong during the install process and messed up the grub configuration on two of the machines, I was able to repair these with the boot-repair disk from sourceforge.  But a third machine was so scribbled I had to restore it from backups.

     At this point everything is back in service.  I apologize for the longer than anticipated outage.

Server Failed to Boot

     One physical server that hosts among other things the mail spool and server failed to boot.  I have to drive down to the co-lo to reboot it so service is going to take longer than expected to restore.

MariaDB Downgraded to 10.5

     We were forced to downgrade MariaDB back to 10.5 because 10.6.1 deprecated some functionality required by NextCloud.  I have filed a bug report with Nextcloud so they can be aware their code is out of date and hopefully fix it.

DoS Attack

     I’ve got about half a dozen of the major sources blocked and while there is still some packets dribbling through from others they are not enough to cause network or router overload so should not further impact network performance.