An in place upgrade to Mint 20 failed, their script for setting up python had inappropriate syntax for the version of python being installed.

     A clean install of Mate 20 also failed.  Mate Desktop did not work, could not fully install because of package dependency issues, on a fresh system.

     Mint has been return to 19.3 which at least functions.

     An upgrade of the Mint shell server from 19.3 to 20.0 is in progress.

     Because this is an upgrade involving an upgrade of the underlying Ubuntu system from 18.04 to 20.04, and thus has a greater chance than normal of going south, I am performing a backup of Mint first.

     This whole process will result in the server being slower than normal while it is ongoing and there will be times when various sub-systems are temporarily dysfunctional.

     If all possible it would be better to use one of the alternate debian based systems, Ubuntu, Debian, MxLinux, Julinux, or Zorin during this period.

     I will make an additional announcement when the upgrade has completed.

     Maintenance on mail, virtual private, and shell servers has been completed.

     Tonight starting at around 11pm I will be taking these machines down for maintenance, zorin, centos8, and manjaro, then vps1, centos7, then mail and Debian. 

     Downtime should be about 1/2 hour except for Debian which usually takes about an hour.

     Tonights’ maintenance is done.  Things did not take as long as I had expected.

     Tonight shortly after midnight I will be rebooting the physical hosts which will reboot everything because recent security updates require a reboot to activate them.  This should take about fifteen minutes.

     Then I will be taking the web server down for about an hour for imaging and then to extend the file system to accommodate a larger database required by a new customer.

     I have the physical hosts up on 5.4.0 now, this is the kernel that shipped with Ubuntu 20.04.  It is not entirely stable on machines which serve as NFS servers which two of these machines do.

     I have built 5.3.0 kernels which were previously stable but can not boot into them because automatic backups are running.  Also I don’t know if MY 5.3 kernel will be stable as Canonical applies some patches that will not be present but the 5.3 kernel is no longer available as 19.10 is past END OF LIFE now.  I will attempt to do so tomorrow evening.

     Sorry I had to do an emergency reboot on servers today.  I discovered what caused this morning’s crash and it was a bug in the kernel I was running that caused it to not recover resources as processes exited and new ones were created, so it would continue to eat memory until it ran out then crash and reboot.

     I don’t know if this present kernel is trouble free but I do know it does not have this particular problem and NFS seems to work correctly in it, which has been the main reason for all the kernel experimentation lately.

      After I installed a newer version of fail2ban on the client mail server, the attackers were no longer successful at overloading the machine as it was able to keep up with changing attacking IP addresses.

      Now they are attacking our incoming mail servers, and although they are increasing the load on these machines they can not elicit login credentials from these machines as they are not setup for authentication.

     I am installing the newer fail2ban on these machines to get the loads down as with the client server.

     I have installed a newer version of fail2ban onto the mail client server.  It seems much better able to keep up with attacking IP’s and has brought the load down to a more normal range.