Maintenance Work Tonight

     Tonight, Wednesday Night to Thursday Morning, the 24th and 25th:

     Going to be returning one server to service, shuffling services around and clearing another server so I can do an operating system reload on it.  Unfortunately the Zesty update has failed more often than succeeded and in cases where it seems to have succeeded I’ve had networking problems afterwards.  So rather than risk it I’m going to clear one server of all services, reload it, then redistribute all the services to balance loads.  This will result in outages of various services lasting from 15-45 minutes each.

Mail & Shells

     There will be some slowness in the mail system and shell servers today as I am making backups of all the virtual machines in preparation for shuffling them around when I return another server to service.

Igloo

      Machines on Igloo include:

  • Debian
  • FTP (which is also the web server)
  • Mx2 (one of two incoming mail servers)
  • Radius1 (a machine used for Radius authentication, not yet in service)
  • Scientific7
  • vps3 (a customer virtual machine)
  • vps4 (another customer virtual machine)

     The crash didn’t break these but while I was at the co-lo I decided to bring Igloo’s host operating system up to snuff.  The latest Ubuntu fixed a number of systemd scripting problems that formerly made remote booting impossible (the scripts would hang at target shutdown reached).  I also suspect that it fixes a number of hacks that they have for obvious reasons not publicized.  So I brought it up from 16.04 to 16.10, the first step, and all the virtual machines broke.

     The reason why they broke is that Canonical decided to change the XML machine names and not bother to tell anyone.  I discovered this by deleting and re-installing one machine and comparing the resultant XML file.  Once I know that it was a simple edit to fix the others.

     Now I am doing the second phase of the upgrade which brings 16.04.2LTS up to 17.04.  When it finishes it will require another reboot and who knows what else they might change that it will break.

Igloo Crash

     Igloo, which is a physical host machine that hosts a number of virtual machines including the web server as well as home directories, crashed.  This is the first time it has ever crashed since it has been an i7-6700k based system (more than a year).  Seems to be a pattern lately.  At any rate, there was some corruption to MySQL but I believe I’ve got that straightened out and I am upgrading the server as long as I am down here which will require two more reboots before it is finished.

Mint Maintenance

     Mint is going down for maintenance for about 40 minutes.  It is being moved to new hardware.  All other shell servers will remain available while Mint is moved.  The closest alternatives in terms of environment are Ubuntu and Debian.

Ubuntu

     I will be taking the Ubuntu Shell server down for about 1/2 hour to move it to new hardware.  This will result in better performance.  I expect long term reliability will be improved as well as I have found the i7-6850 with identical cooler operates at substantially lower temperatures.  The i7-6700k when heavily taxed would get upwards of around 190F and that with an EVO 212 cooler with dual fans which is capable of dissipating around 230 watts, but the i7-6850 with the same cooler (but different motherboard owing to different socket requirements) is running usually at less than 100F and the highest temperature I have seen thus far is 109F. Heat is a big enemy of CPUs so I expect this will make for better reliability in the long term.