OpenSuse.eskimo.com has been upgraded to leap 42.3.  An attempt at an online upgrade failed, it did everything except apparently write the boot block correctly because upon rebooting it could not even start grub.

     So I did a fresh install of 42.3, which is probably just as well as the upgrade from 42.1 to 42.2, while successful, was less than 100% clean.

     I am still installing applications so if there is anything particularly important, please create a ticket using https://www.eskimo.com/ and then the Support drop down menu, select “Tickets”.

     The upgrade of OpenSuse went well right up to the point where a reboot was required and at that point locked up the whole qemu/kvm system, some issue with the address still in use like it did not properly free it up when the virtual host went down.

     Anyway, it required a reboot of iglulik which hosts a number of virtual machines and so took many things out of service for about two minutes around 9pm.

     At this point I am going to do a fresh install of opensuse so it will be out of service for a while.

     The shell server opensuse.eskimo.com is presently in the process of being upgraded from Leap 42.2 to 42.3.  Because of this it may not be fully functional, especially third party software like x2go which may need to be recompiled, or the nx-libs upon which it depends, afterward.

    Everything should be up and running now.  Our web server is now 22% faster making it now faster than 97% of the Internet according to Pingdom, before it was faster than 94%.

     Many other services have been sped up as well but to a lesser degree.  Most of the shell servers are also 22% faster now.

     I hope things will be stable but it will take a couple of weeks to be sure.  When CPU speed is increased, the voltage also needs to be increased.  No chart or other guide as to how much is provided but the rule of thumb is the absolute minimum necessary for stability at a given clock speed since heat is very directly related to voltage.  I ran stress tests for an hour, I’d really like to run them for weeks but can’t do so on in service machines.

I have finished work on the host hardware but have not brought up all the virtual machines yet.  I will post a completion notice here when done.

     I will be taking the server that hosts users home directories and a number of virtual machines down this evening in order to make some adjustments to the bios settings.  This will require multiple adjustments, boots, benchmarking and readjustments in order to achieve the maximum possible performance.  I will start this work around 11pm and it may take several hours to finish.

     The server upgrades were challenging.  The actual upgrades went fine, it’s what newer versions of software broke that was problematic.  For example, nfs-kernel-server and nfs-common, those upgrades broke the NFS server’s ability to talk to eskimo.com which is a SunOS 4.1.4 machine although it worked fine with all, even very old, flavors of Linux.  So I backed out the upgrade for those two packages.

     Then the upgrade also upgraded php from 7.0 to 7.1, but many of our web applications are not yet compatible with 7.1, most notably squirrel mail.  So I had to uninstall 7.1, and add a third party repository to re-install 7.0.

     I believe everything is working but if you find something please generate a ticket, use Support->Ticket’s from https://www.eskimo.com/.  Since I was up until 7:30AM, I will not be available before around 2pm.

     I will be rebooting servers to complete the Ubuntu upgrade by booting into 4.13.x kernel.

     I have upgraded my workstation and one mail server to Ubuntu 17.10 and the upgrades went more smoothly than any prior Ubuntu update I’ve ever done.  My ancient Nvidia graphics card that has always been a hassle to get to work properly required no tweaking at all, drivers worked out of the box, this is a first in five years!

     I am done doing server maintenance for the night.  I kept Ubuntu down longer than expected in order to troubleshoot and fix an issue with the greeter.  I had to uninstall and reinstall a lot of things to get it fixed.  Hopefully did not break something else in the process.