All of the Ubuntu based physical hosts are now upgraded to Jammy 22.04.

     As far as I can tell, everything seems to be up and running.  Did need to fix some things with apparmor breaking my name server which has some configurations they didn’t expect, and various broken NFS mount points, and random services and servers that didn’t start, but I believer everything is good now save for radius running in simplex at the moment because one server died hard (won’t even post).  Working on a fix for that.

     Kernel upgrades are completed.  All NFS and NIS mounts checked and all machines checked to be sure all services are running.

     A Distribution upgrade of the physical hosts from Ubuntu 20.04 to 22.04 is in progress.  In several hours this will also require rebooting everything again and fixing some NFS configuration files which I have found this upgrade breaks.

     This will affect all ‘eskimo.com’ services since these are the physical machines that everything runs on.

     I ordinarily do kernel upgrades on Friday so I have a weekend to recover if something goes wrong before the heavier business usage during the week, however, because my car is in the shop I am going to do them this Sunday evening starting at 11pm.  That day because my wife has it off so if something locks up and I need to go to the co-location facility I can use her car.

     If the kernel upgrade goes well, I may attempt to upgrade some of the physical servers to 22.04 now that I have the NFS issues with 22.04 resolved.

     This will affect all of Eskimo’s Services, various services will be down for as long as 10 minutes but not all at the same time.

     This will affect all sites hosted here including private virtual servers, shared web hosting, Linux shell services, e-mail, and our social media sites, https://friendica.eskimo.com/, https://hubzilla.eskimo.com/, and https://nextcloud.eskimo.com/, as well as our own site https://www.eskimo.com/ but the interruptions should in general be ten minutes or less.

     For anyone maintaining a hetrogeneous network utilizing Linux NFS servers, when I upgraded from 20.04 to 22.04 on our mail server which exports the mail spool to various machines, most using version 4.2 over TCP NFS but some using version 3 UDP and one using version 2 UDP, it broke all of our machines mounting using version 2 or version 3 NFS even though rpcinfo -p showed these protocols exported.

     It turns out that what changed is capabilities and defaults.  All three NFS versions are supported, but all now DEFAULT to TCP only.  This makes ZERO sense as UDP was the only protocol available in the NFS version 2 and version 3 specification.

     However, it is configurable and now everything is back in service and I can move forward with updating the rest of our machines that are 20.04 to 22.04, however I am going to hold off on that until I get my car back from the shop as that such upgrades often result in disasters leaving a machine unbootable and requiring in person attention.

     I wrote zero since when I meant zero sense.

     Not sure why but I have a real problem with homonyms, since and sense, to, too, and two, etc.  I know the correct usage but when I write, my stream of consciousness is in a verbal form and the translation to to text more or less automatic and not taking homonyms into account, so unless I actually proof read, which I do not do often enough, this results.

     I will be doing kernel upgrades over the next week or so but not all at once as per usual.  I will be doing the virtual machines which are most exposed to the external Internet first, and holding off on the physical servers until my car returns from the shop (it ate it’s alternate, an unfortunately not infrequent occurrence for this model.

     The virtual machines I can always reboot remotely but the physical machines I need to be there in person if they lock up during a boot.

     Our router at the co-lo facility is running short of CPU during peak traffic times in the evenings resulting in greater latency and some packet loss.

     I am currently researching a replacement which will have approximately 6x as much CPU to accommodate this traffic and future growth.

     The current router is an Edgerouter Lite which has two 500 Mhz MIPS CPUs, the replacement I’m looking at is a Ubiquiti Dream Machine, which has four 1.7 Ghz cores.  The latter is designed for an enterprise environment.

     The Dream Machine comes in a lot of different flavors and I’m not familiar with all the terminology used in the specifications so I’m waiting on Ubiquiti support to answer a few questions before placing an order.

     All of our mail servers have now been upgraded to 22.04, and one of the things I was unaware of before upgrading to 22.04 is that postfix, the mail transport agent that we use, has changed a bunch of defaults.  This may cause some problems and I am reviewing all the settings to make sure that where the defaults have changed, we explicitly declare the settings we want rather than relying on the now incorrect defaults.

     I’ve also loosened somewhat the fail2ban restrictions on the incoming servers.  Since the primary function of fail2ban is to stop brute force password guessing attacks, and people do not authenticate to the incoming servers, it is not necessary that they be as strict.  We had some instances where outlook servers were banned for repeatedly trying to deliver mail to non-existent addresses.  This happens when spammers use very unclean old address lists.

     A distribution upgrade failed on mail.eskimo.com leaving the machine in an unbootable state where grub failed to install properly.

     An attempt to install grub2 manually results in can not write EFI variables to NvRAM.  I’ve not encountered this on a virtual machine before and not being successful at finding info on how to fix.  The easiest thing may be to just rebuild the virtual machine from scratch and then restore all the files.  I have a few other things I’m going to try first but before I do I’m making a current backup of the mail spool to make sure we do not lose any mail.  This may take several hours.

     An upgrade of the mail server mail.eskimo.com is in progress.

     There may be temporary periods where the server does not respond during this process.  It should be brief.