I successfully got the iomemory drivers working under 5.14.0 on the server where it was important (the one with the flash card and Intel ethernet).  I’ve turned on hardware offloading and so far it has run cleanly.  Assuming it is still clean tomorrow I plan on upgrading the remaining servers to the 5.14.0 kernel tomorrow.  This will involves some downtime for all services between 11pm – Midnight.

     It was a bit of a battle, for some reason the automatic DKMS installation and update-initramfs failed so I had to do everything by hand, but by hand it all worked.  I can’t find any explanation for the failure codes and it did not fail when I did the same steps the makefile did manually.  Gotta love stuff like this, kind of beginning to feel a lot like Windows.

     This will affect ALL services including https://friendica.eskimo.com/, https://hubzilla.eskimo.com/, and https://nextcloud.eskimo.com/.  E-mail coming in won’t be lost but will be delayed during this time.  Outgoing will be refused connection during this time.

     I have determined what I need to do to get the iomemory drivers (for flash card on main server with NFS home directories, web server, and flash drive) with the 5.14.0 kernel.  So I am going to be taking things down in about 15 minutes, between about 12:15 AM and 1:15 AM or so September 2nd.  Everything will be unavailable for a short while during reboots.

     The server maintenance is completed for the night.  The new drivers would not compile against 5.14.0 so we are back to 5.13.12 for now.

      I am going to be booting one of the physical servers (Iglulik) into a new 5.14 kernel tonight and this will take the web server, Mint, Debian, and Ubuntu shell servers down for a short interval, approximately ten minutes, EXCEPT the web server.

     The web server will be down longer because the upgrade to 5.14 requires rebuilding the iomemory drivers for the flash drive used for our MariaDB database.

     This will mean also that https://friendica.eskimo.com/, https://hubzilla.eskimo.com/, and https://nextcloud.eskimo.com/.

     Other web based services such as Squirrelmail will also be unavailable during this time frame.  Home directories for all shell servers will be temporarily unavailable during the reboot but should be available within about five minutes.  You can access e-mail via the shell servers or via IMAP/SMTP while work on the web server is under way.

     5.14 has a new copyless driver for the Intel E1000 Ethernet chipset used on this server and it is my hope that this driver will fix the hardware offloading issues with the existing driver that force us to disable hardware offloading.

     I am receiving complaints from people unable to connect to servers that they used to be able to using putty on versions of Windows prior to Windows 10.

     The reason you are unable to connect is that as exploits are found in various encryption schemes, those schemes are retired in favor of newer more secure encryption algorithms.

     All versions of Windows prior to Windows 10, with the exception of Windows 8.1, are past end of life.  If you are using one of these, you will not be receiving updates that update security issues like encryption vulnerabilities and thus you will not be able to connect with your old encryption methods.

     Putty is an aftermarket ssh client for older version of Windows.  If you have Windows 10, you do not need putty as it comes with openssh, you just need to install it as one of Windows features (it is not installed by default).

     If you are using 8.1, you should upgrade your putty installation to the July 21, 2021 release.  If you are using an older version of Windows you can try this as well but older versions may not have the necessary libraries for it to work properly.

     System kernel upgrades are completed.  All services are back online.

     Debian came up first time properly so I ended up not doing any systemd debugging today.  Whether this was a fluke or they fixed something with upgrades since last kernel upgrade I do not know.  If it messed up again I will debug at that time.

      We will be upgrading all servers to 5.11.12 on Friday at 11pm.  This will take about an hour.

     I will also be adding some debugging code in Debian to try to gather more information with respect to problems with the systemd configuration so it may be down for a longer period while I go through a few tweak / reboot / examine debug log cycles.

     I have received notice that there is a failure of some HVAC equipment at the Isomedia facility where our equipment is located.

     Unlike Integra, they have people monitoring these things and have already dispatched personal to repair.

     I have temperature monitors in the servers and so far they haven’t shown any increase in temperature.  Not too worried about the servers since I have a very large amount of fan in each cabinet except for the old UltraSparc machines used for dial-up radius authentication still.  They will tolerate 104F max, right now it is 82F so still a good margin of safety.

     Debian Bullseye is UP.  The systemd configuration still is not correct so it is very slow to boot but it does eventually start everything just not in the correct order and only after various timeouts.  I will figure out systemd eventually.  I keep hoping Poettering will give up and go away but he seems determined to ruin Linux.

     After upgrade to bulleys, debian will not boot with NIS enabled.  NIS (Network Information Services) is required in order for you to be able to login.  This seems to be yet another Poettering systemd induced snafu.  I am looking for a solution.