Because Ubuntu just locked up and because this time I got an indication it was an issue with the hypervisor and physical host, I am going to be upgrading the following machine kernels early this evening around 11:30PM:

     Iglulik (has NFS /home and /misc directories), ftp, debian, ubuntu, and mint.  These will all be down briefly, the web server longer than the others.

     Web services including all our virtual hosts, https://friendica.eskimo.com/, https://hubzilla.eskimo.com/, https://nextcloud.eskimo.com/, and all webmail services as well as our own website, https://www.eskikmo.com/ will be affected.

     We’ll be doing kernel upgrades of all machines this coming Friday starting at 11pm.  Interruption to most services should be relatively brief, however, web based services may take approximately 1/2 hour to restore.  This is because I’ll be reverting the physical host to a 5.13.19 kernel (presently the only host on 5.14) as 5.14 did not fix the ethernet issue AND has proven to be less than stable.  Because a different driver for the flash card is required for 5.14 to 5.13 I can not use the dkms mechanism to compile this driver and because to compile it fresh the machine needs to be running the kernel upon which the driver is running, I will need to first reboot the physical host to 5.13, (it will be the 5.13.19 EOL kernel) and then recompile the drivers, then bring up the web server.  From previous experience this process requires around 1/2 hour.

     Web service includes all our virtual hosts, https://friendica.eskimo.com/, https://hubzilla.eskimo.com/, https://nextcloud.eskimo.com/, and all webmail services as well as our own website, https://www.eskikmo.com/.

     I plan on doing some kernel upgrades over the evening but probably not the physical hosts or the lesser used virtual machines.  There is some instabilities in the existing kernel that is causing CPU lock-ups.  Mostly these are transient and resolved by watchdog timers but one did hang our web server the other day.

     I do not have access to parking at the co-location facility this weekend because they are re-stripping the parking lot so I’ll save updates of physical hosts for next weekend.

     The reboots of the virtual machines only take a few seconds so interruptions will be brief.  I’ve already done the web server so it will not be impacted, mainly shell and mail servers.

     Also I plan to take some of the private virtual servers down for between 15-60 minutes depending upon their size to image them (a form of backup).

     Apache 2.4.49 was released today and had many important security fixes and also some fixes to help it gracefully shutdown which makes it do so much faster.  This will result in a much shorter interruption during log rotates at midnight.

     We have installed 2.4.49.  Pingdom tests reveal essentially unchanged performance loading our WordPress site in 279ms.

     With respect to Eskimo North’s we services, I found that the issue that hung our web server has existed since May of this year so I’m not real optimistic the next kernel upgrade will resolve it, however, I found a kernel variable I set that will cause the system to panic and reboot if it occurs again so at least it won’t result in hours long outages.

     This note covers several topics, web server crash, kernel upgrades, and 56k dial service.

     The web server crashed today shortly after 9AM but nobody called until 1PM and it was in a mode where it was still consuming CPU and appeared up to the hypervisor, but not talking to the network so I remained unaware of it until I received that call.

     This affected all Eskimo North web services including webmail, https://friendica.eskimo.com/, https://hubzilla.eskimo.com/, https://nextcloud.eskimo.com/ and virtual domain hosting.

     Also, a small aside, unlike our previous attempts at social media, friendica has been largely successful and has acquired an active user base so far of 566 users and is growing at a daily rate of 1-2%.  It is part of a federated network of almost four million users and sports some fairly interesting discussions.

     This is the second time it has crashed since the last kernel upgrade but the first time it just locked without logging anything, this time I did get good logging of what happened.  I will be doing an upgrade on the kernel of this machine tonight in hopes of addressing this but I am not 100% sure if the issue is with the virtual host.

    So the next major kernel upgrade affecting all servers will be on September 24th between 11PM and 12AM.  I will be reverting Iglulik, the machine that hosts ftp, www, ubuntu, debian, and mint to the latest 5.13 kernel since the 5.14 appears flawed and did not fix the Intel E1000 chipset issues they said it would.or the physical host.  Tonight’s upgrade will only be of the virtual host.

     The physical host has not shown any errors however this kernel had been stable on the virtual machine until I upgraded the kernel on the physical machine.  Confusing I know.  But the bottom line is this, it is safe for me to boot a virtual machine remotely because I can control it remotely even in a crashed state, a physical machine I can’t, so the latter will need to wait until Friday the 24th of September (they are re-striping the parking lot this weekend so no parking at the co-lo when I  usually do these).

     So the next major kernel upgrade affecting all servers will be on September 24th between 11PM and 12AM.  I will be reverting Iglulik, the machine that hosts ftp, www, ubuntu, debian, and mint to the latest 5.13 kernel since the 5.14 appears flawed and did not fix the Intel E1000 chipset issues they said it would.

     Then the last note is I do not anticipate 56k dial service continuing after the end of 2023, this per my one remaining wholesale provider that only has large      So the next major kernel upgrade affecting all servers will be on September 24th between 11PM and 12AM.  I will be reverting Issumataq, the machine that hosts ftp, www, ubuntu, debian, and mint to the latest 5.13 kernel since the 5.14 appears flawed and did not fix the Intel E1000 chipset issues they said it would.contracts lasting through that date.  This could change if  one of their larger customers renews their contract but at present this is the plan.

     Eskimo.net was unavailable to the Internet part of Thursday and Friday (Sept 9th-10th) owing to an error I made on the master DNS server.  It didn’t show up right away until the cache expired on the secondaries.  It is corrected now.

     Web server crashed at 22:10 PM Pacific Time today (Sept 9th).  Fortunately I tried to access it right at that time so it was back up after 4 minutes.  Unfortunately, it did not provide any information in the logs that gives a clue as the the cause.  It just hard hung and would not respond until forcibly booted.

     The Usenet News service I have been using to provide news for our shell servers seems to have gone TU without warning.  They sent me no notice but just disappeared from the Web.  They were a US based firm but the domain name is now owned by an outfit in England.  There is no website, phone numbers disconnected, no e-mail addresses working, no way to find out what happened.

     I am looking for a new provider but in the meantime I apologize but no Usenet News available until and unless I can find a replacement.

     If anyone knows of a potential replacement please contact support@eskimo.com or via the Fediverse nanook@friendica.eskimo.com.

     Unfortunately the 5.14.0 kernel did not contain the new E1000 driver advertised in advance, still the broken 2012 vintage broken driver.  So I’m cancelling the kernel upgrade I had planned for this Friday since nothing is to be gained from it.  I do see there is a new 2021 driver on Intel’s website, so I guess I’ll need to figure out how to integrate that into the kernel.