Going to have to do an emergency reboot on the machine that hosts the web server, Ubuntu, and the /home directory owing to a CPU soft lockup that is persistent.

     I am planning a kernel upgrade to 6.0.10 on Friday December 2nd, starting at 11:00PM Pacific Standard Time.  I expect to be finished by 11:30 if everything goes smoothly.  If on the other hand Poettering gets us again and systemd does not shut down or start up properly then it may not be fully operational until around 1AM.

     I have been testing 6.0.9 for the last eight days and it has corrected the problem with squashfs so that snap applications like Firefux are working more reliably now though not nearly as well as they did before Ubuntu forced that atrocity (snap) upon us in 22.04.

     6.0.10 came out before 6.0.9 could be installed, thus we will be installing 6.0.10.  In addition to squashfs, it also fixes numerous memory leaks and a few potential crashes.  It also fixes other things we aren’t using such as specific hardware or IPv6 problems.

     This will affect all of Eskimo North’s services, paid and free, including web hosting, virtual private servers, Linux shell servers, e-mail, and our Fediverse services which include https://friendica.eskimo.com/, https://hubzilla.eskimo.com/, https://yacy.eskimo.com/, and https://nextcloud.eskimo.com/.

     Ubuntu pushed out an upgrade of the nfs-kernel-server today.

     SunOS had a good functional reliable NFS system in SunOS 3.0 in 1986 (they had an earlier version in SunOS 2.0 but not so good).  Linux has been working on it for 20+ years and it’s still flaky even though it is central to so many server clusters including ours.

     Just wanted to make people aware of this, if you see any weird file behaviors please let me know.

     I got authentication working again by taking the version 3.0.0-beta meant for Nextcloud 24, editing the xml file that described it to include support for Nextcloud 25, tried it, and it SEEMS to work.

     Nextcloud 25.0.1 came out, but the app user_external used to authenticate Linux/Unix users hasn’t been ported to 25.x yet, thus if you use a system login for access, until this is corrected there is no way to authenticate.

     I have filed a bug report on Github but if history repeats, it will be dutifully ignored, time will tell.

     I am in the process of upgrading Nextcloud 24.0.7 to 25.0.1, this likely will take awhile as it involves major changes in database schemas, etc.

     However, one plus, I discovered part of the reason Nextcloud has been so slow.  I’ve been using memcache and I’ve been using PHP 8.0, however memcache is configured to use aPCU and aPCU module was not enabled in PHP 8.0 so this broke memcache.

     For some reason previous versions did not complain about this configuration issue, but now that this version did, it is corrected.

     I apologize for the interruption to web services today.

     For reasons unknown, because there were no new dependencies that should have pulled the Ubuntu version of Apache2 in, Ubuntu installed their version of Apache2 which does not have all the necessary capabilities our locally compiled version has and broke our web services.

     I had to uninstall theirs and recompile and install ours, a process which took around fifteen minutes.

     All services are fully operational.

     At 12:54 kernel upgrades are completed.  Sorry this took so long but one physical host failed to boot properly.  The Nvidia graphics card did not initialize properly and systemd brought the machine up into single user mode but not multi-user mode so I could not access from here.  I had to drive to the Co-Location facility which is 22 miles away and there was construction work at the I-5 / I-90 interface that made that take longer than it should have.

     I am still checking NIS/NFS mounts, but all the basic subsystems are up and running.

     I’ve tested 6.07 and 6.08 and both seemed to have resolved the issue with squashfs, so will be doing kernel upgrades.  Provided they haven’t introduced new bugs, this should eliminate all bugs of any operational consequence.  There still is an issue with startup of centos7 and scientific7 but that only generates an error message and is of no operational consequence and, according to developers, that bug will be addressed in 6.2, so still a ways off.

     This will affect off of Eskimo North’s services including our public services https://friendica.eskimo.com/, https://hubzilla.eskimo.com/, https://nextcloud.eskimo.com/, and https://yacy.eskimo.com/ as well as our own website, all the sites we host and virtual private servers.

     Downtime for any one service should not exceed about ten minutes except for yacy which takes around 40 minutes to rebuild it’s database after a reboot.  This operation should be completed by midnight.