Last night when we had a server lock up owing to a kernel bug, there was an unforeseen consequence discovered today.

     It appears the mail server was bound to this server as the NIS server it was using for authentication.  This caused authentication to fail during this interval.  If you attempted to access the mail server too many times during this period, your IP address may be locked out now.

     If you are presently unable to access the mail server, in order to unblock you, I need to know your device IP address for the device or devices that can not access mail.

     To find this out, FROM the affected device go to our website https://www.eskimo.com/, hover over “Web Apps“, and in the pull-down menu that results, select “Your IP address” from the bottom.  Let me know what that IP address is then I can investigate and unblock you if you are blocked.

     Since the file server that crashed contained both home directories, a lot of the web applications, and MariaDB database, it’s sudden refusal to continue screwed up a lot of things which I’ve spent the entire evening fixing, so those of you who made payments today will see your receipts sometime tomorrow.  I apologize for the delay but I’m just too burned out to continue tonight.

     If there is a silver lining it is that that server and all the virtual machines on it now have the new kernel installed so won’t need to do those on Friday.  Also other machines that I needed to reboot to resolve NFS issues, etc, are also updated, so Friday’s upgrades should go faster than normal and have a relatively low chance of failure since failures almost always involve this particular machine both because it has more cores than the other and thus more parallel processes with potential to trip race conditions.

     Going to have to do an emergency reboot on the machine that hosts the web server, Ubuntu, and the /home directory owing to a CPU soft lockup that is persistent.

     I am planning a kernel upgrade to 6.0.10 on Friday December 2nd, starting at 11:00PM Pacific Standard Time.  I expect to be finished by 11:30 if everything goes smoothly.  If on the other hand Poettering gets us again and systemd does not shut down or start up properly then it may not be fully operational until around 1AM.

     I have been testing 6.0.9 for the last eight days and it has corrected the problem with squashfs so that snap applications like Firefux are working more reliably now though not nearly as well as they did before Ubuntu forced that atrocity (snap) upon us in 22.04.

     6.0.10 came out before 6.0.9 could be installed, thus we will be installing 6.0.10.  In addition to squashfs, it also fixes numerous memory leaks and a few potential crashes.  It also fixes other things we aren’t using such as specific hardware or IPv6 problems.

     This will affect all of Eskimo North’s services, paid and free, including web hosting, virtual private servers, Linux shell servers, e-mail, and our Fediverse services which include https://friendica.eskimo.com/, https://hubzilla.eskimo.com/, https://yacy.eskimo.com/, and https://nextcloud.eskimo.com/.

     Ubuntu pushed out an upgrade of the nfs-kernel-server today.

     SunOS had a good functional reliable NFS system in SunOS 3.0 in 1986 (they had an earlier version in SunOS 2.0 but not so good).  Linux has been working on it for 20+ years and it’s still flaky even though it is central to so many server clusters including ours.

     Just wanted to make people aware of this, if you see any weird file behaviors please let me know.

     I got authentication working again by taking the version 3.0.0-beta meant for Nextcloud 24, editing the xml file that described it to include support for Nextcloud 25, tried it, and it SEEMS to work.

     Nextcloud 25.0.1 came out, but the app user_external used to authenticate Linux/Unix users hasn’t been ported to 25.x yet, thus if you use a system login for access, until this is corrected there is no way to authenticate.

     I have filed a bug report on Github but if history repeats, it will be dutifully ignored, time will tell.

     I am in the process of upgrading Nextcloud 24.0.7 to 25.0.1, this likely will take awhile as it involves major changes in database schemas, etc.

     However, one plus, I discovered part of the reason Nextcloud has been so slow.  I’ve been using memcache and I’ve been using PHP 8.0, however memcache is configured to use aPCU and aPCU module was not enabled in PHP 8.0 so this broke memcache.

     For some reason previous versions did not complain about this configuration issue, but now that this version did, it is corrected.

     I apologize for the interruption to web services today.

     For reasons unknown, because there were no new dependencies that should have pulled the Ubuntu version of Apache2 in, Ubuntu installed their version of Apache2 which does not have all the necessary capabilities our locally compiled version has and broke our web services.

     I had to uninstall theirs and recompile and install ours, a process which took around fifteen minutes.

     All services are fully operational.