I have decommissioned julinix.yellow-snow.net because in the last year only two people have logged into it and each of them only once.  It is basically Ubuntu re-themed anyway so doesn’t really offer anything unique.

     I am in the process of decommissioning the eskimo.com shell server because it is an antique SunOS server that has some compatibility issues with Linux NIS and login system.  Linux natively supports usernames up to 16 characters long and passwords with 16 significant characters and much stronger encryption.  SunOS supports only triple-DES encryption, eight significant characters, and maximum login length of eight characters.  Linux supports very large numbers in UID and GID, but SunOS is limited to 65535 as the highest each.  Once the transition to a new NIS master is completed you will no longer be able to access eskimo.com shell server BUT you will be able to have longer usernames and passwords and you will be able to change your password from any server.  A process that is not possible because of these incompatibilities presently.

     One of our customers account was compromised, in spite of all the facilities in place to prevent this, and used to send spam triggering the addition of our mail server IP to abuseix.  Yahoo and AOL have also blocked it for excessive spam.  I have requested delisting at abuseix and with AOL and Yahoo there isn’t much to do but wait.

     Kernel test is completed, 5.16 did not perform adequately.

     We are trialing a new kernel on our web server.  We were running 5.13.19 however it is past end-of-life upstream.  We attempted 5.15.x but the performance of this kernel was garbage.  However, I’ve tested 5.16.0 on my workstation and it performed much better than 5.15 and at least on par if not slightly better than 5.13, so I’m giving it a try on our web server (the most heavily loaded machine).  If it holds up we will be upgrading all the machines next Friday the 21st between 11PM and Midnight.

     If you notice any difference in the performance please let us know.

     Because 2.4.52 version of Apache seems to have a bug that causes it to occasionally stall, and because earlier versions have a root exploit that makes returning to them unfeasible, I wrote an automatic recovery script to recover the server in the event of a failure.

     What it does is automatically test the web server for a proper response every five minutes.  If it fails to respond, it attempts to restart apache, waits a minute to give it time to spool up, then retests.

     If the second test is good it logs the failure so I can keep track of how often it is failing.  If the second test fails, it turns my speaker volume up to 100%, and I have 100-watts per channel so that’s 200 watts worth of noise and then sends the Star Trek Next Generation Red Alert sound effect to the speakers.  I should hear that pretty much no matter what if I’m here, and also sends an e-mail so if I’m out I will be notified via my tablet or laptop.

     Hopefully this will prevent long outages like that which occurred last night when Apache decided to silently hang.

    I am going to kill our web server for about five minutes a couple of time to test some automatic failure recovery scripts I’ve written to try to prevent a repeat of last night’s outage.

     Our fax machine has bitten the dust.

     I’ve got a new one on order but it won’t be here until around the 28th.

     In the meantime, to get payment info, or other material you don’t feel comfortable sending plain-text, to me securely please either send e-mail from your eskimo.com e-mail address or use the ticket system to send the info.

The mail server took longer than I had expected.  And then after the backup was done it would not start properly.  So took some mucking around to figure out why and fix.  There were some things that used files on /misc which is an NFS mounted partition that were not set to wait for the mount.  So mail wasn’t back up to about 12:30AM.

     I am going to be taking the web server, mail server, and some of the shell servers down tonight at 11pm for between 20-60 minutes (mail server closer to 20, web server closer to 60) to image them (a form of backup) as I’ve made substantial changes to fix various issues to all of them and want to capture those changes so I don’t have to re-apply them in the event restoration from a backup is required.

     This will affect all customer websites for about 20 minutes, it will also affect all eskimo.com websites including friendica, hubzilla, and nextcloud, and mail will be unavailable for about an hour, but incoming mail will not be lost, it will be queued and delivered after the server comes back up.

This affected Eskimo North and customer sites as well as friendica.eskimo.com, hubzilla.eskimo.com, and nextcloud.eskimo.com.

We had a web server issue causing intermittent failure of the web server.  I thought this was load related as the server is quite heavily loaded, but it did it today during a time of very light load.  I was able to chase the problem down to a cronjob being run on a wordpress site automatically configured to redirect to secure mode when the certificate had expired.  The cronjobs were never exiting and ran once a minute and simply piled up until memory was exhausted causing failure of other services.  Normally I get automatic notifications when certificates are about to expire but did not in this instance. I’ve ordered a new certificate and commented out the cron job until it arrives and can be installed.