Sorry about the outage tonight between about 3am and 4:30am.  I screwed my workstation up to the point where it became necessary to restore from backups, but that did not work.  Even after re-installing grub and update-grub2, etc, it’s trying to boot a kernel that doesn’t exist, so I’m using an old version of Linux I have on a hard drive in case the flash fails but I’m going to have to re-install.

     Anyway, while connected to our main file server via x2go I forgot I was connected remotely and went to shutdown the machine thinking it was my workstation I was shutting down, then I discovered it was the server.

     So I had to drive to the co-lo and power the server back up.  When I went to leave my car would not start.  Something is screwed with the theft prevention system and it does not recognize either key fob and would not let me start.  So had to spend $332 to tow it to the garage, get my wife out of work early to come get me, and I’ve been up all night, and my workstation is still broken.

     So I will not be available during the day to day.  Sorry but I need to get some sleep.

     So in the last week, two of our cars broke down, one that was broken magically fixed itself, I think it was the fuel pump relay but it started working again so we’re driving it now, and I broke two of my teeth which won’t be fixed until mid to late July because the dentist is so backed up, and I broke my workstation.  Great, so trouble coming not in threes but at least fives or sixes.

     If anything is still down, please generate a ticket via the ticket system (https://www.eskimo.com -> Support -> Tickets) and I will attend to it when I am among the living again.

     I did not get the kernel upgrade done last night, between the fact that I broke a tooth and so had to go to bed early so I can get to the dentist today and the fact that 5.18.0 had some compiler errors, I’ve got patches from the developers that fix them but haven’t had time to make and install the new kernels yet, the kernel upgrade didn’t happen.  So far I haven’t had anyone express connection difficulties, but if you experience them in the meantime, mosh is a possible work-around.  Else I’ll at least get the guest machines done tonight or tomorrow.  Host servers (physical machines) will have to wait until I know I’ve got a working car.

     With respect to the new router, I have it here, still in the process of figuring out how to configure it.  The interface is quite different than the old machine owing to vastly expanded capabilities.  This is really a full fledged multi-media box not just a router although in our case it will be used just as a router.  It is also a video recorded, network controller, etc.

     This affects all eskimo.com services:

     I am planning an emergency kernel upgrade of at least the most used shell servers, the mail server, and the web server tonight.  May not be able to get to everything as I can not stay up late as I broke a tooth last night and need to get up to see a dentist tomorrow.

     This upgrade mostly involves the fallback of MCTCP to TCP.  MCTCP is a multi-path TCP protocol that allows a connection to be maintained even when the end point IP addresses change.  These are mostly used with mobile phones but any device that does not talk this protocol needs to fall back to TCP, these would mostly be older devices without wireless capability.

     The existing kernel has a bug in this fall-back code.  Because some of my customers have antique computers that may be affected, I want to try to get the publicly facing computers upgraded if possible tonight but I can’t stay up late to do it.  I’m going to focus on the physical servers in case they fail to reboot.  I will not have a car available after tonight until sometime next week as my wife’s car needs to go into the shop and she will need mine to get to work.

     I also have the new router but have not had time to install the drive and get it configured.  This should happen in the next week or so.

     Now that duckduckgo.com and swisscows.com/ch are censoring, as bing.com and google.com have always done, and since what else is out there, save for yandex.com which censors but on different content, there is nothing else out there that does not censor except for yacy.com.  Searx DOES only proxy requests from the others above, it provides privacy but does not provide uncensored results.

     So to that end, I’ve installed a federated Yacy server and made it publicly available.

     You can reach it at https://yacy.eskimo.com/ or from https://www.eskimo.com/ (web apps) menu.

     It’s just started indexing so the local database isn’t very complete yet but it will return results from ALL of the federated Yacy peers so you will see good results even now.  They will only get better with time.

     However, use with caution, remember this is NOT a censored search engine so no doubt you will find things in the Not Safe For Work and NOT Family Friendly category as well as ALL political views, even those that offend you.

     I finally received the info I needed from Ubiquiti and we have a new router ordered.  It should be here in 7-10 days and will provide 6.8x the CPU power of the existing router.

     Our existing router is sometimes saturating between 6pm-9pm Pacific Daylight Time, even though the bandwidth is not anywhere near maxed out.  This causes latency and dropped packets.

     The existing router handles high throughput if TCP traffic if packets are large, but it does not handle high UDP traffic or small TCP packets.  The new router should improve this substantially.

     I also bought an optional 1TB hard drive for the router (it has 128GB of flash internally) which will allow me to install a nice desktop and more security software.

     For anyone looking for a router, I really love Ubiquiti products because of their excellent firewall capabilities and interface and the fact that they are Debian based so you can basically install anything from Debian as long as it is present in the MIPS ports, and if it isn’t you can self-compile as it also includes the normal Debian port of GCC and libs.

     Kernel upgrades and reboots are completed.

     All services are active.  All NFS mount points working.  All NIS binding successful.

     Only server where there was a problem was one customers private virtual server and that because he had configured SSL but the SSL certificates were not present.

     Not really sure what changed, but all the machines booted significantly faster this time around.  Not sure if this kernel boots better or if some changes Ubuntu made improved efficiency.

     This impacts all eskimo.com services including shared web hosting, virtual private servers, e-mail, shell access, https://friendica.eskimo.com/, https://hubzilla.eskimo.com/, and https://nextcloud.eskimo.com/ federated social media services.

     In about 1-1/2 hours I will be going to the co-location facility to extract a broken server.  This should not be service impacting except that it is in a physical stack that will require disturbing other machines and whenever that happens there is always to potential for a power cord to fall out or something similar.

     CORRECTION:

     Kernel Updates will be Friday May 20th, NOT tonight May 19th.

We will be doing kernel updates on all systems Friday May 20th starting at 11pm Pacific Daylight Time (-0700 GMT).  I expect interruptions of individual services not to be more than about ten minutes and it should be completed by 12PM (midnight).

This will impact all of Eskimo North’s services including:

     https://www.eskimo.com/

     https://nextcloud.eskimo.com/

     https://friendica.eskimo.com/

     https://hubzilla.eskimo.com/

This kernel upgrade is a point release that fixes some bugs.  It does not change capabilities or interfaces so should have no impact except to prevent crashes and processes hanging.  We haven’t experienced either of these with the current kernel but others have.

     Someone here, and sorry my memory does not remember who, contacted me a while back with respect to Nicholas Heer, if you could please contact me again I would very much appreciate it.