I apologize for the disruption today.  Our gateway router crapped out entirely as in not crashed but dead as a door nail.  I had already purchased a new unit with a lot more CPU and memory to replace it but owing to my unfamiliarity it took some time to get it configured and operational.  But this unit is hands down way more powerful than the old so we shouldn’t see the lag during heavy traffic or be nearly as easily packet flooded as with the old unit.

     Going to reboot all or most machines at 11pm to upgrade kernels.

     This will affect all services, paid and unpaid.  With the exception of yacy which will take about 45 minutes, other services should not be down longer ten minutes.

     I had a problem with Iglulik firewall because quotad would not honor my configuration and listen to the configured port so that forced me to leave all ports open to other machines here.  Not a desirable situation for security.

     I filed a bug report and amazingly enough the Ubuntu folks (Canonical) got back to me with a fix.  Seems the arguments need quotes around them.  Because this file lacked official documentation, I did not know this.

     So need to reboot to make this change effective.  Most of the time this is non-eventful, but owing to the complexity of this particular machine systemd screws up sometimes and then a drive to the co-lo is required.  So outage is expected to be from 10pm-10:02pm but may last until about 10:40pm.  This outage will take the form of all things depending upon /home temporarily freezing.

     Ubuntu spontaneously booted around 13:30 today.  The logs indicated an intentional reboot but I didn’t request one, so don’t know if a signal went to the wrong process (systemd) or exactly what happened.

     Then around 6pm the load on inuvik which hosts one of our web servers and debian and manjaro, went through the roof.  Upon examination, apache was not reaping processes, upon killing apache to force systemd to reap them, systemd also failed to do so.

     At that point I rebooted the machine because something appears to have corrupted the process table and allowing it to continue long would have probably resulted in a total system lockup.

     Later this evening, I am going to take Debian down for around 15 minutes in order to expand the size of the virtual machine image to in turn allow for an expansion of the file system as the root file system is 92% full.

All of the shell servers are now available via ssh, vnc, desktop and terminal under web-apps on our website (initial login screen public, public), and most should also be available via rdp, there a couple I’ve recently re-installed (Manjaro, MxLinux) that I have not tested rdp yet.

And speaking of re-install, Manjaro and MxLinux are now both the current release and both are massive improvements of their prior selves.  The Mate desktop now largely improved over their former selves.  The Mate desktop now works properly under Manjaro though it still lacks mate-tweak so not as customizable, and MxLinux is now a superset of Debian Bookworm and unlike the old MxLinux some of those extensions are actually usable and the whole thing has significantly improved aesthetics.

Fedora is at some point planning on discontinuing support for NIS and at that time will be forced to drop it from our network as this is how authentication is accomplished, so just a heads up on that.  It will still be available for VPS’s if you wish as they have their own authentication and do not depend upon network authentication.

The NIS system failed tonight, NONE of the ypbind can bind to any server on any machine.  I do not know if an update broke this or just what, still investigating.

     At around 11pm this evening, I will be rebooting some machines, some virtual private servers and some shell servers, to enable shared memory in all of them.

     What this does is that a kernel routine scans the machines, and if it finds two identical pages of memory, it maps them both into one physical page, then anytime something tries to update one of those pages it uses a copy-on-write operation to create a separate modified copy.

     This allows identical pages to share one physical page freeing up more memory for things like disk buffering and caching resulting in better overall efficiency.  There are a lot of daemons that sit and run in the background from start-up to shutdown that share 99% of their memory information in common.  By allowing them to physically share them, overall memory usage is lowered.

The web app was working when I went to bed but broken when I got up, so I’m trying to figure out what went wrong over night.

You can now reach a shell server terminal or remote desktop from our website without having to go through the guacamole login.  Just use Web-Appls->Terminal or Desktop, it will display a selection of all the shell servers Terminal or Desktop, click on one and login.  With some servers there are multiple desktops available, not all are able to work with this interface.  The Mate desktop does work on ALL servers.  Sound is not yet working on all servers, I’m still working on configuring it.