When ice, the server that previously experienced the hard drive failure failed, I moved the network connection to another server, inuvik, but, I had been using ice because it is the only machine which has all Intel network interfaces.

     The other machines all have one Intel and one Realtek, thus to use them as a router requires the use of a Realtek NIC on those machines.  The device drivers for Realtek network interfaces under Linux have been dodgy, last time I had to do this we could not get the NIC to see carrier at 1gb/s so had to run at 100mb/s temporarily but this time it saw carrier fine and was stable for a week which is some kind of record.  But when I got to the co-lo the NIC was totally locked up.  Not even a reboot got it going, had to power cycle the machine.

     So now that ice has a new drive, the network connection is moved back to it and things should be stable (I know, famous last words).

     Last night I spent several hours fine tuning our newest machine, Inuvik, I was not able to get any faster CPU speeds, in fact I am running at 4.8Ghz now because I did find with some tests, notably long fft tests, it did show some instabilities even though in a weeks operation these have not manifested.

     But I was able to significantly improve memory performance from 2133Mhz to 3400 Mhz and on an 18 core CPU every memory cycle you can get is precious.

     I was also able increase the Mesh frequency slightly from 2.6Ghz to 2.8Ghz.  The mesh is similar to rings in lower core count CPUs, it is used to provide communications between the cores.  I don’t know how much traffic this actually is in a Linux environment so I don’t know to what degree this helps but every cycle one can get anywhere.

     Ok, after that I turned to kernel upgrades, and the last machine to upgrade was the one used for our router.  It had a drive that had previously shown some SMART errors but after running a diagnostic they went away and it behaved until last night.  Last night it absolutely would not boot.  Strangest damned thing, could read the drive, write the drive, but could not boot off of it.  I’ve never seen a drive failure cause this so I assumed a software issue and re-installed grub, re-installed kernels, re-built the initramfs system, and these are pretty much all the software components you should need to be able to boot but no go.  It would find and load grub but grub couldn’t find the kernel, very very odd.  I was so convinced this had to be software issue that I had to re-install 25 times to convince myself otherwise.  I finally stole one of the drives out of the RAID array and turned it into a new system disk, that worked.  But it does not have everything it needs and it’s not a healthy young pup itself.

     So for now I’ve moved the routing and all the virtual machines off this box.  At present the two services are down are the NIS master which means you can’t change your password or login shell at the moment, and a DNS server, but we have six so that isn’t going to seriously impair things.

     I have a new drive which I had purchased when the first drive started puking out SMART errors, and it also is a 7200 RPM drive with 4x the cache the old drive had.  At present I’m copying all the data off the failed drive to the drive I stole out of the RAID array to bring the machine back up, and then I’m go replace that old drive that has failed with the new one, recover any data I need for NIS and for the name server off that drive, then reformat and return it to the RAID array.

     I am working on a new video conferencing feature to add to our site shortly.  I tried to get another suite working but it depended upon a message protocol that we are not able to get to work.  This one uses infrastructure I am more familiar with so my chances are somewhat better.

     I am also looking at RustDesk as a possible replacement for Guacamole because the developers have really turned Guacamole into an unmaintainable disaster.  It really is oriented towards LDAP auth and our network isn’t, so that doesn’t work so well for us.

Our newest and most powerful server, Inuvik, is now restored to service. It is an i9-10980xe CPU with 256GB of RAM clocked at 4.9Ghz. The big  challenge to getting this operational was finding a motherboard that could reliably supply the enormous power requirements of this chip. While rated at a TDP of 165 watts, this with only a single core not clocked at more than 4.8Ghz and the remaining cores at a baseline of 3Ghz. All cores clocked at 4.9Ghz with a heavy load such as prime95/mprime #16 torture tests test #2, small fft, 36 processes (all 18 cores provide hyperthreading), it can draw 540 watts. With a CPU core voltage of 1.32 volts, this works out to 409 amps, a lot for a PCB trace to handle and in fact on the Asrock motherboard, it melted the solder at the CPU socket. Better boards handle this by having a number of planes dedicated to power and ground.

At this point, friendica, hubzilla, roundcube, yacy, and Manjaro shell server are all again operational. There are issues with Mastodon, an update left it broken and I’m still troubleshooting. Because of it’s refusal to run under a modern OS, I have Ubuntu 20.04 installed on a virtual machine that is then proxied to through the main machine via a private network. Something seems to have gone afoul with this but I’m still trying to nail it down.

     There was some major weirdness this morning and afternoon.  It started with the mail server not responding to NFS requests.  Mail is a virtual machine on the Igloo physical host, so to reboot it I had to login to Igloo however, Ubuntu in their infinite wisdom has some system wide scripts that run when you login and among other things check the mail by looking at the local mail spool which on all of the machines is NFS mounted from mail.  At this time mail was still responding to imap, pop3, and smtp so it was still possible to use via Thunderbird, but this broke shortly after I posted about it.

     It used to be NFS had a timeout and when a server did not respond, if you were patient you would eventually get past this.  But apparently the default is no longer to time out.

     So I had to drive down to the co-lo facility to reboot that machine, I apologize that I did not hear the phone ring, I was sleeping heavy and late owing to being sick last night.  Not sure what upset my stomach but I upchucked in the middle of the night and my upset stomach made it difficult for me to get to sleep for many hours.

     So when I got to the co-lo I could not reboot the physical host even with the three finger salute.  It hung on shutting down guests.  I had to forcibly reboot it with the magic-sys-request key, alt+delete+printscreen+B to force a boot.  Now I had a newer kernel prepare, three issues newer than the one in service and I knew there were some memory leaks among other things fixed, so thought well might as well install the new kernel on the physical hosts and mail while I am here.

     This went ok on Igloo and Iglulik, but when I went to reboot on Ice, it would not come up.  Strangely ice had swapped it’s drive letters between sda and sdb, sda had become sdb and vice versa.  I had not moved the drives.  A while back I had changed the UUID’s to drive numbers because the blkid program at the time was unreliable leading to occasional failed reboots.  Now the machine was randomly swapping drive letter, so I put it back to UUID so it doesn’t care about the drive letters.  If blkid becomes a problem again I’ll probably switch to labels which honestly makes more sense anyway.

     I will be rebooting some of the shell servers and other non-physical hosts tonight to upgrade the kernels on them.  I’m also going to try to find the script that is checking for mail and eliminate it on the physical hosts so I can reliably get into them if mail goes down again.

     At some point libvirtd on igloo, the machine which hosts mail and a number of shell servers, failed.  Libvirtd is the server side virtualization management daemon, it is responsible for starting, stopping, arranging networking, storage, and system resources for kvm/qemu guests (also for xen but we aren’t using xen here).

     This affected a number of machines including mail and because every server NFS mounts the mail spool from mail, it affected them indirectly.

     The message that Igloo gave in syslog relating to libvirt was:

        libvirtd[2271]: internal error: wrong nlmsg len

     The “nlmsg” refers to Netlink, so it would appear something went wrong in networking and libvirtd didn’t know how to handle it and crashed.

     I don’t know exactly how long and how deep the outage was since it was kind of a gradual deterioration situation after libvirtd crashed.  I was going to add an automatic restart to libvirtd in systemd to prevent this specific failure in the future but found it was already in place but incompletely specified so perhaps systemd choked.  I have corrected that.

     I received about eight tickets on this issue, and I really appreciate it that the ticket system is being used, but also with outages of this magnitude a phone call would be good because if I’m not actively at the terminal I may not be aware of issues.

     Rust is a new compiled programming language that users a new memory
management scheme.

     I first learned several assembly languages and then learned C, and because I learned assembly first and thus really think in terms of what the hardware does, I have not had issues with array bounds or de-referenced pointers but a lot of people have. In fact this tends to be what causes the majority of privilege escalation exploits.

     Many languages, Java, Python, Perl, BASIC, etc solved this by using a memory management technique known as garbage collection but this method has severe performance issues. First, it can be difficult for the language to determine if a particular variable will ever be accessed again, thus memory release may be very delayed resulting in wasted memory. But more significant is that garbage collection causes periodic halts in execution that can be very annoying.

     Enter rust, they invented a new method of memory management in which you declare to the compiler how memory is used, in what contexts and time frames, and this enables the compiler to manage memory much as you would do by hand without the human error component.

     This makes Rust an ideal replacement for C, for those who are less disciplined, and for critical tasks, because, like C, it can approach assembler in efficiency, doesn’t introduce the periodic lags of garbage collection, and yet protects you against buffer overruns and pointer de-reference errors.  Now if they will only invent a text editor that corrects run-on sentences.

     I’ve installed the rust compiler rustc on all of the shell servers and working on installing it on the other machines as it will be necessary in the future for kernel compilation.

     The newest version is on Fedora and Rocky8, 1.80, slightly older versions on Ubuntu, and Zorin, 1.75, and even older versions on Debian and MxLinux 1.65. 

     The brief web outage today lasting approximately two minutes was to apply a security update to the Apache server bringing it up to 2.4.62 owing to vulnerabilities found in the previous version.

     At the same time, I also upgraded the kernel to 6.10.0.  The 6.10 kernel has some improvements that speed up encryption.

     Some update pushed on Rocky8 and Fedora broke rwho and ruptime on those machines.  They will still provide user status to other servers but are no longer pulling other servers for rwhod.

     Further, ruptime requires the ‘daemon’ command which is no longer available on these machines.

     At some point NIS will disappear from Fedora and when it does we will be forced to retire the machine at that time.  Because we don’t know when this will happen we can not predict this date.  Therefore we recommend NOT relying on that machine for anything.  If you have cron jobs on it, please move them to rocky8 or if you do not need a Redhat environment, one of the other servers.

     Roundcube mail is back.  It is in the WebApps->Mail->Roundcube menu or the direct path is https://roundcube.eskimo.com/.

     This is running on the new server and will be down between Wednesday evening and possibly Saturday though I may be able to get it back up sooner.  I am replacing the motherboard, AGAIN, as the new motherboard is stable BUT has two dead memory slots so the machine is only seeing 192GB of 256GB of installed memory.

     None the less, it is now an 18 core machine, 36 threads, at 4.8Ghz all cores.

     The way I finally got roundcube working is that I installed Ubuntu 20.04 on a virtual machine on the new server.  I have this virtual machine listening to a non-routeable IP address so that it can not be reached directly from the outside world to provide some degree of security to ancient software.  Then I have it proxied through the main web server on the new machine which listens to both a public IP address and a private one that it can use to talk to internal servers.  Something about 24.04 breaks roundcube, I know not what.