Igloo Crashed

     Igloo, one of the physical hosts that among other things hosts the mail spool, crashed earlier this afternoon.

     I was compiling a new version of GCC (13.1) using eight cores and it just got too hot.  I stress test these machines thoroughly when I initially turn them up but over time CPUs suffer something known as chip rot, essentially even though electrons are 1/2000th the mass of protons, when electrons move one way atoms occasionally move the other ever so slightly and over time this creep causes features to be less well defined and maximum stable CPU speeds become slightly less.  I have already down clocked slightly ice and iglulik, this machine is the only one still at 4.4 GHz, I will need to bump it down to 4.3 or 4.2 the next time I am at the co-lo.

     In the meantime I am completing the compilations on this machine using only four cores so it generates less heat.

     I am also working on a new 10 core / 20 thread server using an i9-10900x that will have 256GB of RAM to handle some of the more CPU and memory intensive workload.  Unlike the other machines which have two memory channels, this one has four.  It will also have everything, even the root partition, on RAID.

Kernel Upgrades Completed

     Kernel upgrades are completed.  Sorry it took longer to get some of the virtual private servers up than normal.  One of the physical servers failed to come back up into the normal run level, it booted but systemd hung (which unfortunately it sometimes does though this is not frequent on the physical servers).

Kernel Upgrade Saturday 4/15/2023 11PM

     I will be performing a kernel upgrade tonight starting at 11pm, expecting to finish by 11:30pm.  Expected downtime for most services less than ten minutes, yacy.eskimo.com may take as much as 45 minutes owing to the rebuilding of the database after reboot.

     This upgrade will affect all of Eskimo North’s services, shell accounts, UUCP, e-mail, web hosting, mail lists, and virtual private servers.

     It will also affect our free services, https://friendica.eskimo.com/, https://hubzilla.eskimo.com/, https://nextcloud.eskimo.com/, and https://yacy.eskimo.com/.

Kernel Upgrades Sat April 1st 11PM

     Will be attempting an upgrade to kernel 6.1.22 tonight starting at 11pm.  Expect to be finished at least with the reboots by 11:30pm and should have all NIS and NFS dependencies restored by midnight if not sooner.  Interruptions of individual services should not exceed about ten minutes with the exception of the Yacy search engine, https://yacy.eskimo.com/, this because Yacy rebuilds it’s database after each reboot and this process takes in excess of half an hour.

     This will affect all Eskimo North services including virtual private servers, shell servers, e-mail, web hosting, and our free services including https://friendica.eskimo.com, https://hubzilla.eskimo.com, https://nextcloud.eskimo.com/, and https://yacy.eskimo.com/.

     The 6.1.22 kernel has some minor fixes relative to the 6.1.20 kernel we are presently running so I do not expect any significant operational changes.

Kernel Upgrade Saturday March 18th 11PM

     We will be upgrading our server kernels on Saturday the 18th starting at 11pm Pacific Daylight Time.  Expected downtime is about ten minutes per service with the exception of Yacy which can take half an hour or longer to rebuild database after a reboot.

     This will affect all of our services, free and non-free, virtual private servers, shell services, e-mail, and web hosting.  The free services are the fediverse services, https://friendica.eskimo.com/, https://hubzilla.eskimo.com/, https://nextcloud.eskimo.com/, and https://yacy.eskimo.com/.

Manjaro, Kernel Issues, etc.

     I had announced previously plans to retire manjaro as it was not getting any use, but in the last three months three people have started using it, therefore I do not intend to retire it at this point.

     On another note, the 6.1 kernel has been declared to be a “long-term” kernel, which means it will be supported for a long time for things like bug fixes and security fixes.  I am very happy about this as it is hands down the best performing kernel we have ever run.  6,2 so far does not perform well enough to use in service.  But we will not be forced out of 6.1 any time soon for security reasons, so this is good news.