I am going to make a second attempt at upgrading the mail server starting at around 6PM tonight. I believe I know what I did wrong to cause it to fail the first time. If all goes well there will be only minimal downtime (a minute or so to reboot). If all doesn’t go well there will be 45 minutes or so to restore from backups again.
Monthly Archives: April 2020
Ubuntu Still Upgrading
Ubuntu is still in the process of upgrading. Probably another two hours or so to go, around 6PM I am estimating. The 20.04 upgrade is large, over 6280 packages to be upgraded and close to 1000 to install.
Ubuntu Out of Service
Ubuntu is out of service owing to an upgrade that blew up. It is in the progress of updating still. In the meantime please use one of the other available shell servers listed here:
https://www.eskimo.com/services/shells/servers/
Debian, Mint, Julinux, and Zorin are all based on similar code and will have most of the same applications available.
Mail is back up however there is something wrong with either opendkim, opendmarc, or both, so for now these features are temporarily disabled.
Our mail server, mail.eskimo.com, is temporarily out of service owing to a failed upgrade. I am working on reverting back to the previous software. Estimated restoral time approximately 3AM Pacific Standard Time.
Iglulik Spontaneous Boot
The machine which holds the home directories and web server spontaneously booted this morning about 9:45 AM.
After it booted, the NFS mounts on the web server did not mount properly requiring it to be rebooted.
As far as I know all services are restored.
Web Server Interruption / Speed
I apologize for the slowdown and short interruption of the web server this afternoon. Normal traffic during the day is around 6 hits/second, but this afternoon it peaked at just under double that and the load crept up pretty high.
I took a look at resources and noticed it was running low on RAM so allocated another 8GB and then it settled down and ran okay but it required a reboot to make that change effective.
5.6 Tickless Kernels
Those of you with any Debian based system, Ubuntu, Mint, Debian, MX-Linux, Zorin, Julinux, etc, who might wish to try a 5.6 kernel, I’ve built tickless kernels. At this point the client kernel is fairly well tested, the server kernel less so but I’m running it now. There are some significant improvements in 5.6 verses 5.5, just as there were with 5.5 verses 5.4.
Tickless operation has significant advantages for both clients and servers. In clients, it means the CPU does not wake-up for clock ticks UNLESS there is other work in the work queue pending. This makes your client slightly more responsive AND saves battery on laptops by not running the CPU when there isn’t work.
On servers, tickless operation is particularly helpful when you have many guest systems on a host. With as few as 12 guests, you can end up using half your CPU cycles servicing clock interrupts, so this can substantially reduce your server load and improve responsiveness with multiple guests.
5.6 introduced the ability for peripherals to DMA directly to each other which can substantially improve things like disk-to-disk copies. There are also some improvements in NFS. There are many new drivers for new devices.
Unlike my 5.5 and 5.4 kernels, this kernel is NOT a simple clone of the default Ubuntu configuration with only a few changes made, the changes made are much more radical providing support for more devices and performance enhancements.
To use these kernels with full functionality of periphery, especially if you have Intel graphics, and NOT to have meltdown or spectre vulnerabilities, you need the very most current Linux firmware available on the Github site, the firmware included with Ubuntu or Asus are NOT current and will leave you with exposed vulnerabilities.
To get this use “get clone https://github.com/NXP/linux-firmware.git“, which will clone to a directory called firmware, then copy all of the directory structure into /usr/lib/firmware.
You can get these kernels from ftp.eskimo.com://pub/kernel/linux-5.6-tickless/[client|server/*.deb or from https://www.eskimo.com/…/linux-5.6-ti…/[client%7Cserver]/*.…. There are three .deb packages in each directory, all must be installed. Download them then type dpkg -i *.deb, then reboot.
The difference between the client and server version is that the client is fully preemptive with a clock rate of 1000HZ (though no interrupts unless there is work), where as the server is non-preemptive and a clock rate of 100HZ to minimize unnecessary interrupts and allow maximum work to be done. The client version will run fine on servers but will waste more CPU in interrupts. The server version will work fine on clients but will have a higher latency that may not be suitable for audio / video applications.
Kernel Updates
Over the next week or two, I will be updating system kernels to 5.6 kernels. This provides some additional performance enhancements verses 5.5. It allows server-to-server copies in NFS and it allows peripheral-to-peripheral DMA on the hardware side which should improve the efficiencies of things like disk to disk copies by not requiring the CPU to intervene other than to setup the DMA transfer. On my workstation it has also been somewhat more stable although I have not seen stability issues with 5.5 on the servers.
Web Server Updated
Our web server is now running Apache 2.4.43 and openssl 1.1.1f. This mainly fixes minor security issues in both packages.