When I changed the mail server IP address, I neglected to change the firewall rules for imap and pop3.  This is fixed now.  If you have further issues please let me know.

     I am going to be making some more IP address changes tonight, these will mainly impact many of the shell servers and mail server briefly, although the IPs of these servers won’t be changed, the physical host they are on will which requires their reboots as the host is rebooted.

     This morning’s outage was caused by a kernel soft CPU lockup on the server that serves the home directories and also one virtual private server.  Because it is a physical host, not a virtual machine, I had to drive to the co-location to power cycle it.

     This is caused by a race condition in the kernel when two or more CPUs attempt to access a resource that has not implemented proper locking.

     I have made some changes to the system configuration that should result in an automatic panic and reboot should this occur again in the future.

     I will be rebooting various machines not to perform kernel upgrades but instead to make changes to IP addresses.  In theory rebooting shouldn’t be necessary but Leonard Poettering has so screwed up Linux with systemd that simply taking the network out of service and returning it to service with the new settings no longer works.

     The router software upgrade completed successfully.

     I changed the IP address of mail and changed it’s IP in DNS, however, I neglected to update /etc/hosts on all the machines and that overrides DNS.  This caused mail to file from shell servers as well as webmail.  This has been corrected.

     I am working on changing the IP addressing scheme for machines at Eskimo.  This will involve some reboots of NFS clients when IP server addresses change.  The mail server address will be changing from 204.122.16.222 to 204.122.16.14.  The name will remain “mail.eskimo.com”.  The old eskimo.com shell server will become “sunos.eskimo.com”.

     There will be a brief loss of connectivity for a router software upgrade tonight shortly after 11pm.  If this is typical it may last about 5-10 minutes.

An automatic update of WordPress broke the way it is displaying our home page, in particular I had a lot of inline CSS that it is for some reason not properly grocking now. So the home page will be a bit ugly for a day or two, my apologies.