I managed to get everything working on new mail servers based upon Ubuntu 18.04 and for me at least spam filtering is working much better.  Prior to the upgrade I was getting between 1/3rd and 1/4th of spam in my INBOX rather than spam box, now I am getting 1/25 spams in my INBOX the rest to the spam box and so far no false positives.

     The new servers bring new versions of just about all of the spam control facilities, new versions of spamassassin, postgres, dkim, spf milter, clamav, etc.  It’s still older version of postfix, procmail, and smartlist because I can not get the newer versions to play together well owing to the fact that they each want to operate in their own chroot jail which unfortunately doesn’t give them access to each other which they need to function.  I think eventually I will be able to get the newer version of postfix working but the new procmail is too broken.

    The shell server julinux.yellow-snow.net is now back in operation.  After going down a lot of rabbit holes, I thought to ask systemd the status of nis, and low and behold it was disabled.  I re-enabled it and now all is well.  So when you do an upgrade of ubuntu or a Ubuntu-derived distribution that is something to lookout for.

     Julinux kicked out a new release based upon Ubuntu-Mate 18.10, but this is about three months premature and predictably it broke things, most notably NIS.  Without NIS users can not authenticate on this machine. So for now that machine is down.  I’ll try to troubleshoot NIS and if I can’t fix it revert to the 18.04 version.

     I have made several changes to mail server configurations and DNS that should reduce spam and viruses.

• No mail will be accepted to user@mail.eskimo.com via any of the mail servers.
• SPF records are now published for mail.eskimo.com.  This should prevent forgeries since SPF records are checked on all of our servers.
• The issue which prevented the clamav anti-virus database from being updated has been repaired.
• Fail2ban was not starting on mail allowing that machine to be used for brute force password guessing attacks.  This has been fixed.

      I have restored mail.eskimo.com from backups, re-applied all of the updates between the time it was backed up and current, and still it continues to operate.  I do not know what broke the init system but it is presently all current again and yet running okay.  I will be taking the system down again about 20 minutes around midnight to make a backup with all the updates in place.

     The client mail server is up but I will need to reboot after bringing it current again with updates to make sure they didn’t break it, and if one of them did, a few more to determine which one.  I suspect an issue with clamav as it was the last thing to start before init explodes.

     The client mail server is currently down being restored from backup.  For reasons I am unable to identify, the init start up script fails to parse the entire /etc.rc.d/rc5.d directory stopping just before postfix even though if I run the scripts they run fine.

     I am restoring the server from backups which I hope will resolve the problem.  Expected downtime is about 20 minutes.  Mail should be functional by 8:20PM Pacific Time.

     Something broke the init system of mail.eskimo.com such that necessary things are not being started on the client mail server when it boots.  No cron jobs ran for some period of time and this caused the anti-virus database not to get updated.  This would have made it possible for a virus to propagate from one infected customers computer to another.  Mail may be interrupted for short intervals as I will need to perform a number of reboots to test and get the start-up logic straightened out.

ISOMEDIA will be performing network maintenance on Wednesday October 24th 2018 from 12:01AM PST to 04:00AM Pacific Time. The work being performed is not expected to be service affecting. As a precaution, please ensure that your Internet equipment is performing no critical updates, backups, or other activities over your connection during this time frame.

iSOMEDIA is the provider of our co-location rack where our Internet servers are located.

     Work is done, upgrades finished, verified proper NFS mounting and NIS binding on all machines.  I had to restart the NFS server that houses home directories twice because the new kernel did not install completely. Postfix on mail neglected to start because I had to force a reboot which did not remove the postfix pid file.  But other than that, all went reasonably smoothly.