The mail server took longer than I had expected.  And then after the backup was done it would not start properly.  So took some mucking around to figure out why and fix.  There were some things that used files on /misc which is an NFS mounted partition that were not set to wait for the mount.  So mail wasn’t back up to about 12:30AM.

     I am going to be taking the web server, mail server, and some of the shell servers down tonight at 11pm for between 20-60 minutes (mail server closer to 20, web server closer to 60) to image them (a form of backup) as I’ve made substantial changes to fix various issues to all of them and want to capture those changes so I don’t have to re-apply them in the event restoration from a backup is required.

     This will affect all customer websites for about 20 minutes, it will also affect all eskimo.com websites including friendica, hubzilla, and nextcloud, and mail will be unavailable for about an hour, but incoming mail will not be lost, it will be queued and delivered after the server comes back up.

This affected Eskimo North and customer sites as well as friendica.eskimo.com, hubzilla.eskimo.com, and nextcloud.eskimo.com.

We had a web server issue causing intermittent failure of the web server.  I thought this was load related as the server is quite heavily loaded, but it did it today during a time of very light load.  I was able to chase the problem down to a cronjob being run on a wordpress site automatically configured to redirect to secure mode when the certificate had expired.  The cronjobs were never exiting and ran once a minute and simply piled up until memory was exhausted causing failure of other services.  Normally I get automatic notifications when certificates are about to expire but did not in this instance. I’ve ordered a new certificate and commented out the cron job until it arrives and can be installed.

     Mail server changes are completed.  Mail that fails SPF, DKIM, or DMARC checks will no longer be rejected outright, instead they will be placed in your spam folder by default.  In the process of implementing this, I discovered many perl modules needed by spamassassin where not installed and so portions of spamassassin were not functioning.  Spam filtering should therefore be more effective now.

     You can now whitelist these as you could other e-mails scored as spam using the spam control facilities described here:

     https://www.eskimo.com/support/mail/spam-control-facilities/

     Similarly you can write procmail rules that handle such failures as you desire.  E-mails will now contain a header line like this:

     Authentication-Results: mx2.eskimo.com; dmarc=pass (p=none dis=none)
header.from=gmail.com

     That dmarc= may be “pass”, “fail”, or “none” and you can write rules to key off of these if you so desire.

     By default, any e-mail that arrives in your “spam” box should not be trusted, forged mails will go here.  So if you get e-mail from your bank saying you need to update your authentication or some such, do NOT click on the link if said mail is in your spam box.

     As a general safety note, I recommend NEVER clicking on these types of links, instead go directly to the site in question and make any necessary changes there.

     About a year ago we implemented opendkim, opendmark, and spf checking in order to reduce mail forgeries.  This did have the intended effects, although it’s not impossible to forge e-mails with these measures in place, it is difficult enough that it prevents the vast majority.

     However, DMARC protocol and to a lesser degree DKIM seems to be too difficult a concept for some mail providers to properly implement causing some legitimate mail to be rejected because it was marked to be checked by the sending sites DNS but they didn’t implement it correctly so it gets rejected.  This has particularly been an issue with one rather large cloud provider, but now we are seeing issues with NewEgg, a computer retailer I do quite a lot of business with and with GoDaddy.

     The existing system provides no effective means of whitelisting individually, and I do not wish to whitelist sites site-wide because then those sites can be forged.  However, I prefer to give the individual the ability to do so.

     Presently, opendmarc is implemented by opendmarc set to reject mode.  I intend to change this so that it only adds a header line to the mail and then add a rule to spamassassin to score the existence of a header indicating a failed dmarc with a really high value so that it will go to the spam folder unless you whitelist the site in your .spamassassin/user_prefs file OR you do something different in your own .procmailrc rules if you choose to override system rules.

     This way people savvy enough to recognize a forged e-mail can override the system wide filtering for themselves if they wish and those that can’t will at least have the option of examining their spam folders for missing mail and odds are good that if you’re expecting the e-mail it probably is legitimate.

     However, I will need to do this in two phases and there will be an interval during the process in which forged e-mail WILL go to your INBOX, therefore I caution you NOT to follow any links that say you need to provide authentication information for this site or any other site you do business with as they may be forged.  I will send a second notice when this is completed.

     During the first phase, I will change the configuration on OpenDmarc milter NOT to reject failed mail but only to add a header line.  Then once I find some examples of forged e-mail or create some, so I know what the headers look like, I will add a rule to spamassassin.  Between changing the configuration and adding the rule, forgeries will get through so be extra cautious with incoming mail until this has been completed.