The kernel upgrades are completed and all NFS and NIS connections have been verified.  The upgrade was relatively uneventful.  Everything is back up.

     I am in the process of upgrading vps3 because the version of Devuan has reached end of life and it no longer has repositories from which to receive security updates.  At the conclusion of this process there will be about 1/2 hour downtime for it to image it.

     The free public services https://friendica.eskimo.com/, https://hubzilla.eskimo.com/, and https://nextcloud.eskimo.com/ are back in service.

     If you get e-mail like this:

     And your user@eskimo.com is still using old version, Please tap the blue button below to upgrade your mailbox to the latest version, and get 25GB free data space with a more organized mailbox to avoid deactivation. 

     It is a scam.  You can forward it to spamtrap@eskimo.com or just delete it.  I’ve already traced the source, it is coming from a Seattle based competitor.  Slimy business practice in my view but I guess that’s our world today.

     I hope to do kernel upgrades tonight from 5.12.4 to 5.12.8, mostly a bug fix upgrade.  Because there is no change in major point release, hopefully it will go smoothly.

     This will result in approximately 10 minute outages of various services during this time frame, this includes shell servers, virtual private servers, web hosting, email, and https://friendica.eskimo.com/, https://hubzilla.eskimo.com/, and https://nextcloud.eskimo.com/.

     I’m going to take vps3, vps5, and vps7 down after 11PM and before 2AM for maintenance (to image, a form of backup) for about 1/2 hour each tonight.

     Maintenance on Fedora, Centos7, Scientific7, and JuLinux has been completed.

     All of these machines will be going down for 15-45 minutes this afternoon to image (a form of backup).  They will be taken down one at a time and I will try to avoid / delay those actively in use.

ISOMEDIA Service Affecting Network Maintenance 6/3/2021

On Thursday, June 3rd, beginning at 12:01 AM PDT and continuing until 05:00  AM PDT, engineers will be performing maintenance on the network backbone ring. This maintenance has the potential to be service affecting, with the possibility of multiple periods of up to five minutes with limited connectivity.

ISOMEDIA is where our equipment is co-located and provides our Internet connectivity.

     Recently we’ve seen huge floods of spam and more disturbing scams and viruses, that are not yet caught by clam-antivirus, from providers that either market to spammers but also have a few legitimate customers, or providers that are simply hacked to death.  Affected sites include Digital Ocean (lots of “.tech” spam), Amazones (random spam, scams, phishing attempts), mailspike and sendgrid (massive amounts of spam).  Many smaller sites primarily in Eastern Europe.

     The floods from these sites has been so severe I’ve been forced to block part of their address space.  However, this has the side effect of also blocking legitimate clients from their servers.  This is less a problem with Google’s cloud because they have all their legitimate customers properly configured with SPF, DKIM, and DMARC and I can simply block anything from their servers not properly configured.  But with these others most of their customers do not implement any of these protocols.

     I have our servers configured so that we collect all the relevant mail address data before either accepting or blocking the connection and then log that data whether accepted or rejected.  And I have a access list for domains and/or e-mail addresses so we can either accept domains from sites that are otherwise blocked, or reject known spam domains.

     But if we are rejecting legitimate addresses, I do not know until you tell me, and an e-mail or phone call saying I am not getting e-mail from XYZ is not helpful unless you have the sending domain and your e-mail address, even then the best way for me to receive this information and act on it is if you generate a ticket.

     To do this you can go to our website, https://www.eskimo.com/ and under the Support pull-down select Tickets.  If you do not already have an account on the ticket system, create one.  You do not have to be a customer to create tickets, so if you are someone trying legitimately to e-mail a customer here and getting a bounce, you also can use this system to report the issue.  For those address spaces we have to block I can create exceptions for domains or e-mail addresses that are legitimate.  Please note however, I CAN NOT create an except that will allow mail from misconfigured sending sites to get through, in those cases the SENDER will have to fix their configuration though I will be happy to assist in identifying the nature of the misconfiguration.

     I’ve completed work on the mail server.  To get it to not hang, I had to remove all the xorg-xserver-video drivers for hardware that wasn’t present (which is all of them, it’s a virtual machine) and that made it happy, no longer hangs when rebooting.

     While I was in there I removed some other cruft and optimized some settings like enabling huge memory pages and write caching, all of which should help performance a tiny bit.  I also discovered the problem with postfwd, between groovy and hirsute they changed group nobody to group nogroup, but postfwd was still expecting group nobody so I just created a duplicate entry in the group file and that made it happy.

      I will be performing multiple reboots of the client mail server tonight after 11pm to troubleshoot the issue that causes it to hang during a systemd initiated reboot.  This is a problem with the new Hirsute release of Ubuntu but I managed to fix my workstation by removing a bunch of unnecessary cruft and hope that I can do the same with the mail server.  Because I did quite a lot at once on my workstation, I don’t know exactly what it was that it was getting stuck on.  This may also cause a delay if you attempt to login to a shell server while a reboot is in progress but if this happens just wait a couple of minutes and try again.  The down times will be relatively brief (2-3 minutes).