I installed new SSL Certificates for eskimo.com today and unfortunately although they worked correctly for Apache they are NOT working for sendmail / postfix. The latter two are saying the key does not match the cert even though it’s the SAME key and certificate apache is using, so not sure what is going on with this but am restoring old certs from backup while I work to resolve this. Mail exchange with other sites or your mail client MAY fail during the time it takes to restore from backups.
Category Archives: Uncategorized
Phishing Scams and Spam Filtering
If you get e-mail saying eskimo.com has blocked X as spam but login and you can get them, this is a phishing scam from someone in Digital Sewer hosting (Digital Ocean) trying to get customers authentication info here. Please never login to anything e-mail tells you to based upon any link contained in that e-mail.
I’ve also received no fewer than three people complaining about spam blocked that isn’t spam, I will re-iterate again, I have no control over the sending sites properly configuring their mail servers and unfortunately often they don’t and if we can’t positively identify a sending site then it is going to be scored as spam. You have control over this. You can white list a specific domain or address or you can set your spam filtering score so high that nothing will be scored as spam. Anything that has improper DKIM, SPF, or DMARC is counted by our site as a forgery so will be scored 50+ depending upon other offending factors. Normal spam will usually score between +5 and +15, that is why the default score is set to 5.
If you want ALL of your spam to come through unfiltered, set the SCORE to 99. If you are seeing very little spam but getting a few false positives, consider setting your score slightly higher than the default say somewhere between 7-10, if you are getting a lot of spam consider setting it lower say ‘3 or 2’ and then whitelisting any false positives that do occur.
The following document describes in detail how to adjust spam filtering, clicking on the document will give you a better formatted version, WordPress somewhat messes the formatting.
We use Clam-AV to block viruses. Mail containing viruses is rejected with a message sent back to the sender specifying the infecting virus.
Clam-AV won’t catch all viruses. Between the time a virus is released into the wild and the time it is detected, analyzed, and a signature created, that virus is undetectable. We recommend that you install an anti-virus program on your computer, especially if using Windows.
Message which clear Clam-AV are then scored by SpamAssassin according to the likelihood they are spam.
If you do not have Procmail rules, system rules will place mail scored as spam in your “spam” folder.
If you have Procmail rules, then your rules to decide what to do with mail scored as spam. Please see System Procmail Rules.
Bayesian Filtering – Training
SpamAssassin includes Bayesian filtering. Bayesian filters learn from examples of what is spam and what is ham (non-spam).
Please send spam to spamtrap@eskimo.com.
Please send non-spam (ham) to hamtrap@eskimo.com. Mail sent to hamtrap must be sent from an eskimo address.
Bayesian filters work best if they have lots of material to compare. Please help with effective training by sending non-spam to hamtrap even if it is not misclassified. Without some ham to compare to spam, the filters can not distinguish between spam and ham.
It is best to use Pine or other mail programs which contain a “bounce” facility that will send the message without adding additional headers. Otherwise, SpamAssasin’s Bayesian filtering may decide that anything you originate and send to other users here is spam.
It is best to send ham (non-spam) to hamtrap after sending spam to spamtrap, as it will allow the Bayesian filters to “unlearn” anything incorrectly learned as spam.
SpamAssassin Preferences
SpamAssassin can be tailored to your preferences. In your “$HOME” directory, there is a hidden directory called “.spamassassin” that will contain a file called “user_prefs“.
The “user_prefs” file is where you can override any system defaults, set the scoring for the spam threshold as low or high as you like, change the scoring of any individual rules, and white_list or black_list any addresses or domains you wish.
The “user_prefs” file is an text file. You can edit it with any text editor, pico, nano, ex, vi, emacs, etc. Anything after a ‘”#” is a comment. There are commented examples in the file of how to do most things.
Examples
Whitelist From
whitelist_from address@domain.com #whitelist a specific address.
whitelist_from *@domain.com #whitelist an entire domain.
Blacklist From
blacklist_from address@domain.com #blacklist a specific address.
blacklist_from *@domain.com #blacklist an entire domain.
Blacklist To
By default, customers can receive mail at four addresses, user@eskimo.com, user@eskimo.net, user@eskimonorth.com, and user@eskimonorth.net. Ola Grande customers can also receive e-mail at user@olagrande.net.
Because eskimo.com has been around the longest, it is more prone to receiving spam than the other addresses. Some customers use eskimo.net for their primary e-mail address. If you wanted to block all e-mail except your eskimo.net address, you could so so with the following rules:
blacklist_to *@eskimo.com
blacklist_to *@eskimonorth.com
blacklist_to *@eskimonorth.net
Required Score
You can adjust the score required for mail to be considered spam. Higher scores increase the likelihood spam will end up in your INBOX. Lower scores increase the likelihood legitimate mail will be placed in your spam folder. “5” is the default value.
required_score 5
Individual Rules
You can set how much a rule contributes to the spam score. A score of zero disables that test. Negative scores reduce the likelihood mail will be considered spam.
Speakers of Asian languages, like Chinese, Japanese, and Korean, will want to add or uncomment the following:
score HTML_COMMENT_BBITS 0
score UPPERCASE_25_50 0
score UPPERCASE_50_75 0
score UPPERCASE_75_100 0
score OBSCURED_EMAIL 0
Speakers of any language that uses non-English accented characters may wish to add or uncomment the following line. These turn off rules that fire on misformatted messages generated by common mail apps in contravention of the email RFCs.
score SUBJ_ILLEGAL_CHARS 0
For a complete list of SpamAssassin tests, please see http://spamassassin.apache.org/tests_3_3_x.html.
Attack? Upcoming Changes
Today around 5pm we had some sort of issue that caused a large number of xrdp programs to be running on Ubuntu, Centos7, and Scientific7 at the same time causing a large CPU load and slow response on these servers. These did not seem to be operating normally but stuck in some sort of loop chewing up CPU.
I also noticed some kernel messages about SYN flooding on port 3389 which is the RDP port. I got into the router to try to do a traffic analysis to find where these were coming from but by the time I did they had stopped.
So either a badly behaving client or a new type of mystery denial of service attack. Don’t know which as I have not seen these before and they did not last long enough to determine the source.
Then with respect to changes, I’ll be changing the SunOS 4.1.4 server currently at “eskimo.com” to “sunos.eskimo.com“, it will still be reachable only from the network internally. This is in preparation for installation of the new router which officially does not support NAT (it actually does just not officially, it is Debian based and uses legacy IP tables so there is nothing that prevents me from using iptables to implement NAT but it is not officially supported). At any rate this will simplify configurations all around.
Sometime in the not too distant future we will be doing another kernel upgrade but it will need to be either after I get my car back from the shop or when I know my wife will be off work as the potential for systemd to hang is always present.
Other people have successfully gotten SunOS 4.1.4 working on qemu emulation so I know it is doable. I recently did succeed in getting a qemu emulation working doing UltraSparc emulation and running Redhat 6.2 for 64-bit Sparc and the performance was actually quite reasonable, so it is my long term plan to move it to an emulator. I’ve got it working to the point where I can boot from the install disk but haven’t been able to figure out the correct numbers to partition and install to a virtual disk just yet. It is not supported properly from virt-manager so I have to create this entirely by hand.
Kernel Upgrade Issues
We experienced quite a few problems tonight ALL of which were Poettering related, that is to say caused by bugs in systemd. One of our physical servers hung during the start up process in systemd, as did half a dozen virtual machines. I wish I could travel back in time and give his father a condom. Everything was back up at about 12:10AM and various services that didn’t start fixed by 1:49AM. With the exception of misconfigurations on two customer virtual private servers, EVERYTHING was the result of systemd errors.
Kernel Upgrades June 4th 11pm-12pm PDT (GMT -0700)
I am planning on doing kernel upgrades tonight. If they go as smooth as last time they will complete by 11:30 but may take another half hour to check NFS/NIS bindings.
This will affect all Eskimo North services including our Fediverse websites:
https://friendica.eskimo.com, https://hubzilla.eskimo.com/, https://nextcloud.eskimo.com/, and our main site https://www.eskimo.com/
The downtime for any given service should not exceed 10-15 minutes.
I Want One!
NIS authentication in NextCloud working again.
I apologize for the long time authentication was not working, I was unaware because my account was setup as a native Nextcloud account before I had NIS authentication working.
It is again working and we are on version 24.0.1. I have many but not all applications re-enabled, new applications that require configuration aren’t. Many old apps are not available for 24.0.1, or will be but presently are only compiled for the arm64 CPU. I will enable these as they become available.
I did not enable dashboard, but because it makes snails look like lightspeed but also because when it was previously enabled NOBODY liked it. That is to say the feedback I received regarding it was universally negative.
NextCloud
About a month ago I posted about upgrade issues with Nextcloud. I did not realize until a few days ago that it broke NIS logins because I had created my account on nextcloud before hooking it into the systems NIS authentication system. A ticket I received a few days ago alerted me of this fact.
In order to fix Nextcloud properly I am essentially going to re-install it except the database and files will remain in place. Then I will need to re-install all the apps including the one that provides authentication to the majority of Eskimo’s users via NIS.
Because the install instructions tell me to stop our web server during the duration but this does not make a lot of sense and I don’t wish a multi-hour and potentially multi-day interruption, I am going to instead disable the NextCloud config in the web server. This will cause any calls to Nextcloud to go back to our home page, then when the new version is installed, undo the web configuration disable, at which point it will be operational but without the applications and you still will not be able to login unless you, like me, created your login before I had NIS wired in.
Because the NIS connection actually involves an application, this will not work until I get the apps re-installed.
As a consequence tonight at some point if you go to nextcloud you will just get our home page. When nextcloud initially comes back you won’t be able to login right away, it may take a day or two before this capability returns because there are a lot of applications and some require some configuration before logins will be operational again.
What is Wrong
Figured out what was wrong with the 5.17.11 kernel, it is an option I selected which strengthens the kernel by zeroing the stack to initialize everything before a call and after, this required an argument to gcc which gcc-12.1 doesn’t understand. It’s only an issue in a module used for Nvidia compatibility but Ubuntu and other debian based releases are depending upon the presence of this particular library and without it act very erratically, the kernel also seems to not be stable with this option so on both notes it was a bad choice. Was just trying to harden the systems against stack exploits.
Kernel Updates Postponed
It was fortunate that kernel upgrades were delayed as I had installed it on my workstation, and the “fixes” broke the kernel much worse than it was. It had the potential to do something wrong, so far unrealized on the servers, but the current release is taking kernel oopses on a regular basis, so much worse.
And Ubuntu, which I’ve been using since 2012, has become way too Microsofty now, just like Redhat before them and really twisted my arm into changing distros again, at least in my work station, but there is no one distro that really does everything I need, so instead of being dual-boot my workstation is going to become triple boot with Win10, Debian, and Manjaro with Manjaro being my daily driver, but Debian for those cases where I need to run commercial software such as Anydesk which is not available for Manjaro because Manjaro builds pretty much everything from source.
The things Ubuntu has done recently is screwed up the libs such that I can’t have 32 bit version of Wine peacefully co-exist with virtual machines, and I can’t have anything gtk3 peacefully co-exist with the rest of the system because they’ve gone to distributing libgtk3 with snap and they are sending a version that does not match what most of the other software was compiled under so I’m getting LD failed to preload libgtk3 missing symbol errors left and right. I don’t like snaps, they are slow as snails, totally insecure, and often unreliable, and they in most cases include their own libs and containerize things which neither have a reason to be containerized nor work well in that environment.
And to add insult to injury, the most recent firefox they distributed, version 100, is not configured the way I like and when I tried to fix that I got a “Managed by Canonical”, which is Ubuntu’s parent company and it did not let me. Well if I wanted that I’d run Windows as my daily driver, NO THANK YOU. So I’m going to be mostly tied up Sunday arranging my machine correctly and kernel upgrades will wait until the next point release and then only after I’ve had a few days to be sure they’ve corrected whatever they broke in 5.17.11, perhaps 5.18.1 will be out and fix the compile problems I found in 5.18 that I did file a bugzilla report on, and in the meantime I will also file a report on the kernel oopses in 5.17.11 to make sure they are aware of them.
I built version 102 of Firefox from source today and unfortunately it has a nasty bug where it will not save the default profile.