The client mail server is still being beaten by bounces from misconfigured sites that, instead of rejecting mail they will not deliver up front, queue it and bounce it later.
This ran the server out of spool space sometime early this morning and as a result, some mail that couldn’t be immediately delivered, may have been lost.
Sites where immediate delivery may not have been possible tended to be large sites like Yahoo or Google which throttle the rate at which they will accept incoming mail.
A hacker used a brute force attack (kept connecting and using auth to try passwords) and successfully guessed the password of one of our customers.
This was followed by a bunch of computers bashing our mail server to send a few million spams using that customers login credentials. A botnet was apparently involved as the spam originated from many IP addresses.
I have disabled the account in question until I can contact the customer and arrange for a new, stronger, password.
I have deleted all the spam that was still in queue.
I’ve checked blacklists and, where we were listed, requested removal. One automatically removes only after seven days and won’t manually remove without a $119 extortion fee. Another has had us listed since 2007 apparently because one of our customers angered someone in IRC. That’s hard to imagine.
At any rate I’ve got things as cleaned up as they can be for now.
The following servers are restored to service:
scientific.eskimo.com
debian.eskimo.com
mx69.eskimo.com
The following servers did not come back up after the power outage and I will need to make a trip to the co-location facility to fix:
scientific.eskimo.com – shell server
debian.eskimo.com – shell server
mx69.eskimo.com – used to provide mail service from misconfigured servers
ipspace.eskimo.com – advertises availability of 207.54.0.0/19
The outage between 3:15 and 3:30pm this afternoon was caused by a power outage at the co-location center where our equipment is located. They do have UPS and backup generators. I do not yet know the cause of the power outage.
The newest version of Java doesn’t like exceptions for self-signed applets served by http protocol. It will allow you to run them but asks for confirmation each time.
I’ve changed the link for Java SSH on our home page to an https link in order to prevent Java from complaining.
Tonight’s maintenance activity is concluded.
I will be taking the web/ftp and mail servers down just after midnight for about 20 minutes each, between about 12:05AM and 1AM, in order to make images with the new encryption certificates so that if it becomes necessary to restore a machine, we will do so with the current encryption certificates and not those tied to private keys that were potentially compromised by the HeartBleed OpenSSL bug.
We use Mammoth Networks to provide DSL back haul for our customers on CenturyLink DSL circuits.
On Wednesday April 16th, starting at 10:59PM our time and going to 1AM April 17th, Mammoth will be grooming ATM “pseudowires” in Seattle. Customers should expect short switch hits (short interruptions in connectivity) during this interval.
Monday, information about a flaw in OpenSSL was released to the public. This flaw allowed an attacker to grab a random 64k segment of memory contents from the server exploited with this flaw. With enough attempts, it is possible they could obtain the private key rendering the encryption ineffective.
I became aware of this Tuesday evening thanks to notes from three of our customers and installed the necessary upgrades to OpenSSL to plug this hole.
However, because a small possibility existed that someone may have obtained the private keys in that period of time, I generated new private keys and CSR’s and asked Comodo to re-issue new certificates which they were willing to do at no charge.
These new encryption certificates were installed today. If you use web mail or the web ssh client, there is a very remote possibility that your password information could have been obtained.
To change your password, ssh to eskimo.com (the old SunOS shell server), and from the command prompt (if you are using esh for a shell, use ‘!’ to get to the command prompt), type “passwd“. (Don’t type the quote marks). It will prompt first for your existing password and then the new password twice.
Even though this exploit has only been known to the public since Monday, and we closed the hole Tuesday, it has existed in the code for approximately two years. My concern is that NSA, KGB, and other such agencies probably have known about it and exploited it for several years.
The chances of a random hacker exploiting it successfully in the day it was open are much smaller since not only would they have to execute the exploit repeatedly to get the private key, then they’d have to be in a network position to intercept that encrypted traffic.