We will be performing upgrades on the majority of our servers shortly after midnight tonight.  Be prepared for outages lasting on average about 15 minutes after midnight and before 6AM on October 13th (early Saturday Morning).

     Mint should be fully operational now.  If you find anything missing please initiate a ticket Support->Tickets.  Thank you.

     Mint is re-installed and Android Studio is now available on it as well but most of the other applications are not yet in place, mail is not configured, etc.  This will take a couple of days.

     Mint is down for maintenance.

     When attempting to install Android Studio I discovered that something was wrong with apt.  I restored from backups, turns out something was wrong with apt the last time I backed it up.  So at this point a fresh install seems in order.

      Android Studio is now installed on Debian.Eskiimo.Com, Julinux.Yellow-Snow.Net, Ubuntu.Eskimo.Com, and Zorin.Eskimo.Com.

     For those interested in developing Android Apps, Android Studios are available on the above three shell servers.  To use it, you must have x2go installed on your computer and use it to connect to one of the above servers.  In x2go, before you connect, select “Mate” as the desktop Manager.

     After you logged in under the Applications pull down menu, select Programming->Android Studio.  It takes a while to start up so be patient.  It will be fastest on Ubuntu.

     Something worth noting, Mint and Ubuntu both have Oracle Java 10 installed, the other machines have JDK 8 installed.  It is known JDK 8 is missing some libraries so some applications may not compile on these machines.

     I will be rebooting a number of servers to upgrade the kernels to a 4.18.0 kernel.  After the issues with NFS file locking caused severe operational problems with the client mail server today I decided to try the new 4.18.0 kernels on a few machines and it appears to be running significantly cleaner.  Several of the other Linux distributions we have available have already upgraded to 4.18.0 and we’ve experienced no NFS troubles on them.  Presently the web servers are now running 4.18.0.  Since that represents the heaviest load of all the servers and stresses the kernel the most, if it runs clean on the web servers it pretty much indicates it will likely run clean on everything else.

     Reboots of the mail server and a restoration from backup did not fix performance issues.

     While there was both a bot-net brute force password guessing attack in progress, which fail2ban found and locked out (over 1200 IPs involved) and an attempt to use the machines name server as an amplifier in a denial of service attack against a third party, (it won’t work because it is rate limited), neither of these represented a significant load or amount of traffic.

     I rebooted the NFS server that holds the mail spool and this fixed the issue.  I do not know what was wrong with it as I was able to access files on the spool directory prior to the boot but it had something to do with file locking as everything was waiting on a lock to clear which never did.

     We are having problems with our client mail server that provides imap, pop3 and smtp server.  Processes show waiting on disk but disk I/O is not busy.  It is not a hardware issue as far as I can tell because the machine is a virtual machine and other machines on the same hardware are fine.

     We are under going brute force password attacks and fail2ban is working to lock out the offending IPs but the botnet is large so there are many, over 1200 so far.

     To test the possibility that a recent software update broke something, I am restoring the machine image from a previous backup and will reboot into it shortly.

     OpenSuse.eskimo.com is going to go down for an upgrade to Leap 15.  The online upgrade is not working owing to some repositories not responding so going to have to do a fresh install and that’s a tedious process that’s going to take a while to get everything working again.

     Actually this is being delayed by an unspecified amount of time by the fact that the OpenSuse website is broken.

     In response to a customer request, imapfilter has been installed on all Linux based shell servers.