Generated by All in One SEO v4.8.8, this is an llms.txt file, used by LLMs to index the site. # Eskimo North Your Home on the Internet +1 206 812-0051 ## Sitemaps - [XML Sitemap](https://www.eskimo.com/sitemap.xml): Contains all public & indexable URLs for this website. ## Posts - [News](https://www.eskimo.com/news/) - [Outgoing Mail Bounces](https://www.eskimo.com/2022/07/outgoing-mail-bounces/) - If you get an e-mail bounced back with a message something to the effect of no inverse DNS for 204.122.16.222, this was not our doing but something that went wrong at ARIN, the entity responsible for assignment of IP space and associated records. Re-send your e-mail and it should go through successfully now unless - [Reboots Around Midnight](https://www.eskimo.com/2022/07/reboots-around-midnight/) - I am going to reboot a bunch of machines tonight sometime midnight or after to change some configuration options. It turns out that there is a bug in virt-manager where if you tell it to copy the host CPU configuration, it doesn't, instead it takes an four core, eight thread, CPU, and turns it - [Encryption Certificates](https://www.eskimo.com/2022/06/encryption-certificates/) - I had the certificates re-issued with a new CSR created using the current openssl 3.0.1, and the resulting certificate worked fine with Apache, Postfix, Dovecot, and MariaDB, so encryption is now secured for another year. - [Old Certs Restored](https://www.eskimo.com/2022/06/old-certs-restored/) - Old certs are restored from backup so things will work for now. I have to get new certificates in place by July 7th as these expire at that date. But I will keep the old certs on disk until I know the new ones are working in mail as well as the web. - [Mail SSL Errors](https://www.eskimo.com/2022/06/mail-ssl-errors/) - I installed new SSL Certificates for eskimo.com today and unfortunately although they worked correctly for Apache they are NOT working for sendmail / postfix. The latter two are saying the key does not match the cert even though it's the SAME key and certificate apache is using, so not sure what is going on - [Phishing Scams and Spam Filtering](https://www.eskimo.com/2022/06/phishing-scams-and-spam-filtering/) - If you get e-mail saying eskimo.com has blocked X as spam but login and you can get them, this is a phishing scam from someone in Digital Sewer hosting (Digital Ocean) trying to get customers authentication info here. Please never login to anything e-mail tells you to based upon any link contained in that - [Attack? Upcoming Changes](https://www.eskimo.com/2022/06/attack-upcoming-changes/) - Today around 5pm we had some sort of issue that caused a large number of xrdp programs to be running on Ubuntu, Centos7, and Scientific7 at the same time causing a large CPU load and slow response on these servers. These did not seem to be operating normally but stuck in some sort of - [Kernel Upgrade Issues](https://www.eskimo.com/2022/06/kernel-upgrade-issues/) - We experienced quite a few problems tonight ALL of which were Poettering related, that is to say caused by bugs in systemd. One of our physical servers hung during the start up process in systemd, as did half a dozen virtual machines. I wish I could travel back in time and give his father - [Kernel Upgrades June 4th 11pm-12pm PDT (GMT -0700)](https://www.eskimo.com/2022/06/kernel-upgrades-june-4th-11pm-12pm-pdt-gmt-0700/) - I am planning on doing kernel upgrades tonight. If they go as smooth as last time they will complete by 11:30 but may take another half hour to check NFS/NIS bindings. This will affect all Eskimo North services including our Fediverse websites: https://friendica.eskimo.com, https://hubzilla.eskimo.com/, https://nextcloud.eskimo.com/, and our main site https://www.eskimo.com/ The - [I Want One!](https://www.eskimo.com/2022/06/i-want-one/) - [NIS authentication in NextCloud working again.](https://www.eskimo.com/2022/06/nis-authentication-in-nextcloud-working-again/) - I apologize for the long time authentication was not working, I was unaware because my account was setup as a native Nextcloud account before I had NIS authentication working. It is again working and we are on version 24.0.1. I have many but not all applications re-enabled, new applications that require configuration aren't. - [NextCloud](https://www.eskimo.com/2022/06/nextcloud-10/) - About a month ago I posted about upgrade issues with Nextcloud. I did not realize until a few days ago that it broke NIS logins because I had created my account on nextcloud before hooking it into the systems NIS authentication system. A ticket I received a few days ago alerted me of this - [Pixelfed.Eskimo.Com Properly Federating](https://www.eskimo.com/2025/07/pixelfed-eskimo-com-properly-federating/) - I finally got pixelfed.eskimo.com to properly federate so that it now shares images with the rest of the fediverse allowing you to see pictures from people you follow on other sites and they your pictures. - [Eskimo North Changes](https://www.eskimo.com/2025/06/eskimo-north-changes/) - I've firewall rules to address recent SYN and UDP floods. So far they have had the intended effect. I've installed a new shell server, anduinos.eskimo.com. This is a debian derived server similar to popOS except the default graphical environment is Gnome configured to look like Windows. Unfortunately this environment does not - [Mail System Maintenance](https://www.eskimo.com/2025/06/mail-system-maintenance/) - The mail system is currently shutdown to image the servers. That is to say to take a copy of the virtual machine image while it is down in case it should get corrupted or lost and needs to be restored in the future. I expect to have the mail sub-system returned to service around - [Web Server Database](https://www.eskimo.com/2025/06/web-server-database/) - Mariadb, the database that is used to provide SQL service for most web apps requiring it (a handful use postgresql) hung when some internal process failed to remove a lock and everything else piled up behind it. I could not restart it with either systemd or mysqladmin, and even a regular kill signal did - [Mail](https://www.eskimo.com/2025/05/mail-23/) - Mail system is down. Something very weird happened when I rebooted the server it is hosted on. It did not find the qemu img for the mail spool at the location specified in the XML file and the location was obviously wrong as the directory did not exist. Yet, I had rebooted the server - [Mail](https://www.eskimo.com/2025/04/mail-22/) - Mail is down presently as it is being moved from the Iglulik server to inuvik. The reason I am doing this is the machine is showing some signs of instability so I want to get mail, a critical service OFF of this machine before it crashes and burns. This was THE only machine that would - [Squirrelmail Moved](https://www.eskimo.com/2025/04/squirrelmail-moved/) - Squirrelmail has been moved to our new web server. The menu items through our home page and web-apps will work as normal, but the direct URL has changed to https://squirrelmail.eskimo.com/. - [Mail Up](https://www.eskimo.com/2025/03/mail-up-4/) - Mail is back up and is now on a stable but slower platform. I will need to reload Linux on the other machine as it has somehow become badly corrupted. Systemd is not acting properly on the larger machine and is starting servi ces, particularly systemd-networkd which needs NOT to be running as I - [Mail Maintenance](https://www.eskimo.com/2025/03/mail-maintenance-7/) - Mail is going to be down for approximately 2 hours to move from the present host that has some software networking instability issues and needs a reload to a stable host. Because this involves copying 400GB of data across a 1Gb/s link, it will take approximately two hours, I expect it to be back up - [Inuvik - Still Trying to Kill me](https://www.eskimo.com/2025/03/inuvik-still-trying-to-kill-me/) - Inuvik, the server that runs all the social media and also runs manjaro and alma shell servers is still out of service. I can't get an Asus X299 board that doesn't have dead memory channels, they are all just shit, and I got a Gigabyte board that did work but one of the SATA controllers - [Dear Monitor Manufacturers](https://www.eskimo.com/2025/03/dear-monitor-manufacturers/) - The DCI-P3 standard is INADEQUATE. To my eyes at least, 620nm light does NOT appear RED, it appears watermelon orangish, NOT a pleasing color to represent say a red rose. I don't understand why it is so difficult to make pixel filters, or quantum dots in the case of Qled technology that make 670nm - [Machine is backup, Raid has finished reconstructing. Hopefully it will be stable but there hints of some software instabilities.](https://www.eskimo.com/2025/03/machine-problems-with-mail-ubuntu-and-some-others/) - The machine which provides the mail spool, ubuntu, and some other systems is having issues rebuilding a RAID array after a disk replacement marking the RAID dirty and preventing it's re-assembly. The machine is back up and operational at the time of it's posting but I do not know yet if it will - [Realtime Kernel Now Available](https://www.eskimo.com/2025/02/realtime-kernel-now-available/) - As some of you may be aware, I've been building kernels for Eskimo's servers and for my own workstation tickless for some time and making them available to the general public. Until now I have not made realtime kernels available because building realtime used to conflict with virtual machines, virtual GPUs, and some other - [Friendica, Hubzilla, Mastodon, Alma, Manjaro, etc.](https://www.eskimo.com/2025/02/friendica-hubzilla-mastodon-alma-manjaro-etc/) - The server Inuvik that provides all of these services has been down for a while. I apologize this but I spent some time in the hospital with a serious infection and another month on home IV and working on computers will tethered to an IV wasn't very tenable. I had ordered a new - [Mail Subsystem Repaired](https://www.eskimo.com/2025/01/mail-subsystem-repaired/) - The grub subsystem on mail.eskimo.com has been repaired. For some reason the Ubuntu update system installed some packages needed for a signed kernel onto a system that is running an unsigned kernel AND installed some arm64 packages on an x86_64 system. I've deleted the inappropriate packages, installed the correct and necessary packages and test booted - [Mail Subsystem](https://www.eskimo.com/2025/01/mail-subsystem/) - I have to take the mail subsystem down for perhaps 45 minutes or so in order to fix an issue with grub that was broken by a Ubuntu update that left it in a half-secure boot and half non-secure boot state. This could, if not fixed, cause a failure to reboot at a more critical - [Roundcube, Manjaro, Yacy, Friendica, Hubzilla, Mastodon, Misskey Down](https://www.eskimo.com/2024/12/roundcube-manjaro-yacy-friendica-hubzilla-mastodon-misskey-down-2/) - Wanted to update folks, the new CPU arrived, I installed it and the machine would not even post. Have not had time to troubleshoot further. However, I have another machine that I recently build that is essentially the same architecture, 10-core rather than 18-cores, but the 18 core machine was only running at less than - [Roundcube, Manjaro, Yacy, Friendica, Hubzilla, Mastodon, Misskey Down](https://www.eskimo.com/2024/12/roundcube-manjaro-yacy-friendica-hubzilla-mastodon-misskey-down/) - The estimated time of return to service is around December 20th. I am waiting upon the arrival of a new cpu. - [Ubuntu](https://www.eskimo.com/2024/12/ubuntu-12/) - Ubuntu is currently down because it is in the process of being moved to new faster hardware. It should be back up about 12:30AM Dec 11th. - [Mail Server](https://www.eskimo.com/2024/12/mail-server-13/) - The mail server is taking longer to copy than I anticipated. This is because the RAID array is still synchronizing disk and so competing with the copy for disk I/O. - [Client Mail Server](https://www.eskimo.com/2024/12/client-mail-server-3/) - The client mail server, mail.eskimo.com, is in the process of being moved from the Igloo server to the Ice server. This is being done because the new Ice has 2-1/2 times as many CPU cores as the old machine and 4x the amount of RAM and mail needs more RAM to handle list serve - [Ubuntu Is Down](https://www.eskimo.com/2024/12/ubuntu-is-down/) - The reason mail did not work on Ubuntu is that I accidentally left a development repository on during the upgrade. This resulted in a mixture of different package versions including a broken libglib2 library which there was no way of fixing save re-installing. I am in the process of re-installing all the packages - [Ubuntu Upgraded but Outgoing Mail is Broken](https://www.eskimo.com/2024/11/ubuntu-upgraded-but-outgoing-mail-is-broken/) - Ubuntu is upgraded to 24.04 Noble but outgoing e-mail is presently broken. Please use another shell server to send or reply to e-mail. I am still working on it. - [Ubuntu - Upgrade in Progress](https://www.eskimo.com/2024/11/ubuntu-upgrade-in-progress-2/) - Ubuntu is currently unresponsive because of an update in progress from 22.04 Jammy to 24.04 noble. This is taking longer than expected largely owing to the huge number of packages installed and automatic backups of the host server taking place simultaneously. In the meantime, please consider using debian, mint, mxlinux, popos, or zorin as alternatives. - [Imapd / Pop3](https://www.eskimo.com/2015/03/imapd-pop3/) - Further research suggests that this isn't going to fix it. I'm going to update Dovecot anyway just to get it current but will probably do this later this evening instead of at 5pm. It appears that this problem is because of the POODLE exploit that came out which RedHat "solved" by disabling SSLv3. - [Friendica.Eskimo.Com - Down for a couple of Hours](https://www.eskimo.com/2024/11/friendica-eskimo-com-down-for-a-couple-of-hours/) - I was not where I thought I was and did an rm -rf removing the friendica directory in the process. So it's being restored from compressed backups. Because the partition that is backed up is 13TB partition, this is going to take a little while. - [New Social Media Service](https://www.eskimo.com/2024/11/new-social-media-service/) - In an attempt to further the freedom of speech, I have added a new Misskey social media site to Eskimo North. It can be reached through our Web-Apps menu on the main website https://www.eskimo.com/ or directly at https://misskey.eskimo.com/. It is a minimal setup at present so not super pretty but I'll fill it - [Ice / Redundancy Setup](https://www.eskimo.com/2024/11/ice-redundancy-setup/) - Just a heads up on how things are going with Ice and general redundancy measures. I've got cron jobs in place to distribute SSL certs across all of the physical machines so any one physical machine being alive will be adequate to get access to SSL certs. I've got a 5th name server running as - [Anki](https://www.eskimo.com/2024/11/anki/) - At the request of one of our customers, "anki" has been installed on all of our shell servers. The version installed is the qt6 version from the ankiweb website with the exception of Mint. On the Mint server, the distribution version is installed because system security would not permit some of the actions - [Alma Linux](https://www.eskimo.com/2024/11/alma-linux/) - A new shell server based upon the Alma Linux distribution is now available. The server name is "alma.eskimo.com". It is available via ssh, vnc, rdp, and guacamole. To access via guacamole (web) go to https://www.eskimo.com/ Then hover over web apps and select Terminal or Desktop. At the login prompt type "public" in - [Some Cable Providers Blocking Port 22](https://www.eskimo.com/2024/10/some-cable-providers-blocking-port-22/) - If you are having issues connecting to a shell server using port 22, we have found at least one cable company is blocking connections to port 22. On most of our shell servers, we also listen to port 443. If ssh times out, try "ssh -p 443 server.eskimo.com". Substitute "server" with the desired host, i.e., - [Inuvik](https://www.eskimo.com/2024/10/inuvik-2/) - Our Inuvik server which hosts manjaro, friendica, hubzilla, mastodon, and yacy, proved to be unstable under 6.11.4 as it was on 6.11.3, so I am headed over to the co-lo facility to reboot back onto 6.11.2 which is stable. Estimated return to service time 01:15 Pacific Daylight Time. - [Maintenance Work Is Complete](https://www.eskimo.com/2024/10/maintenance-work-is-complete/) - Inuvik now has a brand new Seasonic 1200 watt supply, it also has the good thermal paste now which lowered CPU max temps by about 10C, and I wire tied the fans to the heat sink because the fan clips kept slipping off. It took me longer than expected because I forgot two - [Planned Maintenance](https://www.eskimo.com/2024/10/planned-maintenance/) - Just a heads up, I will be taking Inuvik down for about 4 hours tonight to replace a power supply. It was initially very spotty after I brought it back online after a month or so of downtime to get a working motherboard in place, but after a week or so settled down. This is - [Priorities, Immediate Works, Future Plans to Address](https://www.eskimo.com/2024/10/priorities-immediate-works-future-plans-to-address/) - This outage, I learned three resources were particularly important to have available on more than one machine: DNS - Without this mail will returned no such address. SSL Certificates - Without this no encrypted services, mail, web, databases, can be started. Mail spool - Without no mail services. Home directories - Without no mail - [Outage - Post Report](https://www.eskimo.com/2024/10/outage-post-report/) - To the best of my knowledge, everything is back online now but not all of the hardware, thus some things will be not as fast as usual. We have four physical hosts providing the services with numerous virtual machines on these hosts. Two hosts are i7-6700k 4 core/8 thread systems with 64GB RAM, - [Another Machine Died around 4AM](https://www.eskimo.com/2024/09/another-machine-died-around-4am/) - This machine was being our router, it is the only machine aside from Inuvik that has more than one Intel NIC. So presently Iglulik is playing router but it only has one Intel and one Realtek interface, the Realtek is supposed to be capable of 100, 1G, and 2.5G but the Linux drivers for - [Inuvik Restored to Service](https://www.eskimo.com/2024/09/inuvik-restored-to-service/) - Innuvik is back in service. I fixed the cable connector by epoxying the 4-pin to the 20-pin effectively making it a 24 pin connector so the latch on the 20 pin section holds all in. I no longer get intermittent power when I move the cable around. I discovered that about a penny - [Inuvik Update](https://www.eskimo.com/2024/09/inuvik-update/) - I was able to fix the existing power connector with epoxy, it no longer loses power if I wiggle it around. And I took the CPU off and it definitely developed a bad air gap, almost a penny right in the middle of the heat spreader. I could tell because the compound had dried - [Update on Inuvik (the newest server)](https://www.eskimo.com/2024/09/update-on-inuvik-the-newest-server/) - Inuvik presently provides service for: roundcube.eskimo.com, friendica.eskimo.com, hubzilla.eskimo.com, mastodon.eskimo.com, and yacy.eskimo.com (and I am working on some additional services). It has been unstable and after bringing it home I discovered the power connector was flaky. I ordered a new cable but it was the wrong cable. The motherboard has a 24-pin plus two - [Friendica.eskimo.com, Hubzilla.eskimo.com, Mastodon.eskimo.com, Yacy.eskimo.com, roundcube.eskimo.com](https://www.eskimo.com/2024/09/friendica-eskimo-com-hubzilla-eskimo-com-mastodon-eskimo-com-yacy-eskimo-com-roundcube-eskimo-com/) - The above services will be down until some time Saturday evening. This machine has been unstable and getting increasingly so. It does have a heat issue which I will resolve when some new heat sink compound arrives, but even down clocked so it did not get too hot it was still crashing and sometimes - [Inuvik Too Hot](https://www.eskimo.com/2024/09/inuvik-too-hot/) - Inuvik is running too hot. This machine was running at 4.8Ghz small fft torture test 36 threads 2/threads per core before I brought it over to the co-lo but it is exceeding 96C now but only on a couple of cores. When you have a couple of cores running hot on a multi-core - [Network Outage](https://www.eskimo.com/2024/09/network-outage/) - When ice, the server that previously experienced the hard drive failure failed, I moved the network connection to another server, inuvik, but, I had been using ice because it is the only machine which has all Intel network interfaces. The other machines all have one Intel and one Realtek, thus to use them - [Apple IOS 18 and Apple IpadOS 18 MailBug](https://www.eskimo.com/2024/09/apple-ios-18-and-apple-ipados-18-mailbug/) - If you are thinking of upgrading to IOS 18, you may want to hold off as there is a bug in the mail application. You may be able to work around this by adding just a '/' for the mail path and you mail not. If you have already upgraded to IOS 18, you can - [Last Night - Projects](https://www.eskimo.com/2024/09/last-night-projects/) - Last night I spent several hours fine tuning our newest machine, Inuvik, I was not able to get any faster CPU speeds, in fact I am running at 4.8Ghz now because I did find with some tests, notably long fft tests, it did show some instabilities even though in a weeks operation these have - [Inuvik](https://www.eskimo.com/2024/09/inuvik/) - Our newest and most powerful server, Inuvik, is now restored to service. It is an i9-10980xe CPU with 256GB of RAM clocked at 4.9Ghz. The big challenge to getting this operational was finding a motherboard that could reliably supply the enormous power requirements of this chip. While rated at a TDP of 165 watts, this - [System Issues Resolved](https://www.eskimo.com/2024/09/system-issues-resolved/) - There was some major weirdness this morning and afternoon. It started with the mail server not responding to NFS requests. Mail is a virtual machine on the Igloo physical host, so to reboot it I had to login to Igloo however, Ubuntu in their infinite wisdom has some system wide scripts that run when - [System Issues](https://www.eskimo.com/2024/09/system-issues/) - There appears to be an issue with the mail server this morning. None of the NFS mounts to other systems are working and ssh isn't working but mail services are working so mail clients like Thunderbird are working. I am trying to get into the physical servers so I can reboot mail but it's hanging - [System Issues Today](https://www.eskimo.com/2024/08/system-issues-today/) - At some point libvirtd on igloo, the machine which hosts mail and a number of shell servers, failed. Libvirtd is the server side virtualization management daemon, it is responsible for starting, stopping, arranging networking, storage, and system resources for kvm/qemu guests (also for xen but we aren't using xen here). This affected a - [Rust](https://www.eskimo.com/2024/08/rust/) - Rust is a new compiled programming language that users a new memory management scheme. I first learned several assembly languages and then learned C, and because I learned assembly first and thus really think in terms of what the hardware does, I have not had issues with array bounds or de-referenced pointers but - [Brief Web Outage 14:24-14:27 July 17th](https://www.eskimo.com/2024/07/brief-web-outage-1424-1427-july-17th/) - The brief web outage today lasting approximately two minutes was to apply a security update to the Apache server bringing it up to 2.4.62 owing to vulnerabilities found in the previous version. At the same time, I also upgraded the kernel to 6.10.0. The 6.10 kernel has some improvements that speed up encryption. - [Fedora and Rocky8 Info](https://www.eskimo.com/2024/07/fedora-and-rocky8-info/) - Some update pushed on Rocky8 and Fedora broke rwho and ruptime on those machines. They will still provide user status to other servers but are no longer pulling other servers for rwhod. Further, ruptime requires the 'daemon' command which is no longer available on these machines. At some point NIS will disappear - [Don't Buy Epson!](https://www.eskimo.com/2024/07/dont-buy-epson/) - I am a little bit more than pissed off at what Epson pulled. I had a WF2950 all-in-wonder inkjet printer / scanner / fax. It isn't officially supported under Linux but it mostly works. The mostly being the need to boot windows for Firmware updates. Today it just stopped working, the scanner wasn't seen anymore - [Test](https://www.eskimo.com/2024/07/test-4/) - Testing. - [](https://www.eskimo.com/2024/06/8398/) - This shows where the Asrock X299 steel legend motherboard failed. The i9-10980 CPU ran for ten hours then hung. I power cycled the machine and smoke came out. Here is where it originated from, the backside of the motherboard under the cooler backplate. The name goes on before the quality goes in. - [New Server - ARRRRRrRRRRRrrrrggggghhHH!](https://www.eskimo.com/2024/06/new-server-arrrrrrrrrrrrrrrggggghhhh/) - I'm on my 4th MOTHERBOARD with the new server, this one I got running last night, it ran all night but crashed at 10:17AM, not the end of the world as I'd not completely dialed in the operating parameters yet, but I powered it down and back up to reboot since it was hung and - [New Server Still Broke](https://www.eskimo.com/2024/06/new-server-still-broke/) - Replaced the Asus Prime X299 IIA board with an Gigabyte X299 Aorus Master, this board won't even post. Says bad memory, but I put in just two modules and no matter what two I tried it would not work, then I tried some Crucial memory I had on hand, it won't post either, so - [Rocky8](https://www.eskimo.com/2024/06/rocky8/) - Since elm successfully compiled on Rocky8, I decided to try metamail. It successfully compiled as well and is now installed and linked into elm. Also, this is a reminder that friendica.eskimo.com, hubzilla.eskimo.com, mastodon.eskimo.com, yacy.eskimo.com, and manjaro.eskimo.com will all be going down for probably around six hours tonight in order to replace the motherboard. - [Roundcube Is Back](https://www.eskimo.com/2024/06/roundcube-is-back/) - Roundcube mail is back. It is in the WebApps->Mail->Roundcube menu or the direct path is https://roundcube.eskimo.com/. This is running on the new server and will be down between Wednesday evening and possibly Saturday though I may be able to get it back up sooner. I am replacing the motherboard, AGAIN, as the new - [New Server Status](https://www.eskimo.com/2024/06/new-server-status/) - On the newest server, I have replaced the motherboard (was unstable and would only lock an i9-10900x at 4.0Ghz). Replacing the motherboard got that up to 4.3Ghz but did not fix the stability issue AND introduced a new issue, this new motherboard has two dead DIMMS slots and it appears to be a compatibility issue - [New Shell Servers Rocky8 and PopOS](https://www.eskimo.com/2024/06/new-shell-servers-rocky8-and-popos/) - As previously noted, centos7, scientific7, and now centos-stream are going away. Centos7 and scientific7 at the end of this month, and centos-stream in a few hours as Redhat has already dropped the repositories for it ahead of schedule. I've brought two new distros online, rocky8.eskimo.com is primarily a replacement for centos-stream, centos7, and - [Redhat Screws Us Again](https://www.eskimo.com/2024/06/redhat-screws-us-again/) - Redhat screwed us again. Sorry for bad news, but Centos-Stream which originally was touted as a rolling release, i.e. no version numbers just continued updates, instead became a fixed release towards a given platform, i.e., centos-stream8 development for centos8, etc. And so was discontinued along with Centos8 early I might add (end of - [Mail Lists and Some Aliases Out Of Service](https://www.eskimo.com/2024/06/mail-lists-and-some-aliases-out-of-service/) - Mail lists and some aliases are currently out of service owing to the aliases file on the client mail server somehow being truncated and half not there. I am restoring from backups. While this is in progress mail lists will not work as ALL aliases for smartlist are gone and many users aliases are gone - [Git is now installed on Ubuntu](https://www.eskimo.com/2024/05/git-is-now-installed-on-ubuntu/) - Git is now installed on Ubuntu.eskimo.com. At present only the git application itself is installed, there is no GUI or web interface. I plan on installing some web interface after the new server is stable, this will probably be another two weeks. I have shipped back the defective CPU, it should arrive - [New Shell Server - PopOS.Eskimo.Com](https://www.eskimo.com/2024/05/new-shell-server-popos-eskimo-com/) - I've added a new shell server, popos.eskimo.com. Presently it can be accessed only via ssh and x2go, I have not yet setup rdp or vnc or guacamole. Let me know if there is something you would like installed. Pretty much anything available to Ubuntu is available to PopOS. - [New Server](https://www.eskimo.com/2024/05/new-server-2/) - I have the new server operational with the new motherboard and OLD CPU. However, it will not operate at rated speed let alone overclock. There is something wrong with the memory controller and it will not operate at the rated 2933 Mhz under Linux, oddly it will run at over 4300Mhz using memtest-86 so - [Moving Domain, Name Server Changes, Hardware Upgrade](https://www.eskimo.com/2024/05/moving-domain-name-server-changes-hardware-upgrade/) - I am moving eskimo.com domain to a new registry. I have some issues doing this in that my name servers are in the same domain presently. This is problematic as it creates a circular resolution problem where to resolve a name server address here, say ns1.eskimo.com, you first need to access the name server to - [New Web Applications Server](https://www.eskimo.com/2024/05/new-web-applications-server/) - The new server continues to be unstable, it tends to spontaneously reboot when the upper reaches of memory get used. It may be a bad DIMM. I also accidentally over-volted the CPU at one point, I meant to set 1.17 volts and accidentally set 1.7, the max for this CPU is supposed to be - [Manjaro Is Available Again](https://www.eskimo.com/2024/04/manjaro-is-available-again/) - Manjaro finally released an ISO that correctly builds x2goserver, nis, and other necessary tools. It is installed and operational. If there are software packages you would like installed let me know and I'll at least attempt. ELM will not build on it just like most other modern Linux platforms so I can not install - [Older Web Server Outage](https://www.eskimo.com/2024/04/older-web-server-outage/) - Last night the old web server was rebooted by me to change an IP address that for some reason the networking would not let go of even with a network restart. I neglected to check to make sure everything came back up, my bad. Somehow ownership of the encryption key for mariadb got changed - [Mail](https://www.eskimo.com/2024/04/mail-21/) - Owing to the ypbind unbound issues causing mail to be returned as no such address, I've created the following script which is run on all the mail servers out of crontab once a minute. The purpose of this script is to keep track of the status of ypbind, if unbound, shutdown postfix so sending mail - [FTP Server Replaced](https://www.eskimo.com/2024/04/ftp-server-replaced/) - Replaced the old wu-ftp ftp server server with vsftpd, seems to be working. - [FTP Server](https://www.eskimo.com/2024/04/ftp-server/) - The ftp server is broken. This is caused by libraries being replaced by Ubuntu upgrades with libraries no longer compatible with the libraries it was compiled against. Further, the existing source for wu-ftpd will no longer compile in the modern compile environment so I can not fix it as I have done in the - [Drive Replacement Successful](https://www.eskimo.com/2024/04/drive-replacement-successful/) - Drive replacement went extremely smoothly, total down time of 23 minutes. Drive is now replicating the other drive in the raid array. Indications are that it will take another seven hours (it's been going for 45 minutes) so my projection of 6-8 hours seems to be spot on. System may be a bit slow during - [Maintenance April 7th 2:00AM ~2:30AM](https://www.eskimo.com/2024/04/maintenance-april-7th-200am-230am/) - We will be off line, the entire network, for up to about half an hour, starting around 2AM Sunday morning, to replace a failing drive in the machine which is also acting as a router at present. This drive is part of a RAID array so all data is duplicated and none will be lost. - [Mail Back to Normal](https://www.eskimo.com/2024/04/mail-back-to-normal/) - Today after some three hundred updates, the original SPF checker I was using, the phython3 version, still was not working, so I installed the perl versions of policyd. I don't really like perl as I don't find it very readable relative to python, but presently it is working. I also found the clamav virus check - [Eskimo Site Status](https://www.eskimo.com/2024/04/eskimo-site-status/) - Ubuntu is back. Sorry it took so long but many snags along the way. Our old web server is running without a Network Manager because Ubuntu clods broke it. I have to set the network interface manually after a boot. Inuvik is also broken because Ubuntu 24.04 engineers mistakenly put new libs in the main - [Ubuntu Down](https://www.eskimo.com/2024/03/ubuntu-down-2/) - Ubuntu is down owing to a failed update yesterday. I attempted to restore from backups but during the update procedure things went south again. I am trying again, if this fails I will re-install. In the meantime please use any other machine, debian, mint, zorin, mxlinux are all similar debian derived systems. - [New Server and Social Media Sites](https://www.eskimo.com/2024/03/new-server-and-social-media-sites/) - Still working on getting the new server resurrected. Having difficulties getting the RAID to auto-assemble at boot time and haven't figured out why. And while investing $300 on a hardware controller could solve that issue, some pointed out to me a reason to avoid that if possible and that is the hardware controller uses - [Outage Difficulties](https://www.eskimo.com/2024/03/outage-difficulties/) - First some background.. Our old web server was over burdened, particularly when it came to RAM. Also it booted off a rotary disk and only the mariadb was on nvme memory thus it was slow to boot. Linux likes having a lot more RAM than it needs because it uses any not required by - [Outage](https://www.eskimo.com/2024/03/outage-7/) - I still have not restored some services, particularly the newer web server upon which a few customers sites are on. I've been at this constantly for about 33 hours without sleep and I have reached my physical endurance limit. It was necessary to completely re-install one systems operating system as something in the boot - [Maintenance Outage](https://www.eskimo.com/2024/03/maintenance-outage-4/) - I plan to take most services off line between 11pm-12pm tonight, March 12th, 2024, for about twenty minutes to replace a failed network interface card in one of the servers. Because this server provides some disk storage for most of the machines via NFS this means most services will be unavailable during that interval. - [Outage](https://www.eskimo.com/2024/03/outage-6/) - The machine that is doing double duty as a webhost and router is ill. I have it running in a crippled state but if it reboots it will not come up automatically. For those who are interested in the technical details and may have encountered this before and can provide some hints, what - [Server Issues](https://www.eskimo.com/2024/02/server-issues/) - We had an issue with the web server eating itself after an upgrade introduced a library that conflicted with a library I had compiled in order to enable http2 protocol before Ubuntu included it in their distribution. I ended up having to restore this server from backups and bring it forward again, removing - [Carl Jung](https://www.eskimo.com/2023/11/carl-jung/) - “The spirit of evil is fear, negation, the adversary who opposes life in its struggle for eternal duration and thwarts every great deed, who infuses into the body the poison of weakness and age through the treacherous bite of the serpent; he is the spirit of regression, who threatens us with bondage to the mother - [SSH Key Vulnerability](https://www.eskimo.com/2023/11/ssh-key-vulnerability/) - A new ssh key vulnerability has been found affecting RSA keys which are the default in many older Linux implementations. I strongly suggest generating a new ed25519 key using the command ssh-keygen -t ed25519 and remove any RSA keys you may be using. To do this remove relevant lines from your ~/,ssh/authorized_keys file. RSA - [Outage](https://www.eskimo.com/2023/10/outage-5/) - Sorry it took me so long to get this back up. I thought the machine that was serving as router had crashed, it had not, however, the device driver for the ethernet card had unloaded. When I tried to load it it told me invalid argument. Odd since I hadn't changed any arguments, in fact - [6.1.57 Kernel Borked](https://www.eskimo.com/2023/10/6-1-57-kernel-borked/) - I built a new 6.1.57 kernel last night and it did not work. This caused an outage this morning after the machine playing router rebooted. We are back on 6.1.54. - [Stability or the Lack Thereof](https://www.eskimo.com/2023/09/stability-or-the-lack-thereof/) - This morning I finally figured out the source of the most recent instability (since we changed out the bad NIC card). We kept having this incidence where I'd go to the co-lo, thought I had everything working but in minutes or hours or sometimes a few days it would just stop talking to the Internet. - [Outage](https://www.eskimo.com/2023/09/outage-4/) - Sorry for the down time. Our second router to die in two weeks died yesterday afternoon. At first I thought it was something I did but then after restoring from a backup when it was working it still didn't. The vendor had changed the software in such a way that the translation of firewall rules - [While we were not able to add a Diaspora at this time, We DID Add a Mastodon Instance](https://www.eskimo.com/2023/09/while-we-were-not-able-to-add-a-diaspora-at-this-time-we-did-add-a-mastodon-instance/) - You can view our latest social media instance either by using Web Apps on our main web site https://www.eskimo.com/ and selecting Mastodon, or you can go there directly https://mastodon.eskimo.com/. Mastodon is a Twitter/X link interface but unlike Twitter, Mastodon is part of the Fediverse, like Friendica and Hubzilla, which consists of tens of - [Diaspora](https://www.eskimo.com/2023/09/diaspora/) - Someone asked for us to setup a Diaspora node here, unfortunately because of openssh and ruby versions, we can not do this at present. In theory this will be possible after the next Diaspora release which will be ported to ruby3 and openssh 3.x. Ubuntu 22.04 ships with OpenSSL version 3, with no option of - [Interruption Tommorrow](https://www.eskimo.com/2023/09/interruption-tommorrow/) - Sometime tomorrow there will be a brief interruption of Eskimo North's services including https://friendica.eskimo.com/, https://hubzilla.eskimo.com/, https://nextcloud.eskimo.com/, shell servers, e-mail, and web hosting, lasting from one to ten minutes while our Internet backbone connection is moved from a 100mb/s port to a 1Gb/s port. I do not know what time this will occur and I do - [Router Speed+](https://www.eskimo.com/2023/09/router-speed/) - Also forgot to mention we'll also be supporting Jumbo Frames (4500 byte frames) if you have a 1Gbps or better connection and your provider supports them. This will help with performance for things like transfer of large files or streaming. - [DNS Problem Solved](https://www.eskimo.com/2023/09/dns-problem-solved/) - I'm not really sure what I broke but it was something I misconfigured on our new router. - [Mail / DNS Issues](https://www.eskimo.com/2023/09/mail-dns-issues/) - We are having DNS issues all over the place, and even my home machine using non-eskimo serves is getting DNS errors, and no updates queued for Ubuntu seem to be relevant. It is these DNS issues that are causing problems with e-mail. I am working on resolving, I have one incoming mail server working by - [Unplanned Disruption](https://www.eskimo.com/2023/09/unplanned-disruption/) - I apologize for the disruption today. Our gateway router crapped out entirely as in not crashed but dead as a door nail. I had already purchased a new unit with a lot more CPU and memory to replace it but owing to my unfamiliarity it took some time to get it configured and operational. But - [Kernel Upgrade 11PM Sept 1st](https://www.eskimo.com/2023/09/kernel-upgrade-11pm-sept-1st/) - Going to reboot all or most machines at 11pm to upgrade kernels. This will affect all services, paid and unpaid. With the exception of yacy which will take about 45 minutes, other services should not be down longer ten minutes. - [Reboot Iglulik Short (hopefully) Interruption 10PM tonight](https://www.eskimo.com/2023/08/reboot-iglulik-short-hopefully-interruption-10pm-tonight/) - I had a problem with Iglulik firewall because quotad would not honor my configuration and listen to the configured port so that forced me to leave all ports open to other machines here. Not a desirable situation for security. I filed a bug report and amazingly enough the Ubuntu folks (Canonical) got back - [Unintended Interrupts](https://www.eskimo.com/2023/08/unintended-interrupts/) - Ubuntu spontaneously booted around 13:30 today. The logs indicated an intentional reboot but I didn't request one, so don't know if a signal went to the wrong process (systemd) or exactly what happened. Then around 6pm the load on inuvik which hosts one of our web servers and debian and manjaro, went through - [Debian](https://www.eskimo.com/2023/08/debian-14/) - Later this evening, I am going to take Debian down for around 15 minutes in order to expand the size of the virtual machine image to in turn allow for an expansion of the file system as the root file system is 92% full. - [Shell Servers](https://www.eskimo.com/2023/08/shell-servers-3/) - All of the shell servers are now available via ssh, vnc, desktop and terminal under web-apps on our website (initial login screen public, public), and most should also be available via rdp, there a couple I've recently re-installed (Manjaro, MxLinux) that I have not tested rdp yet. And speaking of re-install, Manjaro and MxLinux are - [NIS Broken](https://www.eskimo.com/2023/08/nis-broken/) - The NIS system failed tonight, NONE of the ypbind can bind to any server on any machine. I do not know if an update broke this or just what, still investigating. - [Some Limited Reboots](https://www.eskimo.com/2023/08/some-limited-reboots/) - At around 11pm this evening, I will be rebooting some machines, some virtual private servers and some shell servers, to enable shared memory in all of them. What this does is that a kernel routine scans the machines, and if it finds two identical pages of memory, it maps them both into one - [Host and Terminal](https://www.eskimo.com/2023/07/host-and-terminal/) - The web app was working when I went to bed but broken when I got up, so I'm trying to figure out what went wrong over night. - [Web Based Terminal or Desktop](https://www.eskimo.com/2023/07/web-based-terminal-or-desktop/) - You can now reach a shell server terminal or remote desktop from our website without having to go through the guacamole login. Just use Web-Appls->Terminal or Desktop, it will display a selection of all the shell servers Terminal or Desktop, click on one and login. With some servers there are multiple desktops available, not all - [Kernel Upgrades Tonight 11pm-midnight](https://www.eskimo.com/2023/07/kernel-upgrades-tonight-11pm-midnight-2/) - I will be performing kernel upgrades tonight upgrading from 6.1.37 to 6.1.41. This will affect all of Eskimo North's services, free and paid including e-mail, shells, web hosting, and virtual private servers. It will also affect https://friendica.eskimo.com/, https://hubzilla.eskimo.com/, https://nextcloud.eskimo.com/, and https://yacy.eskimo.com/. The downtime for most services will be less than ten minutes. Yacy - [Fedora 39 -](https://www.eskimo.com/2023/07/fedora-39/) - I have graphical environments via x2go, vnc, or web now working on all machines EXCEPT Fedora. I haven't got rpd working everywhere yet. But since rdp relies on Xvnc, it made since to focus on VNC first. And I haven't setup sound on all yet. With Fedora I have a problem, I need an inetd - [Zorin - Back Up](https://www.eskimo.com/2023/07/zorin-back-up/) - Zorin is back up with a graphical environment working again. - [Zorin - Temporarily Down](https://www.eskimo.com/2023/07/zorin-temporarily-down/) - I am re-installing Zorin because the set of installed software has too many conflicts to easily resolve. So it will be down temporarily while this is performed. I am trying to fix the graphical environment on all servers, most work but some don't and Zorin was one of those that didn't. Having multiple greeters, - [Debian](https://www.eskimo.com/2023/07/debian-13/) - I took Debian down for about 5 minutes starting at 2:52 PM Pacific in order to increase the size of the root partition which was running out of space. I recently compiled gcc 13.1 and new kernels and source for both ate up a lot of space. - [Mail Capacity](https://www.eskimo.com/2023/07/mail-capacity/) - I had to re-install Dovecot over the weekend because something went wrong internally that I was unable to identify. I installed with mostly default settings, changing on a few things I knew were necessary for our installation. This morning it reached some of the internal limits set. I've addressed those issues today. - [Ubuntu Spontaneously Booted 10:08AM Today](https://www.eskimo.com/2023/07/ubuntu-spontaneously-booted-1008am-today/) - The only thing I could find in the logs is that the guest tried to use a VT-x feature not supported by the host CPU, but the host CPU DOES support VT-x and the guest is configured to copy the host CPU configuration so it should not use anything not supported by the host - [New Features](https://www.eskimo.com/2023/07/new-features/) - There are some New Features I'd like to make you aware of: On our website, https://www.eskimo.com/, at the far right on the menu bar there is new button "Issues". Press on this button to see the latest news, whether it be outages or new features or something else. The new NIS server is Linux based - [Summary of What Happened](https://www.eskimo.com/2023/07/summary-of-what-happened/) - Over the past 24 hours we've had three server issues. First, some changes to the network were necessary to accommodate a new router that will be replacing our existing router. The new router does not do port based network address translation which we previously used to provide services from two different servers on a - [Mail Server is back UP](https://www.eskimo.com/2023/07/mail-server-is-back-up/) - The mail server is back up. I don't know what happened to Dovecot but it was generating errors I've never seen before, errors referring to "extra groups" and I was not able to gain any useful insights from Google, Bing, reboot didn't help, debug didn't offer any useful insights, etc. Finally I purged - [Mail](https://www.eskimo.com/2023/07/mail-20/) - The main NIS server is backup but the mail server is still not operating correctly complaining about "extra groups", a message I've never seen before and not having any luck googling. I suspect some old auth info is being cached somewhere but damned if I know where so I've purged the whole thing and am - [Authentication Problems](https://www.eskimo.com/2023/07/authentication-problems/) - I re-configured the NIS system to have a Linux based NIS master last night. It was working last night but the main NIS server crashed during the night and although it should still work from the slave servers it is having problems. So I'm going down to the co-lo facility to restore the crashed server - [Router Replacement](https://www.eskimo.com/2023/07/router-replacement-4/) - The router replacement that was planned for Saturday is probably going to be delayed until Monday. In the meantime, the old SunOS machine is being retired and I am setting up a new NIS master on a Linux machine. Because of differences in the way Linux and SunOS NIS are implemented, SunOS puts - [Network Work this Saturday](https://www.eskimo.com/2023/07/network-work-this-saturday/) - This will impact ALL of eskimo.com's services. If all goes well I am going to replace our router this Saturday evening. It may take a bit to get it to work as the interface is quite different than the old and I've got some concerns about configuring the network side of things. After - [Work Tonight and Ongoing](https://www.eskimo.com/2023/07/work-tonight-and-ongoing/) - This will happen sometime this evening, probably after 10PM but can't be exact because of other commitments with unknown time frames. I'm going to be taking the new server, Inuvik, down for about an hour or so to install some adapters for the drives I replaced last week. The new drives have repurposed pin3 of - [Downtime](https://www.eskimo.com/2023/07/downtime/) - Is taking me longer than I had hoped because Linux duplicated an entry in the /etc/mdadm/mdadm.conf file (a bug that it does once in a while), and when it does this it won't boot up all the way. It gets to where it's time to assemble the RAID device but since the raw devices - [Maintenance Work Tonight](https://www.eskimo.com/2023/07/maintenance-work-tonight-3/) - Just a reminder that I am going to bring a server down to replace two failed hard drives tonight. This will affect https://friendica.eskimo.com/, https://hubzilla.eskimo.com/, https://eskimo.com/ (but NOT https://www.eskimo.com/), and the shell servers Manjaro.eskimo.com and Debian.eskimo.com. Estimated downtime is about an hour and probably will start between 10:30 and 11:30 just depending upon when I - [Web Restored](https://www.eskimo.com/2023/07/web-restored/) - Everything is back in service in terms of the web service here. For some reason when I saved all mysql databases prior to changing the innodb table options, it did not save mysql (the database with all the permissions and grant tables) properly. So I had to ressurrect a backup, dump the mysql database - [Partial Web Outage](https://www.eskimo.com/2023/07/partial-web-outage-2/) - Database is taking longer than I expected to recover. The issue was caused by the fact that innodb-file-per-table was not set to true. This caused all the innodb tables to be stored in the system file. The problem with this is that space is never recovered and the file grows until the disk - [Partial Web Outage](https://www.eskimo.com/2023/07/partial-web-outage/) - We have a partial web outage now. This was caused by a misconfiguration of the mariadb database on the new server that caused it to eat itself. I am in the process of correcting that configuration issue and restoring the server. Unfortunately, owing to the size of the database, this will take some - [Wednesday Evening / Night](https://www.eskimo.com/2023/07/wednesday-evening-night/) - Sometime Wednesday evening or night I am going to take the new machine, Inuvik.eskimo.com down, to replace the two drives I had intended to use for backup space as both have suffered a head crash at some point and have media errors that are too numerous to map-out with two spare tracks provided. Because - [Database](https://www.eskimo.com/2023/07/database/) - Go ahead and make changes, got more work to do before ready to switch over. - [Database Migration](https://www.eskimo.com/2023/07/database-migration/) - I am migrating the mariadb database (you may know it as mysql) from the old server to the new server tonight. Please do not make posts or other database changes that you can not afford to lose tonight. There is 117GB in the database, so this is not something that can happen instantly. - [Unplanned Outages](https://www.eskimo.com/2023/07/unplanned-outages/) - Sorry for the unplanned outages of Debian and Manjaro tonight. As I mentioned earlier, I had moved these two to the new machine primary because I have to build kernels on both, on Debian because Debian signs it's kernels and won't load an unsigned kernel built on a non-Debian machine, on Manjaro also - [Debian, Majaro, Kernel Upgrades](https://www.eskimo.com/2023/07/debian-majaro-kernel-upgrades/) - Manjaro.eskimo.com is now operational again, including Mate Desktop and workspace switcher. Debian.eskimo.com is now fully upgraded to Bookworm. We will be doing a kernel upgrade Saturday July 8th starting at 11pm. This upgrade will require reboots of all servers and hence interruption of all services, paid and free, including mail, web - [Debian Upgrade - Still In the Works - And More...](https://www.eskimo.com/2023/07/debian-upgrade-still-in-the-works-and-more/) - The upgrade from Bullseye to Bookworm is turning out to be a MUCH larger upgrade than was the upgrade from Buster to Bulleye, more than 8200 packages are being updated so it is taking much longer than expected. Then, when it is finished, I am going to try to move this virtual domain off the - [Upgrading Debian](https://www.eskimo.com/2023/07/upgrading-debian/) - I am planning on upgrading Debian from Bullseye 11 to Bookworm 12 tonight. This will require a reboot at the end of the upgrade process. - [No Service Interruption Tonight](https://www.eskimo.com/2023/07/no-service-interruption-tonight/) - After a bit of experimentation, I found having the CPU voltage follow the CPU frequency resulted in about 40 watts of idle power savings but occasionally caused CPU data errors. The latter is not acceptable so that part of the plan has been eliminated. I will be installing the new server tonight though it - [Maintenance Work July 1st 10pm-2am](https://www.eskimo.com/2023/06/maintenance-work-july-1st-10pm-2am/) - I am planning on doing some major maintenance work tomorrow. First will be the installation of our newest server. This should not disturb existing services but there is always the potential for operator error. The second thing I will be doing, and this WILL result in downtime for various services, is to take - [Russian Hacking](https://www.eskimo.com/2023/06/russian-hacking/) - You may have heard on the news items about Russians hacking into various government and business organizations, The Department of Energy for example. Here, I have seen an approximately 3x increase in hacking activity in the last month, the majority of it in the form of e-mail phishing scams, and a non-trivial percentage - [Mail, Web, New Server](https://www.eskimo.com/2023/06/mail-web-new-server/) - Web and SSL Mail were temporarily interrupted earlier this evening as I replaced the SSL certificate for *.eskimo.com that was due to expire on July 6th. Finally got the new server to boot consistently off of RAID. Main issue was EFI system disk needed to be a physical direct device and not RAID or a - [Mail Trouble](https://www.eskimo.com/2023/06/mail-trouble/) - Mail is fixed now. - [Kernel Upgrades Completed](https://www.eskimo.com/2023/06/kernel-upgrades-completed-21/) - Kernel upgrades completed successfully. All services, nfs mounts, nis bindings checked and verified. - [Kernel Upgrade Tonight 11pm-12pm](https://www.eskimo.com/2023/06/kernel-upgrade-tonight-11pm-12pm/) - I will be performing a kernel upgrade requiring reboot of all of our servers starting at 11PM. If all goes well we should finish by 11:30PM with boots and midnight by checks to make sure all services properly started, NIS bindings and NFS mounts properly completed, etc. This will be to kernel 6.1.33. - [New Server](https://www.eskimo.com/2023/06/new-server/) - I've got the new server physically together. It posts, boots, and everything tests out good. Now I've got to load an OS in order to find optimal settings for performance. Someone suggested a good name a while back and I've misplaced it. If you know who you are and could resend, please do. - [Microsoft Issue Revealed an Issue with Our Mail Servers](https://www.eskimo.com/2023/05/microsoft-issue-revealed-an-issue-with-our-mail-servers/) - Microsoft's outlook mail service, servers, starting with *.protection.outlook.com apparently added some servers without adding an SPF record for them. This revealed a problem with our spf client. It is supposed to not reject but only flag the incoming mail and then it should go to your spam box. Unfortunately even though it was - [Today's May Outage](https://www.eskimo.com/2023/05/todays-may-outage/) - Today's May 30th between 1:45 PM Pacific Daylight Time and 4:10 PM Pacific Daylight Time, outage was caused by a circuit breaker failure at the co-location facility where our equipment is co-located. Our equipment did not lose power but their core routers did. This affected all of our paid and free services, e-mail, - [Reboots Completed](https://www.eskimo.com/2023/05/reboots-completed-16/) - Reboots completed, all servers and services are up except manjaro and it's still being fixed since being reinstalled, some things putting up a good fight. - [Kernel Upgrades and Other Happenings](https://www.eskimo.com/2023/05/kernel-upgrades-and-other-happenings/) - Saturday evening starting at 11PM we will be performing a kernel upgrade of all of our servers to version 6.1.30. It has some significant fixes for bugs that, while they haven't bitten us yet, could. I expect reboots to be completed by 11:30PM, various services that don't restart properly and NFS and NIS - [Spam Filtering Change](https://www.eskimo.com/2023/05/spam-filtering-change/) - The majority of spam filters here put spam in a folder named "spam" rather than rejecting it outright. However, there are two types of spam that I manually block when discovered, virii and phishing scams. Virii are various computer viruses, especially ransomware. When I find a server is infected, I block mail from - [Kernel Upgrades Completed](https://www.eskimo.com/2023/05/kernel-upgrades-completed-20/) - Kernel upgrades completed by 12:30, various services that systemd failed to start properly all corrected by 12:46. Everything is back in service at Eskimo. - [Kernel Upgrades May 13 11pm PDT (GMT-0700)](https://www.eskimo.com/2023/05/kernel-upgrades-may-13-11pm-pdt-gmt-0700/) - I am planning a kernel upgrade tonight Saturday May 13th starting at 11PM. This will affect all of Eskimo North's services: web hosting, shell accounts, e-mail, virtual private servers, and our free Fediverse services https://friendica.eskimo.com/, https://hubzilla.eskimo.com/, https://nextcloud.eskimo.com/, and https://yacy.eskimo.com/. Provided all servers boot correctly no service with the exception of yacy should - [Nextcloud Upgrade Experiment](https://www.eskimo.com/2023/05/nextcloud-upgrade/) - I am going to take the web server database offline briefly later this evening to make a backup before trying an experiment with Nextcloud. In the past I have gotten the external_authentication module to work for versions it was not designed simply by altering the definitions file associated with it. I am going to - [New App on Ubuntu Shell Server](https://www.eskimo.com/2023/05/new-app-on-ubuntu-shell-server/) - I've installed a new application for your use on the Ubuntu shell server if you access it visually via x2go, rdp, vnc, or via the website. This new application is called LibreWolf and is under the Internet menu on Gnome or Mate Desktops. LibreWolf is an independent fork of Firefox with enhanced security - [cmatrix](https://www.eskimo.com/2023/05/cmatrix/) - Try expanding a terminal, preferably one that handles ansi colors, to full screen and then in ubuntu type, "cmatrix". - [Manjaro Update](https://www.eskimo.com/2023/04/manjaro-update/) - I found the Manjaro XFCE spin installer works, including with manual partitioning. Sad Mate doesn't but not surprised. It will probably take 4-5 hours to get everything configured but should be working again later this evening. - [Manjaro May Be Down for a While](https://www.eskimo.com/2023/04/manjaro-may-be-down-for-a-while/) - Manjaro may be down for a while, possibly until the next distro comes out. The existing installation was hurt by installing gcc-13.1 which corrupted some of the library files. I restored from backups but the Manjaro update system won't let you update a system that is too old, so that didn't work. - [Kernel Upgrades Completed, EXCEPT Manjaro](https://www.eskimo.com/2023/04/kernel-upgrades-completed-except-manjaro/) - Manjaro failed to boot with the new kernel and I chased the issue to my recent installation of gcc-13.1 on this machine. It changed some system libraries in a way that broke them. I am restoring from backup, then will bring the machine up to date, then will install the new kernel, all - [Kernel Upgrade 4/29 11PM-Midnight](https://www.eskimo.com/2023/04/kernel-upgrade-4-29-11pm-midnight/) - We will be doing a kernel upgrade of all servers tonight starting at 11pm. If all goes well it should conclude by 11:30, if not and we have to make a trip to the co-location facility, then 12:30. No individual service with the exception of yacy should be down for more than about ten - [Igloo Crashed](https://www.eskimo.com/2023/04/igloo-crashed/) - Igloo, one of the physical hosts that among other things hosts the mail spool, crashed earlier this afternoon. I was compiling a new version of GCC (13.1) using eight cores and it just got too hot. I stress test these machines thoroughly when I initially turn them up but over time CPUs suffer - [Eskimo-Aleut languages](https://www.eskimo.com/2023/04/eskimo-aleut-languages/) - I thought this was pretty cool: Eskimo-Aleut languages - [Kernel Upgrades Completed](https://www.eskimo.com/2023/04/kernel-upgrades-completed-19/) - Kernel upgrades are completed. Sorry it took longer to get some of the virtual private servers up than normal. One of the physical servers failed to come back up into the normal run level, it booted but systemd hung (which unfortunately it sometimes does though this is not frequent on the physical servers). - [Kernel Upgrade Saturday 4/15/2023 11PM](https://www.eskimo.com/2023/04/kernel-upgrade-saturday-4-15-2023-11pm/) - I will be performing a kernel upgrade tonight starting at 11pm, expecting to finish by 11:30pm. Expected downtime for most services less than ten minutes, yacy.eskimo.com may take as much as 45 minutes owing to the rebuilding of the database after reboot. This upgrade will affect all of Eskimo North's services, shell accounts, - [Fax Machine Repaired](https://www.eskimo.com/2023/04/fax-machine-repaired/) - The fax machine has been repaired so you can send me faxes now if that is your preferred method of communicating. - [Kernel Upgrade Completed - Except Manjaro](https://www.eskimo.com/2023/04/kernel-upgrade-completed-except-manjaro/) - Kernel upgrades are all completed except for Manjaro (did not have time to finish compiling), it will be done later this morning. - [Kernel Upgrades Sat April 1st 11PM](https://www.eskimo.com/2023/04/kernel-upgrades-sat-april-1st-11pm/) - Will be attempting an upgrade to kernel 6.1.22 tonight starting at 11pm. Expect to be finished at least with the reboots by 11:30pm and should have all NIS and NFS dependencies restored by midnight if not sooner. Interruptions of individual services should not exceed about ten minutes with the exception of the Yacy search - [Kernel Upgrades Complete](https://www.eskimo.com/2023/03/kernel-upgrades-complete/) - Kernel upgrades are completed. Ran into a few minor software issues on some machines but all are resolved now. - [Kernel Upgrade Saturday March 18th 11PM](https://www.eskimo.com/2023/03/kernel-upgrade-saturday-march-18th-11pm/) - We will be upgrading our server kernels on Saturday the 18th starting at 11pm Pacific Daylight Time. Expected downtime is about ten minutes per service with the exception of Yacy which can take half an hour or longer to rebuild database after a reboot. This will affect all of our services, free and - [Manjaro, Kernel Issues, etc.](https://www.eskimo.com/2023/03/manjaro-kernel-issues-etc/) - I had announced previously plans to retire manjaro as it was not getting any use, but in the last three months three people have started using it, therefore I do not intend to retire it at this point. On another note, the 6.1 kernel has been declared to be a "long-term" kernel, which - [CGI's in Virtual Domains / Web Hosting Packages](https://www.eskimo.com/2023/03/cgis-in-virtual-domains-web-hosting-packages/) - A customer had an issue with a CGI that was part of his website not being found. I found a configuration error in our website where the CGI directory was defined globally rather than just for our website causing the server to look in the wrong place for CGI scripts. This is now - [VPS's Back Up](https://www.eskimo.com/2023/03/vpss-back-up/) - Machine is repaired and back in place, everything should be functioning at full speed again. - [Virtual Private Server vps1, 2, 4, 5, and 8](https://www.eskimo.com/2023/03/virtual-private-server-vps1-2-4-5-and-8/) - Got the machine repaired, taking it back to the co-lo, should be there in about 40 minutes. The old fan was doing about 3RPM, new fan worked fine so not a controller issue. - [ice not booting](https://www.eskimo.com/2023/03/ice-not-booting/) - Ice, the physical host with a number of virtual private servers is refusing to boot. This is usually a Poettering thing where systemd either gets stuck shutting something down or starting something up but requires physical intervention so I will need to drive down to the co-lo facility which takes approximately 1/2 hour to - [Kernel Upgrades Tonight 11pm-11:30PM](https://www.eskimo.com/2023/03/kernel-upgrades-tonight-11pm-1130pm/) - Upgrading Eskimo North server kernels to 6.1.16 this evening starting at 11PM. Should be finished by 11:30PM providing no physical server hangs. This involves all of the servers so shell servers, mail servers, web server, virtual private servers, will all each be down about 3-10 minutes depending upon the machine complexity. The - [HTTPS and HTTP2](https://www.eskimo.com/2023/03/https-and-http2/) - A relatively small portion of the domains hosted here has https and http2 enabled by an encryption certificate. The primary advantage of https is so all of the data flowing between the web server and the web client is encrypted so can't easily be monitored by a third party. A secondary advantage is - [Phishing Scams - Expired Passwords, etc.](https://www.eskimo.com/2023/03/phishing-scams-expired-passwords-etc/) - Just a reminder, we do NOT use password aging here so you will NEVER get an e-mail from us telling you that your password has expired. If you receive such a mail it is from a third party attempting to look like us in an attempt to get your authentication information from you. - [Working on New Server](https://www.eskimo.com/2023/03/working-on-new-server/) - Some time ago I started a new build for a new server based upon the i9-9900k. I purchased the motherboard, but subsequent upgrades to Linux improved performance to where it was not necessary. I was going to go with the i9-9900k because it was economically doable while the next generation really wasn't. I do - [Kernel Upgrade 3/3/23 11PM](https://www.eskimo.com/2023/03/kernel-upgrade-3-3-23-11pm/) - Planning on upgrading all the systems to Linux 6.1.14 Friday 3/3/2023 starting at 11PM. If all goes well, should be completed by 11:30PM, if not it maybe another 1-1/2 hours (if a physical host does not reboot properly). This will effect all services including shell accounts, web hosting packages, e-mail, etc. It also - [Outlook E-mail Outage](https://www.eskimo.com/2023/02/outlook-e-mail-outage/) - Some of our customers have had e-mail to sites hosted by Outlook fail and get bounced with something on the order of cannot lookup sender address or some such. Microsoft claims troubles leaving there fine, i.e., they have no issue but Down Detector says otherwise and I've had reports from other sites experiencing - [Denial of Service](https://www.eskimo.com/2023/02/denial-of-service-6/) - Our web server is presently undergoing what is known as a "Slow and Low" denial of service attack. In this type of attack someone initiates a large number of connections from sources which are very slow. This limits our servers ability to finish a connection and so it eats up all available connections. To - [Flash Content](https://www.eskimo.com/2023/02/flash-content/) - If you wish to play the flash games in the Games section of our website, Ybbored.com, Defender-games.net, or any other flash site, now that Adobe has discontinued flash, you can do so with Firefox and an add-on plugin called "Flash Player 2022". This works pretty much with any OS that Firefox works with, including - [Downtime Friday 11PM - 1AM](https://www.eskimo.com/2023/02/downtime-friday-11pm-1am/) - It took me nearly two hours to get everything back up after booting into the new kernels tonight. The issue revolves around some recently added ufw rules to improve security. Even though I have explicit rules permitting machines that need to see each others portmapper (rpcbind) to do so, they aren't working and - [Web Based Terminal and Console Access](https://www.eskimo.com/2023/02/web-based-terminal-and-console-access/) - The existing web based terminal and console access is broken because the Guacamole installed requires some features not present in OpenSSL 3.1, so I will have to compile and install a newer version. I hope to get this corrected later this evening. - [Tonight's Outage](https://www.eskimo.com/2023/02/tonights-outage/) - Tonight's outage was not the result of a hardware or software error, rather the result of an operator error. I had built a new kernel and had intended to try it on my workstation before deployment but I also had a window open on the main file server because that is where I store - [Kernel Upgrade](https://www.eskimo.com/2023/01/kernel-upgrade-3/) - I made an error when I configured the last kernel. While 6.1.7 does appear to be stable AND it appears to have fixed the long standing NFS bug for which I enabled the extra debugging, I accidentally compiled it with premption which I do not want on a server as it adds additional context - [Fedora Messed...](https://www.eskimo.com/2023/01/fedora-messed/) - Fedora is messed beyond repair, going to have to re-install. It will work kind of for non-graphical sessions, but I can't get any display manager to work properly. - [Default PHP is now 8.0](https://www.eskimo.com/2023/01/default-php-is-now-8-0/) - We have changed the default PHP from version 7.4 to version 8.0 now. Everything from 5.6 to 8.2 is available, anything older than 8.0 is no longer receiving security updates so should be avoided IF POSSIBLE but some older applications may not work with 8.0. It is best to update those applications - [Yacy is Down](https://www.eskimo.com/2023/01/yacy-is-down/) - The Yacy search engine "https://yacy.eskimo.com/" is down at present. Something is causing it to eat up all available memory and then die. I have opened a trouble ticket after doing some preliminary troubleshooting and am waiting upon a response. - [Virtual Private Servers](https://www.eskimo.com/2023/01/virtual-private-servers-5/) - I am going to have to reboot virtual private servers later this evening because iptables on the physical host is messed up, ufw reset isn't working, and now I've locked myself out of the machine altogether. - [Kernel Upgrades Jan 20 11PM PST (GMT-0800)](https://www.eskimo.com/2023/01/kernel-upgrades-jan-20-11pm-pst-gmt-0800/) - We will be upgrading to the 6.1.7 kernel this evening at 11pm. Because KASAN caused issues with some of our servers, some would not boot with it, some were slower, we will only be putting it on two NFS servers that have been problematic. I believe however that 6.1.7 has already addressed the bug - [I'm Again Available via Phone / E-mail.](https://www.eskimo.com/2023/01/im-again-available-via-phone-e-mail/) - Comcast has completed their work and I'm back online and phones are operational. - [CentOS-Stream, MxLinux](https://www.eskimo.com/2023/01/centos-stream-mxlinux/) - CentOS Stream has not been used since last March and MxLinux since last July, so I am going to ask if anyone wants to continue these or should I discontinue them and save the resources? - [Customer Service on the 18th](https://www.eskimo.com/2023/01/customer-service-on-the-18th/) - Comcast will be doing work on the 18th that will take my telephone system, fax, and Internet connection away from my home office during part of that day. I do not know what time or how long. I hope to have some internet access via my tablet but it is very limited. If you - [vps1-6. vps8 Physical Server Boot Failed](https://www.eskimo.com/2023/01/vps1-6-vps8-physical-server-boot-failed/) - The machine which hosts vps1-6 and 8 failed to reboot properly. I am headed down to the co-location facility now to troubleshoot. I should arrive there about 12:10AM. - [Server Hang - Kernel Upgrades 1/12/2023](https://www.eskimo.com/2023/01/server-hang-kernel-upgrades-1-12-2023/) - The server that provides home directories and also shared web services as well as two virtual private servers hung with the CPU stall bug even though we're about 12 incarnations down the road from where it first appeared. We know this seems to be a use after free error in the kernel but kexec - [Kernel Upgrade Aborted](https://www.eskimo.com/2023/01/kernel-upgrade-aborted/) - Tonight's kernel upgrade is aborted because it will not build with the debugging options the developers wanted me to include so I've sent the compiler errors back to them and will resume when I have a fix. - [Kernel Upgrade Tonight Sunday 11pm](https://www.eskimo.com/2023/01/kernel-upgrade-tonight-sunday-11pm/) - I am going to be upgrading the kernels on only the physical servers tonight in order to turn on some additional debugging options to help the developers chase down an error in the NFS code that is causing issues for us. Apparently this bug only occurs when you have a mix of NFSv3 and - [Kernel Upgrades](https://www.eskimo.com/2023/01/kernel-upgrades-12/) - The kernel upgrades were successful and relatively trouble free. We are now operating on Linux 6.1.3, which was released yesterday. - [Kernel Upgrades 1/6 11PM PST (GMT-0800)](https://www.eskimo.com/2023/01/kernel-upgrades-1-6-11pm-pst-gmt-0800/) - Planning to upgrade to a 6.1.2 kernel Friday 1/6 at 11pm Pacific Time. The present kernel, 6.0.15 has a nasty bug where it locks hard, no kernel dump, no auto reboot, no magic sys request key, only power cycling the affected machine restores service. The inability to get a kernel dump makes this bug - [Virtual Private Servers](https://www.eskimo.com/2023/01/virtual-private-servers-4/) - The machine that hosts many of the virtual private servers, ice, is down. I am headed down to the co-lo to find out why. - [Fax Down](https://www.eskimo.com/2022/12/fax-down/) - My facsimile machine ran out of paper today and the spare paper I had all got wet from the tree limb through my roof, so it will be nonoperational until I can get more paper. Please e-mail or call instead. - [PHP Upgrade](https://www.eskimo.com/2022/12/php-upgrade-2/) - We are going to attempt to change the default version of PHP to PHP 8.0.26 tonight. The last time I attempted this there were too many applications that did not work but since that time several new releases have come out so we are going to try again. Just a reminder, if your - [6.0.15 Upgrade - Still Unstable](https://www.eskimo.com/2022/12/6-0-15-upgrade-still-unstable/) - Within half an hour of booting 6.0.15, the server which holds home directories and much shared disk space spontaneously booted. While still not good, this is better than the behavior of 6.0.12 which locked up and did not reboot. - [](https://www.eskimo.com/2022/12/6911/) - [Kernel Upgrades, Mail to Gmail, Dial Access](https://www.eskimo.com/2022/12/kernel-upgrades-mail-to-gmail-dial-access/) - Two customers reported mail failing to gmail, with the bounces indicating improper SPF records, however, I changed the SPF record along with the IP of the mail server at the same time AND pushed out the changes manually to all of our name servers. Gmail was caching old data. I have since tested and - [Kernel Upgrades NOT HAPPENING Tonight](https://www.eskimo.com/2022/12/kernel-upgrades-not-happening-tonight/) - I still can't get to my car and by the looks of our street it won't happen tonight so the kernel upgrade is going to be put-off until at least Saturday night. Hopefully the machines will make it another day without seizing because if they do I can't get to the co-lo to fix - [Kernel Upgrades - Maybe](https://www.eskimo.com/2022/12/kernel-upgrades-maybe/) - Kernel upgrades may or may not happen tonight. I let my wife take my car to work last night because it is better in snow and ice than hers. Coming home, she slipped a tiny bit, panic'd, parked it and walked the rest of the way. Now, the road in front of my house - [Virtual Private Servers Outage](https://www.eskimo.com/2022/12/virtual-private-servers-outage/) - The outage of virtual private servers was caused by a bug in the 6.0.12 kernel (the stability issues I mentioned earlier). It has been responsible for locking up four different machines in the last two weeks and was scheduled to be upgrade this Friday, however, since I had already put the new kernel on - [Kernel Upgrades](https://www.eskimo.com/2022/12/kernel-upgrades-11/) - I had gotten confused about the day Christmas was on and had incorrectly announced kernel updates on Christmas Eve. Actually they will be done on Friday evening the 23rd between 11pm-11:30pm providing the roads are still semi-passable. I will be upgrading to 6.0.15 from 6.0.12. - [Kernel Upgrade Christmas Eve 11pm-Midnight](https://www.eskimo.com/2022/12/kernel-upgrade-christmas-eve-11pm-midnight/) - Barring large amounts of snow on the roads, I will be doing a kernel upgrade this Christmas Eve starting at 11pm. Not really the thing I want to be doing on Christmas Eve but the present kernel has shown some instability that lead to one machine locking up. This will impact all of - [Mail](https://www.eskimo.com/2022/12/mail-19/) - When I changed the mail server IP address, I neglected to change the firewall rules for imap and pop3. This is fixed now. If you have further issues please let me know. - [IP address changes continued](https://www.eskimo.com/2022/12/ip-address-changes-continued/) - I am going to be making some more IP address changes tonight, these will mainly impact many of the shell servers and mail server briefly, although the IPs of these servers won't be changed, the physical host they are on will which requires their reboots as the host is rebooted. - [This Morning's Outage](https://www.eskimo.com/2022/12/this-mornings-outage/) - This morning's outage was caused by a kernel soft CPU lockup on the server that serves the home directories and also one virtual private server. Because it is a physical host, not a virtual machine, I had to drive to the co-location to power cycle it. This is caused by a race condition - [Reboots 11PM-4AM Tonight](https://www.eskimo.com/2022/12/reboots-11pm-4am-tonight/) - I will be rebooting various machines not to perform kernel upgrades but instead to make changes to IP addresses. In theory rebooting shouldn't be necessary but Leonard Poettering has so screwed up Linux with systemd that simply taking the network out of service and returning it to service with the new settings no longer - [Router Firmware Upgrade Complete](https://www.eskimo.com/2022/12/router-firmware-upgrade-complete/) - The router software upgrade completed successfully. - [Mail](https://www.eskimo.com/2022/12/mail-18/) - I changed the IP address of mail and changed it's IP in DNS, however, I neglected to update /etc/hosts on all the machines and that overrides DNS. This caused mail to file from shell servers as well as webmail. This has been corrected. - [IP Address Changes](https://www.eskimo.com/2022/12/ip-address-changes/) - I am working on changing the IP addressing scheme for machines at Eskimo. This will involve some reboots of NFS clients when IP server addresses change. The mail server address will be changing from 204.122.16.222 to 204.122.16.14. The name will remain "mail.eskimo.com". The old eskimo.com shell server will become "sunos.eskimo.com". - [Router Upgrade 11PM](https://www.eskimo.com/2022/12/router-upgrade-11pm/) - There will be a brief loss of connectivity for a router software upgrade tonight shortly after 11pm. If this is typical it may last about 5-10 minutes. - [WordPress Eskimo Home Page](https://www.eskimo.com/2022/12/wordpress-eskimo-home-page/) - An automatic update of WordPress broke the way it is displaying our home page, in particular I had a lot of inline CSS that it is for some reason not properly grocking now. So the home page will be a bit ugly for a day or two, my apologies. - [Kernel Upgrade Status and Manjaro](https://www.eskimo.com/2022/12/kernel-upgrade-status-and-manjaro/) - The issues that remained with Centos7 and Scientific7 with the existing 6.0.x kernels were not expected to be fixed until 6.2 according to the developers, however, 6.0.12 appears to have fixed them as both machines booted cleanly after the upgrade last night. Also, I adjusted the clock on ice from 4.3 Ghz to - [Manjaro](https://www.eskimo.com/2022/12/manjaro/) - Manjaro will be down for a while. It is hurt to the point where a fresh install is easier than straightening out dependency loops. - [Tentative Maintenance Work Friday 12/9 11pm-2am](https://www.eskimo.com/2022/12/tentative-maintenance-work-friday-12-9-11pm-2am/) - Depending upon weather and other factors, I am planning to upgrade the kernel 6.0.11, however this depends upon the roads being clear of snow. And then after the kernel upgrade, I am planning to go to the co-location facility and will be making some adjustments to the server that is serving the majority of - [Need Roof Repair Referral for Torch Down Roof Tree Limb Penetration](https://www.eskimo.com/2022/12/need-roof-repair-referral-for-torch-down-roof-tree-limb-penetration/) - I need a referral for someone who repairs torch down roofs. I've got a branch that came down and pierced my roof and having difficulty finding a contractor that does torch down in residential and isn't totally swamped already because of the storm in the Seattle area, I am in Shoreline. I'd rather - [Mystery Actually Deepens](https://www.eskimo.com/2022/12/mystery-actually-deepens/) - Ok, I found that I had X11forwarding set to No in ssh_config. I actually did not but something changed it, probably a Ubuntu upgrade. Setting that to Yes made it work again. But now here is the deeper mystery, there is something in my .home directory also affecting this, because even when set - [Weird X-windows Problem](https://www.eskimo.com/2022/12/weird-x-windows-problem/) - I'm looking for some community wisdom here, particular in light of being totally overwhelmed between roof issues, four days of power outages, water bed failures, dryer failure, heater failure, I've obviously done something to piss-off the creator. Anyway, I use the old eskimo.com system still as the NIS master and also do some - [Kernel Upgrade Postponed](https://www.eskimo.com/2022/12/kernel-upgrade-postponed-2/) - Owing to recent events and the fact that another snow storm is forecast for Friday evening and 6.0.10 seems to have introduced a file descriptor leak, I am putting off kernel upgrades until December 9th starting at 11pm by which time hopefully 6.0.11 will be out and addressed this newly introduced issue. - [Please Be Patient](https://www.eskimo.com/2022/12/please-be-patient/) - I apologize that I have been unresponsive for the last three days. I have been without electricity or an Internet connection and now have a tree limb sticking through my roof and ceiling. We just got power back after a three day outage, one of the very last of City Light's out of - [Mail Bans](https://www.eskimo.com/2022/11/mail-bans/) - Last night when we had a server lock up owing to a kernel bug, there was an unforeseen consequence discovered today. It appears the mail server was bound to this server as the NIS server it was using for authentication. This caused authentication to fail during this interval. If you attempted to access - [Monday's Server Issues - Friday's Kernel Update](https://www.eskimo.com/2022/11/mondays-server-issues-fridays-kernel-update/) - Since the file server that crashed contained both home directories, a lot of the web applications, and MariaDB database, it's sudden refusal to continue screwed up a lot of things which I've spent the entire evening fixing, so those of you who made payments today will see your receipts sometime tomorrow. I apologize for - [Emergency Reboot](https://www.eskimo.com/2022/11/emergency-reboot/) - Going to have to do an emergency reboot on the machine that hosts the web server, Ubuntu, and the /home directory owing to a CPU soft lockup that is persistent. - [Kernel Upgrade 12/2/2022 11PM PST (GMT-0800)](https://www.eskimo.com/2022/11/kernel-upgrade-12-2-2022-11pm-pst-gmt-0800/) - I am planning a kernel upgrade to 6.0.10 on Friday December 2nd, starting at 11:00PM Pacific Standard Time. I expect to be finished by 11:30 if everything goes smoothly. If on the other hand Poettering gets us again and systemd does not shut down or start up properly then it may not be fully - [NFS Kernel Server Upgrade](https://www.eskimo.com/2022/11/nfs-kernel-server-upgrade/) - Ubuntu pushed out an upgrade of the nfs-kernel-server today. SunOS had a good functional reliable NFS system in SunOS 3.0 in 1986 (they had an earlier version in SunOS 2.0 but not so good). Linux has been working on it for 20+ years and it's still flaky even though it is central to - [NextCloud Unborked, Kind Of](https://www.eskimo.com/2022/11/nextcloud-unborked-kind-of/) - I got authentication working again by taking the version 3.0.0-beta meant for Nextcloud 24, editing the xml file that described it to include support for Nextcloud 25, tried it, and it SEEMS to work. - [Nextcloud Borked](https://www.eskimo.com/2022/11/nextcloud-borked/) - Nextcloud 25.0.1 came out, but the app user_external used to authenticate Linux/Unix users hasn't been ported to 25.x yet, thus if you use a system login for access, until this is corrected there is no way to authenticate. I have filed a bug report on Github but if history repeats, it will be - [Nextcloud Upgrade In Progress](https://www.eskimo.com/2022/11/nextcloud-upgrade-in-progress/) - I am in the process of upgrading Nextcloud 24.0.7 to 25.0.1, this likely will take awhile as it involves major changes in database schemas, etc. However, one plus, I discovered part of the reason Nextcloud has been so slow. I've been using memcache and I've been using PHP 8.0, however memcache is configured - [Web Services](https://www.eskimo.com/2022/11/web-services/) - I apologize for the interruption to web services today. For reasons unknown, because there were no new dependencies that should have pulled the Ubuntu version of Apache2 in, Ubuntu installed their version of Apache2 which does not have all the necessary capabilities our locally compiled version has and broke our web services. - [Kernel Upgrades and NIS/NFS Checks Completed](https://www.eskimo.com/2022/11/kernel-upgrades-and-nis-nfs-checks-completed/) - All services are fully operational. - [Kernel Upgrades Done but Not Checked](https://www.eskimo.com/2022/11/kernel-upgrades-done-but-not-checked/) - At 12:54 kernel upgrades are completed. Sorry this took so long but one physical host failed to boot properly. The Nvidia graphics card did not initialize properly and systemd brought the machine up into single user mode but not multi-user mode so I could not access from here. I had to drive to the - [Kernel Upgrades 11pm Pacific Standard Time (GMT -0800)](https://www.eskimo.com/2022/11/kernel-upgrades-11pm-pacific-standard-time-gmt-0800/) - I've tested 6.07 and 6.08 and both seemed to have resolved the issue with squashfs, so will be doing kernel upgrades. Provided they haven't introduced new bugs, this should eliminate all bugs of any operational consequence. There still is an issue with startup of centos7 and scientific7 but that only generates an error message - [FTP Server Restored](https://www.eskimo.com/2022/11/ftp-server-restored/) - The FTP server is restored and somewhat better secured. I do not know what exploit they used because all of the known exploits for wu-ftpd I had fixed, so this is one not known, however, it would appear they only had anonymous user permissions as nothing outside of the ftp directory was disturbed. - [FTP server damaged](https://www.eskimo.com/2022/11/ftp-server-damaged/) - Someone apparently found an exploit that allowed them to really trash the public directory of our ftp server. Consequently, anonymous access is extremely restricted until I've been able to restore the directories from backups, modify some file permissions and create an apparmor profile to limit potential damage in the future. - [No Kernel Upgrade This Weekend](https://www.eskimo.com/2022/11/no-kernel-upgrade-this-weekend/) - 6.0.6 did not fix either of the outstanding bugs affecting the current kernel on our servers, therefore I will NOT be doing a kernel upgrade this Friday. - [Phishing Scams](https://www.eskimo.com/2022/11/phishing-scams-2/) - If you have received an e-mail like this, DO NOT RESPOND to it. It is what is known as as phishing scam, an attempt by a third party to obtain your authentication information and hack your account. There has been a recent significant uptick in these, mostly originating in foreign countries such as Iran and - [Eskimo North Kernel Upgrades](https://www.eskimo.com/2022/10/eskimo-north-kernel-upgrades/) - We will may be doing kernel upgrades again this Friday November 4th at the traditional time starting at 11PM pending testing of Linux-6.0.6 and possibly 6.0.7 if it should come out before then. There are presently two kernel bugs that are impacting us, neither very severely. One is that there are still CPU - [Kernel Upgrade Oct 28th 11PM PDT (GMT-0700)](https://www.eskimo.com/2022/10/kernel-upgrade-oct-28th-11pm-pdt-gmt-0700/) - Starting at 11PM, I will be performing a kernel upgrade to linux-6.0.5 on all machines requiring reboots of all Eskimo North machines. This process should be completed by about 11:30PM. With the exception of https://yacy.eskimo.com/, no service should be down more than about ten minutes. Yacy will take longer because it rebuilds it's index - [Fastest WordPress Site](https://www.eskimo.com/2022/10/fastest-wordpress-site/) - If you are looking for a place to host your WordPress website, I believe we have the fastest WordPress host in existence. Our site will load a WordPress site in under 200ms, our own loads in 148ms, WordPress's OWN site takes 942ms to load, nearly a full second, and six times slower than ours. - [RoundCube](https://www.eskimo.com/2022/10/roundcube/) - A few weeks ago, I upgraded roundcube to make it compliant with php 8.x. I did not notice at the time but this broke compose and settings. At the heart of the issue were database structural changes that the upgrade script did not address. To resolve this, I had to re-install with a clean database. - [Scam Alert](https://www.eskimo.com/2022/10/scam-alert/) - If you get a letter like this: It is NOT from Eskimo, it is a phishing scam attempting to obtain your login credentials. Do NOT respond, or at least do NOT provide them any identifiable information. If you wish to tell them what orifice in which to insert it that is fine, except even - [Kernel Upgrades](https://www.eskimo.com/2022/10/kernel-upgrades-10/) - Kernel upgrades are complete. Mostly went smoothly, some minor self-inflicted issues on a couple of machines but all good now. - [Kernel Upgrades Friday Oct 7 11:00-11:30PM PDT (GMT-0700)](https://www.eskimo.com/2022/10/kernel-upgrades-friday-oct-7-1100-1130pm-pdt-gmt-0700/) - We will be upgrading the remaining systems including the physical hosts to the 6.0.0 kernel release. We've got it presently installed on about a dozen servers and so far only one expedited RCU CPU Stall and that occurred during boot-up on the physical server with the most guest machines. This can happen even with - [Sad Sad](https://www.eskimo.com/2022/10/sad-sad/) - Just found out a long time customer, Dave Bruels who had Interlake China Tours, passed yesterday. I've spent many lunches with him and enjoyed his photos from China. Great man and will miss him. His wife had passed not long before and whatever the medical reason, I'm sure he passed of a broken heart. - [Zorin Borked](https://www.eskimo.com/2022/10/zorin-borked/) - Zorin will be down for a few days, well partially. It won't be fully operational again for several days. Because Zorin 15 is based upon Ubuntu 16.04, there is NO support for bpfilters, and the currently Linux kernel has deprecated the old iptables filtering scheme so fail2ban and other firewall features do not work. - [6.0 Kernel Working Well](https://www.eskimo.com/2022/10/6-0-kernel-working-well/) - The 6.0 kernel is working well, so far there has been only one expedited RCU CPU stall and that was on a virtual host during boot-up on a machine that has quite a few guests. These can occur at this time just because ALL of the guests are busy with start-up and it simply - [Linux Official 6.0 Release](https://www.eskimo.com/2022/10/linux-official-6-0-release/) - At this point I have Linux kernel 6.0 installed on all but private virtual machines and the physical hosts. So far it's running MUCH better than rc5 or rc6, but not perfect. I've gotten ONE expedited RCU CPU stall in a days time on a dozen or so machines. With 6.0rc5-6 I would see - [Kernel Progress?](https://www.eskimo.com/2022/10/kernel-progress/) - I received a note from bugzilla.kernel.org that another user found 6.0.0rc7 no longer had the expedited RCU CPU stalls issue, but before I could grab and try it, the official release of 6.0 came out, so I'm going to build it and give it a try on a few select servers. If it runs - [Service Affecting Kernel Upgrades Completed](https://www.eskimo.com/2022/10/service-affecting-kernel-upgrades-completed/) - Kernel upgrades are completed on the physical hosts and critical machines. They are not done on vps1-vps7, but those machines are all single core and single core machines are not experiencing the CPU stall bugs in 5.16-6.0 kernels. Some of the other shell servers also are not updated yet but these are sufficiently - [Kernel Upgrades 11pm Oct 2nd PST (GMT-0700)](https://www.eskimo.com/2022/10/kernel-upgrades-11pm-oct-2nd-pst-gmt-0700/) - Things are running smoothly but load is high with stock 5.15.0 kernel from Ubuntu. I've configured the latest 5.15 kernel (5.15.71) and installed on a number of machines and it is also running well with no forced preemption, 100HZ clock, and fully tickless kernel. This reduces overhead somewhat and so I am going to - [5.19.12 Broken](https://www.eskimo.com/2022/10/5-19-12-broken/) - Even though 5.19.11 ran okay on four busy servers for a week, 5.19.12 is NOT running well, and 5.19.11 is no longer available so I'm going to go back to a stock 5.15 Ubuntu kernel on most of the machines. Tonight there will be additional reboots between 11pm-11:30pm to this end. It seems - [New Kernel Failed](https://www.eskimo.com/2022/10/new-kernel-failed/) - The new kernel is not running well on our main NFS server so another reboot of iglulik, which is the server that provides the /home directories, was required to load a different kernel. Iglulik is now running the stock Ubuntu 22.04 5.15.0 kernel. Less than idea but 5.19.12 did not run well on it. - [Kernel Upgrades and Expedited RCU CPU Stalls](https://www.eskimo.com/2022/09/kernel-upgrades-and-expedited-rcu-cpu-stalls/) - From kernels 5.15 forward, we've had issues with expedited RCU CPU stalls on our servers. I've experimented with kernels configured per the stock Ubuntu configuration and these same kernels do NOT show expedited RCU CPU stalls. The RCU system is Read-Copy-Update, it is a means to allow read concurrency with updates without - [Manjaro Response](https://www.eskimo.com/2022/09/manjaro-response/) - A customer wrote: I think the only reason manjaro isn't used as much is because it isn't available on eskimo.com, only on yellow-snow. I've tried: $ ssh user@manjaro.eskimo.com ssh: Could not resolve hostname manjaro.eskimo.com: Name or service not known and gave up. So, I've added a CNAME in eskimo.com so you can ALSO reach it - [Manjaro - Anyone Using It?](https://www.eskimo.com/2022/09/manjaro-anyone-using-it/) - According to the logs, nobody has used Manjaro since April, so I am considering removing it if it is just a waste of resources. But the logs do not capture visual sessions because x2go does not write the /etc/wtmp and /etc/utmp files. If anyone has any objection to this machine going away, please - [SquirrelMail](https://www.eskimo.com/2022/09/squirrelmail-3/) - We recently upgraded squirrelmail to be compatible with PHP 8.0+. The upgrade unfortunately disabled some features with respect to theme color and font choices that some of our customers preferred. Consequently I've restored the old version along with configuring this application to use PHP 7.4 for which it is designed. I've - [Kernel Issues](https://www.eskimo.com/2022/09/kernel-issues-4/) - From 5.15 onwards there seems to be an incompatibility between tickless and non-preemptive options. If I select either one by itself the kernel seems to be stable, if I select both, I get RCU expedited CPU stalls. So this is not so easy to sort out because each of these options by itself triggers - [Kernel Issues](https://www.eskimo.com/2022/09/kernel-issues-3/) - The kernel issues are not resolved but I am making progress. I have been running customized kernels for eons because I can get more efficiency and better response than the kitchen sink kernels that are distributed with a Linux distribution like Ubuntu. From 5.15 forward there have been some minor stability issues - [Kernel Issues](https://www.eskimo.com/2022/09/kernel-issues-2/) - The 5.18.19 kernel which previously was stable is now showing instabilities and is showing the same CPU stalls as the 6.0.0 kernel which points to a possible configuration issue on my part. The only things I've intentionally changed between this and previous 5.18.19 kernels was to enable a bunch of Intel reptoline CPU security - [On the 6.0.0-rcX kernel](https://www.eskimo.com/2022/09/on-the-6-0-0-rcx-kernel/) - I've had several people ask about the issues with the 6.0.0-rc5 and 6.0.0-rc6 (so far) kernels, so far the developers have not picked up my bug report. For those interested in the details, here is the URL: https://bugzilla.kernel.org/show_bug.cgi?id=216501 - [Kernel Downgrades](https://www.eskimo.com/2022/09/kernel-downgrades/) - I did not finish downgrading all the machines to 5.18.19 but I did finish the physical hosts, the mail servers (client and both incoming), the web server, ubuntu, debian, centos7, and slinux-7. All the machines still on 6.0.0-rc5 are virtual machines so I can reboot them from here if necessary. I will be - [Correction Time Zone is Pacific Daylight Time (PDT) Not PST.](https://www.eskimo.com/2022/09/correction-time-zone-is-pacific-daylight-time-pdt-not-pst/) - Correction Time Zone is Pacific Daylight Time (PDT) Not PST. - [Emergency Kernel Reversion Tonight/Tomorrow 11PM PDT (GMT-0700)](https://www.eskimo.com/2022/09/emergency-kernel-reversion-tonight-tomorrow-11pm-pst-gmt-0700/) - I am going to revert as many machines as I can tonight back to 5.18.19 kernel starting at 11pm. I do not have time to prepare all the machines but what I do not get done will be finished tomorrow evening. The 6.0.0 kernel up through RC4 was good, but RC5 and later has - [PHP Default Back to 7.4](https://www.eskimo.com/2022/09/php-default-back-to-7-4/) - I've put the PHP default back to 7.4 because the APCu and Memcache PHP modules are not working for versions 8-8.2. - [PHP Upgrade](https://www.eskimo.com/2022/09/php-upgrade/) - I've upgraded the versions of PHP available now to include 5.6, 7.0, 7.1, 7.2, 7.3, 7.4, 8.0, 8.1, and 8.2, 8.2 isn't entirely complete yet so I don't recommend it's use though it is fundamentally functional but no redis cache, no memcache, etc. I have not yet updated the documentation on how to select - [Scientific7 Down](https://www.eskimo.com/2022/09/scientific7-down/) - Scientific7.eskimo.com is down. I broke it in an attempt to convert the /boot file system from 128 bit inodes to 256 bit inodes to accommodate 64-bit timestamps so that it can work beyond 2038, though why, given the end of life is ten years before that, is somewhat lost on me. At any rate, - [Kernel Upgrade Tonight 11PM PDT (GMT-0700)](https://www.eskimo.com/2022/09/kernel-upgrade-tonight-11pm-pdt-gmt-0700/) - I will be performing a kernel upgrade of all Eskimo North servers starting at 11PM tonight, I expect to conclude by 11:30PM. If all goes well, no individual service should be down for more than about ten minutes, most fewer. This will include shell servers, e-mail, web hosting, and fediverse services including https://friendica.eskimo.com/, - [Reboots Completed](https://www.eskimo.com/2022/09/reboots-completed-15/) - Reboots are completed. I did not have any issues with the web server running for five days on the new kernel, but 1/2 hour after reboot it failed, so not sure if a fluke or a real problem. I'll keep an eye on it. - [Kernel Upgrades Friday Sept 9th 11pm PDT (-0700GMT)](https://www.eskimo.com/2022/09/kernel-upgrades-friday-sept-9th-11pm-pdt-0700gmt/) - The release candidate kernel, 6.0.0-rc4 is working better than any kernel I've ever used to date, better than the last three stable releases by far in terms of performance, stability, and security. I've got it running on four of our busiest servers presently, and unlike 5.19.x it has not produced any CPU stalls or - [Kernel Issues](https://www.eskimo.com/2022/09/kernel-issues/) - At present, as near as I can determine there is no Linux kernel which is simultaneously: 1) Up to date with regards to outstanding security issues. 2) Entirely stable and functional. 3) Performs well and doesn't spend more time in the kernel than in userland. It is possible to get - [Kernel Regression Testing](https://www.eskimo.com/2022/09/kernel-regression-testing/) - We will be performing kernel regression testing to try to determine what commit introduced the rcu expedited timeout bug in the 5.19 kernel to assist kernel developers in correcting the situation. Because it can take several weeks for this to manifest on an unbusy workstation, I will be trialing various kernels on our web - [Kernel Upgrades are Completed](https://www.eskimo.com/2022/08/kernel-upgrades-are-completed-2/) - Kernel upgrades have been completed. All servers are back in service. All NFS and NIS relationships are re-established. All services are up. - [Kernel Upgrades August 26 11PM Pacific Daylight Time (GMT-0700)](https://www.eskimo.com/2022/08/kernel-upgrades-august-26-11pm-pacific-daylight-time-gmt-0700/) - We will be upgrading to kernel 5.19.4 this Friday at 11PM. I don't know if this will address the CPU stalls or not. The number of changes to the kernel since 5.19.3 are so voluminous that I don't have time to review all of them to see if any of them impact this problem. - [Some Australia .au Domains Blocked](https://www.eskimo.com/2022/08/some-australia-au-domains-blocked/) - Because of a severe mailbomb I experienced the last few days, it was necessary to block incoming mail from several hundred .au domains with improperly configured mail servers that bounce mail to non-existent addresses by storing the e-mail, looking up the incoming address, and sending it back, rather than refusing it up front and - [Mail Maintenance Completed](https://www.eskimo.com/2022/08/mail-maintenance-completed/) - The mail system maintenance is complete. - [Mail Server - Maintenance 11pm Additional](https://www.eskimo.com/2022/08/mail-server-maintenance-11pm/) - I forgot the most important part of this message, I will be taking mail down for about ten minutes at 11pm tonight to make another backup to capture the corrected NFS configuration. - [I've Filed the Following Kernel Bug Report with Bugzilla.Kernel.Org regarding 5.19.x](https://www.eskimo.com/2022/08/ive-filed-the-following-kernel-bug-report-with-bugzilla-kernel-org-regarding-5-19-x/) - Robert Dinse 2022-08-21 07:37:09 UTC Created attachment 301614 [details] The configuration file used to Comile this kernel. This behavior has persisted across 5.19.0, 5.19.1, and 5.19.2. While the kernel I am taking this example from is tainted (owing to using Intel development drivers for GPU virtualization), it is also occurring on non-tainted kernels on servers - [Mail - Unannounced Boot of Some Machines](https://www.eskimo.com/2022/08/mail-unannounced-boot-of-some-machines/) - I had to reboot igloo which is one of our physical hosts which hosts the mail server among others. It was necessary because after making a backup of mail, it would not complete a boot with the CPU tasks hanging. Something was corrupted with the KVM hypervisor on the physical host. I did discover - [Mail Server Maintenance](https://www.eskimo.com/2022/08/mail-server-maintenance-2/) - I am going to take the mail server down again at around 11pm for about 20 minutes to make a current backup as the existing backup is approximately 300 updates behind and half of those are security related. - [Mail Server Returned To Service](https://www.eskimo.com/2022/08/mail-server-returned-to-service/) - The client mail server is returned to service at approximately 8:10PM however, there will be a brief introduction in approximately 15 minutes once updates are applied to bring it back current from it's backed up state and new kernel is installed (it is running on a kernel from May). - [Mail Server](https://www.eskimo.com/2022/08/mail-server-12/) - The client mail server crashed this evening at approximately 7:25 PM Pacific Daylight Time. It will not boot, something has corrupted the initramfs file system, I am working on repairing it. Estimated downtime 30-90 minutes. - [Kernel Upgrade Date Correction - August 26th 11PM Friday](https://www.eskimo.com/2022/08/kernel-upgrade-date-correction-august-26th-11pm-friday/) - It was brought to my attention that I erred in my announcement of the changed date of the kernel upgrade. I posted it would be Friday August 24th, but the correct date is Friday August 26th. A customer felt this was ambiguous since the 24th was on a Wednesday not Friday and asked me - [Kernel Upgrade Postponed](https://www.eskimo.com/2022/08/kernel-upgrade-postponed/) - A kernel upgrade for all of eskimo.com's services scheduled for August 17th at 11pm PDT has been postponed until August 24th because kernel 5.19.2 came out this Wednesday after I had already built and partially deployed 5.19.1. 5.19.1 contained only three very small fixes so did not require extensive testing but 5.19.2 contains far - [Nextcloud Repaired](https://www.eskimo.com/2022/08/nextcloud-repaired/) - Nextcloud (https://nextcloud.eskimo.com) is repaired and up to the current stable release of 24.0.2.4. There were a three apps that were broken but not listed as incompatible. - [Nextcloud Is Borked](https://www.eskimo.com/2022/08/nextcloud-is-borked/) - Nextcloud is down. Something went horribly wrong during an update last night. I tried to roll back to the previous version and it's not working either. It appears to be an issue with apps but I haven't been able to isolate which yet and this is a painful and slow process. - [Kernel Upgrades Friday August 19th 11PM Pacific Daylight Time](https://www.eskimo.com/2022/08/kernel-upgrades-friday-august-19th-11pm-pacific-daylight-time/) - I am planning a kernel upgrade this Friday, August 19th, at 11pm Pacific Daylight Time (GMT-0700). This will affect all Eskimo North services, shell servers, e-mail, web hosting, other hosting, https://friendica.eskimo.com/, https://hubzilla.eskimo.com/, https://nextcloud.eskimo.com, https://yacy.eskimo.com/, and https://www.eskimo.com/. I do not expect the downtime for any one service to exceed ten minutes and the - [Fedora](https://www.eskimo.com/2022/08/fedora-2/) - Fedora is now again accessible from the outside world. The fail2ban folks have fixed fail2ban so that it works properly (if not a bit slowly) with python 3.11 used by Fedora. - [Kernel Upgrades Completed](https://www.eskimo.com/2022/08/kernel-upgrades-completed-18/) - The kernel upgrade to 5.19.0 went amazingly smooth this evening with only one service (dovecot) not starting on one machine (mail) and that because a dovecot upgrade had overwritten my systemd start-up file so that it tried to start before all of the necessary file systems were mounted. There was also one virtual - [Comcast Router Replaced](https://www.eskimo.com/2022/08/comcast-router-replaced/) - Well I'll have to give Comcast credit where credit is due. They were out here five hours after the router failed with a working replacement. So all good again, for now. I think this is my 15th Comcast modem / router. - [Comcrap Router Borked](https://www.eskimo.com/2022/08/comcrap-router-borked/) - My Comcrap Cable router (with Cisco firmware) has had a melt down and is NAT'ing me to an address of 71.197.179.184. Since many of the servers I need to access to do ordinary maintenance are hard-wired to only allow connections from my IP address, I can not access many of them. Comcrap is supposedly - [Fedora - No Outside Access](https://www.eskimo.com/2022/08/fedora-no-outside-access/) - I am taking outside access away from fedora until fail2ban can be made operational again. Fedora upgraded the python instance to 3.11 but fail2ban will not work with python greater than 3.10, so until this is fixed I am waking away access from the outside to prevent brute force password attacks. You - [Kernel Upgrade Friday August 5th 11pm Pacific Daylight Time (GMT-0700)](https://www.eskimo.com/2022/08/kernel-upgrade-friday-august-5th-11pm-pacific-daylight-time-gmt-0700/) - I am tentatively scheduling a kernel upgrade for all eskimo.com services this Friday pending successful testing of 5.19.0. I've only installed it on two machines at present, the web server because it is really heavy on context switches and uses many computer language used in web programming, including C, C++, java, python, php, ruby, - [Router Maintenance Completed](https://www.eskimo.com/2022/07/router-maintenance-completed/) - Router maintenance at eskimo.com has been completed. The automatic upgrade failed but manually uploading the new firmware succeeded. - [Router Maintenance](https://www.eskimo.com/2022/07/router-maintenance/) - This affects all eskimo.com services including our shell servers, virtual private servers, https://friendica.eskimo.com/, https://hubzilla.eskimo.com/, https://nextcloud.eskimo.com/, and https://www.eskimo.com. This evening there will be several times when I/O will freeze for router reboots. I am trying to upgrade software but having some difficulty so need to follow a procedure given to me by Ubiquiti that - [Mail List Recommendations](https://www.eskimo.com/2022/07/mail-list-recommendations/) - I am looking for some suggestions for open source mail list software capable of running under Linux to replace the SmartList mail lists we are currently hosting. SmartList has several fundamental problems. First, neither it nor procmail which it relies on has not been under active development for at least a decade. This - [Router Software Upgrade 7/13/22 11PM PST (GMT-0700)](https://www.eskimo.com/2022/07/router-software-upgrade-7-13-22-11pm-pst-gmt-0700/) - We are going to do a software upgrade tonight. This requires a router reboot which takes about five minutes during which we will temporarily disconnect from the Internet. This affects ALL eskimo.com services. - [Kernel Upgrades Completed](https://www.eskimo.com/2022/07/kernel-upgrades-completed-17/) - Kernel upgrades were completed with only minor systemd issues like not starting timedate service on some machines. Nothing severe or even consequential beyond requiring restart of those services. - [Kernel Upgrades 7/8/2022 11PM Pacific Daylight (GMT -0700)](https://www.eskimo.com/2022/07/kernel-upgrades-7-8-2022-11pm-pacific-daylight-gmt-0700/) - This will affect ALL of Eskimo North's services including https://friendica.eskimo.com/, https://hubzilla.eskimo.com/, https://nextcloud.eskimo.com/, and https://www.eskimo.com/. We will be upgrading to 5.17.15 which is the last iteration of the 5.17 kernel, we will not be upgrading to 5.18 series yet as it has not been entirely stable on my workstation yet. Keep getting stuck - [What is Wrong](https://www.eskimo.com/2022/05/what-is-wrong/) - Figured out what was wrong with the 5.17.11 kernel, it is an option I selected which strengthens the kernel by zeroing the stack to initialize everything before a call and after, this required an argument to gcc which gcc-12.1 doesn't understand. It's only an issue in a module used for Nvidia compatibility but Ubuntu - [Kernel Updates Postponed](https://www.eskimo.com/2022/05/kernel-updates-postponed/) - It was fortunate that kernel upgrades were delayed as I had installed it on my workstation, and the "fixes" broke the kernel much worse than it was. It had the potential to do something wrong, so far unrealized on the servers, but the current release is taking kernel oopses on a regular basis, so - [Outage](https://www.eskimo.com/2022/05/outage-3/) - Sorry about the outage tonight between about 3am and 4:30am. I screwed my workstation up to the point where it became necessary to restore from backups, but that did not work. Even after re-installing grub and update-grub2, etc, it's trying to boot a kernel that doesn't exist, so I'm using an old version of - [Kernel Upgrade](https://www.eskimo.com/2022/05/kernel-upgrade-2/) - I did not get the kernel upgrade done last night, between the fact that I broke a tooth and so had to go to bed early so I can get to the dentist today and the fact that 5.18.0 had some compiler errors, I've got patches from the developers that fix them but haven't - [Emergency Kernel Upgrade Tonight](https://www.eskimo.com/2022/05/emergency-kernel-upgrade-tonight/) - This affects all eskimo.com services: I am planning an emergency kernel upgrade of at least the most used shell servers, the mail server, and the web server tonight. May not be able to get to everything as I can not stay up late as I broke a tooth last night and need to - [Non-censored Federated Search Engine](https://www.eskimo.com/2022/05/non-censored-federated-search-engine/) - Now that duckduckgo.com and swisscows.com/ch are censoring, as bing.com and google.com have always done, and since what else is out there, save for yandex.com which censors but on different content, there is nothing else out there that does not censor except for yacy.com. Searx DOES only proxy requests from the others above, it provides - [Replacement Router Ordered](https://www.eskimo.com/2022/05/replacement-router-ordered/) - I finally received the info I needed from Ubiquiti and we have a new router ordered. It should be here in 7-10 days and will provide 6.8x the CPU power of the existing router. Our existing router is sometimes saturating between 6pm-9pm Pacific Daylight Time, even though the bandwidth is not anywhere near - [Kernel Upgrades are Completed](https://www.eskimo.com/2022/05/kernel-upgrades-are-completed/) - Kernel upgrades and reboots are completed. All services are active. All NFS mount points working. All NIS binding successful. Only server where there was a problem was one customers private virtual server and that because he had configured SSL but the SSL certificates were not present. Not really sure what changed, - [Maintenance Tonight](https://www.eskimo.com/2022/05/maintenance-tonight-3/) - In about 1-1/2 hours I will be going to the co-location facility to extract a broken server. This should not be service impacting except that it is in a physical stack that will require disturbing other machines and whenever that happens there is always to potential for a power cord to fall out or - [Kernel Updates Friday May 20th at 11PM Pacific Daylight Time (-0700 GMT).](https://www.eskimo.com/2022/05/kernel-updates-friday-may-19th-at-11pm-pacific-daylight-time-0700-gmt/) - We will be doing kernel updates on all systems Friday May 20th starting at 11pm Pacific Daylight Time (-0700 GMT). I expect interruptions of individual services not to be more than about ten minutes and it should be completed by 12PM (midnight). This will impact all of Eskimo North's services including: https://www.eskimo.com/ https://nextcloud.eskimo.com/ - [Correction Kernel Update on May 20th](https://www.eskimo.com/2022/05/correction-kernel-update-on-may-20th/) - CORRECTION: Kernel Updates will be Friday May 20th, NOT tonight May 19th. - [Friend of Nicholas Heer - Please Contact Me](https://www.eskimo.com/2022/05/friend-of-nicholas-heer-please-contact-me/) - Someone here, and sorry my memory does not remember who, contacted me a while back with respect to Nicholas Heer, if you could please contact me again I would very much appreciate it. - [Distro Updates Completed](https://www.eskimo.com/2022/05/distro-updates-completed/) - All of the Ubuntu based physical hosts are now upgraded to Jammy 22.04. As far as I can tell, everything seems to be up and running. Did need to fix some things with apparmor breaking my name server which has some configurations they didn't expect, and various broken NFS mount points, and random - [Eskimo.Com Kernel Upgrades Complete - Distro Upgrades in Progress](https://www.eskimo.com/2022/05/eskimo-com-kernel-upgrades-complete-distro-upgrades-in-progress/) - Kernel upgrades are completed. All NFS and NIS mounts checked and all machines checked to be sure all services are running. A Distribution upgrade of the physical hosts from Ubuntu 20.04 to 22.04 is in progress. In several hours this will also require rebooting everything again and fixing some NFS configuration files which - [Kernel Upgrades Sunday May 14th at 11PM PDT (GMT -0700)](https://www.eskimo.com/2022/05/kernel-upgrades-sunday-may-14th-at-11pm-pdt-gmt-0700/) - I ordinarily do kernel upgrades on Friday so I have a weekend to recover if something goes wrong before the heavier business usage during the week, however, because my car is in the shop I am going to do them this Sunday evening starting at 11pm. That day because my wife has it off - [NFS Kernel Server - Linux Ubuntu 22.04](https://www.eskimo.com/2022/05/nfs-kernel-server-linux-ubuntu-22-04/) - For anyone maintaining a hetrogeneous network utilizing Linux NFS servers, when I upgraded from 20.04 to 22.04 on our mail server which exports the mail spool to various machines, most using version 4.2 over TCP NFS but some using version 3 UDP and one using version 2 UDP, it broke all of our machines - [22.04 NFS](https://www.eskimo.com/2022/05/22-04-nfs/) - I wrote zero since when I meant zero sense. Not sure why but I have a real problem with homonyms, since and sense, to, too, and two, etc. I know the correct usage but when I write, my stream of consciousness is in a verbal form and the translation to to text more - [Kernel Upgrades](https://www.eskimo.com/2022/05/kernel-upgrades-9/) - I will be doing kernel upgrades over the next week or so but not all at once as per usual. I will be doing the virtual machines which are most exposed to the external Internet first, and holding off on the physical servers until my car returns from the shop (it ate it's alternate, - [Network Latency / Packet Loss](https://www.eskimo.com/2022/05/network-latency-packet-loss/) - Our router at the co-lo facility is running short of CPU during peak traffic times in the evenings resulting in greater latency and some packet loss. I am currently researching a replacement which will have approximately 6x as much CPU to accommodate this traffic and future growth. The current router is an - [Mail](https://www.eskimo.com/2022/05/mail-17/) - All of our mail servers have now been upgraded to 22.04, and one of the things I was unaware of before upgrading to 22.04 is that postfix, the mail transport agent that we use, has changed a bunch of defaults. This may cause some problems and I am reviewing all the settings to make - [Mail Problems](https://www.eskimo.com/2022/05/mail-problems-3/) - A distribution upgrade failed on mail.eskimo.com leaving the machine in an unbootable state where grub failed to install properly. An attempt to install grub2 manually results in can not write EFI variables to NvRAM. I've not encountered this on a virtual machine before and not being successful at finding info on how to - [Mail Imap/Pop/SMTP Server Upgrade](https://www.eskimo.com/2022/05/mail-imap-pop-smtp-server-upgrade/) - An upgrade of the mail server mail.eskimo.com is in progress. There may be temporary periods where the server does not respond during this process. It should be brief. - [Web Server Upgraded](https://www.eskimo.com/2022/05/web-server-upgraded/) - Finally got the web server upgraded. Don't really know what went wrong last night, pretty much repeated the same procedure but this time after I had finished the compile environment still worked so I was able to recompile apache2 and have it use the newer openssl 3.0.2. So all of eskimo.com including friendica.eskimo.com, - [Happy Mother's Day + Maintenance Notice](https://www.eskimo.com/2022/05/happy-mothers-day-maintenance-notice/) - I wish to wish all of you who are Mother's a very happy Mother's day! My mother has been gone from this planet for about a decade now. She was the wisest woman I've ever known in my life. Lived a hard life but threw her best effort at it. She is with - [Web Server and Covid Update](https://www.eskimo.com/2022/05/web-server-and-covid-update/) - I was unsuccessful at upgrading the web server last night but learned some more things and am going to make another stab tonight but before I start I am going to take it down for about 1/2 hour to make a backup as there was an upgrade to hubzilla today that I don't want - [Web Server Maintenance](https://www.eskimo.com/2022/05/web-server-maintenance-7/) - I am going to make another attempt at updating the distro on the web server, if it fails there will be about a half-hour downtime to restore from backup, if it succeeds only a few minutes to reboot. Past attempts have failed because of library version mis-matches, I believe I've got a way - [Web Server Maintenance 10pm-11pm](https://www.eskimo.com/2022/05/web-server-maintenance-10pm-11pm/) - Now that I've removed a number of unused PPAs, removed "Proposed" packages, and hopefully gotten the web server better prepared for a distribution update, I am going to take the server down for approximately an hour between 10pm-11pm tonight to make another image backup prior to another distribution upgrade attempt. This will affect - [Ubuntu Upgrade Complete](https://www.eskimo.com/2022/05/ubuntu-upgrade-complete/) - Ubuntu has successfully been upgraded to Jammy Jellyfish 22.04 LTS. Microsoft Visual Studio Code is no longer installed. If you were using this please let me know and I will re-install. - [Ubuntu](https://www.eskimo.com/2022/04/ubuntu-11/) - Because at some point I got the proposed archive enabled on Ubuntu, there are incorrect versions of a lot of software installed on Ubuntu making upgrading impossible. I have to de-install all this stuff, upgrade, and then re-install. So Ubuntu is going to be marginally functional while this is in progress. I hope that - [Ubuntu Upgrade Aborted](https://www.eskimo.com/2022/04/ubuntu-problems/) - I am backing out the partial Ubuntu upgrade and restoring from backups tonight and will attempt again tomorrow. - [Ubuntu and WebServer Maintenance 10PM - 11PM Tonight](https://www.eskimo.com/2022/04/ubuntu-and-webserver-maintenance-10pm-11pm-tonight/) - I am going to take the shell server, ubuntu.eskimo.com, and the web server www.eskimo.com down for maintenance tonight between 10pm-11pm to make backups prior to upgrading from Ubuntu 20.04 Focal to Ubuntu 22.04 Jammy. After the backups I will begin upgrade. I do not anticipate long downtime for Ubuntu for the actual upgrade, - [Covid and Mailed Payments](https://www.eskimo.com/2022/04/covid-and-mailed-payments/) - I have to ask that you not mail payments until further notice as TIna, Raymond, and myself are all down with Covid-19. We are all doing alright so far, staying in our heads and not going into our lungs, good oxygen saturation status, mild fevers. Pretty much the symptoms of your average head - [Kernel Upgrades completed.](https://www.eskimo.com/2022/04/kernel-upgrades-completed-16/) - Kernel upgrades are completed, all machines are operational. - [Kernel Upgrades / Nextcloud](https://www.eskimo.com/2022/04/kernel-upgrades-nextcloud/) - I will be doing another kernel upgrade this Friday between 11pm-midnight requiring reboots of all machines. As I feared, there were bugs introduced in NFS, particular an issue with delayed requests in NFSD causing the daemon to die (nfs server daemon). While I have not encountered this yet it is a documented problem - [NextCloud](https://www.eskimo.com/2022/04/nextcloud-9/) - I apologize if Nextcloud is not working for you at the moment. I attempted an upgrade yesterday and it failed resulting in file integrity check failing for a bunch of files. I opened a ticket for this and found out many others are experiencing the same thing for this release, but as of - [Mail and Web Maintenance](https://www.eskimo.com/2022/04/mail-and-web-maintenance/) - I intent to take the mail and web servers down tonight at 11pm to make backup images. This is just a way to quickly restore the mail server or web server if something catastrophic happens to their file systems. The mail server should take about 1/2 hour and the web server probably similar. Don't - [Reboots Complete](https://www.eskimo.com/2022/04/reboots-complete/) - Tonight's kernel upgrade went smoother than I had anticipated. I had anticipated issues based upon the fact that we were moving to a new major release and I knew that NFS code got somewhat re-worked again. This has always been a problem in Linux, not sure why they can't get a 1995 vintage protocol - [Ubuntu](https://www.eskimo.com/2022/04/ubuntu-10/) - Someone or some thing halted Ubuntu this evening at just after 8pm. I do not know what, could not locate anything in the logs. At this point I've removed public access to systemctl, removed a number of graphical reboot/shutdown tools, and performed an early upgrade of the kernel on this machine in case - [Kernel Upgrades Friday 4/15 11pm-Midnight](https://www.eskimo.com/2022/04/kernel-upgrades-friday-4-15-11pm-midnight/) - I'm going to upgrade all of the system kernels to 5.17.3 on Friday April 15th between 11pm and midnight Pacific time. I would have stuck with the 5.16 kernel until hell froze over if Linus had chose to make it a long term supported kernel, but unfortunately he did not. 5.15 did not - [Ubuntu Crash](https://www.eskimo.com/2022/04/ubuntu-crash/) - Ubuntu crashed around 3PM today while I was applying updates. I do not yet know what caused it. - [Debian / GCC](https://www.eskimo.com/2022/04/debian-gcc/) - Debian is back up but the gcc is back to an older version so I'm going to attempt a compile 11.2 which I know is a good version as I use it for kernel compiles. - [Debian](https://www.eskimo.com/2022/04/debian-12/) - Sorry, I broke Debian tonight. I am going to load backups so I can recover /usr/local which I destroyed by installing a broken version of gcc that overwrote the working version. Then I will reboot back into the current version and replace /usr/local with the backed up version. Sorry this is kind of a - [New Capability for Private Virtual Servers](https://www.eskimo.com/2022/04/new-capability-for-private-virtual-servers/) - We now offer cockpit control panel for our virtual private servers if they are a debian derived system such as debian, ubuntu, mint, mxlinux, zorin, julinux, etc. I've been down on control panels in general for security reason but this one looks good as it does not require any special privileges and only - [NextCloud Upgrades](https://www.eskimo.com/2022/04/nextcloud-upgrades/) - Eskimo North NextCloud located at https://nextcloud.eskimo.com/ has been upgraded to version 22.6 and additionally some new applications have been installed including a shared white board and chat, and quite a few others. Some tables which previously were unindexed are now indexed which provides some performance improvements. - [Comcast](https://www.eskimo.com/2022/03/comcast/) - If you are having problems connecting to any services here, please let me know. We are having some peering issues with Comcast causing some issued for Customers accessing us through Comcast. It seems that peering is going through CenturyLink's network. I've generated tickets for this from each end, but the more information we can - [Kernel Upgrade April Fools Day 11pm-11:30PM Pacific Daylight Time](https://www.eskimo.com/2022/03/kernel-upgrade-april-fools-day-11pm-1130pm-pacific-daylight-time/) - The kernel upgrade last Friday introduced bugs worse than it fixed resulting in some instability. As a result, we'll be doing another upgrade this Friday to fix the bugs the last bug fixes introduced. Hopefully better results this time, only time will tell. - [Reboots Completed](https://www.eskimo.com/2022/03/reboots-completed-14/) - Reboots are completed. All machines up and verified proper NIS and NFS mounts and all subsystems started successfully. Two machines are still on 5.16.14, those are the UUCP machine and manjaro, and this because I'm having difficulties getting 5.16.17 to build properly on those boxes. I have a bug report opened on bugzilla regarding - [Kernel Upgrades 11pm-11:30pm PDT March 25th](https://www.eskimo.com/2022/03/kernel-upgrades-11pm-1130pm-pdt-march-25th/) - I am going to be rebooting all the machines to upgrade the kernels tonight starting at 11PM. I expect all to be done by 11:30PM. Interruption of individual services should be brief. - [tmux](https://www.eskimo.com/2022/03/tmux/) - Per a customers request, tmux is now installed on all shell servers. - [Web and Mail Server work Completed](https://www.eskimo.com/2022/03/web-and-mail-server-work-completed/) - All disks are now good, no more errors, and loads on all machines now are satisfactory, switching to virtio drivers on all the virtual hosts made a HUGE difference. Not hard to get results like below with STATIC HTML content, but find another site that loads PHP based WordPress site in this time frame! - [Web Server Optimization](https://www.eskimo.com/2022/03/web-server-optimization/) - Doing some benchmarking on virtual machines, I discovered on fast drives that virtio drivers are about 10% faster than emulated SATA drives for the SAME hardware. I had erroneously assumed SATA would be faster because it supported hdparm read-ahead where virtio does not support this parameters, but it turns out that the Linux - [Brief Interruption at 5PM Today](https://www.eskimo.com/2022/03/brief-interruption-at-5pm-today/) - The brief interruption of web services at 5PM Pacific Time today March 20th was for the purpose of increasing the memory allocated to the web server from 64GB to 80GB to provide a bit more overhead for load spikes. - [Back to the Original Plan](https://www.eskimo.com/2022/03/back-to-the-original-plan/) - Ok, running good now that I switched the emulation so going to take down around 11pm'ish to backup as it is. Also going to take Ubuntu down to move to a different physical server. Ubuntu and the web server are the two heaviest loads so doesn't make good sense to have them on the - [Virtual Machine Web Server](https://www.eskimo.com/2022/03/virtual-machine-web-server/) - Regarding the web server, I discovered the Ethernet device was set to an emulated device, but by switching to virtio, I was able to get the interrupt load down substantially, so going to watch it run for a while and make a decision later with regards to whether or not to move to physical - [Change in Plans](https://www.eskimo.com/2022/03/change-in-plans/) - Instead of backing up the existing www configuration, I am going to attempt to move the web server from a virtual server to a physical server. The reason for this is that 5.16 kernel is not handling interrupts as efficiently as previous kernels did but previous kernels have an exploit known as a "dirty - [Web / Debian / Ubuntu](https://www.eskimo.com/2022/03/web-debian-ubuntu/) - The work I set out to do last night is completed except the final configuration was not what I had initially intended, however, I was not able to boot from flash owing to the BIOS of the machine only recognizing SATA flash drives and not PCIe drives which is unfortunate since all the SATA - [Maintenance Tonight 11pm - Sometime Saturday Morning](https://www.eskimo.com/2022/03/maintenance-tonight-11pm-sometime-saturday-morning/) - This is a reminder that we will be doing some fairly extensive maintenance on Iglulik, which is the server that hosts ftp/www, ubuntu, debian, and mint virtual machines, the MariaDB database, and /home directory partitions. As a consequence all services that require any of these facilities will be unavailable for a number of hours. - [Contact Info - Important Please Read!](https://www.eskimo.com/2022/03/contact-info-important-please-read/) - If you have not updated your contact information and the telephone number you registered with is no longer current, please contact me to update it. Tonight I've had a spate of people calling to request password changes from out of state numbers that did not match the client at all but they KNEW - [Post Office and Payments](https://www.eskimo.com/2022/03/post-office-and-payments/) - I'm having difficulty accessing my Shoreline Post Office Box because they have taken to locking the building after hours. I've had a PO Box there for around three decades but this has only recently become an issue. So if you are making a payment that is near your expiration date, please use a debit - [More Extended Maintenance](https://www.eskimo.com/2022/03/more-extended-maintenance/) - More extended maintenance is planned for the server that hosts /home directories and web service this Friday March 18th starting at around 11PM through Saturday around 4AM Pacific Daylight Time (GMT-7). This time frame is approximate at best. The mate to the drive that failed developed two bad sectors right after this drive - [Only Accomplished Part of What I Planned](https://www.eskimo.com/2022/03/only-accomplished-part-of-what-i-planned/) - Replaced the failed drive but did not install the new flash owing to the mounting screw was missing from the motherboard and there is a card that is in the way. So instead of using the socket on the motherboard, I'm going to get a PCIe adapter card for a whole of $9, and - [Eskimo North Extended Maintenance Outage 11PM March 12th - 4AM March 13th](https://www.eskimo.com/2022/03/eskimo-north-extended-maintenance-outage-11pm-march-12th-4am-march-13th/) - Tonight I am going to perform hardware surgery on the machine that hosts home directories. As a result, ALL services except virtual private servers will be down for a number of hours. The server which hosts the /home directory partition has an ill drive in the RAID array for this partition. It has - [Kernel Upgrades](https://www.eskimo.com/2022/03/kernel-upgrades-8/) - Kernel upgrades are done except for four machines, one of them is just borked, somehow the image got corrupted so it's being restored from backups. Could not get the drivers for the Fusion I/O drive to compile under 5.16 so I've ordered a more main brand conventional SSD, it won't provide quite the I/O - [Oops](https://www.eskimo.com/2022/03/oops/) - I was doing half a dozen things at once and got into the wrong terminal and accidentally rebooted one of the physical servers before I had intended to and without stopping virtual servers first, and under this circumstance it takes forever and a day to reboot so it will probably be 15-30 minutes before - [Fedora Broken](https://www.eskimo.com/2022/03/fedora-broken/) - Something broke networking on Fedora. I thought it was the new kernel as it failed after rebooting, but booting off the old kernel, networking also does not start. So I am restoring from a backup and then will apply the kernel upgrade and then updates to bring it forward again. - [Kernel Upgrades Between Wed March 9th and Saturday March 12th](https://www.eskimo.com/2022/03/kernel-upgrades-between-wed-march-9th-and-saturday-march-12th/) - Because a very nasty exploit has become known that could lead to privilege escalation, a kernel upgrade will be applied to all servers as soon as possible. Because the exploit requires some sort of inside access to utilize, shell servers will be done first, then virtual private servers, and the rest. The physical - [Unscheduled Reboot](https://www.eskimo.com/2022/03/unscheduled-reboot/) - Around 12:35 I needed to reboot the server that hosts the home directories, web server, ubuntu, debian, and mint because something went wrong with the quota system that I was unable to identify and correct without a reboot. - [Scheduled Maintenance March 5th Cancelled](https://www.eskimo.com/2022/03/scheduled-maintenance-march-5th-cancelled/) - The downtime for March 5th is cancelled. The drive in question has only a single failed sector. SMART estimates more than six years of life remaining. It would not automatically re-allocate the failed sector on a read, apparently it can only do this on a write which is odd since it's part of a - [Eskimo North Maintenance Saturday Morning Midnight - 2AM'ish March 5th Pacific Standard Time](https://www.eskimo.com/2022/03/eskimo-north-maintenance-saturday-morning-midnight-2amish-march-5th/) - I will be taking the main server holding home directories down for up to two hours Sunday to replace a failed drive in a RAID array. No data is lost as the result of this drive being ill since all data is duplicated in the RAID array. Things will be slower for the twelve - [Mint is Repaired but Down for Backup](https://www.eskimo.com/2022/03/mint-is-repaired-but-down-for-backup/) - I've repaired Mint, at the heart of the issue was the glibc package name was somehow missing a '1' at the end and so package version mis-matches resulted. I had to force a manual install and then re-apply several hundred updates. Because chasing this down was not easy I am making a backup now - [Mint Unavailable for a few hours](https://www.eskimo.com/2022/03/mint-unavailable-for-a-few-hours/) - A routine upgrade exploded today on Mint leaving packages in an inconsistent state of dependency loop hell that I can't easily fix so I'm going to restore from a backup and then re-apply missing updates. This will take an hour or more to complete. It will be unavailable during this time. - [Brief Interruptions Around 4pm Today](https://www.eskimo.com/2022/02/brief-interruptions-around-4pm-today/) - I apologize for the brief interruptions of service around 4pm and 4:20pm today. Our router was being attacked by a botnet attempting to brute force guess passwords on it. To combat this I was able to employ fail2ban, the same software we use on our hosts, as the router is based upon Debian - [Outlook E-mail Update](https://www.eskimo.com/2022/02/outlook-e-mail-update/) - The attacks from Microsofts servers have not have not completely stopped but they've slowed considerably. Looking specifically at what is in the logs tripping fail2ban, I find that they are trying to send mail from non-existent domains. This is something our servers will not accept anyway, so for now I've whitelisted the IP - [Outlook E-mail](https://www.eskimo.com/2022/02/outlook-e-mail/) - E-mail from outlook.com is having issues right now because at least 750 of their servers are infected with or affected by some kind of botnet malware. Our servers are equipped with fail2ban which watches logs for abusive behavior and locks those IP addresses out for a period of time. The attacks are still - [Reboots Completed](https://www.eskimo.com/2022/02/reboots-completed-13/) - Reboots are completed, everything at eskimo.com back operational. - [Reboot Servers around 11PM Pacific Time](https://www.eskimo.com/2022/02/reboot-servers-around-11pm-pacific-time/) - I plan on rebooting all servers around 11PM tonight. This is necessary to make various updates, some of which are security related, operational. Downtime for any one machine should not exceed 10 minutes. I expect it to go smooth as other than routine updates supplied by OS vendors, I haven't made any significant - [Scam Calls](https://www.eskimo.com/2022/02/scam-calls/) - Apparently someone is calling my customers with scam calls representing us. I'm not calling people at random. If you happen to get one of these calls, please check your caller ID and collect the number and any other identifying information you can get. Thank you. - [Centos-Stream.Eskimo.Com Graphical Desktop](https://www.eskimo.com/2022/02/centos-stream-eskimo-com-graphical-desktop/) - Graphical Desktop sessions are now working on Centos-Stream.Eskimo.Com via X2go, RDP, VNC, and via the web (host or terminal) Guacamole services. - [Centos8 Being Retired](https://www.eskimo.com/2022/02/centos8-being-retired/) - Redhat has decided to retire Centos8 early. Most of what was on Centos8 has been ported to Centos-Stream which is also available here. So if you were using the hostname Centos8.eskimo.com please change to Centos-Stream.eskimo.com. If there is anything that you need that was on Centos8 but it is not on Centos-Stream, please - [Mint Upgrade Completed](https://www.eskimo.com/2022/01/mint-upgrade-completed/) - The upgrade of Mint completed uneventfully. You'll probably not notice much difference, it only affected 160 packages. This is the smallest distribution upgrade I've ever seen for ANY operating system and makes me wonder if Mint is dead. - [Mint Upgrade](https://www.eskimo.com/2022/01/mint-upgrade-3/) - I am going to take Mint down for about 30-45 minutes to make a backup prior to upgrading from Mint 20.2 Uma to Mint 20.3 Una. Mint upgrades have been fraught with problems in the past so want to make a backup first just in case it ends up in an unrecoverable situation. Once - [Eskimo.Com Service NOT SHUTTING DOWN](https://www.eskimo.com/2022/01/eskimo-com-service-not-shutting-down/) - It became clear from all the feedback I got that a notice I sent out was very misinterpreted. Eskimo North is NOT shutting down nor is our shell service, e-mail, web hosting, friendica, hubzilla, or nextcloud node. The ONLY thing shutting down are two of the shell servers, one because it's absolutely antique - [Julinix.Yellow-Snow.Net and Eskimo.com](https://www.eskimo.com/2022/01/julinix-yellow-snow-net-and-eskimo-com/) - I have decommissioned julinix.yellow-snow.net because in the last year only two people have logged into it and each of them only once. It is basically Ubuntu re-themed anyway so doesn't really offer anything unique. I am in the process of decommissioning the eskimo.com shell server because it is an antique SunOS server that - [Mail to Yahoo, AOL, and any site using hostedmail.com](https://www.eskimo.com/2022/01/mail-to-yahoo-aol-and-any-site-using-hostedmail-com/) - One of our customers account was compromised, in spite of all the facilities in place to prevent this, and used to send spam triggering the addition of our mail server IP to abuseix. Yahoo and AOL have also blocked it for excessive spam. I have requested delisting at abuseix and with AOL and Yahoo - [Kernel Test Concluded](https://www.eskimo.com/2022/01/kernel-test-concluded/) - Kernel test is completed, 5.16 did not perform adequately. - [Web Server - New Kernel Being Tested](https://www.eskimo.com/2022/01/web-server-new-kernel-being-tested/) - We are trialing a new kernel on our web server. We were running 5.13.19 however it is past end-of-life upstream. We attempted 5.15.x but the performance of this kernel was garbage. However, I've tested 5.16.0 on my workstation and it performed much better than 5.15 and at least on par if not slightly better - [Automatic Restart and Alarm for Web Service](https://www.eskimo.com/2022/01/automatic-restart-and-alarm-for-web-service/) - Because 2.4.52 version of Apache seems to have a bug that causes it to occasionally stall, and because earlier versions have a root exploit that makes returning to them unfeasible, I wrote an automatic recovery script to recover the server in the event of a failure. What it does is automatically test the - [Web Server Tests](https://www.eskimo.com/2022/01/web-server-tests/) - I am going to kill our web server for about five minutes a couple of time to test some automatic failure recovery scripts I've written to try to prevent a repeat of last night's outage. - [Fax Inoperative Until January 28th or so](https://www.eskimo.com/2022/01/fax-inoperative-until-january-28th-or-so/) - Our fax machine has bitten the dust. I've got a new one on order but it won't be here until around the 28th. In the meantime, to get payment info, or other material you don't feel comfortable sending plain-text, to me securely please either send e-mail from your eskimo.com e-mail address or - [Mail and Web Server Backup Done](https://www.eskimo.com/2022/01/mail-and-web-server-backup-done/) - The mail server took longer than I had expected. And then after the backup was done it would not start properly. So took some mucking around to figure out why and fix. There were some things that used files on /misc which is an NFS mounted partition that were not set to wait for the - [Maintenance 11pm Tonight Fri Jan 7th](https://www.eskimo.com/2022/01/maintenance-11pm-tonight-fri-jan-7th/) - I am going to be taking the web server, mail server, and some of the shell servers down tonight at 11pm for between 20-60 minutes (mail server closer to 20, web server closer to 60) to image them (a form of backup) as I've made substantial changes to fix various issues to all of - [Web Server Issues](https://www.eskimo.com/2022/01/web-server-issues-3/) - This affected Eskimo North and customer sites as well as friendica.eskimo.com, hubzilla.eskimo.com, and nextcloud.eskimo.com. We had a web server issue causing intermittent failure of the web server. I thought this was load related as the server is quite heavily loaded, but it did it today during a time of very light load. I was able - [Mail Server Changes Completed](https://www.eskimo.com/2022/01/mail-server-changes-completed/) - Mail server changes are completed. Mail that fails SPF, DKIM, or DMARC checks will no longer be rejected outright, instead they will be placed in your spam folder by default. In the process of implementing this, I discovered many perl modules needed by spamassassin where not installed and so portions of spamassassin were not - [Mail System Changes - WARNING!](https://www.eskimo.com/2022/01/mail-system-changes-warning/) - About a year ago we implemented opendkim, opendmark, and spf checking in order to reduce mail forgeries. This did have the intended effects, although it's not impossible to forge e-mails with these measures in place, it is difficult enough that it prevents the vast majority. However, DMARC protocol and to a lesser degree - [Amazon AWS Services](https://www.eskimo.com/2021/12/amazon-aws-services/) - Amazon has lost power to one of their data centers. This is causing issues reaching sites which they host. They do not elaborate on the nature of the power outage but you can read about it here: Major services including Slack, AWS, Hulu, Imgur facing outages I don't envy the folks having - [Web Server Maintenance - Mon Dec 20 12:50-2AM](https://www.eskimo.com/2021/12/web-server-maintenance-mon-dec-20-1250-2am/) - Taking website down to make another backup after finding more missing stuff. This should take approximately one hour to complete. This will affect https://friendica.eskimo.com/ https://hubzilla.eskimo.com/ https://nextcloud.eskimo.com https://www.eskimo.com/ and all customer sites hosted with Eskimo North. - [Web Server Restored](https://www.eskimo.com/2021/12/web-server-restored/) - Sorry for the downtime between about 10pm Dec 18th and 1am Dec 19th. In the process of trying to fix some golang stuff so I could get a new application running, I meant to remove a golang subdirectory in /usr/share and accidentally hit return after typing rm -r /usr before I finished the rest - [Denial of Service](https://www.eskimo.com/2021/12/denial-of-service-5/) - We experienced a very brief denial of service attack around 3pm today. By the time I logged into the router and turned on traffic analysis to determine the nature and origin it had stopped. - [Nextcloud Dashboard](https://www.eskimo.com/2021/12/nextcloud-dashboard/) - A number of users complained about the introduction of Dashboard a while back. It is an app now and could be disabled. Given that nobody had anything good to say about it and it was slow as molasses in liquid helium, I have disabled it. - [Nextcloud Restored](https://www.eskimo.com/2021/12/nextcloud-restored/) - https://nextcloud.eskimo.com/ is restored to service. The automatic updater just would not work. After about a dozen retries and it hanging in various spots during the upgrade, I manually upgraded which was successful. There are a number of older applications that are no longer supported and a number of new available that I've yet to - [Nextcloud Down](https://www.eskimo.com/2021/12/nextcloud-down-2/) - Nextcloud is down tonight. An update blew up and left it in an unusable state. I've been working at it for about eight hours but need to get some sleep. - [Friday Evening Maintenance 11:30PM](https://www.eskimo.com/2021/12/friday-evening-maintenance-1130pm/) - At approximately 11:30PM Friday December 10th, Pacific Time, I will be rebooting the physical hosts which in turn will reboot all the virtual machines. This is necessary because some system library updates that have been recently applied require a system restart to utilize the new code which addresses some Linux security concerns. This should - [ssh-keygen detail](https://www.eskimo.com/2021/12/ssh-keygen-detail/) - One other detail, if you give it a pass phrase then it will ask for that pass phrase every time you use ssh-copyid. This is just an extra layer of security if your originating machine is not secure. - [Eskimo Shell Users - Logging In Without a Password](https://www.eskimo.com/2021/12/eskimo-shell-users-logging-in-without-a-password/) - If ssh-copyid gives you this error: ssh-copy-id user@ubuntu.eskimo.com /usr/bin/ssh-copy-id: ERROR: No identities found It means that you have never created ssh keys on your ssh client. To do so: type: ssh-keygen It will respond with something like: Generating public/private rsa key pair. Enter file in which to save the key (/home/user/.ssh/id_rsa):[Enter key] Enter passphrase (empty - [Web Server / MXLinux Shell Server](https://www.eskimo.com/2021/12/web-server-mxlinux-shell-server/) - Mxlinux is now upgraded to Mxlinux-21. The following describes what I am working on to address slow web response. If you like tech details read on, else just know I'm working on a solution. I am in the process of acquiring components for a new web server. The existing server is no - [News for Shell Users](https://www.eskimo.com/2021/12/news-for-shell-users/) - You can login to any of our shell servers without a password by setting up ssh keys. To do this, from a terminal on your machine type: ssh-copyid username@host.eskimo.com Where username is your login, and host is the shell server you want to connect to, for example, if I wanted to set - [Mail Server](https://www.eskimo.com/2021/11/mail-server-11/) - Our client mail server is having some issues with NFS connectivity to Ubuntu. I am going to reboot it in hopes of clearing this problem however, NFS reconnecting is not 100% after a reboot in Linux so I will have to manually check all the shell servers which may take about twenty minutes after - [Red Hat Based Servers](https://www.eskimo.com/2021/11/red-hat-based-servers/) - A while back vnc and rdp no longer functioned on the various Redhat based servers, centos7, centos8, centos-stream, fedora, and scientific7. This was caused by an update the xrdp that required a newer version of openssl but they did not provider a corresponding upgraded openssl. That has since been resolved and these - [Web Server Tonight](https://www.eskimo.com/2021/11/web-server-tonight/) - Our web server was unavailable or intermittently available this evening for about 1/2 hour owing to some hackers in France first trying to exploit a buffer overflow that existed in Apache 2.4.50, and when that failed just a high traffic denial of service attack that ran the server load over 250 briefly at which - [Debian Restored](https://www.eskimo.com/2021/11/debian-restored-2/) - Debian is now, to the best of my knowledge, fully restored to health. Most of the software that was previously online is there, there are a few packages I removed because I felt they hindered security without providing much use for remote users. If there is something missing you need still let me know. - [Debian is Up](https://www.eskimo.com/2021/11/debian-is-up-2/) - I found what was causing the boot problems, firewalld and ufw were simultaneously installed. In earlier revisions these played good together but now they lock up when both installed. And firewalld by itself does not seem to function anymore, does not add anything into iptables. VNC is just giving a black screen. Installed same as - [Debian](https://www.eskimo.com/2021/11/debian-11/) - I was trying to get vnc working and for some reason the firewall was blocking it. I had ufw allow vnc but still couldn't connect to it off of the machine. So I turned off firewalld and the machine locked up. Obviously this is not correct behavior and I am still troubleshooting. Availability of - [Debian](https://www.eskimo.com/2021/11/debian-10/) - Debian is back up except I do not have all the graphical access working again and much software is not installed. I would appreciate your help. Installing all the software previously installed is not workable as it causes systemd to get stuck in a loop and not boot properly. I ended up re-installing - [Ubuntu](https://www.eskimo.com/2021/11/ubuntu-9/) - My apologies for kicking everyone off of Ubuntu. Went to reboot Debian and accidentally rebooted the wrong server. - [Debian](https://www.eskimo.com/2021/11/debian-9/) - In the process of installing software it exploded again, issue with nvidia drivers. I'm going to have to start-over again. It's 5:30AM so need to get some sleep. - [Debian Up](https://www.eskimo.com/2021/11/debian-up-2/) - Debian is back up but software is still re-installing and I am not re-installing your cron jobs until all the software is re-installed as some of them depend upon packages not yet installed. Also, rdp and vnc and web desktop access are working again and ssh -X redirection is also working again. None - [Debian Re-install](https://www.eskimo.com/2021/11/debian-re-install/) - I am going to do a fresh install of Debian Bullet 11. This will take some time. I apologize for this but systemd has not worked properly since the upgrade and I've spent several hours trying to troubleshoot with no success. I think at this point a clean install is the best bet for - [Debian Broken](https://www.eskimo.com/2021/11/debian-broken/) - Debian is presently broken. Something is scrambled with systemd and it will not come back up properly after a reboot. I may need to do a fresh install on this machine to get it to work properly. Systemd has not functioned correctly since upgrade to bullet. I'm still actively troubleshooting but it may be - [Unanticipated Debian Reboot](https://www.eskimo.com/2021/11/unanticipated-debian-reboot/) - I apologize for the unanticipated Debian reboot today. I am trying to fix some software issues and when installing software the install got stuck during a systemd, the bane of Linux, services restart and there is no way to recover except to reboot. - [Looking for High Capacity CPU Cooler](https://www.eskimo.com/2021/11/looking-for-high-capacity-cpu-cooler/) - I'm looking to build a new machine, primary function will be a faster more robust web server. Because Google, Duckduckgo, Bing, and now even SwissCows has gotten heavily into censoring information about adverse vax events, certain political issues, etc, I want to put a yacy search engine back up, but when I had one - [Ubuntu 21.10](https://www.eskimo.com/2021/10/ubuntu-21-10/) - If you're running the short term release of Ubuntu, 21.04, considering upgrading to 21.10, you may want to hold off a bit. I upgraded my workstation the day 21.10 was released. To day, installing normal system upgrades rendered my system unusable and unrecoverable except to recover from backups. So in my humble opinion, - [Latest Phishing Scam from linode.com forged as other e-mail addresses.](https://www.eskimo.com/2021/10/latest-phishing-scam-from-linode-com-forged-as-other-e-mail-addresses/) - If you get mail like this DO NOT CLICK ON THE LINK. It is a linode.com phishing scam and will attempt to obtain your login credentials under the false guise of increasing your mail storage and in the process probably infect your computer with a virus. - [Debian Bullet - X-fowarding Borked](https://www.eskimo.com/2021/10/debian-bullet-x-fowarding-borked/) - Since upgrading debian to bullet, X-fowarding does not work. One thing that changed in Bullet was a switch from Xorg being the default to Xwayland. However, the Bullet faq claims x-forwarding still works but that hasn't been my experience. I have X-11 Forwarding enabled in sshd_config and I have xauth installed, but still no - [Phishing Scams - Please Read](https://www.eskimo.com/2021/10/phishing-scams-please-read/) - A "Phishing Scam" is a scam designed to get your credentials for things like access to your mail here, or your bank account information, or your Amazon account, etc. Here is an example: ----- ID:nanook@eskimo.com Your nanook@eskimo.com Password expired today at 9/29/2021 11:25:07 a.m. Use this button to continue with the same password as - [Phone Service](https://www.eskimo.com/2021/09/phone-service/) - My apologies to those who called and tried to let me know about the web outage. My Comcast telephone service was apparently also out because I was home and in the office at the time and the phone did not ring. I did not get notifications of your recorded messages until today. I've tested - [Web Service](https://www.eskimo.com/2021/09/web-service/) - Sorry for the downtime in web service today. It appears suexec which is used to set the web server permissions for a given users files lost it's setuid attribute yesterday. Not sure how but the update time on the file coincided with when I did system updates and the Apache Makefile for 2.4.49 now - [Manjaro Available](https://www.eskimo.com/2021/09/manjaro-available/) - Just a few hours later they came out with an update to ypbind and yp-tools that made them compatible with the newer libraries so Manjaro is again available. - [Manjaro Unusable](https://www.eskimo.com/2021/09/manjaro-unusable/) - The morons at Manjaro have broken ypbind by making it and the libraries it depends upon incompatible versions. I thought perhaps something just went wrong with updates so I restored from an old backup and then updated again and it brought it right back to an unusable state. So until the folks at - [Web Server Outage 16:35-16:45 Sept 25th](https://www.eskimo.com/2021/09/web-server-outage-1635-1645-sept-25th/) - A Linux upgrade changed openssl in such a way that the new libraries were incompatible with the existing compilation of apache2. I had to recompile apache2 to get it to listen to the SSL port again. Sorry, Ubuntu does not bother to notify us when they make crushing changes to system libraries. This - [Kernel Upgrades Completed 23:21](https://www.eskimo.com/2021/09/kernel-upgrades-completed-2321/) - Kernel upgrades completed, Mail system and all shell servers and virtual private servers are available. NFS and NIS connectivity has been verified on all machines. - [Kernel Upgrade and Payment Option](https://www.eskimo.com/2021/09/kernel-upgrade-and-payment-option/) - Tonight's Kernel Upgrade will be brief, starting at 11PM and should be concluded by 11:15PM as the big server with home directories, Mint, Debian, Ubuntu, and the Web Server are all done because all got sick under the previous kernel and were upgraded Tuesday. So Web service and those three shell servers will - [gcc 11.2 available on Ubuntu](https://www.eskimo.com/2021/09/gcc-11-2-available-on-ubuntu/) - For those who would like to learn / experiment with the bleeding edge features of C and C++, gcc 11.2 is now available on the Ubuntu shell server. To use it make sure that /usr/local/bin is first in your path before /usr/bin. - [Kernel Upgrades Tonight](https://www.eskimo.com/2021/09/kernel-upgrades-tonight/) - Because Ubuntu just locked up and because this time I got an indication it was an issue with the hypervisor and physical host, I am going to be upgrading the following machine kernels early this evening around 11:30PM: Iglulik (has NFS /home and /misc directories), ftp, debian, ubuntu, and mint. These will all - [Kernel Upgrades - Friday September 24th 11pm-12 midnight Pacific Daylight Time](https://www.eskimo.com/2021/09/kernel-upgrades-friday-september-24th-11pm-12-midnight-pacific-daylight-time/) - We'll be doing kernel upgrades of all machines this coming Friday starting at 11pm. Interruption to most services should be relatively brief, however, web based services may take approximately 1/2 hour to restore. This is because I'll be reverting the physical host to a 5.13.19 kernel (presently the only host on 5.14) as 5.14 - [Kernel Upgrades](https://www.eskimo.com/2021/09/kernel-upgrades-7/) - I plan on doing some kernel upgrades over the evening but probably not the physical hosts or the lesser used virtual machines. There is some instabilities in the existing kernel that is causing CPU lock-ups. Mostly these are transient and resolved by watchdog timers but one did hang our web server the other day. - [Our Web Server Upgraded to Apache 2.4.49](https://www.eskimo.com/2021/09/our-web-server-upgraded-to-apache-2-4-49/) - Apache 2.4.49 was released today and had many important security fixes and also some fixes to help it gracefully shutdown which makes it do so much faster. This will result in a much shorter interruption during log rotates at midnight. We have installed 2.4.49. Pingdom tests reveal essentially unchanged performance loading our Wordpress - [Web Server Kernel Issues](https://www.eskimo.com/2021/09/web-server-kernel-issues/) - With respect to Eskimo North's we services, I found that the issue that hung our web server has existed since May of this year so I'm not real optimistic the next kernel upgrade will resolve it, however, I found a kernel variable I set that will cause the system to panic and reboot if - [Web Server / Kernel Upgrades / Dial Access](https://www.eskimo.com/2021/09/web-server-kernel-upgrades-dial-access/) - This note covers several topics, web server crash, kernel upgrades, and 56k dial service. The web server crashed today shortly after 9AM but nobody called until 1PM and it was in a mode where it was still consuming CPU and appeared up to the hypervisor, but not talking to the network so I - [Eskimo.Net](https://www.eskimo.com/2021/09/eskimo-net/) - Eskimo.net was unavailable to the Internet part of Thursday and Friday (Sept 9th-10th) owing to an error I made on the master DNS server. It didn't show up right away until the cache expired on the secondaries. It is corrected now. - [Web Server Crashed](https://www.eskimo.com/2021/09/web-server-crashed-2/) - Web server crashed at 22:10 PM Pacific Time today (Sept 9th). Fortunately I tried to access it right at that time so it was back up after 4 minutes. Unfortunately, it did not provide any information in the logs that gives a clue as the the cause. It just hard hung and would not - [Usenet News](https://www.eskimo.com/2021/09/usenet-news-2/) - The Usenet News service I have been using to provide news for our shell servers seems to have gone TU without warning. They sent me no notice but just disappeared from the Web. They were a US based firm but the domain name is now owned by an outfit in England. There is no - [E1000 Drivers Still Fuxored](https://www.eskimo.com/2021/09/e1000-drivers-still-fuxored/) - Unfortunately the 5.14.0 kernel did not contain the new E1000 driver advertised in advance, still the broken 2012 vintage broken driver. So I'm cancelling the kernel upgrade I had planned for this Friday since nothing is to be gained from it. I do see there is a new 2021 driver on Intel's website, so - [Kernel Upgrades Friday Sept 3rd, 2021 11pm-Midnight](https://www.eskimo.com/2021/09/kernel-upgrades-friday-sept-3rd-2021-11pm-midnight/) - I successfully got the iomemory drivers working under 5.14.0 on the server where it was important (the one with the flash card and Intel ethernet). I've turned on hardware offloading and so far it has run cleanly. Assuming it is still clean tomorrow I plan on upgrading the remaining servers to the 5.14.0 kernel - [Server Maintenance](https://www.eskimo.com/2021/09/server-maintenance-2/) - I have determined what I need to do to get the iomemory drivers (for flash card on main server with NFS home directories, web server, and flash drive) with the 5.14.0 kernel. So I am going to be taking things down in about 15 minutes, between about 12:15 AM and 1:15 AM or so - [5.14 Maintenance](https://www.eskimo.com/2021/08/5-14-maintenance/) - The server maintenance is completed for the night. The new drivers would not compile against 5.14.0 so we are back to 5.13.12 for now. - [Eskimo Maintenance Work 11PM - 1AM Aug 30-31 Pacific Daylight Time](https://www.eskimo.com/2021/08/eskimo-maintenance-work-11pm-1am-aug-30-31-pacific-daylight-time/) - I am going to be booting one of the physical servers (Iglulik) into a new 5.14 kernel tonight and this will take the web server, Mint, Debian, and Ubuntu shell servers down for a short interval, approximately ten minutes, EXCEPT the web server. The web server will be down longer because the upgrade - [Debian Upgrades - Connecting with Old Putty](https://www.eskimo.com/2021/08/debian-upgrades-connecting-with-old-putty/) - I am receiving complaints from people unable to connect to servers that they used to be able to using putty on versions of Windows prior to Windows 10. The reason you are unable to connect is that as exploits are found in various encryption schemes, those schemes are retired in favor of newer - [Kernel Upgrades Completed](https://www.eskimo.com/2021/08/kernel-upgrades-completed-15/) - System kernel upgrades are completed. All services are back online. Debian came up first time properly so I ended up not doing any systemd debugging today. Whether this was a fluke or they fixed something with upgrades since last kernel upgrade I do not know. If it messed up again I will debug - [Kernel Upgrade and Debian Friday Aug 20th 11PM-Midnight](https://www.eskimo.com/2021/08/kernel-upgrade-and-debian-friday-aug-20th-11pm-midnight/) - We will be upgrading all servers to 5.11.12 on Friday at 11pm. This will take about an hour. I will also be adding some debugging code in Debian to try to gather more information with respect to problems with the systemd configuration so it may be down for a longer period while I - [HVAC failure at Co-Location Facility](https://www.eskimo.com/2021/08/hvac-failure-at-co-location-facility/) - I have received notice that there is a failure of some HVAC equipment at the Isomedia facility where our equipment is located. Unlike Integra, they have people monitoring these things and have already dispatched personal to repair. I have temperature monitors in the servers and so far they haven't shown any increase - [Debian Bullseye Up](https://www.eskimo.com/2021/08/debian-bullseye-up/) - Debian Bullseye is UP. The systemd configuration still is not correct so it is very slow to boot but it does eventually start everything just not in the correct order and only after various timeouts. I will figure out systemd eventually. I keep hoping Poettering will give up and go away but he seems - [Debian Upgrade Not Going Well](https://www.eskimo.com/2021/08/debian-upgrade-not-going-well/) - After upgrade to bulleys, debian will not boot with NIS enabled. NIS (Network Information Services) is required in order for you to be able to login. This seems to be yet another Poettering systemd induced snafu. I am looking for a solution. - [Debian Upgrade In Progress](https://www.eskimo.com/2021/08/debian-upgrade-in-progress/) - I am in the process of upgrading debian.eskimo.com from Buster to Bullseye. During this process ssh sessions may be terminated and new connections will not be accepted until completion. At least one reboot will be required. - [System Maintenance Completed](https://www.eskimo.com/2021/08/system-maintenance-completed/) - System maintenance completed. There were minor issues but no major problems, everything is up and running. - [Kernel Upgrade, Kernels Available](https://www.eskimo.com/2021/08/kernel-upgrade-kernels-available/) - We will be performing kernel upgrades tonight between 11pm and midnight. All machines will be rebooted at some point during that interval. The kernels we use are the most recent upstream kernels compiled for tickless operation. They are available to you at: https://www.eskimo.com/kernel or ftp://pub/kernel They are available in both .deb - [Kernel Upgrades Completed](https://www.eskimo.com/2021/08/kernel-upgrades-completed-14/) - All the machines had finished booting by around 11:20PM however there were several that did not boot correctly and required manual intervention to bring them back up fully and there were some NFS mounts that did not mount automatically. Everything was resolved by midnight. - [Kernel Upgrades Completed](https://www.eskimo.com/2021/07/kernel-upgrades-completed-13/) - Everything was back up by 12:35 except the machine I use for system accounting. For some reason the virtual host network interface would not work. I ended up deleting and re-installing the network interface, that finally got it to talk with a new Mac address. Don't really have any idea why the old one - [Mail Server](https://www.eskimo.com/2021/07/mail-server-10/) - The mail server is fixed, though I still have to check NFS mount points of various servers, however, the CAUSE of the issue with mail was not, as I had suspected, the new kernel. Something has changed the behavior of the system such that the order of data in nsswitch.conf is being ignored, - [Client Mail Server](https://www.eskimo.com/2021/07/client-mail-server-2/) - I am having problems with the mail server being unstable with the new kernel, I am going to try 5.13.5 (came out since we installed 5.13.4) to see if it resolves. This will take about 20 minutes to compile and install. - [Kernel Upgrade Friday July 30th 11PM-Midnight](https://www.eskimo.com/2021/07/kernel-upgrade-friday-july-30th-11pm-midnight/) - Going to do another kernel upgrade this Friday starting at 11PM providing a crash doesn't force one sooner. I didn't want to move to 5.13 so early in the release cycle, actually didn't want to move to it at all but was forced to by the Sequoia exploit. There are lots of - [Kernel Upgrades Completed](https://www.eskimo.com/2021/07/kernel-upgrades-completed-12/) - It has been a long grueling upgrade, made particularly difficult by one server having drivers not officially supported under 5.13.x kernels, but I got it working. All NFS mounts and NIS bindings have been verified. Another issue this time around, at some point some bug(s) have been introduced into systemd causing it - [Tickless Kernels - Sequoia Exploit](https://www.eskimo.com/2021/07/tickless-kernels-sequoia-exploit/) - I have removed all tickless kernels prior to 5.13.4 because they are vulnerable to a recently announced root exploit that goes by "Sequoia", CVE-2021-33909. 5.13.4 tickless kernels are available for all Debian and now for all Redhat based Intel and AMD x86-64 systems (any 64 bit Intel or AMD CPU). Previously I only - [Kernel Upgrades](https://www.eskimo.com/2021/07/kernel-upgrades-6/) - All public facing servers except the private virtual servers have been upgraded to 5.13.4 which has the Sequoia exploit fixed. I will work on getting the virtual private servers done today, so your vps if you have one here, will be rebooted some time today. Tonight 11pm I will upgrade the file - [Manjaro Down](https://www.eskimo.com/2021/07/manjaro-down/) - I'm not sure what broke manjaro but it will not boot into a mode where I can login, not even console, so am restoring from backup. - [Kernel Upgrades](https://www.eskimo.com/2021/07/kernel-upgrades-5/) - Because of the severity of the Sequoia security bug, kernel upgrades for shell servers and the web and mail servers, basically anything exposed to the public, will happen tonight instead of Friday as originally planned, and non-active servers will be upgraded sooner than 11pm. The downtime for these will only be about 1 minute - [Kernel Upgrades Friday Jul 20 11-12pm](https://www.eskimo.com/2021/07/kernel-upgrades-friday-jul-20-11-12pm/) - There are some worthwhile fixes to the kernel and as a result will be doing a kernel upgrade on July 20th, 11PM-Midnight. Most services will experience an outage of about ten minutes during this interval. - [Router Firmware Upgrade 5PM](https://www.eskimo.com/2021/07/router-firmware-upgrade-5pm/) - I am going to upgrade the router firmware shortly after 5pm tonight. This will result in a brief interruption of Internet connectivity, usually lasting 1-2 minutes. Usually this is brief enough that ssh sessions will not be knocked down but no guarantees. - [Kernel Upgrades](https://www.eskimo.com/2021/07/kernel-upgrades-4/) - The reason that there have been no kernel upgrades for the last two weeks or this week is that 5.12.x has reached the point where it has been stable on our platform and I've been monitoring the progress and have not seen any changes that fix bugs that impact us, provide security improvements, or - [Mint VNC Fixed](https://www.eskimo.com/2021/07/mint-vnc-fixed/) - VNC on Mint is fixed. - [Mint VNC Broken](https://www.eskimo.com/2021/07/mint-vnc-broken/) - VNC is currently broken on mint, it will produce only a small fixed-size screen. I am working to resolve. In the meantime, either use a different shell server, or a different protocol such as web based, or x2go, or rdp. - [Mail Issue Resolved](https://www.eskimo.com/2021/07/mail-issue-resolved/) - Mail issue has been resolved. An update altered the permission on the mail spool directory in a way that, while more secure, (using a mail group rather than setuid mail programs) breaks it's compatibility with many applications which are not setgid. Permissions have been reset. - [Mail Imap Pop3](https://www.eskimo.com/2021/07/mail-imap-pop3/) - An update today seems to have broken dovecot resulting in webmail and external imap and pop-3 mail clients failing. Shell mail clients that read the mail spool directly will still work. I am working to resolve this issue. - [Security](https://www.eskimo.com/2021/06/security/) - I am working on some additional security measures, and there is the potential I may break things at least temporarily in the process. If you noticing anything broken, please submit a ticket or e-mail support@eskimo.com. - [Kernel Upgrades Completed](https://www.eskimo.com/2021/06/kernel-upgrades-completed-11/) - Kernel upgrades are completed, all NFS mounts and NIS binds are working. Everything should be in service. - [Kernel Upgrades Tonight 11PM-Midnight](https://www.eskimo.com/2021/06/kernel-upgrades-tonight-11pm-midnight/) - I am planning on doing kernel upgrades tonight starting at 11pm. If things go well they may be concluded by 11:30, if they go like last time might be 2:30, hopefully not. There has been problems with grub configuration being properly updated by the automatic kernel installer lately and when it fails it - [The Denial of Service Attack Concluded](https://www.eskimo.com/2021/06/the-denial-of-service-attack-concluded/) - The denial of service attack seems to have concluded. It was not of sufficient intensity to cause lost packets but it did cause delays of up to two seconds from time to time which would be noticeable if you were typing something interactively like using an ssh terminal or composing e-mail. There - [Denial of Service Attack](https://www.eskimo.com/2021/06/denial-of-service-attack-7/) - We are under a denial of service attack at present. It is affecting my router so heavily I am having difficulty logging in to determine it's origin and block it. - [Interruption in HTTPS webservice](https://www.eskimo.com/2021/06/interruption-in-https-webservice/) - I apologize for the interruption in the encrypted web service (https://) this afternoon. System updates changed something in a way that broke Apache's ability to serve encrypted pages without recompiling. - [Everything is Back Online](https://www.eskimo.com/2021/06/everything-is-back-online/) - For some reason one physical server and two virtual servers would not boot after the new kernel was installed. Something went wrong during the install process and messed up the grub configuration on two of the machines, I was able to repair these with the boot-repair disk from sourceforge. But a third machine was - [Server Failed to Boot](https://www.eskimo.com/2021/06/server-failed-to-boot/) - One physical server that hosts among other things the mail spool and server failed to boot. I have to drive down to the co-lo to reboot it so service is going to take longer than expected to restore. - [MariaDB Downgraded to 10.5](https://www.eskimo.com/2021/06/mariadb-downgraded-to-10-5/) - We were forced to downgrade MariaDB back to 10.5 because 10.6.1 deprecated some functionality required by NextCloud. I have filed a bug report with Nextcloud so they can be aware their code is out of date and hopefully fix it. - [DoS Attack](https://www.eskimo.com/2021/06/dos-attack-2/) - I've got about half a dozen of the major sources blocked and while there is still some packets dribbling through from others they are not enough to cause network or router overload so should not further impact network performance. - [Kernel Upgrades Completed](https://www.eskimo.com/2021/05/kernel-upgrades-completed-10/) - The kernel upgrades are completed and all NFS and NIS connections have been verified. The upgrade was relatively uneventful. Everything is back up. I am in the process of upgrading vps3 because the version of Devuan has reached end of life and it no longer has repositories from which to receive security updates. - [Phishing Scam](https://www.eskimo.com/2021/05/phishing-scam-4/) - If you get e-mail like this: And your user@eskimo.com is still using old version, Please tap the blue button below to upgrade your mailbox to the latest version, and get 25GB free data space with a more organized mailbox to avoid deactivation. It is a scam. You can forward it to spamtrap@eskimo.com - [Kernel Upgrades Tonight 11PM-Midnight Pacific Daylight Time](https://www.eskimo.com/2021/05/kernel-upgrades-tonight-11pm-midnight-pacific-daylight-time/) - I hope to do kernel upgrades tonight from 5.12.4 to 5.12.8, mostly a bug fix upgrade. Because there is no change in major point release, hopefully it will go smoothly. This will result in approximately 10 minute outages of various services during this time frame, this includes shell servers, virtual private servers, web - [Vps3, 5, and 7 Maintenance](https://www.eskimo.com/2021/05/vps3-5-and-7-maintenance/) - I'm going to take vps3, vps5, and vps7 down after 11PM and before 2AM for maintenance (to image, a form of backup) for about 1/2 hour each tonight. - [Fedora, Centos7, Scientific7, JuLinux Completed](https://www.eskimo.com/2021/05/fedora-centos7-scientific7-julinux-completed/) - Maintenance on Fedora, Centos7, Scientific7, and JuLinux has been completed. - [Fedora, Centos7, Scientific7, JuLinux Maintenance](https://www.eskimo.com/2021/05/fedora-centos7-scientific7-julinux-maintenance/) - All of these machines will be going down for 15-45 minutes this afternoon to image (a form of backup). They will be taken down one at a time and I will try to avoid / delay those actively in use. - [Network Backbone Maintenance June 3rd 12AM-5AM](https://www.eskimo.com/2021/05/network-backbone-maintenance-june-3rd-12am-5am/) - ISOMEDIA Service Affecting Network Maintenance 6/3/2021 On Thursday, June 3rd, beginning at 12:01 AM PDT and continuing until 05:00 AM PDT, engineers will be performing maintenance on the network backbone ring. This maintenance has the potential to be service affecting, with the possibility of multiple periods of up to five minutes with limited connectivity. ISOMEDIA - [Spam Filtering Issues](https://www.eskimo.com/2021/05/spam-filtering-issues/) - Recently we've seen huge floods of spam and more disturbing scams and viruses, that are not yet caught by clam-antivirus, from providers that either market to spammers but also have a few legitimate customers, or providers that are simply hacked to death. Affected sites include Digital Ocean (lots of ".tech" spam), Amazones (random spam, - [Mail Server Work Complete](https://www.eskimo.com/2021/05/mail-server-work-complete/) - I've completed work on the mail server. To get it to not hang, I had to remove all the xorg-xserver-video drivers for hardware that wasn't present (which is all of them, it's a virtual machine) and that made it happy, no longer hangs when rebooting. While I was in there I removed some - [Mail Server Interruptions 11PM tonight Unknown Duration](https://www.eskimo.com/2021/05/mail-server-interruptions-11pm-tonight-unknown-duration/) - I will be performing multiple reboots of the client mail server tonight after 11pm to troubleshoot the issue that causes it to hang during a systemd initiated reboot. This is a problem with the new Hirsute release of Ubuntu but I managed to fix my workstation by removing a bunch of unnecessary cruft and - [Mail Server Repaired](https://www.eskimo.com/2021/05/mail-server-repaired/) - I've undone the majority of new bugs introduced by Ubuntu in their Hirsute Hippo release. It took quite a while to chase some of them down. I normally avoid using short term releases but dovecot had some serious problems in 20.04, so upgraded to groovy to resolve, and groovy was a clean update, unlike - [Mail](https://www.eskimo.com/2021/05/mail-16/) - I've identified the issue with mail, but I've managed to hang the physical server in an attempt to fix and it is going to require a drive to the co-location facility so things may be broken for the next 1-2 hours. - [Postfix](https://www.eskimo.com/2021/05/postfix/) - Last night's upgrade broke something in the postfix configuration of the mail client server used for sending mail. Unfortunately it is giving only an extremely generic error making identifying what it is difficult. It claims the server is misconfigured yet postfix check shows no errors. Argh! I am working on it. - [Tonight's Maintenance Completed](https://www.eskimo.com/2021/05/tonights-maintenance-completed/) - All Debian based kernels were upgraded to 5.12.4. This took an hour and 15 minutes to complete on the physical server hosting the web server because the iomemory drivers did not want to compile under 5.12.4. After much pulling of hair I found that it was a libc6 version mismatch between the machine - [Tonight's Upgrades](https://www.eskimo.com/2021/05/tonights-upgrades/) - The web server will be down longer than normal after tonight's kernel updates because I have to recompile a driver used with the flash drive for the newer kernel. The client mail server, mail.eskimo.com, will be intermittently available for several hours after the kernel update as I will also be performing an - [Kernel Upgrade Friday 11PM PDT](https://www.eskimo.com/2021/05/kernel-upgrade-friday-11pm-pdt/) - We will be doing kernel upgrades on all of our servers. Expected time frame 11PM-11:30PM with perhaps a straggler machine or two. We've had a lot of systemd issues recently that has made startup less than 100% reliable. This will affect all of our shell customers and web hosting customers as well as - [Mail from Outlook](https://www.eskimo.com/2021/05/mail-from-outlook/) - Expect mail issues from outlook.jp and perhaps outlook in general, I do not know to what degree they share services, but outlook.jp appears hacked to death recently and we've been receiving a lot of phishing scams from it. I'm blocking scam addresses as I become aware but this sort of thing will eventually cause - [Outgoing Mail Fixed](https://www.eskimo.com/2021/05/outgoing-mail-fixed/) - Outgoing mail is fixed except it is still shy on memory. I meant to reboot last night to increase memory but issues with Debian kept me preoccupied. I will reboot mail later tonight to double the memory allocation. - [Outgoing Mail](https://www.eskimo.com/2021/05/outgoing-mail-2/) - I made some changes to the client mail server yesterday in an attempt to relieve some memory overload issues without rebooting and in the process broke mail in a way that it is causing mail to get stuck in queue. I'm working on correcting this. Your mail is not lost and will get sent - [Debian Back up](https://www.eskimo.com/2021/05/debian-back-up/) - Debian is back up, with home directories and mail spool this time. - [Debian - Will Be Back at 10:30PM](https://www.eskimo.com/2021/05/debian-will-be-back-at-1030pm/) - I made an error in editing the /etc/fstab to fix the swap partition and edited out the NFS mounts for home directory and mail spool. I've fixed that but I'm making a backup of the fixed machine before I bring it back into service. - [Debian Restored](https://www.eskimo.com/2021/05/debian-restored/) - Debian is restored to service and available. - [Mail](https://www.eskimo.com/2021/05/mail-15/) - I am planning on rebooting mail client server around midnight tonight in order to increase memory allocation because there were still a few failures resulting from being unable to allocate enough memory. - [Debian](https://www.eskimo.com/2021/05/debian-8/) - Debian will be back up in about an hour. I identified the problem as a bug in systemd-sysusers which causes it to hang if NIS is used with containers and avahi is not present. Avahi is not present because I do not want it "discovering" potentially insecure machines on the network and prefer - [Debian](https://www.eskimo.com/2021/05/debian-7/) - Even after restoring from backup, debian still has problems, please use one of the other Debian based servers such as Ubuntu, Mint, Julinux, or Zorin while I sort this out. If there is an application on Debian that you need and is not present on one of these other servers, please e-mail support@eskimo.com - [Debian - Restoring from Backup](https://www.eskimo.com/2021/05/debian-restoring-from-backup/) - Seem to be going down one rabbit hole after another. I think somehow the system image for Debian has gotten severely corrupted, so I am restoring it from backups made on May 3rd, then will apply updates and bring it current. I expect Debian to be operational around 1700 Pacific Daylight Time. - [Debian Emergency Maintenance](https://www.eskimo.com/2021/05/debian-emergency-maintenance-2/) - Found cause of mounting issues was disk controller host was changed from Virtio to SATA, I did this intentionally on Mail but not on Debian so not sure how it got changed, but changing it back corrected the swap mounting issue however the Volatile file and directory creation issue re-emerged even with older systemd, - [Debian Emergency Maintenance](https://www.eskimo.com/2021/05/debian-emergency-maintenance/) - With an older systemd, Debian did not fail at that same spot, but there does seem to be a UUID issue with one of the partitions so will require further reboot after some tweaking. - [Debian](https://www.eskimo.com/2021/05/debian-6/) - Debian didn't exactly crash, when it booted it did not mount tmpfs on /tmp, this is an in memory file system used for temporary files for speed and efficiency. Many hours later it mounted it and this broke systemd that needs access to some of those files. I do not know what - [Kernel Upgrades Completed](https://www.eskimo.com/2021/05/kernel-upgrades-completed-9/) - The kernel upgrades went mostly smooth except for the client mail server. Had issues with systemd timing out before it mounted all the disk partitions. This is a virtual machine configured to use a physical partition on a RAID10 device on this host computer and using virtio disk. This has not been a problem - [Kernel Upgrade](https://www.eskimo.com/2021/05/kernel-upgrade/) - We will be performing a kernel upgrade tonight between 11pm and midnight. If all goes well it should be concluded by 11:30pm. Going over the change log, I do not expect it to fix a particular bug that is affecting a couple of our virtual private servers but it does address a bug - [Debian, Centos8, and vps1 Maintenance Completed](https://www.eskimo.com/2021/05/debian-centos8-and-vps1-maintenance-completed/) - Debian, Centos8, and vps1 Maintenance Completed - [Vps1 Maintenance](https://www.eskimo.com/2021/05/vps1-maintenance/) - I am taking vps1.eskimo.net down for approximately 1/2 hour to image. - [Debian Maintenance](https://www.eskimo.com/2021/05/debian-maintenance-8/) - Taking Debian shell server down for about 45-60 minutes to image. Please use one of the other debian based servers in the interim, such as ubuntu, mint, julinux, zorin, or mxlinux. - [Centos8 Maintenance](https://www.eskimo.com/2021/05/centos8-maintenance/) - Taking centos8 shell server down for approximately 1/2 hour for imaging. - [Zorin Maintenance](https://www.eskimo.com/2021/05/zorin-maintenance-6/) - I will be taking the Zorin shell server down for approximately 1/2 hour for imaging. - [Centos-Stream and MxLinux Maintenance Completed](https://www.eskimo.com/2021/05/centos-stream-and-mxlinux-maintenance-completed/) - Maintenance on Centos-Stream and MxLinux is complete. These are back in service. - [Manjaro Maintenance](https://www.eskimo.com/2021/05/manjaro-maintenance-2/) - Taking the Manjaro shell server down for 1/2 hour approximately for imaging. - [MxLinux Maintenance](https://www.eskimo.com/2021/05/mxlinux-maintenance-6/) - Taking the MxLinux shell server down for about 1/2 hour for maintenance. - [CentosStream Maintenance](https://www.eskimo.com/2021/05/centosstream-maintenance/) - Going to be taking the CentosStream shell server down for about 1/2 hour to image it. - [Kernel Upgrades Completed](https://www.eskimo.com/2021/04/kernel-upgrades-completed-8/) - Kernel upgrade for virtual private servers 1-8 and the host machine, "ice", have been completed. All are now running 5.10.33. Hopefully stability will be better. - [Kernel Upgrades on Ice and VPS's only](https://www.eskimo.com/2021/04/kernel-upgrades-on-ice-and-vpss-only/) - We have been having some stability issues on the machine that serves private virtual servers vps1-vps8 since kernel 5.8 where there were some major changes that significantly improved load and responsiveness and hence the unwillingness to revert to 5.4. I have filed a bug report on various issues that have shown up in crash - [Kernel Upgrade Completed](https://www.eskimo.com/2021/04/kernel-upgrade-completed/) - Maintenance is completed and everything is back up and running. We had some issues with both the mail server and the web server not automatically mounting all disk file systems and swap partitions so I had to go around to each machine and check these things. https://friendica.eskimo.com/, https://hubzilla.eskimo.com, and https://nextcloud.eskimo.com/ are all operational. - [Kernel Upgrades Friday 4/22/2021 11PM Pacific Daylight Time](https://www.eskimo.com/2021/04/kernel-upgrades-friday-4-22-2021-11pm-pacific-daylight-time/) - We will be performing another round of kernel updates on April 22, 2021 starting at 11pm. Also at this time we'll be taking mail down for some additional time in order to increase the amount of RAM. This should only take a few additional minutes. The logs are indicating some memory pressure during peak - [NextCloud Moved](https://www.eskimo.com/2021/04/nextcloud-moved/) - I made the announcement some time ago that I was going to move nextcloud. This is because in it's old location it's .htaccess conflicted with WordPress .htaccess when it came to well-known web addresses. To resolve this, I gave NextCloud it's own hostname. The OLD URL was: https://www.eskimo.com/nextcloud/ The NEW URL is: - [Kernel Upgrades Completed](https://www.eskimo.com/2021/04/kernel-upgrades-completed-7/) - Kernel upgrades completed without any major incidents at 11:38 PM. Everything was mostly up by 11:15 PM, the remaining time was just checking NFS mounts and such. https://friendica.eskimo.com/ and https://hubzilla.eskimo.com/ are available. - [Kernel Upgrades 4/16 11PM PST](https://www.eskimo.com/2021/04/kernel-upgrades-4-16-11pm-pst/) - I am planning another kernel upgrade this Friday April 16th starting at 11pm. If all goes well it should be concluded by 11:30pm. If it does not, it may be as late as 12:30pm. These upgrades will affect virtually all of our services including https://friendica.eskimo.com/ and https://hubzilla.eskimo.com/ Fediverse social media sites. There - [Ice Rebooted](https://www.eskimo.com/2021/04/ice-rebooted/) - One of our servers rebooted at 2:30AM with no oops, crash dump, or error. I thought I had crash-dump enabled but did not, so corrected that, also upgraded to the latest point release of the kernel. This server has most of the private virtual machines so is the reason your virtual machines would - [Kernel Upgrades Completed](https://www.eskimo.com/2021/04/kernel-upgrades-completed-6/) - It was not a clean upgrade. Grub got corrupted on one machine, one server didn't properly export the exports until I manually restarted the nfs server. There were quite a few broken NFS mounts. So, some things were back up at 11:16, and the broken systems were gradually restored with restoration completed at 12:07. - [Kernel Upgrades](https://www.eskimo.com/2021/04/kernel-upgrades-3/) - I plan to upgrade kernels on all the servers tonight starting at 11PM. Upgrades should be concluded by midnight. This will result in outages lasting 10-20 minutes for each physical server and perhaps another ten minutes for virtual machines on that server depending upon whether NFS and NIS properly bind after reboots or not. - [Crash & Reboot](https://www.eskimo.com/2021/04/crash-reboot/) - We had one of our physical servers, which mainly services virtual private servers, crash and reboot this morning. The crash was caused by snapd attempting to apply livepatch to our kernel which is not a Canonical kernel. Livepatch was disabled in the software properties but it tried anyway. I've removed snapd from - [Mint is Up - 20.1](https://www.eskimo.com/2021/03/mint-is-up-20-1/) - Mint is back online and is now Mint 20.1, however it is a fresh install and there is not a lot of software installed yet. If there is something you'd like that is missing, please go to our website https://www.eskimo.com/ and select Support -> Tickets and generate a trouble ticket with your request. - [Mint](https://www.eskimo.com/2021/03/mint-7/) - I attempted to upgrade Mint again today and again it failed, this time leaving the machine in an unusable state where python3 wasn't properly installed and python3 is needed for apt so I could not fix it. I could have restored from backups but instead opted to try a fresh install of Mint - [Web Trouble](https://www.eskimo.com/2021/03/web-trouble/) - Our web server was largely unavailable between 12:30AM and about 4AM on March 30th owing to one of the system database tables in MariaDB getting corrupted. I could have restored everything from backups in a shorter period of time but instead loaded backups, rescued the hurt system table structure from backups, then returned the - [Network OK](https://www.eskimo.com/2021/03/network-ok/) - Nobody at Isomedia has gotten back to me but the packet loss has resolved so I assume someone found what was wrong or if it was a DDoS attack, it's ended. At any rate it's cleared up. - [Network Trouble](https://www.eskimo.com/2021/03/network-trouble/) - We are seeing significant packet loss from various locations to our servers in the Bellevue CoLo facility. I have determined the issue to be between Isomedia's Seattle ring router and Bellevue core router and generated a trouble ticket. We are not seeing heavy traffic to our facility but it may be another customer in - [Centos-Stream Broke](https://www.eskimo.com/2021/03/centos-stream-broke/) - Centos-Stream is broken at present. I tried to install the current openssl on it and unfortunately sshd was built against 1.1.1b which has some different symbols than 1.1.1k so does not work. Pulling it out also didn't fix for reasons I do not understand so doing a fresh install. - [OpenSSL and Kernel Upgrades mostly completed](https://www.eskimo.com/2021/03/openssl-and-kernel-upgrades-mostly-completed/) - With the exception of Redhat based machines, all updates are completed. I am going to have to build openssl for the Redhat machines because Redhat seems to be ignoring the openssl exploits so there may be some reboots later this evening of Centos7/8/stream, Fedora, and Scientific7. - [Kernel, Openssl Upgrade / Reboots Tonight](https://www.eskimo.com/2021/03/kernel-openssl-upgrade-reboots-tonight/) - I do not normally do kernel updates mid-week, I prefer to wait until Friday on the off chance something goes horribly wrong, to provide the most time to recover before the business week. However, a serious vulnerability has been discovered in openssl and I'm going to have to reboot all the machines just - [Why Friendica and Hubzilla](https://www.eskimo.com/2021/03/why-friendica-and-hubzilla/) - Farcebook does not tell me WHICH of their policies I've violated, though they seem to think this is worth a 90-day ban on posting or commenting. All I was trying to do is provide some historical perspective to what is happening in Amerika today. This is an example of why I created https://friendica.eskimo.com and https://hubzilla.eskimo.com/ - [Maintenance This Weekend](https://www.eskimo.com/2021/03/maintenance-this-weekend/) - I may or may not do kernel upgrades Friday starting at 11PM, there are two new releases since the last but I'm waiting to see if this is stable or not. If not will do otherwise I'm going to hold off on that until the following weekend. The other thing I will be - [Manjaro Disk Resize Completed](https://www.eskimo.com/2021/03/manjaro-disk-resize-completed/) - The manjaro disk image resizing operation is completed. It is back up and available. - [Manjaro Maintenance](https://www.eskimo.com/2021/03/manjaro-maintenance/) - I am going to be taking Manjaro down for a while this afternoon. I ran out of space during an upgrade and need to increase the size of disk image. - [Mail / Upgrades](https://www.eskimo.com/2021/03/mail-upgrades/) - After servers were rebooted, postfix did not start ont he mail server and I did not notice before I went to bed. This affected the ability to send outgoing e-mail. - [Kernel Upgrade Finished](https://www.eskimo.com/2021/03/kernel-upgrade-finished/) - Kernel upgrades went smoothly. My oldest son Carl, was visiting so I didn't post this earlier, but only had two machines not boot up properly, mxlinux and the web server, and a reboot was all it too for them. ALL NIS bindings and NFS mounts happened correctly the first time around so it feels - [Kernel Upgrade 3/23/21 11pm PST](https://www.eskimo.com/2021/03/kernel-upgrade-3-23-21-11pm-pst/) - I am going to be upgrading kernels on most of our servers starting at 11PM. If things go smoothly we should be done by 11:30PM however the last round we had a couple of machines that had a corrupted Grub configuration that had to be fixed so it took a bit longer. - [Hubzilla / Web Server](https://www.eskimo.com/2021/03/hubzilla-web-server/) - Hubzilla has caused various grief to our web server but I've finally chased things down to a beta worker module that was very broken. Installed the stable version and now it seems to be behaving. - [Web Server Maintenance](https://www.eskimo.com/2021/03/web-server-maintenance-6/) - I messed up during last night's backup of the web server so need to take it down again tonight, starting about 12:15AM (Mon March 8th) for about 45 minutes to make another backup. - [Nextcloud](https://www.eskimo.com/2021/03/nextcloud-8/) - I will be moving Nextcloud from https://www.eskimo.com/nextcloud/ to https://nextcloud.eskimo.com/ to resolve a conflict between it and Wordpress that prevents some of it's features from working properly. I do not yet know the exact time and date this will happen. - [Web Server Maintenance 3/7/22 1AM](https://www.eskimo.com/2021/03/web-server-maintenance-3-7-22-1am/) - On Sunday 3/7/22, I am going to take the web server down for approximately an hour to image it in it's new configuration. - [Flash Database](https://www.eskimo.com/2021/03/flash-database/) - I managed to successfully install hardware and software. Only drawback is that the ioFusion2 drive in some way conflicted with one of my NIC cards, so I had to move the ethernet back to the Intel controller. The E1000 chipset drivers don't work right with hardware offloading in Linux so it is necessary to - [Database Speed Issues](https://www.eskimo.com/2021/03/database-speed-issues/) - Database speed issues may or may not be resolved tomorrow with maintenance. The preferred way of using this drive is to invoke a feature called Atomic Writes. What that does is it bypasses cache on a write and writes directly to flash memory eliminating the need for the database to do a double - [Fusion I/O Flash Drive Hardware Work Friday March 5th](https://www.eskimo.com/2021/03/fusion-i-o-flash-drive-hardware-work-friday-march-5th/) - The Fusion I/O Flash drive that Amazon first told me would arrive on March 5th, then updated and said it will arrive between March 5th and March 12th, arrived today. I will be installing it on Friday evening. During this time most services will be unavailable for probably about 1/2 hour starting around - [Oracle Linux - Not Going to Happen](https://www.eskimo.com/2021/02/redhat-derived-shell-servers-end-of-life-dates/) - I had looked at Oracle Linux as a possible replacement for CentOS down the road, however, Oracle is too broken. First, x2go will not work because of an incomplete perl environment, and second, it is tied to 3.10 kernel which is not safe on modern Intel CPUs and lacks many features of modern kernels. - [Kernel Upgrades Completed](https://www.eskimo.com/2021/02/kernel-upgrades-completed-5/) - Kernel upgrades are finally completed. It took abnormally wrong because something went wrong with the kernel install on mail and it ended up with a grub configuration pointing to two ancient kernels that no longer exist and thus it was unable to boot. I was able to fix the grub configuration with boot - [Kernel and Hardware Upgrades](https://www.eskimo.com/2021/02/kernel-and-hardware-upgrades/) - I am planning yet another kernel upgrade tonight starting at 11pm as there is an issue with the kernel taking memory from the wrong end of a linked list resulting in poor guest performance. Since we use VM's to implement most of our services, guest performance is important. If this goes as smoothly - [PCIe SSD with Atomic Writes Located](https://www.eskimo.com/2021/02/pcie-ssd-with-atomic-writes-located/) - I found one of MariaDB's recommended drive model on Amazon for a very reasonable price and ordered. I don't know if my motherboard will be able to boot from it because I don't know if the BIOS knows about it, but it should be compatible from a hardware standpoint and Linux talks directly bypassing - [Need PCIe SSD with Atomic Writes for MariaDB](https://www.eskimo.com/2021/02/need-pcie-ssd-with-atomic-writes-for-mariadb/) - I recently turned up a Hubzilla in addition to the Friendica Fediverse social media site here. Sunday it seemed to be okay but come Monday the server was seeing loads of around 150 or so at which point a number of web apps turned themselves off. Also it was exhausted on connections which made - [Maintenance](https://www.eskimo.com/2021/02/maintenance-4/) - I am going to be taking our webserver down at midnight to backup and then to re-partition the disk so that I can create a separate partition for mariadb tables so I can better optimize it for performance. This is pacific standard time. It may be down for several hours. - [Wave Cable E-mail](https://www.eskimo.com/2021/02/webserver-mariadb-database/) - Wave cable uses a company called Synacor to provide their e-mail. Synacor is allowing spammers to use their servers. We have had this problem ourselves in the past with hacked accounts being used to spam. We have solved it by installing fail2ban to limit brute force password attacks and postfwd to force a - [Hubzilla](https://www.eskimo.com/2021/02/hubzilla/) - New Federated Social Media Hubzilla We've added a new federated social media platform called "Hubzilla", https://hubzilla.eskimo.com/. It is similar to Friendica https://friendica.eskimo.com/ in that it is a federated social media platform but it has many more capabilities including channels that are similar to groups or blogs or user pages in Facebook but federated across - [Kernel Upgrades Completed](https://www.eskimo.com/2021/02/kernel-upgrades-completed-4/) - Kernel upgrades are completed. All NFS and NIS relationships verified operational. This was the fastest (15 minutes to boot, 9 minutes to check) kernel upgrade yet, all NFS and NIS relationships established themselves successfully first time around. Only problem I ran into is postfix failed to start on mx2 but started fine manually. - [Kernel Upgrades Friday Feb 19th 11pm-Midnight](https://www.eskimo.com/2021/02/kernel-upgrades-friday-feb-19th-11pm-midnight/) - I plan to do kernel upgrades between 11pm and midnight on Friday Feb 18th. Outages of individual services should not last more than about 15 minutes, most things should be back up by 11:30 though some NIS or NFS issues may not be resolved until midnight or so. - [Phishing Scams](https://www.eskimo.com/2021/02/phishing-scams/) - Customer Asks: -- Robert is this from you? style seems wrong... and there was no heads up (unless I missed it) -------- Forwarded Message -------- Subject: New Message Received Date: Fri, 12 Feb 2021 12:18:41 +0100 From: Eskimo.com Notification --------- End Relevant Part of Forwarded Message --------- I'm sending this to everyone because these - [Kernel Upgrade Rescheduled Tonight 11pm-Midnight](https://www.eskimo.com/2021/02/kernel-upgrade-rescheduled-tonight-11pm-midnight/) - I had planned on doing a kernel upgrade Wednesday night but I got things ready sooner than I expected so I am going to do this tonight starting at 11pm rather than Wednesday. I expect all servers to be rebooted before midnight though it may take slightly longer to fix any broken NFS / - [Kernel Upgrade Wednesday 11pm-Midnight](https://www.eskimo.com/2021/02/kernel-upgrade-wednesday-11pm-midnight/) - I expect to be doing another kernel upgrade this Wednesday. These are fixing only minor bugs but I prefer to not allow any known exploits to remain. I know the NSA and many other bad players monitor and develop exploits for kernel bugs as soon as they become known so I prefer to minimize - [How I Came to run a Friendica Node...](https://www.eskimo.com/2021/02/how-i-came-to-run-a-friendica-node/) - I am 62 years old now, but my arrival at running a friendica node really started before friendica or even the Internet or home computers were a thing. When I was three years old, I had a bad bout of pneumonia and was hospitalized for a couple of weeks. During that time I was in - [VNC on Mint, Zorin](https://www.eskimo.com/2021/02/vnc-on-mint-zorin/) - VNC is working on Mint and Zorin. - [VNC Fedora and Julinux](https://www.eskimo.com/2021/02/vnc-fedora-and-julinux/) - VNC is now working on Fedora and Julinux. - [VNC debian centos7](https://www.eskimo.com/2021/02/vnc-debian-centos7/) - VNC is now working on debian and centos7. - [Ubuntu VNC working](https://www.eskimo.com/2021/02/ubuntu-vnc-working/) - VNC is now working on Ubuntu, RDP still broken. - [rdp / vnc broken across the board](https://www.eskimo.com/2021/02/rdp-vnc-broken-across-the-board/) - The problem isn't limited to the redhat machines, it seems EVERYONE pushed out a new version of rdp without bothering to test and the new version expects a newer version of openssl than that which is supplied. So to get it working again I'm going to have to recompile openssl on all the machines. - [Last Night's Maintenance Mint and Mail](https://www.eskimo.com/2021/02/last-nights-maintenance-mint-and-mail/) - I fell asleep while still working on system maintenance which resulted in some problems with sending mail, incoming mail, and mint. I apologize, went past what caffeine could do. - [rdp / vnc on Centos7/8/Stream/Fedora Unavailable](https://www.eskimo.com/2021/02/rdp-vnc-on-centos7-8-stream-fedora-unavailable/) - RDP and VNC (which depends upon RDP) is unavailable on Centos7, Centos8, CentosStream, and Fedora. The reason for this is that they sent an upgrade to the RDP package that was built against openssl that is newer that the version that exists in the repository so it fails. - [Maintenance Work Tonight](https://www.eskimo.com/2021/02/maintenance-work-tonight-2/) - Later this evening, possibly after midnight, I will be taking a number of servers down for a short time to audit packages. One of our customers private virtual server crashed today and after rebooting was consuming excessive CPU. An audit of the package system showed there were many missing and obsolete packages even - [Maintenance Completed](https://www.eskimo.com/2021/01/maintenance-completed-26/) - Tonight's maintenance is complete. - [Reboot Wednesday 11pm PST](https://www.eskimo.com/2021/01/reboot-wednesday-11pm-pst/) - I am planning a server reboot on Wednesday January 27th at 11pm. I expect all services will be restored by 11:30pm. This is for a kernel upgrade from 5.10.4 to 5.10.10. It fixes a number of bugs that are minor and thus far haven't affected us and probably won't but prefer to have - [pam-abl used for DoS attack.](https://www.eskimo.com/2021/01/pam-abl-used-for-dos-attack/) - Pam_abl or libpam-abl is a pam module used to black list users or IP addresses that repeatedly fail authentication. It serves the same function as fail2ban but has a flaw that it can be used to deny service to a user by repeatedly trying passwords for that user from different IP addresses. It was - [Mail and other Login Difficulties](https://www.eskimo.com/2021/01/mail-and-other-login-difficulties/) - We are being hit with such heavy brute force password guessing attempts that it has triggered pam_abl module on a number of hosts, a situation I have not previously encountered, and I am trying to figure out how to reset it and set it high enough that fail2ban will trigger first. - [Eskimo North's History](https://www.eskimo.com/2021/01/eskimo-norths-history/) - When I was around five or six years old, I visited a cousin and he had a low powered AM radio transmitter that would allow him to broadcast to radios nearby. I was fascinated. A few years later, around 4th grade, I took a real interest in electronics and proceeded to read everything I - [New Kernels](https://www.eskimo.com/2021/01/test2/) - I've compiled and placed online the latest mainstream Linux kernels 5.11rc4 and 5.10.9. The 5.11rc4 is only available in client form since it is still a release candidate it is not really suitable for server use yet. When the official release comes out I will place a server version. The kernels can be - [Test](https://www.eskimo.com/2021/01/test-3/) - Test posting. - [Web Server](https://www.eskimo.com/2021/01/web-server-7/) - Last night I managed to repair the PHP environment on our web server except that PHP 8.x is presently uninstalled, it wasn't the default so this should only affect testing. Tonight I will re-install and then take the web server down for about 45 minutes to back it up. - [test](https://www.eskimo.com/2021/01/test-2/) - test - [Admin Stupidity](https://www.eskimo.com/2020/12/admin-stupidity/) - I accidentally deleted my /home directory (what Windows folks call folder) on my workstation yielding it being mostly useless. I am in the process of restoring from backups but I will be unable to do most of the usual stuff while this process is under way. - [Reboots Completed](https://www.eskimo.com/2020/12/reboots-completed-12/) - Reboots are completed. NFS appears to be functioning. Hopefully this kernel will be stable. - [5.10 kernels available](https://www.eskimo.com/2020/12/5-10-kernels-available/) - If you run a Debian based distro (Ubuntu, Debian, Mint, MxLinux, Zorin, Julinux, etc) and would like to try 5.10, I have made .deb packages available in https://www.eskimo.com/kernel/linux-5.10-tickless/[client|server|/...deb Choose client if low latency is the most important thing, typically used in a workstation, gaming, or video production environment, and server if throughput is - [Reboots Tonight 11pm-midnight](https://www.eskimo.com/2020/12/reboots-tonight-11pm-midnight/) - I will be rebooting all the servers tonight starting around 11pm, should be concluded by midnight, to install a new kernel, 5.10.1. Hopefully it will work and one boot will be enough. As with 5.9 which never really worked, there have been significant changes to NFS which we make heavy use of here. - [vps2, vps3, vps4 completed, Maintenance now on vps1, mxlinux, and zorin](https://www.eskimo.com/2020/12/vps2-vps3-vps4-completed-maintenance-now-on-vps1-mxlinux-and-zorin/) - Maintenance completed on vps2, vps3, and vps4. Now I will be taking vps1, mxlinux, and zorin down for about 1/2 hour each for imaging (a form of backup). - [Maintenance Outage](https://www.eskimo.com/2020/12/maintenance-outage-3/) - I'm going to be taking vps2, vps3, and vps4 out of service for about 20-30 minutes each tonight for imaging (a form of backup in which the entire virtual machine image is copied). - [Mail Server](https://www.eskimo.com/2020/11/mail-server-9/) - We have had a problem recently with dovecot corrupting it's index files. I do not know exactly when this started. The version that Ubuntu provides is 2.3.11. I recently changed the mail client server from keeping things on an NFS mounted partition to local disk as the general consensus within the dovecot community - [Mail Server](https://www.eskimo.com/2020/11/mail-server-8/) - Getting the mail server off of NFS did not stop it corrupting it's dovecot-index files. It is down presently as I'm trying to compile the current version to see if it is improved. - [Mail](https://www.eskimo.com/2020/11/mail-14/) - My last attempt to change the way the mail system is organized failed because the kernel we were using had a bug that caused it to incorrectly read the partition tables. The kernel problem has been corrected. I am going to take mail down for about an hour now to move to igloo - [Mission Aborted](https://www.eskimo.com/2020/10/mail-server-down-for-move-of-physical-hosts/) - What I'd hoped to do with mail didn't work. I've got to do a bit more research before making another attempt. - [Emergency Reboots Tonight](https://www.eskimo.com/2020/10/emergency-reboots-tonight/) - We have had some issues with three different customers accessing e-mail. I was unable to replicate this until tonight. When it did fail for me the failures indicate an NFS problem with the new kernels, consequently on the NFS servers and mail clients I am going to revert to a previously known working kernel - [Drive Taking Errors](https://www.eskimo.com/2020/10/drive-taking-errors/) - One of our machines has a drive that is taking some errors. It completely passes the SMART internal diagnostics but the errors indicate problems finding sector headers which can happen if a machine isn't shutdown properly say during a power outage it can clobber some sector headers. While this can be fixed by - [Digital Ocean Spam / Virus](https://www.eskimo.com/2020/10/digital-ocean-spam-virus/) - We have received a large number of spams containing a virus from Digital Ocean address spaces. We are receiving these exclusively from digital address space. For every one of these I have sent e-mail to their published abuse address, abuse@digitalocean.com and to their NOC at noc@digitalocean.com. I have yet to receive a single - [Unannounced Kernel Upgrade](https://www.eskimo.com/2020/10/unannounced-kernel-upgrade/) - I apologize for the unannounced kernel upgrade this morning but it was done rapidly because a security flaw was discovered in 5.8 and earlier kernels that I wanted to eliminate as rapidly as possible. We are now running 5.9 kernels. This took two rounds last night because my first build of 5.9 was - [Web Server Downtime](https://www.eskimo.com/2020/09/web-server-downtime/) - Sorry for the downtime between 2-4:30AM. Something got corrupted and I was unable to login with x2go, and I had gotten friendica working since the last backup so I did not want to just restore from backups and lose all the work I had done. I had to delete and reinstall about half - [Spam Filtering](https://www.eskimo.com/2020/09/spam-filtering-3/) - I am happy to report that at least one phishing scam I've received after installing the anti-phishing scam plugin was successfully diverted to my spam box so it is at least catching some. I've also added a botnet plugin that looks for IP's of known botnet's and deflects those spams to your spambox. - [Phishing Scams - Spam Filter](https://www.eskimo.com/2020/09/phishing-scams-spam-filter/) - I am sure that you, like myself, are getting tired of being inundated by phishing scams. I have added a new plugin to spamassassin which is designed to catch phishing scams. It works by looking for the URL's of known phishing websites in incoming e-mails. It remains to be seen how effective this - [All Servers Restored to Service](https://www.eskimo.com/2020/09/all-servers-restored-to-service/) - Maintenance is complete. All servers are restored to service. - [Mint, Zorin Restored, Vps7, Manjaro, and Zorin Maintenance](https://www.eskimo.com/2020/09/mint-zorin-restored-vps7-manjaro-and-zorin-maintenance/) - Mint. Scientific7, and Zorin are restored to service. vps7.eskimo.net, manjaro, and uucp going down for maintenance. I expect the outages of these to last 15-30 minutes each. - [Fedora Restored, Scientific7 Maintenance](https://www.eskimo.com/2020/09/fedora-restored-scientific7-maintenance-2/) - Fedora has been restored to service. Scientific7 is being taken down for maintenance. Estimated downtime approximately 1/2 hour. - [Mint Maintenance](https://www.eskimo.com/2020/09/mint-maintenance-5/) - Mint will be going down for maintenance. Expected duration approximately one hour. - [Fedora and Zorin Maintenance](https://www.eskimo.com/2020/09/fedora-and-zorin-maintenance/) - Fedora and Zorin are presently down for maintenance. Expected duration approximately 1/2 hour. - [Mail Issues](https://www.eskimo.com/2020/09/mail-issues-4/) - There were problems with the pop/imap/smtp service this morning after the reboots. At some point between this reboot and the previous, a Ubuntu update overwrote my systemd configuration file for dovecot, the imap/pop3 server. My changes to this file were designed to cause dovecot startup to wait until after the /misc file system, where - [Maintenance Completed](https://www.eskimo.com/2020/09/maintenance-completed-25/) - All systems have been rebooted. All Debian derived systems are now running kernel version 5.8.9, no longer patched as NFS fixes have been incorporated into the base code. All systems have been tested for NFS/NIS mounts and binding. - [Reboots 3AM](https://www.eskimo.com/2020/09/reboots-3am/) - I'll be rebooting all servers around 3AM for a kernel upgrade. Downtime for any given service will be around ten minutes. The total process should take about 1/2 hour. - [Web PHP Outage](https://www.eskimo.com/2020/09/web-php-outage/) - Much of the webserver refused to run PHP code randomly today after I compiled in mod_authnz_external. It should not have done this, should not in theory conflict with any of the other apache mods but obviously it did. I will be doing further experimentation to try to nail down the cause. I have had - [Web Server Outage](https://www.eskimo.com/2020/08/web-server-outage-2/) - The web server outage was caused by myself entering an incorrect DELETE statement into mysql and accidentally screwing up the grant tables. I was unable to correct so restored from backups made last night. - [Webserver Fix + New Feature](https://www.eskimo.com/2020/08/webserver-fix-new-feature/) - Issues with the PHP and Apache installation of our web server have been resolved. In addition a new capability has been added. In the past we had one version of PHP and often upgrades broke existing applications that were not compliant with the most recent version. It is now possible to set PHP versions by - [Censorship](https://www.eskimo.com/2020/08/censorship/) - https://childrenshealthdefense.org/news/chd-holds-press-conference-with-legal-team-and-plaintiff-in-lawsuit-against-facebook-mark-zuckerberg-and-three-of-facebooks-so-called-fact-checkers/ I want my customers to be aware of censorship that is happening on the Internet today. Personally, I find this a very disturbing trend. Facebook, Twitter, Instagram, Linked In, pretty much all the major social media are heavily censoring non-mainstream views and I feel this is a very unhealthy thing for a Democratic society. - [Web Server](https://www.eskimo.com/2020/08/web-server-6/) - I apologize for the web server outage this afternoon. An update to ubuntu installed their apache2 server over mine. My version of Apache is very different from the Ubuntu version. It does not look for the config files in the same place and it has some customizations for performance and security. I - [Server Reboots Completed](https://www.eskimo.com/2020/08/server-reboots-completed/) - This evening's server maintenance has been completed. - [Server Reboots](https://www.eskimo.com/2020/08/server-reboots-11/) - Well about an hour and a half into it, iglulik puked out an NFS related kernel oops, four of the eight knsfd's locked up and the machine ground to a halt. I had to go to the co-lo to reboot and bring it back to a 5.4 kernel along with the other NFS servers. - [Between 11pm - 12pm tonight we will be rebooting all Intel based servers](https://www.eskimo.com/2020/08/between-11pm-12pm-tonight-we-will-be-rebooting-all-intel-based-servers/) - Starting at 11pm, I will be rebooting all of our Intel based servers. I expect this to be completed by midnight. I am doing this to install Linux 5.8, testing has shown it to be a huge performance win. I hope that the NFS stability issues of 5.7 are totally resolved, so far no - [Client Mail Server Maintenance - July 28th 2:10AM - 2:30AM](https://www.eskimo.com/2020/07/client-mail-server-maintenance-july-28th-210am-230am/) - I will be taking the mail server down for imaging for approximately 20 minutes. Imaging is making a backup of the virtual machine which must be quiescent during the operation. - [Maintenance Completed](https://www.eskimo.com/2020/07/maintenance-completed-24/) - This morning's maintenance has been completed. - [Reboots All Servers](https://www.eskimo.com/2020/07/reboots-all-servers/) - All servers (except the old eskimo.com server) will be rebooted tonight shortly after midnight. This is necessary because security updates were installed that require a reboot to be effective. This should be completed within about half an hour. - [Mint 20 Failed](https://www.eskimo.com/2020/07/mint-20-failed/) - An in place upgrade to Mint 20 failed, their script for setting up python had inappropriate syntax for the version of python being installed. A clean install of Mate 20 also failed. Mate Desktop did not work, could not fully install because of package dependency issues, on a fresh system. Mint has - [Mint Upgrade 19.3 to 20.0](https://www.eskimo.com/2020/07/mint-upgrade-19-3-to-20-0/) - An upgrade of the Mint shell server from 19.3 to 20.0 is in progress. Because this is an upgrade involving an upgrade of the underlying Ubuntu system from 18.04 to 20.04, and thus has a greater chance than normal of going south, I am performing a backup of Mint first. This whole - [Maintenance Completed.](https://www.eskimo.com/2020/07/maintenance-completed-23/) - Maintenance on mail, virtual private, and shell servers has been completed. - [Zorin, Mail, Vps1, Centos7, Centos8, Debian, Majaro Maintenance 11pm-1am](https://www.eskimo.com/2020/07/zorin-mail-vps1-centos7-centos8-debian-majaro-maintenance-11pm-1am/) - Tonight starting at around 11pm I will be taking these machines down for maintenance, zorin, centos8, and manjaro, then vps1, centos7, then mail and Debian. Downtime should be about 1/2 hour except for Debian which usually takes about an hour. - [Maintenance Is Completed](https://www.eskimo.com/2020/07/maintenance-is-completed/) - Tonights' maintenance is done. Things did not take as long as I had expected. - [Reboots and Imaging - Maintenance Outage Tonight](https://www.eskimo.com/2020/07/reboots-and-imaging-maintenance-outage-tonight/) - Tonight shortly after midnight I will be rebooting the physical hosts which will reboot everything because recent security updates require a reboot to activate them. This should take about fifteen minutes. Then I will be taking the web server down for about an hour for imaging and then to extend the file system - [Stability - Interim](https://www.eskimo.com/2020/07/stability-interim/) - I have the physical hosts up on 5.4.0 now, this is the kernel that shipped with Ubuntu 20.04. It is not entirely stable on machines which serve as NFS servers which two of these machines do. I have built 5.3.0 kernels which were previously stable but can not boot into them because automatic - [Kernel Issue Emergency Reboot](https://www.eskimo.com/2020/07/kernel-issue-emergency-reboot/) - Sorry I had to do an emergency reboot on servers today. I discovered what caused this morning's crash and it was a bug in the kernel I was running that caused it to not recover resources as processes exited and new ones were created, so it would continue to eat memory until it ran - [Mail Servers](https://www.eskimo.com/2020/07/mail-servers/) - After I installed a newer version of fail2ban on the client mail server, the attackers were no longer successful at overloading the machine as it was able to keep up with changing attacking IP addresses. Now they are attacking our incoming mail servers, and although they are increasing the load on these machines - [Mail Server Attack Status](https://www.eskimo.com/2020/07/mail-server-attack-status/) - I have installed a newer version of fail2ban onto the mail client server. It seems much better able to keep up with attacking IP's and has brought the load down to a more normal range. - [Client Mail Server Under Attack](https://www.eskimo.com/2020/07/client-mail-server-under-attack/) - The client mail server is still under attack. The attack consists of a botnet that is trying to elicit login and password for users by brute force methods using postfix auth. So far a little over 3,000 IP addresses are blocked but fail2ban can't work fast enough to get them all in real time. - [Reboots and DDoS](https://www.eskimo.com/2020/07/reboots-and-ddos/) - Iglulik spontaneously rebooted again this morning, still no idea what is causing this. Mail was under a DDoS attack as of around 9:30 this morning, still ongoing at 10AM, fail2ban is essentially saturated locking out attacking IPs as fast as it can. Load is heavy but server is still functional but a bit - [NFS on Linux](https://www.eskimo.com/2020/07/nfs-on-linux/) - I've had problems with NFS on Linux basically forever, and it became significantly better AND worse under NFSv4. It got better in that locking mostly works under version 4.2, so things like alpine work correctly where the mail spool is NFS mounted. It got worse in that sometimes mounts fail to mount, especially if - [2:30 Crash](https://www.eskimo.com/2020/07/230-crash/) - Around 2:30pm everything spontaneously rebooted. Just prior to that our mail server had been under a DDOS attack by a botnet, fail2ban had been working hard to lock out attacking IPs, still the load on the machine was 800, and while I was trying to chase that everything crashed. When things came back - [Iglulik Status](https://www.eskimo.com/2020/07/iglulik-status/) - Sorry the downtime was longer than anticipated. I ran many stress tests and was not able to get the machine to error, freeze, overheat, or otherwise act up. First, I upgraded the BIOS because experience has taught me that newer Asus BIOS software is generally more stable than old Asus software. On my - [Iglulik Web Outage](https://www.eskimo.com/2020/07/iglulik-web-outage/) - I have reverted Iglulik to an older known stable kernel and still it spontaneously booted last night (and did not start the web server upon recovery) so now I know there is a hardware problem. Tonight shortly after midnight I will be taking this machine down for a while to run some diagnostics - [Iglulik Instability](https://www.eskimo.com/2020/07/iglulik-instability/) - I believe I have located the major source of instability but unfortunately at a sacrifice to performance. I have an Nvidia 210 video card in this machine for the console. It's a very low end card but adequate for that purpose, however, in 2019, Nvidia discontinued driver support so I had to switch - [Iglulik Still Unstable](https://www.eskimo.com/2020/07/iglulik-still-unstable/) - 5.7.7 kernel was still unstable, so was 5.8rc3, but at least with the latter it logged some information that showed some memory allocations failed with the contiguous memory allocater, a new feature recently introduced into the Linux kernel. I am building a new kernel with that disabled, it really isn't required since there - [Iglulik Spontaneous Boot](https://www.eskimo.com/2020/07/iglulik-spontaneous-boot-2/) - Iglulik spontaneously rebooted again tonight, this time on 5.7.7 it made it four days between spontaneous boots but this time I discovered what triggered it so I've got a bug report files with bugzilla.kernel.org and I'm going to give 5.8pre4 a try if it proves semi-stable on my workstation. I normally avoid pre-release kernels - [Iglulik Reboot Tonight](https://www.eskimo.com/2020/07/iglulik-reboot-tonight/) - One of our servers has been unstable on 5.7.6 and rebooted spontaneously twice in the last few days. Oddly, only this server seems to be impacted but it is a newer CPU than the others so it may be a kernel problem specific to this CPU. I am going to reboot into 5.7.7 - [OpenSuse Shell Server to be Discontinued](https://www.eskimo.com/2020/06/opensuse-shell-server-to-be-discontinued/) - I am discontinuing opensuse.eskimo.com shell server because users have been unable to authenticate for several months owing to a broken library in opensuse that incorrectly attempts to originate connections to ypserv on an unprivileged port. I filed a bug report with Suse several months ago, nothing has come of it in the way - [Spontaneous Boot and Resulting Issues](https://www.eskimo.com/2020/06/spontaneous-boot-and-resulting-issues/) - Iglulik spontaneously booted today. NFS partitions did not properly remount on one mail server, this may result in some spam not being properly filtered and mail that should have gone to spam and/or other folders, instead being placed in your INBOX. I apologize for this inconvenience. I have modified systemd unit files - [5.7.6 Kernel Upgrade Completed](https://www.eskimo.com/2020/06/5-7-6-kernel-upgrade-completed/) - 5.7.6 appears to have resolved NFS issues that came into being in 5.7.0 through 5.7.4 and is now the active kernel on our server and debian based systems. - [Mail / Vps 1-7](https://www.eskimo.com/2020/06/mail-vps-1-7/) - Going to take the mail subsystem down for about 1/2 hour to troubleshoot kernel problem as well as vps1-7 virtual private servers. Had planned this for yesterday but problems building kernels delayed. - [Physical Host Boot and Testing Tonight 11pm-1am](https://www.eskimo.com/2020/06/physical-host-boot-and-testing-tonight-11pm-1am/) - This evening I will be booting ice into a new kernel to test NFS. I am working with a Linux kernel developer to debug some kernel nfs server issues that are new to the 5.7.x kernels. I plan to boot into the new kernel around 11pm, then things may or may not be - [Ubuntu Out of Service](https://www.eskimo.com/2020/04/ubuntu-shellx-out-of-service/) - Ubuntu is out of service owing to an upgrade that blew up. It is in the progress of updating still. In the meantime please use one of the other available shell servers listed here: https://www.eskimo.com/services/shells/servers/ Debian, Mint, Julinux, and Zorin are all based on similar code and will have most of the - [Maintenance Down Times](https://www.eskimo.com/2020/06/maintenance-down-times/) - Between 11PM Sunday June 21st and 2AM Monday June 22nd: vps2, vps3, vps4, vps5, and vps7 about 1/2 hour each for imaging. scientific7, uucp, and mx1 about 1/2 hour for scientific7 and about 15 minutes for the other two. mx1 will not be service affecting as mx2 will handle the traffic while - [Mail Server 7:15-8:30PM](https://www.eskimo.com/2020/06/mail-server-715-830pm/) - Webmail, imap, and pop were unavailable from 7:15PM to approximately 8:30PM. I restarted mail to test a new kernel before I restart the large servers tonight. For some reason dovecot automatic start failed and I did not realize it at the time. I got a call just before 8:30, checked, and found - [Reboots Early Saturday](https://www.eskimo.com/2020/06/reboots-early-saturday-2/) - Saturday morning just after midnight Pacific time, I will be rebooting servers into a new kernel. If this kernel behaves properly we will remain on 5.7.4, if not I will need to reboot again to 5.6.0. 5.7.0-5.7.2 has not functioned properly as NFS server. - [Reboot is completed](https://www.eskimo.com/2020/06/reboot-is-completed/) - The reboot did not in and of itself fix the problem but everything restarted did point me at the source so I could fix it. It is completed, everything should be functional again. - [Emergency Maintenance](https://www.eskimo.com/2020/06/emergency-maintenance/) - I have to reboot a machine because the NIS subsystem, which is how authentication information is pushed to all machines, is not working properly on this machine. This will briefly make e-mail unavailable as well as the shell servers centos7, centos8, scientific7, and fedora. - [System Reboots 1-2 AM June 13 2020](https://www.eskimo.com/2020/06/system-reboots-1-2-am-june-13-2020/) - I will be rebooting all servers shortly after 1AM to load a new 5.7.2 kernel, the 5.7.0 kernel was slightly unstable in nfs server applications. - [MxLinux back up](https://www.eskimo.com/2020/06/mxlinux-back-up/) - The shell server, mxlinux.eskimo.com, is again available. It has been upgraded from 18.3 to 19.2. So far 19.2 appears to be MUCH more solid and bug free relative to 18.3. I have yet to find any actual bugs in the implementation (but a hell of a lot of operator errors). I have - [Iglulik Spontaneously Booted about 3AM](https://www.eskimo.com/2020/06/iglulik-spontaneously-booted-about-3am/) - Iglulik spontaneously rebooted about 3AM, this is the second time it has done this and Ice, the server which has /mail spool, has also spontaneously booted once. It appears that 5.7 has some stability issues when NFS file systems are exported. The machines not exporting file systems have been completely stable. I know - [MxLinux Down for Upgrade](https://www.eskimo.com/2020/06/mxlinux-down-for-upgrade/) - I am taking mxlinux down to upgrade from MxLinux 18.3 to MxLinux 19.2. Because of a change in the debian code base upon which MxLinux is based and because they've bastardized Debian beyond the point where normal upgrade procedures can be used, a complete re-install is necessary so it may be down for a - [Found / Fixed Mail Issue](https://www.eskimo.com/2020/06/found-fixed-mail-issue/) - Mail issue was a problem with Dovecot. It did not open the SASL socket for some reason. Restarted and now is okay. - [Mail Issues](https://www.eskimo.com/2020/06/mail-issues-3/) - We are having some problems with the mail sub-system that I am still struggling to understand. It is responding slowly and refusing service to some hosts while permitting it to others and I have not yet been able to determine why. - [Ice Spontaneously booted today](https://www.eskimo.com/2020/06/ice-spontaneously-booted-today/) - Ice, a machine which holds the /mail partition as well as mail, mx2, and some private virtual machines, spontaneously rebooted today. There would appear to be some stability issues in 5.7 yet. I suspect these related to NFS as both ice and iglulik export major NFS partitions used by the rest of the - [Iglulik](https://www.eskimo.com/2020/06/iglulik/) - Iglulik, the host which hosts /home directories spontaneously booted about 3pm today. I do not yet know what caused this. I will need to re-check all the NFS mounts on the other hosts since invariably some of them fail to remount correctly. - [Hosts NFS / NIS Mounts / Binding Verified](https://www.eskimo.com/2020/06/hosts-nfs-nis-mounts-binding-verified/) - Hosts NFS mounts have been checked and NIS bindings have been checked. A few hosts failed to come up completely after reboot. All of these problems have been resolved and all hosts are operational except for OpenSuse. OpenSuse has a problem with a library that breaks NIS. I opened a ticket on this - [Reboots Complete - Still Checking Hosts](https://www.eskimo.com/2020/06/reboots-complete-still-checking-hosts/) - The reboots are completed but I am about an hour behind schedule. Two things set me back. First, SOMETHING installed dnsmasq on my stealth master DNS server. It is a master that is hidden behind a firewall so that hackers can't inject nastiness into it and then it supplies all the secondary servers - [Server Reboots](https://www.eskimo.com/2020/06/server-reboots-10/) - I am planning on rebooting physical hosts which will affect all services tonight starting at midnight. I should be complete by about 12:30, and then another hour or so to check all the servers for proper NFS/NIS mounting/binding which is not 100% reliable under Linux. - [Eskimo SSL Certificate Replaced](https://www.eskimo.com/2020/05/eskimo-ssl-certificate-replaced/) - We have been issued and installed new SSL certificates to correct the problems with the previous certificates caused by the issuer. - [Eskimo SSL Certificate](https://www.eskimo.com/2020/05/eskimo-ssl-certificate/) - We use a site-wide SSL certificate for our web and mail servers which we purchased from RAPIDSSL. About two years ago, they were bought out by an outfit called SECTIGO, and asses apparently decided to shut off the intermediate certificate server even though they still had acquired customers using those certificates and without - [Reboots Done](https://www.eskimo.com/2020/05/reboots-done-4/) - Reboots are completed, all NFS mounts and NIS bindings are checked. As near as I can tell everything is operational again. - [Physical Host Reboots](https://www.eskimo.com/2020/05/physical-host-reboots/) - Tonight I have to reboot physical hosts which various virtual machines and NFS file system hosts that hosts things like your mail spool and home directories. I expect to start this around 12AM and the reboots should be completed by about 12:30 but it will take a few more hours to check all the - [Mail to Comcast](https://www.eskimo.com/2020/05/mail-to-comcast/) - A couple of days ago a customers account here was compromised about two PM and used to send about 40,000 spams out before I shut it down. I managed to delete the majority of them from queue before they were processed but some got through. In response, Comcast black listed out entire domain - [Centos 8 -> Centos Stream](https://www.eskimo.com/2020/05/centos-8-centos-stream/) - CentOS8 has been out for approximately a half year and in that time no Desktop interface has been completely ported. And at this time the current release is 8.1, however, Centos folks seem bent on abandoning Centos 8 and replacing it with a rolling release called Centos Stream similar to Fedora and that seems - [Centos 7 / Scientific 7 Additions](https://www.eskimo.com/2020/05/centos-7-scientific-7-additions/) - Good news for those of you who liked the old Gnome environment on Centos6, Gnome is now available on Centos7 and Scientific Linux 7. It has been, up to this point, unavailable because of the unavailability of gnome-flashback, necessary for remote Gnome Desktop access, but I found an "unofficial repository" that contained it so - [Mail Issues](https://www.eskimo.com/2020/05/mail-issues-2/) - When I 'fixed' some DKIM issues a couple of days ago, I used the same file name for two different functions accidentally and this broke the configuration in ways that affected mail lists. That has been corrected. In addition, I misconfigured the client mail server in a way that permitted open relaying and - [Spam Filtering](https://www.eskimo.com/2020/05/spam-filtering-2/) - I found and fixed two issues with spam filtering that closed a couple of holes, hopefully didn't break anything in the process, but if so please use Support->Tickets to report the issue. First thing I found broken was opendkim checking was not working owing to key retrieval. It appears that opendkim does not - [Mail Spool Server Spontaneously Boot about 19:05 PM](https://www.eskimo.com/2020/05/mail-spool-server-spontaneously-boot-about-1905-pm/) - The server with the mail spool spontaneously rebooted around 19:05 PM. Because NFS re-binding after a crash is less than reliable, some shell servers may not have properly remounted their file systems. It may take me several hours to check these. If you run into a server that hangs when you login, hit 'df', - [KDE Not Available on Centos8](https://www.eskimo.com/2020/05/kde-not-available-on-centos8/) - CentOS8 made the decision NOT to include KDE however, it has been available via the EPEL repository. Today they started upgrading the version on the EPEL respository but it is in a half updated broken state, so for now until they complete this project, KDE is not available on Centos8. I had bad - [OpenSuse Not Available](https://www.eskimo.com/2020/05/opensuse-not-available/) - Opensuse is presently unavailable owing to a problem in a library that breaks NIS authentication. It has been reported, they are aware of it but at this point there is no indication of if/when it will be fixed. - [Mail Repaired](https://www.eskimo.com/2020/05/mail-repaired/) - Mail is repaired. It unfortunately required rebooting pretty much everything because after I recreated the file system and exported it, all the other servers which had it mounted gave "Stale File Handle", and would not let me unmount it without a reboot. But that is all done now, everything is good until it's - [Mail Problems](https://www.eskimo.com/2020/05/mail-problems-2/) - There is a problem with the mail spool file system. It is not a hardware problem but a file system corruption problem that fsck does not detect. I am going to have to copy and rebuild the file system. At the rate files are copying this will take approximately 45 minutes to complete. - [Mail Problems](https://www.eskimo.com/2020/05/mail-problems/) - I am having a problem with the mail spool file server. There is something wrong with the file system that fsck -f -y does not find or correct. I am in the process of troubleshooting but it may be necessary to copy and rebuild in which case it may be down for several hours. - [Web Server Issues](https://www.eskimo.com/2020/05/web-server-issues-2/) - If you received an e-mail about your WordPress site like this: Howdy! Since WordPress 5.2 there is a built-in feature that detects when a plugin or theme causes a fatal error on your site, and notifies you with this automated email. In this case, WordPress caught an error with one of your plugins, WPtouch - [Upgrade of the Web Server Done](https://www.eskimo.com/2020/04/upgrade-of-the-web-server-done/) - Upgrade of the web server to Ubuntu-Mate 20.04 with PHP7.4 has been completed. A lot of things are broken. phpBB3 is broken. If you discover other things please use Support->Tickets to generate a trouble ticket. - [Web Server OS Upgrade In Progress](https://www.eskimo.com/2020/04/web-server-os-upgrade-in-progress/) - Our web server is up but upgrades are in process. This involves restarting many things and eventually a reboot and then probably many things will need to be fixed. - [Upgrade of Physical Hosts Completed](https://www.eskimo.com/2020/04/upgrade-of-physical-hosts-completed/) - It did not go as smooth as I would have liked, but the upgrade of physical host machines to Ubuntu Focal 20.04 has been completed. All machines have been checked for NIS/NFS connectivity. OpenSuse is broken until a library they broke is fixed. We do not have an ETA. - [Physical Hosts Upgrade](https://www.eskimo.com/2020/04/physical-hosts-upgrade/) - Because I have a vehicle tonight, I am attempting to upgrade the physical hosts to Ubuntu 20.04, starting with ice. This will involve a, hopefully, brief interruption of mail and virtual private servers later this evening. - [Encryption Compatibility](https://www.eskimo.com/2020/04/encryption-compatibility/) - Encryption is ubiquitous on the Internet and is necessary to protect your privacy and insure the security of your information. Three areas it is commonly used here, on the web server when you connect to one of our web pages with a web browser, on shell servers, when you connect in with ssh, and - [Denial of Service Attack](https://www.eskimo.com/2020/04/denial-of-service-attack-6/) - We were briefly under a denial of service attack around 5:30PM April 23rd, 2020. - [Encryption - Please Keep Your Software Current](https://www.eskimo.com/2020/04/encryption-please-keep-your-software-current/) - After performing upgrades here, I am getting many reports of people not being able to connect to mail.eskimo.com or to a shell server because of TLS or SSL negotiations failing. This is caused because OLD encryption protocols which have been compromised have been eliminated from the newest software on this end and replaced - [Mail Upgraded to Ubuntu 20.04](https://www.eskimo.com/2020/04/mail-upgraded-to-ubuntu-20-04/) - Mail is upgraded to 20.04, this time successfully but I still have not fixed DKIM / DMARC, will work on that later this evening. - [Mail Upgrade](https://www.eskimo.com/2020/04/mail-upgrade/) - The client mail server 'mail.eskimo.com' is in the process of upgrading again. It failed last night but I think I know what I did wrong. It will be unavailable to send mail or receive mail via imap or pop-3 or webmail until completed later this evening. You may read mail via shell mailers like - [Ubuntu Online](https://www.eskimo.com/2020/04/ubuntu-online/) - Ubuntu is now back online with Focal 20.04 release. I'm still doing a bit of clean-up but it should be stable and usable at this point. - [Mail Upgrade - 2nd Attempt](https://www.eskimo.com/2020/04/mail-upgrade-2nd-attempt/) - I am going to make a second attempt at upgrading the mail server starting at around 6PM tonight. I believe I know what I did wrong to cause it to fail the first time. If all goes well there will be only minimal downtime (a minute or so to reboot). If all doesn't go - [Ubuntu Still Upgrading](https://www.eskimo.com/2020/04/ubuntu-still-upgrading/) - Ubuntu is still in the process of upgrading. Probably another two hours or so to go, around 6PM I am estimating. The 20.04 upgrade is large, over 6280 packages to be upgraded and close to 1000 to install. - [Mail](https://www.eskimo.com/2020/04/mail-13/) - Mail is back up however there is something wrong with either opendkim, opendmarc, or both, so for now these features are temporarily disabled. - [Mail](https://www.eskimo.com/2020/04/mail-12/) - Our mail server, mail.eskimo.com, is temporarily out of service owing to a failed upgrade. I am working on reverting back to the previous software. Estimated restoral time approximately 3AM Pacific Standard Time. - [Iglulik Spontaneous Boot](https://www.eskimo.com/2020/04/iglulik-spontaneous-boot/) - The machine which holds the home directories and web server spontaneously booted this morning about 9:45 AM. After it booted, the NFS mounts on the web server did not mount properly requiring it to be rebooted. As far as I know all services are restored. - [Web Server Interruption / Speed](https://www.eskimo.com/2020/04/web-server-interruption-speed/) - I apologize for the slowdown and short interruption of the web server this afternoon. Normal traffic during the day is around 6 hits/second, but this afternoon it peaked at just under double that and the load crept up pretty high. I took a look at resources and noticed it was running low on - [5.6 Tickless Kernels](https://www.eskimo.com/2020/04/5-6-tickless-kernels/) - Those of you with any Debian based system, Ubuntu, Mint, Debian, MX-Linux, Zorin, Julinux, etc, who might wish to try a 5.6 kernel, I've built tickless kernels. At this point the client kernel is fairly well tested, the server kernel less so but I'm running it now. There are some significant improvements in 5.6 verses - [Kernel Updates](https://www.eskimo.com/2020/04/kernel-updates/) - Over the next week or two, I will be updating system kernels to 5.6 kernels. This provides some additional performance enhancements verses 5.5. It allows server-to-server copies in NFS and it allows peripheral-to-peripheral DMA on the hardware side which should improve the efficiencies of things like disk to disk copies by not requiring the - [Web Server Updated](https://www.eskimo.com/2020/04/web-server-updated/) - Our web server is now running Apache 2.4.43 and openssl 1.1.1f. This mainly fixes minor security issues in both packages. - [Pop Before SMTP Support Discontinued](https://www.eskimo.com/2020/04/pop-before-smtp-support-discontinued/) - We recently implemented DMARC, DKIM, and SPF on all of our mail servers to prevent forgeries, reduce spam, and improve mail delivery reliability. DMARC is incompatible with pop-before-smtp therefore we can no longer support this. It is necessary to authenticate on outbound SMTP connections as well as inbound pop-3 and imap-4 connections. - [Pop Before SMTP No Longer Supported](https://www.eskimo.com/2020/04/pop-before-smtp-no-longer-supported/) - We recently implemented DMARC, DKIM, and SPF on all of our mail servers to prevent forgeries, reduce spam, and improve mail delivery reliability. DMARC is incompatible with pop-before-smtp therefore we can no longer support this. It is necessary to authenticate on outbound SMTP connections as well as inbound pop-3 and imap-4 connections. - [Mail List / Procmail / DMarc](https://www.eskimo.com/2020/03/mail-list-procmail-dmarc/) - I made an error in opendmarc configuration in which I neglected to add localhost, 127.0.0.1, to the list of servers to ignore. This broke mail lists and procmail recipes that remailed incoming mail back out. It also could potentially break .vacation forwarding. This has been corrected. - [Ubuntu Reboot and Backup](https://www.eskimo.com/2020/03/ubuntu-reboot-and-backup/) - I will be taking Ubuntu down later this evening for about 45 minutes to change the Ethernet emulation in the virtual machine because the Intel it is currently set to has driver bugs. These don't materially affect service but generate a lot of annoying messages in the logs. I am also going to image - [Incoming Mail](https://www.eskimo.com/2020/03/incoming-mail/) - I've broken something in our incoming mail servers in the process of implementing DKIM and DMARC. It is causing mail to be stuck in queue and not delivered to your INBOX. I am working on resolving this. Mail is not being lost it is just stuck in queue and should deliver once I - [Pop-Before-SMTP](https://www.eskimo.com/2020/03/incoming-mail-2/) - Pop-before-SMTP is no longer supported. It is not compatible with DMARC protocol. To originate e-mail from a computer outside of our shell servers you must configure your e-mail client for plain password authentication. - [2446.doc Virus](https://www.eskimo.com/2020/03/2446-doc-virus/) - If you get an e-mail with an attachment claiming to be a contract from a company you've never heard of, and the attachment is a ".doc" file, don't open, "2446.doc" is a Word macro Virus. It won't hurt Linux because Librewriter is smart enough to recognize it as such but it will break Windows. - [Yahoo Now Working Both Directions](https://www.eskimo.com/2020/03/yahoo-now-working-both-directions/) - I tried to login to Yahoo again and could not so reset my password again. After doing that mail is now working to/from Yahoo, both directions. Cox.net is now the only site I am aware of that is blocking us. - [Mail Update](https://www.eskimo.com/2020/03/mail-update-4/) - The domains msn.com, outlook.com, microsoft.com. and other microsoft domains, yahoo.com, and aol.com are now all allowing e-mail from us. The only site that I am aware that is still blocking us is cox.net. - [There are some Issues...](https://www.eskimo.com/2020/02/there-are-some-issues/) - There were some issues with several of the shell servers but all NFS and NIS is now connected properly, all servers are operational now. - [Reboots Completed](https://www.eskimo.com/2020/02/reboots-completed-11/) - I have completed rebooting everything that needed to be rebooted. I am still checking that NFS partitions mounted properly and NIS bound properly, but otherwise everything is good, most services should be functioning, mail and web both already checked, only shell servers left to check. - [Saturday Reboots 2-3AM](https://www.eskimo.com/2020/02/saturday-reboots-2-3am/) - I need to reboot some of the servers owing to updates to core libraries and utilities. Downtime should be limited. - [Outlook](https://www.eskimo.com/2020/02/outlook/) - Outlook (Microshaft) declined to de-list even though the problem was "fixed" the day it occurred. I guess this is their new way of attacking Linux. I already joined JMRP, I tried to join SNDS but they do not send the e-mail they say they will so I can't complete the sign up process. - [Hotmail, MSN, Cox, Yahoo](https://www.eskimo.com/2020/02/hotmail-msn-cox-yahoo/) - Just an update, I have applied for de-listing of our servers IP address with all of these providers. I have not heard back from any of them yet. - [Mail to Cox, Yahoo, MSN, and Hotmail](https://www.eskimo.com/2020/02/mail-to-cox-yahoo-msn-and-hotmail/) - We had an issue with a customer whose computer was infected with a virus on February 12th. That was resolved the same day however in that time the virus tried to send itself to honeypot addresses on the SORBS real-time black hole list and so got us on that list. I went through - [Mail Work Completed](https://www.eskimo.com/2020/02/mail-work-completed/) - The maintenance work on mail.eskimo.com has been completed. - [Mail Server Change](https://www.eskimo.com/2020/02/mail-server-change/) - I have changed the IP address of mail.eskimo.com to 204.122.16.222. If you have the IP address set numerically in your client please change it to this address. It is presently also still listening to 204.122.16.4 but this will be discontinued in the future. I will be taking mail down for imaging around 4AM - [Mail Up](https://www.eskimo.com/2020/02/mail-11/) - Finished about an hour ago but forgot to post. The mail client server is back online. - [Website Outage](https://www.eskimo.com/2020/02/website-outage/) - Sorry for the outage this afternoon. A PHP upgrade broke one of the plugins I used on our WordPress based website. If you are using: WP-Optimize - Clean, Compress, Cache (version 3.0.16) On your website, you should remove it as it will break under our current environment. Also, testing with and without - [Mail Fixed Again](https://www.eskimo.com/2020/02/mail-fixed-again/) - There was another milter I forgot about, the clamav milter for virus scanning and it was that which was broken. That is now fixed and mail is functioning again. We are still on SORBS so mail to Yahoo.com and Cox Cable is still a problem but that is a problem with SORBS ticket - [Mail](https://www.eskimo.com/2020/02/mail-10/) - Our client mail server is again rejecting mail with a strange try again later message from a milter. The only milter is dkim so I do not know why it is doing this. I am in the process of troubleshooting. I've already restored the server from backups and it did operate for several hours - [Mail Server Recovered](https://www.eskimo.com/2020/02/mail-server-recovered/) - Mail server is restored from backups. We are still on the SORBS list, their system lost the ticket I created earlier today and trying to create a new one generated an error, so can't say how long it will take to get de-listed but this mainly affects Yahoo and Cox addresses. - [mail.eskimo.com wedged](https://www.eskimo.com/2020/02/mail-eskimo-com-wedged/) - The client mail server is totally wedged and I can not figure out why so I am restoring it from backups. I have saved the spool prior so no mail will be lost. - [Mail](https://www.eskimo.com/2020/02/mail-9/) - Last night I became aware that our outbound mail server mail.eskimo.com had been added to the SORBS real time black hole list. I received no warnings or notifications or spam complaints prior so I had no reason to believe we had an active spammer. Afterwards, I was able to locate enough information - [Mindful Awareness](https://www.eskimo.com/2020/02/mindful-awareness/) - I rarely write or post about things which are not highly relevant to our service here, but I came across this and what he is saying here is so relevant to life in general, I thought it worth sharing: - [Server Reboots](https://www.eskimo.com/2020/01/server-reboots-9/) - I am going to be rebooting all Intel based servers here shortly to install a 5.5 kernel. This kernel should provide some performance improvement, it certainly has done so for my workstation. - [Web Apps Ruptime User Count Fixed](https://www.eskimo.com/2020/01/web-apps-ruptime-user-count-fixed/) - The user count in ruptime was wrong, that has been corrected. - [Zorin restored to service](https://www.eskimo.com/2020/01/zorin-restored-to-service/) - Backups of zorin are completed. It has been restored to service. - [Mint Back Up](https://www.eskimo.com/2020/01/mint-back-up/) - Backups are completed on Mint. It has been restored to service. - [Zorin and Mint Maintenance](https://www.eskimo.com/2020/01/machines-checked-ubuntu-restored/) - Zorin and Mint are now going down to be imaged. Debian, JuLinux, MxLinux, and Ubuntu are alternate Debian derived machines available while these are being backed up. - [Reboots Completed](https://www.eskimo.com/2020/01/reboots-completed-10/) - Reboots are completed. I am still checking to make sure NIS bound and NFS mounted on all servers. Shortly I will be taking down some servers for imaging, specifically Ubuntu will be first. - [Rebooting servers 11pm Jan 23 - 1 am Jan 24](https://www.eskimo.com/2020/01/rebooting-servers-january-23-24/) - I will be rebooting almost all the servers tonight (all of the Intel based machines). If everything goes well, downtime should be minimal perhaps 15-20 minutes but there are no guarantees until it's finished. - [Debian Restored](https://www.eskimo.com/2020/01/julinux-restored/) - Debian is back up and running. - [Fedora Restored, JuLinux Maintenance](https://www.eskimo.com/2020/01/debian-maintenance-7/) - Fedora is restored to service. Taking Julinux down for maintenance. Debian will require about an hour because it is a larger machine. - [Centos8 Up, Taking Fedora down for backup](https://www.eskimo.com/2020/01/centos8-up-taking-fedora-down-for-backup/) - Centos8 is back online. Taking Fedora down for imaging for about 1/2 hour. - [Centos7 back up, Centos8 going down for Maintenance](https://www.eskimo.com/2020/01/centos7-back-up-centos8-going-down-for-maintenance/) - Taking Centos8 down for imaging. - [Centos7 Maintenance](https://www.eskimo.com/2020/01/centos7-maintenance-2/) - I will be taking centos7 down for approximately 1/2 hour to image the server (a form of backup). - [Slackware Server Discontinued](https://www.eskimo.com/2020/01/slackware-server-discontinued/) - I decided to remove the Slackware from the shell servers today. 1) It was not being used, about a month between logins on average and then only brief logins. 2) It was not being maintained. I've seen no updates for a long time. 3) I could not get x2go to work - [Ubuntu SSH Re-installed](https://www.eskimo.com/2020/01/ubuntu-ssh-re-installed/) - I needed to re-install ssh server on ubuntu.eskimo.com because there was a problem with the existing installation that caused it not to re-start automatically upon reboot. The result is the openssh encryption keys will have changed. You can remove the old key from your .ssh/known_servers with: ssh-keygen -f "/home/nanook/.ssh/known_hosts" -R "ubuntu.eskimo.com" - [Centos6 / Scientific Retired](https://www.eskimo.com/2020/01/centos6-scientific-retired/) - The shell servers centos6.eskimo.com and scientific.eskimo.com have reached end of life and so have been retired. You can use any of the servers listed here: https://www.eskimo.com/services/shells/servers/ I personally recommend ubuntu.eskimo.com or debian.eskimo.com. These two have the widest assortment of software available. - [Merry Christmas](https://www.eskimo.com/2019/12/merry-christmas-3/) - [Postfix mail.eskimo.com](https://www.eskimo.com/2019/12/postfix-mail-eskimo-com/) - I was unable to get the locally compiled version to even run so now I am restoring the mail server from an earlier known working backup. This may take up to 30 minutes. - [Postfix on mail.eskimo.com](https://www.eskimo.com/2019/12/postfix-on-mail-eskimo-com/) - Postfix is acting up on mail.eskimo.com, I do not know if this is because of an update or exactly what is wrong. Specifically, it is rejecting some clients even though I have explicitly told it to accept SASL authenticated clients regardless of EHLO status. This is causing issues when sending e-mail from some phones - [Spam](https://www.eskimo.com/2019/12/spam-3/) - The additional spam filtering did not do as much as I had hoped, or for that matter as SpamHaus had suggested that it would. I will continue to look for other ways to reduce spam. - [New Spam Filters](https://www.eskimo.com/2019/12/new-spam-filters/) - We will be implementing some new spam filtering capabilities from Spamhaus that I hope will increase the effectiveness of our spam filters. These consist of a domain reputation black list and an IP based system that will prevent our mail server from being used by spam botnets when credentials are compromised which should - [Client mail server: mail.eskimo.com](https://www.eskimo.com/2019/12/client-mail-server-mail-eskimo-com/) - I apologize for the brief interruption of mail.eskimo.com around 9pm tonight. I have encountered in the past several delays when I went to post mail but I could not find a resource shortage to explain it and I had set no upper bounds on the number of processes postfix could launch so that - [OpenSuse](https://www.eskimo.com/2019/12/opensuse-2/) - OpenSuse.eskimo.com is now available to you again. It is no longer "Leap", it is now "Tumbleweed", a rolling distribution which will keep it current and up to date unlike Leap. - [Linux Library Bug Affecting X2Go](https://www.eskimo.com/2019/12/linux-library-bug-affecting-x2go/) - There is a bug in the mesa-libGL library used in the Debian derived Linux's presently that may either break your ability to connect via x2go or break some desktop functionality. This is because when the screen size is queried the library is incorrectly returning a rectangle of 0x0. I know the developers are aware - [New Alternate Site for Status](https://www.eskimo.com/2019/12/new-alternate-site-for-status/) - Please make note of the following URL: https://pocketnet.app/nanookeskimocom This replaces the page on Facebook for status when our website is down. - [Facebook](https://www.eskimo.com/2019/12/facebook/) - It is hard to say if our Facebook page will remain. I referred to Mark Zuckerberg as Fuckerberg after he denied a post in which I stated that I had used Kratom to relieve diabetic neuropathy because nothing that the doctors would give me worked and if it had not been for that I - [Christmas Fun](https://www.eskimo.com/2019/12/christmas-fun/) - [Mail](https://www.eskimo.com/2019/12/mail-8/) - I have found that what kernel resource ran out on the client mail server was the number of open files. I've bumped that up by 5x, hopefully that will be sufficient. Dec 3 08:57:22 mail kernel: [371342.776244] VFS: file-max limit 194527 reached Dec 3 09:43:33 mail kernel: [374113.872327] VFS: file-max limit 194527 reached Dec - [Mail Server](https://www.eskimo.com/2019/12/mail-server-7/) - Our client mail server got flaky between around 11AM and 1PM today. I am pretty sure the cause is brute force password attacks exhausting some kernel resource but I have not been able to identify the resource being exhausted. The reason I believe this is the cause is that in the last - [SMTP](https://www.eskimo.com/2019/11/smtp/) - Last night I was up until 6AM doing reboots and backups. It is not unusual for NFS mount points to not mount or NIS to not bind after a reboot. Those are bugs I am used to and always check for. But postfix not starting is unusual, I didn't check, didn't notice, went to sleep - [Server Reboots Friday 2AM-ish](https://www.eskimo.com/2019/11/server-reboots-friday-2am-ish/) - I will be rebooting the physical host machines Friday morning around 2AM. This will affect most everything. Downtime should be less than about 15 minutes if everything goes as planned. This is to load new 5.4 kernels. - [Web Server Upgrades](https://www.eskimo.com/2019/11/web-server-upgrades-2/) - I've upgraded the encryption suite on our web server to modern encryption. An unfortunate side effect is that it will break compatibility with IE8 on WinXP and versions of Android 2.73 or earlier. - [Tickless Kernels](https://www.eskimo.com/2019/11/tickless-kernels/) - If you are running any Debian derived operating system, Ubuntu, Debian, Mint, Zorin, Julinux, etc, you may wish to try a tickless kernel. What tickless kernels do for you is eliminate the CPU from having to wake-up to service clock interrupts unless there is actual work to be done. This saves a significant - [Fedora is now Rawhide](https://www.eskimo.com/2019/11/guacamole-up/) - Fedora.eskimo.com is now RawHide, a rolling release, rather than 29, 30, etc. This means it's always on the bleeding edge. If you need a more stable Redhat based release I suggest Centos7.eskimo.com or Centos8.yellow-snow.net. - [Fedora Down for Upgrade](https://www.eskimo.com/2019/11/fedora-down-for-upgrade-3/) - Fedora is in the process of an operating system upgrade. Unlike Debian based systems, RedHat systems are brain dead and can't be upgraded while in use. It should be available Friday. - [Guacamole Operational](https://www.eskimo.com/2019/11/guacamole-operational/) - Guacamole is again operational. I am going to take the web server down for about a half hour shortly after midnight to image it so I do not lose the work I've done today. - [Guacamole](https://www.eskimo.com/2019/11/guacamole-3/) - Guacamole web access to shell server console and terminals is presently unavailable. Restoring the web server and then upgrading to 19.10 broke it to the point where it needs to be completely re-installed. I am working on that. I apologize for the interruptions earlier but needed to fix various things so that - [Web Server](https://www.eskimo.com/2019/11/web-server-5/) - Our web server is currently reverted to the software state it was in on October 24th. This is because a reboot was required after a software update this morning and upon rebooting it would not come back up. It didn't even provide the normal grub menu. I attempted to fix this with - [Taking Web Server Down for Maintenance](https://www.eskimo.com/2019/10/taking-web-server-down-for-maintenance/) - Taking the web server down to image prior to upgrading Linux in case something goes wrong. It will be back online in approximately 1/2 hour. - [MxLinux](https://www.eskimo.com/2019/10/mxlinux-3/) - Nobody has used MxLinux this month at all. They are preparing a new release but it's based upon an antique 4.19 kernel and an antique 1.20 version of Mate, and consequently I've decided rather than upgrading the machine I am just going to retire it and load some other version of Linux on it. - [Brief Outage](https://www.eskimo.com/2019/10/brief-outage/) - We had a brief outage around 7:35PM caused by one of our physical host servers spontaneously rebooting. I do not know what caused the reboot as there was nothing logged of the event. - [Web Mail Outage Fixed](https://www.eskimo.com/2019/10/web-mail-outage-fixed/) - We had a brief problem with webmail which resulted from my failure to add our own IP space to the ignoreip line of the fail2ban configuration in the new mail server. This has been fixed. - [Ubuntu Down](https://www.eskimo.com/2019/10/ubuntu-down/) - Because an upgrade exploded and left Ubuntu in a bad state, I need to restore it from backup and start over. It will be down for about an hour. - [Web Server Maintenance](https://www.eskimo.com/2019/10/web-server-maintenance-5/) - I'm going to attempt an upgrade of the web server to Ubuntu 19.10 tonight. This will include an upgrade to PHP 7.3, however, this is likely to break a few things initially. If web mail is not working late tonight, be patient. Typically PHP makes syntactical changes that requires the upgrading of some applications - [Raid Replication Finished](https://www.eskimo.com/2019/10/raid-replication-finished/) - Machine is again running with full RAID. - [Failed Drive Replaced](https://www.eskimo.com/2019/10/failed-drive-replaced/) - The failed drive has been replaced. The RAID array is copying to the new drive, it will take about ten hours to complete. - [Server Maintenance October 16 Midnight-2AM](https://www.eskimo.com/2019/10/server-maintenance-october-16-midnight-2am/) - I will be taking most services down around midnight for approximately two hours to replace a failed hard drive on the server that provides NFS service for everyone's home directories. - [Mail Up](https://www.eskimo.com/2019/10/mail-up-3/) - Maintenance completed. Mail is available. - [Mail](https://www.eskimo.com/2019/10/mail-7/) - I am taking the client mail server down for about 1/2 hour to image it after getting smartlists working again. - [Server Maintenance](https://www.eskimo.com/2019/10/server-maintenance/) - I will be taking various shell servers and the client mail server down for about 1/2 hour each tonight, either to move to a different host machine to do some load balancing, or for backups in the case of the mail server. - [Smartlist's](https://www.eskimo.com/2019/10/smartlists/) - When I installed the new mail server, I changed the ownership of smartlists from "smartlist" to "list" because the newer Linux distros use a different permissions scheme. However, some files, like the "dist" file were inadvertently changed to and this broke the ability for people to administer their lists. I am working - [Iglulik Raid Degraded](https://www.eskimo.com/2019/10/iglulik-raid-degraded/) - One disk has failed in the /dev/md0 on Iglulik. The RAID is currently running in degraded mode, one disk has no mirror at present. As long as that drive holds up this is not service affecting. A new drive has been ordered and should be here Wednesday. If the mirrored drive should fail - [Client Mail Server Status](https://www.eskimo.com/2019/10/client-mail-server-status/) - I got part of but not all of what I had hoped to accomplish done tonight. Our client mail server is now chroot'd as are the incoming servers. Our client mail server now signs mail with a DKIM signature and checks mail that comes to it with a signature though from a - [Mail Server Maintenance Completed](https://www.eskimo.com/2019/10/mail-server-maintenance-completed-5/) - Mail server maintenance is completed. - [Client Mail Server Maintenance](https://www.eskimo.com/2019/10/client-mail-server-maintenance-2/) - October 8th, midnight - 12:30AM, I will be taking this machine down to image it. This should take approximately 1/2 hour. During this time you will be unable to send mail or view mail via IMAP/POP/Webmail, you can still view mail via shell mailers. - [Lists Operational](https://www.eskimo.com/2019/10/lists-operational/) - Lists are operational with the new mail server now. I was somehow missing procmail_wrapper from the smartlist.bin binaries. - [New Mail Server](https://www.eskimo.com/2019/10/new-mail-server/) - Our client mail server, "mail.eskimo.com", has been replaced with a newer server based upon Ubuntu 19.10. I am having difficulty getting smartlist to work right but not only with this server but the two existing incoming servers. I believe it is a permissions problem, smartlist on CentOS is owned by "smartlst" but under - [New Shell Server - centos8.yellow-snow.net](https://www.eskimo.com/2019/10/new-shell-server-centos8-yellow-snow-net/) - We have a new shell server available for your use. Centos8.yellow-snow.net. This is not available via the web yet but is available via "ssh centos8.yellow-snow.net" or X2go using "centos8.yellow-snow.net" as the host name. Please also note the installed software is extremely limited right now, the only desktops available at this time are KDE - [Mail Server](https://www.eskimo.com/2019/10/mail-server-6/) - The new mail server is up. I am not sure if list mail is working. I will do some further testing after I have a few hours of sleep. - [Mail Server Replacement](https://www.eskimo.com/2019/10/mail-server-replacement/) - I am going to replace the existing Centos6 based system powering mail.eskimo.com with a brand new Ubuntu 19.10 based system. This brings with it immediately only some minor benefits, new stronger key exchange methods for key exchange for example. In the longer term it will also bring better spam and virus filtering. - [Reboots Done](https://www.eskimo.com/2019/10/reboots-done-3/) - All affected systems have been rebooted. - [News](https://www.eskimo.com/2019/10/news/) - I will be rebooting all Intel-based servers later this evening or very early Saturday morning (near midnight), in order to load new kernels that address yet another newly discovered Intel exploit. When I recompiled the client mail server the day before yesterday, it overwrote the existing aliases file breaking all aliases and mail - [Firewall](https://www.eskimo.com/2019/10/firewall/) - At some point, Canonical, the folks behind the Ubuntu Linux distribution that we use on most of our servers, opted to move where firewalld looks for iptables, iptables-restore, ip6tables, and ip6tables-restore from /sbin to /usr/sbin but neglected to move the actual commands there, thus causing firewalld to fail upon startup. And rather than no - [Sending Mail Fixed](https://www.eskimo.com/2019/10/sending-mail-fixed/) - Mail.eskimo.com should be completely fixed now but I will be replacing it entirely shortly as it is an old server based upon CentOS6, the only one not modernized, and I am preparing a new server to completely replace it which will be based upon the current release of Ubuntu and upgraded as Ubuntu upgrades. - [Sending Mail - An Update](https://www.eskimo.com/2019/10/sending-mail-an-update/) - Authentication was broken, that is now fixed. Sending mail will work from local shell servers, from web mail, but NOT from devices off of our local network with encryption. I am working on fixing this but unfortunately it is somewhat involved so will take a little bit yet. - [Sending Mail - Postfix](https://www.eskimo.com/2019/10/sending-mail-postfix/) - I have discovered that what happened is that last night I recompiled postfix in order to add a needed feature to make it work in some situations that previously it did not. I was missing a necessary library on the system when I first compiled it and this caused it to omit adding that - [Mail Sending with SSL / TLS Broken](https://www.eskimo.com/2019/10/mail-sending-with-ssl-tls-broken/) - Last night I made some changes to postfix and inadvertently broke start tls and tls. I am working on correcting this. You can send mail from shell using alpine but presently web mail and portable devices using encryption will fail. - [OpenSuse Re-install](https://www.eskimo.com/2019/09/opensuse-re-install/) - Upgrade is failing to get packages from the repository and so I am going to re-install OpenSuse. It will be down until some time tomorrow evening. - [OpenSuse](https://www.eskimo.com/2019/09/opensuse/) - Ran out of disk during an attempted update of OpenSuse from 15.0 to 15.1. Had to halt machine in order to resize the disk image and partitions to accommodate the update requirements. - [Reboots Completed but More Today](https://www.eskimo.com/2019/09/reboots-completed-but-more-today/) - All the planned reboots have been completed and NIS/NFS connectivity verified, but Centos7, Scientific7, and MxLinux were all a little slow out the gate with kernel updates so I have to reboot those today. - [Reboots Saturday Morning](https://www.eskimo.com/2019/09/reboots-saturday-morning-2/) - Sometime after tonight I'll be rebooting the majority of servers to install kernel upgrades that address some newly discovered security concerns in Intel processors. Not all services will be down at once except for a brief period when we reboot the server with the /home user partition. That should take about fifteen minutes. This - [Windows X2Go Audio](https://www.eskimo.com/2019/09/windows-x2go-audio/) - X2Go may fail saying pulseaudio not running on either Windows 7 or Windows 10. The problem is that the mixer is not enabled on windows. Go to Control Panel –> Sound –> Sound devices –> Record –> Right button –> Show hidden devices –> Mixer activate - [Reboots Completed](https://www.eskimo.com/2019/09/reboots-completed-9/) - Reboots are complete, no further interruptions anticipated tonight. - [Server Reboots 2AM-3AM Saturday Morning](https://www.eskimo.com/2019/09/server-reboots-2am-3am-saturday-morning/) - We will be rebooting most of our servers (all the Intel based machines) to load a new kernel to address recently discovered security vulnerabilities. - [OpenSuse Up](https://www.eskimo.com/2019/08/opensuse-up-2/) - OpenSuse.eskimo.com is back online but is back on version 15.0 owing to the inability for x2goserver to run on 15.1 because x2goagent is missing. - [NextCloud](https://www.eskimo.com/2019/08/nextcloud-7/) - Most NextClouds are back online. Unrecoverable for the time being are OnlyOffice and ZimbraDrive. - [NextCloud](https://www.eskimo.com/2019/08/nextcloud-6/) - Presently a large number of NextCloud applications are unavailable. This is because in a recent update something went wrong breaking one or more of the apps. With all of the apps enabled it just loads a white page, no errors or clues as to what is wrong even if I turn on php - [OpenSuse Down](https://www.eskimo.com/2019/08/opensuse-down-3/) - OpenSuse.eskimo.com is going to be down longer than I thought, days, maybe weeks. 15.1 is installed but I can't get into any of the graphical tools I need to complete the configuration because x2goserver is broken on it, no x2goagent necessary for x2goserver. No idea yet how I'm going to work around - [OpenSuse Down for Upgrades](https://www.eskimo.com/2019/08/opensuse-down-for-upgrades/) - Opensuse.eskimo.com will be unavailable for several hours during upgrade from Leap 15.0 to leap 15.1. Unfortunately, an in-place upgrade failed requiring a fresh install. This has never been a simple or straight forward process with Suse so will take a few hours. - [OwnCloud](https://www.eskimo.com/2019/08/owncloud-6/) - OwnCloud has once again pissed itself during the upgrade process. Checking usage, I see nobody has made use of this in the last 90 days so I am going to discontinue support for OwnCloud and leave NextCloud alone to provide cloud services. - [NextCloud In Service](https://www.eskimo.com/2019/08/nextcloud-in-service/) - NextCloud upgrade has completed and it is back in service. - [OwnCloud and NextCloud Updating](https://www.eskimo.com/2019/08/owncloud-and-nextcloud-updating/) - Updates are in progress for both OwnCloud and NextCloud cloud services at Eskimo. Both of these involve a maintenance period where cloud services will be unavailable. In the case of OwnCloud, because it's been a very long time since an upgrade was available, this is likely to be a lengthy process. - [Shell Server Maintenance](https://www.eskimo.com/2019/08/shell-server-maintenance/) - CentOS6, Debian, and Ubuntu will be down for a short period tonight, probably 11pm-1am time frame, to change some disk parameters to improve performance and reduce overall load on the physical hosts by enabling read-ahead and write-back caching. - [Web Server Upgrades](https://www.eskimo.com/2019/08/web-server-upgrades/) - We are always trying to improve the speed, reliability, and functionality of our products. To this end we've made the following upgrades to our web server: Upgraded to Apache 2.4.41 Added read-ahead and writeback caching to the file systems. Added gzip compression to our server configuration. Enabled transparent huge pages for both web - [Web Server Crashed](https://www.eskimo.com/2019/08/web-server-crashed/) - It appears the web server crashed at 17:09 but nobody called until 19.35. It has been restored to service. I am unable to determine the cause at this point. I will be changing the kernel later tonight on the off chance it is a kernel issue. It left the machine in a state without - [Name Server 204.122.16.8 ns1.eskimo.com Down](https://www.eskimo.com/2019/08/name-server-204-122-16-8-ns1-eskimo-com-down/) - An update to bind 9 broke the name server by requiring a very different configuration. I had been meaning to move it to a more modern platform anyway so this is in the works. Until this is completed, this name server will not function. - [Maintenance Completed Aug 5th 4:30AM](https://www.eskimo.com/2019/08/maintenance-completed-aug-5th-430am/) - Maintenance was completed around 4:30 AM this morning. Anything down is broken. Please report anything down using either https://www.eskimo.com/ -> Support -> Tickets or e-mail support@eskimo.com or call 206-812-0051. Thank you. - [Manjaro / Social Maintenance](https://www.eskimo.com/2019/08/manjaro-social-maintenance/) - I did not complete all the maintenance I had intended last night so manjaro and social are presently down for maintenance. Expected down time less than one hour. - [Tonight 11pm to ~4AM Interruptions](https://www.eskimo.com/2019/08/tonight-11pm-to-4am-interruptions/) - I will be performing maintenance activity tonight after about 11PM that will involve the reboot of various servers and taking some down for backup imaging. This requires about 15-45 minutes per machine. Where possible I will attempt to leave functional mates up but this will not always be possible, for example when rebooting igloo - [Mostly Up](https://www.eskimo.com/2019/07/mostly-up/) - Things are mostly back up now. I will be checking for proper NFS/NIS mounts over the next couple of hours but things should be basically functional. - [Mail Server and other hosts down](https://www.eskimo.com/2019/07/mail-server-and-other-hosts-down/) - The machine which hosts the mail spool, client mail server, and some other hosts is currently down. It failed to reboot because of missing software prohibiting it from accessing the Internet. Since it can not access the Internet I can not restore the missing software that way. I attempted to restore from backups but - [Rebooting All Servers](https://www.eskimo.com/2019/07/rebooting-all-servers/) - Rebooting all servers this morning (Sunday July 21st) to make upgrades effective. - [Borg Backup](https://www.eskimo.com/2019/07/borg-backup/) - Borg Backup is installed on all of our shell servers at the request of one of our customers. We make backups automatically of all user home directories and e-mail weekly. And both of these things are also on RAID10 partitions so data will not be lost in the event of a drive failure. - [Debian is Back](https://www.eskimo.com/2019/07/debian-is-back/) - Mail is now operational on Debian however, elm will not be available. I can not get elm-2.5.8, the last official version, to compile. C syntactical requirements have just changed too much for even clang to properly grok the code anymore and that's always been my go-to compiler for antique code. If there are - [Debian Back Up - Not Fully Configured](https://www.eskimo.com/2019/07/debian-back-up-not-fully-configured/) - Debian is back up. Mail is not yet configured. Many applications still being installed. All the desktops are presently installed. Mail is not yet configured. You will not yet be able to send or receive e-mail from Debian. Applications are still installing. There are lots. This will take some time. - [Fedora Down / Up](https://www.eskimo.com/2019/07/fedora-down-up/) - Fedora was shut down unintentionally, it is back online. - [Name Server ns3.eskimo.com](https://www.eskimo.com/2019/07/name-server-ns3-eskimo-com/) - One of our name servers, ns3.eskimo.com, was down earlier. I have restored it to service. There were three others that were operational so this should have only caused a temporary delay in resolution if it happened to be the first chosen. - [Debian](https://www.eskimo.com/2019/07/debian-5/) - A short while after I fixed Debian it ate itself again. This time leaving itself in an unbootable state. Rather than restoring from a stretch backup, I am doing a fresh install. So much changed between stretch and buster that it is not surprising the upgrade wasn't entirely smooth. It may take several - [Debian Outage](https://www.eskimo.com/2019/07/debian-outage/) - Around 7:15 PM this evening Debian networking went away. I logged into the console and everything at first appeared normal but I could not ping or connect to anything outside. I tried restarting networking, it was uneventful but did not restore connectivity. I think restarted NetworkManager but it failed to start with - [Debian Upgrade Completed](https://www.eskimo.com/2019/07/debian-upgrade-completed-2/) - The Debian upgrade has been completed. The shell server debian.eskimo.com is now "buster", otherwise known as Debian 10. It seems generally crisper. I only had to sacrifice two programs to get it fully upgraded, eclipse an ide that wants jdk8, and buster is now jdk11, and a game, the name of which slips my - [Debian](https://www.eskimo.com/2019/07/debian-4/) - Debian.eskmo.com is in the process of an upgrade from "stretch" to "buster". I did not expect it to interfere with service but it is currently not allowing logins. I expect the upgrade to be completed sometime early this evening. - [X2Go on Zorin Fixed](https://www.eskimo.com/2019/07/x2go-on-zorin-fixed/) - X2go is fixed on Zorin. - [Guacamole](https://www.eskimo.com/2019/07/guacamole-2/) - I'm banned from x2go-users list because I dared to disagree with Lord King Master Stephan Baur, the project owner. What will I do with the rest of my life? I know, I'll install Guacamole on all my servers which provides the same functionality without the need for special client software except for an ordinary - [Pine on Shell Servers](https://www.eskimo.com/2019/07/pine-on-shell-servers/) - One customer, who was migrating from centos6 to ubuntu, brought it to my attention that pine was not present on ubuntu and alpine had an unworkable default configuration. I have fixed this not only on Ubuntu but on the rest of the shell servers as well. - [GoodVibes on Zorin](https://www.eskimo.com/2019/06/goodvibes-on-mxlinux-2/) - GoodVibes is now installed on zorin.eskimo.com. As with the other systems, x2go is required to use this or it can be accessed via the visual or terminal access in our Web Apps menu on our website. In the process of installing this I found that the meson, gstreamer, and x2go applications and - [x2go broken on Zorin](https://www.eskimo.com/2019/06/x2go-broken-on-zorin/) - The installation of x2goserver on zorin.eskimo.com is presently broken. Unlike other systems in which this occurred simply purging and re-installing did not fix. I am working on correcting this problem. In the meantime, if you need graphical access to this machine, use the web interface under web-apps on our website at https://www.eskimo.com/. - [GoodVibes on MxLinux](https://www.eskimo.com/2019/06/goodvibes-on-mxlinux/) - GoodVibes is now installed on mxlinux.eskimo.com. As with the other systems, x2go is required to use this or it can be accessed via the visual or terminal access in our Web Apps menu on our website. In the process of installing this I found that the meson, gstreamer, and x2go applications and - [GoodVibes on Mint](https://www.eskimo.com/2019/06/goodvibes-on-mint/) - GoodVibes is now installed on mint.eskimo.com. As with the other systems, x2go is required to use this or it can be accessed via the visual or terminal access in our Web Apps menu on our website. In the process of installing this I found that the meson, gstreamer, and x2go applications and - [GoodVIbes on JuLinux](https://www.eskimo.com/2019/06/goodvibes-on-julinux/) - GoodVibes is now on julinux.yellow-snow.net. To use it you need to connect with a graphical client that supports audio. I recommend X2Go from https://x2go.org/. X2Go is free and open source. It is available for Linux, Windows, and MacOS. If you use either Mate or Gnome Desktops, GoodVibes is available under the Internet -> Audio - [GoodVibes](https://www.eskimo.com/2019/06/goodvibes/) - I've installed GoodVibes, an internet radio player that allows you to listen to a number of mostly French speaking Internet radio stations on Ubuntu.eskimo.com. If you login with x2go (see https://x2go.org/ if you do not already have x2go installed on your computer, it is a graphical client that enables you to utilize a much - [GoodVibes](https://www.eskimo.com/2019/06/goodvibes-2/) - GoodVibes is also now installed on debian.eskimo.com. Using x2go (https://www.x2go.org/) with either Mate or Gnome Desktops, it is under Applications -> Sound and Video. Other desktops you'll need to locate it yourself. You can also launch it from a terminal, as long as you somehow have X11 forwarded, as "goodvibes". GoodVibes is an - [Eskimo.Com Shell Server Down](https://www.eskimo.com/2019/06/eskimo-com-shell-server-down-2/) - The "eskimo.com" shell server will be down until sometime this evening. Please use an alternative such as "ubuntu.eskimo.com" in the interim. All your files and e-mail will be the same. - [vps4, vps5, and vps6, debian, scientific, and ubuntu](https://www.eskimo.com/2019/06/vps4-vps5-and-vps6-debian-scientific-and-ubuntu/) - vps4, vps5, and vps6, debian, scientific, and ubuntu will be down for approximately 1/2 hour each , debian and ubuntu somewhat longer, the others somewhat less, for backups. - [Mail Completed](https://www.eskimo.com/2019/06/mail-completed/) - Maintenance on mail is finished, it is restored to service. - [Mail Maintenance](https://www.eskimo.com/2019/06/mail-maintenance-6/) - Our mail server will be unavailable on June 15th between 02:40AM - 03:00AM for backup. - [Server](https://www.eskimo.com/2019/06/server/) - Found that the entire web server was not down, only mariadb (what used to be mysql). MariaDB is necessary to our site and any site which uses Wordpress or other content management engines. I've added a script to crontab now that checks once a minute and restarts it if it dies. But - [Web Server Crash](https://www.eskimo.com/2019/06/web-server-crash/) - Our web server crashed some time this morning. Only one call so I was unaware that it was down. I'm going to try to put together some means of automatic monitoring. I do not yet know the cause of the crash. We have been hit with denial of service attacks for the last - [ns3.eskimo.com](https://www.eskimo.com/2019/06/ns3-eskimo-com/) - One of our name servers, ns3.eskimo.com, is currently down. This was due to a combination of DNSSEC root-key change and my attempts to being lazy in resolving it. I attempted to purge / re-install the bind package thinking that would get me fresh conf files including the DNS key file however, Redhat had - [Zorin Upgrade](https://www.eskimo.com/2019/06/zorin-upgrade/) - Zorin is presently down for an operating system upgrade. Unfortunately, Zorin provides no online upgrade path, it is necessary to do a full re-install so it will be down for a few hours. - [Web Server Upgrade](https://www.eskimo.com/2019/06/web-server-upgrade-2/) - I upgraded our web server operating system from Ubuntu 17.10 to Ubuntu 18.04 tonight. I replaced our previous database, MySql 5.7 with MariaDB 10.3. This was not entirely smooth owing to the existence of mysql in the NIS database conflicting with the local user that the install script wanted to create and NOTHING - [Router Replaced](https://www.eskimo.com/2019/06/router-replaced/) - Router arrived last night. Configured and brought it up to current software release and installed tonight. All work is done, everything is back up and running normally. - [Router Replacement](https://www.eskimo.com/2019/05/router-replacement-3/) - The router I ordered from Amazon did not arrive yesterday. When I checked shipping, it had not even shipped. I called Amazon and they said they were out of stock and it was back ordered, estimated to be here on June 6th. I asked to have my one day shipping fee refunded - [Host Desktop or Terminal (Guacamole)](https://www.eskimo.com/2019/05/host-desktop-or-terminal-guacamole/) - Several recent changes I made to our web server, compiling and installing the most recent versions of openssl and gnupg, and installing new SSL certificates with a slightly different more unified naming convention, broke guacd, the daemon that interfaces to the server to provide mapping of vnc or ssh to a desktop or terminal. - [Outage - Router Failure](https://www.eskimo.com/2019/05/outage-router-failure/) - Our border router failed today, this is the router that connects the outside world to our servers, This particular model used to be carried by Fry's but as luck would have it, they were sold out. There are no other dealers that I can find listed locally so I've ordered a replacement from Amazon. - [Microsoft Visual Studio Code](https://www.eskimo.com/2019/05/microsoft-visual-studio-code/) - Microsoft Visual Studio Code is available now in all the Debian derived servers, this would include debian, mint, julinux, mxlinux, ubuntu, and zorin. Although it is 'code' from the command line, on most machines it will only start from the menu's. It is under Applications -> Programming -> Visual Studio Code - [New SSL Certs Installed](https://www.eskimo.com/2019/05/new-ssl-certs-installed/) - I apologize for the brief interruptions in mail and web services today. It was necessary to install new SSL encryption certificates as the old were about to expire. Comodo is now Sectigo and they only shipped me the domain cert and one of two intermediate certificates and no root certs so the bundle-ca file - [Web Server](https://www.eskimo.com/2019/05/web-server-4/) - I received a call today asking when the web server would be back up. The operating system was responsive, Apache was running but some pages would come up and some would not. I wasn't able to determine the cause but rebooted the server and it hung during the reboot attempt. I then hard forced - [Work on Scientif7 and Julinux completed](https://www.eskimo.com/2019/05/work-on-scientif7-and-julinux-completed/) - These two virtual machines are moved to their new hosts. - [I am taking julinux and scientific7 down for a couple of hours](https://www.eskimo.com/2019/05/i-am-taking-julinux-and-scientific7-down-for-a-couple-of-hours/) - I am taking julinux and scientific7 down for a couple of hours to move them to different physical hosts so that public name servers are distributed across different physical host machines so at least one will be up at any given time. - [Maintenance Completed](https://www.eskimo.com/2019/05/maintenance-completed-22/) - Tonight I completed the task of configuring the new drives in and moving data to them. This required a lengthy downtime of the virtual machines including web service on this particular host because data can't be accessed while it is being moved from one RAID array to another. I still have to check - [Maintenance Completed](https://www.eskimo.com/2019/05/maintenance-completed-21/) - New drives are successfully installed. Haven't got them configured into a RAID array yet but the machine sees them so the hardware is good. - [Maintenance May 10th / 11th 11:30-1:30](https://www.eskimo.com/2019/05/maintenance-may-10th-11th-1130-130/) - I will be taking the server which houses home directories down to attempt again to add disks again. This time equipped with 3-1/2 to 5-1/4 adapters and power cables. Expected downtime is about an hour but I will also be rebooting the other machines to get them off of the 5.0.0 kernel which has - [NextCloud Terms of Service / Login Problems](https://www.eskimo.com/2019/05/nextcloud-terms-of-service-login-problems/) - Login problems were caused by the fact that the syntax required by the external users module, used to allow login with Eskimo system logins, changed between 14.10 and 15.07 but nothing bothered to check this or tell me about it. The config.php syntax has been corrected to work with the newer version. You - [Nextcloud Updates Completed](https://www.eskimo.com/2019/05/nextcloud-updates-completed/) - NextCloud updates were completed last night. We are now on version 15.07. Some applications were removed because no updated versions were available. Other new applications were added. - [NextCloud Update](https://www.eskimo.com/2019/05/nextcloud-update/) - An upgrade from 14.06 to 14.10 broke several applications for which no update was available. I will be cleaning those up and then proceeding with an upgrade to 15.x this evening. This is somewhat involved so may not be finished for several days. - [Maintenance Friday Evening](https://www.eskimo.com/2019/05/maintenance-friday-evening/) - I will be taking the big server that houses all users home directory off line to add additional hard drives Friday evening probably starting around 10:30PM. This will require pausing incoming e-mail as well since .procmailrc files and local e-mail boxes reside in home directories. Shell servers will be unavailable and all web services - [Service Issues May 9th](https://www.eskimo.com/2019/05/service-issues-may-9th/) - Sometime this morning Igloo, the server that hosts the mail spool, just stopped serving files in /mail via NFS. It wasn't a case of NFS dying altogether as other files systems still worked. I wasn't able to troubleshoot much further because when I got to the co-location facility, the console was dead and - [Update to Ubuntu 19.04](https://www.eskimo.com/2019/05/update-to-ubuntu-19-04/) - For workstations 19.04 is mostly okay, for servers it's a piece of crap and has kept me up all night last night and tonight fixing things it broke. The new kernel, 5.0.0-14 is basically non-functional, networking either doesn't work at all or randomly goes away. One machine went into hibernate mode all by it's - [Maintenance](https://www.eskimo.com/2019/05/maintenance-3/) - I got about 30% of what I set out to do tonight done. Consequently there will be additional downtime tonight and next weekend. I'm going to have to order an external case. I thought I had more internal drive bays than I do. - [Dial Access Fully Restored](https://www.eskimo.com/2019/04/dial-access-fully-restored/) - We found servers in our logs that were not in our server configuration. Further investigation determined that GlobalPOPs had added additional radius servers but had not informed us. Requests coming from new servers not in our configuration were rejected causing authentication failures. I obtained a complete list of their radius servers, added - [Dial-up Access Down](https://www.eskimo.com/2019/04/dial-up-access-down/) - Presently, our dial-up customers are unable to authenticate. We have tested our radius authentication servers and those of our wholesale provider and found the issue to be with the providers authentication servers. I attempted to generate a ticket via the normal means at GlobalPops loginto.us website and found the ticket link broken - [Maintenance Rescheduled](https://www.eskimo.com/2019/04/maintenance-rescheduled/) - Work I had planned for Friday Evening / Saturday Morning will be rescheduled for Saturday Evening / Sunday morning owing to it taking me longer to get things prepared than I had anticipated. - [Maintenance Friday - Saturday](https://www.eskimo.com/2019/04/maintenance-friday-saturday/) - Late Friday evening into early Saturday there will be some downtime for system maintenance. First probably around 10:30, I will be taking the server that hosts home directories, out of service for up to an hour. The reason for this is I will be adding a pair of 10TB drives to allow us to - [Linux Security Exploit (Iglulik was attacked)](https://www.eskimo.com/2019/03/linux-security-exploit-iglulik-was-attacked/) - Warning!!!! This security exploit has not been widely published but it IS actively being exploited. Someone caused my server that houses our customers /home directories to spontaneously reboot trying to exploit it. Fortunately the kernel logged their attempts. See: https://www.kernel.org/doc/html/latest/admin-guide/l1tf.html In our cause I performed measurements on system load, web page loading times and latency - [Iglulik Crash](https://www.eskimo.com/2019/03/iglulik-crash/) - Iglulik, our system which houses /home directories and some virtual machines spontaneously rebooted. Examination of the logs suggest someone tried to take advantage of a CPU exploit similar to Meltdown. I am looking into several possible mitigation strategies, all of them have performance impacts. - [Debian Repaired](https://www.eskimo.com/2019/03/debian-repaired-2/) - Debian is fixed and restored to service. - [Debian](https://www.eskimo.com/2019/03/debian-3/) - Debian's key database has gotten badly corrupted. The machine is down at present and is being restored from backups. It should be available again in about an hour. In the meantime, please use one of the other debian derived machines such as julinux.yellow-snow.net, mint.eskimo.com, mxlinux.eskimo.com, ubuntu.eskimo.com, or zorin.eskimo.com. - [Centos 7](https://www.eskimo.com/2019/03/centos-7/) - CentOS 7 stopped accepting connections today. I found ypbind was not bound so it could not access the NIS database to get user authentication information. However, ypbind would not rebind on restart. Usually the cause of this is portmapper (rpcbind) not running. I checked, it was but I restarted it anyway, still - [Manjaro Maintenance Tuesday Morning 03:00-04:00](https://www.eskimo.com/2019/03/manjaro-maintenance-tuesday-morning-0300-0400/) - Taking Manjaro.Yellow-Snow.Net down to backup and grow disk. Should be back up in about one hour. - [Maintenance Sunday Morning Midnight - 0400 AM](https://www.eskimo.com/2019/03/maintenance-sunday-morning-midnight-0400-am/) - I will be rebooting all servers early Sunday morning to make a kernel upgrade effective, and then taking select servers down for a period to perform imaging, a form of backups that basically captures the entire state of a machine so it can be quickly restored in the case of severe corruption. Brought - [Zorin](https://www.eskimo.com/2019/03/zorin-3/) - I was never able to get Gnome to work on Zorin, Gnome is usually one of the easier desktops to get functioning. I am going to re-install Zorin on the assumption that something probably went wrong. It will be down for a couple of days. We will also be going from the free Zorin - [MxLinux](https://www.eskimo.com/2019/03/mxlinux-2/) - The shell server mxlinux is now back online. It is also now available from the the web and audio works. Not all of the application software has been re-installed yet, this will take some time. If there is something you need, please let me know and I will prioritize it. - [CentOS 6 - Sublime-Text](https://www.eskimo.com/2019/03/centos-6-sublime-text/) - The application sublime-text has been removed from CentOS 6 because the authors have changed it to required GTK-3 and CentOS 6 has Gnome-2 and thus no GTK-3. If you need this application use ANY other shell server EXCEPT Centos 6 and Scientific Linux both of which have Gnome 2. - [MxLinux will be down until further notice](https://www.eskimo.com/2019/03/mxlinux-will-be-down-until-further-notice/) - In the process of trying to re-install mxlinux, one of the software repositories is broken so I am able to obtain all of the necessary components. As a result, mxlinux.eskimo.com will be down until they fix the issue with their repository. In the meantime please use debian.eskimo.com, julinux.yellow-snow.net, mint.eskimo.com, or ubuntu.eskimo.com which are - [MxLinux Maintenance](https://www.eskimo.com/2019/03/mxlinux-maintenance-5/) - Something is broken in MxLinux and I'm not sure if an update broke it or my attempt to configure it to work with guacamole. There are too many variables to troubleshoot each individually. I am going to restore it from backup. There will be a short downtime while it reboots from a backed up - [Connecting to eskimo.com with a modern ssh](https://www.eskimo.com/2019/03/connecting-to-eskimo-com-with-a-modern-ssh/) - Because modern ssh has disabled some older less secure encryption protocols, by default you will not be able to connect with the shell server 'eskimo.com'. To fix this add the following to your /etc/ssh/ssh_config file: Host eskimo.com KexAlgorithms +diffie-hellman-group1-sha1 This will enable diffie-hellman-group1-sha1 encryption protocol just for this specific host. - [Guacamole Fixed](https://www.eskimo.com/2019/03/guacamole-fixed/) - Fixed the boot start up problems with guacamole, in the process learned a bit more about systemd unit files including how to specify the user ID the program runs as and how to specify what must be started and running before the program in question is started. The root of the problem is - [Guacamole / Web Availability](https://www.eskimo.com/2019/03/guacamole-web-availability/) - I am working to correct some issues that cause guacamole not to start automatically on reboot. This unfortunately requires rebooting of the server to test and finding the issue causing it to fail has been quite difficult requiring many reboots. My apologies for the interruptions but it is necessary to insure that this service - [Scientific 7 Added to Host Desktop or Terminal](https://www.eskimo.com/2019/03/scientific-7-added-to-host-desktop-or-terminal/) - The feature Host Desktop or Terminal allows you to connect to one of our shell servers either as a full graphical desktop or a simple text terminal from the web. This feature is full encrypted end to end. The only possible security threat with a public system is if a keystroke logger is - [Maintenance Completed](https://www.eskimo.com/2019/03/maintenance-completed-20/) - Maintenance work for tonight has been completed. - [Maintenance Saturday Midnight - 04:00AM](https://www.eskimo.com/2019/03/maintenance-saturday-midnight-0400am/) - Going to reboot a number of servers that need to be rebooted in order to make some security updates effective. Then I will be taking the web server down and possibly some other servers to make backups so I do not loose the work I've done to get guacamole (the software that allows you - [Fedora Upgrade in Progress](https://www.eskimo.com/2019/03/fedora-upgrade-in-progress-2/) - Fedora.eskimo.com is presently offline because it is in the process of being upgraded from Fedora 28 to Fedora 29, and Fedora's update process does not allow it to be done while online. - [CentOS7 Desktop via Web](https://www.eskimo.com/2019/03/centos7-desktop-via-web/) - CentOS7 Desktop is now working via the web. - [Mint via Web](https://www.eskimo.com/2019/03/mint-via-web/) - Tonight I added Mint to the hosts you can get to, either terminal or full graphical desktop, via the web. - [Guacamole](https://www.eskimo.com/2019/03/guacamole/) - This is an example of accessing Debian from our website and opening up Gimp within debian. This makes editing your website online super easy! - [Connect to Servers Terminal or Full Desktop via Web!](https://www.eskimo.com/2019/03/connect-to-servers-terminal-or-full-desktop-via-web/) - New Feature! A while back I asked for your suggestions. What follows is an implementation of one of them that I think is a great addition to Eskimo. I hope that you agree. If you go to our website, https://www.eskimo.com/, under Web Apps, you will notice there is a new item called Host Terminal or - [Web Server](https://www.eskimo.com/2019/02/web-server-3/) - The web server was not responding because the encryption key and certificate of one of my customers sites did not match. I was unaware this condition would affect the entire server. - [Server Reboots Saturday Morning](https://www.eskimo.com/2019/02/server-reboots-saturday-morning/) - We will be rebooting the majority of our servers early in the morning shortly after midnight Saturday February 16th, 2019, for a kernel upgrade. - [Apache Update Complete](https://www.eskimo.com/2019/02/reboots-saturday-morning/) - The update of Apache to 2.4.38 has been completed. - [Apache Upgrade](https://www.eskimo.com/2019/02/apache-upgrade/) - There will be a brief interruption in web servers (less than 60 seconds) to upgrade from Apache 2.4.37 to 2.4.38 later this afternoon. - [Debian, Fedora, Scientific7](https://www.eskimo.com/2019/02/debian-fedora-scientific7/) - These machines had NFS and/or NIS issues after reboot last night but have been restored to service. - [Server Reboots](https://www.eskimo.com/2019/02/server-reboots-8/) - I'm going to reboot all the servers sometime after midnight tonight, assuming power is still on and stable at that time, to make effective a kernel upgrade. - [Interruption In Service](https://www.eskimo.com/2019/02/interruption-in-service/) - I apologize for the brief interruption in services around 2:30PM today. After I disabled quotas, all the free disk space went away. Only problem is disk usage did not show it allocated to anyone. Something in that machines kernel or file system was corrupted. After I rebooted the machine, disk usage was back to - [Quota System Misbehaving Still](https://www.eskimo.com/2019/02/quota-system-misbehaving-still/) - Still the quota system is acting weird. I've turned it off (in theory) for now and will reboot the server and re-enable later tonight if it stays sane while turned off. Because it's a kernel process and not behaving properly I really can't be sure that it is turned off so if it continues - [Quota](https://www.eskimo.com/2019/02/quota/) - Not sure why but the quota system went berserk today and decided I had about ten times as much disk usage as I actually had. I do not know if other accounts were affected but I ran a system wide audit to correct the quota values. - [Olive Video Editor](https://www.eskimo.com/2019/02/olive-video-editor/) - Olive video editor has been installed on Debian, JuLinux, Mint, Ubuntu, and Zorin. Note julinux is under yellow-snow.net, julinux.yellow-snow.net. It is available under Applications -> Sound and Video -> Olive. Larger video windows may display somewhat jumpy over x2go depending upon your Internet connection but this will not affect the playback quality - [Clipgrab](https://www.eskimo.com/2019/02/clipgrab/) - Because Chrome and Firefox's plug-in architecture is effectively neutered in the interest of security (mostly because Windows security is poor), I have not been able to find a good youtube download plugin. The plugins that exist under the new interface require a third party website be involved and those seem limited and unreliable. - [Sublime-Text Installed](https://www.eskimo.com/2019/01/sublime-text-installed/) - I've installed another new text editor, this, like atom, is a graphical editor featuring syntax highlighting and many other modern features. It is called Sublime Text. Sublime Text is available on Centos6, Centos7, Debian, Fedora, Julinux, Mint, MxLinux, Scientific6, Scientific7, Ubuntu, and Zorin. It is available under Applications -> Programming -> Sublime - [Atom](https://www.eskimo.com/2019/01/atom-2/) - A new graphical text editor, very capable and configurable, called "atom", has been installed on Debian, Mint, Ubuntu, MxLinux, Julinux, Zorin, Fedora, Centos7, and Scientific7. Centos6 and Scientific Linux6 lack a library required so I can not install it on those servers, and there is no package available for Opensuse. This is a - [Reboots Done](https://www.eskimo.com/2019/01/reboots-done-2/) - Reboots are completed and all machines tested. I'm too tired to go on so will work on backups tomorrow. - [Saturday Early AM Maintenance](https://www.eskimo.com/2019/01/saturday-early-am-maintenance/) - Shortly after midnight Saturday morning, I will be rebooting all of the Intel Linux based servers for a kernel upgrade. Between the actual reboots and checking to make sure NIS and NFS mounts properly re-establish after rebooting, this will probably take about two hours but no one service should be done for more than - [RecordMyDesktop](https://www.eskimo.com/2019/01/recordmydesktop/) - I have installed RecordMyDesktop on Centos6.eskimo.com, Centos7.eskimo.com, Debian.eskimo.com, Fedora.eskimo.com, Julinux.yellow-snow.net, Mint.eskimo.com, Mxlinux.eskimo.com, Opensuse.eskimo.com, Scientific.eskimo.com, Scientific7.eskimo.com, Ubuntu.eskimo.com, and Zorin.eskimo.com. On Debian based machines, it is available under the Applications -> Sound & Video menu in mate and gnome. On Redhat based machine it is available only from the command line. - [High Wind Advisory Tonight](https://www.eskimo.com/2019/01/high-wind-advisory-tonight/) - There is a high wind advisory which predicts wind gusts up to 60 MPH for Shoreline (where I am located). This may cause outages of my telephone and / or Internet connection and thus ability to provide customer service but should not affect online services as the latter are co-located in a robust co-location - [msmtp](https://www.eskimo.com/2018/12/msmtp/) - Per a customer request, msmtp has been installed on ubuntu.eskimo.com. To provide redundancy for times when ubuntu is out of service, it has also been installed on debian.eskimo.com, mint.eskimo.com, julinux.yellow-snow.net, and zorin.eskimo.com, all of which are debian derived machines. It is not at this time installed on mxlinux because it is still in the - [Privacy Darknet](https://www.eskimo.com/2018/12/privacy-darknet/) - Our web server is fully capable of strong encryption as well as http 2.0 protocol. However, if you have your own hosted domain with us, you need a digital encryption certificate. You can obtain a certificate from any vendor but some work better than others in terms of browser compatibility and encryption strength. https://www.youtube.com/watch?v=luvthTjC0OI - [Christmas](https://www.eskimo.com/2018/12/christmas-2/) - Matthew 22:36-40 New International Version (NIV) 36 “Teacher, which is the greatest commandment in the Law?” 37 Jesus replied: “‘Love the Lord your God with all your heart and with all your soul and with all your mind.’[a] 38 This is the first and greatest commandment. 39 And the second is like it: ‘Love your neighbor as yourself.’[b] 40 All - [MxLinux - Reinstall](https://www.eskimo.com/2018/12/mxlinux-reinstall/) - MxLinux.eskimo.com will be down for several hours to install MX18. I attempted an in place upgrade from MX17 to MX18 and it failed. At first only lightdm would not run, then it refused to mount NFS partitions which is critical here as both the /home directory and /var/mail spook directories are NFS mounted - [Diet](https://www.eskimo.com/2018/12/diet/) - This is a really good discussion on diet, what and WHEN to eat, and how bad diet manifests diabetes, metabolic syndrome, etc. https://youtu.be/m6KClPkotxM - [Reboots Early Saturday Morning 12:01 - 03:00](https://www.eskimo.com/2018/12/reboots-early-saturday-morning-1201-0300/) - Some time after midnight tonight I will be rebooting various servers that require a reboot to make recent security updates take effect. This will result in short down times of between 2-10 minutes. - [Mint Upgraded to Tessa 19.1](https://www.eskimo.com/2018/12/mint-upgraded-to-tessa-19-1/) - Mint.eskimo.com has been upgraded to Tessa 19.1. This is the first in-place upgrade of Mint that has succeeded for me. All previous in place upgrades failed requiring a fresh install. This one went very clean. - [MxLinux now MX18](https://www.eskimo.com/2018/12/mxlinux-now-mx18/) - MxLinux.eskimo.com has been upgraded from MX17 to MX18. The kernel has been upgraded to a non-xen version 4.19.10 SMP preemptive kernel with security fixes. - [Telegram-Desktop on Debian, JuLinux, Mint, Ubuntu, and Zorin](https://www.eskimo.com/2018/12/telegram-desktop-on-debian-julinux-mint-ubuntu-and-zorin/) - Official desktop client for the Telegram messenger Pure instant messaging — simple, fast, secure, and synced across all your devices. Over 100 million active users in two and a half years. FAST: Telegram is the fastest messaging app on the market, connecting people via a unique, distributed network of data centers around the globe. SYNCED: - [New Features - Discord *, purple-discord, Steam](https://www.eskimo.com/2018/12/new-features-discord-purple-discord-steam/) - We've installed Discord on Ubuntu.eskimo.com, Debian.eskimo.com, Julinux.yellow-snow.net, Mint.eskimo.com (no menu entry, launch from a terminal by typing discord), and Zorin.eskimo.com (no menu entry, launch from a terminal by typing discord). On the systems with menu entries for discord, it is found in the "Internet" sub-menu. Discord is a audio / video conferencing system very - [Web and FTP Changes](https://www.eskimo.com/2018/12/web-and-ftp-changes/) - We upgraded our Wordpress to 5.01 today. Hopefully this will not cause problems. ' Our virtual domain system for ftp is changing and it will no longer automatically change to your directory when someone logs in anonymously. They will need to cd to /home/user to get to your directory. This is in preparation for - [Mint Completed](https://www.eskimo.com/2018/12/mint-completed-2/) - Mint is completed. - [MxLinux Completed](https://www.eskimo.com/2018/12/backup/) - MxLinux has completed and is back in service. - [Server Reboots 12/9/18 Midnight - 2AM](https://www.eskimo.com/2018/12/server-reboots-12-9-18-midnight-2am/) - I will be rebooting servers between midnight - 2AM. Downtime for any given service should not be more than 20 minutes per service, most will be shorter than that. This is to upgrade kernel and apply some security patches. - [Mail from support ](https://www.eskimo.com/2018/11/mail-from-support/) - If you receive e-mail from: support This mail is NOT from us. It is a phishing scam attempting to get your authentication so they can take over your account. Do NOT click on the link. Delete it. Any mail that comes from outside domains is not from us. - [Mail Abuse - New Fail2ban Rules](https://www.eskimo.com/2018/11/mail-abuse-new-fail2ban-rules/) - I found people connecting to our mail server, giving the auth command, then disconnecting. I found out that this was a Chinese botnet attempting to deliver spam but why they would issue an auth command and then not provide arguments is not clear. Perhaps they are aware of some bug in postfix that I - [Happy Thanksgiving - and Spam](https://www.eskimo.com/2018/11/happy-thanksgiving-and-spam/) - First I want to wish everyone a Happy Thanksgiving. I've noticed the same explosion of spam here that I am sure you are all suffering from. I have not been able to find anything wrong here but there are a couple of viruses going around building botnets these folks are using. To try to improve - [Server Reboots](https://www.eskimo.com/2018/11/server-reboots-7/) - I will be rebooting servers to affect a kernel upgrade in just a few minutes. This will affect all machines but downtime for any given machine should not be more than about 15 minutes. - [New Links](https://www.eskimo.com/2018/11/new-links/) - I've added a couple of links to our links pull down on our web site. One is to something called "Tech Rights" which deal with things like copyrights, patents, and particularly how they affect the open source and technological user community. The second link that I have added is to "Open Source", a - [Scientific Linux 7 Broken - Restoration In Progress](https://www.eskimo.com/2018/11/scientific-linux-7-broken-restoration-in-progress/) - I broke scientific7.eskimo.com today. It is in the process of being restored from backups. Please use centos7.eskimo.com, which is based upon the same RHEL7 code base, in the interim. - [Reboots Completed](https://www.eskimo.com/2018/11/reboots-completed-8/) - Reboots to make active security patches have been completed. Sorry it took me somewhat longer than anticipated owing to I had a couple of machines that were stubborn and did not want to go down for reboot. - [Reboots - Nov 7th 01:30-02:00](https://www.eskimo.com/2018/11/reboots-nov-7th-0130-0200/) - I will be rebooting all of the Intel based machines between 1:30AM - 2:00AM November 7th to make active several security updates that patch serious issues with openssl, systemd, and DNS. Downtime for any given machine should not exceed about 15 minutes. Most will be less. - [Name Service Issues Early Morning November 3rd](https://www.eskimo.com/2018/11/name-service-issues-early-morning-november-3rd/) - I did not expect last night to be a maintenance night but it had other ideas. Around 3AM name service for our own hosts failed although our name servers were still resolving external hosts fine. The issue was caused by an update that replaced the named apparmor profile on the master name server. - [All Done](https://www.eskimo.com/2018/10/all-done/) - Clock speed back to 4.3 Ghz, everything tested. Sleep time now. - [One More Reboot](https://www.eskimo.com/2018/10/one-more-reboot/) - Sorry, need to do one more reboot of the machine that serves home directories to fix a clock speed issue. - [Upgrades Completed](https://www.eskimo.com/2018/10/upgrades-completed-2/) - All of the physical hosts have now been upgraded to Ubuntu 18.10. There were a couple of issues, the upgrade replaced /etc/defaults/nis and turned the NIS servers off. It took me several hours to find and correct that. Second problem was that some of the old kvm machine types, including the one that - [Mail](https://www.eskimo.com/2018/10/mail-6/) - Since we recently upgraded the mail servers, it comes with a newer postfix that supports more rules aimed at stopping spam and forgery and I've added some of those. Hopefully they will not break any legitimate mail, but if you do get reports of mail bounced, if you can get copies of the bounced - [Mint Restored](https://www.eskimo.com/2018/10/mint-restored/) - Mint is restored to service with an additional 100GB Disk. - [Mint](https://www.eskimo.com/2018/10/mint-6/) - I have to take Mint down for a short period to increase available disk space as it has grown to where there is no space left in the root partition. - [Mail Forgeries](https://www.eskimo.com/2018/10/mail-forgeries/) - I've discovered a situation in which it is possible to forge the from address with some mail clients. The person wishing to forge the e-mail provides a fake "From: " header but a real address in the envelope. Because postfix only checks the envelope it does not prevent these kind of forgeries if - [Saturday Morning Upgrades](https://www.eskimo.com/2018/10/saturday-morning-upgrades/) - I will be upgrading the physical host machines to Ubuntu 18.10 early Saturday morning. The upgrade process is rather long and grueling and during at least part of it breaks NIS. However, these machines are all NIS slaves and I will time the upgrades such that only one will be affected at any given - [Possible FTP / WWW IP Lockout](https://www.eskimo.com/2018/10/possible-ftp-www-ip-lockout/) - After the upgrade caused NIS issues, I've discovered some users are locked out of access to web and ftp services here. If you're reading this you are probably okay but if you've got a weather station or something else automatically doing periodic logins to ftp, you may be blocked. If this is the case - [Web Server Issues](https://www.eskimo.com/2018/10/web-server-issues/) - We upgraded our main web server to Ubuntu 18.10 Cosmic Cuttlefish early this morning. I did not become aware until later this morning that the upgrade broke NIS which is how the system propagates information about users between machines. This has been corrected as of about 10:45 AM this morning the 22nd of October, - [Incoming Mail Servers Upgraded](https://www.eskimo.com/2018/10/incoming-mail-servers-upgraded/) - I managed to get everything working on new mail servers based upon Ubuntu 18.04 and for me at least spam filtering is working much better. Prior to the upgrade I was getting between 1/3rd and 1/4th of spam in my INBOX rather than spam box, now I am getting 1/25 spams in my INBOX - [Julinux.Yellow-Snow.Net Fixed](https://www.eskimo.com/2018/10/julinux-yellow-snow-net-fixed/) - The shell server julinux.yellow-snow.net is now back in operation. After going down a lot of rabbit holes, I thought to ask systemd the status of nis, and low and behold it was disabled. I re-enabled it and now all is well. So when you do an upgrade of ubuntu or a Ubuntu-derived distribution that - [Julinux Temporarily O/S](https://www.eskimo.com/2018/10/julinux-temporarily-o-s/) - Julinux kicked out a new release based upon Ubuntu-Mate 18.10, but this is about three months premature and predictably it broke things, most notably NIS. Without NIS users can not authenticate on this machine. So for now that machine is down. I'll try to troubleshoot NIS and if I can't fix it revert to - [Mail Server Configuration Changes](https://www.eskimo.com/2018/10/mail-server-configuration-changes/) - I have made several changes to mail server configurations and DNS that should reduce spam and viruses. No mail will be accepted to user@mail.eskimo.com via any of the mail servers. SPF records are now published for mail.eskimo.com. This should prevent forgeries since SPF records are checked on all of our servers. The issue which - [Mail Server Operational and Current](https://www.eskimo.com/2018/10/mail-server-operational-and-current/) - I have restored mail.eskimo.com from backups, re-applied all of the updates between the time it was backed up and current, and still it continues to operate. I do not know what broke the init system but it is presently all current again and yet running okay. I will be taking the system down again - [Client Mail Server Up - But More Interruptions](https://www.eskimo.com/2018/10/client-mail-server-up-but-more-interruptions/) - The client mail server is up but I will need to reboot after bringing it current again with updates to make sure they didn't break it, and if one of them did, a few more to determine which one. I suspect an issue with clamav as it was the last thing to start before - [Mail Down - Being Repaired](https://www.eskimo.com/2018/10/mail-down-being-repaired/) - The client mail server is currently down being restored from backup. For reasons I am unable to identify, the init start up script fails to parse the entire /etc.rc.d/rc5.d directory stopping just before postfix even though if I run the scripts they run fine. I am restoring the server from backups which I - [Client Mail Server Issues](https://www.eskimo.com/2018/10/client-mail-server-issues/) - Something broke the init system of mail.eskimo.com such that necessary things are not being started on the client mail server when it boots. No cron jobs ran for some period of time and this caused the anti-virus database not to get updated. This would have made it possible for a virus to propagate from - [Network Maintenance Oct 24th 00:01-04:00am](https://www.eskimo.com/2018/10/network-maintenance-oct-24th-0001-0400am/) - ISOMEDIA will be performing network maintenance on Wednesday October 24th 2018 from 12:01AM PST to 04:00AM Pacific Time. The work being performed is not expected to be service affecting. As a precaution, please ensure that your Internet equipment is performing no critical updates, backups, or other activities over your connection during this time frame. iSOMEDIA - [Kernel Upgrades Completed](https://www.eskimo.com/2018/10/kernel-upgrades-completed-3/) - Work is done, upgrades finished, verified proper NFS mounting and NIS binding on all machines. I had to restart the NFS server that houses home directories twice because the new kernel did not install completely. Postfix on mail neglected to start because I had to force a reboot which did not remove the postfix - [System Upgrade Reboots](https://www.eskimo.com/2018/10/system-upgrade-reboots/) - We will be performing upgrades on the majority of our servers shortly after midnight tonight. Be prepared for outages lasting on average about 15 minutes after midnight and before 6AM on October 13th (early Saturday Morning). - [Mint Back](https://www.eskimo.com/2018/10/mint-back/) - Mint should be fully operational now. If you find anything missing please initiate a ticket Support->Tickets. Thank you. - [Android Studio](https://www.eskimo.com/2018/10/android-studio/) - Android Studio is now installed on Debian.Eskiimo.Com, Julinux.Yellow-Snow.Net, Ubuntu.Eskimo.Com, and Zorin.Eskimo.Com. For those interested in developing Android Apps, Android Studios are available on the above three shell servers. To use it, you must have x2go installed on your computer and use it to connect to one of the above servers. In x2go, before - [Mint](https://www.eskimo.com/2018/10/mint-5/) - Mint is re-installed and Android Studio is now available on it as well but most of the other applications are not yet in place, mail is not configured, etc. This will take a couple of days. - [Mint](https://www.eskimo.com/2018/10/mint-4/) - Mint is down for maintenance. When attempting to install Android Studio I discovered that something was wrong with apt. I restored from backups, turns out something was wrong with apt the last time I backed it up. So at this point a fresh install seems in order. - [Server Reboots Tonight After Midnight](https://www.eskimo.com/2018/10/server-reboots-tonight-after-midnight/) - I will be rebooting a number of servers to upgrade the kernels to a 4.18.0 kernel. After the issues with NFS file locking caused severe operational problems with the client mail server today I decided to try the new 4.18.0 kernels on a few machines and it appears to be running significantly cleaner. Several - [Client Mail Server - In Service](https://www.eskimo.com/2018/10/client-mail-server-in-service/) - Reboots of the mail server and a restoration from backup did not fix performance issues. While there was both a bot-net brute force password guessing attack in progress, which fail2ban found and locked out (over 1200 IPs involved) and an attempt to use the machines name server as an amplifier in a denial - [Client Mail Server - Imap/Pop/SMTP](https://www.eskimo.com/2018/10/client-mail-server-imap-pop-smtp/) - We are having problems with our client mail server that provides imap, pop3 and smtp server. Processes show waiting on disk but disk I/O is not busy. It is not a hardware issue as far as I can tell because the machine is a virtual machine and other machines on the same hardware are - [OpenSuse Upgrade](https://www.eskimo.com/2018/10/opensuse-upgrade/) - OpenSuse.eskimo.com is going to go down for an upgrade to Leap 15. The online upgrade is not working owing to some repositories not responding so going to have to do a fresh install and that's a tedious process that's going to take a while to get everything working again. Actually this is being - [Imapfilter](https://www.eskimo.com/2018/10/imapfilter/) - In response to a customer request, imapfilter has been installed on all Linux based shell servers. - [Virtual Private Server Backups Completed](https://www.eskimo.com/2018/09/virtual-private-server-backups-completed/) - Backups of all virtual private servers has been completed. - [Virtual Private Server Maintenance](https://www.eskimo.com/2018/09/virtual-private-server-maintenance/) - I will be taking virtual private servers down for about 15-20 minutes each to image the machines (a form of backup that makes a copy of the machine state). If you are doing anything critical please e-mail nanook@eskimo.com right away. - [Reboot's Completed](https://www.eskimo.com/2018/09/reboots-completed-7/) - System reboots are completed. NFS and NIS remounts and binds verified. Everything is operational. New microcode for i7-6850k still broken but I've gathered information Canonical needed so hopefully this got us closer to a fix. - [Server Reboots Early Saturday Morning](https://www.eskimo.com/2018/09/server-reboots-early-saturday-morning/) - I will be rebooting servers early Saturday morning, shortly after midnight. The server which houses home directories will be rebooted possibly more than once. Most of the servers will be rebooted to load a new kernel. The server housing the home directories will also be testing a new microcode package from Ubuntu - [Fedora 28](https://www.eskimo.com/2018/09/fedora-28-2/) - The shell server fedora.eskimo.com has been upgraded to Fedora 28. Gnome Flash back and KDE Plasma no longer exist with this distribution so are not available on X2Go, however Mate, LXDE, LXQT, and XFCE still work. - [NextCloud is updated to Version 14](https://www.eskimo.com/2018/09/fedora-28/) - Nextcloud has been updated to Version 14, however the database was badly broken. If you had files there, login, then let me know and I should be able to add them back into the database. You will have to reset any application passwords and the like. - [NextCloud](https://www.eskimo.com/2018/09/nextcloud-5/) - NextCloud is updated to version 14, tonnes of apps now but the upgrade blew up and I had to do a mostly fresh install. If you used your system login here, that should work but initially you won't see your files. If you created a login then you will need to re-register. Once you - [NextCloud Down](https://www.eskimo.com/2018/09/nextcloud-down/) - I updated some out of date applications on Nextcloud and after I finished it would no longer provide a login, just a blank white page. I am attempting to diagnose and repair. - [Mail Back Up](https://www.eskimo.com/2018/09/mail-back-up-2/) - On the mail server this morning we were hit with a brute force password guessing attack by a large botnet. For some unknown reason fail2ban died and then the bots swamped the dovecot imap server to the point of totally wedging it. Fail2ban is restarted and has locked out most of the botnet - [Mail Server Problems](https://www.eskimo.com/2018/09/mail-server-problems/) - We are having problems with the mail subsystem. I do not yet know the cause but am working to determine it. Service on some hosts besides mail may be interrupted briefly during the troubleshooting process. - [Social.Eskimo.Com - Down for Modifications](https://www.eskimo.com/2018/09/social-eskimo-com-down-for-modifications/) - I have taken social.eskimo.com down as I am trying to make the video upload function work properly as well as fix some user interface issues. As this involves some major code modifications, it is off-line temporarily. - [Our phpBB3 Forums](https://www.eskimo.com/2018/09/our-phpbb3-forums/) - phpBB3 has improved considerably with time and there are many extensions available and I have installed many of these. With the Google Captcha we finally have spam under control. We now have the ability of adding images, video, and sound to posts. Please sign up and participate: https://www.eskimo.com/forums - [Piwigo Photo Gallery](https://www.eskimo.com/2018/09/piwigo-photo-gallery/) - In the past I recommended Coppermine but Coppermine broke with 7.2. It really died do to lack of interest and maintenance. The authors did not keep it up to date with modern PHP standards. I recently replaced my Coppermine Gallery with Piwigo. Piwigo is MUCH more capable, it has a plugin extension method - [Pizza Meeting Place](https://www.eskimo.com/2018/09/pizza-meeting-place/) - This is the Pizza place we used to meet at after some asshole torched it. - [Hardware Work Done](https://www.eskimo.com/2018/09/hardware-work-done/) - I've replaced the Intel NIC's with TP-Link NIC's. The first machine took close to two hours because at first I was not able to get it to work. I finally chased it down to lack of patience on my part, these cards take approximately 1 minute to initialize. I was not able to - [Hardware Maintenance Late Sept 17th to Early Sept 18th](https://www.eskimo.com/2018/09/hardware-maintenance-late-sept-17th-to-early-sept-18th/) - I will be taking various machines down tonight for about fifteen minutes each to install new NIC cards with non-Intel chipsets. From 4.15.0 forward, the Linux kernel has had a bug in the Intel E-1000 drivers that cause the cards to lock-up when hardware offloading is used. Usually these lock-ups are transient resulting in - [Outage Sept 16th 2018 05:15 - 12:46](https://www.eskimo.com/2018/09/outage-sept-16th-2018-0515-1246/) - The ethernet controller on the server that provides /home pages wedged today. Most services depend upon being able to access /home and were unavailable as a result. As near as I can tell looking at the logs, the ethernet wedged shortly after 5AM but I did not receive any telephone calls until around - [JUlinux Maintenance Done](https://www.eskimo.com/2018/09/julinux-maintenance-done/) - Completed imaging julinux.yellow-snow.net. Give this machine a try, it's a very nice Ubuntu derived server. It has the stability and software base of Ubuntu but with nicer artwork. - [Taking JULinux down temporarily to backup.](https://www.eskimo.com/2018/09/taking-julinux-down-temporarily-to-backup/) - I am taking julinux down for a few minutes to make an image to capture all the recently installed software in the image. - [Investigating New Location for User Meetings](https://www.eskimo.com/2018/09/investigating-new-location-for-user-meetings/) - I am investigating a new location for our user meetings. This would be Amante Pizza on 123rd and Roosevelt Northeast in Seattle. The Pizza is excellent, they have spirits for those who care to imbibe, and they have a big screen TV that in theory can be connected to a computer and we - [Mush on All Servers](https://www.eskimo.com/2018/09/mush-on-all-servers/) - I managed to get Mush to compile on modern Linux machines. It is now available on ALL of our Linux servers. - [New Server - JuLinux.Yellow-Snow.Net](https://www.eskimo.com/2018/09/new-server-julinux-yellow-snow-net/) - We have a new server available for your use but this one is in the yellow-snow.net domain. The full server name is julinux.yellow-snow.net. If you use this server, e-mail you send will by default by username@yellow-snow.net. E-mail to this address will also come to your INBOX. This server is a new Linux distribution - [Debian Login Trouble](https://www.eskimo.com/2018/09/debian-login-trouble/) - Received a report that people were unable to login to Debian this afternoon. I found that ypbind had died but there was no indication in the logs as to why. I restarted it and logins are functioning normally again. - [Slow Service Early Wednesday Morning (1-4AM)](https://www.eskimo.com/2018/09/slow-service-early-wednesday-morning-1-4am/) - Slow service early this morning and the temporary unavailability of mail.eskimo.com was the result of a denial of service attack where upon our name servers were used as amplifiers in a denial of service attack aimed at us. I had to lower the external view rate limit because of this, hopefully it is still - [Reboots Completed about 1AM](https://www.eskimo.com/2018/09/reboots-completed-about-1am/) - Sorry I neglected to post the completion. Reboots went smoothly, only Debian failed to properly remount file systems. - [Reboots Starting](https://www.eskimo.com/2018/09/reboots-starting/) - I will be starting reboots now. - [Reboots and other Maintenance](https://www.eskimo.com/2018/08/reboots-and-other-maintenance/) - I will be doing another round of reboots sometime after midnight Friday night / Saturday morning and then some machines will be out of service 20-30 minutes for imaging. ## Pages - [Home](https://www.eskimo.com/) - Eskimo North Fast Hosting Linux Friendly Virtual Private Servers, Shell Accounts, and Hosting. Tired of dealing with Godaddy, Hostgater, BlueHost? Contact Us! Our Servers have full support for https and http/2. Knowledgeable human assistance via phone, e-mail, or tickets. Call Us Now at (206) 812-0051 or (800) 246-6874. Web Apps Web based applications. These are - [Server Down](https://www.eskimo.com/server-down/) - The server with the social media sites friendica, hubzilla, mastodon, misskey, and the yacy search engine and, roundcube, manjaro, and alma, is currently down, and because we have a vet appointment for our dogs rabbis shot and I'm down to one car at present waiting for a new wheel to get here from Alabama, - [Encryption Fixed](https://www.eskimo.com/encryption-fixed/) - We received a new certificate with currently operable intermediate certificate servers and installed it. Mail TLS/SSL is functional again. - [Social Is Down for a Rebuild](https://www.eskimo.com/social-is-down-for-a-rebuild/) - I was unable to get the ELGG content management system to do some of the things I needed, like inline videos and images in the right contexts, so I've decided to scrap it and try something different. The forums are still up however and now do support inline images and video. - [Privacy Policy](https://www.eskimo.com/privacy-policy/) - Who we are Our website address is: https://www.eskimo.com. What personal data we collect and why we collect it EXTERNAL APPS Many external applications also collect data. If you do not wish your real name collected you can use a pseudonym in these applications. Many do not have provisions for dumping or erasing data so please - [Zorin, Same as it ever was...](https://www.eskimo.com/zorin-same-as-it-ever-was/) - I was unable to get gnome-flashback to work on Zorin. I can't find any obvious reason it does not work. I've put the machine back the way it was. - [gtmetrix](https://www.eskimo.com/gtmetrix/) - [Eskimo North User Meeting](https://www.eskimo.com/eskimo-north-user-meeting/) - Our next meeting will be held on October 29th 2016 4PM-7PM at Spiros Pizza on Aurora Avenue North and 185th ST N in Shoreline (in the Fred Meyer Parking Lot). It is located on the Northwest corner of the Intersection. There is a parking lot behind the building and also on-street parking (several - [test](https://www.eskimo.com/test/) - [php snippet=1] - [Chat](https://www.eskimo.com/chat/) - Chat Right now this page is just a place holder. I am trying to get an XMPP/Jabber server and web based chat client to work under WordPress. The idea is that you will either be able to originate a chat session on the web or using a XMPP/Jabber chat app on a mobile - [Subscriptions](https://www.eskimo.com/subscriptions/) - Subscribing allows every post to Eskimo North News to appear in your INBOX. - [New-Apply](https://www.eskimo.com/new-apply/) - Apply for a Linux Shell Account [vfb id=1] - [environment](https://www.eskimo.com/environment/) - Environment [environment] - [Facebook Comments](https://www.eskimo.com/facebook-comments/) - [Reviews](https://www.eskimo.com/reviews/) - Reviews We appreciate it if you would take the time to write a review at any of the sites below. BBB Business Review City Pages City Squares Cylex enQuira Find Us Here Google+ Host Search Hubbiz Local Database Merchant Circle Mojo Pages Super Pages SwitchBoard Web Hosting Top Yahoo Yellow Bot Yellow Pages Go Green - [Kudos](https://www.eskimo.com/kudos/) - I imagine that most of the email you get is because of problems and that people rarely go to the effort of taking the time to commend great service. I'm simply writing to tell you that your service has been great since you've completed your updates to the systems. I can only describe it as - [Improve](https://www.eskimo.com/improve/) - --- improve --- Additional documentation and maybe a forum to ask other users for suggestions on what to use or how to go about getting things set up would be nice. Half of my challenge is knowing what already exists and what it does. If I know that, I can read the documentation. If I - [Device](https://www.eskimo.com/device/) - --- device --- Desktop Computer --- device --- Desktop Computer --- device --- Desktop Computer --- device --- Desktop Computer --- device --- Other --- device --- Desktop Computer --- device --- Desktop Computer --- device --- Desktop Computer --- device --- Desktop Computer --- device --- Desktop Computer --- device --- Desktop Computer --- - [Complexity](https://www.eskimo.com/complexity/) - --- complexity --- Needs Documentation --- complexity --- Just Right --- complexity --- Just Right --- complexity --- Just Right --- complexity --- Needs Documentation --- complexity --- Just Right --- complexity --- Just Right --- complexity --- Just Right --- complexity --- Just Right --- complexity --- Needs Documentation --- complexity --- Needs Documentation - [Dislikes](https://www.eskimo.com/dislikes/) - --- dislikes --- I don't do as much Unix as I used to so I'm not keeping up and it isn't as easy to remember stuff I used to do. --- dislikes --- can't think of anything. --- dislikes --- Sometimes I have trouble understanding the technical aspects of a server --- dislikes --- Nothing - [Likes](https://www.eskimo.com/likes/) - --- likes --- As a developer I like access to the shell. I run a couple of small jobs here so I don't need a server at home. I also run some stuff on the web here. I have recommended it to other IT folks I know because of the features and knowledgeable folks. I - [Account](https://www.eskimo.com/account/) - [Directory](https://www.eskimo.com/directory/) - [Friends](https://www.eskimo.com/friends/) - [Activity](https://www.eskimo.com/activity/) - [Signup](https://www.eskimo.com/signup/) - After you sign up, please take a moment to fill out your profile. Users with empty profiles, particularly those with non-human sounding names and throw-away addresses like hotmail.com, are likely to be deleted quickly. - [Inbox](https://www.eskimo.com/inbox/) - [External Links](https://www.eskimo.com/external-links/) - Items in this drop-down menu are external to Eskimo North. Facebook Page Twitter Page Yahoo Groups - [Subscribe2](https://www.eskimo.com/subscribe2/) - You can subscribe or unsubscribe to e-mail notifications of new posts. ## Categories - [Uncategorized](https://www.eskimo.com/category/uncategorized/) - [Eskimo](https://www.eskimo.com/category/eskimo/) ## Tags - [whitelist](https://www.eskimo.com/tag/whitelist/) - [blacklist](https://www.eskimo.com/tag/blacklist/) - [ffff00](https://www.eskimo.com/tag/ffff00/) - [ff6600](https://www.eskimo.com/tag/ff6600/) - [ff99cc](https://www.eskimo.com/tag/ff99cc/) - [ff00ff](https://www.eskimo.com/tag/ff00ff/) - [000080](https://www.eskimo.com/tag/000080/) - [000000](https://www.eskimo.com/tag/000000/) - [003300](https://www.eskimo.com/tag/003300/) - [0000ff](https://www.eskimo.com/tag/0000ff/) - [800000](https://www.eskimo.com/tag/800000/) - [16](https://www.eskimo.com/tag/16/) - [2](https://www.eskimo.com/tag/2/) - [333399](https://www.eskimo.com/tag/333399/) - [ff0000](https://www.eskimo.com/tag/ff0000/)