Web Server Upgrades

     We are always trying to improve the speed, reliability, and functionality of our products.

     To this end we’ve made the following upgrades to our web server:

  • Upgraded to Apache 2.4.41
  • Added read-ahead and writeback caching to the file systems.
  • Added gzip compression to our server configuration.
  • Enabled transparent huge pages for both web server and our physical hosts.
  • Changed the way the network interface is configured.
  • Made changes to systemd configuration to insure prerequisites are up and running before Apache is started.

     The last two items are primarily aimed at improving boot time so during a server reboot downtime is reduced.

     The upgrade to Apache 2.4.41 fixes a couple of flaws in Apache that could be used in a denial of service attack against our web server.

     Gzip compression will improve page load time over slower links by reducing total page time.  Note that Pingdom tests will report that we do not have gzip enabled.  This is untrue but because we include some code that calls Twitter and Twitter does not compress, it causes this test to fail.

     If you have a WordPress plugin that does gzip compression, please disable the gzip function in this plugin.  Otherwise double gzip compression will result which browsers will not know how to handle resulting in gibberish.

     Transparent huge pages is largely a major win for MariaDB used to provide database operations for WordPress, other content management systems, and other applications on our website and on customer websites.  The combination of having this enabled on our physical host machine and web server improves the response time of MariaDB by approximately 400%.

Web Server Crashed

     It appears the web server crashed at 17:09 but nobody called until 19.35.  It has been restored to service.  I am unable to determine the cause at this point.  I will be changing the kernel later tonight on the off chance it is a kernel issue.  It left the machine in a state without any errors on the console and nothing logged in the way of errors that provided any clues.

     It is restored to service.

Tonight 11pm to ~4AM Interruptions

     I will be performing maintenance activity tonight after about 11PM that will involve the reboot of various servers and taking some down for backup imaging.  This requires about 15-45 minutes per machine.  Where possible I will attempt to leave functional mates up but this will not always be possible, for example when rebooting igloo or iglulik which have the mail spool and home directories on them respectively, both of these will result in unavoidable interruptions system wide.

Mail Server and other hosts down

     The machine which hosts the mail spool, client mail server, and some other hosts is currently down.  It failed to reboot because of missing software prohibiting it from accessing the Internet.  Since it can not access the Internet I can not restore the missing software that way.  I attempted to restore from backups but do not know all of the files involved in the package.  I am burning a DVD and will be heading back down to the co-lo to complete the restoration once I have that in hand.