It appears the web server crashed at 17:09 but nobody called until 19.35. It has been restored to service. I am unable to determine the cause at this point. I will be changing the kernel later tonight on the off chance it is a kernel issue. It left the machine in a state without any errors on the console and nothing logged in the way of errors that provided any clues.
It is restored to service.
An update to bind 9 broke the name server by requiring a very different configuration. I had been meaning to move it to a more modern platform anyway so this is in the works. Until this is completed, this name server will not function.
Maintenance was completed around 4:30 AM this morning. Anything down is broken. Please report anything down using either https://www.eskimo.com/ -> Support -> Tickets or e-mail support@eskimo.com or call 206-812-0051. Thank you.
I did not complete all the maintenance I had intended last night so manjaro and social are presently down for maintenance. Expected down time less than one hour.
I will be performing maintenance activity tonight after about 11PM that will involve the reboot of various servers and taking some down for backup imaging. This requires about 15-45 minutes per machine. Where possible I will attempt to leave functional mates up but this will not always be possible, for example when rebooting igloo or iglulik which have the mail spool and home directories on them respectively, both of these will result in unavoidable interruptions system wide.