Around 2:30pm everything spontaneously rebooted.  Just prior to that our mail server had been under a DDOS attack by a botnet, fail2ban had been working hard to lock out attacking IPs, still the load on the machine was 800, and while I was trying to chase that everything crashed.

     When things came back up, NFS did not work right, mail had quota on the /root partition AND had mounted it read-only.

     I’ve got the physical hosts, web server, mail infrastructure, and ubuntu shell server up and running, I am working on checking the rest of the servers for proper NIS/NFS connectivity.