Around 7pm tonight I found one of my computers was unable to reach our website.  After some investigation I found that DNS had failed, none of our servers responded to queries with information about our own servers although they still would resolve outside addresses okay.

     I chased the problem down to apparmor on our master DNS server which is a stealth DNS server and not publicly accessible.

     In the upgrade from Ubuntu 15.10 to Ubuntu 16.04LTS, Ubuntu renamed all of the DNS directories from /var/named/* to /var/bind/* but did not physically move the directories from their existing location.

     This resulted in our master DNS server being unable to access it’s conf file, log file, or any of the zone records.  This did not immediately cause problems but once the records in the slave servers expired, then they were unable to answer queries regarding those records.

     I corrected the paths in apparmor and now it is able to access it’s conf file.  The master records have been propagated to the slave servers which are now correctly answering queries.