I did not expect last night to be a maintenance night but it had other ideas.  Around 3AM name service for our own hosts failed although our name servers were still resolving external hosts fine.

     The issue was caused by an update that replaced the named apparmor profile on the master name server.  We use a hidden master (a master which is not reachable from the outside world) for security reasons.  I use a file system layout for named that is different from the default Ubuntu layout.  I had to modify the apparmor profile accordingly.  When the update replaced my modified profile, it resulted in named not being able to read some of it’s configuration files and then failing to resolve local host names as a result.

     This has been corrected.