When ice, the server that previously experienced the hard drive failure failed, I moved the network connection to another server, inuvik, but, I had been using ice because it is the only machine which has all Intel network interfaces.

     The other machines all have one Intel and one Realtek, thus to use them as a router requires the use of a Realtek NIC on those machines.  The device drivers for Realtek network interfaces under Linux have been dodgy, last time I had to do this we could not get the NIC to see carrier at 1gb/s so had to run at 100mb/s temporarily but this time it saw carrier fine and was stable for a week which is some kind of record.  But when I got to the co-lo the NIC was totally locked up.  Not even a reboot got it going, had to power cycle the machine.

     So now that ice has a new drive, the network connection is moved back to it and things should be stable (I know, famous last words).