The machine that is doing double duty as a webhost and router is ill.  I have it running in a crippled state but if it reboots it will not come up automatically.

     For those who are interested in the technical details and may have encountered this before and can provide some hints, what is happening is that when it boots it tried to create system users with systemd-sysusers.service which runs systemd-sysusers as a one-shot, it does not complete however and times out.

     The other thing that is not working is mdmonitor which uses mdadm to build the RAID devices (everything on this machine is on RAID).

     So after both of these timeout, I can login to single user shell, run them by hand, and they run fine.  Hence the big mystery.  I am going to go back tonight and try rebuilding initramfs on the off chance it is broken.  If that fails I’m going to attempt an upgrade to 24.04, I did manage to get the old web server which was doing something similar working this way.

     But if worst comes to worst I am going to have to re-install the machine and that may take a while.  So service may be spotty tonight.  I apologize for that but sometimes things just don’t give you an option.