I made an error in editing the /etc/fstab to fix the swap partition and edited out the NFS mounts for home directory and mail spool.  I’ve fixed that but I’m making a backup of the fixed machine before I bring it back into service.

     Debian is restored to service and available.

     I am planning on rebooting mail client server around midnight tonight in order to increase memory allocation because there were still a few failures resulting from being unable to allocate enough memory.

     Debian will be back up in about an hour.  I identified the problem as a bug in systemd-sysusers which causes it to hang if NIS is used with containers and avahi is not present.

      Avahi is not present because I do not want it “discovering” potentially insecure machines on the network and prefer the machine knows only about what I tell it.

      Since NIS is not optional and snap is, I’ve removed snap and that made Debian boot properly.  I am now making a backup to capture the current working configuration.  That will conclude in about an hour at which point the machine will be returned to service.

     Even after restoring from backup, debian still has problems, please use one of the other Debian based servers such as Ubuntu, Mint, Julinux, or Zorin while I sort this out.

     If there is an application on Debian that you need and is not present on one of these other servers, please e-mail support@eskimo.com or submit a ticket with https://www.eskimo.com/support/osTicket/.

     There will be multiple interruptions to Debian while I troubleshoot this issue further.

     Seem to be going down one rabbit hole after another.  I think somehow the system image for Debian has gotten severely corrupted, so I am restoring it from backups made on May 3rd, then will apply updates and bring it current.

     I expect Debian to be operational around 1700 Pacific Daylight Time.

     Found cause of mounting issues was disk controller host was changed from Virtio to SATA, I did this intentionally on Mail but not on Debian so not sure how it got changed, but changing it back corrected the swap mounting issue however the Volatile file and directory creation issue re-emerged even with older systemd, so upgraded everything back to current and will continue to troubleshoot.

     With an older systemd, Debian did not fail at that same spot, but there does seem to be a UUID issue with one of the partitions so will require further reboot after some tweaking.

     Debian didn’t exactly crash, when it booted it did not mount tmpfs on /tmp, this is an in memory file system used for temporary files for speed and efficiency.

     Many hours later it mounted it and this broke systemd that needs access to some of those files.

     I do not know what is causing this delayed mounting of partitions but I had a similar problem with mail when I booted it.  I suspected the newer kernel so I booted off the older kernel, same issue.  Then I installed 5.12.2, the most current, still the same.

     After about four attempts it succeeded (same with mail) and is up and running but until I identify this problem reboots are going to be an issue.  This kind of smells like a Poettering systemd issue but, Googling, I can find no other reports of a similar issue.

     I will continue to investigate but for now the server is back up.

     The kernel upgrades went mostly smooth except for the client mail server.  Had issues with systemd timing out before it mounted all the disk partitions.  This is a virtual machine configured to use a physical partition on a RAID10 device on this host computer and using virtio disk.  This has not been a problem in the past but did not want to work today.  Changing the disk emulation to SATA made it work so apparently there is some ugliness with the virtio devices in the newest kernel although it was not a problem with any of the virtual machines that used a file rather than a partition for their virtual disk.

     At this point the virtual private servers and host computer is now running 5.12.2, this because 5.10.x has not been stable on that machine.  The rest are running 5.10.35.

     https://friendica.eskimo.com/, https://hubzilla.eskimo.com/, https://nextcloud.eskimo.com/ and all other Eskimo North services are back online and operational.  With the exception of client mail access pretty much everything else was online by 11:30PM, but it took slightly longer than that to bring the mail server back online.