Database is taking longer than I expected to recover.

     The issue was caused by the fact that innodb-file-per-table was not set to true.  This caused all the innodb tables to be stored in the system file.  The problem with this is that space is never recovered and the file grows until the disk is full and then the database crashes.  For this reason, Mariadb has shipped with this set by default after version 10.2 (we are on 10.6).  However, Ubuntu, in their infinite wisdom, ships this with the distro with it NOT set.  I’ve been bit with this before when I installed the last server so should have known to check but it’s been a few years and memory cells are aging.

      To fix this I first needed to copy the entire /var/lib/mysql directory to a larger disk since I can’t even start the database with disk full.  With more than 600GB of data, this took a while.  Then I’ve got to dump all the databases, and there is around 160GB of legitimate data to dump so this also takes a while.  After that I can delete the ibdata1 file and log, then copy all the remaining system files back to the original disk, then with this configuration option correctly set, restore the database from the dump.  There is no shortcut, without losing data, that I am aware of.