I will be taking the server which houses home directories down to attempt again to add disks again. This time equipped with 3-1/2 to 5-1/4 adapters and power cables. Expected downtime is about an hour but I will also be rebooting the other machines to get them off of the 5.0.0 kernel which has proven to be flakey.
Login problems were caused by the fact that the syntax required by the external users module, used to allow login with Eskimo system logins, changed between 14.10 and 15.07 but nothing bothered to check this or tell me about it. The config.php syntax has been corrected to work with the newer version.
You will need to agree to terms of service, if you do not, service will not be provided. If you are using automated apps, before they will work you will need to login to the website:
https://www.eskimo.com/nextcloud/
And tick the ‘Accept Terms of Service’ box.
NextCloud updates were completed last night. We are now on version 15.07. Some applications were removed because no updated versions were available. Other new applications were added.
An upgrade from 14.06 to 14.10 broke several applications for which no update was available. I will be cleaning those up and then proceeding with an upgrade to 15.x this evening. This is somewhat involved so may not be finished for several days.
I will be taking the big server that houses all users home directory off line to add additional hard drives Friday evening probably starting around 10:30PM. This will require pausing incoming e-mail as well since .procmailrc files and local e-mail boxes reside in home directories. Shell servers will be unavailable and all web services will be down. I expect this maintenance to take less than an hour.
Later in the morning I will be booting some servers into 5.1 kernels. 5.0.0 has proven to be an extremely unstable kernel so trying to get everything off of it before it breaks anything else like it did this morning. This will only involve brief outages.
Sometime this morning Igloo, the server that hosts the mail spool, just stopped serving files in /mail via NFS. It wasn’t a case of NFS dying altogether as other files systems still worked.
I wasn’t able to troubleshoot much further because when I got to the co-location facility, the console was dead and it would not allow remote logins. I was forced to power cycle to reboot after which everything appeared normal.
These machines were recently upgraded from Ubuntu 18.10 to 19.04 and that upgrade included the first 5.0.0 Linux kernel. I have had issues with that kernel on other machines and have resolved them by compiling 5.1 which appears to be more stable. I am doing that for this machine now which means there will be an additional interruption when I boot into the new kernel.
For workstations 19.04 is mostly okay, for servers it’s a piece of crap and has kept me up all night last night and tonight fixing things it broke. The new kernel, 5.0.0-14 is basically non-functional, networking either doesn’t work at all or randomly goes away. One machine went into hibernate mode all by it’s lonesome. I am so disappointed that Canonical would release such a half-baked distribution, not at all like them.
Well hopefully I will get home and things will be working this time. I am tired and hungry.
I got about 30% of what I set out to do tonight done. Consequently there will be additional downtime tonight and next weekend. I’m going to have to order an external case. I thought I had more internal drive bays than I do.