Mail

     I have found that what kernel resource ran out on the client mail server was the number of open files.  I’ve bumped that up by 5x, hopefully that will be sufficient.

Dec 3 08:57:22 mail kernel: [371342.776244] VFS: file-max limit 194527 reached
Dec 3 09:43:33 mail kernel: [374113.872327] VFS: file-max limit 194527 reached
Dec 3 09:43:33 mail kernel: [374113.905264] VFS: file-max limit 194527 reached
Dec 3 09:45:17 mail kernel: [374217.999876] VFS: file-max limit 194527 reached
Dec 3 09:45:23 mail kernel: [374223.425007] VFS: file-max limit 194527 reached
Dec 3 09:45:24 mail kernel: [374224.590837] VFS: file-max limit 194527 reached
Dec 3 10:21:11 mail kernel: [376372.038404] VFS: file-max limit 194527 reached
Dec 3 10:21:24 mail kernel: [376385.096118] VFS: file-max limit 194527 reached

Mail Server

     Our client mail server got flaky between around 11AM and 1PM today.

     I am pretty sure the cause is brute force password attacks exhausting some kernel resource but I have not been able to identify the resource being exhausted.

     The reason I believe this is the cause is that in the last two days the number of IP addresses we lock out for these sorts of attacks has increased from a typical number of several hundred to over 15,000.  This is probably the result of a new Windows virus that is allowing the creation of huge botnets.  This is something we see periodically.

     I rebooted the server which restored it to normal functionality and will continue to try to determine what is being exhausted and correct it.

SMTP

Last night I was up until 6AM doing reboots and backups. It is not
unusual for NFS mount points to not mount or NIS to not bind after a reboot.  Those are bugs I am used to and always check for.

But postfix not starting is unusual, I didn’t check, didn’t notice,
went to sleep and so it didn’t get fixed until someone called around 2PM.

I’ll work on some sort of automated monitoring solution.

Tickless Kernels

     If you are running any Debian derived operating system, Ubuntu, Debian, Mint, Zorin, Julinux, etc, you may wish to try a tickless kernel.

     What tickless kernels do for you is eliminate the CPU from having to wake-up to service clock interrupts unless there is actual work to be done.  This saves a significant amount of CPU time on a single machine these days, particularly if Intel, because the overhead of context switching has increased significantly as the result of various CPU flaws that have to be worked around.

     This can significantly improve battery life in a laptop or tablet.

     Where this kernel really shines though is on the server side when hosting multiple virtual machines.  Each virtual machine has it’s own clock ticks wasting CPU, plus the host.  This can end up eating more CPU than the actual work the machines are doing.

     I have made available two kernels both based upon the recently released 5.4 final. One is called “client” and is intended for end user systems where interactive response and low latency are important.  This kernel is entirely preemptive.

     The second is called “server” and is for server loads like physical hosts hosting virtual machines, web servers, mail servers, etc.  It is the same as the client kernel except that it is non-preemptive.

     Both of these are based upon Ubuntu 19.10 configuration except modified to be completely tickless and modified to work on the i7-6850k / Asus z-190 systems.  The stock kernels do not talk to the network on these systems.  ALL CPUs and hardware supported by the Ubuntu kernels plus the above is supported by these.

     You can download these via ftp from ftp.eskimo.com:

        Login: anonymous

        Password: You@whereever.net (your e-mail address).

     Complaints, suggestions, questions can be sent to: nanook@eskimo.com

     After logging in cd /pub/kernel/linux-5.4-tickless/[client|server]

     Then prompt off

     mget *.deb

     After you’ve retrieved the files install with:

     dpkg -i *.deb

     These kernels aren’t signed.  Sorry but I’m not willing to buy in to the Microsoft Secure Boot extortion scheme, so you will need to have secure boot turned off to use these.