Eskimo North Users Meeting
Saturday December 19th 2pm
Spiro’s on 185th North and Aurora Shoreline

Look For This Sign on a Table

     I am done with tonight’s maintenance.  The host machines are now up and running on the current kernel.

     Dear Eskimo North Customer,

     This is a reminder that the first Eskimo North users meeting will be
held this Saturday starting at 2PM and ending whenever it does at Spiro’s
Pizza in the Fred Meyer parking lot on Aurora Ave N just south of 185th:

        https://www.eskimo.com/eskimo-north-user-meeting/

     My eldest Son, Carl, will be probably be there for the first hour,
between 2pm-3pm, after that he has a wedding party to attend, so if you’d
like to meet the person who held down the fort in my absence that is the
time frame he will be there.

     So far only 31 people have signed up for the new forums and only
three have left introductions.

     I really want Eskimo North to meet your needs and to that end I
really need your input. For that input to be useful, it’s best in a
public setting as it allows ideas to build upon ideas and for some
congruence to emerge.

     Two ways to get to forums:

        https://www.eskimo.com/forums/

     Or go to the main website:

        https://www.eskimo.com

     Web Apps -> Forums

     Thank you and have a Merry Christmas!

        https://www.biblegateway.com/passage/?search=Psalm+150

     nanook@eskimo.com

     Sometime after 10PM tonight I will be rebooting the host machines to load new kernels and libraries. This will interrupt all services for a 10-15 minute period.

     I am going to go down to the co-location facility as there are presently bugs in the systemd scripts that can cause the server to hang during shutdown.  This way I can monitor it in person and if it hangs immediately force a reboot so there won’t be a prolonged interruption.

     Even if the shutdown goes smoothly, it will still be 10-15 minutes because of the time it takes to save the guests before the reboot and restore them after.

     I will be out of the office today as I have a doctors appointment during this time frame.

     The solution I’ve got in place now is modreqtimeout with header and body timeouts in place so people can’t just create a connection and stall it indefinitely.  This should mitigate the slow httpd style attacks we received yesterday.  This module is included in the stock Apache distribution so my home is that it is more stable and better tested.

     I’ve got a partial fix in for the type of attacks yesterday that does not require mod_qos but while it will keep someone from eating all the available slots, it won’t stop them from placing a heavy load on the server as mod_qos does.

     I am still looking for a more complete but stable fix.  It’s too bad mod_qos wasn’t stable as it did mitigate the attacks very effectively.

     The web server stopped serving all traffic this morning even though it was still running.  I’m assuming this is a bug in mod_qos which I installed yesterday to mitigate a type of denial of service attack as I’ve never seen this behavior from apache before.

     Consequently I’ve removed that module but still have some timeouts in place that should at least reduce the severity of this type of attack and will research the problem with mod_qos to see if either there is a fix or an alternate solution to this problem.

     Had some problems with MySQL but it is repaired.

Current Time: Sunday, 06-Dec-2015 13:50:16 PST

Restart Time: Sunday, 06-Dec-2015 13:38:00 PST

Parent Server Config. Generation: 1

Parent Server MPM Generation: 0

Server uptime: 12 minutes 15 seconds

Server load: 15.81 19.11 21.48

Total accesses: 16901 – Total Traffic: 103.4 MB

CPU Usage: u28.99 s33.72 cu2241.71 cs305.58 – 355% CPU load

23 requests/sec – 144.1 kB/second – 6.3 kB/request

24 requests currently being processed, 476 idle workers

     They are hammering the server now.  Normal traffic is between 4-7 requests/sec save for the occasional special event but it’s hanging in there just fine.

     The 355% CPU load is deceptive as the web server doesn’t know there are multiple cores and take that into account in it’s calculation.

     It is slowing things slightly but still very usable where as prior to installing mod_qos, it was unusuable.