posted on 2016-04-23 23:13
When running loadbalanced applications, in particular a redundant webserver, you have several approaches at your disposal.
In the following it is assumed, that you have a setup with a dedicated Firewall facing externally like this:
* ---FIREWALL --- LOADBALANCER --- WEB SERVER 1 | WEB SERVER 2
SetEnvIf <custom-flag>in apache, i.e. use the origin IP in question
In case you have had a running setup, which stopped working some times after a while:
Especially, if your configuration were complexer, like the web servers were the front-ends for an application server backend which in turn fronts a database, you might as well check all your timeouts. Maybe you have longer running queries than you did when the application was freshly set up, and you now hit certain tresholds.
A good rule of thumb is have all timeouts set up with equal values. Of course it's a nice idea to change all timeouts when changing the front-end up front...
Besides setting up your main ip onto the loadbalancer, give each of the webservers dedicated ips, too. So you can reach them directly, in case you need to test nodes indepently.
Since you usually don't want the web servers to be publicly reachable, they are within their own private subnetwork. Set up further public ips up onto the firewall, that all of these let you reach the firewall.
Beside the main IP which is 1:1 NAT-ted to the LOADBALANCER, do 1:1 NAT's towards the web servers with the other two IP's. Just make sure you restrict access by filtering all IP's besides your own on the firewall.
Now even when a server is removed from the loadbalancer and thus not publicly reachable anymore, for testing purposes it can still be accessed.
If you can terminate your HTTPS encryption at the loadbalancer, do it. Besides lessening your server load, it also helps you with not having to decrypt packets when anaylzing packetdumps.
There are scenarios where you will not want that, but if you know that to be the case, you know the solution anyway, too.
If you wonder wether you can reach both web servers at all, and 'sticky' sessions are enabled on the loadbalancer, clear your browser cache. Cookies are then used to lead you always onto the same webserver.
Redo it several times, if you do not succeed at first. That however implicates you know the loadbalancing strategy to work somehow alternating both servers.
If the loadbalancer is using a somehow 'fixed' distribution algorithm, it may effectively create an active-passive setup: Thus you can only reach the second webserver, if the primary one is either removed or simply turned off.
When wondering where you lose packets, to long-running packet dumps at the firewall as well as the loadbalancer and the webservers. So you can compare where the network eats your packets or which node is misconfigured.
Don't forget to filter the packets by your local workstation IP (or the ip of the gateway where it is behind), so you don't have to put up with visual information overload.
Special bonus tip here, if you want real time server debugging with wireshark:
I can't do a more detailed description as I am currently in a hurry, but I might do so once I need this again when a
tcpdump -vvv -XXX does not suffice.
View posts from 2017-05, 2017-04, 2017-03, 2017-02, 2017-01, 2016-12, 2016-11, 2016-10, 2016-09, 2016-08, 2016-07, 2016-06, 2016-05, 2016-04, 2016-03, 2016-02, 2016-01, 2015-12, 2015-11, 2015-10, 2015-09, 2015-08, 2015-07, 2015-06, 2015-05, 2015-04, 2015-03, 2015-02, 2015-01, 2014-12, 2014-11, 2014-10, 2014-09, 2014-08, 2014-07, 2014-06, 2014-05, 2014-04, 2014-03, 2014-01, 2013-12, 2013-11, 2013-10