Posts tagged loadbalancing

On troubleshooting load-balanced applications

posted on 2016-04-23 23:13

When running loadbalanced applications, in particular a redundant webserver, you have several approaches at your disposal.

In the following it is assumed, that you have a setup with a dedicated Firewall facing externally like this:


Add a custom HTTP Header via the Webserver

  • can be witnessed in browser dev console
  • can be seen in packetdumps like tcpdumps or in wireshark
  • SetEnvIf <custom-flag> in apache, i.e. use the origin IP in question

check timeouts

In case you have had a running setup, which stopped working some times after a while:

Especially, if your configuration were complexer, like the web servers were the front-ends for an application server backend which in turn fronts a database, you might as well check all your timeouts. Maybe you have longer running queries than you did when the application was freshly set up, and you now hit certain tresholds.

A good rule of thumb is have all timeouts set up with equal values. Of course it's a nice idea to change all timeouts when changing the front-end up front...

Turn one node off

  • simply deactivate one node in the loadbalancer and see if you can spot a difference

make all nodes directly reachable

Besides setting up your main ip onto the loadbalancer, give each of the webservers dedicated ips, too. So you can reach them directly, in case you need to test nodes indepently.

Since you usually don't want the web servers to be publicly reachable, they are within their own private subnetwork. Set up further public ips up onto the firewall, that all of these let you reach the firewall.

Beside the main IP which is 1:1 NAT-ted to the LOADBALANCER, do 1:1 NAT's towards the web servers with the other two IP's. Just make sure you restrict access by filtering all IP's besides your own on the firewall.

Now even when a server is removed from the loadbalancer and thus not publicly reachable anymore, for testing purposes it can still be accessed.

terminate SSL connections at the loadbalancer

If you can terminate your HTTPS encryption at the loadbalancer, do it. Besides lessening your server load, it also helps you with not having to decrypt packets when anaylzing packetdumps.

There are scenarios where you will not want that, but if you know that to be the case, you know the solution anyway, too.

clear your cache

If you wonder wether you can reach both web servers at all, and 'sticky' sessions are enabled on the loadbalancer, clear your browser cache. Cookies are then used to lead you always onto the same webserver.

Redo it several times, if you do not succeed at first. That however implicates you know the loadbalancing strategy to work somehow alternating both servers.

If the loadbalancer is using a somehow 'fixed' distribution algorithm, it may effectively create an active-passive setup: Thus you can only reach the second webserver, if the primary one is either removed or simply turned off.


When wondering where you lose packets, to long-running packet dumps at the firewall as well as the loadbalancer and the webservers. So you can compare where the network eats your packets or which node is misconfigured.

Don't forget to filter the packets by your local workstation IP (or the ip of the gateway where it is behind), so you don't have to put up with visual information overload.

Special bonus tip here, if you want real time server debugging with wireshark:

  • set up an ssh tunnel, so you can stream data back to your workstation
  • create a FIFO queue file
  • start netcat server locally and pipe its output to the queue
  • open this file in wireshark
  • on the server start the tcpdump piped into netcat sending all data through the tunnel so it reaches your workstation

Happy debugging. I can't do a more detailed description as I am currently in a hurry, but I might do so once I need this again when a tcpdump -vvv -XXX does not suffice.

This blog covers .csv, .htaccess, .pfx, .vmx, /etc/crypttab, /etc/network/interfaces, /etc/sudoers, /proc, 10.04, 14.04, AS, ASA, ControlPanel, DS1054Z, GPT, HWR, Hyper-V, IPSEC, KVM, LSI, LVM, LXC, MBR, MTU, MegaCli, PHP, PKI, R, RAID, S.M.A.R.T., SNMP, SSD, SSL, TLS, TRIM, VEEAM, VMware, VServer, VirtualBox, Virtuozzo, XenServer, acpi, adaptec, algorithm, ansible, apache, apachebench, apple, applet, arcconf, arch, architecture, areca, arping, asa, asdm, autoconf, awk, backup, bandit, bar, bash, benchmarking, binding, bitrate, blackarmor, blockdev, blowfish, bochs, bond, bonding, booknotes, bootable, bsd, btrfs, buffer, c-states, cache, caching, ccl, centos, certificate, certtool, cgdisk, cheatsheet, chrome, chroot, cisco, clamav, cli, clp, clush, cluster, coleslaw, colorscheme, common lisp, configuration management, console, container, containers, controller, cron, cryptsetup, csync2, cu, cups, cygwin, d-states, database, date, db2, dcfldd, dcim, dd, debian, debug, debugger, debugging, decimal, desktop, df, dhclient, dhcp, diff, dig, display manager, dm-crypt, dmesg, dmidecode, dns, docker, dos, drivers, dtrace, dtrace4linux, du, dynamictracing, e2fsck, eBPF, ebook, efi, egrep, emacs, encoding, env, error, ess, esx, esxcli, esxi, ethtool, evil, expect, exportfs, factory reset, factory_reset, factoryreset, fail2ban, fbsd, fdisk, fedora, file, filesystem, find, fio, firewall, firmware, fish, flashrom, forensics, free, freebsd, freedos, fritzbox, fsck, fstrim, ftp, ftps, g-states, gentoo, ghostscript, git, git-filter-branch, github, gitolite, global, gnutls, gradle, grep, grml, grub, grub2, guacamole, hardware, haskell, hdd, hdparm, hellowor, hex, hexdump, history, howto, htop, htpasswd, http, httpd, https, i3, icmp, ifenslave, iftop, iis, imagemagick, imap, imaps, init, innoDB, innodb, inodes, intel, ioncube, ios, iostat, ip, iperf, iphone, ipmi, ipmitool, iproute2, ipsec, iptables, ipv6, irc, irssi, iw, iwconfig, iwlist, iwlwifi, jailbreak, jails, java, javascript, javaws, js, juniper, junit, kali, kde, kemp, kernel, keyremap, kill, kpartx, krypton, lacp, lamp, languages, ldap, ldapsearch, less, leviathan, liero, lightning, links, linux, linuxin3months, lisp, list, livedisk, lmctfy, loadbalancing, locale, log, logrotate, looback, loopback, losetup, lsblk, lsi, lsof, lsusb, lsyncd, luks, lvextend, lvm, lvm2, lvreduce, lxc, lxde, macbook, macro, magento, mailclient, mailing, mailq, manpages, markdown, mbr, mdadm, megacli, micro sd, microsoft, minicom, mkfs, mktemp, mod_pagespeed, mod_proxy, modbus, modprobe, mount, mouse, movement, mpstat, multitasking, myISAM, mysql, mysql 5.7, mysql workbench, mysqlcheck, mysqldump, nagios, nas, nat, nc, netfilter, networking, nfs, nginx, nmap, nocaps, nodejs, numberingsystem, numbers, od, onyx, opcode-cache, openVZ, openlierox, openssl, openvpn, openvswitch, openwrt, oracle linux, org-mode, os, oscilloscope, overview, parallel, parameter expansion, parted, partitioning, passwd, patch, pct, pdf, performance, pfsense, php, php7, phpmyadmin, pi, pidgin, pidstat, pins, pkill, plasma, plesk, plugin, posix, postfix, postfixadmin, postgres, postgresql, poudriere, powershell, preview, profiling, prompt, proxmox, ps, puppet, pv, pveam, pvecm, pvesm, pvresize, python, qemu, qemu-img, qm, qmrestore, quicklisp, quickshare, r, racktables, raid, raspberry pi, raspberrypi, raspbian, rbpi, rdp, redhat, redirect, registry, requirements, resize2fs, rewrite, rewrites, rhel, rigol, roccat, routing, rs0485, rs232, rsync, s-states, s_client, samba, sar, sata, sbcl, scite, scp, screen, scripting, seafile, seagate, security, sed, serial, serial port, setup, sftp, sg300, shell, shopware, shortcuts, showmount, signals, slattach, slip, slow-query-log, smbclient, snmpget, snmpwalk, software RAID, software raid, softwareraid, sophos, spacemacs, spam, specification, speedport, spi, sqlite, squid, ssd, ssh, ssh-add, sshd, ssl, stats, storage, strace, stronswan, su, submodules, subzone, sudo, sudoers, sup, swaks, swap, switch, switching, synaptics, synergy, sysfs, systemd, systemtap, tar, tcpdump, tcsh, tee, telnet, terminal, terminator, testdisk, testing, throughput, tmux, todo, tomcat, top, tput, trafficshaping, ttl, tuning, tunnel, tunneling, typo3, uboot, ubuntu, ubuntu 16.04, udev, uefi, ulimit, uname, unetbootin, unit testing, upstart, uptime, usb, usbstick, utf8, utm, utm 220, ux305, vcs, vgchange, vim, vimdiff, virtualbox, virtualization, visual studio code, vlan, vmstat, vmware, vnc, vncviewer, voltage, vpn, vsphere, vzdump, w, w701, wakeonlan, wargames, web, webdav, weechat, wget, whois, wicd, wifi, windowmanager, windows, wine, wireshark, wpa, wpa_passphrase, wpa_supplicant, x11vnc, x2x, xfce, xfreerdp, xmodem, xterm, xxd, yum, zones, zsh

Unless otherwise credited all material Creative Commons License by sjas