Posts tagged performance

linux: systemcheck in 60 seconds

posted on 2016-01-21 00:03

This post is a completely copied from @brendangregg from here. Just in shorter and typed by me in hope I can memorize it easier that way, plus a little change with including htop.

Summary: check a linux system for problems immediatly after ssh'ing onto it.

  1. htop - uptime, core diversity, load, swap on first sight via a TUI.
  2. uptime - for load checking, likely unnnecesary after htop
  3. dmesg | tail - check for errors like out of memory
  4. vmstat 1 - check amount of processes (r) and kernel/userland distribution and swap
  5. mpstat -P ALL 1 - check for a single hot core
  6. pidstat 1 - check for high load on single process
  7. iostat -xz 1 - high r/w load? awaits? util%?
  8. free -m - memory available, likely unneded after htop
  9. sar -n DEV 1 - rxkb/s or txkb/s is 125mbytes max for 1G NICs, util% ok?
  10. sar -n TCP,ETCP 1 - act = egress, pasv = ingress traffic, retransmits = bad, usually
  11. top - zxcV and 1 and < and > are your best friends, along with knowing status indices.


At 7. buffers = block device caching, cache = page cache for file system.

At 10. just switch columns through the angle brackets keys and have a look the waits (wa) to see if there are disk related issues, after having pressed 1 to show all available cores. d with a number after changes the refresh time to x seconds. In general everything concerning top can be found in the manual.

Lastly, a list of the process states from the mentioned top man page:

D = uninterruptible sleep <<-- waiting for disk
R = running
S = sleeping
T = traced or stopped
Z = zombie

Linux performance observability tools

posted on 2015-01-17 18:50:42

This is an alphabetical list which will serve as a reminder, what programs are there to be looked up for me. :)

All this started when I stumbled across a picture on the web, which was from a presentation from Brendan Gregg at LinuxCon14 as I later found out. It was called Linux Performance Tools and it's worth its words in gold, platin and whatever material you see as highly valuable. The slides are here, get your copy and study them. If you want some serious linux sysadmin skills, there is no possible excuse for not doing it.



Another two incentives can be found here and here. These may only use a small portion of the later mentioned programs, but either walk the extra miles, or raise your hands in defeat once things get tough, everybody gets to choose man's own path.

Alphetically sorted:

blktrace (8)         - generate traces of the i/o traffic on block devices
dstat (1)            - versatile tool for generating system resource statistics
dtrace (1)           - Dtrace compatibile user application static probe generation tool.
ebpf: nothing appropriate.
ethtool (8)          - query or control network driver and hardware settings
free (1)             - Display amount of free and used memory in the system
ftrace: nothing appropriate.
iostat (1)           - Report Central Processing Unit (CPU) statistics and input/output statistics for devices and partitions.
iotop (8)            - simple top-like I/O monitor
ip (8)               - show / manipulate routing, devices, policy routing and tunnels
iptraf (8)           - Interactive Colorful IP LAN Monitor
ktap: nothing appropriate.
lldptool (8)         - manage the LDP settings and status of lldpad
lsof (8)             - list open files
ltrace (1)           - A library call tracer
lttng: nothing appropriate.
mpstat (1)           - Report processors related statistics.
netstat (8)          - Print network connections, routing tables, interface statistics, masquerade connections, and multicast memberships.
nicstat: nothing appropriate.
pcstat: nothing appropriate.
perf (1)             - Performance analysis tools for Linux
pidstat (1)          - Report statistics for Linux tasks.
/proc: nothing appropriate.
ps (1)               - report a snapshot of the current processes.
rdmsr: nothing appropriate.
sar (1)              - Collect, report, or save system activity information.
slabtop (1)          - display kernel slab cache information in real time
snmpget (1)          - communicates with a network entity using SNMP GET requests
ss (8)               - another utility to investigate sockets
stap (1)             - systemtap script translator/driver
strace (1)           - trace system calls and signals
swapon (8)           - enable/disable devices and files for paging and swapping
sysdig ()            - the definitive system and process troubleshooting tool
tcpdump (8)          - dump traffic on a network
tiptop (1)           - display hardware performance counters for Linux tasks
top (1)              - display Linux processes
uptime (1)           - Tell how long the system has been running.
vmstat (8)           - Report virtual memory statistics

First some more explanations on the ones listed above with "nothing appropriate":

ebpf, ftrace, ktap, lttng, nicstat, pcstat, /proc, rdmsr are usually all too new. New like either in bleeding edge, or at least not available in CentOS 7 or Debian 7. If you grab the sources, you might get along. The manpage headlines are actually from a CentOS 7. (Only exception is sysdig, which I installed via the one-liner its github page provided.) /proc is of course not a command, but mentions the /proc folder linux uses where a lot of useful information can be found.

Here are some other sortings, by 'types' now. (Maybe this improves readability, or makes it easier to remember, who knows. It's worth trying, still.)

'stat', 'top', 'trace', 'tap':

dstat      iotop      blktrace     ktap
iostat     slabtop    dtrace       stap
mpstat     tiptop     ftrace
netstat    top        ltrace
nicstat               strace

the rest:


This were only the 'observability' tools from the presentation. There are also some more listed on 'benchmarking' and 'tuning', and maybe 'tracing'.

Just go an read up on them. NOW.

This blog covers .csv, .htaccess, .pfx, .vmx, /etc/crypttab, /etc/network/interfaces, /etc/sudoers, /proc, 10.04, 14.04, 16.04, AS, ASA, ControlPanel, DS1054Z, GPT, HWR, Hyper-V, IPSEC, KVM, LSI, LVM, LXC, MBR, MTU, MegaCli, PHP, PKI, PS1, R, RAID, S.M.A.R.T., SNMP, SSD, SSL, TLS, TRIM, VEEAM, VMware, VServer, VirtualBox, Virtuozzo, XenServer, acpi, adaptec, algorithm, ansible, apache, apache2.4, apachebench, apple, applet, arcconf, arch, architecture, areca, arping, asa, asdm, autoconf, awk, backup, bandit, bar, bash, benchmarking, binding, bitrate, blackarmor, blockdev, blowfish, bochs, bond, bonding, booknotes, bootable, bsd, btrfs, buffer, c-states, cache, caching, ccl, centos, certificate, certtool, cgdisk, cheatsheet, chrome, chroot, cisco, clamav, cli, clp, clush, cluster, cmd, coleslaw, colorscheme, common lisp, configuration management, console, container, containers, controller, cron, cryptsetup, csync2, cu, cups, cygwin, d-states, database, date, db2, dcfldd, dcim, dd, debian, debug, debugger, debugging, decimal, desktop, df, dhclient, dhcp, diff, dig, display manager, dm-crypt, dmesg, dmidecode, dns, docker, dos, drivers, dtrace, dtrace4linux, du, dynamictracing, e2fsck, eBPF, ebook, efi, egrep, emacs, encoding, env, error, ess, esx, esxcli, esxi, ethtool, evil, expect, exportfs, factory reset, factory_reset, factoryreset, fail2ban, fakeroot, fbsd, fdisk, fedora, file, files, filesystem, find, fio, firewall, firmware, fish, flashrom, forensics, free, freebsd, freedos, fritzbox, fsck, fstrim, ftp, ftps, g-states, gentoo, ghostscript, git, git-filter-branch, gitbucket, github, gitolite, global, gnutls, gradle, grep, grml, grub, grub2, guacamole, hardware, haskell, hdd, hdparm, hellowor, hex, hexdump, history, howto, htop, htpasswd, http, httpd, https, i3, icmp, ifenslave, iftop, iis, imagemagick, imap, imaps, init, innoDB, innodb, inodes, intel, ioncube, ios, iostat, ip, iperf, iphone, ipmi, ipmitool, iproute2, ipsec, iptables, ipv6, irc, irssi, iw, iwconfig, iwlist, iwlwifi, jailbreak, jails, java, javascript, javaws, js, juniper, junit, kali, kde, kemp, kernel, keyremap, kill, kpartx, krypton, lacp, lamp, languages, ldap, ldapsearch, less, leviathan, liero, lightning, links, linux, linuxin3months, lisp, list, livedisk, lmctfy, loadbalancing, locale, log, logrotate, looback, loopback, losetup, lsblk, lsi, lsof, lsusb, lsyncd, luks, lvextend, lvm, lvm2, lvreduce, lxc, lxde, macbook, macro, magento, mailclient, mailing, mailq, make-jpkg, manpages, markdown, mbr, mdadm, megacli, micro sd, microsoft, minicom, mkfs, mktemp, mod_pagespeed, mod_proxy, modbus, modprobe, mount, mouse, movement, mpstat, multitasking, myISAM, mysql, mysql 5.7, mysql workbench, mysqlcheck, mysqldump, nagios, nas, nat, nc, netfilter, networking, nfs, nginx, nmap, nocaps, nodejs, numberingsystem, numbers, od, onyx, opcode-cache, openVZ, openlierox, openssl, openvpn, openvswitch, openwrt, oracle linux, org-mode, os, oscilloscope, overview, parallel, parameter expansion, parted, partitioning, passwd, patch, pct, pdf, performance, pfsense, php, php7, phpmyadmin, pi, pidgin, pidstat, pins, pkill, plasma, plesk, plugin, posix, postfix, postfixadmin, postgres, postgresql, poudriere, powershell, preview, profiling, prompt, proxmox, ps, puppet, pv, pveam, pvecm, pvesm, pvresize, python, python3, qemu, qemu-img, qm, qmrestore, quicklisp, quickshare, r, racktables, raid, raspberry pi, raspberrypi, raspbian, rbpi, rdp, redhat, redirect, registry, requirements, resize2fs, rewrite, rewrites, rhel, rigol, roccat, routing, rs0485, rs232, rsync, s-states, s_client, samba, sar, sata, sbcl, scite, scp, screen, scripting, seafile, seagate, security, sed, serial, serial port, setup, sftp, sg300, shell, shopware, shortcuts, showmount, signals, slattach, slip, slow-query-log, smbclient, snmpget, snmpwalk, software RAID, software raid, softwareraid, sophos, spacemacs, spam, specification, speedport, spi, sqlite, squid, ssd, ssh, ssh-add, sshd, ssl, stats, storage, strace, stronswan, su, submodules, subzone, sudo, sudoers, sup, swaks, swap, switch, switching, synaptics, synergy, sysfs, systemd, systemtap, tar, tcpdump, tcsh, tee, telnet, terminal, terminator, testdisk, testing, throughput, tmux, todo, tomcat, top, tput, trafficshaping, ttl, tuning, tunnel, tunneling, typo3, uboot, ubuntu, ubuntu 16.04, ubuntu16.04, udev, uefi, ulimit, uname, unetbootin, unit testing, upstart, uptime, usb, usbstick, utf8, utm, utm 220, ux305, vcs, vgchange, vim, vimdiff, virtualbox, virtualization, visual studio code, vlan, vmstat, vmware, vnc, vncviewer, voltage, vpn, vsphere, vzdump, w, w701, wakeonlan, wargames, web, webdav, weechat, wget, whois, wicd, wifi, windowmanager, windows, wine, wireshark, wpa, wpa_passphrase, wpa_supplicant, x11vnc, x2x, xfce, xfreerdp, xmodem, xterm, xxd, yum, zones, zsh

Unless otherwise credited all material Creative Commons License by sjas