Posts tagged kernel

iptables and netfilter chains diagram

posted on 2016-08-10 18:56

This is a NICE diagram I stumbled across here:

 +---------------------+                              +-----------------------+
 | NETWORK INTERFACE   |                              | NETWORK INTERFACE     |
 +----------+----------+                              +-----------------------+
            |                                                    ^
            |                                                    |
            |                                                    |
            v                                                    |
 +---------------------+                                         |
 | PREROUTING          |                                         |
 +---------------------+                                         |
 |                     |                                         |
 | +-----------------+ |                                         |
 | | raw             | |                                         |
 | +--------+--------+ |                                         |
 |          v          |                                         |
 | +-----------------+ |                              +----------+------------+
 | | conn. tracking  | |                              | POSTROUTING           |
 | +--------+--------+ |                              +-----------------------+
 |          v          |                              |                       |
 | +-----------------+ |                              | +-------------------+ |
 | | mangle          | |                              | | source NAT        | |
 | +--------+--------+ |                              | +-------------------+ |
 |          v          |                              |          ^            |
 | +-----------------+ |                              | +--------+----------+ |
 | | destination NAT | |                              | | mangle            | |
 | +-----------------+ |                              | +-------------------+ |
 +----------+----------+  +------------------------+  +-----------------------+
            |             | FORWARD                |             ^
            |             +------------------------+             |
            v             |                        |             |
     +-------------+      | +--------+  +--------+ |             |
     | QOS ingress +----->| | mangle +->| filter | |------------>+
     +------+------+      | +--------+  +--------+ |             |
            |             |                        |             |
            |             +------------------------+             |
            |                                                    |
            |                                                    |
            v                                                    |
 +---------------------+                              +----------+------------+
 | INPUT               |                              | OUTPUT                |
 +---------------------+                              +-----------------------+
 |                     |                              |                       |
 |  +---------------+  |                              |  +-----------------+  |
 |  | mangle        |  |                              |  | filter          |  |
 |  +-------+-------+  |                              |  +-----------------+  |
 |          v          |                              |          ^            |
 |  +---------------+  |                              |  +-------+---------+  |
 |  | filter        |  |                              |  | destination NAT |  |
 |  +---------------+  |                              |  +-----------------+  |
 +----------+----------+                              |          ^            |
            |                                         |  +-------+---------+  |
            |                                         |  | mangle          |  |
            |                                         |  +-----------------+  |
            |                                         |          ^            |
            |                                         |  +-------+---------+  |
            |                                         |  | conn. tracking  |  |
            |                                         |  +-----------------+  |
            |                                         |          ^            |
            |                                         |  +-------+---------+  |
            |                                         |  | raw             |  |
            |                                         |  +-----------------+  |
            |                                         +-----------------------+
            v                                                    ^
+----------------------------------------------------------------+------------+
|                             LOCAL PROCESS                                   |
+-----------------------------------------------------------------------------+

debian: build newest kernel

posted on 2016-06-05 19:23

prerequisites

You should have like 50G of harddisk space available. Since I currently don't (SSD), I use an external harddisk.

Also:

apt install build-essential kernel-package libssl-dev xz-utils ncurses-dev

These are like ~1,3G of additional files on your main system, if you install without the --no-recommends flag.

get sources

Head over to kernel.org and download the source of the kernel of your choice. You should know what stable means and wether you want to use a release client or not, else get the stable kernel.

I will use the current RC release.

extract, copy current config, start compiling

tar xJvf linux-4.7-rc1.tar.xz
cd linux-4.7-rc1/

# copy currently used configuration. otherwise use `make menuconfig`.
cp /boot/config-`uname -r` .
yes "" | make oldconfig

# see how many cores you can use for compiling
# count amount of cores on top, then press 'q'
htop

# build, pass corecount with '-j' flag
make-kpkg -j4 --append-to-version "-sjas" --initrd buildpackage
# if the previous step failed, do these
## make clean
## rm -rf .config
## make menuconfig # save to .config  and exit immediatly
## then redo the previous make-kpkg

I also had a more specific error last time:

make[4]: *** No rule to make target 'debian/certs/benh@debian.org.cert.pem', needed by 'certs/x509_certificate_list'.  Stop.
Makefile:985: recipe for target 'certs' failed

Solution was to comment the CONFIG_SYSTEM_TRUSTED_KEYS line out from my .config.

Free Ebook: Linux Kernel in a Nutshell

posted on 2016-04-21 20:47

To get a free version of Greg Koah Hartman's 'Linux Kernel in a Nutshell', he hosts pdfs of all sections of the book on his homepage here.

There you can download the files bundled in a .zip file and merge them via pdfsam, the best PDF split/merge tool on the planet.

The assembling order of all single files is like this:

  • title.pdf
  • LKNSTOC.fm.pdf
  • part1.pdf
  • ch00.pdf
  • ch01.pdf
  • ch02.pdf
  • ch03.pdf
  • ch04.pdf
  • ch05.pdf
  • ch06.pdf
  • part2.pdf
  • ch07.pdf
  • ch08.pdf
  • part3.pdf
  • ch09.pdf
  • ch10.pdf
  • ch11.pdf
  • colo.pdf
  • part4.pdf
  • appa.pdf
  • appb.pdf
  • LKNSIX.fm.pdf

Linux Kernel: hello world

posted on 2016-03-12 17:26

intro

Easiest this is done via kernel modules. (TBH I don't know if it is possible otherwise in a feasible way, besides building a completely new kernel?)

So for this you should know how to handle kernel modules:

  • lsmod = show loaded kernel modules

  • insmod <module> = load kernel module

  • rmmod <module> = unload kernel module

  • modprobe <module> = load kernel module and, if needed, its dependencies

  • modprobe -r <module> = unload kernel module and unneded dependencies

This guide is debian-specific

prerequisites

#install build environment
apt install build-essential

# look up your kernel version
uname -a 
apt search linux-headers | grep headers
apt install linux-headers-<YOUR_VERSION_HERE>
mkdir /lib/modules/$(uname -r)/build/

actual module

Linux and dynamic tracing rants

posted on 2016-01-27 21:46:34

This is current work in heavy progress.

tracing is just not overly accessible

When wanting to start with dynamic kernel tracing, usual problems are similar, no matter what technology you want to use:

  • "I don't know where to start."
  • "I don't know how XYZ is done in tracing tool ABC."
  • "I don't know what probes exist."
  • "I don't know what syscalls are existing."
  • "I installed the packages, but this doesn't work?"
  • "I need to copy-paste scripts to make this work?"
  • "Heck, I don't even know what syscalls are."
  • "What can I do with all this stuff?"

Usually the syntax ain't even too bad, it's the points above hindering the further spreading of these tools. There is a pattern there to be found, so this post should do this:

  • Show what tracing is and in what shape the tooling landscape is currently.
  • Provide small examples which are usable to get a proper starting point.
  • Provide one-liners for getting overviews over the currently available tools for all probes and trace-points.
  • Provide one-liners to show how to catch syscalls which took place.
  • Provide detailed install instructions where necessary, but rather search non-invasive tools. Some tools are completely integrated into the kernel and thus directly accessible, so the focus is on these.
  • Rather than running script files, statements can directly be run from the command-line when provided correctly.

The last two can be explained rather shortly:

  • Syscalls are the C functions which make up the API by which applications can access the kernel's functions. These are documented in the type (2) man pages, if you did't know yet.

Here's a list, even though they may be called a little differently at times:

SYSCALL   WHAT IT DOES

read      read bytes from a file descriptor (file, socket)
write     write bytes from a file descriptor (file, socket)
open      open a file (returns a file descriptor)
close     close a file descriptor

fork      create a new process (current process is forked)
exec      execute a new program

connect   connect to a network host
accept    accept a network connection

stat      read file statistics
ioctl     set I/O properties, or other miscellaneous functions

mmap      map a file to the process memory address space

brk       extend the heap pointer

If a complete code audit is too heavy (All branches have to be checked after all. And later you find out you overlooked something.), dynamic tracing is for you. You can either find out how many syscalls were run, or what values variables were set to, you can collect data and create graphs from it, ... Actually you can do more than you need, so providing the most use cases should do well enough.

intro to dynamic tracing

What exactly is this dynamic tracing thing? Let's start with some terms which I shamelessly rephrase from a lesser-known but very able russian guy named Sergey Klyaus and his github stuff here:

  • Looking solely at code = static code analysis, sadly this is error-prone and a damn lot of work. There's a reason not many people do kernel development.
  • Watching a system's behaviour at runtime is dynamic analysis, but there are different types of introspection.

There are several methologies:

TODO

  • instrumentalizing
  • sampling
  • profiling
  • tracing

Sergey is truly awesome and knows his stuff. His ebook, though 'it may never be finished' as he said somewhere IIRC, is an outstanding piece of work and has already over 200 pages. The best part is that it is still freely available, and besides some little typos (English is not his mother tongue.) it is a damn good read.

So what technologies are available there will be provided in a short overview. The examples are purposefully short for copy-pasting, so starting with this stuff is easier.

DTrace

After I read a lot of stuff lately from the man, the myth, the legend, @brendangregg, it looks like DTrace is plain awesome. But since adoption on linux may take forever (if it will even happen at all since the open DTrace4Linux port by Paul Fox seems to be pretty much a one-man-show and Oracle's DTrace is just a wrapper around SystemTap, sadly I have no link where I read this), going with the alternatives seems the way to go on linux.

On FreeBSD it seems: 'Just use DTrace.'

On Linux the answer is not just as simple, thus this post might grow quite a bit over the following paragraphs.

usage

For the sake of completeness, here is a bunch of dtrace scripts:

# process plus its arguments
dtrace -n 'proc:::exec-success { trace(curpsinfo->pr_psargs); }'

# files opened by a process
dtrace -n 'syscall::open*:entry { printf("%s %s",execname,copyinstr(arg0)); }'

# syscall count of a program
dtrace -n 'syscall:::entry { @num[execname] = count(); }'

# syscall count by the system
dtrace -n 'syscall:::entry { @num[probefunc] = count(); }'

# syscall count of a process
dtrace -n 'syscall:::entry { @num[pid,execname] = count(); }'

# used memory of a progress
dtrace -n 'io:::start { printf("%d %s %d",pid,execname,args[0]->b_bcount); }'

# count of pages which were swapped by a process
dtrace -n 'vminfo:::pgpgin { @pg[execname] = sum(arg0); }'

eBPF

eBPF is under active development within the linux kernel, latest changes in version 4.4 you can read about here, but kernel developers call these things scary stuff.

Somewhere in a presentation Brendan compared DTrace to eBPF like a kitty hawk to a jet engine, which, besides it being 'in-kernel', should be the reason why it might be most likely be the most important tracer in linux some day.

A little presentation on BPF can be found here.

SystemTap

Until Linux' extended Berkeley Packet Filter (eBPF) is real prime time material, stap should do well, Brendan thought, as could be seen here.

SystemTap has got two modes:

  • Awk/C like language, gets the job done
  • Embedded C mode aka "guru mode" in case you need it

install

Most distributions have prepackaged what you want. Well, at least Debian did, and maybe CentOS, too, IIRC. Afterwards run stap-prep, which should tell you what else you have to install. (Usually you need the debug headers for your kernel, to make systemtap work.)

usage

TODO place some useful oneliners here

# show processes opening files in realtime
# Brendan wrote in his 'Systems Performance' book: "I've never actually seen this work."
# I feel proud, it did for me. ;)
stap -ve 'probe syscall.open { printf ("%30s %-100s\n", execname(), user_string($filename)); }'

explore

# NICE FULL OVERVIEWS

# PROBE TYPES
stap --dump-probe-types | awk -F. 'BEGIN {current=""; print "\n\033[31;1mstap -ve \"global s; probe ... {...}\"\033[0m\n"} {if (current != $1) { current=$1; printf "\n\033[33;1m%s\033[0m\n",current } else {print $0}}' | less -R
# PROBE ALIASES
stap --dump-probe-aliases | awk -F. 'BEGIN {current=""; print "\n\033[31;1mstap -ve \"global s; probe ... {...}\"\033[0m\n"} {if (current != $1) { current=$1; printf "\n\033[33;1m%s\033[0m\n",current } else {print $0}}' | less -R
# PROBE FUNCTIONS
echo $'\n\e[31;1mstap --dump-functions\e[0m\n'; stap --dump-functions

# some other examples, for the sake of completeness
stap -l 'kernel.function("acpi_*")' | sort
stap -l 'module("ohci1394").function("*")' | sort
stap -L 'module("thinkpad_acpi").function("brightness*")' | sort

further stuff

A pretty new example on Heatmaps using stap can be found here and here.

Further you also can export histograms directly to console, which is a damn awesome feature.

perf

According to Brendan, they quite heavily use perf over at netflix. Interestingly neflix runs no own infrastructure anymore, but completely relies on amazon's cloud services instead, I learned somewhere last week. You really got to know how to measure your available performance when doing such stunts, so perf sure sounds like a good idea.

Most stuff which helped me with perf here in a nutshell:

usage

What syscalls are run the most?

perf top

Let's do some profiling. In short, create a baseline data-set of your system first, then start your application and collect a second set of data from your 'system under test' (SUT). Afterwards just compare both collected sets:

perf record -p <PID> -o baseline.data sleep 30
perf record -p <PID> -o SUT.data sleep 30
perf diff baseline.data SUT.data

TODO
perf report -n --stdio

If regular strace is too heavy on your system, give perf trace a try.

This is all you need if you don't want to go down the rabbit hole. If sure, just do proceed:

explore

# check what probes exist at all
perf test

# helps with exploring what is actually possible
## alphabetically, from Brendan
perf list | awk -F':' '/Tracepoint event/ { lib[$1]++ } END { for (i in lib) { printf " %-16s %d\n",i,lib[i] } }' | sort | column
## by count
perf list | awk -F':' '/Tracepoint event/ { lib[$1]++ } END { for (i in lib) { printf " %-16s %d\n",i,lib[i] } }' | sort -nk2 | tac | column
## A PROPER COLORED LIST OF ALL TRACE GROUPS PLUS ITS TRACE POINTS (UGLY AS CAN BE)
perf list | awk -F'[: \t]+' 'BEGIN {current=""} /Tracepoint event/ {if (current != $2) { current=$2; print $2, "\n\t", $3 } else {print "\t", $3}}' | sed -r ''s/^[[:graph:]]+/$(printf "\033[33;1m&\033[0m")/'' | less -R

## SIMILAR, BUT ONLY FOR SYSTEM CALLS (UGLY? HELL, THIS IS EVEN WORSE)
perf list | awk -F'[: \t]+' 'BEGIN {current=""} /Tracepoint event/ {if (current != $2) { current=$2; print $2, "\n\t", $3 } else {print "\t", $3}}' | grep -e syscalls -e sys_enter -e sys_exit | sed -r -e 's/^syscalls/& ( with prefixes: sys_enter_ \/ sys_exit_ )/' -e ''s/^[[:graph:]]+/$(printf "\033[33;1m&\033[0m")/'' -e 's/sys_enter_([[:graph:]])/\1/' -e 's/sys_exit_([[:graph:]])/\1/' | uniq | awk 'BEGIN { flag = 1; id = 0 } /with prefixes:/ { print $0; flag = 0; next; print $0 } { if (flag) {print $0} else {array[id]=$0; id++}} END { for (i in array){print array[i] | "sort" }}' | less -R

Linux: install most recent kernel on CentOS 7

posted on 2015-06-15 21:25:04

Proceed at your own risk. You should have good reasons to use a server distribution with the most recent kernel in production.

To keep this sweet and short, do as root:

rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm
yum --enablerepo=elrepo-kernel install -y kernel-ml 

The downloading part might take a while.

Afterwards update grub:

grub2-mkconfig -i /boot/grub2/grub.cfg
grub2-install

Good Luck. Regression errors may lurk out there, waiting for you.

Linux performance observability tools

posted on 2015-01-17 18:50:42

This is an alphabetical list which will serve as a reminder, what programs are there to be looked up for me. :)

All this started when I stumbled across a picture on the web, which was from a presentation from Brendan Gregg at LinuxCon14 as I later found out. It was called Linux Performance Tools and it's worth its words in gold, platin and whatever material you see as highly valuable. The slides are here, get your copy and study them. If you want some serious linux sysadmin skills, there is no possible excuse for not doing it.

Seriously.

DO. IT. NOW.

Another two incentives can be found here and here. These may only use a small portion of the later mentioned programs, but either walk the extra miles, or raise your hands in defeat once things get tough, everybody gets to choose man's own path.

Alphetically sorted:

blktrace (8)         - generate traces of the i/o traffic on block devices
dstat (1)            - versatile tool for generating system resource statistics
dtrace (1)           - Dtrace compatibile user application static probe generation tool.
ebpf: nothing appropriate.
ethtool (8)          - query or control network driver and hardware settings
free (1)             - Display amount of free and used memory in the system
ftrace: nothing appropriate.
iostat (1)           - Report Central Processing Unit (CPU) statistics and input/output statistics for devices and partitions.
iotop (8)            - simple top-like I/O monitor
ip (8)               - show / manipulate routing, devices, policy routing and tunnels
iptraf (8)           - Interactive Colorful IP LAN Monitor
ktap: nothing appropriate.
lldptool (8)         - manage the LDP settings and status of lldpad
lsof (8)             - list open files
ltrace (1)           - A library call tracer
lttng: nothing appropriate.
mpstat (1)           - Report processors related statistics.
netstat (8)          - Print network connections, routing tables, interface statistics, masquerade connections, and multicast memberships.
nicstat: nothing appropriate.
pcstat: nothing appropriate.
perf (1)             - Performance analysis tools for Linux
pidstat (1)          - Report statistics for Linux tasks.
/proc: nothing appropriate.
ps (1)               - report a snapshot of the current processes.
rdmsr: nothing appropriate.
sar (1)              - Collect, report, or save system activity information.
slabtop (1)          - display kernel slab cache information in real time
snmpget (1)          - communicates with a network entity using SNMP GET requests
ss (8)               - another utility to investigate sockets
stap (1)             - systemtap script translator/driver
strace (1)           - trace system calls and signals
swapon (8)           - enable/disable devices and files for paging and swapping
sysdig ()            - the definitive system and process troubleshooting tool
tcpdump (8)          - dump traffic on a network
tiptop (1)           - display hardware performance counters for Linux tasks
top (1)              - display Linux processes
uptime (1)           - Tell how long the system has been running.
vmstat (8)           - Report virtual memory statistics

First some more explanations on the ones listed above with "nothing appropriate":

ebpf, ftrace, ktap, lttng, nicstat, pcstat, /proc, rdmsr are usually all too new. New like either in bleeding edge, or at least not available in CentOS 7 or Debian 7. If you grab the sources, you might get along. The manpage headlines are actually from a CentOS 7. (Only exception is sysdig, which I installed via the one-liner its github page provided.) /proc is of course not a command, but mentions the /proc folder linux uses where a lot of useful information can be found.

Here are some other sortings, by 'types' now. (Maybe this improves readability, or makes it easier to remember, who knows. It's worth trying, still.)

'stat', 'top', 'trace', 'tap':

dstat      iotop      blktrace     ktap
iostat     slabtop    dtrace       stap
mpstat     tiptop     ftrace
netstat    top        ltrace
nicstat               strace
pcstat
pidstat
vmstat

the rest:

ebpf
ethtool
free
ip
iptraf
lldptool
lsof
lttng
perf
/proc
ps
rdmsr
sar
snmpget
ss
swapon
sysdig
tcpdump
uptime

This were only the 'observability' tools from the presentation. There are also some more listed on 'benchmarking' and 'tuning', and maybe 'tracing'.

Just go an read up on them. NOW.

OS schema

posted on 2015-01-17 04:55:48

While toying around with the vim DrawIt plugin (which simplyfies the task of creating ascii art diagrams immensely), I needed something to test it with.

After having come across an operating system layout and which of linux tools' are to be used at which spot, I 'redrew' this:

      +---------------------------------------------+
      |              APPLICATIONS                   |
      |          db's, all server types, ...        |
      |                   +-------------------------+
      |                   |       system libs       |
      +=============================================+
      |              system call interface          |
      +=============================================+
 l    |  VFS           ||  sockets   ||             |     cpu     +-------+
 i    +----------------++------------++  scheduler  |-------------| CPU 1 |
 n  k |  FS            ||  TCP/UDP   ||             |  intercon.  +-------+
 u  e +----------------++------------++-------------+                 |
 x  r |  volume        ||  IP        ||             |                 |
    n |  managers      ||            ||  virtual    |                 |
    e +----------------++------------++  memory     |                 |
    l |  block dev if  ||  ethernet  ||             |             +------+
      +=============================================+             | DRAM |
      |              device drivers                 |             +------+
      +=============================================+
                             |
                             |  I/O bus
                             |
                       +------------+
                       | I/O bridge |
       expander        +------------+
       interconnect     /           \
                       /             \
       +----------------+          +--------------------+
       | I/O controller |          | network controller |
       +----------------+          +--------------------+
               /\                            /\
              /  \     if transports        /  \
             /    \                        /    \
       +------+    +------+         +------+   +------+
       | disk |    | swap |         | port |   | port |
       +------+    +------+         +------+   +------+

This blog covers .csv, .htaccess, .pfx, .vmx, /etc/crypttab, /etc/network/interfaces, /etc/sudoers, /proc, 10.04, 14.04, AS, ASA, ControlPanel, DS1054Z, GPT, HWR, Hyper-V, IPSEC, KVM, LSI, LVM, LXC, MBR, MTU, MegaCli, PHP, PKI, R, RAID, S.M.A.R.T., SNMP, SSD, SSL, TLS, TRIM, VEEAM, VMware, VServer, VirtualBox, Virtuozzo, XenServer, acpi, adaptec, algorithm, ansible, apache, apachebench, apple, applet, arcconf, arch, architecture, areca, arping, asa, asdm, autoconf, awk, backup, bandit, bar, bash, benchmarking, binding, bitrate, blackarmor, blockdev, blowfish, bochs, bond, bonding, booknotes, bootable, bsd, btrfs, buffer, c-states, cache, caching, ccl, centos, certificate, certtool, cgdisk, cheatsheet, chrome, chroot, cisco, clamav, cli, clp, clush, cluster, coleslaw, colorscheme, common lisp, configuration management, console, container, containers, controller, cron, cryptsetup, csync2, cu, cups, cygwin, d-states, database, date, db2, dcfldd, dcim, dd, debian, debug, debugger, debugging, decimal, desktop, df, dhclient, dhcp, diff, dig, display manager, dm-crypt, dmesg, dmidecode, dns, docker, dos, drivers, dtrace, dtrace4linux, du, dynamictracing, e2fsck, eBPF, ebook, efi, egrep, emacs, encoding, env, error, ess, esx, esxcli, esxi, ethtool, evil, expect, exportfs, factory reset, factory_reset, factoryreset, fail2ban, fbsd, fdisk, fedora, file, filesystem, find, fio, firewall, firmware, fish, flashrom, forensics, free, freebsd, freedos, fritzbox, fsck, fstrim, ftp, ftps, g-states, gentoo, ghostscript, git, git-filter-branch, github, gitolite, global, gnutls, gradle, grep, grml, grub, grub2, guacamole, hardware, haskell, hdd, hdparm, hellowor, hex, hexdump, history, howto, htop, htpasswd, http, httpd, https, i3, icmp, ifenslave, iftop, iis, imagemagick, imap, imaps, init, innoDB, innodb, inodes, intel, ioncube, ios, iostat, ip, iperf, iphone, ipmi, ipmitool, iproute2, ipsec, iptables, ipv6, irc, irssi, iw, iwconfig, iwlist, iwlwifi, jailbreak, jails, java, javascript, javaws, js, juniper, junit, kali, kde, kemp, kernel, keyremap, kill, kpartx, krypton, lacp, lamp, languages, ldap, ldapsearch, less, leviathan, liero, lightning, links, linux, linuxin3months, lisp, list, livedisk, lmctfy, loadbalancing, locale, log, logrotate, looback, loopback, losetup, lsblk, lsi, lsof, lsusb, lsyncd, luks, lvextend, lvm, lvm2, lvreduce, lxc, lxde, macbook, macro, magento, mailclient, mailing, mailq, manpages, markdown, mbr, mdadm, megacli, micro sd, microsoft, minicom, mkfs, mktemp, mod_pagespeed, mod_proxy, modbus, modprobe, mount, mouse, movement, mpstat, multitasking, myISAM, mysql, mysql 5.7, mysql workbench, mysqlcheck, mysqldump, nagios, nas, nat, nc, netfilter, networking, nfs, nginx, nmap, nocaps, nodejs, numberingsystem, numbers, od, onyx, opcode-cache, openVZ, openlierox, openssl, openvpn, openvswitch, openwrt, oracle linux, org-mode, os, oscilloscope, overview, parallel, parameter expansion, parted, partitioning, passwd, patch, pct, pdf, performance, pfsense, php, php7, phpmyadmin, pi, pidgin, pidstat, pins, pkill, plasma, plesk, plugin, posix, postfix, postfixadmin, postgres, postgresql, poudriere, powershell, preview, profiling, prompt, proxmox, ps, puppet, pv, pveam, pvecm, pvesm, pvresize, python, qemu, qemu-img, qm, qmrestore, quicklisp, quickshare, r, racktables, raid, raspberry pi, raspberrypi, raspbian, rbpi, rdp, redhat, redirect, registry, requirements, resize2fs, rewrite, rewrites, rhel, rigol, roccat, routing, rs0485, rs232, rsync, s-states, s_client, samba, sar, sata, sbcl, scite, scp, screen, scripting, seafile, seagate, security, sed, serial, serial port, setup, sftp, sg300, shell, shopware, shortcuts, showmount, signals, slattach, slip, slow-query-log, smbclient, snmpget, snmpwalk, software RAID, software raid, softwareraid, sophos, spacemacs, spam, specification, speedport, spi, sqlite, squid, ssd, ssh, ssh-add, sshd, ssl, stats, storage, strace, stronswan, su, submodules, subzone, sudo, sudoers, sup, swaks, swap, switch, switching, synaptics, synergy, sysfs, systemd, systemtap, tar, tcpdump, tcsh, tee, telnet, terminal, terminator, testdisk, testing, throughput, tmux, todo, tomcat, top, tput, trafficshaping, ttl, tuning, tunnel, tunneling, typo3, uboot, ubuntu, ubuntu 16.04, udev, uefi, ulimit, uname, unetbootin, unit testing, upstart, uptime, usb, usbstick, utf8, utm, utm 220, ux305, vcs, vgchange, vim, vimdiff, virtualbox, virtualization, visual studio code, vlan, vmstat, vmware, vnc, vncviewer, voltage, vpn, vsphere, vzdump, w, w701, wakeonlan, wargames, web, webdav, weechat, wget, whois, wicd, wifi, windowmanager, windows, wine, wireshark, wpa, wpa_passphrase, wpa_supplicant, x11vnc, x2x, xfce, xfreerdp, xmodem, xterm, xxd, yum, zones, zsh


Unless otherwise credited all material Creative Commons License by sjas