Posts from 2017-04

ansible introduction

posted on 2017-04-29 10:20

prerequisites for the next section

You know how to use ssh,ssh-keygen, ssh-copy-id so you can achieve your desired destination server of choice with passwordless keyauth for user root.

toying with ansible commandline, first needed notations

This is just for getting singe action run on single host or a group of hosts.

  • without using a playbook (= ansible's name for a detailed blueprint for what you want to install)
  • showing modules (module = ansible part implementing some function)
  • showing how to group contents with a hosts inventory file
  • why idempotence really makes this better than shellscripts

Some available modules:

  • command: run a specified shell command, default if nothing specified with -m
  • ping: does what it says and returns result
  • apt: debian/ubuntu package manager frontend
  • copy: copy local files -> remote hosts
  • file: change attributes
  • service: server daemon handling
  • template: create file from template and copy to remote host
  • lineinfile: make sure a certain line is found i certain config file

Run a shell command remotely:

## run uptime on host(s)
# -i: 'inventory', hostname or location of hosts file or a folder containing inventory files
# -m: 'module', ansible part that does something, defa
ansible -i HOSTNAME -m command -a uptime
ansible -i HOSTNAME_1,HOSTNAME_2 -a uptime

Group hosts:

## lets create a hosts file for less typing
cd ~
mkdir ansible
cd !$
echo "
" > hosts
# update hosts packages
ansible -i hosts -m date

Let's read the hosts file automatically be read and set explicitly some defaults. There's more to the hosts file like setting host/group-specific vars, groups-of-groups, but for now this will do.

echo "
host_key_checking = False
ansible_ssh_user = root
ansible_ssh_port = 22
ansible_connection = smart
ansible_shell_type = sh
hostfile = hosts
" > ansible.cfg
ansible hosts-group -a 'ip a'
ansible all -m ping  ## run on all hosts in the inventory file

-s does sudo, in case you'd want to work with a nonroot user.

more notation and some reasoning behind (proper) configuration management

All this can be done with bash, too, of course, except that you get some fancy output with colors and host grouping for free. But until now nothing really has changed. So why the fuss?

To better explain that, some more notation stuff:

  • A playbook consists of plays.
  • A play consists of tasks to run on (a) specific host(s).

And: Idempotence? The official definition from the manual's glossary:
An operation is idempotent if the result of performing it once is exactly the same as the result of performing it repeatedly without any intervening actions.


  • The same results even if rerunning as often as you wish! (Want to implement errorhandling yourself in bash yourself for making sure you have a line in a file? Ansible may not do everything for you, but it helps rather nicely.)

But what is not expressed through this term, but what you get along the way compared to shell scripting?

  • host grouping (sure, go back to while read i'ing that files consisting lists of hosts.)
  • batteries included (no more thinking about handling connections or timeouts usually, modules for everything)
  • pretty fast time-to-production
  • error handling usually included, or providing you with enough readable info to work it out
  • automatically colored output while running
  • quite readable overview of what worked or didn't after running through
  • automatically documented systems if done properly

Especially the last two points make any maintenance work a breeze. Either recreate a configuration on a new host through provisioning. Or just quickly read the task lists (grouping them properly helps for sure here), to get the idea what the system's purpose was/is. Keep the last bullet point in mind when creating and structuring playbooks, it really pays off in the end. If I don't forget, I will include a virtual banana for scale or something, this post series is going to end up a long one anyway.

hating puppet

posted on 2017-04-29 10:20

a long rant on configuration management and a healthy dose of puppet hate

In the beginning there were shellscripts for provisioning. ('Go away or I will replace you with...')

Eventually highlevel language code appeared for this purpose. I did that myself with python, too reinventing the wheel, wrote some rather sophisticated wrappers in bash (shudder) for another project and have seen a nontrivial configuration management codebase in... PHP. I'm not kidding, that codebase still gives me the chills.

Due to the more or less obvious shortcomings with these methods, theoretically some proper configuration management tool, relieving one of the duty to really implement everything by hand while being readable, just had to arrive, especially with all these computers being everywhere nowadays.

From all the news and testimonals, puppet seemed to be the all-around best contender and thus lesser of all evils while being enterprisey enough.

Well puppet...

  • is just overhyped
  • has an awful documentation which looks much much better than it really is
  • which is making you end up regularily on github in the sources to find out what the heck its doing this time
  • because its internals do seem to change from time to time
  • along with questionable backwards compatibility (was also being told this by people working with it everyday with different versions in rather big different setups running in production.)
  • and it getting really ugly as soon as you try to do non-trivial parts (never try updating data structures with a loop),
  • but the other competition (chef, cfengine) is also just as bloated / crappy / ruby it seems / looks / people's testimonals tell,

... so you likely may up with ansible at last.

The fact that friends of mine jump through hoops with it at work in comparably big setups (and doing things with it in the process which the external specialized puppet consultants could not make do), also does not really inspire confidence. Mind you, they take the configuration management theme far, far more serious than I or my colleagues do. (The goal is single button presses for everything, from single hosts to complete clusters. The Litmus test wether the goal is achieved: You don't ssh onto the servers to do anything, but fix it in the configuration management's database. No more clustershell or for-looping across the fleets, no more manual work in the field. Same results, always, bla bla bla.)

And on the tool front, finally there's salt which a lot of people seem to dig, but ansible wins with simplicity, albeit being slower due to using ssh (salt has an encrypted rabbitmq as communication engine). But that hasn't been an issue with some several hundred servers so far, thanks to the ssh multiplexing.

But Salt basically needs a master due to being pull-based, but the push-based approach of ansible is good enough by far for provisioning with our workflow. Not every change has to go through ansible at our place (yet), and we just like having fewer moving parts.

On a last note, I really really really tried to dig (hoping even love) puppet, having had my doubts (scratch that, read: 'I still have') about ansible being the best tool available for our outlet, but I have looked too often at its documentation, just to end up finally in the sources to get whats happening. It doesn't matter that it is ruby instead of python (I am not one of THESE people), but It simply does not deliver, even after spending some considerable time with it. Sure, after a while you can even read all of its output pretty easily, but ansible's is just easier to read. You get up to speed rather easily, and until the ssh speed becomes the bottleneck, it will just do for now. May be that with 10.000s of servers this will change, lets see wether we will find out.

We've had our share with different ansible problems, too, of course (it's all software, after all). But all an all that went rather smoothly comparing the amounts of time we spent with fixing these. This included i.e. completely random bugs with extracting zipfiles and version incompatibilies, too, but all of them were resolved in an order of magnitude less pain and head-desk-banging happenings with puppet.

Dear future self, if considering puppet again, read this three times, thank you - and quit the stockholm syndrome bullshit.

proxmox and lxc containers

posted on 2017-04-28 20:48

Here is the bare minimum when working with the shell to create containers, at least thats the intention of this post.

check storage, where to save it

pvesm help  ## show help
pvesm help <command>  ## show help for specific <command>
pvesm status  ## show storages and free space
pvesm list <storage>  ## show current contents

handle image(s) from which to create containers

pveam help  ## show help
pveam help <command>  ## show help for specific <command>
pveam update  ## update template list
pveam available  ## show downloadable images
pveam available -section system  ## you will want this for blank os containers without applications
pveam download <image>  ## get <image>
pveam list <storage>  ## show templates on <storage>
pveam remove <image>  ## delete

provisioning containers

pct help  ## show help
pct help <command>  ## show help for specific <command>
pct list  ## show existing containers on current HV, other HVs' ones will not be shown
pveam list <storage> ## show available image templates

pct create ... ## don't do this, use the gui. seriously.
pct list  ## get ids
pct destroy <cid>  ## delete container by <id>

pct config <cid>  ## get current config for <containerid>, helps if you really want to shell-script containers

handling containers

At first you will want the gui to learn your way around. Later on you will read pct help and pct help <command> quite often, the cli api is pretty, pretty decent.

Also you might like using pct help set and using pct set ... for changing container settings.

For now this is sufficient to get up and running.

On blockdevices and their blocksizes

posted on 2017-04-27 06:00

Disclaimer: This is a draft and waiting to be verified once I get around plugging all old disks in. As of now, it mainly tries to specify the problem of disk sector size being report differently by linux on OS level and from hdparm via the firmware, and wether dd can really specify arbitrary blocksizes.

Getting it from the kernel via /sys/dev/, lsblk, or fdisk did show different sizes from hdparm when tested.

It's the only seemingly correct way to determine the sector sizes, since it queries the firmware on disk directly. Getting it from the kernel via /sys/dev/ (lsblk or fdisk should use that same info) did show different sizes from hdparm -I when tested.

hdparm sports -i and -I:

-i     Display the identification info which the kernel drivers (IDE, libata) have stored from boot/config
       uration time.  This may differ from the current  information  obtainable  directly  from  the  drive
       itself with the -I option.  The data returned may or may not be current, depending on activity since
       booting the system.  For a more detailed interpretation of the  identification  info,  refer  to  AT
       Attachment  Interface  for Disk Drives, ANSI ASC X3T9.2 working draft, revision 4a, April 19/93, and
       later editions.

- I     Request identification info directly from the drive, which is displayed in  a  new  expanded  format
        with considerably more detail than with the older -i option.

An overview on the information hdparm gets, can be found here. I did not find the actual block size in there, so my guess is the sizes are calculated from the information found in the drives registers.

-i shows pretty much a dump of the firware registers it seems, while -I seems to do additional calculations to present more information in human-friendly form.

To overwrite the last 1mb of a disk via dd, on a disk with 512b sectors this will do:

dd bs=512 if=/dev/zero of=/dev/sda count=2048 seek=$((`blockdev --getsz /dev/sda` - 2048))

bs is in bytes, count and seek are blockcounts.

Questions for the future are:

  • does this work reliably with devices having 4k sectors?
  • what does blockdev --getbsz print in these cases?
  • does it use the size reported by the kernel?
  • do mismatches come from different physical/logical blocksizes reported by the disk?
  • were the mismatches I witnessed really in between disk and OS, or just within the firmware?

Mind you, the numbers I saw earlier (2 years?) were with a dozen directly connected IDE and SATA drives, not with a raid controller in between, on disks of differenent manufacturers and sizes I had left over, and some showed in-kernel 512b even though they were 4k ones.

This really intrigues me, since the linux tools all seem to 'just work'. But using dd in the past I ran into problems were I simply could not manually find data during forensic work on disks where its offset was calculated to be.

gnu global

posted on 2017-04-26 20:56

Get the current version from here. Alternatively, use the use the anonymous access to CVS, but no Idea wether the patches would work there.

The steps to version 6.5.6 running on a current gentoo install were, including the patches follow. I hope I don't forget something, this write was done a week later.

needed packages

emerge -atv these:

  • dev-python/pygments
  • dev-util/ctags
  • sys-devel/automake
  • sys-devel/m4

getting and compiling

mkdir asdf
cd asdf
curl -O
tar xzvf global-6.5.6.tar.gz
cd global-6.5.6

applying these two patches:

diff --git a/ b/
index cda52e4..d7d1a73 100644
--- a/
+++ b/
@@ -24,6 +24,7 @@ dnl
 AC_INIT([GNU GLOBAL],[6.5.6])
 AC_DEFINE(COPYRIGHT_YEAR,["1996-2016"],[Copyright Year])
 AM_INIT_AUTOMAKE([1.9.3 gnu subdir-objects])

diff --git a/ b/
index 43585a6..f6ab5c5 100644
--- a/
+++ b/

 EXTRA_DIST = ${gtags_DATA} mainpage.dox

    GTAGSCONF=' --suggest:' doxygen

Then back to the shell:

autoreconf -fvi
./configure --with-universal-ctags=`which ctags`
make check
sudo make install

setup and test

sudo updatedb
sudo locate gtags.conf
cp -va /usr/local/share/gtags/gtags.conf ~/.globalrc
gtags --version
global --version
cd ../..
rm -rf asdf


Now onto actual usage:


gtags -v        # generate tag database (initially, rerun with -i everytime code is changed!)

global FUNC     # finds file where FUNC is defined, regexp's work, too
global -r FUNC      # find files where FUNC is used in
global -x FUNC      # == ctags -x, name+line+path+line_content

Of course you only want to look at source, not generated code. If git is used, doing this from the project root will do the trick:

git ls-files > gtags.files

There's more to it, but it should be sufficiant for now. Usually global should be integrated into your dev environment of choice.

Out of the box, less should work. vim and emacs need plugins to work, they won't be covered at this time here.


export LESSGLOBALTAGS=global  ## this is needed so less knows it can work with global
less -t FUNC        # opens file where FUNC is defined

installing quicklisp

posted on 2017-04-22 20:48

Each lisp install can be considered as a variable image. To add quicklisp for proper library management, there are some simple steps needed.

This is done on a steel bank common lisp install.

curl -O
sbcl --load quicklisp.lisp

* (quicklisp-quickstart:install)
* (ql:add-to-init-file)
* (sb-ext:quit)

rm quicklisp.lisp

openvpn dns pushed on linux 2

posted on 2017-04-15 15:08

Last time I got this to work on debian (see here), now it was time to port this stuff to gentoo (there was more to it, this just to document that openvpn part).

needed packages

emerge -atv net-vpn/openvpn
emerge -atv net-dns/openresolv

resolvconf packages are deprecated it seems, openresolv is the new hotness.

openvpn config

These to lines need to be present in your .ovpn file at the end:

up  /etc/openvpn/
down    /etc/openvpn/

This differs from /etc/openvpn/update-resolv-conf in debian.

openvpn cannot find /dev/net/tun

Simple solution, just create the device if its not already present:

mkdir /dev/net
mknod /dev/net/tun c 10 200

Afterwards everything should work as expected, including the updated dns settings. Use watch -n1 -d cat /etc/resolv.conf to see things changing upon establishing and terminating the tunnel.

make it stick

After every reboot you'd have to recreate the /dev/net folder and device node. To let this happen automatically, create a file called /etc/local.d/create_tun_device.start containing:

[ ! -d /dev/net ] && mkdir /dev/net
[ ! -e /dev/net/tun ] && mknod /dev/net/tun c 10 200

This blog covers .csv, .htaccess, .pfx, .vmx, /etc/crypttab, /etc/network/interfaces, /etc/sudoers, /proc, 10.04, 14.04, 16.04, AS, ASA, ControlPanel, DS1054Z, GPT, HWR, Hyper-V, IPSEC, KVM, LSI, LVM, LXC, MBR, MTU, MegaCli, PHP, PKI, PS1, R, RAID, S.M.A.R.T., SNMP, SSD, SSL, TLS, TRIM, VEEAM, VMware, VServer, VirtualBox, Virtuozzo, XenServer, acpi, adaptec, algorithm, ansible, apache, apache2.4, apachebench, apple, applet, arcconf, arch, architecture, areca, arping, asa, asdm, autoconf, awk, backup, bandit, bar, bash, benchmarking, binding, bitrate, blackarmor, blockdev, blowfish, bochs, bond, bonding, booknotes, bootable, bsd, btrfs, buffer, c-states, cache, caching, ccl, centos, certificate, certtool, cgdisk, cheatsheet, chrome, chroot, cisco, clamav, cli, clp, clush, cluster, cmd, coleslaw, colorscheme, common lisp, configuration management, console, container, containers, controller, cron, cryptsetup, csync2, cu, cups, cygwin, d-states, database, date, db2, dcfldd, dcim, dd, debian, debug, debugger, debugging, decimal, desktop, df, dhclient, dhcp, diff, dig, display manager, dm-crypt, dmesg, dmidecode, dns, docker, dos, drivers, dtrace, dtrace4linux, du, dynamictracing, e2fsck, eBPF, ebook, efi, egrep, emacs, encoding, env, error, ess, esx, esxcli, esxi, ethtool, evil, expect, exportfs, factory reset, factory_reset, factoryreset, fail2ban, fakeroot, fbsd, fdisk, fedora, file, files, filesystem, find, fio, firewall, firmware, fish, flashrom, forensics, free, freebsd, freedos, fritzbox, fsck, fstrim, ftp, ftps, g-states, gentoo, ghostscript, git, git-filter-branch, gitbucket, github, gitolite, global, gnutls, gradle, grep, grml, grub, grub2, guacamole, hardware, haskell, hdd, hdparm, hellowor, hex, hexdump, history, howto, htop, htpasswd, http, httpd, https, i3, icmp, ifenslave, iftop, iis, imagemagick, imap, imaps, init, innoDB, innodb, inodes, intel, ioncube, ios, iostat, ip, iperf, iphone, ipmi, ipmitool, iproute2, ipsec, iptables, ipv6, irc, irssi, iw, iwconfig, iwlist, iwlwifi, jailbreak, jails, java, javascript, javaws, js, juniper, junit, kali, kde, kemp, kernel, keyremap, kill, kpartx, krypton, lacp, lamp, languages, ldap, ldapsearch, less, leviathan, liero, lightning, links, linux, linuxin3months, lisp, list, livedisk, lmctfy, loadbalancing, locale, log, logrotate, looback, loopback, losetup, lsblk, lsi, lsof, lsusb, lsyncd, luks, lvextend, lvm, lvm2, lvreduce, lxc, lxde, macbook, macro, magento, mailclient, mailing, mailq, make-jpkg, manpages, markdown, mbr, mdadm, megacli, micro sd, microsoft, minicom, mkfs, mktemp, mod_pagespeed, mod_proxy, modbus, modprobe, mount, mouse, movement, mpstat, multitasking, myISAM, mysql, mysql 5.7, mysql workbench, mysqlcheck, mysqldump, nagios, nas, nat, nc, netfilter, networking, nfs, nginx, nmap, nocaps, nodejs, numberingsystem, numbers, od, onyx, opcode-cache, openVZ, openlierox, openssl, openvpn, openvswitch, openwrt, oracle linux, org-mode, os, oscilloscope, overview, parallel, parameter expansion, parted, partitioning, passwd, patch, pct, pdf, performance, pfsense, php, php7, phpmyadmin, pi, pidgin, pidstat, pins, pkill, plasma, plesk, plugin, posix, postfix, postfixadmin, postgres, postgresql, poudriere, powershell, preview, profiling, prompt, proxmox, ps, puppet, pv, pveam, pvecm, pvesm, pvresize, python, python3, qemu, qemu-img, qm, qmrestore, quicklisp, quickshare, r, racktables, raid, raspberry pi, raspberrypi, raspbian, rbpi, rdp, redhat, redirect, registry, requirements, resize2fs, rewrite, rewrites, rhel, rigol, roccat, routing, rs0485, rs232, rsync, s-states, s_client, samba, sar, sata, sbcl, scite, scp, screen, scripting, seafile, seagate, security, sed, serial, serial port, setup, sftp, sg300, shell, shopware, shortcuts, showmount, signals, slattach, slip, slow-query-log, smbclient, snmpget, snmpwalk, software RAID, software raid, softwareraid, sophos, spacemacs, spam, specification, speedport, spi, sqlite, squid, ssd, ssh, ssh-add, sshd, ssl, stats, storage, strace, stronswan, su, submodules, subzone, sudo, sudoers, sup, swaks, swap, switch, switching, synaptics, synergy, sysfs, systemd, systemtap, tar, tcpdump, tcsh, tee, telnet, terminal, terminator, testdisk, testing, throughput, tmux, todo, tomcat, top, tput, trafficshaping, ttl, tuning, tunnel, tunneling, typo3, uboot, ubuntu, ubuntu 16.04, ubuntu16.04, udev, uefi, ulimit, uname, unetbootin, unit testing, upstart, uptime, usb, usbstick, utf8, utm, utm 220, ux305, vcs, vgchange, vim, vimdiff, virtualbox, virtualization, visual studio code, vlan, vmstat, vmware, vnc, vncviewer, voltage, vpn, vsphere, vzdump, w, w701, wakeonlan, wargames, web, webdav, weechat, wget, whois, wicd, wifi, windowmanager, windows, wine, wireshark, wpa, wpa_passphrase, wpa_supplicant, x11vnc, x2x, xfce, xfreerdp, xmodem, xterm, xxd, yum, zones, zsh

Unless otherwise credited all material Creative Commons License by sjas