Posts from 2014-09

list of all shell shortcuts (bash / zsh)

posted on 2014-09-30 14:09:12

zsh:

bindkey -L

bash:

bind -P

## alternative: (improved readability!)
bind -P | grep -v "is not" | sed 's/can be found on/:/' | column -s: -t

write-back vs. write-through caching

posted on 2014-09-29 12:15:40

When putting hardware RAID controller to use, you usually have to choose between the option in the post title.

write-back

Once the hardware-RAID-controller (HWR) has the data cached, which the operating system sent, it tells the OS to go on. (The OS is told the disk write is finished.)

OS          --          HWR cache           --          HDD / SSD
1. send data to write
                        2. data is cached
                        3. OS is told to go on
4. OS goes on
                                                        5. data is written to disk,
                                                           when system load is low.

write-through

Only when the data has been written to disk, the OS is told so.

OS          --          HWR cache           --          HDD / SSD
1. send data to write
                        2. data is cached
                                                        3. data is written to disk
                        4. OS is told to go on
5. OS goes on

Test from the Proxmox wiki: (here)

Single SATA WD 400GB: 1360.17
3 x 15K rpm sas RAID5 with write-through: 159.03 (YES, only 159!)
same as above but with write-back enabled: 3133.45

TL;DR

Use write-back on controllers with a battery backup unit (BBU) or a flash memory, depending on which functionality is available.

Adaptec's lower series cannot be equipped with neither, the 5xxx series can have one of each (or even both?). The 6xxx series doesn't need a BBU.

Take note, that BBU's need attention, whereas flash-based solutions are maintenance-free.

With SSD's however, you can forget about all this and just use write-through, since it's fast enough.

CIDR by hand

posted on 2014-09-28 17:26:43

For calculating ipv4 subnetmasks for classless inter-domain routing, there exist calculators. I just grew tired of them.

the powers of 2

Since they are elemental:

2^0  = 1
2^1  = 2
2^2  = 4
2^3  = 8
2^4  = 16
2^5  = 32
2^6  = 64
2^7  = 128
2^8  = 256
2^9  = 512
2^10 = 1024
2^11 = 2048
2^12 = 4096
2^13 = 8192
2^14 = 16384
2^15 = 32768
2^16 = 65536

The tricky bit later is to remember that these are the possible values, but to not fall for the off-by-one errors later on.

ip adresses and subnet masks

Any IP address or subnet mask consists of 32 bits, or more to the point, four octetts. Lets look at some examples, for training reasons:

# in decimal
10.0.0.1
10.255.255.254
# in binary
00001010.00000000.00000000.00000000
00001010.11111111.11111111.11111110

# in decimal
192.168.0.1
192.168.255.254
# in binary
11000000.10101000.00000000.00000001
11000000.10101000.11111111.11111110

# in decimal
255.255.255.0
255.255.255.255
# in binary
11111111.11111111.11111111.00000000
11111111.11111111.11111111.11111111

So the subnets are numbered just by the count of 1's.

regular subnets

An overlook on the 'regualar' ones, the amount of hosts is explained later:

255.255.255.255 = 11111111.11111111.11111111.11111111
                = 32 x 1
                = /32 subnet (one single host)

255.255.255.0   = 11111111.11111111.11111111.00000000
                = 24 x 1
                = /24 subnet (254 hosts, class C net)

255.255.0.0     = 11111111.11111111.00000000.00000000
                = 16 x 1
                = /16 subnet (65.535 hosts, class B net)

255.0.0.0       = 11111111.00000000.00000000.00000000
                = 8 x 1
                = /8 subnet (16.777.216 hosts, class A net)

class c subnets in detail

And an in-depth look at the ones for splitting class C nets, and why hosts (and a whole subnet) seem to be missing:

11111111.11111111.11111111.11111111 = /32 = 2^0 IP's = 1 host

11111111.11111111.11111111.11111100 = /30 = 2^2-2 IP's = 2 hosts
11111111.11111111.11111111.11111000 = /29 = 2^3-2 IP's = 6 hosts
11111111.11111111.11111111.11110000 = /28 = 2^4-2 IP's = 14 hosts
11111111.11111111.11111111.11100000 = /27 = 2^5-2 IP's = 30 hosts
11111111.11111111.11111111.11000000 = /26 = 2^6-2 IP's = 63 hosts
11111111.11111111.11111111.10000000 = /25 = 2^7-2 IP's = 126 hosts
11111111.11111111.11111111.00000000 = /24 = 2^8-2 IP's = 254 hosts

and

11111111.11111111.11111111.11111111 = /32 = 255.255.255.255

11111111.11111111.11111111.11111100 = /30 = 255.255.255.252
11111111.11111111.11111111.11111000 = /29 = 255.255.255.248
11111111.11111111.11111111.11110000 = /28 = 255.255.255.240
11111111.11111111.11111111.11100000 = /27 = 255.255.255.224
11111111.11111111.11111111.11000000 = /26 = 255.255.255.192
11111111.11111111.11111111.10000000 = /25 = 255.255.255.128
11111111.11111111.11111111.00000000 = /24 = 255.255.255.0

no '/31' ?

Why is /31 'missing'? See here:

11111111.11111111.11111111.11111110 = /31 = 2^1-2 IP's = 0 hosts = NONSENSE
11111111.11111111.11111111.11111110 = /31 = NONSENSE

/32 is a network without a single host, and thus fine. But any network with more than one host needs a network base address and a broadcast adress. For the net the .0 is used, and .255 for broadcast, thus two IP's are already taken in every subnet.

TOIlet / FIGlet: fancy headlines

posted on 2014-09-16 11:57:12

Install the toilet package and you will be able to this:

[sjas@ctr-014 ~]% toilet asdf

                   #    m"" 
  mmm    mmm    mmm#  mm#mm 
 "   #  #   "  #" "#    #   
 m"""#   """m  #   #    #   
 "mm"#  "mmm"  "#m##    #   


[sjas@ctr-014 ~]% 

Install the figlet package and you will be able to do this:

[sjas@ctr-014 ~]% figlet qwer

  __ ___      _____ _ __ 
 / _` \ \ /\ / / _ \ '__|
| (_| |\ V  V /  __/ |   
 \__, | \_/\_/ \___|_|   
    |_|                  
[sjas@ctr-014 ~]%    

bash table output

posted on 2014-09-10 16:34:02

Pipe the output of a command to column, which 'columniates' lists.

Before:

[sjas@ctr-014 ~]% ip r
default via 10.20.1.1 dev eth1.9 
10.0.0.0/16 dev eth1.3  proto kernel  scope link  src 10.0.255.14 
10.20.1.0/24 dev eth1.9  proto kernel  scope link  src 10.20.1.14 
169.254.0.0/16 dev eth1.2  scope link  metric 1000 
192.168.184.0/24 dev eth1.2  proto kernel  scope link  src 192.168.184.42 
192.168.189.0/24 dev eth1.522  proto kernel  scope link  src 192.168.189.42 
[sjas@ctr-014 ~]%

After:

[sjas@ctr-014 ~]% ip r | column -t
default           via  10.20.1.1  dev    eth1.9
10.0.0.0/16       dev  eth1.3     proto  kernel  scope   link  src  10.0.255.14
10.20.1.0/24      dev  eth1.9     proto  kernel  scope   link  src  10.20.1.14
169.254.0.0/16    dev  eth1.2     scope  link    metric  1000
192.168.184.0/24  dev  eth1.2     proto  kernel  scope   link  src  192.168.184.42
192.168.189.0/24  dev  eth1.522   proto  kernel  scope   link  src  192.168.189.42
[sjas@ctr-014 ~]%

Choose which you like better.

linux: configure networking temporary from shell without ifup/ifdown

posted on 2014-09-09 13:00:21

In Debian-based distros, usually you change /etc/network/interfaces accordingly, then use ifdown and ifup to bring the changes into action. (Do not think of service networking restart or similar, these approaches will most likely NOT WORK PROPERLY!)

In general ip a is short for ip addr, ip l is short for ip link, ip r is short for ip route.

NOTE:
All which is done here, will only be temporary!
All settings will be gone after the next reboot.
These steps are presented here in case you have to debug a failing install process (You actually can get to a console during a Debian install!) or when troubleshooting network problems.

Only exception are the DNS settings, these should stick.

If you need the settings to stick, just edit the config files.

overview up front

  1. add IP to interface
  2. take interface down
  3. take interface up
  4. add gateway route
  5. fix dns if needed

setup

Use these settings, how you need them. Maybe you need to flush the interface first, maybe just remove a single ip, I do not know. Without further ado, these are what you most likely will need:

Check current settings:

ip a

To just test ip's on single NIC's, add them to the interface:

ip a a 10.0.0.2/24 dev eth0

To remove a single address:

ip a d 10.0.0.2/24 dev eth0

To remove all addresses:

ip a f dev eth0

deactivating / activating the interface

This means basically deactivating and reactivating the interface in question.

ip l s eth0 down
ip l s eth0 up

Check what happened:

ip a

# also maybe helpful: ('no carrier'!)
ip l

If there is written 'UP', the interface is in it's desired state. If however there is written 'NO CARRIER', there is no networking cable attached.

routing

Also a gateway is needed. Usually like this:

ip r add default via 10.0.0.1 dev eth0

dns

Now you should be set, maybe you are missing dns resolution, if no dns servers were set prior to this. Verify this: ping 8.8.8.8 will work in this case, whereas ping google.com will not.

echo 'nameserver 8.8.8.8' >> /etc/resolv.conf

lessons learned

Instead of using /etc/network/interfaces, where the configuration from above would look like this:

auto iface eth0
iface eth0 inet static
    address 10.0.0.2
    netmask 255.255.255.0
    network 10.0.0.0
    broadcast 10.0.0.255
    gateway 10.0.0.1

and where you'd have to follow up the editing with a ifdown eth0; wait; ifup eth0, or using the already-or-about-to-be-deprecated ifconfig, you do this, purely on the shell:

ip a f eth0
ip l s eth0 down
ip l s eth0 up
ip a a 10.0.0.2/24 dev eth0
ip r a default via 10.0.0.1 dev eth0
echo 'nameserver 8.8.8.8' >> /etc/resolv.conf

which is in long form: (if you prefer to type more and need detailed explanations)

# delete all addresses from interface eth0
ip addr flush eth0

# deactivate eth0
ip link set eth0 down

# activate eth0
ip link set eth0 up

# set ip address and netmask in interface eth0
# netmask is done by specifying the /24 subnet (...)
ip addr add 10.0.0.2/24 dev eth0

# add the gateway
ip route add default via 10.0.0.1 dev eth0

# fix dns, if needed, as root
echo 'nameserver 8.8.8.8' >> /etc/resolv.conf

A list for all this abbreviations is due, it seems. And thou shalt reread this here again and again. Seriously.

common lisp's 'format'

posted on 2014-09-08 21:50:28

Common Lisp swiss army knife of printing, format, shall be looked at in depth here.

usual layout

(format <destination_parameter> "<control_string>" <value_parameter>)

The format keyword starts the formatted string.

"control_string" is just a regular string in between double quotes, usually containing some control sequences.

Value parameters are the parts substituting the control sequences within the control string.

the destination parameter

Three possible values:

  • nil = just create a string, return the string as value, print nothing
  • t = print to console, function returns nil as value
  • stream = output to a stream

standard control sequences layouts

general layout:

~x,y,z,'p@a

where x, y, z are number-counts, 'p is a character, @ is optional, a could as well be an s or any other control sequence type.

x   : base count of characters
y   : if given, and base size is too small, increase in steps of y
z   : exact number of spaces added to string
'p  : padding character
@   : make the introduced value appear AFTER the padding



~a    absolute value, no delimiters
~s    string value, including delimiters
~5a   absolute value, size of 5, real string is leftbound
~5@a  absolute value, size of 5, real string is rightbound
~5,2a absolute value, size of 5, if value parameter is larger
      increase result in steps of 2 characters

float layout:

~a,b,cf

where a is the sum of integers, the comma and the amount of decimals possible. b is just the count of decimals to be shown. c scale number in factors of 10.

available number control sequences

~b  binary
~x  hexadecimal
~d  decimal
~:d decimal with 1000er dots in between
~f  floats
~$  currency control sequence

newlines

~%  terpri = hard newline, if number is given in front of it, it's the count of newlines
~&  fresh-line = soft newline, only newline if none has been inserted just before yet.

tables and justified output

The following part is not really easy at first, but this vanishes as soon as you grok the system. Have fun learning these...

~xt 

where x is an integer, at which position the current table column should start.

~< ... ~>

In between these the amount of spaces gets adjusted to be evenly distributed.

:@

Make the columns being adjusted right.

~x

x is an integer, and this describes the complete size of a column.

~{ ... ~} 

Use these to loop through a list.

~a,b:;

Break text into parts of equal length, print a only if there are more characters than a certain amount of integers b.

mysql reset root password

posted on 2014-09-08 11:27:38

To reset mysql passwords, connect from the commandline to the mysql daemon.

But like this:

  1. service mysql stop
  2. mysqld_safe --skip-grant-tables --skip-networking. That way no pw checks are made, and network connections are disabled, only localhost connections are enabled. (Security!)
  3. change password: mysql -Ne "update mysql.user set password=password('this_is_the_new_password') where user='root'; flush privileges;"
  4. stop mysqld with killall -9 mysqld_safe
  5. service mysql start
  6. try the account with the new password

If it is not working, service mysql stop again, start mysqld_safe --skip-grant-tables --skip-networking and fix the password, as you likely had a copy-paste or typing in there the last time.

bash kill signals

posted on 2014-09-07 14:42:58

preface

bash uses these kinds of kill signals, along with many others. In fact, signals are rather part of the OS and not the shell itself, and are present in all unices.

Examples:

SIGHUP hangup (nowadays original intentions not needed, often used for config reloads) SIGINT keyboard interrupt (Ctrl-c) SIGKILL kill signal SIGTERM termination signal SIGSTOP stop the process SIGTSTP suspend (Ctrl-z)

SIGKILL and SIGSTOP cannot be caught, blocked or ignored.

traps

Look up bash traps in the manual on how to caption signals, i.e. in scripts. Since trap is a bash-builtin, use help trap to do so. (Or search in man bash).

list

Besides these, there are A LOT of others, just in case you wondered what else is out there.

kill -l will show a list of all available signals, depending on your OS.

[sjas@nb ~]$ kill -l
 1) SIGHUP       2) SIGINT       3) SIGQUIT      4) SIGILL       5) SIGTRAP
 6) SIGABRT      7) SIGBUS       8) SIGFPE       9) SIGKILL     10) SIGUSR1
11) SIGSEGV     12) SIGUSR2     13) SIGPIPE     14) SIGALRM     15) SIGTERM
16) SIGSTKFLT   17) SIGCHLD     18) SIGCONT     19) SIGSTOP     20) SIGTSTP
21) SIGTTIN     22) SIGTTOU     23) SIGURG      24) SIGXCPU     25) SIGXFSZ
26) SIGVTALRM   27) SIGPROF     28) SIGWINCH    29) SIGIO       30) SIGPWR
31) SIGSYS      34) SIGRTMIN    35) SIGRTMIN+1  36) SIGRTMIN+2  37) SIGRTMIN+3
38) SIGRTMIN+4  39) SIGRTMIN+5  40) SIGRTMIN+6  41) SIGRTMIN+7  42) SIGRTMIN+8
43) SIGRTMIN+9  44) SIGRTMIN+10 45) SIGRTMIN+11 46) SIGRTMIN+12 47) SIGRTMIN+13
48) SIGRTMIN+14 49) SIGRTMIN+15 50) SIGRTMAX-14 51) SIGRTMAX-13 52) SIGRTMAX-12
53) SIGRTMAX-11 54) SIGRTMAX-10 55) SIGRTMAX-9  56) SIGRTMAX-8  57) SIGRTMAX-7
58) SIGRTMAX-6  59) SIGRTMAX-5  60) SIGRTMAX-4  61) SIGRTMAX-3  62) SIGRTMAX-2
63) SIGRTMAX-1  64) SIGRTMAX
[sjas@nb ~]$

All others can try man 7 signal.

usage

If you know the process by id (either from ps or top or the like):

kill -<signalname> <PID>

#i.e. these are the same
kill -kill 1234
kill -9 1234

If you know only the process name:

pkill -<signalname>

#i.e.
pkill -hup dnsmasq

vim fast execution bind

posted on 2014-09-07 12:32:37

To test quick throwaway scripts in vim try this: (linux-only, for windows this will work only via cygwin)

nnoremap <Leader>fr :w<CR>:!clear && ./%<CR>

Upon pressing the <leader>fr key combination (vim's leaderkey is \ by default), this happens:

  • the file you are currently editing will be saved
  • vim drops to a shell and clears the screen
  • the script gets executed

This works as long as the script has a shebang line and is executable (don't forget to chmod), since \% is the wildcard for the currently opened file

The best things in life are for free. :)

Fix git mergeconflicts (with vim)

posted on 2014-09-06 13:11:07

If, during a pull (or rebase), git fails to finish its currently run command, this one-liner might be helpful:

vim `git diff --diff-filter=U --name-only`

It will open all files, that git could not merge, in vim.

Fix the first one, save, :n (next window), fix the next one, and so forth.

Once you are done, git add . them all, git commit, and start again where git left off. (The rebase or push or whatever.)

proxmox partitioning

posted on 2014-09-05 07:12:04

When using the proxmox installer image, usually the partitioning settings are percentages of the disk size.

In cases where you want to have fixed sizes i.e. for swap and '/', choose custom boot options when proxmox prompts you with 'boot':

linux ext4 maxroot=10 swapsize=2

will give you:

  • ext4 as partitioning format, usually ext3 is used
  • 10 GB /
  • 2 GB swap

A list on the available options: (shamelessly stolen here)

At install time, at the boot prompt you can specify "linux" followed by one or more optional parameters, see below:

Default partitioning uses ext3, with only ext4 as option. If you want ext4 just type: ext4

To define the total size of hdd to be used (only Proxmox > 2.3): hdsize=X (where X is in GB, and affects /dev/sda2 size (/dev/sda1 is small and only for boot). This way you can save free space on the HDD for further partitioning (i.e. for an additional PV and VG on the same hard disk that can be used for LVM storage type).

To define the amount of root partition's disk space: maxroot=X (where X is in GB).

To define the amount of swap partition's disk space: swapsize=X (where X is in GB. Default is the same size as installed RAM, with 4GB minimum and HDSIZE/8 as maximum)

To define the amount of logical volume "data" space (/var/lib/vz): maxvz=X (where X is in GB).

To define the amount of free space left in LVM VG 'pve': minfree=X (where X is in GB, 16GB is the default if storage available > 128GB, 1/8 otherwise)

Example: linux ext4 maxroot=25 swapsize=8 maxvz=400 minfree=32

Last notes, also from the link:

Follow the instructions as always. If you screwed up grub and can't boot from hdd anymore, boot from the installer cdrom and at "boot:" prompt just type: pveboot (NOT preceded by "linux"). This will boot from cdrom and mount LVM root and data partitions.

This blog covers .csv, .htaccess, .pfx, .vmx, /etc/crypttab, /etc/network/interfaces, /etc/sudoers, /proc, 10.04, 14.04, AS, ASA, ControlPanel, DS1054Z, GPT, HWR, Hyper-V, IPSEC, KVM, LSI, LVM, LXC, MBR, MTU, MegaCli, PHP, PKI, R, RAID, S.M.A.R.T., SNMP, SSD, SSL, TLS, TRIM, VEEAM, VMware, VServer, VirtualBox, Virtuozzo, XenServer, acpi, adaptec, algorithm, ansible, apache, apachebench, apple, arcconf, arch, architecture, areca, arping, asa, asdm, awk, backup, bandit, bar, bash, benchmarking, binding, bitrate, blackarmor, blowfish, bochs, bond, bonding, booknotes, bootable, bsd, btrfs, buffer, c-states, cache, caching, ccl, centos, certificate, certtool, cgdisk, cheatsheet, chrome, chroot, cisco, clamav, cli, clp, clush, cluster, coleslaw, colorscheme, common lisp, console, container, containers, controller, cron, cryptsetup, csync2, cu, cups, cygwin, d-states, database, date, db2, dcfldd, dcim, dd, debian, debug, debugger, debugging, decimal, desktop, df, dhclient, dhcp, diff, dig, display manager, dm-crypt, dmesg, dmidecode, dns, docker, dos, drivers, dtrace, dtrace4linux, du, dynamictracing, e2fsck, eBPF, ebook, efi, egrep, emacs, encoding, env, error, ess, esx, esxcli, esxi, ethtool, evil, expect, exportfs, factory reset, factory_reset, factoryreset, fail2ban, fbsd, fedora, file, filesystem, find, fio, firewall, firmware, fish, flashrom, forensics, free, freebsd, freedos, fritzbox, fsck, fstrim, ftp, ftps, g-states, gentoo, ghostscript, git, git-filter-branch, github, gitolite, gnutls, gradle, grep, grml, grub, grub2, guacamole, hardware, haskell, hdd, hdparm, hellowor, hex, hexdump, history, howto, htop, htpasswd, http, httpd, https, i3, icmp, ifenslave, iftop, iis, imagemagick, imap, imaps, init, innoDB, innodb, inodes, intel, ioncube, ios, iostat, ip, iperf, iphone, ipmi, ipmitool, iproute2, ipsec, iptables, ipv6, irc, irssi, iw, iwconfig, iwlist, iwlwifi, jailbreak, jails, java, javascript, javaws, js, juniper, junit, kali, kde, kemp, kernel, keyremap, kill, kpartx, krypton, lacp, lamp, languages, ldap, ldapsearch, less, leviathan, liero, lightning, links, linux, linuxin3months, lisp, list, livedisk, lmctfy, loadbalancing, locale, log, logrotate, looback, loopback, losetup, lsblk, lsi, lsof, lsusb, lsyncd, luks, lvextend, lvm, lvm2, lvreduce, lxc, lxde, macbook, macro, magento, mailclient, mailing, mailq, manpages, markdown, mbr, mdadm, megacli, micro sd, microsoft, minicom, mkfs, mktemp, mod_pagespeed, mod_proxy, modbus, modprobe, mount, mouse, movement, mpstat, multitasking, myISAM, mysql, mysql 5.7, mysql workbench, mysqlcheck, mysqldump, nagios, nas, nat, nc, netfilter, networking, nfs, nginx, nmap, nocaps, nodejs, numberingsystem, numbers, od, onyx, opcode-cache, openVZ, openlierox, openssl, openvpn, openvswitch, openwrt, oracle linux, org-mode, os, oscilloscope, overview, parallel, parameter expansion, parted, partitioning, passwd, patch, pdf, performance, pfsense, php, php7, phpmyadmin, pi, pidgin, pidstat, pins, pkill, plesk, plugin, posix, postfix, postfixadmin, postgres, postgresql, poudriere, powershell, preview, profiling, prompt, proxmox, ps, puppet, pv, pvecm, pvresize, python, qemu, qemu-img, qm, qmrestore, quicklisp, r, racktables, raid, raspberry pi, raspberrypi, raspbian, rbpi, rdp, redhat, redirect, registry, requirements, resize2fs, rewrite, rewrites, rhel, rigol, roccat, routing, rs0485, rs232, rsync, s-states, s_client, samba, sar, sata, sbcl, scite, scp, screen, scripting, seafile, seagate, security, sed, serial, serial port, setup, sftp, sg300, shell, shopware, shortcuts, showmount, signals, slattach, slip, slow-query-log, smbclient, snmpget, snmpwalk, software RAID, software raid, softwareraid, sophos, spacemacs, spam, specification, speedport, spi, sqlite, squid, ssd, ssh, ssh-add, sshd, ssl, stats, storage, strace, stronswan, su, submodules, subzone, sudo, sudoers, sup, swaks, swap, switch, switching, synaptics, synergy, sysfs, systemd, systemtap, tar, tcpdump, tcsh, tee, telnet, terminal, terminator, testdisk, testing, throughput, tmux, todo, tomcat, top, tput, trafficshaping, ttl, tuning, tunnel, tunneling, typo3, uboot, ubuntu, ubuntu 16.04, udev, uefi, ulimit, uname, unetbootin, unit testing, upstart, uptime, usb, usbstick, utf8, utm, utm 220, ux305, vcs, vgchange, vim, vimdiff, virtualbox, virtualization, visual studio code, vlan, vmstat, vmware, vnc, vncviewer, voltage, vpn, vsphere, vzdump, w, w701, wakeonlan, wargames, web, webdav, weechat, wget, whois, wicd, wifi, windowmanager, windows, wine, wireshark, wpa, wpa_passphrase, wpa_supplicant, x11vnc, x2x, xfce, xfreerdp, xmodem, xterm, xxd, yum, zones, zsh


Unless otherwise credited all material Creative Commons License by sjas