Posts from 2016-03

arping: duplicate ip address detection
posted on 2016-03-31 22:50

Duplicate IP's within your subnet are a problem that you can detect via arping. It sends a layer2 ARP REQUEST to detect if an IP is already known within the network.

Usually only this is sufficient for usage from the shell:

arping -D <IP>`

When you simply receive a response on the commandline, the IP is in use already. If you use vlans, you have to specify your interface with -I, too.

If you want to use this from within scripts, you might want this:

arping -D -w2 -c2 -I <INTERFACE> <IP>
echo $?

arping returns zero if there's exists a duplicate IP.

One thing to keep in mind is that some linux distributions have several packages available, but only one it the arping. See on debian, for example, you got these two on jessie:

arping/stable 2.14-1 amd64
  sends IP and/or ARP pings (to the MAC address)

iputils-arping/stable,now 3:20121221-5+b2 amd64 [installed]
  Tool to send ICMP echo requests to an ARP address

You need the iputils-arping one, if you happen to use debian.

mdadm cheatsheet
posted on 2016-03-29 07:44

Since I have had too like this crap up one time too often...

# create new multiple device disk
mdadm --create MD_DEV options...
    -l1 -n2 --metadata=0.90 DEV1 DEV2

# assemble previously created multiple device disk
mdadm --assemble MD_DEV options...
    --scan / -s
    --run / -R
    --force / -f
    --update=? / -U
    --readonly / -o

# similar to --create, but...
mdadm --build MD_DEV options...
    DO NOT USE ANYMORE

# bread and butter command
mdadm --manage MD_DEV options...
    --add / -a
    --re-add
    --remove / -r
    --fail / -f
    --replace
    --run / -R
    --stop / -S
    --readonly / -o
    --readwrite / -w

# also bread and butter command
mdadm --misc options... DEVICES
    --query / -Q    (MD_DEV)
    --detail / -D   (MD_DEV)
    --examine / -E  (DEV)
    --examine-bitmap / -X (DEV)
    --run / -R 
    --stop / -S
    --readonly / -o
    --readwrite / -w
    --test / -t
    --wait / -w
    --zero-superblock (DEV)

# havent used these yet
mdadm --grow options device
mdadm --incremental device
mdadm --monitor options...
btrfs subvolume folder list
posted on 2016-03-27 12:16

A list of good folder candidates for being placed within subvolumes being separate from the root filesystem is here:

  • /boot/grub2/*
  • /opt
  • /srv
  • /tmp
  • /usr/local
  • /var/crash
  • /var/lib/{mailman,named,pgsql,mysql}
  • /var/log
  • /var/opt
  • /var/spool
  • /var/tmp
informatics: dual vs. decimal numbering system
posted on 2016-03-26 16:04:41

Saturday afternoon fun... after having watched Kristian Koehntopp's extremely awesome set of mysql presentations here (beware, in german, but he does the math on a flipchart), I finally set out to calculate all the binary numbers with the goal of memorizing them for once and all times to come.

Here we go.

0 [1001] 3 [ jl@jl.dyn.sjas.de ~ ] 15:40:36 $ bc
bc 1.06.95
Copyright 1991-1994, 1997, 1998, 2000, 2004, 2006 Free Software Foundation, Inc.
This is free software with ABSOLUTELY NO WARRANTY.
For details type `warranty'.

2^0
1

2^1
2

2^2
4

2^3
8

2^4
16

2^5
32

2^6
64

2^7
128

2^8
256

2^9
512

2^10
1024

2^11
2048

2^12
4096

2^13
8192

2^14
16384

2^15
32768

2^16
65536

So far so good. Now comes the handy stuff:

KILO    thousand                   1000^1     2^10                          10^3
                                              1024                          1000

MEGA    million                    1000^2     2^20                          10^6
                                              1048576                       1000000

GIGA    billion     / milliard     1000^3     2^30                          10^9
                                              1073741824                    1000000000

TERA    trillion    / billion      1000^4     2^40                          10^12
                                              1099511627776                 1000000000000

PETA    quadrillion / billiard     1000^5     2^50                          10^15
                                              1125899906842624              1000000000000000

EXA     quintillion / trillion     1000^6     2^60                          10^18
                                              1152921504606846976           1000000000000000000

ZETTA   sextillion  / trilliard    1000^7     2^70                          10^21
                                              1180591620717411303424        1000000000000000000000

YOTTA   septillion  / quadrillion  1000^8     2^80                          10^24
                                              1208925819614629174706176     1000000000000000000000000

And for some furter mental classification of the sizes:

2^8
256

2^16                 = KILO = thousands
65536

2^32                 = GIGA = millions
4294967296

2^64                 = EXA  = quintillion / trillion
18446744073709551616

This should do for now.

csync2 setup
posted on 2016-03-21 17:19:01

This is done without SSL, since all servers are within their intranet anyway.

install

apt install csync2 -y

generate key

csync2 -k /etc/csync2.key

/etc/csync2.cfg

nossl * *;

group MYGROUP
{
        host NODE1;
        host NODE2;

        key /etc/csync2.key;

        include /www/htdocs;
        exclude *~ .*;
}

/etc/xinetd.d/csync2

  service csync2
  {
      flags = IPv4
      socket_type         = stream
      protocol            = tcp
      wait                = no
      user                = root
      server              = /usr/sbin/csync2
      server_args         = -i
      disable             = no
  }

copy all files to all nodes

scp /etc/csync2* node2:/etc/

restart daemon

service xinetd restart

usage

# sync stuff
csync2 -xv

# show differences
csync2 -T
csync2 -TT

# dry-run
csync2 -xvd

# force sync everything
csync2 -rf /
pfsense: traffic shaping
posted on 2016-03-20 18:12:20

why?

Over a small broadband connection, to a backend which is split physically into two networks, guarantee that one network can not eat up all the bandwidth. This shall be achieved by per-interface settings.

setup

On the PFSense, there are 3 port:

  • WAN
  • LAN
  • OPT1

LAN is the important network here, so the OPT1 interface shall be cut back.

how?

In the 2.x series, you have to do it like this:

  • Firewall >> Traffic Shaper
  • Click on OPT1 interface in the tree on the left
  • Checkbox 'enable', enter your available bandwidth (likely 1Gbit)
  • Button 'save'
  • Button 'apply new changes'
  • Button 'add new queue'
  • Enter a queue name (this is important to do before saving! Else you have to ssh into your PFS and fix the config via viconfig. Search for queue in there and remove the old setting. If you have trouble finding it, add another queue with a unique name and search for it then.)
  • Checkbox 'enable'
  • Checkbox 'default queue'
  • Bandwidth here is again '1 Gbit' then
  • Service Curve, checkbox 'upper limit' and enter your limit in the m2 field, i.e. '10Kb'
  • Button 'save'
  • Button 'apply new changes'

Then you should be done.

testing

On the main page of the PFS webgui, add the 'traffic graphs' to the front page dashboard. There you see your throughput easily.

Load a large HD video on youtube from a host on the limited subnet, to have a completely used connection. This will not work with mobile devices, since you cannot set the desired quality there. (!)

Changing the m2 value in a separate window (and applying the changes) should show its limiting capability rather nicely.

openssl: s_client to check certificates
posted on 2016-03-18 13:47:07

In short:

openssl s_client -connect <domain.de>:443
git: split large repo into several smaller repositories
posted on 2016-03-15 17:23:56

Here splitting a single bigger repository into several repositories will be described, preserving to each its relevant history.

This guide assumes you have:

  • A running git server. (Here gitolite was used.)
  • Sufficient system administration knowledge so you won't fuck up your companies version control.

create new repositories

First step is creating the repositores which will later be used after the split. Backup the fresh repositories up is a rather smart idea in case you ned to restart from scratch, so you do not have to recreate them through gitolite.

clone the initial repository

git clone --no-hardlinks <src_repo> <name_of_new_smaller_repo>

METHOD 1: extract the relevant subrepo data

cd <name_of_new_smaller_repo>
git filter-branch --subdirectory-filter <path_to_relevant_subfolder> HEAD

METHOD 2: remove folders but keep the rest

This is untestet, but kept here for documentation purposes.

git filter-branch --tree-filter "rm -rf <name_of_folder_to_be_removed>" --prune-empty HEAD

remove old unneeded refs and other unneeded git metadata

git for-each-ref --format="%(refname)" refs/original/ | xargs -n 1 git update-ref -d
git reflog expire --expire=now --all
git reset --hard
git gc

set the origin of your newly create subrepo

... so it will use the remote repo created in the first step.

git remote rm origin
git remote add origin <your_git_user>@<your_git_server>/<reponame>
git push origin master
UBOOT: boot from SATA device
posted on 2016-03-13 20:59:13

This is a complete ripoff from a forum post here so I do not need to fear losing this snippet of information:

setenv sataroot "/dev/sda1"
setenv satadev 0
setenv satapart 1
setenv loaduimagesata 'ext2load sata ${satadev}:${satapart} ${loadaddr} ${uimage}'
setenv sataargs 'setenv bootargs console=${console},${baudrate} root=${sataroot} ${hdmi_patch} fbmem=24M video=mxcfb0:dev=hdmi,1920x1080M@60,bpp=32'
setenv sataboot 'echo Booting from sata ...; run sataargs ; bootm'
setenv bootcmd 'sata dev ${satadev}; run loaduimagesata; run sataboot;'
saveenv
reset

To switch back to the MMC card being the boot device, just change the bootcmd back to default like so:

setenv bootcmd 'mmc dev ${mmcdev}; if mmc rescan; then if run loadbootscript; then run bootscript; else if run loaduimage; then run mmcboot; else run netboot; fi; fi; else run netboot; fi'
saveenv
reset
Linux: mount LUKS / encrypted lvm btrfs subvolume partition
posted on 2016-03-13 20:37:55

When fixing more complex linux installations, you may come across LUKS partitions. Here is the workflow for a luks + lvm + btrfs setup:

# first identify your partition
lsblk -f

# open the encrypted container
# tabbing helps, if you tend to forget commands
cryptsetup luksOpen /dev/sdX1 my_encrypted_partition
# now after you entered the password, it should pop up under /dev/mapper/my_encrypted_partition

# activate all the volume groups
vgchange -aay

# create your mount destinations
mkdir /mnt/asdf
mkdir /mnt/qwer

# mount the lvm partitions, so you can work with them
# VGname = your LVM volume group
# LVname = your LVM logical volume
# SVname = your btrfs subvolume name
mount /dev/mapper/VGname/LVname /mnt/asdf
mount /dev/mapper/VGname/LVname /mnt/qwer -o subvol=@SVname

That should be all you need to fix things, in case you need it. If it is useful to have both LVM and btrfs, may be doubted. btrfs does handle volume management by itself, too.

Linux Kernel: hello world
posted on 2016-03-12 17:26

intro

Easiest this is done via kernel modules. (TBH I don't know if it is possible otherwise in a feasible way, besides building a completely new kernel?)

So for this you should know how to handle kernel modules:

  • lsmod = show loaded kernel modules

  • insmod <module> = load kernel module

  • rmmod <module> = unload kernel module

  • modprobe <module> = load kernel module and, if needed, its dependencies

  • modprobe -r <module> = unload kernel module and unneded dependencies

This guide is debian-specific

prerequisites

#install build environment
apt install build-essential

# look up your kernel version
uname -a 
apt search linux-headers | grep headers
apt install linux-headers-<YOUR_VERSION_HERE>
mkdir /lib/modules/$(uname -r)/build/

actual module

bash: add/remove leading zero to all filenames
posted on 2016-03-12 10:39:35

add leading zero to all filenames in current folder

for i in *; do mv $i 0$i; done

remove leading zero to all filenames starting with four digits in current folder

for i in $(ls -1 0{0..9}{0..9}{0..9}*); do mv $i ${i#0}; done
typo3: fix dark pictures
posted on 2016-03-10 12:39:14

If after an update, a migration or for whatever reason your typo3 installation shows pictures being too dark, your installation very likely uses the wrong color space. Like RGB instead of sRGB.

To confirm this, grep for colorspace RGB in your typo3 installation files.

You very likely have to change like three occurences in t3lib/class.t3lib_stdgraphic.php:

-colorspace RGB

is to be replaced with

-colorspace sRGB

Log into the Backend afterwards, und click on the spark symbol on top to clear all caches. (After you have chosen your site in typo3's file tree.)

If you do not have a login, create a file called ENABLE_INSTALL_TOOL in typo3conf, and comment the original line out and add this line:

$TYPO3_CONF_VARS['BE']['installToolPassword'] = 'bacb98acf97e0b6112b1d1b650b84971';

in typo3conf/localconfiguration.php, so you can access the install tool at domainname.de/typo3/install with the default password joh316. There you can add a new admin user.

After having cleared the caches and confirming everything still works as expected, remove the ENABLE_INSTALL_TOOL file and delete your newly created backend user, and fix the install tool password in localconfiguration.php back again.

To be exact there is no reason to use graphicsmagick or change any configuration variables besides the color spaces for image rendering and clearing the caches afterwards.

Linux: find deleted files with open filehandles
posted on 2016-03-09 18:47:25
lsof -nP +L1
apache: .htaccess redirect all
posted on 2016-03-08 17:11:39

To redirect every incoming request to a new URL:

RewriteEngine on 
RewriteRule ^(.*)$ http://www.mynewdomain.com/$1 [R=301,L]

This will redirect everything, try it with 302 first, instead of 301. 301 happens to be permanent, if you mess something, people have to clear their browser caches...

GNU screen: how to scroll
posted on 2016-03-03 00:31:26

Since I forgot this so very often:

CTRL+a [

use PGUP + PGDN

hit ENTER to escape again
GNU screen: log to file
posted on 2016-03-02 00:34:29

This sequence starts logging, repeat it to stop logging again. From the manpage:

C-a H       (log)         Begins/ends logging of the current window to the file "screenlog.n".

See the folder where you started screen from for screenlog.0 usually.

This can be turned on/off, will append to an existing log file.

mysql: read .csv file into table
posted on 2016-03-01 07:48:37

Also sweet and simple, the compagnion to the opposite here:

load data infile '<FILENAME>.csv' into table <TABLE> fields terminated by ';' enclosed by '"' lines terminated by '\n' ignore 1 rows;"'

<FILENAME>.csv has to be located at /var/lib/mysql, if no other path is specified besides the filename.

This blog covers .csv, .htaccess, .pfx, .vmx, /etc/crypttab, /etc/network/interfaces, /etc/sudoers, /proc, 10.04, 14.04, AS, ASA, ControlPanel, DS1054Z, GPT, HWR, Hyper-V, IPSEC, KVM, LSI, LVM, LXC, MBR, MTU, MegaCli, PHP, PKI, R, RAID, S.M.A.R.T., SNMP, SSD, SSL, TLS, TRIM, VEEAM, VMware, VServer, VirtualBox, Virtuozzo, XenServer, acpi, adaptec, algorithm, ansible, apache, apachebench, apple, arcconf, arch, architecture, areca, arping, asa, asdm, awk, backup, bandit, bar, bash, benchmarking, binding, bitrate, blackarmor, blowfish, bochs, bond, bonding, booknotes, bootable, bsd, btrfs, buffer, c-states, cache, caching, ccl, centos, certificate, certtool, cgdisk, cheatsheet, chrome, chroot, cisco, clamav, cli, clp, clush, cluster, coleslaw, colorscheme, common lisp, console, container, containers, controller, cron, cryptsetup, csync2, cu, cups, cygwin, d-states, database, date, db2, dcfldd, dcim, dd, debian, debug, debugger, debugging, decimal, desktop, df, dhclient, dhcp, diff, dig, display manager, dm-crypt, dmesg, dmidecode, dns, docker, dos, drivers, dtrace, dtrace4linux, du, dynamictracing, e2fsck, eBPF, ebook, efi, egrep, emacs, encoding, env, error, ess, esx, esxcli, esxi, ethtool, evil, expect, exportfs, factory reset, factory_reset, factoryreset, fail2ban, fbsd, fedora, file, filesystem, find, fio, firewall, firmware, fish, flashrom, forensics, free, freebsd, freedos, fritzbox, fsck, fstrim, ftp, ftps, g-states, gentoo, ghostscript, git, git-filter-branch, github, gitolite, gnutls, gradle, grep, grml, grub, grub2, guacamole, hardware, haskell, hdd, hdparm, hellowor, hex, hexdump, history, howto, htop, htpasswd, http, httpd, https, i3, icmp, ifenslave, iftop, iis, imagemagick, imap, imaps, init, innoDB, inodes, intel, ioncube, ios, iostat, ip, iperf, iphone, ipmi, ipmitool, iproute2, ipsec, iptables, ipv6, irc, irssi, iw, iwconfig, iwlist, iwlwifi, jailbreak, jails, java, javascript, javaws, js, juniper, junit, kali, kde, kemp, kernel, keyremap, kill, kpartx, krypton, lacp, lamp, languages, ldap, ldapsearch, less, leviathan, liero, lightning, links, linux, linuxin3months, lisp, list, livedisk, lmctfy, loadbalancing, locale, log, logrotate, looback, loopback, losetup, lsblk, lsi, lsof, lsusb, lsyncd, luks, lvextend, lvm, lvm2, lvreduce, lxc, lxde, macbook, macro, magento, mailclient, mailing, mailq, manpages, markdown, mbr, mdadm, megacli, micro sd, microsoft, minicom, mkfs, mktemp, mod_pagespeed, mod_proxy, modbus, modprobe, mount, mouse, movement, mpstat, multitasking, myISAM, mysql, mysql 5.7, mysql workbench, mysqlcheck, mysqldump, nagios, nas, nat, nc, netfilter, networking, nfs, nginx, nmap, nocaps, nodejs, numberingsystem, numbers, od, onyx, opcode-cache, openVZ, openlierox, openssl, openvpn, openvswitch, openwrt, oracle linux, org-mode, os, oscilloscope, overview, parallel, parameter expansion, parted, partitioning, passwd, patch, pdf, performance, pfsense, php, php7, phpmyadmin, pi, pidgin, pidstat, pins, pkill, plesk, plugin, posix, postfix, postfixadmin, postgres, postgresql, poudriere, powershell, preview, profiling, prompt, proxmox, ps, puppet, pv, pvecm, pvresize, python, qemu, qemu-img, qm, qmrestore, quicklisp, r, racktables, raid, raspberry pi, raspberrypi, raspbian, rbpi, rdp, redhat, redirect, registry, requirements, resize2fs, rewrite, rewrites, rhel, rigol, roccat, routing, rs0485, rs232, rsync, s-states, s_client, samba, sar, sata, sbcl, scite, scp, screen, scripting, seafile, seagate, security, sed, serial, serial port, setup, sftp, sg300, shell, shopware, shortcuts, showmount, signals, slattach, slip, slow-query-log, smbclient, snmpget, snmpwalk, software RAID, software raid, softwareraid, sophos, spacemacs, spam, specification, speedport, spi, sqlite, squid, ssd, ssh, ssh-add, sshd, ssl, stats, storage, strace, stronswan, su, submodules, subzone, sudo, sudoers, sup, swaks, swap, switch, switching, synaptics, synergy, sysfs, systemd, systemtap, tar, tcpdump, tcsh, tee, telnet, terminal, terminator, testdisk, testing, throughput, tmux, todo, tomcat, top, tput, trafficshaping, ttl, tuning, tunnel, tunneling, typo3, uboot, ubuntu, ubuntu 16.04, udev, uefi, ulimit, uname, unetbootin, unit testing, upstart, uptime, usb, usbstick, utf8, utm, utm 220, ux305, vcs, vgchange, vim, vimdiff, virtualbox, virtualization, visual studio code, vlan, vmstat, vmware, vnc, vncviewer, voltage, vpn, vsphere, vzdump, w, w701, wakeonlan, wargames, web, webdav, weechat, wget, whois, wicd, wifi, windowmanager, windows, wine, wireshark, wpa, wpa_passphrase, wpa_supplicant, x2x, xfce, xfreerdp, xmodem, xterm, xxd, yum, zones, zsh

View posts from 2017-02, 2017-01, 2016-12, 2016-11, 2016-10, 2016-09, 2016-08, 2016-07, 2016-06, 2016-05, 2016-04, 2016-03, 2016-02, 2016-01, 2015-12, 2015-11, 2015-10, 2015-09, 2015-08, 2015-07, 2015-06, 2015-05, 2015-04, 2015-03, 2015-02, 2015-01, 2014-12, 2014-11, 2014-10, 2014-09, 2014-08, 2014-07, 2014-06, 2014-05, 2014-04, 2014-03, 2014-01, 2013-12, 2013-11, 2013-10


Unless otherwise credited all material Creative Commons License by sjas