Posts tagged dig

determine ASN from shell
posted on 2016-06-29 01:35

whois can actually use different whois servers for querying. Their output differs, but whois.cymru.com is pretty decent:

whois -h whois.cymru.com <IP-OR-ASN>

I.e.

sjas@zen:~$ whois -h whois.cymru.com $(host sjas.de | head -1 | awk '{print $4}')
AS      | IP               | AS Name
24940   | 78.47.176.149    | HETZNER-AS , DE

In reverse, for looking up which organization is behind a specific AS number:

sjas@zen:~$ whois -h whois.cymru.com AS24940 | tail -1
HETZNER-AS , DE

Downside is, it will not lookup domains, only IP's or ASN's.

If no whois server is specified via the -h flag, whois.arin.net will be used for domains, IP addresses and ASN numbers. whois.cymru.com is however more terse and often more preferable.

DNS: resolution and reverse resolution script
posted on 2016-05-21 20:54

This is a quick-and-dirty for loop for checking a list of dns A resource records using dig. CNAME's are not handled like they should, thus are printed not in the same line, so this can not be used for being parsed without first having a look at the output and curating it first, if these are in use.

for i in sjas.de blog.sjas.de; do echo -n $'\e[33;1m'$i$'\e[0m '; TEMP=`dig +short $i`; echo -n "$TEMP "; TEMP=`dig -x $TEMP +short`; echo ${TEMP%.}; done | column -t

Instead of editing the for-loop, it might be helpful using a heredoc instead:

echo; cat << EOF | while read i; do echo -n $'\e[33;1m'$i$'\e[0m '; TEMP=`dig +short $i`; echo -n "$TEMP "; TEMP=`dig -x $TEMP +short`; echo ${TEMP%.}; done | column -t; echo

Paste this into the shell, followed by a paste of lines of the domains you want, and type EOF afterwards.

Example:

sjas@sjas:~/blog$ echo; cat << EOF | while read i; do echo -n $'\e[33;1m'$i$'\e[0m '; TEMP=`dig +short $i`; echo -n "$TEMP "; TEMP=`dig -x $TEMP +short`; echo ${TEMP%.}; done | column -t; echo
sjas.de
ix.de
asdf.de
EOF

sjas.de  78.47.176.149  static.149.176.47.78.clients.your-server.de
ix.de    193.99.144.80  redirector.heise.de
asdf.de  80.237.132.85  wp078.webpack.hosteurope.de

sjas@sjas:~/blog$ 
Linux: website migration guide
posted on 2015-06-19 19:53:32

Migrating a website can be a tedious task, if you have problems keeping several things at once inside your head. This aims to solve this problem by presenting some proper guidelines.

Here we have a standard dynamic website with a mysql backend, served through an apache httpd.

For other databases/webservers the steps may differ in particular, but essentially this is the same theory everytime.

Mailmigration will as of now not be a part of this here, since it's gonna be long enough anyway.

Read this completely prior, as alternative ways are suggested sometimes.

preparations

This part is almost the most important, actual copying is usually not that hard if you know what you are doing. It's often harder to remember everything.

Before we start, the server can serve data of three kinds which are handled all the same way.

web data, just copy the website code
database, copy the database dump file
emails, copy the mailfiles

The server is accessed via the globally available...:

dns

Basically these are the things you have to copy/adjust so things will go smooth.

preparations

open questions

Putting most of these questions plus the answers to them into a spreadsheed is not the worst idea. Maybe I will come up with a shell one-liner to create a .csv later.

Also it is helpful if you are able to do FXP (transfer files from one host directly to the other, without temporary saving the data/files locally), if you do not have SSH access.

  • server access via ssh is possible?

  • ssh works via key? or password only?

  • root account? (a lot of this guide assumes root privileges, I might have missed points there are no alternatives)

  • if not, do you have all necessary account credentials for all folders etc.?

  • DO THESE WORK?

  • if no ssh, do you have ftp credentials?

  • do the credentials actually work?

  • do you get a database dump you can transfer? (If you cannot access the server, you can't make a dump.)

  • are the folder accurately named?

  • how BIG is the webfolder? (so how long will copying take?)

  • which database management system is used? (i.e. mysql or postgres)

  • database credentials for it are?

  • what is the database the site is using actually called?

  • just how BIG is the database? (and so how long will copying take?)

  • what domains are pointing to the server?

  • are these actually active?

  • and can you change the DNS RR?

  • what are the DNS TTL times?

  • is mailing configured?

  • don't forget the DNS MX RR/RR's while at the last point

DNS: aquiring information active resource records

For finding out about the dns, if you have several virtual hosts on the same machine, try grepping them all there.

When having an apache, grep all vhost files for ServerName and ServerAlias. Here's a kind-of snippet, which will work if your apache vhost configs are in default locations and indented:

\grep -e '^\s\+Server' /etc/apache2/sites-enabled/*

This shows only active sites, check sites-available if you have to migrate sites which are currently turned off, too.

The resulting list, if sanitized, can be piped on the shell and used with something like host/nslookup/echo + dig +short, to easily check which domains are still running. Check all the records, not just the A/AAAA (quad-A is ipv4, single-A is ipv4) records, also MX and whatever is set. If the exit code is non-zero, no dns anymore and less work for you. Providing a script here would not help much, since you should know what you are doing here anyway and it would most likely not help you much.

and maybe prepare the webserver, too

In case the apache config is, lets say, 'adventurous', do apache2ctl -S (Debian/Ubuntu) or httpd -S to see which domains are hosted, and in which file these are defined. Then search there for ServerName/ServerAlias directives.

If the webserver happens to have all vhosts defined in one huge file (which ist just... very not great), remove the configuration and place them into a separated file. In Debian-based Linuces you can use a2ensite <vhost-config-filename> / a2dissite <vhost-config-filename> to enable/disable single websites easily. On Redhat-based ones you create the symlinks to the configfolder apache is configured to load manually and delete them also by hand. (This isn't any different from what a2en/dissite do.) All this only for the sites you want to migrate.

Of course, you can just comment out the information on your vhosts from the config, but just... don't.

For other webservers all this is different, of course, but you get the idea.

DNS: get the domains and the website together, information-wise

Refer to the website via its main link. (apache ServerName from above.) But make sure to note all other aliases there, too. (apache ServerAlias from above.) Since you can only migrate one site after another, this helps to keep track. Write all this down, each alias in another row. Maybe put the inactive ones into an extra column there, too. Could be that these should be prolonged again, or were incorrectly set. (I.e. it did not point to the webserver when you checked.)

Write the set TTL into the next column, along with the current date. (Usually TTL is 86400, which means 24 hours, which is exactly how long it will take until your change to 1800 seconds becomes finally active. If the TTL was longer than 86400 for whatever reason, note that into your list, too!)

DNS: lower TTL the day before the migration

After having created a list and checked which domains are currently active, set the default TTL time to 1800. (Just don't go below, 30 mins are short while you do the migration. Also the registrar might prefer you not to.)

DNS: plan b in case you have dozens of websites to migrate

If you have A LOT of websites that should go from one server to the next, try migrating and testing everything (via entries in the hosts file). Then switch the ip's of the servers with each other. That way no dns changes are needed (except if you have dead domains), because this shit can become tedious, too.

TBD / todo

Nothing more here now, until i am motivated again to write more stuff up.

bash: combined dns-reverse-dns-lookup
posted on 2015-02-20 12:27:15

On dns lookups at work

While working with domains, you often need to to a dns lookup, to find out the ip of the machine in question (at least when working with several hundred web servers ;)), followed by a reverse dns lookup on the ip to find out the actual hostname. The regular hostname is just easier to remember than the IP. It's bad enough with IPv4, and will become worse with IPv6.

I.e. usually you do something like this:

[sjas@ctr-014 ~]% host ix.de
ix.de has address 193.99.144.80
ix.de has IPv6 address 2a02:2e0:3fe:1001:302::
ix.de mail is handled by 10 relay.heise.de.
ix.de mail is handled by 50 secondarymx.heise.de.
[sjas@ctr-014 ~]% host 193.99.144.80
80.144.99.193.in-addr.arpa domain name pointer redirector.heise.de.
[sjas@ctr-014 ~]%

or this:

[sjas@ctr-014 ~]% dig ix.de +short
193.99.144.80
[sjas@ctr-014 ~]% dig -x 193.99.144.80 +short
redirector.heise.de.
[sjas@ctr-014 ~]%

This can be 'shortened' into a single step with proper output:

[sjas@ctr-014 ~]% echo ${$(dig -x $(dig ix.de +short) +short)%?}
redirector.heise.de
[sjas@ctr-014 ~]%

proper solution

Since this is kind of unhandy (and let's be honest, bash sucks sometimes), just place it into a function definition in your .bashrc:

rdns() {
    echo ${$(dig -x $(dig $1 +short) +short)%?}
}

An in-depth explanation of this bash 'gem' will be added here, if I do not forget to add it in the near future. :)

Which let's you do:

[sjas@ctr-014 ~]% rdns ix.de
redirector.heise.de

script explanation

In short:

echo "${$(dig -x $(dig $1 +short) +short)%?}"

Echo a string...

echo "                                      "

... which stems out from a combination of parameter expansion...

      ${               $1                  }"

... wherein also a suffix is removed, in this case ? representing a single char.

                                         %?

There a subshell is used to run the dig command in a subshell...

        $(dig                     +short)

... which in turn runs another dig call in another subshell...

                 $(dig    +short)

Use man bash to get further info on this stuff.

This blog covers .csv, .htaccess, .pfx, .vmx, /etc/crypttab, /etc/network/interfaces, /etc/sudoers, /proc, 10.04, 14.04, AS, ASA, ControlPanel, DS1054Z, GPT, HWR, Hyper-V, IPSEC, KVM, LSI, LVM, LXC, MBR, MTU, MegaCli, PHP, PKI, R, RAID, S.M.A.R.T., SNMP, SSD, SSL, TLS, TRIM, VEEAM, VMware, VServer, VirtualBox, Virtuozzo, XenServer, acpi, adaptec, algorithm, ansible, apache, apachebench, apple, arcconf, arch, architecture, areca, arping, asa, asdm, awk, backup, bandit, bar, bash, benchmarking, binding, bitrate, blackarmor, blowfish, bochs, bond, bonding, booknotes, bootable, bsd, btrfs, buffer, c-states, cache, caching, ccl, centos, certificate, certtool, cgdisk, cheatsheet, chrome, chroot, cisco, clamav, cli, clp, clush, cluster, coleslaw, colorscheme, common lisp, console, container, containers, controller, cron, cryptsetup, csync2, cu, cups, cygwin, d-states, database, date, db2, dcfldd, dcim, dd, debian, debug, debugger, debugging, decimal, desktop, df, dhclient, dhcp, diff, dig, display manager, dm-crypt, dmesg, dmidecode, dns, docker, dos, drivers, dtrace, dtrace4linux, du, dynamictracing, e2fsck, eBPF, ebook, efi, egrep, emacs, encoding, env, error, ess, esx, esxcli, esxi, ethtool, evil, expect, exportfs, factory reset, factory_reset, factoryreset, fail2ban, fbsd, fedora, file, filesystem, find, fio, firewall, firmware, fish, flashrom, forensics, free, freebsd, freedos, fritzbox, fsck, fstrim, ftp, ftps, g-states, gentoo, ghostscript, git, git-filter-branch, github, gitolite, gnutls, gradle, grep, grml, grub, grub2, guacamole, hardware, haskell, hdd, hdparm, hellowor, hex, hexdump, history, howto, htop, htpasswd, http, httpd, https, i3, icmp, ifenslave, iftop, iis, imagemagick, imap, imaps, init, innoDB, inodes, intel, ioncube, ios, iostat, ip, iperf, iphone, ipmi, ipmitool, iproute2, ipsec, iptables, ipv6, irc, irssi, iw, iwconfig, iwlist, iwlwifi, jailbreak, jails, java, javascript, javaws, js, juniper, junit, kali, kde, kemp, kernel, keyremap, kill, kpartx, krypton, lacp, lamp, languages, ldap, ldapsearch, less, leviathan, liero, lightning, links, linux, linuxin3months, lisp, list, livedisk, lmctfy, loadbalancing, locale, log, logrotate, looback, loopback, losetup, lsblk, lsi, lsof, lsusb, lsyncd, luks, lvextend, lvm, lvm2, lvreduce, lxc, lxde, macbook, macro, magento, mailclient, mailing, mailq, manpages, markdown, mbr, mdadm, megacli, micro sd, microsoft, minicom, mkfs, mktemp, mod_pagespeed, mod_proxy, modbus, modprobe, mount, mouse, movement, mpstat, multitasking, myISAM, mysql, mysql 5.7, mysql workbench, mysqlcheck, mysqldump, nagios, nas, nat, nc, netfilter, networking, nfs, nginx, nmap, nocaps, nodejs, numberingsystem, numbers, od, onyx, opcode-cache, openVZ, openlierox, openssl, openvpn, openvswitch, openwrt, oracle linux, org-mode, os, oscilloscope, overview, parallel, parameter expansion, parted, partitioning, passwd, patch, pdf, performance, pfsense, php, php7, phpmyadmin, pi, pidgin, pidstat, pins, pkill, plesk, plugin, posix, postfix, postfixadmin, postgres, postgresql, poudriere, powershell, preview, profiling, prompt, proxmox, ps, puppet, pv, pvecm, pvresize, python, qemu, qemu-img, qm, qmrestore, quicklisp, r, racktables, raid, raspberry pi, raspberrypi, raspbian, rbpi, rdp, redhat, redirect, registry, requirements, resize2fs, rewrite, rewrites, rhel, rigol, roccat, routing, rs0485, rs232, rsync, s-states, s_client, samba, sar, sata, sbcl, scite, scp, screen, scripting, seafile, seagate, security, sed, serial, serial port, setup, sftp, sg300, shell, shopware, shortcuts, showmount, signals, slattach, slip, slow-query-log, smbclient, snmpget, snmpwalk, software RAID, software raid, softwareraid, sophos, spacemacs, spam, specification, speedport, spi, sqlite, squid, ssd, ssh, ssh-add, sshd, ssl, stats, storage, strace, stronswan, su, submodules, subzone, sudo, sudoers, sup, swaks, swap, switch, switching, synaptics, synergy, sysfs, systemd, systemtap, tar, tcpdump, tcsh, tee, telnet, terminal, terminator, testdisk, testing, throughput, tmux, todo, tomcat, top, tput, trafficshaping, ttl, tuning, tunnel, tunneling, typo3, uboot, ubuntu, ubuntu 16.04, udev, uefi, ulimit, uname, unetbootin, unit testing, upstart, uptime, usb, usbstick, utf8, utm, utm 220, ux305, vcs, vgchange, vim, vimdiff, virtualbox, virtualization, visual studio code, vlan, vmstat, vmware, vnc, vncviewer, voltage, vpn, vsphere, vzdump, w, w701, wakeonlan, wargames, web, webdav, weechat, wget, whois, wicd, wifi, windowmanager, windows, wine, wireshark, wpa, wpa_passphrase, wpa_supplicant, x2x, xfce, xfreerdp, xmodem, xterm, xxd, yum, zones, zsh

View posts from 2017-02, 2017-01, 2016-12, 2016-11, 2016-10, 2016-09, 2016-08, 2016-07, 2016-06, 2016-05, 2016-04, 2016-03, 2016-02, 2016-01, 2015-12, 2015-11, 2015-10, 2015-09, 2015-08, 2015-07, 2015-06, 2015-05, 2015-04, 2015-03, 2015-02, 2015-01, 2014-12, 2014-11, 2014-10, 2014-09, 2014-08, 2014-07, 2014-06, 2014-05, 2014-04, 2014-03, 2014-01, 2013-12, 2013-11, 2013-10


Unless otherwise credited all material Creative Commons License by sjas