Posts tagged rsync

linux p2v via rsync
posted on 2016-08-03 13:09

To virtualize an existing and running linux system via rsync, install a fresh linux system. (Or do just the partitioning of the disk of the VM which you want the system to run on afterwards.) It helps to just use a single partition for /, otherwise you have to sync the mountpoints individually. In that case, create a script, in case you have to redo the data sync. (Which is likely to happen.)

If you installed a complete system first, you might consider backup up its /etc/fstab, in case you do not want to fix it afterwards by hand, but just copy-pasting the config back.

Also, if you did not install a complete linux installation on the destination VM, you will have to fix the bootmanager (read: grub2 nowadays) after the initial sync. If you did a complete install, just exclude /boot from rsync.

Boot from a live disk like GRML and mount your partition(s) where you want the data to end up(s) where you want the data to end up.

cd into the folder where you mounted the destination system's / to in your live-disk.


rsync -av --delete --progress --exclude=/dev --exclude=/sys --exclude=/proc --exclude=/mnt --exclude=/media--exclude=/boot <source-server>:/* .
Linux: website migration guide
posted on 2015-06-19 19:53:32

Migrating a website can be a tedious task, if you have problems keeping several things at once inside your head. This aims to solve this problem by presenting some proper guidelines.

Here we have a standard dynamic website with a mysql backend, served through an apache httpd.

For other databases/webservers the steps may differ in particular, but essentially this is the same theory everytime.

Mailmigration will as of now not be a part of this here, since it's gonna be long enough anyway.

Read this completely prior, as alternative ways are suggested sometimes.


This part is almost the most important, actual copying is usually not that hard if you know what you are doing. It's often harder to remember everything.

Before we start, the server can serve data of three kinds which are handled all the same way.

web data, just copy the website code
database, copy the database dump file
emails, copy the mailfiles

The server is accessed via the globally available...:


Basically these are the things you have to copy/adjust so things will go smooth.


open questions

Putting most of these questions plus the answers to them into a spreadsheed is not the worst idea. Maybe I will come up with a shell one-liner to create a .csv later.

Also it is helpful if you are able to do FXP (transfer files from one host directly to the other, without temporary saving the data/files locally), if you do not have SSH access.

  • server access via ssh is possible?

  • ssh works via key? or password only?

  • root account? (a lot of this guide assumes root privileges, I might have missed points there are no alternatives)

  • if not, do you have all necessary account credentials for all folders etc.?


  • if no ssh, do you have ftp credentials?

  • do the credentials actually work?

  • do you get a database dump you can transfer? (If you cannot access the server, you can't make a dump.)

  • are the folder accurately named?

  • how BIG is the webfolder? (so how long will copying take?)

  • which database management system is used? (i.e. mysql or postgres)

  • database credentials for it are?

  • what is the database the site is using actually called?

  • just how BIG is the database? (and so how long will copying take?)

  • what domains are pointing to the server?

  • are these actually active?

  • and can you change the DNS RR?

  • what are the DNS TTL times?

  • is mailing configured?

  • don't forget the DNS MX RR/RR's while at the last point

DNS: aquiring information active resource records

For finding out about the dns, if you have several virtual hosts on the same machine, try grepping them all there.

When having an apache, grep all vhost files for ServerName and ServerAlias. Here's a kind-of snippet, which will work if your apache vhost configs are in default locations and indented:

\grep -e '^\s\+Server' /etc/apache2/sites-enabled/*

This shows only active sites, check sites-available if you have to migrate sites which are currently turned off, too.

The resulting list, if sanitized, can be piped on the shell and used with something like host/nslookup/echo + dig +short, to easily check which domains are still running. Check all the records, not just the A/AAAA (quad-A is ipv4, single-A is ipv4) records, also MX and whatever is set. If the exit code is non-zero, no dns anymore and less work for you. Providing a script here would not help much, since you should know what you are doing here anyway and it would most likely not help you much.

and maybe prepare the webserver, too

In case the apache config is, lets say, 'adventurous', do apache2ctl -S (Debian/Ubuntu) or httpd -S to see which domains are hosted, and in which file these are defined. Then search there for ServerName/ServerAlias directives.

If the webserver happens to have all vhosts defined in one huge file (which ist just... very not great), remove the configuration and place them into a separated file. In Debian-based Linuces you can use a2ensite <vhost-config-filename> / a2dissite <vhost-config-filename> to enable/disable single websites easily. On Redhat-based ones you create the symlinks to the configfolder apache is configured to load manually and delete them also by hand. (This isn't any different from what a2en/dissite do.) All this only for the sites you want to migrate.

Of course, you can just comment out the information on your vhosts from the config, but just... don't.

For other webservers all this is different, of course, but you get the idea.

DNS: get the domains and the website together, information-wise

Refer to the website via its main link. (apache ServerName from above.) But make sure to note all other aliases there, too. (apache ServerAlias from above.) Since you can only migrate one site after another, this helps to keep track. Write all this down, each alias in another row. Maybe put the inactive ones into an extra column there, too. Could be that these should be prolonged again, or were incorrectly set. (I.e. it did not point to the webserver when you checked.)

Write the set TTL into the next column, along with the current date. (Usually TTL is 86400, which means 24 hours, which is exactly how long it will take until your change to 1800 seconds becomes finally active. If the TTL was longer than 86400 for whatever reason, note that into your list, too!)

DNS: lower TTL the day before the migration

After having created a list and checked which domains are currently active, set the default TTL time to 1800. (Just don't go below, 30 mins are short while you do the migration. Also the registrar might prefer you not to.)

DNS: plan b in case you have dozens of websites to migrate

If you have A LOT of websites that should go from one server to the next, try migrating and testing everything (via entries in the hosts file). Then switch the ip's of the servers with each other. That way no dns changes are needed (except if you have dead domains), because this shit can become tedious, too.

TBD / todo

Nothing more here now, until i am motivated again to write more stuff up.

This blog covers .csv, .htaccess, .pfx, .vmx, /etc/crypttab, /etc/network/interfaces, /etc/sudoers, /proc, 10.04, 14.04, AS, ASA, ControlPanel, DS1054Z, GPT, HWR, Hyper-V, IPSEC, KVM, LSI, LVM, LXC, MBR, MTU, MegaCli, PHP, PKI, R, RAID, S.M.A.R.T., SNMP, SSD, SSL, TLS, TRIM, VEEAM, VMware, VServer, VirtualBox, Virtuozzo, XenServer, acpi, adaptec, algorithm, ansible, apache, apachebench, apple, arcconf, arch, architecture, areca, arping, asa, asdm, awk, backup, bandit, bar, bash, benchmarking, binding, bitrate, blackarmor, blowfish, bochs, bond, bonding, booknotes, bootable, bsd, btrfs, buffer, c-states, cache, caching, ccl, centos, certificate, certtool, cgdisk, cheatsheet, chrome, chroot, cisco, clamav, cli, clp, clush, cluster, coleslaw, colorscheme, common lisp, console, container, containers, controller, cron, cryptsetup, csync2, cu, cups, cygwin, d-states, database, date, db2, dcfldd, dcim, dd, debian, debug, debugger, debugging, decimal, desktop, df, dhclient, dhcp, diff, dig, display manager, dm-crypt, dmesg, dmidecode, dns, docker, dos, drivers, dtrace, dtrace4linux, du, dynamictracing, e2fsck, eBPF, ebook, efi, egrep, emacs, encoding, env, error, ess, esx, esxcli, esxi, ethtool, evil, expect, exportfs, factory reset, factory_reset, factoryreset, fail2ban, fbsd, fedora, file, filesystem, find, fio, firewall, firmware, fish, flashrom, forensics, free, freebsd, freedos, fritzbox, fsck, fstrim, ftp, ftps, g-states, gentoo, ghostscript, git, git-filter-branch, github, gitolite, gnutls, gradle, grep, grml, grub, grub2, guacamole, hardware, haskell, hdd, hdparm, hellowor, hex, hexdump, history, howto, htop, htpasswd, http, httpd, https, i3, icmp, ifenslave, iftop, iis, imagemagick, imap, imaps, init, innoDB, inodes, intel, ioncube, ios, iostat, ip, iperf, iphone, ipmi, ipmitool, iproute2, ipsec, iptables, ipv6, irc, irssi, iw, iwconfig, iwlist, iwlwifi, jailbreak, jails, java, javascript, javaws, js, juniper, junit, kali, kde, kemp, kernel, keyremap, kill, kpartx, krypton, lacp, lamp, languages, ldap, ldapsearch, less, leviathan, liero, lightning, links, linux, linuxin3months, lisp, list, livedisk, lmctfy, loadbalancing, locale, log, logrotate, looback, loopback, losetup, lsblk, lsi, lsof, lsusb, lsyncd, luks, lvextend, lvm, lvm2, lvreduce, lxc, lxde, macbook, macro, magento, mailclient, mailing, mailq, manpages, markdown, mbr, mdadm, megacli, micro sd, microsoft, minicom, mkfs, mktemp, mod_pagespeed, mod_proxy, modbus, modprobe, mount, mouse, movement, mpstat, multitasking, myISAM, mysql, mysql 5.7, mysql workbench, mysqlcheck, mysqldump, nagios, nas, nat, nc, netfilter, networking, nfs, nginx, nmap, nocaps, nodejs, numberingsystem, numbers, od, onyx, opcode-cache, openVZ, openlierox, openssl, openvpn, openvswitch, openwrt, oracle linux, org-mode, os, oscilloscope, overview, parallel, parameter expansion, parted, partitioning, passwd, patch, pdf, performance, pfsense, php, php7, phpmyadmin, pi, pidgin, pidstat, pins, pkill, plesk, plugin, posix, postfix, postfixadmin, postgres, postgresql, poudriere, powershell, preview, profiling, prompt, proxmox, ps, puppet, pv, pvecm, pvresize, python, qemu, qemu-img, qm, qmrestore, quicklisp, r, racktables, raid, raspberry pi, raspberrypi, raspbian, rbpi, rdp, redhat, redirect, registry, requirements, resize2fs, rewrite, rewrites, rhel, rigol, roccat, routing, rs0485, rs232, rsync, s-states, s_client, samba, sar, sata, sbcl, scite, scp, screen, scripting, seafile, seagate, security, sed, serial, serial port, setup, sftp, sg300, shell, shopware, shortcuts, showmount, signals, slattach, slip, slow-query-log, smbclient, snmpget, snmpwalk, software RAID, software raid, softwareraid, sophos, spacemacs, spam, specification, speedport, spi, sqlite, squid, ssd, ssh, ssh-add, sshd, ssl, stats, storage, strace, stronswan, su, submodules, subzone, sudo, sudoers, sup, swaks, swap, switch, switching, synaptics, synergy, sysfs, systemd, systemtap, tar, tcpdump, tcsh, tee, telnet, terminal, terminator, testdisk, testing, throughput, tmux, todo, tomcat, top, tput, trafficshaping, ttl, tuning, tunnel, tunneling, typo3, uboot, ubuntu, ubuntu 16.04, udev, uefi, ulimit, uname, unetbootin, unit testing, upstart, uptime, usb, usbstick, utf8, utm, utm 220, ux305, vcs, vgchange, vim, vimdiff, virtualbox, virtualization, visual studio code, vlan, vmstat, vmware, vnc, vncviewer, voltage, vpn, vsphere, vzdump, w, w701, wakeonlan, wargames, web, webdav, weechat, wget, whois, wicd, wifi, windowmanager, windows, wine, wireshark, wpa, wpa_passphrase, wpa_supplicant, x2x, xfce, xfreerdp, xmodem, xterm, xxd, yum, zones, zsh

View posts from 2017-02, 2017-01, 2016-12, 2016-11, 2016-10, 2016-09, 2016-08, 2016-07, 2016-06, 2016-05, 2016-04, 2016-03, 2016-02, 2016-01, 2015-12, 2015-11, 2015-10, 2015-09, 2015-08, 2015-07, 2015-06, 2015-05, 2015-04, 2015-03, 2015-02, 2015-01, 2014-12, 2014-11, 2014-10, 2014-09, 2014-08, 2014-07, 2014-06, 2014-05, 2014-04, 2014-03, 2014-01, 2013-12, 2013-11, 2013-10

Unless otherwise credited all material Creative Commons License by sjas