Posts tagged file

xxd vs hexdump vs od for examining disk dumps from a VMware image

posted on 2016-09-14 21:49

the problem

The problem at hand was, VEEAM backup could not be restored. Neither could the backup be restored, nor could the the backup be opened in from the GUI. So how to verify wether something could be rescued from there?

getting the disk image out of the VEEAM backup

VEEAM lets you extract single images from the complete backup with its Extract.exe utility. Simply locate the executable on disk and start it without parameters. Then you are prompted for the full path to the complete .vbk backup file, afterward select the image you want to extract.

first look at the disk dump

After copying the folder with all the extracted contents, onto a linux box, the fun could start.

  • The VMware image is in the `diskname-###.vmdk' file.
  • .vmdk is the disk configuration file.
  • .nvram is the virtual machine's BIOS.
  • .vmx is the primary configuration file.
  • .vmxf is supplemental configuration.

examining the disk image

Easiest this is done through parted, showing once the size in sectors. This helps when using dd later and skipping over the first x sectors. Afterwards in bytes, for the offset in losetup, which will be easier than dd-skipping around..

Sectors:

root@workstation:/home/sjas/ftp# parted my_server-flat.vmdk u s p
Error: Can't have a partition outside the disk!
Ignore/Cancel? i                                                          
Error: Can't have a partition outside the disk!
Ignore/Cancel? i                                                          
Model:  (file)
Disk /home/sjas/ftp/my_server-flat.vmdk: 83869185s
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 

Number  Start     End        Size       Type     File system     Flags
 1      2048s     8390655s   8388608s   primary  linux-swap(v1)
 2      8390656s  83886079s  75495424s  primary                  boot

Bytes:

root@workstation:/home/sjas/ftp# parted my_server-flat.vmdk u b p
Error: Can't have a partition outside the disk!
Ignore/Cancel? i                                                          
Error: Can't have a partition outside the disk!
Ignore/Cancel? i                                                          
Model:  (file)
Disk /home/sjas/ftp/my_server-flat.vmdk: 42941022720B
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 

Number  Start        End           Size          Type     File system     Flags
 1      1048576B     4296015871B   4294967296B   primary  linux-swap(v1)
 2      4296015872B  42949672959B  38653657088B  primary                  boot

setting up the loop device, so the filesystem from within the file could be read

losetup  # should show nothing, so the first loop device we will use will be loop0
losetup -f # can alternatively be used to find the first free loop device
losetup /dev/loop0 my_server-flat.vmdk

To have easier access to the second partition (so we can use dd without having to use the skip flag all the time), we will loop the second partition, too. Offset is passed by -o in sectors, see the parted output above:

losetup -o 8390656 /dev/loop1 /dev/loop0

Then losetup should look like this:

root@workstation:/home/sjas/ftp# losetup 
NAME       SIZELIMIT     OFFSET AUTOCLEAR RO BACK-FILE
/dev/loop0         0          0         0  0 /home/sjas/ftp/my_server-flat.vmdk
/dev/loop1         0 4296015872         0  0 /dev/loop0

Alternatively you can use losetup -a to show the currently used loop devices.

Once you are done with everything, the loop devices could be deleted via losetup -d /dev/loopX for each one in use.

Alternatively, kpartx can be used, too. It would create device mappings automatically when run like kpartx -av my_server-flat.vmdk. The next free loop device under /dev/loopX would be chosen, and its partition could then be found under loopXp1, loopXp2, etc. Afterwards it could be deleted via kpartx -d my_server-flat.vmdk. However I prefer doing it manually, as with broken partitions kpartx cannot work properly, of course.

examination

Via dd the blocks can be read directly from the loop device'd disk. hexdump, xxd or od will make visible what is actually on there.

Initially this post grew out of the will to document the differences between them, but grew to include how to use the loop device stuff, too.

First lets have a look at the MBR, which is on the first block / 512 bytes on the device

root@workstation:/home/sjas/ftp# dd if=/dev/loop0 bs=512 count=1 2>/dev/null | file -
/dev/stdin: DOS/MBR boot sector; GRand Unified Bootloader, stage1 version 0x3, boot drive 0x80, 1st sector stage2 0x2443e60, GRUB version 0.94

Now lets check wether a VBR is present on the second partition or not, which is not the cause:

root@workstation:/home/sjas/ftp# dd if=/dev/loop1 bs=512 count=1 2>/dev/null | file -
/dev/stdin: data

For illustration here are the three tools in action, showing the MBR of loop0. Lets have a look at the actual disk contents:

xxd:

root@workstation:/home/sjas/ftp# dd if=/dev/loop0 bs=512 count=1 2>/dev/null | xxd
0000000: eb48 9010 8ed0 bc00 b0b8 0000 8ed8 8ec0  .H..............
0000010: fbbe 007c bf00 06b9 0002 f3a4 ea21 0600  ...|.........!..
0000020: 00be be07 3804 750b 83c6 1081 fefe 0775  ....8.u........u
0000030: f3eb 16b4 02b0 01bb 007c b280 8a74 0302  .........|...t..
0000040: 8000 0080 603e 4402 0008 fa90 90f6 c280  ....`>D.........
0000050: 7502 b280 ea59 7c00 0031 c08e d88e d0bc  u....Y|..1......
0000060: 0020 fba0 407c 3cff 7402 88c2 52f6 c280  . ..@|<.t...R...
0000070: 7454 b441 bbaa 55cd 135a 5272 4981 fb55  tT.A..U..ZRrI..U
0000080: aa75 43a0 417c 84c0 7505 83e1 0174 3766  .uC.A|..u....t7f
0000090: 8b4c 10be 057c c644 ff01 668b 1e44 7cc7  .L...|.D..f..D|.
00000a0: 0410 00c7 4402 0100 6689 5c08 c744 0600  ....D...f.\..D..
00000b0: 7066 31c0 8944 0466 8944 0cb4 42cd 1372  pf1..D.f.D..B..r
00000c0: 05bb 0070 eb7d b408 cd13 730a f6c2 800f  ...p.}....s.....
00000d0: 84f0 00e9 8d00 be05 7cc6 44ff 0066 31c0  ........|.D..f1.
00000e0: 88f0 4066 8944 0431 d288 cac1 e202 88e8  ..@f.D.1........
00000f0: 88f4 4089 4408 31c0 88d0 c0e8 0266 8904  ..@.D.1......f..
0000100: 66a1 447c 6631 d266 f734 8854 0a66 31d2  f.D|f1.f.4.T.f1.
0000110: 66f7 7404 8854 0b89 440c 3b44 087d 3c8a  f.t..T..D.;D.}<.
0000120: 540d c0e2 068a 4c0a fec1 08d1 8a6c 0c5a  T.....L......l.Z
0000130: 8a74 0bbb 0070 8ec3 31db b801 02cd 1372  .t...p..1......r
0000140: 2a8c c38e 0648 7c60 1eb9 0001 8edb 31f6  *....H|`......1.
0000150: 31ff fcf3 a51f 61ff 2642 7cbe 7f7d e840  1.....a.&B|..}.@
0000160: 00eb 0ebe 847d e838 00eb 06be 8e7d e830  .....}.8.....}.0
0000170: 00be 937d e82a 00eb fe47 5255 4220 0047  ...}.*...GRUB .G
0000180: 656f 6d00 4861 7264 2044 6973 6b00 5265  eom.Hard Disk.Re
0000190: 6164 0020 4572 726f 7200 bb01 00b4 0ecd  ad. Error.......
00001a0: 10ac 3c00 75f4 c300 0000 0000 0000 0000  ..<.u...........
00001b0: 0000 0000 0000 0000 9b09 0b00 0000 0020  ............... 
00001c0: 2100 824b 810a 0008 0000 0000 8000 804b  !..K...........K
00001d0: 820a 83fe ffff 0008 8000 00f8 7f04 0000  ................
00001e0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00001f0: 0000 0000 0000 0000 0000 0000 0000 55aa  ..............U.

hexdump:

root@workstation:/home/sjas/ftp# dd if=/dev/loop0 bs=512 count=1 2>/dev/null | hexdump -vC
00000000  eb 48 90 10 8e d0 bc 00  b0 b8 00 00 8e d8 8e c0  |.H..............|
00000010  fb be 00 7c bf 00 06 b9  00 02 f3 a4 ea 21 06 00  |...|.........!..|
00000020  00 be be 07 38 04 75 0b  83 c6 10 81 fe fe 07 75  |....8.u........u|
00000030  f3 eb 16 b4 02 b0 01 bb  00 7c b2 80 8a 74 03 02  |.........|...t..|
00000040  80 00 00 80 60 3e 44 02  00 08 fa 90 90 f6 c2 80  |....`>D.........|
00000050  75 02 b2 80 ea 59 7c 00  00 31 c0 8e d8 8e d0 bc  |u....Y|..1......|
00000060  00 20 fb a0 40 7c 3c ff  74 02 88 c2 52 f6 c2 80  |. ..@|<.t...R...|
00000070  74 54 b4 41 bb aa 55 cd  13 5a 52 72 49 81 fb 55  |tT.A..U..ZRrI..U|
00000080  aa 75 43 a0 41 7c 84 c0  75 05 83 e1 01 74 37 66  |.uC.A|..u....t7f|
00000090  8b 4c 10 be 05 7c c6 44  ff 01 66 8b 1e 44 7c c7  |.L...|.D..f..D|.|
000000a0  04 10 00 c7 44 02 01 00  66 89 5c 08 c7 44 06 00  |....D...f.\..D..|
000000b0  70 66 31 c0 89 44 04 66  89 44 0c b4 42 cd 13 72  |pf1..D.f.D..B..r|
000000c0  05 bb 00 70 eb 7d b4 08  cd 13 73 0a f6 c2 80 0f  |...p.}....s.....|
000000d0  84 f0 00 e9 8d 00 be 05  7c c6 44 ff 00 66 31 c0  |........|.D..f1.|
000000e0  88 f0 40 66 89 44 04 31  d2 88 ca c1 e2 02 88 e8  |..@f.D.1........|
000000f0  88 f4 40 89 44 08 31 c0  88 d0 c0 e8 02 66 89 04  |..@.D.1......f..|
00000100  66 a1 44 7c 66 31 d2 66  f7 34 88 54 0a 66 31 d2  |f.D|f1.f.4.T.f1.|
00000110  66 f7 74 04 88 54 0b 89  44 0c 3b 44 08 7d 3c 8a  |f.t..T..D.;D.}<.|
00000120  54 0d c0 e2 06 8a 4c 0a  fe c1 08 d1 8a 6c 0c 5a  |T.....L......l.Z|
00000130  8a 74 0b bb 00 70 8e c3  31 db b8 01 02 cd 13 72  |.t...p..1......r|
00000140  2a 8c c3 8e 06 48 7c 60  1e b9 00 01 8e db 31 f6  |*....H|`......1.|
00000150  31 ff fc f3 a5 1f 61 ff  26 42 7c be 7f 7d e8 40  |1.....a.&B|..}.@|
00000160  00 eb 0e be 84 7d e8 38  00 eb 06 be 8e 7d e8 30  |.....}.8.....}.0|
00000170  00 be 93 7d e8 2a 00 eb  fe 47 52 55 42 20 00 47  |...}.*...GRUB .G|
00000180  65 6f 6d 00 48 61 72 64  20 44 69 73 6b 00 52 65  |eom.Hard Disk.Re|
00000190  61 64 00 20 45 72 72 6f  72 00 bb 01 00 b4 0e cd  |ad. Error.......|
000001a0  10 ac 3c 00 75 f4 c3 00  00 00 00 00 00 00 00 00  |..<.u...........|
000001b0  00 00 00 00 00 00 00 00  9b 09 0b 00 00 00 00 20  |............... |
000001c0  21 00 82 4b 81 0a 00 08  00 00 00 00 80 00 80 4b  |!..K...........K|
000001d0  82 0a 83 fe ff ff 00 08  80 00 00 f8 7f 04 00 00  |................|
000001e0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 55 aa  |..............U.|
00000200

od:

root@workstation:/home/sjas/ftp# dd if=/dev/loop0 bs=512 count=1 2>/dev/null | od -v -A d -t x2z
0000000 48eb 1090 d08e 00bc b8b0 0000 d88e c08e  >.H..............<
0000016 befb 7c00 00bf b906 0200 a4f3 21ea 0006  >...|.........!..<
0000032 be00 07be 0438 0b75 c683 8110 fefe 7507  >....8.u........u<
0000048 ebf3 b416 b002 bb01 7c00 80b2 748a 0203  >.........|...t..<
0000064 0080 8000 3e60 0244 0800 90fa f690 80c2  >....`>D.........<
0000080 0275 80b2 59ea 007c 3100 8ec0 8ed8 bcd0  >u....Y|..1......<
0000096 2000 a0fb 7c40 ff3c 0274 c288 f652 80c2  >. ..@|<.t...R...<
0000112 5474 41b4 aabb cd55 5a13 7252 8149 55fb  >tT.A..U..ZRrI..U<
0000128 75aa a043 7c41 c084 0575 e183 7401 6637  >.uC.A|..u....t7f<
0000144 4c8b be10 7c05 44c6 01ff 8b66 441e c77c  >.L...|.D..f..D|.<
0000160 1004 c700 0244 0001 8966 085c 44c7 0006  >....D...f.\..D..<
0000176 6670 c031 4489 6604 4489 b40c cd42 7213  >pf1..D.f.D..B..r<
0000192 bb05 7000 7deb 08b4 13cd 0a73 c2f6 0f80  >...p.}....s.....<
0000208 f084 e900 008d 05be c67c ff44 6600 c031  >........|.D..f1.<
0000224 f088 6640 4489 3104 88d2 c1ca 02e2 e888  >..@f.D.1........<
0000240 f488 8940 0844 c031 d088 e8c0 6602 0489  >..@.D.1......f..<
0000256 a166 7c44 3166 66d2 34f7 5488 660a d231  >f.D|f1.f.4.T.f1.<
0000272 f766 0474 5488 890b 0c44 443b 7d08 8a3c  >f.t..T..D.;D.}<.<
0000288 0d54 e2c0 8a06 0a4c c1fe d108 6c8a 5a0c  >T.....L......l.Z<
0000304 748a bb0b 7000 c38e db31 01b8 cd02 7213  >.t...p..1......r<
0000320 8c2a 8ec3 4806 607c b91e 0100 db8e f631  >*....H|`......1.<
0000336 ff31 f3fc 1fa5 ff61 4226 be7c 7d7f 40e8  >1.....a.&B|..}.@<
0000352 eb00 be0e 7d84 38e8 eb00 be06 7d8e 30e8  >.....}.8.....}.0<
0000368 be00 7d93 2ae8 eb00 47fe 5552 2042 4700  >...}.*...GRUB .G<
0000384 6f65 006d 6148 6472 4420 7369 006b 6552  >eom.Hard Disk.Re<
0000400 6461 2000 7245 6f72 0072 01bb b400 cd0e  >ad. Error.......<
0000416 ac10 003c f475 00c3 0000 0000 0000 0000  >..<.u...........<
0000432 0000 0000 0000 0000 099b 000b 0000 2000  >............... <
0000448 0021 4b82 0a81 0800 0000 0000 0080 4b80  >!..K...........K<
0000464 0a82 fe83 ffff 0800 0080 f800 047f 0000  >................<
0000480 0000 0000 0000 0000 0000 0000 0000 0000  >................<
0000496 0000 0000 0000 0000 0000 0000 0000 aa55  >..............U.<
0000512

When i know where to look at, I prefer od, as it lets you see the position in decimal bytes (first column, compare to previous output). This helps A LOT when using dd input where you skip-ed the first N blocks / sectors, since you can read wether you are looking at the part which you wanted to examine.

Some notes on its parameters:

  • -A d = show position in decimals. Use x for hexadecimal.
  • -t x1z = show hex output, double-byte-wise (x2, use x1 for single-byte-wise output), z shows the data in the rightmost column.
  • --endian=little = choose endianness. Since this is a x86_64 intel cpu, we need little endian. I could have omitted this, but didn't for illustrating.
  • While still searching on the disk for data (using the dd-to-od from above, but piped ot less, using the -v flag with od is pratical, as it will condense lines consisting only of zeroes, showing only an asterisk.

Now that the basics are covered here, the rest should be easy, so only some more notes along the way:

  • No need to specify the blocksize with dd, since its 512 bytes by default.
  • Change it, in case you know how many bytes you want to jump around and want to be able to calculate easier (use bs=1024 and count=20 to read 20KiB from disk, instead of thinking it's count=40 what you need.
  • Using dd with the skip option, jumps so-and-so many blocks forward. For the sake of brevity, assume that both blocks and sectors are 512 bytes long. Remeber the output of parted in sectors from above?
  • Do use losetup, when kpartx fails.

result

I was able to discern that the backup was indeed broken, as there was not magic ext4 number present anywhere.

0xEF53 was nowhere to be found at the 0x38 offset after the initial padding of 1024 bytes in front of the start of the filesystem. Such info can be found here, for example.

At least I got some training with that stuff, been a while I got around to do so.

This blog covers .csv, .htaccess, .pfx, .vmx, /etc/crypttab, /etc/network/interfaces, /etc/sudoers, /proc, 10.04, 14.04, AS, ASA, ControlPanel, DS1054Z, GPT, HWR, Hyper-V, IPSEC, KVM, LSI, LVM, LXC, MBR, MTU, MegaCli, PHP, PKI, R, RAID, S.M.A.R.T., SNMP, SSD, SSL, TLS, TRIM, VEEAM, VMware, VServer, VirtualBox, Virtuozzo, XenServer, acpi, adaptec, algorithm, ansible, apache, apachebench, apple, arcconf, arch, architecture, areca, arping, asa, asdm, autoconf, awk, backup, bandit, bar, bash, benchmarking, binding, bitrate, blackarmor, blockdev, blowfish, bochs, bond, bonding, booknotes, bootable, bsd, btrfs, buffer, c-states, cache, caching, ccl, centos, certificate, certtool, cgdisk, cheatsheet, chrome, chroot, cisco, clamav, cli, clp, clush, cluster, coleslaw, colorscheme, common lisp, console, container, containers, controller, cron, cryptsetup, csync2, cu, cups, cygwin, d-states, database, date, db2, dcfldd, dcim, dd, debian, debug, debugger, debugging, decimal, desktop, df, dhclient, dhcp, diff, dig, display manager, dm-crypt, dmesg, dmidecode, dns, docker, dos, drivers, dtrace, dtrace4linux, du, dynamictracing, e2fsck, eBPF, ebook, efi, egrep, emacs, encoding, env, error, ess, esx, esxcli, esxi, ethtool, evil, expect, exportfs, factory reset, factory_reset, factoryreset, fail2ban, fbsd, fdisk, fedora, file, filesystem, find, fio, firewall, firmware, fish, flashrom, forensics, free, freebsd, freedos, fritzbox, fsck, fstrim, ftp, ftps, g-states, gentoo, ghostscript, git, git-filter-branch, github, gitolite, global, gnutls, gradle, grep, grml, grub, grub2, guacamole, hardware, haskell, hdd, hdparm, hellowor, hex, hexdump, history, howto, htop, htpasswd, http, httpd, https, i3, icmp, ifenslave, iftop, iis, imagemagick, imap, imaps, init, innoDB, innodb, inodes, intel, ioncube, ios, iostat, ip, iperf, iphone, ipmi, ipmitool, iproute2, ipsec, iptables, ipv6, irc, irssi, iw, iwconfig, iwlist, iwlwifi, jailbreak, jails, java, javascript, javaws, js, juniper, junit, kali, kde, kemp, kernel, keyremap, kill, kpartx, krypton, lacp, lamp, languages, ldap, ldapsearch, less, leviathan, liero, lightning, links, linux, linuxin3months, lisp, list, livedisk, lmctfy, loadbalancing, locale, log, logrotate, looback, loopback, losetup, lsblk, lsi, lsof, lsusb, lsyncd, luks, lvextend, lvm, lvm2, lvreduce, lxc, lxde, macbook, macro, magento, mailclient, mailing, mailq, manpages, markdown, mbr, mdadm, megacli, micro sd, microsoft, minicom, mkfs, mktemp, mod_pagespeed, mod_proxy, modbus, modprobe, mount, mouse, movement, mpstat, multitasking, myISAM, mysql, mysql 5.7, mysql workbench, mysqlcheck, mysqldump, nagios, nas, nat, nc, netfilter, networking, nfs, nginx, nmap, nocaps, nodejs, numberingsystem, numbers, od, onyx, opcode-cache, openVZ, openlierox, openssl, openvpn, openvswitch, openwrt, oracle linux, org-mode, os, oscilloscope, overview, parallel, parameter expansion, parted, partitioning, passwd, patch, pct, pdf, performance, pfsense, php, php7, phpmyadmin, pi, pidgin, pidstat, pins, pkill, plesk, plugin, posix, postfix, postfixadmin, postgres, postgresql, poudriere, powershell, preview, profiling, prompt, proxmox, ps, puppet, pv, pveam, pvecm, pvesm, pvresize, python, qemu, qemu-img, qm, qmrestore, quicklisp, r, racktables, raid, raspberry pi, raspberrypi, raspbian, rbpi, rdp, redhat, redirect, registry, requirements, resize2fs, rewrite, rewrites, rhel, rigol, roccat, routing, rs0485, rs232, rsync, s-states, s_client, samba, sar, sata, sbcl, scite, scp, screen, scripting, seafile, seagate, security, sed, serial, serial port, setup, sftp, sg300, shell, shopware, shortcuts, showmount, signals, slattach, slip, slow-query-log, smbclient, snmpget, snmpwalk, software RAID, software raid, softwareraid, sophos, spacemacs, spam, specification, speedport, spi, sqlite, squid, ssd, ssh, ssh-add, sshd, ssl, stats, storage, strace, stronswan, su, submodules, subzone, sudo, sudoers, sup, swaks, swap, switch, switching, synaptics, synergy, sysfs, systemd, systemtap, tar, tcpdump, tcsh, tee, telnet, terminal, terminator, testdisk, testing, throughput, tmux, todo, tomcat, top, tput, trafficshaping, ttl, tuning, tunnel, tunneling, typo3, uboot, ubuntu, ubuntu 16.04, udev, uefi, ulimit, uname, unetbootin, unit testing, upstart, uptime, usb, usbstick, utf8, utm, utm 220, ux305, vcs, vgchange, vim, vimdiff, virtualbox, virtualization, visual studio code, vlan, vmstat, vmware, vnc, vncviewer, voltage, vpn, vsphere, vzdump, w, w701, wakeonlan, wargames, web, webdav, weechat, wget, whois, wicd, wifi, windowmanager, windows, wine, wireshark, wpa, wpa_passphrase, wpa_supplicant, x11vnc, x2x, xfce, xfreerdp, xmodem, xterm, xxd, yum, zones, zsh


Unless otherwise credited all material Creative Commons License by sjas