Posts tagged hardware

On blockdevices and their blocksizes

posted on 2017-04-27 06:00

Disclaimer: This is a draft and waiting to be verified once I get around plugging all old disks in. As of now, it mainly tries to specify the problem of disk sector size being report differently by linux on OS level and from hdparm via the firmware, and wether dd can really specify arbitrary blocksizes.

Getting it from the kernel via /sys/dev/, lsblk, or fdisk did show different sizes from hdparm when tested.

It's the only seemingly correct way to determine the sector sizes, since it queries the firmware on disk directly. Getting it from the kernel via /sys/dev/ (lsblk or fdisk should use that same info) did show different sizes from hdparm -I when tested.

hdparm sports -i and -I:

-i     Display the identification info which the kernel drivers (IDE, libata) have stored from boot/configā€
       uration time.  This may differ from the current  information  obtainable  directly  from  the  drive
       itself with the -I option.  The data returned may or may not be current, depending on activity since
       booting the system.  For a more detailed interpretation of the  identification  info,  refer  to  AT
       Attachment  Interface  for Disk Drives, ANSI ASC X3T9.2 working draft, revision 4a, April 19/93, and
       later editions.

- I     Request identification info directly from the drive, which is displayed in  a  new  expanded  format
        with considerably more detail than with the older -i option.

An overview on the information hdparm gets, can be found here. I did not find the actual block size in there, so my guess is the sizes are calculated from the information found in the drives registers.

-i shows pretty much a dump of the firware registers it seems, while -I seems to do additional calculations to present more information in human-friendly form.

To overwrite the last 1mb of a disk via dd, on a disk with 512b sectors this will do:

dd bs=512 if=/dev/zero of=/dev/sda count=2048 seek=$((`blockdev --getsz /dev/sda` - 2048))

bs is in bytes, count and seek are blockcounts.

Questions for the future are:

  • does this work reliably with devices having 4k sectors?
  • what does blockdev --getbsz print in these cases?
  • does it use the size reported by the kernel?
  • do mismatches come from different physical/logical blocksizes reported by the disk?
  • were the mismatches I witnessed really in between disk and OS, or just within the firmware?

Mind you, the numbers I saw earlier (2 years?) were with a dozen directly connected IDE and SATA drives, not with a raid controller in between, on disks of differenent manufacturers and sizes I had left over, and some showed in-kernel 512b even though they were 4k ones.

This really intrigues me, since the linux tools all seem to 'just work'. But using dd in the past I ran into problems were I simply could not manually find data during forensic work on disks where its offset was calculated to be.

write-back vs. write-through caching

posted on 2014-09-29 12:15:40

When putting hardware RAID controller to use, you usually have to choose between the option in the post title.

write-back

Once the hardware-RAID-controller (HWR) has the data cached, which the operating system sent, it tells the OS to go on. (The OS is told the disk write is finished.)

OS          --          HWR cache           --          HDD / SSD
1. send data to write
                        2. data is cached
                        3. OS is told to go on
4. OS goes on
                                                        5. data is written to disk,
                                                           when system load is low.

write-through

Only when the data has been written to disk, the OS is told so.

OS          --          HWR cache           --          HDD / SSD
1. send data to write
                        2. data is cached
                                                        3. data is written to disk
                        4. OS is told to go on
5. OS goes on

Test from the Proxmox wiki: (here)

Single SATA WD 400GB: 1360.17
3 x 15K rpm sas RAID5 with write-through: 159.03 (YES, only 159!)
same as above but with write-back enabled: 3133.45

TL;DR

Use write-back on controllers with a battery backup unit (BBU) or a flash memory, depending on which functionality is available.

Adaptec's lower series cannot be equipped with neither, the 5xxx series can have one of each (or even both?). The 6xxx series doesn't need a BBU.

Take note, that BBU's need attention, whereas flash-based solutions are maintenance-free.

With SSD's however, you can forget about all this and just use write-through, since it's fast enough.

This blog covers .csv, .htaccess, .pfx, .vmx, /etc/crypttab, /etc/network/interfaces, /etc/sudoers, /proc, 10.04, 14.04, AS, ASA, ControlPanel, DS1054Z, GPT, HWR, Hyper-V, IPSEC, KVM, LSI, LVM, LXC, MBR, MTU, MegaCli, PHP, PKI, R, RAID, S.M.A.R.T., SNMP, SSD, SSL, TLS, TRIM, VEEAM, VMware, VServer, VirtualBox, Virtuozzo, XenServer, acpi, adaptec, algorithm, ansible, apache, apachebench, apple, applet, arcconf, arch, architecture, areca, arping, asa, asdm, autoconf, awk, backup, bandit, bar, bash, benchmarking, binding, bitrate, blackarmor, blockdev, blowfish, bochs, bond, bonding, booknotes, bootable, bsd, btrfs, buffer, c-states, cache, caching, ccl, centos, certificate, certtool, cgdisk, cheatsheet, chrome, chroot, cisco, clamav, cli, clp, clush, cluster, coleslaw, colorscheme, common lisp, configuration management, console, container, containers, controller, cron, cryptsetup, csync2, cu, cups, cygwin, d-states, database, date, db2, dcfldd, dcim, dd, debian, debug, debugger, debugging, decimal, desktop, df, dhclient, dhcp, diff, dig, display manager, dm-crypt, dmesg, dmidecode, dns, docker, dos, drivers, dtrace, dtrace4linux, du, dynamictracing, e2fsck, eBPF, ebook, efi, egrep, emacs, encoding, env, error, ess, esx, esxcli, esxi, ethtool, evil, expect, exportfs, factory reset, factory_reset, factoryreset, fail2ban, fbsd, fdisk, fedora, file, filesystem, find, fio, firewall, firmware, fish, flashrom, forensics, free, freebsd, freedos, fritzbox, fsck, fstrim, ftp, ftps, g-states, gentoo, ghostscript, git, git-filter-branch, github, gitolite, global, gnutls, gradle, grep, grml, grub, grub2, guacamole, hardware, haskell, hdd, hdparm, hellowor, hex, hexdump, history, howto, htop, htpasswd, http, httpd, https, i3, icmp, ifenslave, iftop, iis, imagemagick, imap, imaps, init, innoDB, innodb, inodes, intel, ioncube, ios, iostat, ip, iperf, iphone, ipmi, ipmitool, iproute2, ipsec, iptables, ipv6, irc, irssi, iw, iwconfig, iwlist, iwlwifi, jailbreak, jails, java, javascript, javaws, js, juniper, junit, kali, kde, kemp, kernel, keyremap, kill, kpartx, krypton, lacp, lamp, languages, ldap, ldapsearch, less, leviathan, liero, lightning, links, linux, linuxin3months, lisp, list, livedisk, lmctfy, loadbalancing, locale, log, logrotate, looback, loopback, losetup, lsblk, lsi, lsof, lsusb, lsyncd, luks, lvextend, lvm, lvm2, lvreduce, lxc, lxde, macbook, macro, magento, mailclient, mailing, mailq, manpages, markdown, mbr, mdadm, megacli, micro sd, microsoft, minicom, mkfs, mktemp, mod_pagespeed, mod_proxy, modbus, modprobe, mount, mouse, movement, mpstat, multitasking, myISAM, mysql, mysql 5.7, mysql workbench, mysqlcheck, mysqldump, nagios, nas, nat, nc, netfilter, networking, nfs, nginx, nmap, nocaps, nodejs, numberingsystem, numbers, od, onyx, opcode-cache, openVZ, openlierox, openssl, openvpn, openvswitch, openwrt, oracle linux, org-mode, os, oscilloscope, overview, parallel, parameter expansion, parted, partitioning, passwd, patch, pct, pdf, performance, pfsense, php, php7, phpmyadmin, pi, pidgin, pidstat, pins, pkill, plasma, plesk, plugin, posix, postfix, postfixadmin, postgres, postgresql, poudriere, powershell, preview, profiling, prompt, proxmox, ps, puppet, pv, pveam, pvecm, pvesm, pvresize, python, qemu, qemu-img, qm, qmrestore, quicklisp, quickshare, r, racktables, raid, raspberry pi, raspberrypi, raspbian, rbpi, rdp, redhat, redirect, registry, requirements, resize2fs, rewrite, rewrites, rhel, rigol, roccat, routing, rs0485, rs232, rsync, s-states, s_client, samba, sar, sata, sbcl, scite, scp, screen, scripting, seafile, seagate, security, sed, serial, serial port, setup, sftp, sg300, shell, shopware, shortcuts, showmount, signals, slattach, slip, slow-query-log, smbclient, snmpget, snmpwalk, software RAID, software raid, softwareraid, sophos, spacemacs, spam, specification, speedport, spi, sqlite, squid, ssd, ssh, ssh-add, sshd, ssl, stats, storage, strace, stronswan, su, submodules, subzone, sudo, sudoers, sup, swaks, swap, switch, switching, synaptics, synergy, sysfs, systemd, systemtap, tar, tcpdump, tcsh, tee, telnet, terminal, terminator, testdisk, testing, throughput, tmux, todo, tomcat, top, tput, trafficshaping, ttl, tuning, tunnel, tunneling, typo3, uboot, ubuntu, ubuntu 16.04, udev, uefi, ulimit, uname, unetbootin, unit testing, upstart, uptime, usb, usbstick, utf8, utm, utm 220, ux305, vcs, vgchange, vim, vimdiff, virtualbox, virtualization, visual studio code, vlan, vmstat, vmware, vnc, vncviewer, voltage, vpn, vsphere, vzdump, w, w701, wakeonlan, wargames, web, webdav, weechat, wget, whois, wicd, wifi, windowmanager, windows, wine, wireshark, wpa, wpa_passphrase, wpa_supplicant, x11vnc, x2x, xfce, xfreerdp, xmodem, xterm, xxd, yum, zones, zsh


Unless otherwise credited all material Creative Commons License by sjas