Posts tagged lsblk

On blockdevices and their blocksizes

posted on 2017-04-27 06:00

Disclaimer: This is a draft and waiting to be verified once I get around plugging all old disks in. As of now, it mainly tries to specify the problem of disk sector size being report differently by linux on OS level and from hdparm via the firmware, and wether dd can really specify arbitrary blocksizes.

Getting it from the kernel via /sys/dev/, lsblk, or fdisk did show different sizes from hdparm when tested.

It's the only seemingly correct way to determine the sector sizes, since it queries the firmware on disk directly. Getting it from the kernel via /sys/dev/ (lsblk or fdisk should use that same info) did show different sizes from hdparm -I when tested.

hdparm sports -i and -I:

-i     Display the identification info which the kernel drivers (IDE, libata) have stored from boot/config
       uration time.  This may differ from the current  information  obtainable  directly  from  the  drive
       itself with the -I option.  The data returned may or may not be current, depending on activity since
       booting the system.  For a more detailed interpretation of the  identification  info,  refer  to  AT
       Attachment  Interface  for Disk Drives, ANSI ASC X3T9.2 working draft, revision 4a, April 19/93, and
       later editions.

- I     Request identification info directly from the drive, which is displayed in  a  new  expanded  format
        with considerably more detail than with the older -i option.

An overview on the information hdparm gets, can be found here. I did not find the actual block size in there, so my guess is the sizes are calculated from the information found in the drives registers.

-i shows pretty much a dump of the firware registers it seems, while -I seems to do additional calculations to present more information in human-friendly form.

To overwrite the last 1mb of a disk via dd, on a disk with 512b sectors this will do:

dd bs=512 if=/dev/zero of=/dev/sda count=2048 seek=$((`blockdev --getsz /dev/sda` - 2048))

bs is in bytes, count and seek are blockcounts.

Questions for the future are:

  • does this work reliably with devices having 4k sectors?
  • what does blockdev --getbsz print in these cases?
  • does it use the size reported by the kernel?
  • do mismatches come from different physical/logical blocksizes reported by the disk?
  • were the mismatches I witnessed really in between disk and OS, or just within the firmware?

Mind you, the numbers I saw earlier (2 years?) were with a dozen directly connected IDE and SATA drives, not with a raid controller in between, on disks of differenent manufacturers and sizes I had left over, and some showed in-kernel 512b even though they were 4k ones.

This really intrigues me, since the linux tools all seem to 'just work'. But using dd in the past I ran into problems were I simply could not manually find data during forensic work on disks where its offset was calculated to be.

luks and lvm and partitioning and filesystem from the shell

posted on 2016-08-24 20:58

Don't overwrite your devices via cp. But we've all been there, done that.

If you don't want to reinstall 'just because', an idea might be to use testdisk depending on what you did.

Getting nice partition layout I tend to use parted (see below), for creating partitions cgdisk (for GPT stuff) or cfdisk (for MBR creation only IIRC) are decent choices.

Back on topic.

preparation

Partitions were still present in my cause, so no need create them anew.

If you have to, do parted /dev/sda p and parted /dev/sdX u b p and use your phone to make photos, in case you have to redo something.

luks

Create and open the cryptocontainer to hold the complete partition, wherein the LVM and your filesystems will lie.

cryptsetup --cipher=aes-xts-plain64 luksFormat /dev/sdXN --force-password
cryptsetup open /dev/sdXN sdXN_crypt

Did you really type an uppercased YES when you were promted? The password you were prompted for is the one you will have to enter in the future.

In case you did something wrong:

cryptsetup close
cryptsetup erase /dev/sdaX

Then start by recreating the container. Did you really type an uppercased YES when you were promted?

lvm2

After the crypto device was opened, you can reference it through the device mapper. Now create the physical volume (PV), volume group (VG) and logical volumes (LV's) where your system will be installed later on:

pvcreate /dev/mapper/sdXN_crypt
vgcreate `hostname` /dev/mapper/sdXN_crypt
lvcreate -L 2G -n swap `hostname`
lvcreate -l 100%FREE -n root `hostname`

Here is a catch: I did not have to recreate a separate /boot partition, as I already had one. If you don't create one first. It has to be located outside the crypto container, else you won't be able too boot after your installation.

If something went wrong, here's how to delete things, too. Choose what you need in particular:

pvremove /dev/sdXN_crypt
vgremove `hostname`
lvremove /dev/`hostname`/<LVname>

filesystems and swap

Create swap:

mkswap /dev/mapper/`hostname`-swap

Create root filesystem:

mkfs -t ext4 /dev/mapper/`hostname`-root

This is pretty much it. From here on you can chroot or do whatever else you want.

Maybe you only want the container for data but for installing a system on there. In that case not calling the LV 'root' and omitting the swap partition up there would have been a wise choice.

linux: resize vm to full disk size

posted on 2016-06-11 10:38

fter resizing the the virtual harddisk of your virtual machine, several other steps are needed so you can utilize this additional space within the VM. This will only talk about increased sizing, which will usually just work. Unlike with downsizing, which are the same steps just in reverse order, but where you can easily kill your currently still running system. Handle downsizing with very, very much care.

This guide assumes you have a single partition, which is used by LVM, where in you have your filesystem(s) in different logical volumes.

resize the partition

I have a vm with hostname called 'test', which has a single disk (/dev/vda), with a single partition (/dev/vda1), which is used by lvm. LVM volume groups are usually called like the hostname (best approach I know of, so here test), and the logical volumes what they are used for (root, swap), or where they are mounted (i.e. var_lib_vz, not shown here).

root uses ext4 as file system.

Initially the disk size was 50G and was increased to 500G.

After the disk size was increased, you can see the available space on the device:

root@test:~# lsblk
NAME             MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sr0               11:0    1 1024M  0 rom
vda              253:0    0  500G  0 disk
+-vda1           253:1    0   50G  0 part
  +-test-root 252:0    0 14.3G  0 lvm  /
  +-test-swap 252:1    0  976M  0 lvm  [SWAP]

Use a partition manager of your choice (fdisk or cfdisk for disks with an MBR, gdisk or cgdisk for disks using a GPT, or parted if you know what you are doing.), delete your partition. Recreate it with the maximum size, reboot.

Then it should look like this, with adjusted partition size:

root@test:~# lsblk
NAME             MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sr0               11:0    1 1024M  0 rom
vda              253:0    0  500G  0 disk
+-vda1           253:1    0  500G  0 part
  +-test-root 252:0    0   49G  0 lvm  /
  +-test-swap 252:1    0  976M  0 lvm  [SWAP]

resize PV, LV, file system

First make LVM format the addition free space: (It will 'partition' it so it can work with it, effectively splitting it into junks of like 4MB if I recall correctly.)

root@test:~# pvresize /dev/vda1
  Physical volume "/dev/vda1" changed
  1 physical volume(s) resized / 0 physical volume(s) not resized

Since the PV was already a member of the VG, no need to extend the VG.

Now for the actual volume:

root@test:~# lvextend -L 499G /dev/test/root
  Size of logical volume test/root changed from 49.04 GiB (12555 extents) to 499.00 GiB (127744 extents).
  Logical volume root successfully resized.

Here I specified it to be resized to 499GB. If I wanted to just use all available space, I'd do:

lvextend -l +100%FREE /dev/mapper/test-root

root@test:~# lvextend -l +100%FREE /dev/mapper/test-root
  Size of logical volume test/root changed from 450.00 GiB (115200 extents) to 499.04 GiB (127755 extents).
  Logical volume root successfully resized.

The -L is just easier to remember.

Lastly, resize the used filesystem:

root@test:~# resize2fs -p /dev/mapper/test-root
resize2fs 1.42.13 (17-May-2015)
Dateisystem bei /dev/mapper/test-root ist auf / eingehaengt; Online-Groessenaenderung ist
erforderlich
old_desc_blocks = 1, new_desc_blocks = 32
Das Dateisystem auf /dev/mapper/test-root is nun 130821120 (4k) Bloecke lang.

Verify it:

root@test:~# df -h
Dateisystem              Groesse Benutzt Verf. Verw% Eingehaengt auf
udev                      983M       0  983M    0% /dev
tmpfs                     201M    3.2M  197M    2% /run
/dev/mapper/test-root  492G    2.3G  469G    1% /
tmpfs                    1001M       0 1001M    0% /dev/shm
tmpfs                     5.0M       0  5.0M    0% /run/lock
tmpfs                    1001M       0 1001M    0% /sys/fs/cgroup

Linux: show all block devices with lsblk

posted on 2014-11-05 00:01:23

To see all currently connected devices like HDD's, SSD's, CD-Rom's and USB sticks, try lsblk.

Usually it looks like this:

sjas@mb:~/ISO/UBCD$ lsblk
NAME                         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                            8:0    0  29.8G  0 disk
|-sda1                         8:1    0   487M  0 part /boot/efi
|-sda2                         8:2    0  25.6G  0 part /
`-sda3                         8:3    0   3.8G  0 part [SWAP]
sr0                           11:0    1 589.2M  0 rom
sdb                            8:16   0 596.2G  0 disk
|-sdb1                         8:17   0   200M  0 part
|-sdb2                         8:18   0   500M  0 part
`-sdb3                         8:19   0 595.5G  0 part
  |-fedora_debra-root (dm-0) 254:0    0    50G  0 lvm
  |-fedora_debra-home (dm-1) 254:1    0   542G  0 lvm
  `-fedora_debra-swap (dm-2) 254:2    0   3.5G  0 lvm
sjas@mb:~/ISO/UBCD$

For a better overview, try a better selection of -o flags. Heres an overview on the possible options on an arbitrary system:

[jl@jerrylee ~]% \lsblk --help | \grep Available -A999 | sed -e '1d' -e '$d' | sed '$d'
        NAME  device name
       KNAME  internal kernel device name
     MAJ:MIN  major:minor device number
      FSTYPE  filesystem type
  MOUNTPOINT  where the device is mounted
       LABEL  filesystem LABEL
        UUID  filesystem UUID
   PARTLABEL  partition LABEL
    PARTUUID  partition UUID
          RA  read-ahead of the device
          RO  read-only device
          RM  removable device
       MODEL  device identifier
      SERIAL  disk serial number
        SIZE  size of the device
       STATE  state of the device
       OWNER  user name
       GROUP  group name
        MODE  device node permissions
   ALIGNMENT  alignment offset
      MIN-IO  minimum I/O size
      OPT-IO  optimal I/O size
     PHY-SEC  physical sector size
     LOG-SEC  logical sector size
        ROTA  rotational device
       SCHED  I/O scheduler name
     RQ-SIZE  request queue size
        TYPE  device type
    DISC-ALN  discard alignment offset
   DISC-GRAN  discard granularity
    DISC-MAX  discard max bytes
   DISC-ZERO  discard zeroes data
       WSAME  write same max bytes
         WWN  unique storage identifier
        RAND  adds randomness
      PKNAME  internal parent kernel device name
        HCTL  Host:Channel:Target:Lun for SCSI
        TRAN  device transport type
         REV  device revision
      VENDOR  device vendor

You can of course take this listing and try it directly:

\lsblk -o$(\lsblk --help | \grep Available -A999 | sed -e '1d' -e '$d' | sed '$d' | awk '{print $1}' | tr '\n' ',' | sed 's/,$//')

But if you do not have two widescreen monitors and a PTY shell drawn across it, you wont be recognizing much. Just for the record, on a Kali LIVE the above command won't even show all the output but jsut die gracefully without even showing an error.

So you might try this:

root@mb:/home/sjas/ISO/UBCD# lsblk -i -o name,label,mountpoint,fstype,model,size,type,state,uuid
NAME                         LABEL    MOUNTPOINT FSTYPE      MODEL              SIZE TYPE STATE   UUID
sda                                                          SSDSA2SH032G1GN   29.8G disk running
|-sda1                                /boot/efi  vfat                           487M part         5604-FDAF
|-sda2                                /          ext4                          25.6G part         ded0e7a8-23af-4deb-9b9d-9d63a26904aa
`-sda3                                [SWAP]     swap                           3.8G part         b022846d-e4b5-475b-b087-c4d5b486601f
sr0                          UBCD532             iso9660     DVDRW  GS21N     589.2M rom  running
sdb                                                          Name             596.2G disk running
|-sdb1                       untitled            hfsplus                        200M part         0fdc7456-f171-3490-9d41-671b43d70db3
|-sdb2                                           ext4                           500M part         66a8ed1c-e56e-4707-bbd5-15bcde2fa5a0
`-sdb3                                           LVM2_member                  595.5G part         BIC2hD-zS3w-yvtC-oNEG-yec1-4Q7h-qb4gwN
  |-fedora_debra-root (dm-0)                     ext4                            50G lvm  running e35f5406-0cc0-4646-86f8-c4031005580a
  |-fedora_debra-home (dm-1)                     ext4                           542G lvm  running 8bc9c1a4-a5c1-4e1b-9dfc-7aeb6437c708
  `-fedora_debra-swap (dm-2)                     swap                           3.5G lvm  running 2ddddb46-8b6d-4fb4-ae19-6390e1015b76
root@mb:/home/sjas/ISO/UBCD#

Of course, I have lsblk -o name,label,mountpoint,fstype,model,size,type,state,uuid aliased in my .bashrc:

alias lsblk='lsblk -o name,label,mountpoint,fstype,model,size,type,state,uuid'

UPDATE:

lsblk -i -o kname,mountpoint,fstype,size,maj:min,rm,name,state,rota,ro,type,label,model,serial

is what i stick with.

parted

posted on 2014-10-11 15:27:56

Sometimes you have to partition disks by hand. For some people, parted is the weapon of choice.

If so, keep in mind you only use it for partitioning. Filesystems are not implemented and parted may tell you so, too. Besides you won't get filesystems created by it. Just use the mkfs.xxx tools at hand. It's just sad that half finished programs are distributed... banana software - it matures in the works of the customer. :(

Back to the posts intention: There are two ways to use parted. Either via the interactive parted-shell, or directly from the commandline.

Most needed commands in the shell might be:

a   = align-check = check alignment to sectors (min or opt as params)
p   = print       = show info for chosen disk
sel = select      = choose disk (i.e. /dev/sdb)
u   = unit        = measuring of sizes (i.e. %,MB,GB,...)
mkl = mklabel     = create disk label (i.e. gpt) (mktable = mkt = the same)
mkp = mkpart      = create partitions
rm  = remove      = delete a partition
q   = quit        = exit parted

An overview on the possible sizes you'd most likely use:

MiB  Mebibyte (1048576 bytes)
GiB  Gibibyte (1073741824 bytes)
MB   Megabyte (1000000 bytes)
GB   Gigabyte (1000000000 bytes)
%    procentual (between 0% and 100%) 
s    sectors (logical sector size)

So usually you would want MiB and GiB, I guess.

Sadly, I haven't found out yet how to optimally align the partition onto the harddisks (so it matches it's sectors best). For this I actually used to use parted from the regular bash shell:

parted /dev/sdb mklabel gpt
parted -a optimal /dev/sdb unit gb mkpart primary ext4 2 100%

Nowadays I mostly use parted interactive with procentual parameters.

But for when I don't, here two example workflows

# create a single MBR  partition with an ext4 on the usb stick /dev/sdb
lsblk -f                ## make sure the stick is really on /dev/sdb
parted /dev/sdb p           ## check whats already on the stick
parted /dev/sdb mkl ms y        ## create MBR not GPT
parted /dev/sdb mkp p 0% 100%   ## single primary partition, percentages are used for proper alignment
mkfs.ext4 /dev/sdb1         ## create filesystem at last
parted /dev/sdb a opt 1     ## check alignment to be optimal for partition 1

# create the same, but with gpt
lsblk -f
parted /dev/sdb p           
parted /dev/sdb mkl g y     ## create GPT
parted /dev/sdb mkp asdf 0% 100%    ## creates partition with name 'asdf', a name must be specified, ext4 already on it

It seems that currently creating the filesystem only works when using GPT's, and only ext4's can be created. But somewhere I heard that it was officially said that you should use mkfs anyway, and not let parted create filesystems.

This blog covers .csv, .htaccess, .pfx, .vmx, /etc/crypttab, /etc/network/interfaces, /etc/sudoers, /proc, 10.04, 14.04, 16.04, AS, ASA, ControlPanel, DS1054Z, GPT, HWR, Hyper-V, IPSEC, KVM, LSI, LVM, LXC, MBR, MTU, MegaCli, PHP, PKI, PS1, R, RAID, S.M.A.R.T., SNMP, SSD, SSL, TLS, TRIM, VEEAM, VMware, VServer, VirtualBox, Virtuozzo, XenServer, acpi, adaptec, algorithm, ansible, apache, apache2.4, apachebench, apple, applet, arcconf, arch, architecture, areca, arping, asa, asdm, autoconf, awk, backup, bandit, bar, bash, benchmarking, binding, bitrate, blackarmor, blockdev, blowfish, bochs, bond, bonding, booknotes, bootable, bsd, btrfs, buffer, c-states, cache, caching, ccl, centos, certificate, certtool, cgdisk, cheatsheet, chrome, chroot, cisco, clamav, cli, clp, clush, cluster, cmd, coleslaw, colorscheme, common lisp, configuration management, console, container, containers, controller, cron, cryptsetup, csync2, cu, cups, cygwin, d-states, database, date, db2, dcfldd, dcim, dd, debian, debug, debugger, debugging, decimal, desktop, df, dhclient, dhcp, diff, dig, display manager, dm-crypt, dmesg, dmidecode, dns, docker, dos, drivers, dtrace, dtrace4linux, du, dynamictracing, e2fsck, eBPF, ebook, efi, egrep, emacs, encoding, env, error, ess, esx, esxcli, esxi, ethtool, evil, expect, exportfs, factory reset, factory_reset, factoryreset, fail2ban, fakeroot, fbsd, fdisk, fedora, file, files, filesystem, find, fio, firewall, firmware, fish, flashrom, forensics, free, freebsd, freedos, fritzbox, fsck, fstrim, ftp, ftps, g-states, gentoo, ghostscript, git, git-filter-branch, gitbucket, github, gitolite, global, gnutls, gradle, grep, grml, grub, grub2, guacamole, hardware, haskell, hdd, hdparm, hellowor, hex, hexdump, history, howto, htop, htpasswd, http, httpd, https, i3, icmp, ifenslave, iftop, iis, imagemagick, imap, imaps, init, innoDB, innodb, inodes, intel, ioncube, ios, iostat, ip, iperf, iphone, ipmi, ipmitool, iproute2, ipsec, iptables, ipv6, irc, irssi, iw, iwconfig, iwlist, iwlwifi, jailbreak, jails, java, javascript, javaws, js, juniper, junit, kali, kde, kemp, kernel, keyremap, kill, kpartx, krypton, lacp, lamp, languages, ldap, ldapsearch, less, leviathan, liero, lightning, links, linux, linuxin3months, lisp, list, livedisk, lmctfy, loadbalancing, locale, log, logrotate, looback, loopback, losetup, lsblk, lsi, lsof, lsusb, lsyncd, luks, lvextend, lvm, lvm2, lvreduce, lxc, lxde, macbook, macro, magento, mailclient, mailing, mailq, make-jpkg, manpages, markdown, mbr, mdadm, megacli, micro sd, microsoft, minicom, mkfs, mktemp, mod_pagespeed, mod_proxy, modbus, modprobe, mount, mouse, movement, mpstat, multitasking, myISAM, mysql, mysql 5.7, mysql workbench, mysqlcheck, mysqldump, nagios, nas, nat, nc, netfilter, networking, nfs, nginx, nmap, nocaps, nodejs, numberingsystem, numbers, od, onyx, opcode-cache, openVZ, openlierox, openssl, openvpn, openvswitch, openwrt, oracle linux, org-mode, os, oscilloscope, overview, parallel, parameter expansion, parted, partitioning, passwd, patch, pct, pdf, performance, pfsense, php, php7, phpmyadmin, pi, pidgin, pidstat, pins, pkill, plasma, plesk, plugin, posix, postfix, postfixadmin, postgres, postgresql, poudriere, powershell, preview, profiling, prompt, proxmox, ps, puppet, pv, pveam, pvecm, pvesm, pvresize, python, python3, qemu, qemu-img, qm, qmrestore, quicklisp, quickshare, r, racktables, raid, raspberry pi, raspberrypi, raspbian, rbpi, rdp, redhat, redirect, registry, requirements, resize2fs, rewrite, rewrites, rhel, rigol, roccat, routing, rs0485, rs232, rsync, s-states, s_client, samba, sar, sata, sbcl, scite, scp, screen, scripting, seafile, seagate, security, sed, serial, serial port, setup, sftp, sg300, shell, shopware, shortcuts, showmount, signals, slattach, slip, slow-query-log, smbclient, snmpget, snmpwalk, software RAID, software raid, softwareraid, sophos, spacemacs, spam, specification, speedport, spi, sqlite, squid, ssd, ssh, ssh-add, sshd, ssl, stats, storage, strace, stronswan, su, submodules, subzone, sudo, sudoers, sup, swaks, swap, switch, switching, synaptics, synergy, sysfs, systemd, systemtap, tar, tcpdump, tcsh, tee, telnet, terminal, terminator, testdisk, testing, throughput, tmux, todo, tomcat, top, tput, trafficshaping, ttl, tuning, tunnel, tunneling, typo3, uboot, ubuntu, ubuntu 16.04, ubuntu16.04, udev, uefi, ulimit, uname, unetbootin, unit testing, upstart, uptime, usb, usbstick, utf8, utm, utm 220, ux305, vcs, vgchange, vim, vimdiff, virtualbox, virtualization, visual studio code, vlan, vmstat, vmware, vnc, vncviewer, voltage, vpn, vsphere, vzdump, w, w701, wakeonlan, wargames, web, webdav, weechat, wget, whois, wicd, wifi, windowmanager, windows, wine, wireshark, wpa, wpa_passphrase, wpa_supplicant, x11vnc, x2x, xfce, xfreerdp, xmodem, xterm, xxd, yum, zones, zsh


Unless otherwise credited all material Creative Commons License by sjas