Posts tagged zsh

mysql: output layout

posted on 2015-03-04 17:51:07

For big mysql tables with a lot of columns, the regular screen output is kind of hard to read at times.

Regularily you call queries like this:

select * from <tablename>;

There are several ways to fix this:

Within the client:

select * from <tablename>\G

At client startup:

## always use alternative output
mysql --vertical

## choose output depending on console width
mysql --auto-vertical-output

How does this look?

Regular:

mysql> show tables;
+---------------------------+
| Tables_in_mysql           |
+---------------------------+
| columns_priv              |
| db                        |
| event                     |
| func                      |
| general_log               |
| help_category             |
| help_keyword              |
| help_relation             |
| help_topic                |
| host                      |
| ndb_binlog_index          |
| plugin                    |
| proc                      |
| procs_priv                |
| proxies_priv              |
| servers                   |
| slow_log                  |
| tables_priv               |
| time_zone                 |
| time_zone_leap_second     |
| time_zone_name            |
| time_zone_transition      |
| time_zone_transition_type |
| user                      |
+---------------------------+
24 rows in set (0.00 sec)

Alternative:

mysql> show tables\G
*************************** 1. row ***************************
Tables_in_mysql: columns_priv
*************************** 2. row ***************************
Tables_in_mysql: db
*************************** 3. row ***************************
Tables_in_mysql: event
*************************** 4. row ***************************
Tables_in_mysql: func
*************************** 5. row ***************************
Tables_in_mysql: general_log
*************************** 6. row ***************************
Tables_in_mysql: help_category
*************************** 7. row ***************************
Tables_in_mysql: help_keyword
*************************** 8. row ***************************
Tables_in_mysql: help_relation
*************************** 9. row ***************************
Tables_in_mysql: help_topic
*************************** 10. row ***************************
Tables_in_mysql: host
*************************** 11. row ***************************
Tables_in_mysql: ndb_binlog_index
*************************** 12. row ***************************
Tables_in_mysql: plugin
*************************** 13. row ***************************
Tables_in_mysql: proc
*************************** 14. row ***************************
Tables_in_mysql: procs_priv
*************************** 15. row ***************************
Tables_in_mysql: proxies_priv
*************************** 16. row ***************************
Tables_in_mysql: servers
*************************** 17. row ***************************
Tables_in_mysql: slow_log
*************************** 18. row ***************************
Tables_in_mysql: tables_priv
*************************** 19. row ***************************
Tables_in_mysql: time_zone
*************************** 20. row ***************************
Tables_in_mysql: time_zone_leap_second
*************************** 21. row ***************************
Tables_in_mysql: time_zone_name
*************************** 22. row ***************************
Tables_in_mysql: time_zone_transition
*************************** 23. row ***************************
Tables_in_mysql: time_zone_transition_type
*************************** 24. row ***************************
Tables_in_mysql: user
24 rows in set (0.00 sec)

Linux: 'top' explained

posted on 2015-03-04 12:54:59

To get a fast overview on what is running on your linux box, use top. (If you want some fancy graphics, try htop, but it has less intuitive shortcuts and is not always installed.)

Sad thing is, at first you don't really know what you are doing. So some guidance:

start and sane defaults

After starting top, press: z, x, c. This will color top (z), show current sort column (x) and the full application path (c).

1 will show stats for all individual cpus.

If you have no idea, use h for getting the help shown.

If you have a newer version of top, V will also work:
This gives you a nice process-tree view.

d changes the update delay, which is at three seconds per default.

cpu stats explained

Straight from the manpage, the CPU statistics show the times spent in:

us = user mode
sy = system mode
ni = low priority user mode (nice)
id = idle task
wa = I/O waiting
hi = servicing IRQs
si = servicing soft IRQs
st = steal (time given to other DomU instances)

If you have low cpu and ram usage but the system is unresponsive, have a look at the wait times.

sorting and searching

Changing the sort column can be done via < and >.

Also available: (not shown in help)

N sort by PID
P sort by CPU usage
M sort by memory usage
T sort by time

R will reverse the output.

u to choose user name, show only this user's processes.

S for cululative time toggling.

columns

f will toggle a window in which you can choose the info fields to be shown. Pressing the character will toggle its state. (Shown or not shown.)

o also opens a window, in there you can reorder the columns. Press the character of the column you want to move, depending on it being upper- or lowercase it gets moved up and down.

manipulate tasks

These should be self-explanatory:

k kill task

r renice task

Linux 'less', advantages, disadvantages, keys, options

posted on 2014-12-01 07:46:36

Being the default pager on linux, and thus the tool you use to look manpages at usually, less is worth some more attention.

key points

Unlike editors or IDE's (vi, emacs, nano, eclipse), pagers (at least less) do not have to load a file completely into memory and thus are faster when displaying huge files. If you happen to think you will never have to open files bigger than some KB size, what about some error logs? (Once I saw a machine write like one additional GB per minute. In this case, you should maybe refrain from less and just use like tail -n1000.)

Also, compared to more, less can also scroll backwards. (!!!)

disadvantages

Pagers cannot edit text. That's what editors are for.

keys

system

q                       quit
h                       show help
= or ctrl-g             show current file name
r                       redraw screen
s                       save file (if input comes from a pipe, not a file)

v                       edit file with $VISUAL or $EDITOR

!<command>              execute <command> in $SHELL
!<mark><command>        pipe text contents between cursor and <mark> to <command>

movement

f or ctrl-f or space    move forward one page
b or ctrl-b             move backward one page 

g                       top of first page
G                       bottom of last page

<count>p                go to <count> percent line in text

d                       forward half a page
u                       backward half a page

m<char>                 mark line with <char>
'<char>                 jump to mark <char>o
''                      goto previous position 

search

/<pattern>              search forward for <pattern>
?<pattern>              search backward for <pattern>

n                       next match
N                       previous match

! or ^N                 prior to <pattern>, will search for non-matching lines
^K                      prior to <pattern>, just mark lines but don't move cursor
^R                      don't use regexes for searching

&<pattern>              SHOW ONLY MATCHES (about the best less command ever)

Especially the | hotkey might be interesting.

To pipe the complete buffer content into a file, do this:

1. g (go to top of file)
2. | (start pipe)
3. $ (pipe until the end of buffer)
4. tee [name of logfile].log

Afterwards you should have a new file. This works both with piped input as well as opened files.

options

startup options

All options with dashes can be used while running less, or as startup commands.

I.e.

+F                      same as 'tail -f', but with less
+/<pattern>             open file at <pattern>

+ is needed during startup, from within less its not needed except when you want to reset a option to its default value.

search options

-A                      search starts after target line
-g                      highlight last search result
-G                      highlight search results
-I                      completely case insensitive searching
-i                      smartcase: case-insensitive if search string contains no upper case
-J                      show status column (to mark lines with search results)
                        left of the the text, lines with matches are marked.
F                       'Waiting for data... (interrupt to abort)' (means ^C)
                        This is basically a 'tail -f' on stereoids!

system options

preface:

`-` prior sets / changes the option
`_` just shows it's current state

-e                      quit at EOF
-M                      toggle long prompt (filename, lines, line %)
-m                      toggle medium prompt (line %)
-N                      show line numbers
-Q                      quiet all terminal bells (!!!)
-R                      output raw control chars = SHOW COLORS
-s                      squeeze multiple blank lines into one
-S                      don't wrap long lines

-P                      define custom promtps
                        See last section here about further information.

custom prompts

   %bX      Replaced by the byte offset into the current input file.   
            The  b  is followed by a single character (shown as X above) 
            which specifies the line whose byte offset is to be used.  
            If the character is a "t", the byte  offset of the top line in 
            the display is used, an "m" means use the middle line, a "b" 
            means use the bottom line, a "B" means use the line  just  after  
            the  bottom line, and a "j" means use the "target" line, 
            as specified by the -j option.

   %B       Replaced by the size of the current input file.

   %c       Replaced by the column number of the text appearing in the 
            first column of the screen.

   %dX      Replaced by the page number of a line in the input file.  
            The line to be used is determined by the X, as with the %b option.

   %D       Replaced by the number of pages in the input file, or quivalently, 
            the page number of the last line in the input file.

   %E       Replaced by the name of the editor (from the VISUAL 
            environment variable, or the EDITOR environment variable 
            if VISUAL is  not  defined).
            See the discussion of the LESSEDIT feature below.

   %f       Replaced by the name of the current input file.

   %F       Replaced by the last component of the name of the current input file.

   %i       Replaced by the index of the current file in the list of input files.

   %lX      Replaced by the line number of a line in the input file.  
            The line to be used is determined by the X, as with the %b option.

   %L       Replaced by the line number of the last line in the input file.

   %m       Replaced by the total number of input files.

   %pX      Replaced by the percent into the current input file,  
            based on byte offsets.  
            The line used is determined by the X as with the %b option.

   %PX      Replaced  by  the  percent into the current input file, 
            based on line numbers.   
            The line used is determined by the X as with the %b option.

   %s       Same as %B.

   %t       Causes any trailing spaces to be removed.  
            Usually used at the end of the string, but may appear anywhere.

   %x       Replaced by the name of the next input file in the list.

Linux: proper tempfiles

posted on 2014-11-20 10:24:43

mktemp creates randomly named files, a recurringly needed appliance.

create a tempfile and save name to a variable

VARIABLE_NAME=`mktemp`

(That was rather easy, wasn't it?)

Querying dd progress

posted on 2014-11-16 17:44:33

UPDATE: use pkill instead of kill: pkill -usr1 dd is all you need.


Usually dd will only show information about the transfer it did, AFTER its completion.

Or try a second shell, and sending a USR1 signal to the dd process.

First, lets startd a demo dd process:

[sjas@mb ~]$ dd if=/dev/random of=/dev/null

Then we need to find out the process id of this dd process. For this you can use pgrep, but i prefer grepping ps auxf:

[sjas@mb ~]$ ps auxf | grep dd
2:root         2  0.0  0.0      0     0 ?        S    06:25   0:00 [kthreadd]
91:sjas      3351  0.0  0.0  30588  1704 ?        Ss   06:25   0:01 /usr/bin/dbus-daemon --fork --print-pid 5 --print-address 7 --session
134:sjas      6501  0.0  0.0  31980  3496 pts/1    S+   17:43   0:00          \_ vim 180-querying-dd-for-progress.post
143:sjas      3580  0.0  0.0   9228  1248 ?        S    06:26   0:00  \_ ksysguardd
169:sjas      6560  0.0  0.0   9868   636 pts/2    S+   17:46   0:00  |   \_ dd if=/dev/random of=/dev/null
172:sjas      6660  0.0  0.0   7836   892 pts/3    S+   17:49   0:00      \_ grep -i -n --color dd
[sjas@mb ~]$ 

So in this example, the PID is 6560.

From the second shell:

kill -usr1 6560

will then show additionally this in the first shell:

0+99 records in
1+0 records out
512 bytes (512 B) copied, 250.022 s, 0.0 kB/s

Of course, you could also pipe the data through pv or bar, to have a continouus status bar. But maybe you don't want that (will slow down things a bit), or you just forgot, and so you still can query the process for the current progress.

Linux: show all block devices with lsblk

posted on 2014-11-05 00:01:23

To see all currently connected devices like HDD's, SSD's, CD-Rom's and USB sticks, try lsblk.

Usually it looks like this:

sjas@mb:~/ISO/UBCD$ lsblk
NAME                         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                            8:0    0  29.8G  0 disk
|─sda1                         8:1    0   487M  0 part /boot/efi
|─sda2                         8:2    0  25.6G  0 part /
`─sda3                         8:3    0   3.8G  0 part [SWAP]
sr0                           11:0    1 589.2M  0 rom
sdb                            8:16   0 596.2G  0 disk
|─sdb1                         8:17   0   200M  0 part
|─sdb2                         8:18   0   500M  0 part
`─sdb3                         8:19   0 595.5G  0 part
  |─fedora_debra-root (dm-0) 254:0    0    50G  0 lvm
  |─fedora_debra-home (dm-1) 254:1    0   542G  0 lvm
  `─fedora_debra-swap (dm-2) 254:2    0   3.5G  0 lvm
sjas@mb:~/ISO/UBCD$

For a better overview, try a better selection of -o flags. Heres an overview on the possible options on an arbitrary system:

[jl@jerrylee ~]% \lsblk --help | \grep Available -A999 | sed -e '1d' -e '$d' | sed '$d'
        NAME  device name
       KNAME  internal kernel device name
     MAJ:MIN  major:minor device number
      FSTYPE  filesystem type
  MOUNTPOINT  where the device is mounted
       LABEL  filesystem LABEL
        UUID  filesystem UUID
   PARTLABEL  partition LABEL
    PARTUUID  partition UUID
          RA  read-ahead of the device
          RO  read-only device
          RM  removable device
       MODEL  device identifier
      SERIAL  disk serial number
        SIZE  size of the device
       STATE  state of the device
       OWNER  user name
       GROUP  group name
        MODE  device node permissions
   ALIGNMENT  alignment offset
      MIN-IO  minimum I/O size
      OPT-IO  optimal I/O size
     PHY-SEC  physical sector size
     LOG-SEC  logical sector size
        ROTA  rotational device
       SCHED  I/O scheduler name
     RQ-SIZE  request queue size
        TYPE  device type
    DISC-ALN  discard alignment offset
   DISC-GRAN  discard granularity
    DISC-MAX  discard max bytes
   DISC-ZERO  discard zeroes data
       WSAME  write same max bytes
         WWN  unique storage identifier
        RAND  adds randomness
      PKNAME  internal parent kernel device name
        HCTL  Host:Channel:Target:Lun for SCSI
        TRAN  device transport type
         REV  device revision
      VENDOR  device vendor

You can of course take this listing and try it directly:

\lsblk -o$(\lsblk --help | \grep Available -A999 | sed -e '1d' -e '$d' | sed '$d' | awk '{print $1}' | tr '\n' ',' | sed 's/,$//')

But if you do not have two widescreen monitors and a PTY shell drawn across it, you wont be recognizing much. Just for the record, on a Kali LIVE the above command won't even show all the output but jsut die gracefully without even showing an error.

So you might try this:

root@mb:/home/sjas/ISO/UBCD# lsblk -i -o name,label,mountpoint,fstype,model,size,type,state,uuid
NAME                         LABEL    MOUNTPOINT FSTYPE      MODEL              SIZE TYPE STATE   UUID
sda                                                          SSDSA2SH032G1GN   29.8G disk running
|─sda1                                /boot/efi  vfat                           487M part         5604-FDAF
|─sda2                                /          ext4                          25.6G part         ded0e7a8-23af-4deb-9b9d-9d63a26904aa
`─sda3                                [SWAP]     swap                           3.8G part         b022846d-e4b5-475b-b087-c4d5b486601f
sr0                          UBCD532             iso9660     DVDRW  GS21N     589.2M rom  running
sdb                                                          Name             596.2G disk running
|─sdb1                       untitled            hfsplus                        200M part         0fdc7456-f171-3490-9d41-671b43d70db3
|─sdb2                                           ext4                           500M part         66a8ed1c-e56e-4707-bbd5-15bcde2fa5a0
`─sdb3                                           LVM2_member                  595.5G part         BIC2hD-zS3w-yvtC-oNEG-yec1-4Q7h-qb4gwN
  |─fedora_debra-root (dm-0)                     ext4                            50G lvm  running e35f5406-0cc0-4646-86f8-c4031005580a
  |─fedora_debra-home (dm-1)                     ext4                           542G lvm  running 8bc9c1a4-a5c1-4e1b-9dfc-7aeb6437c708
  `─fedora_debra-swap (dm-2)                     swap                           3.5G lvm  running 2ddddb46-8b6d-4fb4-ae19-6390e1015b76
root@mb:/home/sjas/ISO/UBCD#

Of course, I have lsblk -o name,label,mountpoint,fstype,model,size,type,state,uuid aliased in my .bashrc:

alias lsblk='lsblk -o name,label,mountpoint,fstype,model,size,type,state,uuid'

UPDATE:

lsblk -i -o kname,mountpoint,fstype,size,maj:min,rm,name,state,rota,ro,type,label,model,serial

is what i stick with.

bash for loops like in C

posted on 2014-11-03 13:50:31

To have 'counting' bash loops, try the following.

Directly in a shell:

[sjas@ctr-014 ~]% for (( i=0; i<5; i++ )); do echo $i; done
0
1
2
3
4
[sjas@ctr-014 ~]%

As a script:

#!/bin/bash

for (( i=0; i<5; i++ ))
do
    echo $i
done

dd progress bar

posted on 2014-11-03 13:48:27

To get a proper progess bar when using dd, try using pv. Maybe apt-get install'ing it is needed, if yes, just go ahead.

Usage shown on the example of copying an .iso onto an usb stick:

[sjas@ctr-014 ~/Downloads]% pv -tpreb CentOS-6.6-x86_64-minimal.iso | dd of=/dev/sdc
 383MB 0:04:09 [1.53MB/s] [========================================>] 100%
 784384+0 records in
 784384+0 records out
 401604608 bytes (402 MB) copied, 265.133 s, 1.5 MB/s
[sjas@ctr-014 ~/Downloads]%

Usually you don't see the second+ lines, and would have to wait 4 minutes until you see your copying was successful.

For small devices this is fine, but when copying whole disks this behaviour becomes VERY annoying.

Another utility would be bar:

bar -if=CentOS-6.6-x86_64-minimal.iso | dd of=/dev/sdc

Same principle as pv, handing it an inputfile and piping it to dd.

list of all shell shortcuts (bash / zsh)

posted on 2014-09-30 14:09:12

zsh:

bindkey -L

bash:

bind -P

## alternative: (improved readability!)
bind -P | grep -v "is not" | sed 's/can be found on/:/' | column -s: -t

Distinguish builtin shell functions, aliases, functions and commands

posted on 2014-01-20 10:04:41

If you have a grown .bashrc and wonder what commands you did define in the past, these are helpful:

bash

type -t

Will show you what exactly you are dealing with. (Builtin, alias, function, regular command.)

[jl@jerrylee ~]$ type -t git
file

[jl@jerrylee ~]$ type -t export
builtin

[jl@jerrylee ~]$ type -t gc
alias

gc is an alias which I have locally defined in my .bashrc. It has a function bound to itself as we will see.

Built-ins are looked up a the main man page. (I.e. man bash or man zsh.)

alias

[jl@jerrylee ~]$ alias gc
alias gc='gitcommit'

If alias is used with no string afterwards, it will push out a complete list of all defined aliases.

declare / typeset

To look up functions:

[jl@jerrylee ~]$ declare -f gitcommit
gitcommit () 
{ 
    git c "$*"
}
[jl@jerrylee ~]$ typeset -f gitcommit
gitcommit () 
{ 
    git c "$*"
}

declare -f and typeset -f are synonymous.

zsh

Here you easiest start with which.

Besides, all other commands are the same.

This blog covers .csv, .htaccess, .pfx, .vmx, /etc/crypttab, /etc/network/interfaces, /etc/sudoers, /proc, 10.04, 14.04, AS, ASA, ControlPanel, DS1054Z, GPT, HWR, Hyper-V, IPSEC, KVM, LSI, LVM, LXC, MBR, MTU, MegaCli, PHP, PKI, R, RAID, S.M.A.R.T., SNMP, SSD, SSL, TLS, TRIM, VEEAM, VMware, VServer, VirtualBox, Virtuozzo, XenServer, acpi, adaptec, algorithm, ansible, apache, apachebench, apple, arcconf, arch, architecture, areca, arping, asa, asdm, autoconf, awk, backup, bandit, bar, bash, benchmarking, binding, bitrate, blackarmor, blockdev, blowfish, bochs, bond, bonding, booknotes, bootable, bsd, btrfs, buffer, c-states, cache, caching, ccl, centos, certificate, certtool, cgdisk, cheatsheet, chrome, chroot, cisco, clamav, cli, clp, clush, cluster, coleslaw, colorscheme, common lisp, console, container, containers, controller, cron, cryptsetup, csync2, cu, cups, cygwin, d-states, database, date, db2, dcfldd, dcim, dd, debian, debug, debugger, debugging, decimal, desktop, df, dhclient, dhcp, diff, dig, display manager, dm-crypt, dmesg, dmidecode, dns, docker, dos, drivers, dtrace, dtrace4linux, du, dynamictracing, e2fsck, eBPF, ebook, efi, egrep, emacs, encoding, env, error, ess, esx, esxcli, esxi, ethtool, evil, expect, exportfs, factory reset, factory_reset, factoryreset, fail2ban, fbsd, fdisk, fedora, file, filesystem, find, fio, firewall, firmware, fish, flashrom, forensics, free, freebsd, freedos, fritzbox, fsck, fstrim, ftp, ftps, g-states, gentoo, ghostscript, git, git-filter-branch, github, gitolite, global, gnutls, gradle, grep, grml, grub, grub2, guacamole, hardware, haskell, hdd, hdparm, hellowor, hex, hexdump, history, howto, htop, htpasswd, http, httpd, https, i3, icmp, ifenslave, iftop, iis, imagemagick, imap, imaps, init, innoDB, innodb, inodes, intel, ioncube, ios, iostat, ip, iperf, iphone, ipmi, ipmitool, iproute2, ipsec, iptables, ipv6, irc, irssi, iw, iwconfig, iwlist, iwlwifi, jailbreak, jails, java, javascript, javaws, js, juniper, junit, kali, kde, kemp, kernel, keyremap, kill, kpartx, krypton, lacp, lamp, languages, ldap, ldapsearch, less, leviathan, liero, lightning, links, linux, linuxin3months, lisp, list, livedisk, lmctfy, loadbalancing, locale, log, logrotate, looback, loopback, losetup, lsblk, lsi, lsof, lsusb, lsyncd, luks, lvextend, lvm, lvm2, lvreduce, lxc, lxde, macbook, macro, magento, mailclient, mailing, mailq, manpages, markdown, mbr, mdadm, megacli, micro sd, microsoft, minicom, mkfs, mktemp, mod_pagespeed, mod_proxy, modbus, modprobe, mount, mouse, movement, mpstat, multitasking, myISAM, mysql, mysql 5.7, mysql workbench, mysqlcheck, mysqldump, nagios, nas, nat, nc, netfilter, networking, nfs, nginx, nmap, nocaps, nodejs, numberingsystem, numbers, od, onyx, opcode-cache, openVZ, openlierox, openssl, openvpn, openvswitch, openwrt, oracle linux, org-mode, os, oscilloscope, overview, parallel, parameter expansion, parted, partitioning, passwd, patch, pct, pdf, performance, pfsense, php, php7, phpmyadmin, pi, pidgin, pidstat, pins, pkill, plesk, plugin, posix, postfix, postfixadmin, postgres, postgresql, poudriere, powershell, preview, profiling, prompt, proxmox, ps, puppet, pv, pveam, pvecm, pvesm, pvresize, python, qemu, qemu-img, qm, qmrestore, quicklisp, r, racktables, raid, raspberry pi, raspberrypi, raspbian, rbpi, rdp, redhat, redirect, registry, requirements, resize2fs, rewrite, rewrites, rhel, rigol, roccat, routing, rs0485, rs232, rsync, s-states, s_client, samba, sar, sata, sbcl, scite, scp, screen, scripting, seafile, seagate, security, sed, serial, serial port, setup, sftp, sg300, shell, shopware, shortcuts, showmount, signals, slattach, slip, slow-query-log, smbclient, snmpget, snmpwalk, software RAID, software raid, softwareraid, sophos, spacemacs, spam, specification, speedport, spi, sqlite, squid, ssd, ssh, ssh-add, sshd, ssl, stats, storage, strace, stronswan, su, submodules, subzone, sudo, sudoers, sup, swaks, swap, switch, switching, synaptics, synergy, sysfs, systemd, systemtap, tar, tcpdump, tcsh, tee, telnet, terminal, terminator, testdisk, testing, throughput, tmux, todo, tomcat, top, tput, trafficshaping, ttl, tuning, tunnel, tunneling, typo3, uboot, ubuntu, ubuntu 16.04, udev, uefi, ulimit, uname, unetbootin, unit testing, upstart, uptime, usb, usbstick, utf8, utm, utm 220, ux305, vcs, vgchange, vim, vimdiff, virtualbox, virtualization, visual studio code, vlan, vmstat, vmware, vnc, vncviewer, voltage, vpn, vsphere, vzdump, w, w701, wakeonlan, wargames, web, webdav, weechat, wget, whois, wicd, wifi, windowmanager, windows, wine, wireshark, wpa, wpa_passphrase, wpa_supplicant, x11vnc, x2x, xfce, xfreerdp, xmodem, xterm, xxd, yum, zones, zsh


Unless otherwise credited all material Creative Commons License by sjas