Posts tagged bash

mysql describe all tables from database
posted on 2017-01-23 12:51

This can be used directly in bash:

DB=your_database_name_here; for i in $(mysql $DB -Ne 'show tables' | cat); do echo; echo $i; mysql $DB -te "describe $i"; done

Just adjust your database.

linux shell number converters
posted on 2016-11-19 15:26

These are interactive promts from converting between the different number formats to decimal and reverse.

# hex-dec
h2d() {
    echo
    echo TO DEC, ctrl+c to end
    echo
    while :
    do
        read -p "hex> " i
        echo "ibase=16; $i" | bc
        echo
    done
}
d2h() {
    echo
    echo TO HEX, ctrl+c to end
    echo
    while :
    do
        read -p "dec> " i
        echo "obase=16; $i" | bc 
        echo
    done
}

# oct-dec
o2d() {
    echo
    echo TO DEC, ctrl+c to end
    echo
    while :
    do
        read -p "hex> " i
        echo "ibase=8; $i" | bc
        echo
    done
}
d2o() {
    echo
    echo TO OCT, ctrl+c to end
    echo
    while :
    do
        read -p "dec> " i
        echo "obase=8; $i" | bc 
        echo
    done
}

# bin-dec
b2d() {
    echo
    echo TO DEC
    echo
    while :
    do
        read -p "bin> " i
        echo "ibase=2; $i" | bc
        echo
    done
}
d2b() {
    echo
    echo TO BIN, ctrl+c to end
    echo
    while :
    do
        read -p "dec> " i
        echo "obase=2; $i" | bc 
        echo
    done
}

Put these into your ~/.bashrc.

Enjoy.

imap via linux shell
posted on 2016-11-09 23:52

Connect to server:

# IMAP
nc SERVERNAME-OR-IP 143
# IMAPS
openssl s_client -connect SERVERNAME-OR-IP:993

IMAP commands:

  • enumerate/prefix commands with arbitrary labels or simply a '.'
  • login USERNAME "PASSWORD" # login
  • list "" "*" # show all mailboxes
  • status [mailbox]
  • select "MAILBOX" # switch to mailbox
  • fetch FIRST:LAST FLAGS
  • fetch MAILID BODY[HEADER]
  • fetch MAILID BODY[TEXT]
  • uid search all
  • uid store MAILID +flags (\Deleted) # mark as deleted
  • expunge # actual delete
  • logout # logout
proper bash history logging
posted on 2016-09-14 23:32

By appending these to your .bashrc:

HISTTIMEFORMAT="%s "
PROMPT_COMMAND="${PROMPT_COMMAND:+$PROMPT_COMMAND ; }"'echo $$ $USER "$(history 1)" >> ~/.bash_history2'

you get a proper history looking like that from all shells connected to a server for each individual user:

root@fahi:~# cat .bash_history2
2786 root    91  1473887187 echo test
2786 root    92  1473887262 l
2786 root    93  1473887267 rm .bash_eternal_history 
2806 root    98  1473887148 tail -f .bash_eternal_history 
2806 root    99  1473888769 cat .bash_history2
2806 root   100  1473888788 lsblk
2821 root    98  1473887148 tail -f .bash_eternal_history 
2821 root    99  1473888794 history 
2821 root   100  1473888809 cat .bash_history2
2835 root   102  1473888809 cat .bash_history2

From the first look it looks good so far, but I fear there is some testing due to make sure there are no bad edge cases. One that I've found so far, was the last command gets repeated on login, maybe, and also when ctrl-c'ing commands. But this could have been the cause due to different shells being active with and without the prompt_command.

More alternatives can be found here.

gnu parallel instead of bash for loops
posted on 2016-05-24 18:15

If you happen to have to iterate over a list of files/strings/whatever, gnu parallel comes in handy after you installed it from your linux distro's package manager.

Then instead of:

for i in *; do echo "test $i"; done

You can simply do:

ls -1 | parallel echo "test "

# alternatively:
parallel echo 'test ' ::: `ls -1`

If you happen to have more complex scripts, simply double-quote the commands handed to parallel. Use {} if you happen to need a reference to the current variable.

I.e.

parallel "echo 'number {} echoed'" ::: `seq 1 10`

which gives:

number 1 echoed
number 2 echoed
number 3 echoed
number 4 echoed
number 5 echoed
number 6 echoed
number 7 echoed
number 8 echoed
number 9 echoed
number 10 echoed

At first this does not look like much, but how often have you messed up for loops? The example is rather made up, but does work better the more complex your examples become.

current blogpost-creation shortcut
posted on 2016-05-22 18:42

This is put here for documentation purposes:

createpost(){
TITLE=$1
TEMP=$(date --rfc-3339=seconds) 
CURRENT_COUNT=$(basename $(find . -iname "*.post" | sort | tail -1 ) | cut -c1- | sed 's/-.\+//g')
COUNT=$(( $CURRENT_COUNT + 1 ))
DATE=${TEMP:0:16}

FINAL_FILENAME=$COUNT-$TITLE.post
FINAL_TITLE=$(echo $1 | sed 's/[[:digit:]]\+-//' | sed 's/-/ /g')

cat << EOF >> $FINAL_FILENAME
;;;;;
title: $FINAL_TITLE
tags: todo
date: $DATE
format: md
;;;;;

EOF

vim $FINAL_FILENAME
}
DNS: resolution and reverse resolution script
posted on 2016-05-21 20:54

This is a quick-and-dirty for loop for checking a list of dns A resource records using dig. CNAME's are not handled like they should, thus are printed not in the same line, so this can not be used for being parsed without first having a look at the output and curating it first, if these are in use.

for i in sjas.de blog.sjas.de; do echo -n $'\e[33;1m'$i$'\e[0m '; TEMP=`dig +short $i`; echo -n "$TEMP "; TEMP=`dig -x $TEMP +short`; echo ${TEMP%.}; done | column -t

Instead of editing the for-loop, it might be helpful using a heredoc instead:

echo; cat << EOF | while read i; do echo -n $'\e[33;1m'$i$'\e[0m '; TEMP=`dig +short $i`; echo -n "$TEMP "; TEMP=`dig -x $TEMP +short`; echo ${TEMP%.}; done | column -t; echo

Paste this into the shell, followed by a paste of lines of the domains you want, and type EOF afterwards.

Example:

sjas@sjas:~/blog$ echo; cat << EOF | while read i; do echo -n $'\e[33;1m'$i$'\e[0m '; TEMP=`dig +short $i`; echo -n "$TEMP "; TEMP=`dig -x $TEMP +short`; echo ${TEMP%.}; done | column -t; echo
sjas.de
ix.de
asdf.de
EOF

sjas.de  78.47.176.149  static.149.176.47.78.clients.your-server.de
ix.de    193.99.144.80  redirector.heise.de
asdf.de  80.237.132.85  wp078.webpack.hosteurope.de

sjas@sjas:~/blog$ 
bash: add/remove leading zero to all filenames
posted on 2016-03-12 10:39:35

add leading zero to all filenames in current folder

for i in *; do mv $i 0$i; done

remove leading zero to all filenames starting with four digits in current folder

for i in $(ls -1 0{0..9}{0..9}{0..9}*); do mv $i ${i#0}; done
SSL certificate check from shell
posted on 2016-02-05 12:53:33

This will show the complete certificate:

echo | openssl s_client -connect google.com:443 2>/dev/null | openssl x509 -noout -text

Exchange the -text flag with any other object which are present in the certificate to get different results. (I.e. -subject or -dates.)

Magento: find out version from shell
posted on 2016-02-03 13:37:57

From within the docroot of your installation:

find . -iname Mage.php | xargs grep "public static function getVersionInfo" -A10

This should do until the code gets changed, tested with the 1.9.0.1 release.

nmap: show available ssl ciphers of a server
posted on 2016-01-04 19:39:00

command

nmap --script ssl-enum-ciphers -p <PORT> <URL>

example

Starting Nmap 6.47 ( http://nmap.org ) at 2016-01-04 15:37 CET
Nmap scan report for sjas.de (78.47.176.149)
Host is up (0.0047s latency).
rDNS record for 78.47.176.149: static.149.176.47.78.clients.your-server.de
PORT    STATE SERVICE
443/tcp open  https
| ssl-enum-ciphers: 
|   SSLv3: 
|     ciphers: 
|       TLS_DHE_RSA_WITH_AES_128_CBC_SHA - strong
|       TLS_DHE_RSA_WITH_AES_256_CBC_SHA - strong
|       TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA - strong
|       TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA - strong
|       TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA - strong
|       TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA - strong
|       TLS_RSA_WITH_AES_128_CBC_SHA - strong
|       TLS_RSA_WITH_AES_256_CBC_SHA - strong
|       TLS_RSA_WITH_CAMELLIA_128_CBC_SHA - strong
|       TLS_RSA_WITH_CAMELLIA_256_CBC_SHA - strong
|     compressors: 
|       NULL
|_  least strength: strong

Nmap done: 1 IP address (1 host up) scanned in 30.54 seconds
linux: show all cronjob files' contents
posted on 2015-12-31 11:50:22

Why didn't I think of that earlier???

for i in $(find /etc/cron*); do echo $'\e[33;1m'$i$'\e[0m'; cat $i; done | less -R

Or, if in doubt and you suspect evil doings happening:

for i in /var/spool/cron/* $(find /etc/cron*/); do echo $'\e[33;1m'$i$'\e[0m'; cat $i; done | less -R
freebsd: static bash
posted on 2015-12-29 07:43:55

To get a static bash executable which is always available, try this:

make -C /usr/ports/shells/bash -D WITH_STATIC_BASH -DWITHOUT_NLS PREFIX=/ install
bash prompt deluxe
posted on 2015-10-12 00:31:04

For quite a long time I have had the same prompt on and off, like:

[user@host ~/folder]$ 

This one was already colored. However quite a while ago I read about Steve Losh and his ZSH prompt, where he also used to show git or mercurial repository information.

After quite a while (making the exit code colored depending on wether it is zero or not is harder than it seems...), this was also added. Without further ado (or any explanation how the colors look like, here are some exmples:

REGULAR, GIT, SVN:
0 [256] 1 [ sjas@nb.dyn.sjas.de ~] 00:06:39 $ cd repo/gitolite-admin/
0 [257] 2 [ sjas@nb.dyn.sjas.de ~/repo/gitolite-admin git:[master] ] 00:06:45 $ cd ../non-modal-swing-dialog-read-only/
0 [258] 3 [ sjas@nb.dyn.sjas.de ~/repo/non-modal-swing-dialog-read-only svn:[Rev 41] ] 00:06:50 $ 

ERROR CODE as first number:
0 [258] 3 [ sjas@nb.dyn.sjas.de ~/repo/non-modal-swing-dialog-read-only svn:[Rev 41] ] 00:07:05 $ asdf
bash: asdf: command not found
127 [259] 4 [ sjas@nb.dyn.sjas.de ~/repo/non-modal-swing-dialog-read-only svn:[Rev 41] ] 00:07:07 $

The second number is the history count altogether like in the history file, the third one the count of the current session. Everything is colored, and for me it is not too long due to the colors.

This goes into the ~/.bashrc:

promptfunction() {
    local EXIT="$?"
    local VCS=""
    PS1=""
    if git branch &>/dev/null
    then
        VCS=" git:$(git show-branch | awk '{print $1}') "
    else
        if svn info &>/dev/null
        then
            VCS=' svn:[Rev '"$(svn info | \grep -i revision | awk '{print $2}')"'] '
        fi
    fi
    PS1="\[\e[3$(if [ $EXIT = 0 ]; then echo '2'; else echo '1'; fi);1m\]\$?\[\e[0m\] [\!] \# \[\e[31;1m\][\[\e[37;1m\] \u\[\e[33;1m\]@\[\e[37;1m\]$(hostname -f) \[\e[32;1m\]\w\[\e[36;1m\]$VCS\[\e[0m\]\[\e[31;1m\]]\[\e[0m\] \[\e[33;1m\]\t\[\e[0m\] \[\e[36;1m\]\\$ \[\e[0m\]"
}
export PROMPT_COMMAND=promptfunction

I could have changed the coloring such that i'd have used variables for the coloring, but by now I can read them just as well. If you want to know more about the coloring, google 'ansi escape codes'. :)

awk: show postfix mailq mail ID's for specific mail
posted on 2015-09-28 00:46:44

In short, replace <searchterm> with a regex for the adress you want:

mailq | awk 'BEGIN { RS = "" } /<searchterm>/ {print $1} '
MySQL: Check used storage engine
posted on 2015-07-13 15:32:01

Something to copy paste, in case you already have a .my.cnf for your root user with his password.

This only for tables you created:

less < <({ for i in $(mysql -e "show databases;" | cat | grep -v -e Database -e information_schema -e mysql -e performance_schema); do echo "--------------------$i--------------------";  mysql -e "use $i; show table status;"; done } | column -t)

This will show all tables, including the mysql ones:

less < <({ for i in $(mysql -e "show databases;" | cat | grep -v -e Database); do echo "--------------------$i--------------------";  mysql -e "use $i; show table status;"; done } | column -t)

To make it a little more readable, hitting -S in less turns or wordwrapping in less. Thus the lines which are too long are simply cut.

In a little more detail, this cannot be copy pasted in this form as it's missing the line break escapes, sorry this time not:

less < <(
            { 
                for i in $(mysql -e "show databases;" | 
                cat | 
                grep -v -e Database -e information_schema -e mysql -e performance_schema);
                do echo "--------------------$i--------------------"; 
                    mysql -e "use $i; show table status;";
                done 
            } | 
            column -t
        )

The cat piping is needed so the output will be without borders. I honestly have no idea why this cat here works the way it does. :)

bash: check MTU
posted on 2015-06-29 17:30:20

To check which MTU works, here's a one-liner. Will have colored output

for (( i=1520; i>1400; i=i-2 )); do if ping -c 1 -M do -s "$i" 8.8.8.8 &>/dev/null; then echo $'\e[32m'; else echo $'\e[31m'; fi; echo "$i ($(( $i + 28 )))"; done

Or easier to read:

for (( i=1520; i>1400; i=i-2 ))
do
    if ping -c 1 -M do -s "$i" 8.8.8.8 &>/dev/null
        then echo $'\e[32m'
        else echo $'\e[31m'
    fi
    echo "$i ($(( $i + 28 )))"
done
bash: fun with programming
posted on 2015-04-11 22:37:59

While strolling around and doing some readup on FreeBSD and it's man pages, I came across the intro pages. There exist man 1 intro to man 9 intro. After having read all, I wanted to have an overview, which manpages were referenced from these, which lead to all this in the end.

With some messing around, this is what I ended up with finally:

[sjas@stv ~]$ MATCH=\\w\\+\([[:digit:]]\); MANPAGE="intro"; for (( i=1;i<10;i++ )); do echo "^[[33;1mman $i $MANPAGE^[[0m"; grep "$MATCH" <(man "$i" "$MANPAGE") | grep -v $(echo "$MANPAGE" | tr '[:lower:]' '[:upper:]') | grep --color "$MATCH"; done

Sidenote: Simply copy-pasting this will not work, see the ansi escape sequences part below on why. If you cannot wait, exchange the two occurences of ^[ characters with a literal escape. Insert via Ctrl-V followed by hitting Esc.

Since this makes use of really a lot of bash tricks, a write-up might be some fun and this post is the result. In case you don't understand something, try googling the term in question for further reference. This post is intended as a pointer on what to search at all.

As this grew quite long I could not be bothered to copy contents of man pages or insert links of wikipedia pages, so bear with me.

preface

As most people do not have a BSD installation ready, a reference the manpages of a linux command would help? A command with several pages would be needed, so how about:

man -k . | awk '{print $1}' | sort | uniq -c | grep -v -e 1 -e 2

Which will give:

  3 info
  3 open

So lets just use the 'info' man pages.

man -k will search all manpages for a given string, in our case for a literal dot which should be included in every page. Of the output only the first column is needed, which is done via awk '{print $1}'. (Do not use cut -d' ' -f1 for things like this, won't work if you have commands separated by several spaces.) sort the output, so double commands are listed in a row, followed by uniq -c which will list all the unique occurences as well as their count. grep -v excludes all occurences of either 1 or 2. (That is why -e is used for providing these, instead of piping through grep -v 1 | grep -v 2, which would work the same.)

overview

Now onto the real beef, which will look like this:

[sjas@nb ~]$ MATCH=\\w\\+\(.\); MANPAGE="info"; for (( i=1;i<10;i++ )); do echo "^[[33;1mman $i $MANPAGE^[[0m"; grep "$MATCH" <(man "$i" "$MANPAGE") | grep -v $(echo "$MANPAGE" | tr '[:lower:]' '[:upper:]') | grep --color "$MATCH"; done
man 1 info
man 2 info
No manual entry for info in section 2
man 3 info
No manual entry for info in section 3
man 4 info
No manual entry for info in section 4
man 5 info
       The Info file format is an easily-parsable representation for online documents.  It can be read by emacs(1) and info(1) among other programs.
       Info files are usually created from texinfo(5) sources by makeinfo(1), but can be created from scratch if so desired.
       info(1), install-info(1), makeinfo(1), texi2dvi(1),
       texindex(1).
       emacs(1), tex(1).
       texinfo(5).
man 6 info
No manual entry for info in section 6
man 7 info
No manual entry for info in section 7
man 8 info
No manual entry for info in section 8
man 9 info
No manual entry for info in section 9

The headlines are printed in bold yellow, the matched manpages are printed in red.

For a better explanation, the one-liner above transformed into a bash script with line numbers:

1  #!/bin/bash
2  MATCH=\\w\\+\([[:digit:]]\)
3  MANPAGE="open"
4  for (( i=1;i<10;i++ ))
5  do 
6      echo "^[[33;1mman $i $MANPAGE^[[0m"
7      grep "$MATCH" <(man "$i" "$MANPAGE") | grep -v $(echo "$MANPAGE" | tr '[:lower:]' '[:upper:]') | grep --color "$MATCH"
8  done

shebang

The shebang in line 1 consists of the magic number #!, meaning the first byte of the file represents # and the second byte !. Unix systems scan files which have their executable bit set for these. When they are found, the rest of the line is treated as the path to the interpreter with which the script should be used. Its maximum lenght is 128 characters due to a compile time restraint, at least in FreeBSD.

variable declaration, definition

Lines 2 and 3 declare and define two variables. These are arbitrarily called MATCH and MANPAGE by me. By convention, these are uppercase, but lowercase will work as well. When a not-yet-present var is introduced (the shell does not know of one with the same name already) via its name and a =, it is declared (memory is reserved and it is created) and assigned the null string. When something follows after the =, it is also defined at once, and will hold the string which follows. Bash variables are usually untyped, when used like this (it's all strings), but with the declare or typeset built-ins (see man bash and search there) you can also define a 'variable' to be an integer, an indexed or associated function, a nameref (means it's a symlink to another variable), to be read-only, to be exported, to automatically uppercase the string of it's definition and such. But I disgress...

quoting and quotation marks (or lack thereof)

"quoting" is the act of 'removing the special meaning of certain characters or words to the shell'.

The second var is just the string 'open' in double quotation marks, whereas the first is also a string, just not enclosed within any quotation marks.

There are quite some variants that can be used:

'
"
(nothing)
\'
\"

In bash, everything in between single quotes is taken literally, no EXPANSION or other substitutions will take place in between the marks. There are these kinds of expansions or substituions:

- brace expansion
- tilde expansion
- parameter and variable expansion
- command substitution
- arithmetic expansion
- word splitting
- and pathname expansion

Look them up in the bash manual, if you are not already second-guessing your decision to read this posting.

Double quotes are used for enclosing strings, but letting bash be able to recognize these:

$ = most expansions
` = command substitions
\ = escapes
! = history expansion

That way, the expansion mechanisms mentioned above are possible to create strings dynamically.

No single quote may be used within double quotation marks, and if you need a literal quotation mark (i.e. for using a string of parameters for a command which is wrapped within another command) you can use pairs of \' or \".

If quoting is omitted, escape spaces and other special signs via the already mentioned escape character alias \, to get a coherent string, like shown in the first variable.

shell escaping and special characters

Since \, ( and ) are special characters in bash, and we want to end up with this string for the regular expression to match our manpage mentions:

'\w\+([[:digit:]])'

they have to be escaped.

regular expressions and character classes

The string itself is a regexp expressing 'match one or several (\+) word characters but no whitespace (\w), followed by an opening parens ((), an element belonging to the character class of digits, which means a number ([[:digit:]]) and finally an closing parens ()). Character classes are part of the POSIX standard and nice to know, since they are easier to use than \s or \w and will just work regardless of implementation as long as your system is POSIX-compliant.

for loop

Line 4 is the header of the for loop, whereas 5 and 8 enclose its body. The header is looped for all eternity while all statements return true. Usually bash's for is used like for i in <number-sequence>; do ..., but this is not everything which is possible.

i is the control variable, which is referenced via "$i" later on, just as the other variables are. ($MANPAGE, $MATCH)

arithmetic evaluation

The (( )) parentheses trigger arithmetic evaluation for what is contained in between, which are three statements in a row. The second statement is also an expression, while it evaluates to 'true', the loop's condition is satisfied and will run. Besides, the c-style for-loop should be self-explanatory.

This is basically the same as $(( ... )) (arithmetic expansion), the difference is the missing $. In bash $ denotes most kind of expansions or substitutions, references to a variable's definition are preluded with a $, too. Whereas in regular expressions it denotes the end of the line, just for the record.

ansi escape sequences

Line 6 is for getting some color into the shell. The ^[ is a literal escape sign, and needed to get bash to recognize the usage of ANSI escape sequences. To insert it, use Ctrl-v followed by Esc, and is a single character internally, even if its representation on the screen is given via two characters. You can see this when you delete it via backspace.

Usually the ANSI sequence part goes like this: <esc>[ <some numbers> m, where the [ denotes the start and m denotes the end of the escape-numbers list. 33 happens to be the number for yellow, red would be i.e. 31. The 1 just means bold. Depending on the feature set the console/terminal emulator you use, you could use the corresponding numbercode to make text underlined or let it blink. The 0 disables all non-standard settings again, so the text afterwards is regular colored and non-bold again.

Since the next part is a little bit more complex, here line number seven from above for easier reference:

7      grep "$MATCH" <(man "$i" "$MANPAGE") | grep -v $(echo "$MANPAGE" | tr '[:lower:]' '[:upper:]') | grep --color "$MATCH"

piping

The | character denotes piping. This simply means the part left of it is executed and the part to its right takes left's output as it's input via a character stream. (I hope this is correct, no warranty on that. :)) Internally a pipe is created through linking two file descriptors of two processes together.

process substitution

In the following, xyz will denote an arbitrary linux/unix command producing some output to the shell, in hope that this will help understanding.

<( xyz ) denotes process substitution (also look it up in man bash ;)), where the output of the command xyz is written to file referenced by a file descriptor which name is passed as argument to the calling command grep.
If >( xyz ) were used, xyz would read and not write to the file referenced by the file descriptor.

Phew. This sounds way harder than it actually is.

grep <searchterm> <( xyz ) means, grep the file descriptor naming the open file where xyz has written it's output to for <searchterm>.

Process substitution and the file descriptor are used, as grep can search only within files, not within an output stream which our xyz command above being man <number> <manpage-name> usually provides.

command substitution (through a subshell)

$( ... ) denotes a sub-shell, which will pass its result to it's parent shell. An older form is to use a pair of backticks, but this form is deprecated:

` ... `

Prior to executing grep -v on the input it is given from the pipe, the subshell is executed as a forked process of the calling process (the invoking shell) which will wait, and the result is handed back to its parent process (grep -v), which will resume execution again then.

This may sound like a contradiction to 'grep can only search in files', but it ain't. The searchterm of grep can be returned from another expression's evaluation, but the location in which to search has to be a file. As the input of grep comes from the pipe, which uses the connection of two processes' file descriptors, we close the circle.

It may be also noted, that if the search term is handed from an expression which hands back a list of several results, only the first result is used and searched for.

Proof:

[sjas@stv ~/test]$ grep --color $(ls -aF | grep '/' | grep './') <(ls -alhF)
/dev/fd/63:drwxr-xr-x  2 sjas  sjas     2B Apr 12 10:40 ./
/dev/fd/63:drwxr-xr-x  6 sjas  sjas    18B Apr 12 10:40 ../

The colored part of the output is just ./, as grep won't search for ../. In case you would want to achieve something like this, you'd have to use a for loop like for i in <command>; do grep --color "$i" <file>; done.

the rest

tr is just used to change every character matched with another one, here via the character classes. Each lowercase char will be exchanged with its uppercase equivalent.

For all die-hards that see this, thank you for reading.

mysql: output layout
posted on 2015-03-04 17:51:07

For big mysql tables with a lot of columns, the regular screen output is kind of hard to read at times.

Regularily you call queries like this:

select * from <tablename>;

There are several ways to fix this:

Within the client:

select * from <tablename>\G

At client startup:

## always use alternative output
mysql --vertical

## choose output depending on console width
mysql --auto-vertical-output

How does this look?

Regular:

mysql> show tables;
+---------------------------+
| Tables_in_mysql           |
+---------------------------+
| columns_priv              |
| db                        |
| event                     |
| func                      |
| general_log               |
| help_category             |
| help_keyword              |
| help_relation             |
| help_topic                |
| host                      |
| ndb_binlog_index          |
| plugin                    |
| proc                      |
| procs_priv                |
| proxies_priv              |
| servers                   |
| slow_log                  |
| tables_priv               |
| time_zone                 |
| time_zone_leap_second     |
| time_zone_name            |
| time_zone_transition      |
| time_zone_transition_type |
| user                      |
+---------------------------+
24 rows in set (0.00 sec)

Alternative:

mysql> show tables\G
*************************** 1. row ***************************
Tables_in_mysql: columns_priv
*************************** 2. row ***************************
Tables_in_mysql: db
*************************** 3. row ***************************
Tables_in_mysql: event
*************************** 4. row ***************************
Tables_in_mysql: func
*************************** 5. row ***************************
Tables_in_mysql: general_log
*************************** 6. row ***************************
Tables_in_mysql: help_category
*************************** 7. row ***************************
Tables_in_mysql: help_keyword
*************************** 8. row ***************************
Tables_in_mysql: help_relation
*************************** 9. row ***************************
Tables_in_mysql: help_topic
*************************** 10. row ***************************
Tables_in_mysql: host
*************************** 11. row ***************************
Tables_in_mysql: ndb_binlog_index
*************************** 12. row ***************************
Tables_in_mysql: plugin
*************************** 13. row ***************************
Tables_in_mysql: proc
*************************** 14. row ***************************
Tables_in_mysql: procs_priv
*************************** 15. row ***************************
Tables_in_mysql: proxies_priv
*************************** 16. row ***************************
Tables_in_mysql: servers
*************************** 17. row ***************************
Tables_in_mysql: slow_log
*************************** 18. row ***************************
Tables_in_mysql: tables_priv
*************************** 19. row ***************************
Tables_in_mysql: time_zone
*************************** 20. row ***************************
Tables_in_mysql: time_zone_leap_second
*************************** 21. row ***************************
Tables_in_mysql: time_zone_name
*************************** 22. row ***************************
Tables_in_mysql: time_zone_transition
*************************** 23. row ***************************
Tables_in_mysql: time_zone_transition_type
*************************** 24. row ***************************
Tables_in_mysql: user
24 rows in set (0.00 sec)
Linux: 'top' explained
posted on 2015-03-04 12:54:59

To get a fast overview on what is running on your linux box, use top. (If you want some fancy graphics, try htop, but it has less intuitive shortcuts and is not always installed.)

Sad thing is, at first you don't really know what you are doing. So some guidance:

start and sane defaults

After starting top, press: z, x, c. This will color top (z), show current sort column (x) and the full application path (c).

1 will show stats for all individual cpus.

If you have no idea, use h for getting the help shown.

If you have a newer version of top, V will also work:
This gives you a nice process-tree view.

d changes the update delay, which is at three seconds per default.

cpu stats explained

Straight from the manpage, the CPU statistics show the times spent in:

us = user mode
sy = system mode
ni = low priority user mode (nice)
id = idle task
wa = I/O waiting
hi = servicing IRQs
si = servicing soft IRQs
st = steal (time given to other DomU instances)

If you have low cpu and ram usage but the system is unresponsive, have a look at the wait times.

sorting and searching

Changing the sort column can be done via < and >.

Also available: (not shown in help)

N sort by PID
P sort by CPU usage
M sort by memory usage
T sort by time

R will reverse the output.

u to choose user name, show only this user's processes.

S for cululative time toggling.

columns

f will toggle a window in which you can choose the info fields to be shown. Pressing the character will toggle its state. (Shown or not shown.)

o also opens a window, in there you can reorder the columns. Press the character of the column you want to move, depending on it being upper- or lowercase it gets moved up and down.

manipulate tasks

These should be self-explanatory:

k kill task

r renice task

colored iptables output
posted on 2015-02-27 00:32:21

To get colored iptables output, try this monster:

iptables -L -vnx --line-numbers | sed ''/Chain.*/s//$(printf "\033[33;1m&\033[0m")/'' | sed ''/[ds]pt:.*/s//$(printf "\033[31;1m&\033[0m")/'' | sed ''/[ds]pts:.*/s//$(printf "\033[31;1m&\033[0m")/'' | sed -r ''/\([0-9]\{1,3\}\\.\)\{3\}[0-9]\{1,3\}\(\\/\([0-9]\)\{1,3\}\)\{0,1\}/s//$(printf "\033[36;1m&\033[0m")/g''

Ugly as shit could ever be, but only way I found out how this can be done. Also a little buggy, as some colors are a bit off, but still better than vanilla.

UPDATE: some fixes and better coloring and way more regex madness

iptables -L -vnx --line-numbers | \
sed ''/Chain[[:space:]][[:graph:]]*/s//$(printf "\033[33;1m&\033[0m")/'' | \
sed ''/^num.*/s//$(printf "\033[33m&\033[0m")/'' | \
sed ''/[[:space:]]DROP/s//$(printf "\033[31m&\033[0m")/'' | \
sed ''/REJECT/s//$(printf "\033[31m&\033[0m")/'' | \
sed ''/ACCEPT/s//$(printf "\033[32m&\033[0m")/'' | \
sed -r ''/\([ds]pt[s]\?:\)\([[:digit:]]\+\(:[[:digit:]]\+\)\?\)/s//$(printf "\\\1\033[33;1m\\\2\033[0m")/''| \
sed -r ''/\([0-9]\{1,3\}\\.\)\{3\}[0-9]\{1,3\}\(\\/\([0-9]\)\{1,3\}\)\{0,1\}/s//$(printf "\033[37;1m&\033[0m")/g'' | \
sed -r ''/\([^n][[:space:]]\)\(LOGDROP\)/s//$(printf "\\\1\033[1;33m\\\2\033[0m")/'' | \
sed -r ''/[[:space:]]LOG[[:space:]]/s//$(printf "\033[36;1m&\033[0m")/''

And something to copy paste more easily, slightly modified again:

iptables -L -vnx --line-numbers | sed ''/Chain[[:space:]][[:graph:]]*/s//$(printf "\033[33;1m&\033[0m")/'' | sed ''/^num.*/s//$(printf "\033[33m&\033[0m")/'' | sed ''/[[:space:]]DROP/s//$(printf "\033[31m&\033[0m")/'' | sed ''/REJECT/s//$(printf "\033[31m&\033[0m")/'' | sed ''/ACCEPT/s//$(printf "\033[32m&\033[0m")/'' | sed -r ''/\([ds]pt[s]\?:\)\([[:digit:]]\+\(:[[:digit:]]\+\)\?\)/s//$(printf "\\\1\033[33;1m\\\2\033[0m")/''| sed -r ''/\([0-9]\{1,3\}\\.\)\{3\}[0-9]\{1,3\}\(\\/\([0-9]\)\{1,3\}\)\{0,1\}/s//$(printf "\033[36;1m&\033[0m")/g'' | sed -r ''/\([^n][[:space:]]\)\(LOGDROP\)/s//$(printf "\\\1\033[1;33m\\\2\033[0m")/'' | sed -r ''/[[:space:]]LOG[[:space:]]/s//$(printf "\033[36;1m&\033[0m")/''| sed ''/CATCH-DROP/s//$(printf "\033[31m&\033[0m")/''
bash: combined dns-reverse-dns-lookup
posted on 2015-02-20 12:27:15

On dns lookups at work

While working with domains, you often need to to a dns lookup, to find out the ip of the machine in question (at least when working with several hundred web servers ;)), followed by a reverse dns lookup on the ip to find out the actual hostname. The regular hostname is just easier to remember than the IP. It's bad enough with IPv4, and will become worse with IPv6.

I.e. usually you do something like this:

[sjas@ctr-014 ~]% host ix.de
ix.de has address 193.99.144.80
ix.de has IPv6 address 2a02:2e0:3fe:1001:302::
ix.de mail is handled by 10 relay.heise.de.
ix.de mail is handled by 50 secondarymx.heise.de.
[sjas@ctr-014 ~]% host 193.99.144.80
80.144.99.193.in-addr.arpa domain name pointer redirector.heise.de.
[sjas@ctr-014 ~]%

or this:

[sjas@ctr-014 ~]% dig ix.de +short
193.99.144.80
[sjas@ctr-014 ~]% dig -x 193.99.144.80 +short
redirector.heise.de.
[sjas@ctr-014 ~]%

This can be 'shortened' into a single step with proper output:

[sjas@ctr-014 ~]% echo ${$(dig -x $(dig ix.de +short) +short)%?}
redirector.heise.de
[sjas@ctr-014 ~]%

proper solution

Since this is kind of unhandy (and let's be honest, bash sucks sometimes), just place it into a function definition in your .bashrc:

rdns() {
    echo ${$(dig -x $(dig $1 +short) +short)%?}
}

An in-depth explanation of this bash 'gem' will be added here, if I do not forget to add it in the near future. :)

Which let's you do:

[sjas@ctr-014 ~]% rdns ix.de
redirector.heise.de

script explanation

In short:

echo "${$(dig -x $(dig $1 +short) +short)%?}"

Echo a string...

echo "                                      "

... which stems out from a combination of parameter expansion...

      ${               $1                  }"

... wherein also a suffix is removed, in this case ? representing a single char.

                                         %?

There a subshell is used to run the dig command in a subshell...

        $(dig                     +short)

... which in turn runs another dig call in another subshell...

                 $(dig    +short)

Use man bash to get further info on this stuff.

Pause bash shell
posted on 2015-01-23 13:08:32

If you have a long running command with a lot of output where you just got a glimpse on something and you need a closer look but the shell won't let you scroll? (Due to new printouts appearing all the time.)

Use Ctrl-s to pause (and you can scroll up all you want, in case your terminal emulator will let you).
Afterwards Ctrl-q will 'unpause' it again.

The shell is not really put on hold, just the visual updating of the standard output is paused. After the unpausing, everything that has happened in the meantime will become updated again.

bash completion shortcuts
posted on 2015-01-23 11:23

The bash shell also has more shortcuts, than just the ones like for emacs or vi movement.

The other interesting completions are:

C-x /     filename completion
C-x $     bash variable completion
C-x @     hostname completion
C-x !     command completion

Meta-~ username completion
Meta-/ filename completion
Meta-$ bash variable completion
Meta-@ hostname completion
Meta-! which does command completion
Running bash scripts
posted on 2015-01-16 23:47:45

There are several ways, how bash scripts can be invoked.

Here are the basic ones along with some lesser known ones:

  1. If your script has a proper shebang and is executable:

    ./SCRIPTNAME.sh

  2. If its missing the x bit:

    bash SCRIPTNAME.sh

  3. Echo commands after processing:

    bash -x SCRIPTNAME.sh

  4. Syntax checking / dry-running:

    bash -n SCRIPTNAME.sh

Linux 'less', advantages, disadvantages, keys, options
posted on 2014-12-01 07:46:36

Being the default pager on linux, and thus the tool you use to look manpages at usually, less is worth some more attention.

key points

Unlike editors or IDE's (vi, emacs, nano, eclipse), pagers (at least less) do not have to load a file completely into memory and thus are faster when displaying huge files. If you happen to think you will never have to open files bigger than some KB size, what about some error logs? (Once I saw a machine write like one additional GB per minute. In this case, you should maybe refrain from less and just use like tail -n1000.)

Also, compared to more, less can also scroll backwards. (!!!)

disadvantages

Pagers cannot edit text. That's what editors are for.

keys

system

q                       quit
h                       show help
= or ctrl-g             show current file name
r                       redraw screen
s                       save file (if input comes from a pipe, not a file)

v                       edit file with $VISUAL or $EDITOR

!<command>              execute <command> in $SHELL
!<mark><command>        pipe text contents between cursor and <mark> to <command>

movement

f or ctrl-f or space    move forward one page
b or ctrl-b             move backward one page 

g                       top of first page
G                       bottom of last page

<count>p                go to <count> percent line in text

d                       forward half a page
u                       backward half a page

m<char>                 mark line with <char>
'<char>                 jump to mark <char>o
''                      goto previous position 

search

/<pattern>              search forward for <pattern>
?<pattern>              search backward for <pattern>

n                       next match
N                       previous match

! or ^N                 prior to <pattern>, will search for non-matching lines
^K                      prior to <pattern>, just mark lines but don't move cursor
^R                      don't use regexes for searching

&<pattern>              SHOW ONLY MATCHES (about the best less command ever)

Especially the | hotkey might be interesting.

To pipe the complete buffer content into a file, do this:

1. g (go to top of file)
2. | (start pipe)
3. $ (pipe until the end of buffer)
4. tee [name of logfile].log

Afterwards you should have a new file. This works both with piped input as well as opened files.

options

startup options

All options with dashes can be used while running less, or as startup commands.

I.e.

+F                      same as 'tail -f', but with less
+/<pattern>             open file at <pattern>

+ is needed during startup, from within less its not needed except when you want to reset a option to its default value.

search options

-A                      search starts after target line
-g                      highlight last search result
-G                      highlight search results
-I                      completely case insensitive searching
-i                      smartcase: case-insensitive if search string contains no upper case
-J                      show status column (to mark lines with search results)
                        left of the the text, lines with matches are marked.
F                       'Waiting for data... (interrupt to abort)' (means ^C)
                        This is basically a 'tail -f' on stereoids!

system options

preface:

`-` prior sets / changes the option
`_` just shows it's current state

-e                      quit at EOF
-M                      toggle long prompt (filename, lines, line %)
-m                      toggle medium prompt (line %)
-N                      show line numbers
-Q                      quiet all terminal bells (!!!)
-R                      output raw control chars = SHOW COLORS
-s                      squeeze multiple blank lines into one
-S                      don't wrap long lines

-P                      define custom promtps
                        See last section here about further information.

custom prompts

   %bX      Replaced by the byte offset into the current input file.   
            The  b  is followed by a single character (shown as X above) 
            which specifies the line whose byte offset is to be used.  
            If the character is a "t", the byte  offset of the top line in 
            the display is used, an "m" means use the middle line, a "b" 
            means use the bottom line, a "B" means use the line  just  after  
            the  bottom line, and a "j" means use the "target" line, 
            as specified by the -j option.

   %B       Replaced by the size of the current input file.

   %c       Replaced by the column number of the text appearing in the 
            first column of the screen.

   %dX      Replaced by the page number of a line in the input file.  
            The line to be used is determined by the X, as with the %b option.

   %D       Replaced by the number of pages in the input file, or quivalently, 
            the page number of the last line in the input file.

   %E       Replaced by the name of the editor (from the VISUAL 
            environment variable, or the EDITOR environment variable 
            if VISUAL is  not  defined).
            See the discussion of the LESSEDIT feature below.

   %f       Replaced by the name of the current input file.

   %F       Replaced by the last component of the name of the current input file.

   %i       Replaced by the index of the current file in the list of input files.

   %lX      Replaced by the line number of a line in the input file.  
            The line to be used is determined by the X, as with the %b option.

   %L       Replaced by the line number of the last line in the input file.

   %m       Replaced by the total number of input files.

   %pX      Replaced by the percent into the current input file,  
            based on byte offsets.  
            The line used is determined by the X as with the %b option.

   %PX      Replaced  by  the  percent into the current input file, 
            based on line numbers.   
            The line used is determined by the X as with the %b option.

   %s       Same as %B.

   %t       Causes any trailing spaces to be removed.  
            Usually used at the end of the string, but may appear anywhere.

   %x       Replaced by the name of the next input file in the list.
Linux: proper tempfiles
posted on 2014-11-20 10:24:43

mktemp creates randomly named files, a recurringly needed appliance.

create a tempfile and save name to a variable

VARIABLE_NAME=`mktemp`

(That was rather easy, wasn't it?)

create / delete raids with Adaptec's arcconf CLI
posted on 2014-11-17 18:11:41

When working with the CLI for the sole purpose of handling RAID's, usually these commands are needed:

  1. task
  2. create
  3. delete

This will be sort of a lazy posting, no screenpastes will find their way in here, I beg your pardon.

preparation

First make your live a lot easier:

alias asdf=/usr/StorMan/arcconf  ## or where your executable is located

Get an overview on what hardware is available:

asdf getconfig 1 pd | less

This is important, so you can locate the channels / slots of the drives you want to handle. The command is piped through less, since usually the output is too big to fit on a screen. (At least on an 19" 8-bay server, where all slots are filled.)

Similarily, you can see the already created RAID's via

asdf getconfig 1 ld

initialize drives

Once you got your information and you decided your layout, initialize the drives.

If you have nothing you need, and want to prepare all drives at once, do:

asdf task start 1 device all initialize

Else specify the channel and drive id, instead of using 'all':

asdf task start 1 device 0 0 initialize

This will erase the metadata from the drive in slot 0, if your setup is correctly assembled. (Else you are in for trouble, sooner or later, but if you do not know this, you might want to consider a different career path anyway...)

create a logical device

Lets have two examples, one raid1 spanning drives 0 0 and 0 1, and a raid10 on drives 0 4 to 0 7:

asdf create 1 logicaldrive name my_raid1 method quick max 1 0 0 0 1
asdf create 1 logicaldrive name my_raid10 method quick max 10 0 4 0 5 0 6 0 7

While the syntax is cryptical at first, this should become pretty clear once you did this several times.

create is self-explanatory, the first 1 means the controller, and in 99% of all cases you only have a setup with a single controller. logicaldrive is always a present keyword here (except you want to create a jbod), a name always helps. method quick initializing is usually also the best way to go. max specifies maximum size of the raid (that is, as big as the disks let it be).

The numbers afterwards are then:

  1. the raid level
  2. all the channel and slot number tuples

deleting a logical device

asdf delete 1 logicaldrive all

deletes all raids which were created prior.

asdf delete 1 logicaldrive 2

deletes the logical volume with the id 2. (Remember asdf getconfig 1 ld!)

That should be about it in short. modify for raid migration or online capacity expansion is reserved for another post for the time being.

Querying dd progress
posted on 2014-11-16 17:44:33

UPDATE: use pkill instead of kill: pkill -usr1 dd is all you need.


Usually dd will only show information about the transfer it did, AFTER its completion.

Or try a second shell, and sending a USR1 signal to the dd process.

First, lets startd a demo dd process:

[sjas@mb ~]$ dd if=/dev/random of=/dev/null

Then we need to find out the process id of this dd process. For this you can use pgrep, but i prefer grepping ps auxf:

[sjas@mb ~]$ ps auxf | grep dd
2:root         2  0.0  0.0      0     0 ?        S    06:25   0:00 [kthreadd]
91:sjas      3351  0.0  0.0  30588  1704 ?        Ss   06:25   0:01 /usr/bin/dbus-daemon --fork --print-pid 5 --print-address 7 --session
134:sjas      6501  0.0  0.0  31980  3496 pts/1    S+   17:43   0:00          \_ vim 180-querying-dd-for-progress.post
143:sjas      3580  0.0  0.0   9228  1248 ?        S    06:26   0:00  \_ ksysguardd
169:sjas      6560  0.0  0.0   9868   636 pts/2    S+   17:46   0:00  |   \_ dd if=/dev/random of=/dev/null
172:sjas      6660  0.0  0.0   7836   892 pts/3    S+   17:49   0:00      \_ grep -i -n --color dd
[sjas@mb ~]$ 

So in this example, the PID is 6560.

From the second shell:

kill -usr1 6560

will then show additionally this in the first shell:

0+99 records in
1+0 records out
512 bytes (512 B) copied, 250.022 s, 0.0 kB/s

Of course, you could also pipe the data through pv or bar, to have a continouus status bar. But maybe you don't want that (will slow down things a bit), or you just forgot, and so you still can query the process for the current progress.

bash ranging
posted on 2014-11-13 12:05:02

Using ranges in bash, you can avoid more complicated for loop constructs (which aren't needed 99% of the time anyway...):

[sjas@mb ~]$ for i in {1..5}; do echo $i; done
1
2
3
4
5
[sjas@mb ~]$ 

This also works with characters:

[sjas@mb ~]$ for i in {z..q}; do echo $i; done
z
y
x
w
v
u
t
s
r
q
[sjas@mb ~]$ 

Even backwards!

Minimal dotfiles
posted on 2014-11-06 05:40:21

Without further ado, this is put here for documentary reasons. I paste this into the current configs and am glad.

.bashrc

#######################################################################################################################
### aliases
#######################################################################################################################
alias l='ls -lahF --color'
alias ..='cd ..'
alias lsblk='lsblk -o name,label,mountpoint,fstype,model,size,type,state,uuid'
alias v='vim'
alias grep='grep -i -n --color'
alias ev='vim ~/.vimrc'
alias eb='vim ~/.bashrc'
alias eg='vim ~/.gitconfig'
alias ggfc='git add .; git commit -m "fastcommit"; git pull origin master; git push origin master'


#######################################################################################################################
### env
#######################################################################################################################
export EDITOR=vim
export VISUAL=vim
export PS1='\[\e[31;1m\][\[\e[37;1m\]\u\[\e[33;1m\]@\[\e[37;1m\]\H \[\e[32;1m\]\w\[\e[31;1m\]]\[\e[36;1m\]\$ \[\e[0m\]'
export PATH=/home/sjas/bin:$PATH
export LESS_TERMCAP_mb=$'\E[1;31m'
# COMMAND AND FLAG COLOR
export LESS_TERMCAP_md=$'\E[1;32m'
# PARAMETER COLOR
export LESS_TERMCAP_us=$'\E[1;33m'
export LESS_TERMCAP_me=$'\E[0m'
export LESS_TERMCAP_se=$'\E[0m'
export LESS_TERMCAP_so=$'\E[38m'
export LESS_TERMCAP_ue=$'\E[0m'

#######################################################################################################################
# blogging
#######################################################################################################################
function createpost() {
    if [ $# -ne 1 ]
    then 
        echo "Usage :: post filename as parameter without .post extension"
        exit 1
    fi

    cat << EOHD > ~/blog/$1.post
;;;;;
title: 
tags: 
date: 
format: md
;;;;;
content
EOHD
}

.vimrc

let mapleader=' '
let maplocalleader=' '
syn on
se hls
se is
se ic
se gd
se ai
se et
se sts=4
se ts=4
se sw=4
se backspace=indent,eol,start
se sm
se wildmenu
se wildmode=list:longest,full


nnoremap s/ :s/
nnoremap ss/ :%s/
vnoremap s/ :s/
nnoremap <leader><backspace> /qwerqwerasdfasdf<cr><esc>
nnoremap <leader>ul yypVr=0<cr>
nnoremap <leader>fs :.!date --rfc-3339=seconds<cr><esc>$xxxxxx
nnoremap <leader><leader>fs Sdate:<esc>o<esc>:.!date --rfc-3339=seconds<cr><esc>kJ$xxxxxx
nnoremap <leader><leader>i :%s/^\s*<cr>/qwerqwer<cr><esc>
vnoremap <leader><leader>i :s/^\s*<cr>/qwerqwer<cr><esc>
nnoremap <leader><leader>s :%s/\s*$<cr>/qwerqwer<cr><esc>
vnoremap <leader><leader>s :s/\s*$<cr>/qwerqwer<cr><esc>
vnoremap <leader>ln !nl -ba<cr>
nnoremap <leader>n :next<cr>
nnoremap <leader>b :prev<cr>
nnoremap <leader><c-d> :sh<cr>

nnoremap <Leader>wq :wq<CR>
nnoremap <Leader>fw :se bt=<cr>:w<CR>
nnoremap <Leader>fa :wa<CR>
nnoremap <Leader>fq :q<CR>
nnoremap <Leader>rq :q!<CR>

cmap w!! w !sudo tee % > /dev/null
Linux: show all block devices with lsblk
posted on 2014-11-05 00:01:23

To see all currently connected devices like HDD's, SSD's, CD-Rom's and USB sticks, try lsblk.

Usually it looks like this:

sjas@mb:~/ISO/UBCD$ lsblk
NAME                         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                            8:0    0  29.8G  0 disk
|─sda1                         8:1    0   487M  0 part /boot/efi
|─sda2                         8:2    0  25.6G  0 part /
`─sda3                         8:3    0   3.8G  0 part [SWAP]
sr0                           11:0    1 589.2M  0 rom
sdb                            8:16   0 596.2G  0 disk
|─sdb1                         8:17   0   200M  0 part
|─sdb2                         8:18   0   500M  0 part
`─sdb3                         8:19   0 595.5G  0 part
  |─fedora_debra-root (dm-0) 254:0    0    50G  0 lvm
  |─fedora_debra-home (dm-1) 254:1    0   542G  0 lvm
  `─fedora_debra-swap (dm-2) 254:2    0   3.5G  0 lvm
sjas@mb:~/ISO/UBCD$

For a better overview, try a better selection of -o flags. Heres an overview on the possible options on an arbitrary system:

[jl@jerrylee ~]% \lsblk --help | \grep Available -A999 | sed -e '1d' -e '$d' | sed '$d'
        NAME  device name
       KNAME  internal kernel device name
     MAJ:MIN  major:minor device number
      FSTYPE  filesystem type
  MOUNTPOINT  where the device is mounted
       LABEL  filesystem LABEL
        UUID  filesystem UUID
   PARTLABEL  partition LABEL
    PARTUUID  partition UUID
          RA  read-ahead of the device
          RO  read-only device
          RM  removable device
       MODEL  device identifier
      SERIAL  disk serial number
        SIZE  size of the device
       STATE  state of the device
       OWNER  user name
       GROUP  group name
        MODE  device node permissions
   ALIGNMENT  alignment offset
      MIN-IO  minimum I/O size
      OPT-IO  optimal I/O size
     PHY-SEC  physical sector size
     LOG-SEC  logical sector size
        ROTA  rotational device
       SCHED  I/O scheduler name
     RQ-SIZE  request queue size
        TYPE  device type
    DISC-ALN  discard alignment offset
   DISC-GRAN  discard granularity
    DISC-MAX  discard max bytes
   DISC-ZERO  discard zeroes data
       WSAME  write same max bytes
         WWN  unique storage identifier
        RAND  adds randomness
      PKNAME  internal parent kernel device name
        HCTL  Host:Channel:Target:Lun for SCSI
        TRAN  device transport type
         REV  device revision
      VENDOR  device vendor

You can of course take this listing and try it directly:

\lsblk -o$(\lsblk --help | \grep Available -A999 | sed -e '1d' -e '$d' | sed '$d' | awk '{print $1}' | tr '\n' ',' | sed 's/,$//')

But if you do not have two widescreen monitors and a PTY shell drawn across it, you wont be recognizing much. Just for the record, on a Kali LIVE the above command won't even show all the output but jsut die gracefully without even showing an error.

So you might try this:

root@mb:/home/sjas/ISO/UBCD# lsblk -i -o name,label,mountpoint,fstype,model,size,type,state,uuid
NAME                         LABEL    MOUNTPOINT FSTYPE      MODEL              SIZE TYPE STATE   UUID
sda                                                          SSDSA2SH032G1GN   29.8G disk running
|─sda1                                /boot/efi  vfat                           487M part         5604-FDAF
|─sda2                                /          ext4                          25.6G part         ded0e7a8-23af-4deb-9b9d-9d63a26904aa
`─sda3                                [SWAP]     swap                           3.8G part         b022846d-e4b5-475b-b087-c4d5b486601f
sr0                          UBCD532             iso9660     DVDRW  GS21N     589.2M rom  running
sdb                                                          Name             596.2G disk running
|─sdb1                       untitled            hfsplus                        200M part         0fdc7456-f171-3490-9d41-671b43d70db3
|─sdb2                                           ext4                           500M part         66a8ed1c-e56e-4707-bbd5-15bcde2fa5a0
`─sdb3                                           LVM2_member                  595.5G part         BIC2hD-zS3w-yvtC-oNEG-yec1-4Q7h-qb4gwN
  |─fedora_debra-root (dm-0)                     ext4                            50G lvm  running e35f5406-0cc0-4646-86f8-c4031005580a
  |─fedora_debra-home (dm-1)                     ext4                           542G lvm  running 8bc9c1a4-a5c1-4e1b-9dfc-7aeb6437c708
  `─fedora_debra-swap (dm-2)                     swap                           3.5G lvm  running 2ddddb46-8b6d-4fb4-ae19-6390e1015b76
root@mb:/home/sjas/ISO/UBCD#

Of course, I have lsblk -o name,label,mountpoint,fstype,model,size,type,state,uuid aliased in my .bashrc:

alias lsblk='lsblk -o name,label,mountpoint,fstype,model,size,type,state,uuid'

UPDATE:

lsblk -i -o kname,mountpoint,fstype,size,maj:min,rm,name,state,rota,ro,type,label,model,serial

is what i stick with.

bash for loops like in C
posted on 2014-11-03 13:50:31

To have 'counting' bash loops, try the following.

Directly in a shell:

[sjas@ctr-014 ~]% for (( i=0; i<5; i++ )); do echo $i; done
0
1
2
3
4
[sjas@ctr-014 ~]%

As a script:

#!/bin/bash

for (( i=0; i<5; i++ ))
do
    echo $i
done
dd progress bar
posted on 2014-11-03 13:48:27

To get a proper progess bar when using dd, try using pv. Maybe apt-get install'ing it is needed, if yes, just go ahead.

Usage shown on the example of copying an .iso onto an usb stick:

[sjas@ctr-014 ~/Downloads]% pv -tpreb CentOS-6.6-x86_64-minimal.iso | dd of=/dev/sdc
 383MB 0:04:09 [1.53MB/s] [========================================>] 100%
 784384+0 records in
 784384+0 records out
 401604608 bytes (402 MB) copied, 265.133 s, 1.5 MB/s
[sjas@ctr-014 ~/Downloads]%

Usually you don't see the second+ lines, and would have to wait 4 minutes until you see your copying was successful.

For small devices this is fine, but when copying whole disks this behaviour becomes VERY annoying.

Another utility would be bar:

bar -if=CentOS-6.6-x86_64-minimal.iso | dd of=/dev/sdc

Same principle as pv, handing it an inputfile and piping it to dd.

list of all shell shortcuts (bash / zsh)
posted on 2014-09-30 14:09:12

zsh:

bindkey -L

bash:

bind -P

## alternative: (improved readability!)
bind -P | grep -v "is not" | sed 's/can be found on/:/' | column -s: -t
bash table output
posted on 2014-09-10 16:34:02

Pipe the output of a command to column, which 'columniates' lists.

Before:

[sjas@ctr-014 ~]% ip r
default via 10.20.1.1 dev eth1.9 
10.0.0.0/16 dev eth1.3  proto kernel  scope link  src 10.0.255.14 
10.20.1.0/24 dev eth1.9  proto kernel  scope link  src 10.20.1.14 
169.254.0.0/16 dev eth1.2  scope link  metric 1000 
192.168.184.0/24 dev eth1.2  proto kernel  scope link  src 192.168.184.42 
192.168.189.0/24 dev eth1.522  proto kernel  scope link  src 192.168.189.42 
[sjas@ctr-014 ~]%

After:

[sjas@ctr-014 ~]% ip r | column -t
default           via  10.20.1.1  dev    eth1.9
10.0.0.0/16       dev  eth1.3     proto  kernel  scope   link  src  10.0.255.14
10.20.1.0/24      dev  eth1.9     proto  kernel  scope   link  src  10.20.1.14
169.254.0.0/16    dev  eth1.2     scope  link    metric  1000
192.168.184.0/24  dev  eth1.2     proto  kernel  scope   link  src  192.168.184.42
192.168.189.0/24  dev  eth1.522   proto  kernel  scope   link  src  192.168.189.42
[sjas@ctr-014 ~]%

Choose which you like better.

bash kill signals
posted on 2014-09-07 14:42:58

preface

bash uses these kinds of kill signals, along with many others. In fact, signals are rather part of the OS and not the shell itself, and are present in all unices.

Examples:

SIGHUP hangup (nowadays original intentions not needed, often used for config reloads) SIGINT keyboard interrupt (Ctrl-c) SIGKILL kill signal SIGTERM termination signal SIGSTOP stop the process SIGTSTP suspend (Ctrl-z)

SIGKILL and SIGSTOP cannot be caught, blocked or ignored.

traps

Look up bash traps in the manual on how to caption signals, i.e. in scripts. Since trap is a bash-builtin, use help trap to do so. (Or search in man bash).

list

Besides these, there are A LOT of others, just in case you wondered what else is out there.

kill -l will show a list of all available signals, depending on your OS.

[sjas@nb ~]$ kill -l
 1) SIGHUP       2) SIGINT       3) SIGQUIT      4) SIGILL       5) SIGTRAP
 6) SIGABRT      7) SIGBUS       8) SIGFPE       9) SIGKILL     10) SIGUSR1
11) SIGSEGV     12) SIGUSR2     13) SIGPIPE     14) SIGALRM     15) SIGTERM
16) SIGSTKFLT   17) SIGCHLD     18) SIGCONT     19) SIGSTOP     20) SIGTSTP
21) SIGTTIN     22) SIGTTOU     23) SIGURG      24) SIGXCPU     25) SIGXFSZ
26) SIGVTALRM   27) SIGPROF     28) SIGWINCH    29) SIGIO       30) SIGPWR
31) SIGSYS      34) SIGRTMIN    35) SIGRTMIN+1  36) SIGRTMIN+2  37) SIGRTMIN+3
38) SIGRTMIN+4  39) SIGRTMIN+5  40) SIGRTMIN+6  41) SIGRTMIN+7  42) SIGRTMIN+8
43) SIGRTMIN+9  44) SIGRTMIN+10 45) SIGRTMIN+11 46) SIGRTMIN+12 47) SIGRTMIN+13
48) SIGRTMIN+14 49) SIGRTMIN+15 50) SIGRTMAX-14 51) SIGRTMAX-13 52) SIGRTMAX-12
53) SIGRTMAX-11 54) SIGRTMAX-10 55) SIGRTMAX-9  56) SIGRTMAX-8  57) SIGRTMAX-7
58) SIGRTMAX-6  59) SIGRTMAX-5  60) SIGRTMAX-4  61) SIGRTMAX-3  62) SIGRTMAX-2
63) SIGRTMAX-1  64) SIGRTMAX
[sjas@nb ~]$

All others can try man 7 signal.

usage

If you know the process by id (either from ps or top or the like):

kill -<signalname> <PID>

#i.e. these are the same
kill -kill 1234
kill -9 1234

If you know only the process name:

pkill -<signalname>

#i.e.
pkill -hup dnsmasq
less syntax highlighting
posted on 2014-08-08 23:44:09

This works on debian:

$ apt-get install source-highlight -y
$ export LESSOPEN="| /usr/share/source-highlight/src-hilite-lesspipe.sh %s"
$ export LESS=' -R '

Put the export lines into your .bashrc to make them stick.

bash heredoc to file
posted on 2014-08-08 09:36:34

To append a heredoc to an existing file:

$ cat << EOHD >> myfile.txt
> line 1
> line 2
> line 3
> line 4
> EOHD

This works on a prompt, and will add this:

line 1
line 2
line 3
line 4

to file 'myfile.txt'.

This was shown on a prompt. To have the same effect from within a script, insert this into your file:

#!/bin/bash
cat << EOHD >> myfile.txt
line 1
line 2
line 3
line 4
EOHD

Make it executable (chown 755 <name-of-script.sh>), and run it (./<name-of-script.sh>).

Create password and copy it into clipboard
posted on 2014-08-07 09:51:29

To create a password and put it into you clipboard immediatly, use this line in your window manager's global shortcuts:

echo `\pwgen -cn 18 -1` | cut -d' ' -f1 | tr -d "\n" | xclip   
List file contents of all files in a folder
posted on 2014-08-06 18:52:29

To show all file contents of all files contained within a folder, you can of course use a loop:

for i in *; do cat $i; done

However, it might be helpful to know from which file which content originated:

for i in *; do echo $i; cat $i; done'

Since this is better, maybe still a bit crowded on the screen, how about coloring the file names?

for i in *; do echo $'\e[1;33m'$i$'\e[0m'; cat $i; done

This will show the same output as previously, but the filenames will show up in yellow.

This is due to the shell escape codes $'\e[1;33m' and $'\e[0m'. The first one tells the shell to use yellow font, the second tells to quit all extra formatting. Else everything would show up yellow.

Besides 1;33 for yellow, these exist: (and maybe even more, I do not know)

Black         0;30
Dark Gray     1;30
Red           0;31
Light Red     1;31
Green         0;32
Light Green   1;32
Brown         0;33
Yellow        1;33
Blue          0;34
Light Blue    1;34
Purple        0;35
Light Purple  1;35
Cyan          0;36
Light Cyan    1;36
Light Gray    0;37
White         1;37

\e[ stands for a literal escape symbol, IIRC.

More info on the shell control sequences can be found here and here.

linux file attributes
posted on 2014-08-03 11:32:52

These should be present for ext2 / ext3 file systems. No idea for ext4 or if these are present across all unices.

a : append only
c : compressed
d : no dump
e : extent format
i : immutable
j : data journalling
s : secure deletion
t : no tail-merging
u : undeletable
A : no atime updates
C : no copy on write
D : synchronous directory updates
S : synchronous updates
T : top of directory hierarchy
ssh tricks links
posted on 2014-07-31 11:30:59

Really nice articles and comments:

  1. https://news.ycombinator.com/item?id=1624010
  2. http://www.symkat.com/ssh-tips-and-tricks-you-need
  3. https://news.ycombinator.com/item?id=1536126
  4. https://pthree.org/2011/07/22/openssh-best-practice/
Bash dollar sign shell variables
posted on 2014-07-25 11:51:54

Cheatsheet shamelessly stolen from here.

VARIABLE    MEANING
$0          Filename of script
$1          Positional parameter #1
$2 - $9     Positional parameters #2 - #9
${10}       Positional parameter #10
$#          Number of positional parameters
"$*"        All the positional parameters (as a single word) ***
"$@"        All the positional parameters (as separate strings)
${#*}       Number of positional parameters
${#@}       Number of positional parameters
$?          Return value
$$          Process ID (PID) of script
$-          Flags passed to script (using set)
$_          Last argument of previous command
$!          Process ID (PID) of last job run in background

*** Must be quoted, otherwise it defaults to $@.
useradd cheatsheet
posted on 2014-07-23 11:38:55

This topic is already covered in more depth in an earlier post here, but now I figured a cheatsheet would help.

SYSTEM USER
    system user privileges? (UID below 1000)
    -r

HOME FOLDER
    create home folder in /home/<username>?
    -m
    no home folder creation:
    -M
    add existing folder as home:
    -d <folder>

    add contents to created home?
    -k <'skeleton' folder containing data>

GROUPING
    create new user group?
    -U
    add to existing group?
    -g <id or groupname>
    add several groups?
    -G <groups separated by comma's>
    don't create group? (user will be added to group with id 100 usually, see manpage)
    -N 

SHELL
    shell access? (use appropriate shell, /bin/sh for system users if login is needed)
    -s /bin/bash
    no shell access?
    -s /bin/false
    no shell access, with notification?
    -s /sbin/nologin

COMMENT
    -c 'comment explaining user usage'
linux: cat to clipboard
posted on 2014-07-21 11:13:46

To put the contents of a file directly into the clipboard, there exist several different ways. One possibility is to mark, CTRL-C or SHIFT-DEL, or whatever is used in you application for copying.

Applications like Klipper, besides providing the functionality of having a memory, also enable the system to copy every selection you make (with your mouse) into the clipboard.

All this is helpful, but once you have content that spans several screen pages, this gets old pretty fast.

Solution on debian: xclip

$ sudo apt-get install xclip

Usage:

$ echo test | xclip     ## clipboard contains now string 'test'
$ cat file.txt | xclip  ## clipboard contains content of file 'test.txt'
bash emacs shell shortcuts
posted on 2014-07-04 13:19:19

Linux comes around usually with the bash shell. By default bash comes with the possibility to enable vi or bash shortcuts. (Google set -o emacs vs. set -o vi for more info.)

Since vi mode is a bit strange to use (No possibility to see which mode you are in, maybe with zsh this could be changed?) I stick to emacs bindings.

Most useful are:

# CHOOSING
CTRL - P    previous command (previous line)
CTRL - N    next command (next line)

# SEARCHING
CTRL - R    incremental search in command history

# MOVING
CTRL - A    beginning of line
CTRL - E    end of line
ALT - B     backward one word
ALT - F     forward one word
CTRL - B    backward one character
CTRL - F    forward one character
ALT - A beginning of sentence
ALT - E end of sentence

# DELETING / CUTTING / "KILLING"
CTRL - D    next char
CTRL - H    previous char
ALT - D     next word
ALT - BSPC previous word
CTRL - W    previous word (not preferred, as it won't work in emacs with evil-mode enabled :o))
CTRL - K    from cursor to end of line
CTRL - U    from cursor to beginning of line

# INSERTING / "YANKING"
CTRL - Y    put paste buffer contents back at cursor location

# UPPERCASE
ALT - U     uppercase next word
ALT - L     lowercase next word

# TRANSPOSE
CTRL - T    last two characters
ALT - T     last two words

# AUTOCOMPLETION
ALT - *     insert all possible completions
ALT - ?     show all possible completions
grep 'or' syntax
posted on 2014-06-03 14:19:34

To search for several matches at once, try one of the following:

$ grep -e '<first_search_term>' -e '<second_search_term>' <filename>

or

$ egrep '<first_search_term> | <second_search_term>' <filename>

Lastly, there are the several regexp specifiers.

$ grep -E '<first_search_term> | <second_search_term>' <filename>

-E flags the search terms to be interpreted as POSIX regular expressions. There's also -P for perl-regexp's, and -F and -B. -B is the default, if omitted, you can just use:

$ grep '<first_search_term> \| <second_search_term>' <filename>

Choose wisely! ;)

Create mails in bash
posted on 2014-05-09 20:17:25

Write this directly on your command prompt:

/usr/bin/mail -s "testmail" root 'mailaddress@domain.tld' -a "From: mail_daemon" <<< "ti ta testmail"

Which will create this:

To: <root@hostname>, <mailaddress@domain.tld>
Subject: testmail
From: <mail_daemon@hostname>

ti ta testmail

This is useful when you already have a postfix (or whatever maildaemon) running, and you need email notification in your scripts.

dd Howto and some MBR tricks
posted on 2014-05-09 19:00:18

dd is used to "convert and copy files" under linux.

Basic Stuff

Read: Use it for disk images. I.e. put an .ISO on an USB stick. Or copy whole HDD's. Or create ISO's.

# use 'mount' to find out where the stick is mounted

# copy iso onto usb stick
dd if=<isoname>.iso of=/dev/sdX

# create an iso
dd if=/dev/cdrom of=/home/<username>/Desktop/<isoname>.iso

# wipe clean
dd if=/dev/zero of=/dev/sdX

# make a file of 100 random bytes
dd if=/dev/urandom of=/home/<username>/my.random bs=1 count=100

That was it with the usual suspects. Now onto more serious stuff.

Serious Stuff

# PRO: create image from one host and stream to the other
#on target host
netcat -l -p 1234 | dd of=/dev/hdc bs=16065b
#on source host (where the ISO will be created and streamed)
dd if=/dev/hda bs=16065b | netcat <targethost-IP> 1234

#or use this:
# from remote to local
rsh 192.168.xx.yy "dd if=/dev/sda ibs=4096 conv=notrunc,noerror" | dd of=/dev/sda obs=4096
# or from local to remote
dd if=/dev/sda ibs=4096 conv=notrunc,noerror | (rsh 192.168.xx.yy dd of=/dev/sda obs=4096)

In the above you may use ssh, or rsh. Do as you please. Of course you may use different IP's.

And now...

Very Serious Stuff

# PRO: show your MBR
dd if=/dev/sda count=1 | hexdump -C

# PRO: back up MBR
dd if=/dev/sda of=mbr.bin count=1

    #put this on a floppy you make with
    dd if=boot.img of=/dev/fd0
    #along with dd. Boot from the floppy and
    dd if=mbr.bin of=/dev/sda count=1
    #will restore the MBR.


# PRO: command to read your BIOS and all interfaces
dd if=/dev/mem bs=1k skip=768 count=256 2>/dev/null | strings -n 8

on decrypting an MBR

Use file. Just do it:

dd if=/dev/sda of=mbr.bin
file mbr.bin

will give you something like this:

mbr.bin: x86 boot sector; partition 1: ID=0xee, starthead 0, startsector 1, 234441647 sectors, extended partition table (last)\011, code offset 0x0

which is easily read and understandable.

rsync howto
posted on 2014-05-07 14:50:43

Usually when using rsync you want to use it like this:

rsync -avzh --progress server1:/path/to/file server2:/path

This does not preserve hard links, use the -H option, if you need this, too.

man [
posted on 2014-05-04 23:30:22

Whenever writing bash if-clauses, this will be your new best friend:

$ man [
bash heredoc
posted on 2014-05-04 22:45:26

Bash's heredoc functionality provides functionality to work with stream literals.

Usages:

# 1. basic case with parameter expansion
command << DELIMITER_YOU_CHOOSE
this here
is input
which is
line for line
piped into command
DELIMITER_YOU_CHOOSE

# 2. no parameter expansion due to the " "
command << "DELIMITER_YOU_CHOOSE"
    echo $SHELL
DELIMITER_YOU_CHOOSE

# 3. same as above, but will strip leading TAB's
# Use TAB's. DON'T use spaces!
command <<- "DELIMITER_YOU_CHOOSE"
    echo $SHELL
DELIMITER_YOU_CHOOSE

# 4. parameter expansion, strip leading TAB's
command <<- DELIMITER_YOU_CHOOSE
    echo $SHELL
DELIMITER_YOU_CHOOSE

# 5.  command <<< evaluated_command, some kind of shortcut
command <<< cat $SHELL

Results:

# 1.
this here
is input
which is
line for line
piped into command

# 2.
    echo $SHELL

# 3.
echo $SHELL

# 4.
/bin/bash

# 5.
/bin/bash

In general, the usage depends on the type of the input source: - < is for files - << is for typed stream literals - <<< is for evaluated commands

This is a possibility to pass several lines of arguments to command expecting input in several stages. I.e. for scripted certificate generation this comes in handy.

bash redirection
posted on 2014-05-04 21:18:29

The bash shell provides three differnt file handles by default. These are called file descriptors in bash.

  • /dev/stdin = 0 = read input from shell prompt
  • /dev/stdout = 1 = stuff printed to shell
  • /dev/stderr = 2 = error messages channel

& is 1 and 2 combined.

For I/O redirection all these can be accessed through piping:

command  <  file        # redirect file to STDIN
command <<  heredoc     # redirect heredoc to command STDIN
command <<< herestring  # redirect herestring to command STDIN
         >  file        # BEWARE: truncates/deletes file contents!
command 1>  file        # redirect command STDOUT to file
command 2>  file        # redirect command STDERR to file
command &>  file        # redirect command STDOUT and STDERR to file
command  >| file        # redirect command STDOUT forcefully
command  >> file        # like '>', but will append instead of overwrite in file

Forceful redirection will write, even when bash has noclobber set. 'noclobber' prevents files from getting overwritten when redirection operators are used.

heredoc's and herestrings's will go into a separate post, soon.

Sidenotes:

# redirection between handles, these are equivalent:
command &>  file        # redirect STDOUT and STDERR
command >   file 2>&1   # redirect STDOUT as well as STDERR to STDOUT

In more detail:

# redirects "i" to "j", if "i" is omitted, defaults to "1"
        i>&j

i,j range from 1 to 9, 3-9 are free to use. Beware, "5" is inherited by child processes and exec usage.

Plus:

# close file output descriptor "i"
        i>&-
# close file input descriptor "j"
        j<&-

and

# these are equivalent

command < input-file > output-file

< input-file command > output-file   # Although this is non-standard.

Files can also be opened for reading AND writing simultanously via <>. Also any file descriptor ID can be used with.

exec can be used to redirection for the complete current shell. See here.

Also you can much more with duplicating, closing or moving file descriptors. But this is stuff for another post when I need it. In the meantime, this bash reverse proxy is really 'wickedly cool'.

bash multitasking
posted on 2014-05-04 17:50:46

When working in the shell under linux and having started a long-running process, which blocks the shell (and you working in it), gives you some options:

  • wait until process is finished, you land at the prompt, you can work on (NO!)
  • open a new shell window, which is no problem when working under a graphical window manager (MAYBE...)
  • use bash's multitasking capabilities (YES!)

How?

First the shortcuts:

Ctrl-C kills the process currently in the foreground.
Ctrl-Z "suspends"/pauses the process currently running in the foreground and puts it into background.
Ctrl-Y suspend job the next time it asks for user input.

The last two differ in such a way, as suspending via Ctrl-Z may swallow pending shell output, whereas Ctrl-Y will not.

Now the commands:

$ jobs              # list all background processes
$ fg                # start marked process in foreground (see '+' on `jobs` list)
$ bg                # start marked process in background (see '+' on `jobs` list)
$ fg %x             # start process with id 'x' in foreground
$ %n                # alias for fg %n
$ bg %x             # start process with id 'x' in background
$ %n &              # alias for bg %n
$ kill %x           # kill process with id 'x'
$ <processname> &   # '&' will start a process and let it run in background

Example usage: (demonstrated via bash's sleep, which just waits for the specified time in seconds)

[sjas@lorelei ~]% sleep 100
    CTRL-Z
[1]  + 7848 suspended  sleep 100

[sjas@lorelei ~]% sleep 200
    CTRL-Z
[2]  + 10920 suspended  sleep 200

[sjas@lorelei ~]% sleep 300
    CTRL-Z
[3]  + 3676 suspended  sleep 300

[sjas@lorelei ~]% sleep 400
    CTRL-Z
[4]  + 10012 suspended  sleep 400

[sjas@lorelei ~]% jobs
[1]    suspended  sleep 100
[2]    suspended  sleep 200
[3]  - suspended  sleep 300
[4]  + suspended  sleep 400

[sjas@lorelei ~]% bg
[4]    10012 continued  sleep 400

[sjas@lorelei ~]% jobs
[1]    suspended  sleep 100
[2]  - suspended  sleep 200
[3]  + suspended  sleep 300
[4]    running    sleep 400

[sjas@lorelei ~]% kill %1
[1]    7848 terminated  sleep 100

[sjas@lorelei ~]% jobs
[2]  - suspended  sleep 200
[3]  + suspended  sleep 300
[4]    running    sleep 400

[sjas@lorelei ~]% fg
[3]    3676 continued  sleep 300
    CTRL-Z

[sjas@lorelei ~]% jobs
[2]  - suspended  sleep 200
[3]  + suspended  sleep 300
[4]    running    sleep 400

[sjas@lorelei ~]%

There is also disown for removing jobs from the list of jobs and other actions. In case of interest man bash and search for disown.

Have fun.

Linux file packers
posted on 2014-05-03 08:15:08

Cheatsheet for zipping/unzipping:

# if you can install programs
$ unp <archive>

unp will determine the filetype etc. by itself and just works.

Else:

## EXTRACT

# .tar.gz
$ tar xzvf <archive>

# .tar.bz2
$ tar xjvf <archive>

And:

## COMPRESS

# .tar.gz
$ tar czvf <archive>

# .tar.bz2
$ tar cjvf <archive>
Working with linux users
posted on 2014-05-01 18:49:21

Usually you want this when creating a new basic user on a linux system:

$ useradd -m -U <username>

-m creates a homefolder in /home/<username>, -U creates a group with the same name as the specified. -U can be omitted, since it's the default, but what if the defaults on the system you are working on have been changed by someone else?

In case you want to initially revoke login rights, use

$ useradd -m -U -s /usr/sbin/nologin <username>

where -s sets the shell to /usr/sbin/nologin. Another possibility is to use /bin/false. The /bin/false setting will just prevent logging in without an error message, whereas /usr/sbin/nologin will either print This account is currently not available. or whatever is specified in /etc/nologin.txt.

For 'system' users you should use /bin/false if no login is needed, or /bin/sh if it is.

But what if these need no homefolder? Or a group with the same name as their username?

$ useradd -r -d /tmp -G <group> -s /bin/false <username>

The example above will create a user that cannot login and has no dedicated homefolder, and a user id residing within the system user id range. /etc/passwd will show /tmp as homefolder, but thats about it. -r classifies the user as a system user, means he has an user / group id below 1000 usually.
This depends on the linux distribution you are using, IIRC, i.e. in Debian see here.

This has to do with internal filtering, so regulars can be distinguished from system users by id.

Maybe you have to have to set an account for a real person, so lets also specify also his full name in a comment via -c and give him access to a proper shell:

$ useradd -m -U -G <group1>,<group2> -s /bin/bash -c "<REAL NAME>" <username>

Specifying the shell is important, else he will have only a sh, not a bash. If you want, you can change this behaviour in /etc/defaults/useradd.

How about adding the user to groups, too?

$ useradd -m -U -G <group1>,<group2> -s /bin/bash -c "<REAL NAME>" <username>

Changing user settings can either be done editing /etc/passwd, but DO NOT DO THIS DIRECTLY!
Use vipw instead, it will lock the file so concurrent updates are impossible. Same goes for /etc/groups, use vigr for this.
For editing the shadow files like /etc/shadow (users) or /etc/gshadow (groups) use vipw -s and vigr -s.
Changing sudo rights is done via visudo.

You can also use one of the following:

chsh = change a user's shell
chfn = change user information such as real name and more
passwd = change user's password
usermod = change users's properties
userdel = delete a user

For userdel you usually want to use userdel -r <username> so the mail and the homefolder will be deleted. Keep in mind this might also delete the dedicated user group .

For creating users in batch use newusers.

A proper du / disk usage alias
posted on 2014-04-26 21:18:50

This finds you all files in the current folder, sorts them from biggest to lowest, and puts human readable file sizes on it.

function dus () {
du --max-depth=0 -k * | sort -nr | awk '{ if($1>=1024*1024) {size=$1/1024/1024; unit="G"} else if($1>=1024) {size=$1/1024; unit="M"} else {size=$1; unit="K"}; if(size<10) format="%.1f%s"; else format="%.0f%s"; res=sprintf(format,size,unit); printf "%-8s %s\n",res,$2 }'}

Usage:

$ dus

Sample output:

[sjas@ctr-014 ~]% dus
3.1G     Downloads
1.7G     VMware-vCenter-Server-Appliance-5.5.0.5100-1312297-system.vmdk
1.4M     blog
576K     work
80K      hs_err_pid25560.log
80K      hs_err_pid24938.log
8.0K     bin
4.0K     yankring_history_v2.txt

This should be included in all linux distros by default.

Editing shell commands in vim
posted on 2014-04-26 18:41:02

Having a long shell command, wishing for an easier way to edit it? Not wanting to have the vim bindings (set -o vi IIRC) enabled in bash, since they, lets face it, could use alot of improvement? (Or a proper zsh prompt, so you see which mode you are currently in, maybe?)

$ Ctrl+X, Ctrl+E

Press this while being in the bash console. Once you save and exit vim, you have your output changes in the shell again, and off you go.

Generating system noise
posted on 2014-04-26 17:50:17

When generating certificates with openssl, trouble might arise as there is not enough randomness on your system.

Fire up a new console and try:

$ find /

This should do. Ctrl+C,Ctrl+D when the keygen is done.

Find out which linux distro & version you are running
posted on 2014-04-08 09:21:20

Ever wondered what linux distribution or which version you are running?
Try:

$ lsb_release -a

or

$ cat /etc/*-release

This works on Fedora and Debian at least, haven't tested it on other distributions.

bash screen essentials
posted on 2014-03-17 14:41:15

According to its man page GNU screen is a

full-screen window manager that multiplexes a physical terminal between several processes [...] There is a scrollback history buffer for each virtual terminal and a copy-and-paste mechanism that allows moving text regions between windows.

which sounds quite interesting.
All this means you can get several terminals into a single shell window. Other nice functions are detachable windows, which is a killer feature for unstable ssh sessions. Tmux (http://blog.hawkhost.com/2010/06/28/tmux-the-terminal-multiplexer/) seems to be the the better alternative nowadays, but sometimes you might have to stick to screen.

$ screen

will invoke the program. There are lots of options possible here, see man screen.

Ctrl-a is by default the global hotkey. All screen commands have to be prefixed with it.

In screen there exist windows and regions. Windows are like several shells running in parallel. Regions are like the two window areas when you use split screen.

basic usage

Most important are:

? for help
| for creating a vertical split region
S for creating a horizontal split region
tab for switching to another region
w for showing a list of windows in the titlebar
" for showing a list of windows in a window, use j or k and enter for navigating
c for starting a new shell
a for changing the buffer within a window
X for killing a region (leaving the window alive)
k for killing a window (leaving the region intact)
\ for exiting screen

That is about it.

Open screen, create a region, switch to it, create a new shell window, be happy.

ssh

## create screen with session named '<sessionname>'
screen -S <sessionname>

## detach a session (while running screen)
ctrl-a d

## show available screen sessions
screen -list

## reattach a session (from the shell)
screen -r <sessionname>

## reattach if session wasn't detached earlier
## happens when you accidentally closed the window
## or when connection went broke
screen -d -r <sessionname>

If you have to copy huge amounts of data or have other long running screen sessions that should not be interrupted, screen will have you covered, literally.

Distinguish builtin shell functions, aliases, functions and commands
posted on 2014-01-20 10:04:41

If you have a grown .bashrc and wonder what commands you did define in the past, these are helpful:

bash

type -t

Will show you what exactly you are dealing with. (Builtin, alias, function, regular command.)

[jl@jerrylee ~]$ type -t git
file

[jl@jerrylee ~]$ type -t export
builtin

[jl@jerrylee ~]$ type -t gc
alias

gc is an alias which I have locally defined in my .bashrc. It has a function bound to itself as we will see.

Built-ins are looked up a the main man page. (I.e. man bash or man zsh.)

alias

[jl@jerrylee ~]$ alias gc
alias gc='gitcommit'

If alias is used with no string afterwards, it will push out a complete list of all defined aliases.

declare / typeset

To look up functions:

[jl@jerrylee ~]$ declare -f gitcommit
gitcommit () 
{ 
    git c "$*"
}
[jl@jerrylee ~]$ typeset -f gitcommit
gitcommit () 
{ 
    git c "$*"
}

declare -f and typeset -f are synonymous.

zsh

Here you easiest start with which.

Besides, all other commands are the same.

Extract list of classes being used in legacy java project
posted on 2013-12-28 09:55:59

Currently I am working with a small sized legacy code base. To get a better overview, the actual LOC (lines of code) might be of interest:

# all lines including the whitespace
time \grep '.*' * -rc | cut -d ':' -f 2 | paste -sd+ | bc

Stripping the blank lines is left as an exercise to the reader.

This is ugly, but blazingly fast. time is just in there to see how fast things actually are.

Also a sorted list of all self-defined classes might make for a handy overview:

ack -h 'public class' | sed -e 's/^\s*//g' | cut -d " " -f 3-5 | sort | sort -k 2,3

Do yourself a favor, and use ack instead of grep. Nothing to regret in 99% of all use cases...

"git commit <commitmessage>" without quotation marks
posted on 2013-12-09 23:05:46

Using git exactly how you want it to, is best done from within a shell. (At least in my former experiences, maybe the git clients improved vastly by now?)

EGit (the eclipse plugin), SmartGit, the Github clients all did not satisfy me. Eiter the clients lacked functions (which were needed and I was back to commandline anyway) or the functions did not behave as expected: Changing files permissions, but nothing otherwise, failing merges that were completely doable in vanilla git, ...) I have had my share, especially with EGit.

But typing 'git commit -am "this is my commitmessage"' gets old over time, too.

A way to fix this is to put this in your .bashrc:

function gc { git commit -am "$*" }

If you use another shell, you might want to put it in whatever initialization file gets executed upon shell startup. On a Mac, you might want to remember this difference to usual linux.

This will make for a nice shortcut:

# OLD
$ git commit -am "this is a long commit message and i do not want to type the quotation marks"

# NEW (I have 'gc' as memo for 'git commit', choose as you like.)
$ gc so this works now without quotation marks

This has to be done through a bash function, with an alias definition it will not work. Of course you can define bash aliases for other git commands you use more often (You bet I did. Actually a ton if it, considering you much I use git nowadays.).

Somewhere in the bash manual it is said to be written [that you should prefer functions over aliases]. Got not linux at hand (cygwin at this moment) and no bash manual for grepping here, so you have to look it up yourself in case you do not trust my hearsay.

Run `git pull` on all folders in current folder
posted on 2013-11-12 04:07:16

To pull newest changes into each of the repositories in your current folder, try this: (All folders lie directly one level below. No recursive check.)

for i in *; do cd $PWD/"${i}" && git pull | cat && cd ..; done

This was kind of tricky to figure out. Important was $PWD else the cd will not work properly, and the cat instead of echo so the console output gets shown.

Run bash from Java
posted on 2013-11-11 08:12:38

Snippet from stackoverflow:

process p = Runtime.getRuntime().exec("bash_script.sh");
BufferedReader in = new BufferedReader(new InputStreamReader(p.getInputStream()));
String line = null;

while ((line = in.readLine()) != null) {
   // use bash script line output
}

I hope I do never have to resort this... but I have doubts. Since this works not only for shellscripts, but basically any executable producing console output. Will have to test this on different platforms i bet, though. Just to make it sure it runs where it has to.

bash array essentials
posted on 2013-10-20 18:24:10

bash is not just the name of a shell terminal, it is also a full-blown programming language. It has some very ugly sides from my point of view. This here resulted from some commandline work, where a number of filenames was needed. I know I will need a refresher on this again.

Note: In the following, arr is the name of our array. Choose whatever you fancy.

get all folder/filenames into an array

arr=(/path/to/wherever/*)

Do not use ls for this, google for the reasons. I forgot why and do not have a link. ;)

access the array to return an element, i.e. the first one. arrays start at 0.

${arr[0]}

If in doubt, use braces and not parentheses.

print first array element to console

echo ${arr[0]}

Just for the record, since the above does just return the item, but bash cannot process it without another command.

show array contents (i.e. for checking the actual data)

declare -p arr

Handiest command for checking arrays' contents. Force yourself to use it!

number of items in the array

${#arr[*]}

Useful for numbered for-loops.

all items in the array

${arr[*]}

Useful for foreach-type loops.

all of the indexes in the array

${!arr[*]}

Useful for... Off the hat I got no idea how this could help.

length of item zero

${#arr[0]}

See last description.

This blog covers .csv, .htaccess, .pfx, .vmx, /etc/crypttab, /etc/network/interfaces, /etc/sudoers, /proc, 10.04, 14.04, AS, ASA, ControlPanel, DS1054Z, GPT, HWR, Hyper-V, IPSEC, KVM, LSI, LVM, LXC, MBR, MTU, MegaCli, PHP, PKI, R, RAID, S.M.A.R.T., SNMP, SSD, SSL, TLS, TRIM, VEEAM, VMware, VServer, VirtualBox, Virtuozzo, XenServer, acpi, adaptec, algorithm, ansible, apache, apachebench, apple, arcconf, arch, architecture, areca, arping, asa, asdm, awk, backup, bandit, bar, bash, benchmarking, binding, bitrate, blackarmor, blowfish, bochs, bond, bonding, booknotes, bootable, bsd, btrfs, buffer, c-states, cache, caching, ccl, centos, certificate, certtool, cgdisk, cheatsheet, chrome, chroot, cisco, clamav, cli, clp, clush, cluster, coleslaw, colorscheme, common lisp, console, container, containers, controller, cron, cryptsetup, csync2, cu, cups, cygwin, d-states, database, date, db2, dcfldd, dcim, dd, debian, debug, debugger, debugging, decimal, desktop, df, dhclient, dhcp, diff, dig, display manager, dm-crypt, dmesg, dmidecode, dns, docker, dos, drivers, dtrace, dtrace4linux, du, dynamictracing, e2fsck, eBPF, ebook, efi, egrep, emacs, encoding, env, error, ess, esx, esxcli, esxi, ethtool, evil, expect, exportfs, factory reset, factory_reset, factoryreset, fail2ban, fbsd, fedora, file, filesystem, find, fio, firewall, firmware, fish, flashrom, forensics, free, freebsd, freedos, fritzbox, fsck, fstrim, ftp, ftps, g-states, gentoo, ghostscript, git, git-filter-branch, github, gitolite, gnutls, gradle, grep, grml, grub, grub2, guacamole, hardware, haskell, hdd, hdparm, hellowor, hex, hexdump, history, howto, htop, htpasswd, http, httpd, https, i3, icmp, ifenslave, iftop, iis, imagemagick, imap, imaps, init, innoDB, innodb, inodes, intel, ioncube, ios, iostat, ip, iperf, iphone, ipmi, ipmitool, iproute2, ipsec, iptables, ipv6, irc, irssi, iw, iwconfig, iwlist, iwlwifi, jailbreak, jails, java, javascript, javaws, js, juniper, junit, kali, kde, kemp, kernel, keyremap, kill, kpartx, krypton, lacp, lamp, languages, ldap, ldapsearch, less, leviathan, liero, lightning, links, linux, linuxin3months, lisp, list, livedisk, lmctfy, loadbalancing, locale, log, logrotate, looback, loopback, losetup, lsblk, lsi, lsof, lsusb, lsyncd, luks, lvextend, lvm, lvm2, lvreduce, lxc, lxde, macbook, macro, magento, mailclient, mailing, mailq, manpages, markdown, mbr, mdadm, megacli, micro sd, microsoft, minicom, mkfs, mktemp, mod_pagespeed, mod_proxy, modbus, modprobe, mount, mouse, movement, mpstat, multitasking, myISAM, mysql, mysql 5.7, mysql workbench, mysqlcheck, mysqldump, nagios, nas, nat, nc, netfilter, networking, nfs, nginx, nmap, nocaps, nodejs, numberingsystem, numbers, od, onyx, opcode-cache, openVZ, openlierox, openssl, openvpn, openvswitch, openwrt, oracle linux, org-mode, os, oscilloscope, overview, parallel, parameter expansion, parted, partitioning, passwd, patch, pdf, performance, pfsense, php, php7, phpmyadmin, pi, pidgin, pidstat, pins, pkill, plesk, plugin, posix, postfix, postfixadmin, postgres, postgresql, poudriere, powershell, preview, profiling, prompt, proxmox, ps, puppet, pv, pvecm, pvresize, python, qemu, qemu-img, qm, qmrestore, quicklisp, r, racktables, raid, raspberry pi, raspberrypi, raspbian, rbpi, rdp, redhat, redirect, registry, requirements, resize2fs, rewrite, rewrites, rhel, rigol, roccat, routing, rs0485, rs232, rsync, s-states, s_client, samba, sar, sata, sbcl, scite, scp, screen, scripting, seafile, seagate, security, sed, serial, serial port, setup, sftp, sg300, shell, shopware, shortcuts, showmount, signals, slattach, slip, slow-query-log, smbclient, snmpget, snmpwalk, software RAID, software raid, softwareraid, sophos, spacemacs, spam, specification, speedport, spi, sqlite, squid, ssd, ssh, ssh-add, sshd, ssl, stats, storage, strace, stronswan, su, submodules, subzone, sudo, sudoers, sup, swaks, swap, switch, switching, synaptics, synergy, sysfs, systemd, systemtap, tar, tcpdump, tcsh, tee, telnet, terminal, terminator, testdisk, testing, throughput, tmux, todo, tomcat, top, tput, trafficshaping, ttl, tuning, tunnel, tunneling, typo3, uboot, ubuntu, ubuntu 16.04, udev, uefi, ulimit, uname, unetbootin, unit testing, upstart, uptime, usb, usbstick, utf8, utm, utm 220, ux305, vcs, vgchange, vim, vimdiff, virtualbox, virtualization, visual studio code, vlan, vmstat, vmware, vnc, vncviewer, voltage, vpn, vsphere, vzdump, w, w701, wakeonlan, wargames, web, webdav, weechat, wget, whois, wicd, wifi, windowmanager, windows, wine, wireshark, wpa, wpa_passphrase, wpa_supplicant, x2x, xfce, xfreerdp, xmodem, xterm, xxd, yum, zones, zsh

View posts from 2017-03, 2017-02, 2017-01, 2016-12, 2016-11, 2016-10, 2016-09, 2016-08, 2016-07, 2016-06, 2016-05, 2016-04, 2016-03, 2016-02, 2016-01, 2015-12, 2015-11, 2015-10, 2015-09, 2015-08, 2015-07, 2015-06, 2015-05, 2015-04, 2015-03, 2015-02, 2015-01, 2014-12, 2014-11, 2014-10, 2014-09, 2014-08, 2014-07, 2014-06, 2014-05, 2014-04, 2014-03, 2014-01, 2013-12, 2013-11, 2013-10


Unless otherwise credited all material Creative Commons License by sjas