Posts from 2016-01

AWK
posted on 2016-01-29 23:54:30

intro

awk is one hell of a beast. It's named after it's inventors Aho, Weinberger and Kernighan, there exist different implementations. This post is to give a rough overview, there is more to it.

Either look up the official documentation for your installed implementation (i.e. mawk is just not gawk), or try heading over here, which is where I look things sometimes up when I happen to need it.

Pro's:

  • It's fast.
  • You have a programming language at your disposal in the shell.
  • One-liners are pretty quickly written.

Con's:

  • rather steep learning curve

When working with text fragments, it can truly speed up things. Since beside a lot of good tutorials always missed a proper introduction for me, this i my shot at creating one myself.

These nice things happen to exist, or can at least be created.

  • variables
  • conditions
  • loops
  • associative arrays
  • functions
  • a profiler (Yes, the software comes with its own profiler built in, depending on the implementation you use)
  • pipes (You can pipe awk arrays directly to shell commands WITHIN awk, which is a nice feature.)
  • arithmetic operators

This list is likely not complete, as this post comes almost completely out of my head.

creating scripts vs. executing statements in the shell

Both is easily possible. The shebang for scripts:

#!/usr/bin/awk -f

Within scripts statements (block stuff within the braces) are separated through newlines, whereas in the shell you need semicolons. You don't need semicolons outside the blocks.

structuring

Usually awk are based on the following structure:

#!/usr/bin/awk -f

BEGIN { ... }
BEGIN { ... }
BEGIN { ... }
CONDITION { ... }
CONDITION { ... }
CONDITION { ... }
CONDITION { ... }
END { ... }
END { ... }
END { ... }

Or on the shell:

awk 'BEGIN { ... } BEGIN { ... } BEGIN { ... } CONDITION { ... } CONDITION { ... } CONDITION { ... } CONDITION { ... } END { ... } END { ... } END { ... }'

The misleading part is, you don't need the BEGIN blocks, the END blocks or the CONDITION's themselves from the CONDITION blocks.

So the program could as well be looking just like this:

awk '{ ... }'

or

awk 'CONDITION'

So in very short:

  • awk processes input line by line usually.
  • BEGIN / END blocks are executed prior or after input processing.
  • The middle part is executed while traversing the input, depending if the condition evaluates to 'true'.
  • If no CONDITION is specified, the block is always executed.
  • several blocks can be used together, all are evaluated.
  • If a condition is true, the current line (called a RECORD, consisting of columns called FIELDS) is printed, even when no block was specified. That also happens to be the case with variable definition. (Sooner or later you will have to debug exactly this case.)

built-in variables

To repeat:

  • RECORD = a single row of your dataset
  • FIELD = a single data entry of a column from the current row

Which explains the variables a bit:

FS          field separator (delimiter for input data, usually ' ')
OFS         output field separator (delimiter for output data)

NF          number of fields = column count
NR          number of record = row count

RS          record separator (how input is delimited, usually '\n')
ORS         output record separator (how output is delimited)

FILENAME    name of the input file

There may be more, but these are mostly implementation-dependant and thus omitted.

user-defined variables

Unlike in bash variables, you omit the $ prepended to the variable names. You have input data, which are usually just strings ("like this"), so you declare variables like this:

example_var_1=""; example_var_2=""

This can either be done in BEGIN or within the main code paths. See next paragraph for some examples.

arrays

All arrays are associative, which lets you emulate regular arrays, too. For these you simply create a variable, defined with value 0. When iterating, simply increase the running index, which is the key for your array values.

Associative arrays are rather easy, one of your fields is a key, the other the value which gets set.

An example for a emulated 'normal' array is this:

awk 'BEGIN {index=0; array=""} {array[index]=$1; index++}'

An example for regular 'associative' usage:

awk 'BEGIN {array=""} {array[$1]=$2}'

CONDITIONS

These could either be simple assignments, or regexes (/ ... /)

built-in functions

This is a quick overview, so you know these exist:

next     jump to next record
exit     quit exit program, an exit code can be specified
getline  for when you need to control getting input
print    self-explanatory
printf   formatted printing like in C
++       increment
function keywork for when you explicitly need user-defined functions

one-liners

Some handy examples are provided here:

## print second column (counting starts with one, not zero)
## this is what you will use awk the most for, don't use `cut`
awk 'print $2'

## print everything EXCEPT second column
awk '$2=""'

## remove empty lines
awk 'NF>0'
# or
awk '$1'

## add a header (printf is analogous to C language)
## the regular print statement could be used as well, of course
awk 'BEGIN {printf "%s %s %s\n","1stcol","2ndcol", "3rdcol"} {print $0}'

## print several columns, use several different delimiters
## both of these work with arbitrary counts for tab, space or colon as delemiter
awk -F'[\t :]+' '{print $1 $2 $3}'
Linux and dynamic tracing rants
posted on 2016-01-27 21:46:34

This is current work in heavy progress.

tracing is just not overly accessible

When wanting to start with dynamic kernel tracing, usual problems are similar, no matter what technology you want to use:

  • "I don't know where to start."
  • "I don't know how XYZ is done in tracing tool ABC."
  • "I don't know what probes exist."
  • "I don't know what syscalls are existing."
  • "I installed the packages, but this doesn't work?"
  • "I need to copy-paste scripts to make this work?"
  • "Heck, I don't even know what syscalls are."
  • "What can I do with all this stuff?"

Usually the syntax ain't even too bad, it's the points above hindering the further spreading of these tools. There is a pattern there to be found, so this post should do this:

  • Show what tracing is and in what shape the tooling landscape is currently.
  • Provide small examples which are usable to get a proper starting point.
  • Provide one-liners for getting overviews over the currently available tools for all probes and trace-points.
  • Provide one-liners to show how to catch syscalls which took place.
  • Provide detailed install instructions where necessary, but rather search non-invasive tools. Some tools are completely integrated into the kernel and thus directly accessible, so the focus is on these.
  • Rather than running script files, statements can directly be run from the command-line when provided correctly.

The last two can be explained rather shortly:

  • Syscalls are the C functions which make up the API by which applications can access the kernel's functions. These are documented in the type (2) man pages, if you did't know yet.

Here's a list, even though they may be called a little differently at times:

SYSCALL   WHAT IT DOES

read      read bytes from a file descriptor (file, socket)
write     write bytes from a file descriptor (file, socket)
open      open a file (returns a file descriptor)
close     close a file descriptor

fork      create a new process (current process is forked)
exec      execute a new program

connect   connect to a network host
accept    accept a network connection

stat      read file statistics
ioctl     set I/O properties, or other miscellaneous functions

mmap      map a file to the process memory address space

brk       extend the heap pointer

If a complete code audit is too heavy (All branches have to be checked after all. And later you find out you overlooked something.), dynamic tracing is for you. You can either find out how many syscalls were run, or what values variables were set to, you can collect data and create graphs from it, ... Actually you can do more than you need, so providing the most use cases should do well enough.

intro to dynamic tracing

What exactly is this dynamic tracing thing? Let's start with some terms which I shamelessly rephrase from a lesser-known but very able russian guy named Sergey Klyaus and his github stuff here:

  • Looking solely at code = static code analysis, sadly this is error-prone and a damn lot of work. There's a reason not many people do kernel development.
  • Watching a system's behaviour at runtime is dynamic analysis, but there are different types of introspection.

There are several methologies:

TODO

  • instrumentalizing
  • sampling
  • profiling
  • tracing

Sergey is truly awesome and knows his stuff. His ebook, though 'it may never be finished' as he said somewhere IIRC, is an outstanding piece of work and has already over 200 pages. The best part is that it is still freely available, and besides some little typos (English is not his mother tongue.) it is a damn good read.

So what technologies are available there will be provided in a short overview. The examples are purposefully short for copy-pasting, so starting with this stuff is easier.

DTrace

After I read a lot of stuff lately from the man, the myth, the legend, @brendangregg, it looks like DTrace is plain awesome. But since adoption on linux may take forever (if it will even happen at all since the open DTrace4Linux port by Paul Fox seems to be pretty much a one-man-show and Oracle's DTrace is just a wrapper around SystemTap, sadly I have no link where I read this), going with the alternatives seems the way to go on linux.

On FreeBSD it seems: 'Just use DTrace.'

On Linux the answer is not just as simple, thus this post might grow quite a bit over the following paragraphs.

usage

For the sake of completeness, here is a bunch of dtrace scripts:

# process plus its arguments
dtrace -n 'proc:::exec-success { trace(curpsinfo->pr_psargs); }'

# files opened by a process
dtrace -n 'syscall::open*:entry { printf("%s %s",execname,copyinstr(arg0)); }'

# syscall count of a program
dtrace -n 'syscall:::entry { @num[execname] = count(); }'

# syscall count by the system
dtrace -n 'syscall:::entry { @num[probefunc] = count(); }'

# syscall count of a process
dtrace -n 'syscall:::entry { @num[pid,execname] = count(); }'

# used memory of a progress
dtrace -n 'io:::start { printf("%d %s %d",pid,execname,args[0]->b_bcount); }'

# count of pages which were swapped by a process
dtrace -n 'vminfo:::pgpgin { @pg[execname] = sum(arg0); }'

eBPF

eBPF is under active development within the linux kernel, latest changes in version 4.4 you can read about here, but kernel developers call these things scary stuff.

Somewhere in a presentation Brendan compared DTrace to eBPF like a kitty hawk to a jet engine, which, besides it being 'in-kernel', should be the reason why it might be most likely be the most important tracer in linux some day.

A little presentation on BPF can be found here.

SystemTap

Until Linux' extended Berkeley Packet Filter (eBPF) is real prime time material, stap should do well, Brendan thought, as could be seen here.

SystemTap has got two modes:

  • Awk/C like language, gets the job done
  • Embedded C mode aka "guru mode" in case you need it

install

Most distributions have prepackaged what you want. Well, at least Debian did, and maybe CentOS, too, IIRC. Afterwards run stap-prep, which should tell you what else you have to install. (Usually you need the debug headers for your kernel, to make systemtap work.)

usage

TODO place some useful oneliners here

# show processes opening files in realtime
# Brendan wrote in his 'Systems Performance' book: "I've never actually seen this work."
# I feel proud, it did for me. ;)
stap -ve 'probe syscall.open { printf ("%30s %-100s\n", execname(), user_string($filename)); }'

explore

# NICE FULL OVERVIEWS

# PROBE TYPES
stap --dump-probe-types | awk -F. 'BEGIN {current=""; print "\n\033[31;1mstap -ve \"global s; probe ... {...}\"\033[0m\n"} {if (current != $1) { current=$1; printf "\n\033[33;1m%s\033[0m\n",current } else {print $0}}' | less -R
# PROBE ALIASES
stap --dump-probe-aliases | awk -F. 'BEGIN {current=""; print "\n\033[31;1mstap -ve \"global s; probe ... {...}\"\033[0m\n"} {if (current != $1) { current=$1; printf "\n\033[33;1m%s\033[0m\n",current } else {print $0}}' | less -R
# PROBE FUNCTIONS
echo $'\n\e[31;1mstap --dump-functions\e[0m\n'; stap --dump-functions

# some other examples, for the sake of completeness
stap -l 'kernel.function("acpi_*")' | sort
stap -l 'module("ohci1394").function("*")' | sort
stap -L 'module("thinkpad_acpi").function("brightness*")' | sort

further stuff

A pretty new example on Heatmaps using stap can be found here and here.

Further you also can export histograms directly to console, which is a damn awesome feature.

perf

According to Brendan, they quite heavily use perf over at netflix. Interestingly neflix runs no own infrastructure anymore, but completely relies on amazon's cloud services instead, I learned somewhere last week. You really got to know how to measure your available performance when doing such stunts, so perf sure sounds like a good idea.

Most stuff which helped me with perf here in a nutshell:

usage

What syscalls are run the most?

perf top

Let's do some profiling. In short, create a baseline data-set of your system first, then start your application and collect a second set of data from your 'system under test' (SUT). Afterwards just compare both collected sets:

perf record -p <PID> -o baseline.data sleep 30
perf record -p <PID> -o SUT.data sleep 30
perf diff baseline.data SUT.data

TODO
perf report -n --stdio

If regular strace is too heavy on your system, give perf trace a try.

This is all you need if you don't want to go down the rabbit hole. If sure, just do proceed:

explore

# check what probes exist at all
perf test

# helps with exploring what is actually possible
## alphabetically, from Brendan
perf list | awk -F':' '/Tracepoint event/ { lib[$1]++ } END { for (i in lib) { printf " %-16s %d\n",i,lib[i] } }' | sort | column
## by count
perf list | awk -F':' '/Tracepoint event/ { lib[$1]++ } END { for (i in lib) { printf " %-16s %d\n",i,lib[i] } }' | sort -nk2 | tac | column
## A PROPER COLORED LIST OF ALL TRACE GROUPS PLUS ITS TRACE POINTS (UGLY AS CAN BE)
perf list | awk -F'[: \t]+' 'BEGIN {current=""} /Tracepoint event/ {if (current != $2) { current=$2; print $2, "\n\t", $3 } else {print "\t", $3}}' | sed -r ''s/^[[:graph:]]+/$(printf "\033[33;1m&\033[0m")/'' | less -R

## SIMILAR, BUT ONLY FOR SYSTEM CALLS (UGLY? HELL, THIS IS EVEN WORSE)
perf list | awk -F'[: \t]+' 'BEGIN {current=""} /Tracepoint event/ {if (current != $2) { current=$2; print $2, "\n\t", $3 } else {print "\t", $3}}' | grep -e syscalls -e sys_enter -e sys_exit | sed -r -e 's/^syscalls/& ( with prefixes: sys_enter_ \/ sys_exit_ )/' -e ''s/^[[:graph:]]+/$(printf "\033[33;1m&\033[0m")/'' -e 's/sys_enter_([[:graph:]])/\1/' -e 's/sys_exit_([[:graph:]])/\1/' | uniq | awk 'BEGIN { flag = 1; id = 0 } /with prefixes:/ { print $0; flag = 0; next; print $0 } { if (flag) {print $0} else {array[id]=$0; id++}} END { for (i in array){print array[i] | "sort" }}' | less -R
linux: systemcheck in 60 seconds
posted on 2016-01-21 00:03

This post is a completely copied from @brendangregg from here. Just in shorter and typed by me in hope I can memorize it easier that way, plus a little change with including htop.

Summary: check a linux system for problems immediatly after ssh'ing onto it.

  1. htop - uptime, core diversity, load, swap on first sight via a TUI.
  2. uptime - for load checking, likely unnnecesary after htop
  3. dmesg | tail - check for errors like out of memory
  4. vmstat 1 - check amount of processes (r) and kernel/userland distribution and swap
  5. mpstat -P ALL 1 - check for a single hot core
  6. pidstat 1 - check for high load on single process
  7. iostat -xz 1 - high r/w load? awaits? util%?
  8. free -m - memory available, likely unneded after htop
  9. sar -n DEV 1 - rxkb/s or txkb/s is 125mbytes max for 1G NICs, util% ok?
  10. sar -n TCP,ETCP 1 - act = egress, pasv = ingress traffic, retransmits = bad, usually
  11. top - zxcV and 1 and < and > are your best friends, along with knowing status indices.

Sidenotes:

At 7. buffers = block device caching, cache = page cache for file system.

At 10. just switch columns through the angle brackets keys and have a look the waits (wa) to see if there are disk related issues, after having pressed 1 to show all available cores. d with a number after changes the refresh time to x seconds. In general everything concerning top can be found in the manual.

Lastly, a list of the process states from the mentioned top man page:

D = uninterruptible sleep <<-- waiting for disk
R = running
S = sleeping
T = traced or stopped
Z = zombie
Weechat + FiSH walkthrough vs. irssi
posted on 2016-01-17 00:30

foreword

For quite some time I used irssi as a console irc client and was quite happy with it. However all good things come to an end, and after a restart of the client (I tend to run it on a server withing a screen session.), irssi would not start anymore.

On another server I compiled both from scratch from the source cloned from the official git repositories, but I could not get FiSH to play with irssi at all. It just complained about ABI version mismatches. Since irssi is not known for its documentation (Honestly, it just sucks.), nothing could be found there, nor through a quick grep through the source or via google. Only a github ticket mentioned ABI problems but that was about it.

Weechat kept coming up as a mentioned alternative, so here we go now.

weechat vs. irssi

  • both are console-based
  • weechat is WAY easier to setup and extend
  • weechat needs less plugins out of the box
  • nicklist integrated automatically (irssi needs screen and a plugin for a buggy implementation of that)
  • bar separator to show last read line automatically in weechat
  • irssi seems to have more drawing bugs
  • weechat has more colors, but this improves readability (at least for me)

install and setup weechat

After getting the last version from pkg (its a FBSD system), simply start it on the shell via weechat.

Then from within, run through the quickstart guide for setting up your network and channel auto settings and stuff as described here.

Don't forget to set up your nick to be highlighted. :)

fish

When in weechat:

# start scripts plugin
/script

# search plugin
via pgdn and up/down search for 'fish.py'

# install
# afterwards 'ia r' is next to the plugin
# which means: 'installed,autostarted,running'
alt + i to install

# close the window afterwards
/close

Now after the plugin is usable, only setting your keys is left to do, like for a channel:

/blowkey set -server NETWORK #CHANNEL PASSWORD

# general help
/help blowkey

usage

In general its almost the same as with irssi, only some things are a bit different:

ctrl-n           next window
ctrl-p           previous window
alt-<number>     jump to window <number>

# pressing several times toggles string and regex search
ctrl-r           search current chat history
alt-n            next search match
alt-p            previous search match

/close           close window
/quit            close weechat
/q               open query
/j               join channel

ctrl-n/p seem not to be mentioned in the weechat documentation, for switching channel windows. Still, you should be using Alt with numbers anyway, helps you a lot when using the hotlist.

Also, Alt+<number> pressed a second time does switch you back to your original window.

copy pasting stuff

alt-l is toggling the copy mode. (Hides the nick list and coloring, awesome.)

Also this might help, "so that new line char is not added at the end of each line displayed (it will not break URL selection)":

/set weechat.look.eat_newline_glitch on

tuning your hotlist

The hotlist is are the cryptic numbers popping up in the statusbar (over your input bar) when activity in other windows beside the currently active one. (It's the [H: ...] part, to be exact. There are numbers shown, signaling the window number followed by a colon, afterwards are the count of new lines is shown in parentheses.)

Since it'd be nice to have alerts only for new messages (and not join/part notifications) and also get the channel names shown, this will come in handy:

/set weechat.look.hotlist_names_level = 14
/set weechat.look.hotlist_names_merged_buffers on
spacemacs: essentials
posted on 2016-01-16 12:20

foreword

Spacemacs is a truly awesome piece of software. It is literally the best of both worlds from vim and emacs.

In the last years I have heavily used (and customized) vim (my .vimrc is about 900 lines). But as soon as you want to do more complex stuff, which involves a lot of plugins, things get ugly and slow. Often you also happen to need a custom compiled vim (so it has python bindings enabled) and bad hackery (vimscript is an absolute traffic accident).

After quite some stints with emacs in the last years (interest in lisp plus the customization which is possible with it, compared to vimscript) and a lot of pulling hair out due to debugging problems (When is a major/minor-mode actually loaded? Packages broken when using the different package universes like melpa/elpa/marmalade together? Evil-mode not working in all buffers even though it was set globally?) I stumbled upon Spacemacs.

common solutions for common problems

helm keybind popup too slow

See ~/.spacemacs:

dotspacemacs-which-key-delay 0.01

syntax highlighting for special files lacking

Put this into ~/.spacemacs in the user-config() section:

(autoload 'markdown-mode "markdown-mode"
  "Major mode for editing Markdown files" t)
(add-to-list 'auto-mode-alist '("\\.post\\'" . markdown-mode))
(add-to-list 'auto-mode-alist '("\\.md\\'" . markdown-mode))

font too large

See ~/.spacemacs, same section:

(defun dotspacemacs/user-config ()
       (set-face-attribute 'default nil :height 80)
)

Here 10 ~= 1pt, so 80 is like 8pt from what I read.

start maximized

See ~/.spacemacs:

dotspacemacs-maximized-at-startup t

custom bindings

Canonical example here shall be using 'jk' simultanously to exit from INSERT mode. For that the key-chord package is needed, plus a little configuration.

See ~/.spacemacs:

(defun dotspacemacs/user-config ()
       (key-chord-define-global "jk" 'evil-normal-state)
       (key-chord-mode t)
       ...
)
(defun dotspacemacs/user-config ()

is the place to go. If you for example need additional packages without grouping them into a layer, put them under dotspacemacs-additional-packages, also in .spacemacs.

dotspacemacs/user-init vs. dotspacemacs/user-config

user-config gets loaded at the end of the parsing of the config file contents, whereas user-init right at the beginning. Usually you want to place code into user-config.

keybinds essentials list (beside original vim stuff)

ESC             abort current menu
C-g             abort current menu
C-h u           go back to previous helm menu

SPC f f         open file
SPC f s         save file

C-x C-c         close spacemacs

# window management
SPC w /         create vertical window split
SPC w -         create horizontal window split
SPC w c         window close

SPC TAB         switch to last buffer
SPC b b         buffer list
SPC b n         next buffer
SPC b p         previous buffer
SPC f d         kill/delete buffer

SPC <number>    jump to window <number>
SPC w arrow     move to window up/left/righ/down
C-w hjkl        move to window up/left/righ/down
SPC w S-arrow   move window up/left/righ/down
SPC w c         close window

# open and reload dotfile
SPC f e d       open
SPC f e R       reload dotfile

# using the help for looking up things, the shortcut is:
SPC h d f       help describe function
SPC h d k       help describe key
SPC h d p       help describe package
SPC h d m       help describe mode
SPC h d v       help describe variable
C-h a           emacs 'apropos'

keybinds additionals list (non-geeks safely forget everything past here...)

HUD

SPC t f         toggle show indicator at 80 chars and disable soft-wrapping
SPC t h h       toggle show cursorline
SPC t i         toggle show indent guide
SPC t h a       toggle automatic symbol highlighting
SPC f t         toggle neo-tree (folder browser)

editing

SPC n r         narrow region (just show current selection)
SPC n f         narrow function (just show current function)
SPC n w         widen again

syntax checking

SPC t s         enable flycheck
grub: boot bsd from iso file
posted on 2016-01-13 01:11

From this gist here I found an example, which is put here for documantation purposes:

menuentry "FreeBSD 9.0 Release" {
set isofile="/boot/iso/FreeBSD-9.0.iso"
loopback loop $isofile
kfreebsd (loop)/boot/kernel/kernel
kfreebsd_module (loop)/boot/mfsroot.gz type=mfs_root
set kFreeBSD.vfs.root.mountfrom=ufs:/dev/md0
}
menuentry "ArchLinux Azriel Remix i686" {
set isofile="/boot/iso/archlinux-2012-07-13-core-i686.is"
loopback loop $isofile
linux (loop)/arch/boot/i686/vmlinuz img_loop=$isofile img_dev=/dev/disk/by-uuid/337A-A94A archisolabel=ARCH_201207 earlymodules=loop
initrd (loop)/arch/boot/i686/archiso.img
}

menuentry "Memtest 86+" {
linux16 /boot/iso/memtest86+-4.20.bin
}
nmap: show available ssl ciphers of a server
posted on 2016-01-04 19:39:00

command

nmap --script ssl-enum-ciphers -p <PORT> <URL>

example

Starting Nmap 6.47 ( http://nmap.org ) at 2016-01-04 15:37 CET
Nmap scan report for sjas.de (78.47.176.149)
Host is up (0.0047s latency).
rDNS record for 78.47.176.149: static.149.176.47.78.clients.your-server.de
PORT    STATE SERVICE
443/tcp open  https
| ssl-enum-ciphers: 
|   SSLv3: 
|     ciphers: 
|       TLS_DHE_RSA_WITH_AES_128_CBC_SHA - strong
|       TLS_DHE_RSA_WITH_AES_256_CBC_SHA - strong
|       TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA - strong
|       TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA - strong
|       TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA - strong
|       TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA - strong
|       TLS_RSA_WITH_AES_128_CBC_SHA - strong
|       TLS_RSA_WITH_AES_256_CBC_SHA - strong
|       TLS_RSA_WITH_CAMELLIA_128_CBC_SHA - strong
|       TLS_RSA_WITH_CAMELLIA_256_CBC_SHA - strong
|     compressors: 
|       NULL
|_  least strength: strong

Nmap done: 1 IP address (1 host up) scanned in 30.54 seconds
mysql: dump table as .csv
posted on 2016-01-04 07:38:01

Sweet and simple:

select * into outfile '<FILENAME>.csv' fields terminated by ';' optionally enclosed by '"' lines terminated by '\n' from <TABLE>;

<FILENAME>.csv is the name of the saved file, which can be found at /var/lib/mysql afterwards. <TABLE> is the name name of the database table you want to export.

RBPI: cpu temp and memory reading
posted on 2016-01-02 17:38:40
# shows the temperature of the cpu
vcgencmd measure_temp

# shows the memory split between the cpu and gpu
vcgencmd get_mem arm && vcgencmd get_mem gpu
flashrom tutorial
posted on 2016-01-02 16:24:55

To directly dump contents of a NOR flash chip directly via the serial peripheral interface bus (SPI), a tool called flashrom will help.

disclaimer

If you read this and want to do what is described, you dont need a disclaimer to know you can kill your hardware through electrostatic discharge or whatever. Else you should not be doing this anyway, except you can afford grilling things and/or insist on learning things. This is likely the only disclaimer on this site for quite some time.

reasons

Why would you want to do that at all?

Flashing new content onto flash chip usually takes place after the chips contents (also containing either the operating system or at least the bootloader or some part of it) were loaded into the RAM. With that OS running, the flash content gets exchanged with a new image. So if the image is faulty, or the flashing process gets interrupted through power loss, you won't have a bootable system anymore. A simple live disk or bootable USB stick won't help much if you can't even find the USB bus (or your other devices with the bootable operation system image) can be found.

In other words, your computer (or if you do stuff with your smartphone, your device) is bricked.

Basically, it becomes a very expensive paper weight.

If you however use the SPI bus directly for ISP (in-systems programming/in-situ programming), you do not have to care for breaking things through faulty images if you have a working one already. This enables you to test things without having to fear you will render your hardware unusable. Which leaves room for trying out things which were impossible prior.

Like fiddling directly with proprietary software which wants to prohibit you from booting a proper operating system on some hardware of choice from you. I don't know when this hobby project will be finished, but I sure learned a lot about electronics within the last half year.

needed tools

  • raspberry pi (revision does not matter, but just get a B 2 in case you don't have one yet.)
  • sd flash card (this is where you will dd your OS image onto)
  • soic clip (google that, in case you want to work on NOR flash chip, so you don't have to solder wires onto the chip directly which is ugly)
  • female-to-female jumper cables (six ones minimum for working with SPI, maybe more)

In my case a debian installation was put onto an SD card of a raspberry pi (which is ARM based, as one might know), only to find out that the existing flashrom package exists for intel architecture based processors only.

Bummer.

compile and install

Ok, so lets install a proper environment and build stuff by hand then, as root user:

apt install build-essential
apt install libusb-dev
apt install pciutils-dev
apt install bzip2

wget http://download.flashrom.org/releases/flashrom-0.9.8.tar.bz2
tar xjvf flashrom-0.9.8.tar.bz2
cd flashrom-0.9.8

make -j4
make install

wiring

Google the chip you want to work on, and look after a description of its pins. (Chances are you already did this, which told you that you could use the SPI bus at all.) Put the SOIC clip on the chip.

Google a raspberry pinout table, and connect the SPI pins (MISO, MOSI, CE0, CLOCK, GND, 3.3V) accordingly.

Use short cables, long ones may cause connection problems.

usage example

All the following was done without a power supply being connected to the board, as the chip got the power from the raspberry's 3.3V Vcc pin.

As I had no prior knowledge on how to use flashrom ('i dont even know what im doing here'[TM]), this is what I tried:

# go to $HOME and create a temp folder
cd
mkdir flashromming
cd flashromming

# show help
flashrom -h

# try directly
flashrom

# try using the programmer which might work
flashrom --programmer linux_spi

# search for spi device
ls -alh /dev/spidev0.*

# use appropriate programmer, which then found my chip
flashrom --programmer linux_spi:dev=/dev/spidev0.0

# look up help to find out how to dump the flash content into a file
flashrom
flashrom -h

# actual dumping (-r = READ flash content into file)
flashrom --programmer linux_spi:dev=/dev/spidev0.0 -r nas-flash-original.bin

# always work on copies, not originals!!!
cp nas-flash-original.bin nas-flash-copy.bin

# have a look at the dumps contents
dd if=nas-flash-copy.bin | hexdump -vC | less

For starters, this worked. There is more:

## OTHER STUFF:
# flash new content onto chip (-w = WRITE file to chip)
flashrom --programmer linux_spi:dev=/dev/spidev0.0 -w newimage.bin

# erase chip contents (-E)
flashrom --programmer linux_spi:dev=/dev/spidev0.0 -E

# verify chip contents against file (-v)
# this is only needed when in doubt which file got flashed, verifying is done automatically after each flashing procedure
flashrom --programmer linux_spi:dev=/dev/spidev0.0 -v newimage.bin

issues

The motherboard which was used also had a serial interface (UART/RS232) which I used have a look at the boot process and for console access. When the SOIC clip was connected to the chip, it just would not boot.

wget: download all linked files from URL
posted on 2016-01-01 21:57:55

This will also create a lot of index files...

wget -m -p -E -k -K -np http://website.tld/path/

... which can be removed like this aftertwards, for example:

find . -iname '*index*' -exec rm {} \;

Don't delete files containing the string 'index' which you still need, check what you do before blindly copying commands from sites you don't know. :)

This blog covers .csv, .htaccess, .pfx, .vmx, /etc/crypttab, /etc/network/interfaces, /etc/sudoers, /proc, 10.04, 14.04, AS, ASA, ControlPanel, DS1054Z, GPT, HWR, Hyper-V, IPSEC, KVM, LSI, LVM, LXC, MBR, MTU, MegaCli, PHP, PKI, R, RAID, S.M.A.R.T., SNMP, SSD, SSL, TLS, TRIM, VEEAM, VMware, VServer, VirtualBox, Virtuozzo, XenServer, acpi, adaptec, algorithm, ansible, apache, apachebench, apple, arcconf, arch, architecture, areca, arping, asa, asdm, awk, backup, bandit, bar, bash, benchmarking, binding, bitrate, blackarmor, blowfish, bochs, bond, bonding, booknotes, bootable, bsd, btrfs, buffer, c-states, cache, caching, ccl, centos, certificate, certtool, cgdisk, cheatsheet, chrome, chroot, cisco, clamav, cli, clp, clush, cluster, coleslaw, colorscheme, common lisp, console, container, containers, controller, cron, cryptsetup, csync2, cu, cups, cygwin, d-states, database, date, db2, dcfldd, dcim, dd, debian, debug, debugger, debugging, decimal, desktop, df, dhclient, dhcp, diff, dig, display manager, dm-crypt, dmesg, dmidecode, dns, docker, dos, drivers, dtrace, dtrace4linux, du, dynamictracing, e2fsck, eBPF, ebook, efi, egrep, emacs, encoding, env, error, ess, esx, esxcli, esxi, ethtool, evil, expect, exportfs, factory reset, factory_reset, factoryreset, fail2ban, fbsd, fedora, file, filesystem, find, fio, firewall, firmware, fish, flashrom, forensics, free, freebsd, freedos, fritzbox, fsck, fstrim, ftp, ftps, g-states, gentoo, ghostscript, git, git-filter-branch, github, gitolite, gnutls, gradle, grep, grml, grub, grub2, guacamole, hardware, haskell, hdd, hdparm, hellowor, hex, hexdump, history, howto, htop, htpasswd, http, httpd, https, i3, icmp, ifenslave, iftop, iis, imagemagick, imap, imaps, init, innoDB, innodb, inodes, intel, ioncube, ios, iostat, ip, iperf, iphone, ipmi, ipmitool, iproute2, ipsec, iptables, ipv6, irc, irssi, iw, iwconfig, iwlist, iwlwifi, jailbreak, jails, java, javascript, javaws, js, juniper, junit, kali, kde, kemp, kernel, keyremap, kill, kpartx, krypton, lacp, lamp, languages, ldap, ldapsearch, less, leviathan, liero, lightning, links, linux, linuxin3months, lisp, list, livedisk, lmctfy, loadbalancing, locale, log, logrotate, looback, loopback, losetup, lsblk, lsi, lsof, lsusb, lsyncd, luks, lvextend, lvm, lvm2, lvreduce, lxc, lxde, macbook, macro, magento, mailclient, mailing, mailq, manpages, markdown, mbr, mdadm, megacli, micro sd, microsoft, minicom, mkfs, mktemp, mod_pagespeed, mod_proxy, modbus, modprobe, mount, mouse, movement, mpstat, multitasking, myISAM, mysql, mysql 5.7, mysql workbench, mysqlcheck, mysqldump, nagios, nas, nat, nc, netfilter, networking, nfs, nginx, nmap, nocaps, nodejs, numberingsystem, numbers, od, onyx, opcode-cache, openVZ, openlierox, openssl, openvpn, openvswitch, openwrt, oracle linux, org-mode, os, oscilloscope, overview, parallel, parameter expansion, parted, partitioning, passwd, patch, pdf, performance, pfsense, php, php7, phpmyadmin, pi, pidgin, pidstat, pins, pkill, plesk, plugin, posix, postfix, postfixadmin, postgres, postgresql, poudriere, powershell, preview, profiling, prompt, proxmox, ps, puppet, pv, pvecm, pvresize, python, qemu, qemu-img, qm, qmrestore, quicklisp, r, racktables, raid, raspberry pi, raspberrypi, raspbian, rbpi, rdp, redhat, redirect, registry, requirements, resize2fs, rewrite, rewrites, rhel, rigol, roccat, routing, rs0485, rs232, rsync, s-states, s_client, samba, sar, sata, sbcl, scite, scp, screen, scripting, seafile, seagate, security, sed, serial, serial port, setup, sftp, sg300, shell, shopware, shortcuts, showmount, signals, slattach, slip, slow-query-log, smbclient, snmpget, snmpwalk, software RAID, software raid, softwareraid, sophos, spacemacs, spam, specification, speedport, spi, sqlite, squid, ssd, ssh, ssh-add, sshd, ssl, stats, storage, strace, stronswan, su, submodules, subzone, sudo, sudoers, sup, swaks, swap, switch, switching, synaptics, synergy, sysfs, systemd, systemtap, tar, tcpdump, tcsh, tee, telnet, terminal, terminator, testdisk, testing, throughput, tmux, todo, tomcat, top, tput, trafficshaping, ttl, tuning, tunnel, tunneling, typo3, uboot, ubuntu, ubuntu 16.04, udev, uefi, ulimit, uname, unetbootin, unit testing, upstart, uptime, usb, usbstick, utf8, utm, utm 220, ux305, vcs, vgchange, vim, vimdiff, virtualbox, virtualization, visual studio code, vlan, vmstat, vmware, vnc, vncviewer, voltage, vpn, vsphere, vzdump, w, w701, wakeonlan, wargames, web, webdav, weechat, wget, whois, wicd, wifi, windowmanager, windows, wine, wireshark, wpa, wpa_passphrase, wpa_supplicant, x2x, xfce, xfreerdp, xmodem, xterm, xxd, yum, zones, zsh

View posts from 2017-03, 2017-02, 2017-01, 2016-12, 2016-11, 2016-10, 2016-09, 2016-08, 2016-07, 2016-06, 2016-05, 2016-04, 2016-03, 2016-02, 2016-01, 2015-12, 2015-11, 2015-10, 2015-09, 2015-08, 2015-07, 2015-06, 2015-05, 2015-04, 2015-03, 2015-02, 2015-01, 2014-12, 2014-11, 2014-10, 2014-09, 2014-08, 2014-07, 2014-06, 2014-05, 2014-04, 2014-03, 2014-01, 2013-12, 2013-11, 2013-10


Unless otherwise credited all material Creative Commons License by sjas