Posts from 2014-05

FreeBSD/PFSense via Grub2
posted on 2014-05-23 17:21:18

After installing PFSense (which is based on FreeBSD) on a dedicated machine for firewalling, a debian install was decided to be set up along with it. This should enable shorter setup times in case PFSense would not suffice: Just boot debian and fix the firewall via iptables in /etc/init.d/firewall.

Long story short, after the debian install and the grub, PFSense was lacking a boot entry. (Who would have guessed.)

Output of fdisk -l from within debian gave this:

Disk /dev/sja: 4011 MB, 4011614208 bytes
16 heads, 63 sectors/track, 7773 cylinders, total 7835184 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x90909090

    Device  Boot      Start         End      Blocks     Id  System
/dev/sda1     *          63     1639999      818968+    a5  FreeBSD
/dev/sda2           1638000     7835183     3098592     83  Linux

So the first partition has the PFSense on it.

Putting the lower part of this into /etc/grub.d/40_custom in debian did the trick:

#!/bin/sh
exec tail -n +3 $0
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.
menuentry "PFSense" {
    # discern partition name easiest via grub shell
    set root=(hd0,1)
    chainloader +1
    # instead of `chainloader +1`, this should work, too
    #kfreebsd /boot/loader
}

Do update-grub afterwards, reboot, and be done.

Debian java update-alternatives
posted on 2014-05-21 06:23:43

If you need javaws with oracle java (not that IcedTea crap), and have it installed already, but lost your settings due to an update, do:

$ update-alternatives --config javaws

This will show you something like this:

There are 6 choices for the alternative javaws (providing /usr/bin/javaws).

  Selection    Path                                              Priority   Status
------------------------------------------------------------
* 0            /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/javaws   1071      auto mode
  1            /usr/lib/jvm/j2re1.7-oracle/bin/javaws             316       manual mode
  2            /usr/lib/jvm/j2sdk1.7-oracle/jre/bin/javaws        317       manual mode
  3            /usr/lib/jvm/java-6-openjdk-amd64/jre/bin/javaws   1061      manual mode
  4            /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/javaws   1071      manual mode
  5            /usr/lib/jvm/java-7-oracle/bin/javaws              9         manual mode
  6            /usr/lib/jvm/java-7-oracle/jre/bin/javaws          1064      manual mode

Press enter to keep the current choice[*], or type selection number: 

Choose the according number and be happy.

nagstamon setup
posted on 2014-05-19 22:29:27

According to apt-get: nagstamon - Nagios status monitor which takes place in systray or on desktop

This is a tool to provide a usable interface to nagios messages, and after some setup you might quite like it.

Install (Debian):

$ apt-get install nagstamon -y

After installing, it might be a good idea to put it into autostart.

Then some settings will be quite practical:

  • connect to nagios (else no messages)
  • apply a filterlist (so you just see what is important to you)
  • expand/collapse not set to hover (quite buggy in KDE)
  • ssh access (fix console setting if you are not using gnome)
  • adjust downtime default time span to your liking
Runlevel configuration in Debian
posted on 2014-05-15 14:43:12

Use sysv-rc-conf for this. It will show an curses GUI, where you can edit the present settings.

Get it from apt:

$ apt-get install sysv-rc-conf
Compile vim on Debian 7
posted on 2014-05-14 17:13:39

bare vim

First install mercurial if you do not have it already. Use hg --version to check.

$ apt-get install mercurial

Then get the vim sources and compile:

$ hg clone https://vim.googlecode.com/hg/ vim
$ cd vim/src
$ make

If it fails with an error like this one:

checking for tgetent in -ltinfo... no
checking for tgetent in -lncurses... no
checking for tgetent in -ltermlib... no
checking for tgetent in -ltermcap... no
checking for tgetent in -lcurses... no
no terminal library found
checking for tgetent()... configure: error: NOT FOUND!
      You need to install a terminal library; for example ncurses.
            Or specify the name of the library with
--with-tlib.
            make: *** [auto/config.mk] Error 1

Try installing an ncurses library:

$ apt-get install libncurses5

Afterwards all should be fine.

vim with python 3 support enabled

If however you desire specific functions (like using the powerline statusbar), you might need python support compiled into vim. Along with this there are some other things I need, too, which are shown below.

Enable it by preparing via ./configure:

./configure \
    --enable-python3interp \
    --with-python-config-dir=/usr/lib/python3.2/config-3.2mu \
    --with-x=yes \
    --with-features=normal \
    --disable-gui \
    --prefix=$HOME/dev/vim

This enables python 3 (for python 2, the command is the same, just lacking the '3'), the config dir is important, too.

X support is baked in for clipboard stuff, IIRC.
The featureset is normal. (See :h +feature for what is included.)
GUI is disabled since I never use gvim.
At last, the install directory is in $HOME/dev, because I prefer the vim install being there along with other dev tools.

python support install problems in detail

missing python dev packages

The tricky stuff for enabling python is to not forget the python dev packages, in my case via apt-get install python3-dev.
If python does not want to work, vim needs python's config dir location, and there has to be the config.c file located. Else installation will work, but you will not notice why it did now work as expected.

Try appending grep python -C3 via a pipe, i.e.

$ ./configure \
    --enable-python3interp \
    --with-python-config-dir=/usr/lib/python3.2/config-3.2mu \
    --with-x=yes \
    --with-features=normal \
    --disable-gui \
    --prefix=$HOME/dev/vim | grep python -C3`

That way you will be able to notice if the ./configure step is borked.

./configure's caching

Also, ./configure caches the steps it has made. This is the case when the ./configure ... output has (cached) appended. In this case just delete the config.cache file within the vim/src/auto folder.

Editing remote files in local vim
posted on 2014-05-14 14:39:09

If you have a highly configured vim that you hold dearly and love to bits, and you have to edit a lot of remote files, try vim-over-ssh. scp is used to transfer files, and there is a slight delay, but it does work awesome.

Syntax:

vim scp://<user>@<host>//<path/to/file>

Note the double slashes // after the hostname. These are used so the absolute path to your file can be specified.

vim open file with cursor positioned
posted on 2014-05-13 15:54:34

To open a file in vim, with the cursor positioned on the first search match, use:

$ vim +/<searchterm> <filename>

To just open vim with the cursor being placed on a specific line, do:

$ vim +<linenumber> <filename>
Home networking and WLAN Routers
posted on 2014-05-10 12:38:52

This is the current setup I have at home, this is mainly for documentation purposes.

# main router, used for dial-up and DHCP
192.168.178.1
# WLAN
192.168.178.2
# my dlink
192.168.178.3
# printer
192.168.178.4

# DHCP range
192.168.178.10 to 254

DHCP has to run on the dial-up router! If it does not, the DHCP server will not know where the DNS can be found and will not be able to tell its clients about it.

But it is about time to test IPv6 at home... so this may be not too long in use.

Create mails in bash
posted on 2014-05-09 20:17:25

Write this directly on your command prompt:

/usr/bin/mail -s "testmail" root 'mailaddress@domain.tld' -a "From: mail_daemon" <<< "ti ta testmail"

Which will create this:

To: <root@hostname>, <mailaddress@domain.tld>
Subject: testmail
From: <mail_daemon@hostname>

ti ta testmail

This is useful when you already have a postfix (or whatever maildaemon) running, and you need email notification in your scripts.

dd Howto and some MBR tricks
posted on 2014-05-09 19:00:18

dd is used to "convert and copy files" under linux.

Basic Stuff

Read: Use it for disk images. I.e. put an .ISO on an USB stick. Or copy whole HDD's. Or create ISO's.

# use 'mount' to find out where the stick is mounted

# copy iso onto usb stick
dd if=<isoname>.iso of=/dev/sdX

# create an iso
dd if=/dev/cdrom of=/home/<username>/Desktop/<isoname>.iso

# wipe clean
dd if=/dev/zero of=/dev/sdX

# make a file of 100 random bytes
dd if=/dev/urandom of=/home/<username>/my.random bs=1 count=100

That was it with the usual suspects. Now onto more serious stuff.

Serious Stuff

# PRO: create image from one host and stream to the other
#on target host
netcat -l -p 1234 | dd of=/dev/hdc bs=16065b
#on source host (where the ISO will be created and streamed)
dd if=/dev/hda bs=16065b | netcat <targethost-IP> 1234

#or use this:
# from remote to local
rsh 192.168.xx.yy "dd if=/dev/sda ibs=4096 conv=notrunc,noerror" | dd of=/dev/sda obs=4096
# or from local to remote
dd if=/dev/sda ibs=4096 conv=notrunc,noerror | (rsh 192.168.xx.yy dd of=/dev/sda obs=4096)

In the above you may use ssh, or rsh. Do as you please. Of course you may use different IP's.

And now...

Very Serious Stuff

# PRO: show your MBR
dd if=/dev/sda count=1 | hexdump -C

# PRO: back up MBR
dd if=/dev/sda of=mbr.bin count=1

    #put this on a floppy you make with
    dd if=boot.img of=/dev/fd0
    #along with dd. Boot from the floppy and
    dd if=mbr.bin of=/dev/sda count=1
    #will restore the MBR.


# PRO: command to read your BIOS and all interfaces
dd if=/dev/mem bs=1k skip=768 count=256 2>/dev/null | strings -n 8

on decrypting an MBR

Use file. Just do it:

dd if=/dev/sda of=mbr.bin
file mbr.bin

will give you something like this:

mbr.bin: x86 boot sector; partition 1: ID=0xee, starthead 0, startsector 1, 234441647 sectors, extended partition table (last)\011, code offset 0x0

which is easily read and understandable.

TTL values
posted on 2014-05-08 09:50:18

A list of commonly used TTL values (i.e. in DNS) settings.

(  60         =     1 m  )
   1800       =    30 m
   3600       =     1 h
   10800      =     3 h
   14400      =     4 h
   21600      =     6 h
   43200      =    12 h
   86400      =     1 d
   259200     =     3 d
   604800     =     7 d
   31536000   =   365 d

Going below 0,5h is usually forbidden. Mostly 24h are used for RR's (resource records, like A / NS / MX), to reduce traffic for the DNS servers.

List of private networks
posted on 2014-05-07 14:57:30

private address spaces

IPv4

IP address ranges for private networks, as by RFC 1918.

10 . 0 . 0 . 0 / 8
172. 16. 0 . 0 / 12
192.168. 0 . 0 / 16

IPv6

Same as above, for IPv6. See RFC 4193.

fc00:: / 7

link-local address spaces

These are for private networks being on the same link. (Read: switch)

Will not work through routers (hosts on different subnets are on different links), but bridges will.

These are intended for address autoconfiguration, i.e. via DHCP.

IPv4

See RFC 6890 and RFC 3927.

169.254. 0 . 0 /16 - except the first and last /24-er subnet.

169.254.1.0 to 169.254.254.255 may be assigned pseudorandomly on ethernet networks, if an IP cannot be obtained.

IPv6

See RFC 4862 and RFC 4291.

fe80:: / 10
rsync howto
posted on 2014-05-07 14:50:43

Usually when using rsync you want to use it like this:

rsync -avzh --progress server1:/path/to/file server2:/path

This does not preserve hard links, use the -H option, if you need this, too.

VMWare ESXi 5.5 create datastore error
posted on 2014-05-07 13:50:18

When trying to create a new datastore on a HDD which was partitioned previously, and you have chosen VMFS-5 the creating will fail with some strange partitioning error.

Solution is to create the datastore with VMFS-3 file format, which will work.
Afterwards delete the datastore, and try recreating it with VMFS-5, which will work then.

Configuring vlan ports on a juniper EX 2200 switch via SSH
posted on 2014-05-05 18:38:13

This will serve as a quick reminder since I don't do that too often.

theory

Over a physical NIC there can run an arbitrary amount of VLAN's. Only requirements are the switch port and your NIC being attached to it being able running VLAN's, and being properly configured. Else only one VLAN can run at once on one port.

Trunking is the possibility to 'tag' packets. If this cannot be done (switch just has not got the functionality), you need a dedicated cable running from switch to switch for EACH VLAN. Via trunk mode the switching cannot be done on just port level, but even across switches, utilizing only a single interconnect.

switchport schemata and config access

ge-X/Y/ZZ.A is a dummy for the following:

X   = the switch number
Y   = backplane number
ZZ  = port number
A   = Unit

Switch number (X) is clear, backplane number (Y) not so. Sometimes you came across switches that are extendable. I.e., you can insert a second panel with a second set of ports into the existing switch. Port number (ZZ) should be clear again. A Unit (A) is a logical NIC. This is needed for layer-3-switching.

If a VLAN is created which spans several subnets, the port has to have logical adresses in both subnets. These are differentiated via the Unit. I.e. the first IP from the first subnet gets Unit '0', the second IP from the second subnet gets '1'... You get the idea. More on this here.

configuration

Here all that is ever done happens on layer 2. No layer-3-switching/routing is done, which is why setup is simpler and only Unit 0 is put to use.

After ssh'ing onto the machine with user 'admin', these are your first helpers:

> ?
> help
> help ?

Whenever you do not know what to do, use ?, or append a ? to the line you are typing currently.

> show vlans
> show interfaces
> show interfaces terse
> show interfaces | display set
> show interfaces ge-X/Y/ZZ
> show interfaces ge-X/Y/ZZ.A
> show interfaces ge-X/Y/ZZ terse
> show interfaces ge-X/Y/ZZ unit A family ethernet-switching vlan members 
> show interfaces ge-X/Y/ZZ unit A | display set
> show chassis

From edit / configure` mode:

> run show interfaces descriptions

Some of these commands can be run in regular (non-edit) mode only, some only in edit mode. To get around this restriction, prefix the command in question with run or set, IIRC.

Show who else is editing what and where:

> status

Then the modes for making changes:

> configure
> edit

> configure exclusive
> edit exclusive

Change to the proper position of the directory tree:

> edit interfaces ge-X/Y/ZZ unit A family ethernet-switching
> up
> top

In general, you can hop around the data tree via edit <path>. This serves readability and will save you quite some typing.

Making changes:

> set port-mode 
> set vlan members 
> delete port-mode

Otherwise, you can use the output of show interfaces | display set directly via copy/paste, if changed accordingly. This is also rather helpful, once you got used to it.

Exiting:

> quit

If you want to throwaway your changes prior to committing:

> rollback

This will load the last committed configuration and clear all pending commits.

Check if your changes worked, and apply them:

> show | compare
> commit check
> commit
> commit at
> commit and-quit

That should be about it.

copying setting

Do:

edit
show interfaces | display set

And just copy paste the shown configuration lines.

logs

show log ?
show log <logname>

temperature and load

show chassis routing-engine 

shortcuts

By the way, backspace works.
Delete will not, but CTRL-D will.
CTRL-C will not, but CTRL-U and CTRL-K will.

shell

If you really desire a shell: start shell
And you will get into a cozy... /usr/sbin/cli?

sh and (t)csh are also available.

man [
posted on 2014-05-04 23:30:22

Whenever writing bash if-clauses, this will be your new best friend:

$ man [
bash heredoc
posted on 2014-05-04 22:45:26

Bash's heredoc functionality provides functionality to work with stream literals.

Usages:

# 1. basic case with parameter expansion
command << DELIMITER_YOU_CHOOSE
this here
is input
which is
line for line
piped into command
DELIMITER_YOU_CHOOSE

# 2. no parameter expansion due to the " "
command << "DELIMITER_YOU_CHOOSE"
    echo $SHELL
DELIMITER_YOU_CHOOSE

# 3. same as above, but will strip leading TAB's
# Use TAB's. DON'T use spaces!
command <<- "DELIMITER_YOU_CHOOSE"
    echo $SHELL
DELIMITER_YOU_CHOOSE

# 4. parameter expansion, strip leading TAB's
command <<- DELIMITER_YOU_CHOOSE
    echo $SHELL
DELIMITER_YOU_CHOOSE

# 5.  command <<< evaluated_command, some kind of shortcut
command <<< cat $SHELL

Results:

# 1.
this here
is input
which is
line for line
piped into command

# 2.
    echo $SHELL

# 3.
echo $SHELL

# 4.
/bin/bash

# 5.
/bin/bash

In general, the usage depends on the type of the input source: - < is for files - << is for typed stream literals - <<< is for evaluated commands

This is a possibility to pass several lines of arguments to command expecting input in several stages. I.e. for scripted certificate generation this comes in handy.

bash redirection
posted on 2014-05-04 21:18:29

The bash shell provides three differnt file handles by default. These are called file descriptors in bash.

  • /dev/stdin = 0 = read input from shell prompt
  • /dev/stdout = 1 = stuff printed to shell
  • /dev/stderr = 2 = error messages channel

& is 1 and 2 combined.

For I/O redirection all these can be accessed through piping:

command  <  file        # redirect file to STDIN
command <<  heredoc     # redirect heredoc to command STDIN
command <<< herestring  # redirect herestring to command STDIN
         >  file        # BEWARE: truncates/deletes file contents!
command 1>  file        # redirect command STDOUT to file
command 2>  file        # redirect command STDERR to file
command &>  file        # redirect command STDOUT and STDERR to file
command  >| file        # redirect command STDOUT forcefully
command  >> file        # like '>', but will append instead of overwrite in file

Forceful redirection will write, even when bash has noclobber set. 'noclobber' prevents files from getting overwritten when redirection operators are used.

heredoc's and herestrings's will go into a separate post, soon.

Sidenotes:

# redirection between handles, these are equivalent:
command &>  file        # redirect STDOUT and STDERR
command >   file 2>&1   # redirect STDOUT as well as STDERR to STDOUT

In more detail:

# redirects "i" to "j", if "i" is omitted, defaults to "1"
        i>&j

i,j range from 1 to 9, 3-9 are free to use. Beware, "5" is inherited by child processes and exec usage.

Plus:

# close file output descriptor "i"
        i>&-
# close file input descriptor "j"
        j<&-

and

# these are equivalent

command < input-file > output-file

< input-file command > output-file   # Although this is non-standard.

Files can also be opened for reading AND writing simultanously via <>. Also any file descriptor ID can be used with.

exec can be used to redirection for the complete current shell. See here.

Also you can much more with duplicating, closing or moving file descriptors. But this is stuff for another post when I need it. In the meantime, this bash reverse proxy is really 'wickedly cool'.

bash multitasking
posted on 2014-05-04 17:50:46

When working in the shell under linux and having started a long-running process, which blocks the shell (and you working in it), gives you some options:

  • wait until process is finished, you land at the prompt, you can work on (NO!)
  • open a new shell window, which is no problem when working under a graphical window manager (MAYBE...)
  • use bash's multitasking capabilities (YES!)

How?

First the shortcuts:

Ctrl-C kills the process currently in the foreground.
Ctrl-Z "suspends"/pauses the process currently running in the foreground and puts it into background.
Ctrl-Y suspend job the next time it asks for user input.

The last two differ in such a way, as suspending via Ctrl-Z may swallow pending shell output, whereas Ctrl-Y will not.

Now the commands:

$ jobs              # list all background processes
$ fg                # start marked process in foreground (see '+' on `jobs` list)
$ bg                # start marked process in background (see '+' on `jobs` list)
$ fg %x             # start process with id 'x' in foreground
$ %n                # alias for fg %n
$ bg %x             # start process with id 'x' in background
$ %n &              # alias for bg %n
$ kill %x           # kill process with id 'x'
$ <processname> &   # '&' will start a process and let it run in background

Example usage: (demonstrated via bash's sleep, which just waits for the specified time in seconds)

[sjas@lorelei ~]% sleep 100
    CTRL-Z
[1]  + 7848 suspended  sleep 100

[sjas@lorelei ~]% sleep 200
    CTRL-Z
[2]  + 10920 suspended  sleep 200

[sjas@lorelei ~]% sleep 300
    CTRL-Z
[3]  + 3676 suspended  sleep 300

[sjas@lorelei ~]% sleep 400
    CTRL-Z
[4]  + 10012 suspended  sleep 400

[sjas@lorelei ~]% jobs
[1]    suspended  sleep 100
[2]    suspended  sleep 200
[3]  - suspended  sleep 300
[4]  + suspended  sleep 400

[sjas@lorelei ~]% bg
[4]    10012 continued  sleep 400

[sjas@lorelei ~]% jobs
[1]    suspended  sleep 100
[2]  - suspended  sleep 200
[3]  + suspended  sleep 300
[4]    running    sleep 400

[sjas@lorelei ~]% kill %1
[1]    7848 terminated  sleep 100

[sjas@lorelei ~]% jobs
[2]  - suspended  sleep 200
[3]  + suspended  sleep 300
[4]    running    sleep 400

[sjas@lorelei ~]% fg
[3]    3676 continued  sleep 300
    CTRL-Z

[sjas@lorelei ~]% jobs
[2]  - suspended  sleep 200
[3]  + suspended  sleep 300
[4]    running    sleep 400

[sjas@lorelei ~]%

There is also disown for removing jobs from the list of jobs and other actions. In case of interest man bash and search for disown.

Have fun.

Linux file system structure, boot process, partitioning
posted on 2014-05-03 08:25:21

The linux directory layout comes from the Filesystem Hierarchy Standard. It is used in linux and the unix derivates. The draft of version 3.0 can be found here. Also helpful might be man hier, the description of the linux file system hierarchy.

In short, there follows an explanation of its depicted structure, plus the folders usually present in a regular linux distro. The list is also annotated with information on how these are used during OS startup, why these are separated the way they are, partitioning considerations. First some background a the boot process in IBM compatible (server) PC's.

Booting procedure, first stage

Power on

When you boot a computer through pressing the power button/IPMI/ILO/whatever, the power supply powers up all hardware with electricity. The BIOS/EFI/UEFI on the motherboard kicks in and searches all known (hardware) drives, in order of the boot ordering set in the bios, for a boot sector. (Sector means a sector on a harddrive, for example.) When a MBR/VBR is found there, the BIOS loads the data which is stored there into the RAM on a fixed position. The software being responsible for this is a bootloader, and comes with the BIOS. This is the first stage of the boot loading.

The boot sector

The boot sector is always located in the first sector on the drive. In case of a MBR being in use, its on the first sector and also at the first head on the first cylinder of the drive.

The boot sector may contain a MBR (Master Boot Record), or a VBR (Volume Boot Record). This differentiation is depending on the boot sector being located outside of the partition containing the other data (like the OS). The boot record, regardless of its type has the same structure.

The boot record

On the data structure of a boot record:
Its starting mark is identified by the hex code 0x55 0xAA on disk, or AA55h in memory. See here.. The hex sequences may differ for BIOS'es for non-x86-CPU's, just for the record. Then it contains the partitioning table, providing basic information on the divisioning into primary partitions. The other code needed for bootstrapping will follow. The order on disk is reversed compared to the one in memory due to Little Endianness usually. No idea how this is on Big Endian based systems.

Usually this creates the following layout:
Size-wise it is 446 bytes for Bootstrap code, 64 bytes for Partition table, 2 bytes for Signature (= 0x55 0xAA). Summing up to 512 byte, when the disks sector size is 512 Bytes. Of course, this can differ, too, A LOT, depending on the OS in use, sector size, ... See here.

The partition table

Because of the 64 byte maximum size of the partition table, a maximum limit of 2 TiB adressable space exists for a disk with 512 bytes per sector: 2^32 * 512 Bytes. This led to the development of GPT (GUID Partition Table, GUID = globally unique identifiers), to address bigger storage devices. But this is just for the record. More info here.

The boot record will tell the system which single partition is currently marked as active, indicating where the operating system or bootloader can be found.

Booting procedure, second stage

o In the case of Linux, the boot record tells the BIOS on which partition the sector containing /boot is located.

/boot

/boot can be placed on the same partition as the other data of the operating system or user data. There exist several reasons on why to put /boot on a separate partition from / and others like /home etc..

  • for full disk encryption like TrueCrypt: if /boot is encrypted, how to decrypt the whole drive?
  • your root file system is administrated via LVM, RAID
  • BIOS limitations (Maybe your BIOS can only read like the first 1022 cylinders of your disk? Don't laugh...)
  • Or the operating system uses a file system which cannot be read by the BIOS.

Some notes on partitioning under linux are here.

Boot manager/loader

Only after things went that way, the boot manager, if one is installed, comes into play. Examples of boot managers are grub/lilo/syslinux on linux, NTLDR/BOOTMGR/winload.exe on Windows and whatever MacOS uses. With the boot loader having been found and loaded into RAM, the second stage of boot loading starts.

On a sidenote, bootloaders can boot make booting into other partitions beside the active one, possible. Also booting from logical volumes is possible through them.

An entry in a boot manager must point to kernel file, or another boot sector on another partition or disk. When another boot sector is chosen, the one currently loaded into memory will be replaced with the one being pointed to. The other boot sector must contain another bootloader or kernel. Else your system won't be able to boot an operating system. ;) This procedure is what is called chain-loading or bootstrapping. This also took place from first to second boot stage.

Now after a long prelude on how bootstrapping works, why placing /boot on a separate partition is useful and how it and its contents will be accessed during boot time... Finally we are near Linux' directory structure.

General partitioning

In the following I tried to logically group folders which are to be placed together onto one partition. In theory the OS can mount all folders here from separate partitions. This schema depicts a useful partitioning layout, but may be overkill. For a usual desktop system, use /boot, /, /home.

swap

During partitioning for linux also a swap partition needs to be created. This is from where linux accesses the virtual memory, but it won't show up in the directory tree below. In the past double the amount of your RAM size was a good value. However this was once upon a time where PC's had like 512 MB RAM. Nowaydays use half the RAM size, that should do fine.

The swapspace could as well be located on the same partition as the root file system. But if the swapspace is corrupted the system may be rendered unusable. This is more likely when / and /swap lie on the same partition.

Linux directory layout

BOOT STUFF, PARTITION 1
=============================================================================================================
/boot           static boot files: bootloader, kernel, ramdisk image ("early userspace")
=============================================================================================================


ROOT FILESYSTEM ("primary hierarchy") with executables / configs / system folders, PARTITION 2
=============================================================================================================
/               root node of the file system, mounted after boot

/lib            kernel modules and dynamic libraries
/lib*           i.e. /lib64 for 64-bit-specific libraries

/bin            binaries, available in single user mode and to all users. 
                used to bring up the system or repair it
                i.e. cat, ls

/sbin           system binaries for system maintenance/adminning
                same as /bin, but for files having to be used with root priviledges
                for boot, restore, recover tasks, just of no concern for regular users
                i.e. sysctl, mkfs, ifup/ifdown

/dev            device files referencing physical interfaces for accessing hardware
/proc           virtual pseudo-file system, for process information
                used as an interface to the kernel's data structures.
/run            data relevant to running processes, files placed earlier in 
                /etc, /var/run, /var/lock, /dev/hcmem etc.
                /var/lock and /var/run were segregated temporal file systems earlier.
                sometimes even mounted prior to /var.
                now with /run is only a single tmpfs (temporal file system) needs to be initiated.
                a RAM-disk is a virtual disk, hosting a file system and thus not the same.

/etc            host-specific configuration files
/tmp            temporal files, may be deleted at any time. (in theory, at last)
/var            variable data
    /var/log        all logfiles you may need sooner than you might like
    /var/tmp        see /tmp

/mnt            mount temporal external devices here
/media          same as /mnt, for removeable devices, will show up on Desktop when used for mounts

/lost+found     via fsck recovered damaged files

/root           home folder of the admin, the root user
-------------------------------------------------------------------------------------------------------------
/usr            could also be here, if not mounted separatedly.
/opt            could also be here, if not mounted separatedly.
/home           could also be here, if not mounted separatedly.
/srv            could also be here, if not mounted separatedly.
=============================================================================================================


USER SYSTEM RESOURCES ("secondary hierarchy"), SHARED APPLICATIONS, PARTITION 3
=============================================================================================================
/usr            secondary hierarchy, 'user system resources'
                also contains i.e. /usr/etc and other folders present in '/'.
                static files that may reside on a separate partition.
                i.e. for being shared for use among several linux systems.
                all the package-manager-installed files go here.
    /usr/bin        binaries of user programs and user-installed applications
    /usr/sbin       same as sbin, system files to be used with root priviledges
                    i.e. the package manager executables/binaries live here
    /usr/lib/       libraries for user programs
    /usr/tmp/       obsolete, use /var/tmp instead
    /usr/include/   include files for C sources
=============================================================================================================

USER SYSTEM RESOURCES ("tertiary hierarchy"), LOCAL APPLICATIONS, PARTITION 4
=============================================================================================================
/usr/local          tertiary hierarchy for local data, specific to this host.
                    also has /usr/local/bin, /usr/local/sbin, /usr/local/etc and so forth
                    usually third party programs concerning the OS go here.
                    be it self-compiled or packaged software, all not maintained by your distribution
                    this is useful because these folders are not touched during OS updates/upgrades
                    everything else you install goes into /opt

                    no idea if it is of use to mount this one on a separate partition?
=============================================================================================================

USER INSTALLED SOFTWARE, PARTITION 5
=============================================================================================================
/opt            additional installed software, addon packages containing static files
                no OS specific stuff, just application software?
                /etc/opt, /var/opt are compagnion folders

                /opt vs. /usr/local/ is a decision like vim vs. emacs...

                if /etc/opt is used for configs, it might make sense putting this on an extra partition, too
=============================================================================================================

USER DATA, PARTITION 6
=============================================================================================================
/home           root point for user's home folders
=============================================================================================================


SERVER DATA, PARTITION 7
=============================================================================================================
/srv            data from daemons/services of this system.
                site-specific data served by this system.
=============================================================================================================

Reasons for the separation of files by design

  • static or variable
  • shareable or unshareable

Shareable means just "is storable on one host and usable on another". Unshareable cannot be.

Static data usually does not change much opposed to variable data, thus it can be stored on read-only media, for example.

Also there is no need to back it up as much as variable data. (Which explains why this distinction actually makes sense, even though this adds additional complexity.)

Some examples for unshareable:
/boot is static and not shareable.
/var/run and /var/lock are variable and not shareable.

Some examples for shareable data:
/var/mail is variable and shareable. /usr or /opt are static and shareable.

Static / read-only being cleanly separated is becoming more important nowadays, since SSD's work better with RO data.

This grew quite a bit longer than expected.

Linux file packers
posted on 2014-05-03 08:15:08

Cheatsheet for zipping/unzipping:

# if you can install programs
$ unp <archive>

unp will determine the filetype etc. by itself and just works.

Else:

## EXTRACT

# .tar.gz
$ tar xzvf <archive>

# .tar.bz2
$ tar xjvf <archive>

And:

## COMPRESS

# .tar.gz
$ tar czvf <archive>

# .tar.bz2
$ tar cjvf <archive>
Linux system users
posted on 2014-05-02 09:35:11

Finding substantiated info on this topic via google is really hard...

Here's what the user id's mean, depending on the number range where they are from:

 The UID and GID numbers are divided into classes as follows:

 0-99:
 Globally allocated by the Debian project, the same on every Debian system.
 These ids will appear in the passwd and group files of all Debian systems.
 New ids in this range are added automatically as the base-passwd package updates.

 Packages which need a single statically allocated uid or gid should use these.
 Their maintainers should ask the base-passwd maintainer for ids.

 100-999:
 Dynamically allocated system users and groups. 
 Packages which need a user or group, but can have this user or group allocated 
 dynamically and differently on each system.
 Should use adduser --system to create the group and/or user. 
 adduser will check for the existence of the user or group.
 If necessary choose an unused id based on the ranges specified in adduser.conf.


 1000-59999:
 Dynamically allocated user accounts. 
 By default adduser will choose UIDs and GIDs for user accounts in this range, 
 though adduser.conf may be used to modify this behavior.


 60000-64999:
 Globally allocated by the Debian project, but only created on demand. 
 The ids are allocated centrally and statically, but the actual accounts are 
 only created on users' systems on demand.

 These ids are for packages which are obscure or which require many 
 statically-allocated ids.
 These packages should check for and create the accounts in /etc/passwd or 
 /etc/group (using adduser if it has this facility) if necessary.
 Packages which are likely to require further allocations should have a "hole" 
 left after them in the allocation, to give them room to grow.


 65000-65533:
 Reserved.


 65534:
 User nobody. The corresponding gid refers to the group nogroup.


 65535:
 (uid_t)(-1) == (gid_t)(-1) must not be used, because it is the error return 
 sentinel value.

Source for this was the Debian Policy Manual. Above is almost 1:1 like from in the source.

Working with linux users
posted on 2014-05-01 18:49:21

Usually you want this when creating a new basic user on a linux system:

$ useradd -m -U <username>

-m creates a homefolder in /home/<username>, -U creates a group with the same name as the specified. -U can be omitted, since it's the default, but what if the defaults on the system you are working on have been changed by someone else?

In case you want to initially revoke login rights, use

$ useradd -m -U -s /usr/sbin/nologin <username>

where -s sets the shell to /usr/sbin/nologin. Another possibility is to use /bin/false. The /bin/false setting will just prevent logging in without an error message, whereas /usr/sbin/nologin will either print This account is currently not available. or whatever is specified in /etc/nologin.txt.

For 'system' users you should use /bin/false if no login is needed, or /bin/sh if it is.

But what if these need no homefolder? Or a group with the same name as their username?

$ useradd -r -d /tmp -G <group> -s /bin/false <username>

The example above will create a user that cannot login and has no dedicated homefolder, and a user id residing within the system user id range. /etc/passwd will show /tmp as homefolder, but thats about it. -r classifies the user as a system user, means he has an user / group id below 1000 usually.
This depends on the linux distribution you are using, IIRC, i.e. in Debian see here.

This has to do with internal filtering, so regulars can be distinguished from system users by id.

Maybe you have to have to set an account for a real person, so lets also specify also his full name in a comment via -c and give him access to a proper shell:

$ useradd -m -U -G <group1>,<group2> -s /bin/bash -c "<REAL NAME>" <username>

Specifying the shell is important, else he will have only a sh, not a bash. If you want, you can change this behaviour in /etc/defaults/useradd.

How about adding the user to groups, too?

$ useradd -m -U -G <group1>,<group2> -s /bin/bash -c "<REAL NAME>" <username>

Changing user settings can either be done editing /etc/passwd, but DO NOT DO THIS DIRECTLY!
Use vipw instead, it will lock the file so concurrent updates are impossible. Same goes for /etc/groups, use vigr for this.
For editing the shadow files like /etc/shadow (users) or /etc/gshadow (groups) use vipw -s and vigr -s.
Changing sudo rights is done via visudo.

You can also use one of the following:

chsh = change a user's shell
chfn = change user information such as real name and more
passwd = change user's password
usermod = change users's properties
userdel = delete a user

For userdel you usually want to use userdel -r <username> so the mail and the homefolder will be deleted. Keep in mind this might also delete the dedicated user group .

For creating users in batch use newusers.

Using rdesktop
posted on 2014-05-01 11:34:57

Either use a graphical frontend, or in case you want the console command for binding it to a shortcut:

rdesktop -g 800x600 -d <domain> -u <user> -p <pw> 123.123.123.123

g is short for geometry and means the resolution, which can be arbitrarily chosen. It can also be set as a percentage value, i.e. 80%.
d, u, p mean domain, user, password.

This blog covers .csv, .htaccess, .pfx, .vmx, /etc/crypttab, /etc/network/interfaces, /etc/sudoers, /proc, 10.04, 14.04, AS, ASA, ControlPanel, DS1054Z, GPT, HWR, Hyper-V, IPSEC, KVM, LSI, LVM, LXC, MBR, MTU, MegaCli, PHP, PKI, R, RAID, S.M.A.R.T., SNMP, SSD, SSL, TLS, TRIM, VEEAM, VMware, VServer, VirtualBox, Virtuozzo, XenServer, acpi, adaptec, algorithm, ansible, apache, apachebench, apple, arcconf, arch, architecture, areca, arping, asa, asdm, awk, backup, bandit, bar, bash, benchmarking, binding, bitrate, blackarmor, blowfish, bochs, bond, bonding, booknotes, bootable, bsd, btrfs, buffer, c-states, cache, caching, ccl, centos, certificate, certtool, cgdisk, cheatsheet, chrome, chroot, cisco, clamav, cli, clp, clush, cluster, coleslaw, colorscheme, common lisp, console, container, containers, controller, cron, cryptsetup, csync2, cu, cups, cygwin, d-states, database, date, db2, dcfldd, dcim, dd, debian, debug, debugger, debugging, decimal, desktop, df, dhclient, dhcp, diff, dig, display manager, dm-crypt, dmesg, dmidecode, dns, docker, dos, drivers, dtrace, dtrace4linux, du, dynamictracing, e2fsck, eBPF, ebook, efi, egrep, emacs, encoding, env, error, ess, esx, esxcli, esxi, ethtool, evil, expect, exportfs, factory reset, factory_reset, factoryreset, fail2ban, fbsd, fedora, file, filesystem, find, fio, firewall, firmware, fish, flashrom, forensics, free, freebsd, freedos, fritzbox, fsck, fstrim, ftp, ftps, g-states, gentoo, ghostscript, git, git-filter-branch, github, gitolite, gnutls, gradle, grep, grml, grub, grub2, guacamole, hardware, haskell, hdd, hdparm, hellowor, hex, hexdump, history, howto, htop, htpasswd, http, httpd, https, i3, icmp, ifenslave, iftop, iis, imagemagick, imap, imaps, init, innoDB, innodb, inodes, intel, ioncube, ios, iostat, ip, iperf, iphone, ipmi, ipmitool, iproute2, ipsec, iptables, ipv6, irc, irssi, iw, iwconfig, iwlist, iwlwifi, jailbreak, jails, java, javascript, javaws, js, juniper, junit, kali, kde, kemp, kernel, keyremap, kill, kpartx, krypton, lacp, lamp, languages, ldap, ldapsearch, less, leviathan, liero, lightning, links, linux, linuxin3months, lisp, list, livedisk, lmctfy, loadbalancing, locale, log, logrotate, looback, loopback, losetup, lsblk, lsi, lsof, lsusb, lsyncd, luks, lvextend, lvm, lvm2, lvreduce, lxc, lxde, macbook, macro, magento, mailclient, mailing, mailq, manpages, markdown, mbr, mdadm, megacli, micro sd, microsoft, minicom, mkfs, mktemp, mod_pagespeed, mod_proxy, modbus, modprobe, mount, mouse, movement, mpstat, multitasking, myISAM, mysql, mysql 5.7, mysql workbench, mysqlcheck, mysqldump, nagios, nas, nat, nc, netfilter, networking, nfs, nginx, nmap, nocaps, nodejs, numberingsystem, numbers, od, onyx, opcode-cache, openVZ, openlierox, openssl, openvpn, openvswitch, openwrt, oracle linux, org-mode, os, oscilloscope, overview, parallel, parameter expansion, parted, partitioning, passwd, patch, pdf, performance, pfsense, php, php7, phpmyadmin, pi, pidgin, pidstat, pins, pkill, plesk, plugin, posix, postfix, postfixadmin, postgres, postgresql, poudriere, powershell, preview, profiling, prompt, proxmox, ps, puppet, pv, pvecm, pvresize, python, qemu, qemu-img, qm, qmrestore, quicklisp, r, racktables, raid, raspberry pi, raspberrypi, raspbian, rbpi, rdp, redhat, redirect, registry, requirements, resize2fs, rewrite, rewrites, rhel, rigol, roccat, routing, rs0485, rs232, rsync, s-states, s_client, samba, sar, sata, sbcl, scite, scp, screen, scripting, seafile, seagate, security, sed, serial, serial port, setup, sftp, sg300, shell, shopware, shortcuts, showmount, signals, slattach, slip, slow-query-log, smbclient, snmpget, snmpwalk, software RAID, software raid, softwareraid, sophos, spacemacs, spam, specification, speedport, spi, sqlite, squid, ssd, ssh, ssh-add, sshd, ssl, stats, storage, strace, stronswan, su, submodules, subzone, sudo, sudoers, sup, swaks, swap, switch, switching, synaptics, synergy, sysfs, systemd, systemtap, tar, tcpdump, tcsh, tee, telnet, terminal, terminator, testdisk, testing, throughput, tmux, todo, tomcat, top, tput, trafficshaping, ttl, tuning, tunnel, tunneling, typo3, uboot, ubuntu, ubuntu 16.04, udev, uefi, ulimit, uname, unetbootin, unit testing, upstart, uptime, usb, usbstick, utf8, utm, utm 220, ux305, vcs, vgchange, vim, vimdiff, virtualbox, virtualization, visual studio code, vlan, vmstat, vmware, vnc, vncviewer, voltage, vpn, vsphere, vzdump, w, w701, wakeonlan, wargames, web, webdav, weechat, wget, whois, wicd, wifi, windowmanager, windows, wine, wireshark, wpa, wpa_passphrase, wpa_supplicant, x2x, xfce, xfreerdp, xmodem, xterm, xxd, yum, zones, zsh

View posts from 2017-03, 2017-02, 2017-01, 2016-12, 2016-11, 2016-10, 2016-09, 2016-08, 2016-07, 2016-06, 2016-05, 2016-04, 2016-03, 2016-02, 2016-01, 2015-12, 2015-11, 2015-10, 2015-09, 2015-08, 2015-07, 2015-06, 2015-05, 2015-04, 2015-03, 2015-02, 2015-01, 2014-12, 2014-11, 2014-10, 2014-09, 2014-08, 2014-07, 2014-06, 2014-05, 2014-04, 2014-03, 2014-01, 2013-12, 2013-11, 2013-10


Unless otherwise credited all material Creative Commons License by sjas