Posts tagged testing

On test layers

posted on 2016-04-24 15:38

After I have seen one too many diagrams on this, I decided to do my own writeup and get to know all corresponding english terms in the progress. The listing is chronologically ordered in regard to a usual project progressing.

Depending on the progress model being used, it may be less waterfall-y and more incremental. If you happen to make sense of that last sentence, you should know where to make adjustments.

This is more or less a rant from experience and just here for pure entertainment value.

development and quality assurance happening (so you know wether you need only dev's, or testers, too)

1. requirements tests

"First there has to be clear what exactly shall be built."

This can be done via:

  • prototyping (throw-away software implementations)
  • user interface designs and validating these with the customer

What shall be verified here:

  • software requirements specification to be approved by the customer afterwards as a contractual basis
  • do not implement any nice-to-haves unless you really have free space for such gimmickry you so identified!
  • domain object models consisting of domain classes
  • user scenarios (UML might actually help here)

2. design tests

This write up may not reflect it, but errors in 1. and 2. are the most costly. Let the devs begin, but make sure the eventually get a rock solid specification and design.

"Is the current design feasible at all?"

Discussions, cost estimations, a vague feeling of despair creeps in.

The easy way to make sure most of your estimations will work is talking with your (willing AND able) devs, and ask them for estimations. Then simply double the said time (if the dev is good, 1day->2days), or double it and increase by an magnitude of order (1day->2weeks).

Start priorizing features in accordance to the results from the requirements tests, especially WRT the part which was written in CAPSLOCK above.

  • profitability (will the budget really work?)
  • correctness (will it what it should do?)
  • tracability (can each design artefact be traced back to an origin in the SRS?)

Triple-check traceability. SERIOUSLY. Each error will have a huge string of other problems following afterwards. Correctness errors should be realized in the progress, and capacity planning can be done in parallel.

Errors during this 'phase' will be costly due to all the manpower going into development, releases and acceptance testing.

development happening (testing too, if dev's don't write their unit tests)

3. unit tests, main classes

"Watch the programmers a little..."

Use the unit testing framework of your choice.

  • correctness (does it what it should do?)
  • completeness (does it everything it should?)
  • test coverage (do we test all code that was written and all use case scenarios?)

Here, all main classes should get complete test coverage.

4. unit tests, all other classes

"Watch the programmmers even more..."

Use the unit testing framework of your choice.

  • correctness (does it what is should do?)
  • completeness (does it everything it should do?)
  • test coverage (do we test all code that was written and all use case scenarios?)

Less important code may be put off for complete testing, even though 100% coverage might feel better. But you have to pay everything that gets written, the best code is the one not needed, after all.

development and quality assurance happening

5. code reviews

"Make sure no swearwords can be found anywhere..."

Best these are done by pair programming and integration people, but this depends on your version control system workflow. If all people can commit to your main repository, there will be a lot of swearing in development. Tools like gerrit will help.

  • correctness (does it what it should do?)
  • comprehensibility (will the next employee be able to fix it?)
  • accordance to coding style guide lines (a uniform code layout prevents unneeded mistakes)
  • maintainability (can the code be patched if needed in feasible time without everything breaking each time?)

quality assurance happening

6. integration tests

"Does this even work?"

Try sticking all parts together by packing all code together and compiling it or set up your web application to run.

  • stability (does it run without freezing?)
  • completeness (does it do everything it should?)
  • correctness (does it do what it should?)
  • functionality (can the customer work with it?)
  • deliverability (can we deliver the software, or is release engineering more like playing the lottery?)
  • maintainability (can the code be patched if needed in feasible time without everything breaking each time?)
  • integrability (can all modules be integrated / do the defined interfaces work?)

7. system tests

"Can we sell that?"

Actually running the product on test installations by your people..

  • requirements (is everything in the SRS covered in the product?)
  • usability (can the customer work with it EASILY? If not,your errors lie in 2. above...)
  • completeness (does it do everything it should?)
  • stability (does it run without freezing?)
  • robustness (is it failure-tolerant, or do mysql injections works...)
  • performance (does it run fast, or do you have to profile the application and exchange parts of it with C code?)

8. acceptancs tests

"Let's see how long we will be over time or budget, if we can't drop some of the open issues..."

Actually running the product and letting the customer embarras your company as he finds obvious flaws within minutes which you should have seen months' ago.

  • user acceptance testing for planning final delivery or rollout by simply running the product

For now, most relevant stuff should have gone into here. However do not believe this is complete, I doubt I really have covered here by this quick write-up.

This blog covers .csv, .htaccess, .pfx, .vmx, /etc/crypttab, /etc/network/interfaces, /etc/sudoers, /proc, 10.04, 14.04, AS, ASA, ControlPanel, DS1054Z, GPT, HWR, Hyper-V, IPSEC, KVM, LSI, LVM, LXC, MBR, MTU, MegaCli, PHP, PKI, R, RAID, S.M.A.R.T., SNMP, SSD, SSL, TLS, TRIM, VEEAM, VMware, VServer, VirtualBox, Virtuozzo, XenServer, acpi, adaptec, algorithm, ansible, apache, apache2.4, apachebench, apple, applet, arcconf, arch, architecture, areca, arping, asa, asdm, autoconf, awk, backup, bandit, bar, bash, benchmarking, binding, bitrate, blackarmor, blockdev, blowfish, bochs, bond, bonding, booknotes, bootable, bsd, btrfs, buffer, c-states, cache, caching, ccl, centos, certificate, certtool, cgdisk, cheatsheet, chrome, chroot, cisco, clamav, cli, clp, clush, cluster, coleslaw, colorscheme, common lisp, configuration management, console, container, containers, controller, cron, cryptsetup, csync2, cu, cups, cygwin, d-states, database, date, db2, dcfldd, dcim, dd, debian, debug, debugger, debugging, decimal, desktop, df, dhclient, dhcp, diff, dig, display manager, dm-crypt, dmesg, dmidecode, dns, docker, dos, drivers, dtrace, dtrace4linux, du, dynamictracing, e2fsck, eBPF, ebook, efi, egrep, emacs, encoding, env, error, ess, esx, esxcli, esxi, ethtool, evil, expect, exportfs, factory reset, factory_reset, factoryreset, fail2ban, fbsd, fdisk, fedora, file, filesystem, find, fio, firewall, firmware, fish, flashrom, forensics, free, freebsd, freedos, fritzbox, fsck, fstrim, ftp, ftps, g-states, gentoo, ghostscript, git, git-filter-branch, github, gitolite, global, gnutls, gradle, grep, grml, grub, grub2, guacamole, hardware, haskell, hdd, hdparm, hellowor, hex, hexdump, history, howto, htop, htpasswd, http, httpd, https, i3, icmp, ifenslave, iftop, iis, imagemagick, imap, imaps, init, innoDB, innodb, inodes, intel, ioncube, ios, iostat, ip, iperf, iphone, ipmi, ipmitool, iproute2, ipsec, iptables, ipv6, irc, irssi, iw, iwconfig, iwlist, iwlwifi, jailbreak, jails, java, javascript, javaws, js, juniper, junit, kali, kde, kemp, kernel, keyremap, kill, kpartx, krypton, lacp, lamp, languages, ldap, ldapsearch, less, leviathan, liero, lightning, links, linux, linuxin3months, lisp, list, livedisk, lmctfy, loadbalancing, locale, log, logrotate, looback, loopback, losetup, lsblk, lsi, lsof, lsusb, lsyncd, luks, lvextend, lvm, lvm2, lvreduce, lxc, lxde, macbook, macro, magento, mailclient, mailing, mailq, manpages, markdown, mbr, mdadm, megacli, micro sd, microsoft, minicom, mkfs, mktemp, mod_pagespeed, mod_proxy, modbus, modprobe, mount, mouse, movement, mpstat, multitasking, myISAM, mysql, mysql 5.7, mysql workbench, mysqlcheck, mysqldump, nagios, nas, nat, nc, netfilter, networking, nfs, nginx, nmap, nocaps, nodejs, numberingsystem, numbers, od, onyx, opcode-cache, openVZ, openlierox, openssl, openvpn, openvswitch, openwrt, oracle linux, org-mode, os, oscilloscope, overview, parallel, parameter expansion, parted, partitioning, passwd, patch, pct, pdf, performance, pfsense, php, php7, phpmyadmin, pi, pidgin, pidstat, pins, pkill, plasma, plesk, plugin, posix, postfix, postfixadmin, postgres, postgresql, poudriere, powershell, preview, profiling, prompt, proxmox, ps, puppet, pv, pveam, pvecm, pvesm, pvresize, python, qemu, qemu-img, qm, qmrestore, quicklisp, quickshare, r, racktables, raid, raspberry pi, raspberrypi, raspbian, rbpi, rdp, redhat, redirect, registry, requirements, resize2fs, rewrite, rewrites, rhel, rigol, roccat, routing, rs0485, rs232, rsync, s-states, s_client, samba, sar, sata, sbcl, scite, scp, screen, scripting, seafile, seagate, security, sed, serial, serial port, setup, sftp, sg300, shell, shopware, shortcuts, showmount, signals, slattach, slip, slow-query-log, smbclient, snmpget, snmpwalk, software RAID, software raid, softwareraid, sophos, spacemacs, spam, specification, speedport, spi, sqlite, squid, ssd, ssh, ssh-add, sshd, ssl, stats, storage, strace, stronswan, su, submodules, subzone, sudo, sudoers, sup, swaks, swap, switch, switching, synaptics, synergy, sysfs, systemd, systemtap, tar, tcpdump, tcsh, tee, telnet, terminal, terminator, testdisk, testing, throughput, tmux, todo, tomcat, top, tput, trafficshaping, ttl, tuning, tunnel, tunneling, typo3, uboot, ubuntu, ubuntu 16.04, udev, uefi, ulimit, uname, unetbootin, unit testing, upstart, uptime, usb, usbstick, utf8, utm, utm 220, ux305, vcs, vgchange, vim, vimdiff, virtualbox, virtualization, visual studio code, vlan, vmstat, vmware, vnc, vncviewer, voltage, vpn, vsphere, vzdump, w, w701, wakeonlan, wargames, web, webdav, weechat, wget, whois, wicd, wifi, windowmanager, windows, wine, wireshark, wpa, wpa_passphrase, wpa_supplicant, x11vnc, x2x, xfce, xfreerdp, xmodem, xterm, xxd, yum, zones, zsh

Unless otherwise credited all material Creative Commons License by sjas