= Cluster Test Harness =

== What is this? ==

This is meant to be a generic Cluster Test Harness (CTH).
Its main purpose is to simulate hardware failure.
Since the Hardware aspects are abstracted, this can be used to test any
software/hardware interaction in presence of hardware failures.
Because it is used to test DRBDs behaviour, some DRBD specifics are built in.

The CTH expects
 * to run on one controlling box
 * which has exclusive access to at least two test nodes via ssh
 * that the ssh login will not ask for a password, but just let me through
 * the test nodes run linux kernel 2.6 (may work with 2.4, too)
 * the test nodes to have "dm" available
 * the test nodes to have "iptables" available
 * the test nodes to '''not''' start heartbeat or drbd
 from their init scripts, or simulating a node failure
 won't work as intended

Test nodes may well be UML sessions.

'''Note''' that since some path's are hardcoded,
the CTH expects to be run from the current directory.

=== Requirements ===

Some of the internal scripting was easier done with versions of grep and ping,
that Debian (and maybe other distros) has not installed by default.

In particular, I expect
 * ping from iputils-ping instead of netkit-ping
 * grep > 2.5 (e.g. from 2.5.1.ds1-1.backports.org.1)

For stress testing, I like to use tiobench, and wbtest.
If you want to use them, you should compile and install them
into e.g. /root/bin on the test nodes.

== File List ==

 LGE_CTH*::
 The perl module. Only adventurouse people should have a look at the internals here.
 You should be at least perl "journeyman", otherwise please stay out.

 functions.sh::
 The core bash scripts, which are used and triggered by the CTH.

 generic_test.pl::
 Example of how to use the LGE_CTH perl module, use a certain configuration,
 and generate randome hardware failures and test resource relocations

 Since some of it is still hardcoded, you have to edit it,
 choose a config file, and some parameters.

 some sample configuration files:
 * uml-minna.conf
 * bloodymary.conf

 CTH_bash.sh::
 Example of how to use the core functions directly from bash.
 In contrast to generic_test.pl and the LGE_CTH perl modules,
 this is meant to script one particular failure scenario.

 The idea is that you create a config file matching your setup
 (see the next two for example), then script a simple test case 
 you suspect to be a problem for DRBD.
 
 Then get you an extra bash instance (because the internal functions may
 explicitly call "exit", which otherwise will kick your seesion out),
 and then "source CTH_bash.sh your.sh.conf T-00X.sh", which will run the
 test case, and drops you back to your extra bash instance, which now
 has an "extended command set" and environment, so you can interactively
 investigate what went wrong (if something went wrong), or reapeat
 certain steps, or do additional weird stuff.
 
 CTH_bash.conf::
 sample configuration for the CTH_bash.sh similar to uml-minna.conf

 bloodymary.sh.conf::
 sample configuration for the bash CTH similar to bloodymary.conf

 CTH_bash.helpers:: guess what ... to keep the CTH_bash.sh clean.

 T-###.sh::
 simple example test cases to show how this is intended to be used.

=== Test Resources ===
 * tiobench
 * wbtest

== AUTHOR ==

of all this crap is Lars Ellenberg <l.g.e@web.de>,
if not otherwise stated.

In case you care for an explicit '''license statement''':
This is and needs to be GPL.

# if you don't know wiki-moin.vim: http://linuxha.trick.ca/WikiTricks
# vim: set ft=wiki-moin :
