Erebos Tester ============= Developed for testing the [Erebos protocol implementation](https://erebosprotocol.net/erebos) and [C++ library](https://erebosprotocol.net/cpp), generally intended mainly for testing networking code – it provides an easy way to execute the tested program simultaneously on multiple nodes within a virtual network. These nodes can be configured in a single or multiple subnets, and their properties, like whether they are connected, can be changed during test run according to a given script. The test framework uses two components from the tested project: 1. Test tool – executable that accepts commands on standard input, executes the tested functionality based on those, and provides output on standard output. The output can be asynchronous, i.e. some events may trigger a message to be generated while waiting for output of a command; interpretation is up to the provided test script. The test tool can be set either: * by the `--tool` command-line parameter of `erebos-tester`, or * in the `erebos-tester.yaml` configuration file, or * by the `EREBOS_TEST_TOOL` environment variable. 2. Test script – defines how to run the instances of test tool and in what kind of network topology. Contains commands to send to the test tool instances and rules to interpret the responses. The script is written in a custom language described below. Usage ----- The `erebos-tester` tool, when executed without any arguments, looks for a `erebos-tester.yaml` file in the current or any parent directory (see below for details). Run `erebos-tester --help` for details about command-line parameters. The tester can be installed from sources or directly via cabal: ``` cabal install erebos-tester ``` When available in the `PATH`, it can be run to test the [Haskell Erebos implementation](https://erebosprotocol.net/erebos): ``` git clone git://erebosprotocol.net/erebos cd erebos cabal build erebos-tester --tool="$(cabal list-bin erebos) test" --verbose ``` or the [C++ one](https://erebosprotocol.net/cpp): ``` git clone git://erebosprotocol.net/cpp cd cpp cmake -B build cmake --build build erebos-tester --verbose ``` To run tests from a given test file, pass it as command-line argument: ``` erebos-tester path/to/script.test ``` To select single test from a file, use `:` separator: ``` erebos-tester path/to/script.test:TestName ``` Configuration ------------- To allow running `erebos-tester` without the need to supply project-specific configuration on command line, per-project configuration can be done using `erebos-tester.yaml` file placed in the root of the project (or other directory from which the tests will be executed). This is a YAML file with following fields: * `tool`: path to the test tool, which may be overridden by the `--tool` command-line option. * `tests`: glob pattern that expands to all the test script files that should be used. Script language --------------- The test script language uses indentation to define the command blocks, e.g. to define a test body or denote the scope of variables. Each command is on its own line, terminated by newline. Commands accept arguments preceded by name/keyword in arbitrary order, to make the behavior clear without the need to know the expected order of the parameters. For examples, see tests within the [Erebos implementation repository](https://code.erebosprotocol.net/erebos/tree/test). Each test script file consists of one or more test cases, started with `test` keyword, with its body within indented block: ``` test []: ``` Test name is optional, but if present can be used to run the single test from a file that contains multiple tests. ### Types The script language is strictly typed without any implicit conversions, although types can not be (as of now) declared explicitly and are always inferred. Each expression has specific concrete type, polymorphic types are not supported (yet). #### integer Integer numbers. Entered as decimal literals and used in arithmetic expressions: ``` let x = 2 let y = 3 let z = x * 2 + y ``` #### number Arbitrary-precision numbers. Entered as literals with decimal point or percentage and used in arithmetic expressions: ``` let x = 2.1 let y = 34% let z = x * 2.0 + y ``` #### string String literals are enclosed in double quotes (`"`), using backslash to escape special characters (`"`, `\` and `$`) and to represent some others (`\n` for newline). ``` let s = "some text" ``` Dollar sign (`$`) can be used to expand variables (numbers are expanded to decimal representation). ``` let a = "abc" let b = 4 let c = "$a $b" # = "abc 4" ``` Arbitrary expression can be used within additional curly braces: ``` let a = 2 let b = 3 let s = "abc ${2*a + b}" # = "abc 7" ``` #### regex Regular expression literals are enclosed in slash characters (`/`): ``` let re = /a.*/ # match any string starting with 'a' ``` Dollar-expansion can be used here as well. Strings expand to regular expressions matching the exact string, regular expressions expand are used directly. ``` let str = "." let re1 = /./ let re2 = /$str$re1/ # match '.' followed by any character ``` #### boolean Result of comparison operators `==` and `/=`. #### network Represents network/subnet, created by `subnet` command and used by `subnet`, `node`, `spawn` and network configuration commands. #### node Represents network node, created by `node` command or implicitly by `spawn`, and used by `spawn` or network configuration commands. Members: `ip` : String representation of node's IP address. `network` : The network which the node belogs to. #### process Represents running process. Created by `spawn`, used by `send` and `expect` commands. Members: `node` : Node on which the process is running. #### list Lists are written using bracket notation: ``` let numbers = [1, 2, 4] ``` List elements can be of any type, but all elements of a particular list must have the same type. Used in the `for` command. ### Build-in commands ``` subnet [of ] ``` Create a subnet within a `` (or context network if omitted) and assign the new network to the variable ``. ``` node [on ] ``` Create a node on network `` (or context network if omitted) and assign the new node to the variable ``. ``` spawn as [on ( | )] ``` Spawn a new test process on `` or `` (or one from context) and assign the new process to variable ``. When spawning on network, create a new node for this process. The process is terminated when the variable `` goes out of scope (at the end of the block in which it was created) by closing its stdin. When the process fails to terminate successfully within a timeout, the test fails. ``` send to ``` Send line with `` to the standard input of ``. ``` expect from [capture [, ... ]] ``` Check whether `` produces line matching `` on standard output, and if this does not happen within current timeout, the test fails. Output lines produced before starting this command and not matched by some previous `expect` are accepted as well. Output lines not matching `` are ignored by this `expect` call, i.e. do not cause the `expect` call to fail. Regular expressions are anchored on both sides, so must match the entire line. If e.g. only the beginning should be matched, the passed regular expression needs to end with `.*`. The regular expression can contain capture groups – parts enclosed in parentheses (`(`, `)`). In that case the expect command has to have the `capture` clause with matching number of variable names. Results of the captures are then assigned to the newly created variables as strings. ``` flush [from ] [matching ] ``` Flush memory of `` output, so no following `expect` command will match anything produced up to this point. If the `matching` clause is used, discard only output lines matching ``. ``` guard ``` Check whether boolean expression `` is true; if not, the test fails. ``` disconnect_node [] ``` Disconnect `` from network – state of the veth network link from the node is set to down. The effect lasts until the end of the block. ``` disconnect_nodes [] ``` Disconnect all nodes of ``. The network bridge interface state is set to down. The effect lasts until the end of the block. ``` disconnect_upstream [] ``` Disconnect network upstream – state of the veth network link connecting network bridge to the upstream network is set to down. The effect lasts until the end of the block. ``` packet_loss [on ] ``` Set the packet loss rate on the node's veth link to `` as decimal number or percentage, e.g. `0.2` or `20%` for 20% packet loss rate. The effect lasts until the end of the block. ``` for in : ``` Execute `` for each element of list ``, with current element assigned to ``. ``` local: ``` Execute `` in a new local scope. Used to restrict scope of variables or duration of effects. ``` with : ``` Execute `` with `` as context. ``` wait ``` Wait for user input before continuing. Useful mostly for debugging or test development. ### Functions When calling a function, parameters are usually passed using argument keywords (in the case of built-in commands, those keywords are typically prepositions like `on`, `from`, etc.), and apart from those, there can be at most one parameter passed without a keyword. This is done in order to avoid the need to remember parameter order and to make the behavior of each call as clear as possible, even without looking up the documentation. To make the syntax unambiguous, the keywordless parameter can be passed as the last parameter, as a literal (number, string, etc.), or using parentheses. So this is ok: ``` expect /something/ from p ``` but if the regular expression is stored in a variable, the parameter needs to move to the end: ``` let re = /something/ expect from p re ``` or be enclosed in parentheses: ``` expect (re) from p ``` or in a literal: ``` expect /$re/ from p ``` ### Defining functions Custom functions can be defined on the top level using `def` keyword, and with the parameters either followed by `=` sign to return a value: ``` def twice x = 2 * x ``` or followed by `:` to define test block: ``` def greet p: send "hello" to p expect /hi/ from p ``` Those then can be invoked elsewhere: ``` test: spawn as p greet p ``` Optional dependencies --------------------- The test framework can use some other tools to help with debugging or development. ### GDB If GDB is installed, it's possible to use `--gdb` command-line switch of the `erebos-tester` tool to use the debugger. The GDB session is started in background and tester uses the GDB machine interface to communicate with it. Whenever a new process is spawned, it is attached to the debugger as a new inferior. In case any process is terminated by a signal, e.g. crashes with segfault, interactive GDB session is opened. ### tcpdump If `tcpdump` binary is found in the `PATH`, it is used to generate network log in the pcap capture format within the test directory, separately for each virtual network (specifically its bridge interface).