diff options
Diffstat (limited to 'README.md')
| -rw-r--r-- | README.md | 216 |
1 files changed, 184 insertions, 32 deletions
@@ -31,17 +31,15 @@ Usage ----- The `erebos-tester` tool, when executed without any arguments, -looks for a `erebos-tester.yaml` file in the current or any parent directory (see below for details). -Run `erebos-tester --help` for details about command-line parameters. +looks for an `erebos-tester.yaml` file in the current or any parent directory (see below for details). +Run `erebos-tester --help` for details about available command-line parameters. -The tester can be installed from sources or directly via cabal: -``` -cabal install erebos-tester -``` +### Examples -When available in the `PATH`, it can be run to test the [Haskell Erebos implementation](https://erebosprotocol.net/erebos): +When available in the `PATH`, it can be run, for example, to test +the [Haskell Erebos implementation](https://erebosprotocol.net/erebos): ``` -git clone git://erebosprotocol.net/erebos +git clone https://code.erebosprotocol.net/erebos cd erebos cabal build erebos-tester --tool="$(cabal list-bin erebos) test" --verbose @@ -49,23 +47,51 @@ erebos-tester --tool="$(cabal list-bin erebos) test" --verbose or the [C++ one](https://erebosprotocol.net/cpp): ``` -git clone git://erebosprotocol.net/cpp +git clone https://code.erebosprotocol.net/cpp cd cpp cmake -B build cmake --build build erebos-tester --verbose ``` -To run tests from a given test file, pass it as command-line argument: +### Running + +To run all tests from project configuration (see below), run the tester without any argument: +``` +erebos-tester +``` + +To run only some named tests, list the names on command line: +``` +erebos-tester FirstTest SecondTest +``` + +To run tests from a given test file, pass it as command-line argument (the path +must contain a slash, so use e.g. `./script.et` for script in the current +directory): ``` -erebos-tester path/to/script.test +erebos-tester path/to/script.et ``` To select single test from a file, use `:` separator: ``` -erebos-tester path/to/script.test:TestName +erebos-tester path/to/script.et:TestName ``` +### Reports + +By default, `erebos-tester` stops when a test fails, showing backtrace and +values of used variables. That can be changed with the following command-line +options: + +* `--report`: run all the tests, continuing even in the case of error, and + print a short summary of the number of passed and failed test, and a list of + those that failed. + +* `--junit-report=<path>`: run all the tests, and write the report to the file + in `<path>` using the JUnit XML format. + + Configuration ------------- @@ -76,6 +102,11 @@ This is a YAML file with following fields: * `tool`: path to the test tool, which may be overridden by the `--tool` command-line option. * `tests`: glob pattern that expands to all the test script files that should be used. +* `select`: tests or tags to be selected for running by default (if not provided, all tests will be executed); + given as a single `string` or a list of `string`s. +* `exclude`: tests or tags to be excluded from running (unless requested explicitly on command line); + given as a single `string` or a list of `string`s. +* `timeout`: initial timeout in seconds for test steps like `expect`, given as `int` or `float`; defaults to `1` if not specified. Script language --------------- @@ -106,7 +137,14 @@ The script language is strictly typed without any implicit conversions, although types can not be (as of now) declared explicitly and are always inferred. Each expression has specific concrete type, polymorphic types are not supported (yet). -#### integer +Generally, types of expressions should be inferred, but they can also be given +explicitly to any (sub)expression using the `:` notation: +``` +let x = 1 : Integer +let y = (2 : Integer) + (x : Integer) +``` + +#### `Integer` Integer numbers. Entered as decimal literals and used in arithmetic expressions: ``` @@ -115,7 +153,7 @@ let y = 3 let z = x * 2 + y ``` -#### number +#### `Number` Arbitrary-precision numbers. Entered as literals with decimal point or percentage and used in arithmetic expressions: ``` @@ -124,7 +162,7 @@ let y = 34% let z = x * 2.0 + y ``` -#### string +#### `String` String literals are enclosed in double quotes (`"`), using backslash to escape special characters (`"`, `\` and `$`) @@ -147,7 +185,7 @@ let b = 3 let s = "abc ${2*a + b}" # = "abc 7" ``` -#### regex +#### `Regex` Regular expression literals are enclosed in slash characters (`/`): ``` @@ -163,15 +201,16 @@ let re1 = /./ let re2 = /$str$re1/ # match '.' followed by any character ``` -#### boolean +#### `Bool` Result of comparison operators `==` and `/=`. +Values are `True` and `False`. -#### network +#### `Network` Represents network/subnet, created by `subnet` command and used by `subnet`, `node`, `spawn` and network configuration commands. -#### node +#### `Node` Represents network node, created by `node` command or implicitly by `spawn`, and used by `spawn` or network configuration commands. @@ -185,9 +224,9 @@ Members: : String representation of the node primary IP address. `network` -: The network which the node belogs to. +: The network which the node belongs to. -#### process +#### `Process` Represents running process. Created by `spawn`, used by `send` and `expect` commands. @@ -196,7 +235,10 @@ Members: `node` : Node on which the process is running. -#### asset +`pid` +: PID of the corresponding system process, `0` if there is none. + +#### `Asset` Represents an asset (file or directory), which can be used during test execution. @@ -205,16 +247,30 @@ Members: `path` : Path to the asset valid during the test execution. +#### `Tag` + +Tag, which can be assigned to a test using the `tag: <Tag>` declaration. + +#### `Signal` + +Type representing unix signals sent to processes. Values are `SIGINT`, `SIGTERM`, etc. + #### list -Lists are written using bracket notation: +Lists are written using bracket notation, and brackets are also used to express the type: ``` -let numbers = [1, 2, 4] +let numbers = [1, 2, 4] : [Integer] ``` List elements can be of any type, but all elements of a particular list must have the same type. - -Used in the `for` command. +They can be concatenated using the `concat` function, which takes a list of lists as argument: +``` +let list = concat [[1], [2, 3], [4]] # = [1, 2, 3, 4] +``` +Or with the `++` operator: +``` +let list = [1] ++ [2, 3] ++ [4] # = [1, 2, 3, 4] +``` ### Built-in commands @@ -231,13 +287,15 @@ node <name> [on <network>] Create a node on network `<network>` (or context network if omitted) and assign the new node to the variable `<name>`. ``` -spawn as <name> [on (<node> | <network>)] +spawn as <name> [on (<node> | <network>)] [args <arguments>] [killwith <signal>] ``` Spawn a new test process on `<node>` or `<network>` (or one from context) and assign the new process to variable `<name>`. When spawning on network, create a new node for this process. +Extra `<arguments>` to the tool can be given as a list of strings using the `args` keyword. The process is terminated when the variable `<name>` goes out of scope (at the end of the block in which it was created) by closing its stdin. +If the `killwith` clause is present, it is also sent the given `<signal>` at that point. When the process fails to terminate successfully within a timeout, the test fails. ``` @@ -246,7 +304,7 @@ send <string> to <process> Send line with `<string>` to the standard input of `<process>`. ``` -expect <regex> from <process> [capture <var1> [, <var2> ... ]] +expect <regex> from <process> [timeout <timeout>] [capture <var1> [, <var2> ... ]] ``` Check whether `<process>` produces line matching `<regex>` on standard output, and if this does not happen within current timeout, the test fails. Output lines produced before starting this command and not matched by some previous `expect` are accepted as well. @@ -259,6 +317,9 @@ The regular expression can contain capture groups – parts enclosed in parenthe In that case the expect command has to have the `capture` clause with matching number of variable names. Results of the captures are then assigned to the newly created variables as strings. +If the `timeout` clause is used, the current timeout value is multiplied by the given `<timeout>` for this `expect` call. +Timeout of zero can be used to expect a matching output line to have been already produced in the past. + ``` flush [from <proc>] [matching <regex>] ``` @@ -267,6 +328,15 @@ Flush memory of `<proc>` output, so no following `expect` command will match any If the `matching` clause is used, discard only output lines matching `<regex>`. ``` +ignore [from <proc>] [matching <regex>] +``` + +Ignore output lines from `<proc>` (or context process) that match the given +`<regex>` (or all lines if the `matching` clause is not used). Affects both +past and future output of the process; the effect lasts until the end of +the block. + +``` guard <expr> ``` @@ -322,12 +392,55 @@ with <expr>: Execute `<test block>` with `<expr>` as context. ``` +multiply_timeout by <multiplier> +``` + +Modify the timeout used for commands like `expect` by multiplying it with `<multiplier>`. +The effect lasts until the end of the block. + +``` wait ``` Wait for user input before continuing. Useful mostly for debugging or test development. +### Shell interpreter + +**Experimental feature**: Functionality is not fully implemented and behavior may change in incompatible ways between releases. + +Using the `shell` expression, it's possible to embed a shell script inside a test script. +The shell script is not passed to an external interpreter, but rather executed by the tester itself, +which allows the use of variables from the rest of the test script: + +``` +test: + node some_node + let x = "abc" + shell as sh on some_node: + echo $x > some_file + echo ${some_node.ip} >> some_file + cat some_file | sed 's/a/A/' > other_file +``` + +The syntax is intended to be generally similar to the classic Bourne shell, +however, only limited functionality is implemented so far (that includes executing commands, pipelines or input/output redirection). + +The general form of the `shell` expression is: + +``` +shell [as <name>] on <node>: + <shell commands> +``` + +Where `<node>` is the network node on which to run the script (it will be run in the network namespace of the node, and with working directory set to the node root), +and `<name>`, if given, is the name of the variable that will refer to the shell process (this can be used e.g. in the `expect` command to check the standard output of the script). +As with the `spawn` command, the resulting process is terminated at the end of the current scope. + +By default the shell process exists with failure whenever any command exits with non-zero status. +This behavior can be disabled using the `set +e` command (and re-enabled with `set -e`). + + ### Functions When calling a function, parameters are usually passed using argument keywords @@ -382,6 +495,12 @@ parentheses: def twice (x) = 2 * x ``` +Type of a given parameter can be also given explicitly using the `:` notation: +``` +def say (what : String) to (p : Process): + send what to p +``` + ### Modules, exports and imports Each test script file constitutes a module. As such, it can export definitions @@ -449,13 +568,46 @@ Such defined asset object can then be used in expressions within tests or functi ``` test: - spawn p + spawn as p send to p "use-asset ${my_asset.path}" ``` -The `my_asset.path` expression expands to a strict containing path to the asset -that can be used by the spawn process `p`. The process should not try to modify -the file. +The `my_asset.path` expression expands to a string containing path to the asset +that can be used by the spawned process `p`. The process should not try to +modify the file. + +Assets can be exported for use in other modules using the `export` keyword, +just like other definitions: + +``` +export asset my_asset: + path: ../path/to/file +``` + +### Tags + +Tags are a way to refer to a group of tests, instead of needing to list all their names individually; +for example to mark broken tests, which can then be easily excluded from running until fixed. +Tags are declared using the `tag` keyword on the top level of a module, +and need to be `export`ed if they are to be referenced from outside of that module: + +``` +export tag Broken +``` + +Tags can be assigned to tests in a the test preamble before the first test steps +using `tag: <Tag>` declaration, which can also be given multiple times: + +``` +test SomeBrokenTest: + tag: Broken + tag: OtherTag + spawn as p + ... +``` + +Such tags can then be used instead of test names to select or exclude tests on +command line or in the configuration file. Optional dependencies |