diff options
Diffstat (limited to 'README.md')
| -rw-r--r-- | README.md | 144 |
1 files changed, 133 insertions, 11 deletions
@@ -56,14 +56,26 @@ cmake --build build erebos-tester --verbose ``` -To run tests from a given test file, pass it as command-line argument: +To run all tests from project configuration (see below), run the tester without any argument: ``` -erebos-tester path/to/script.test +erebos-tester +``` + +To run only some named tests, list the names on command line: +``` +erebos-tester FirstTest SecondTest +``` + +To run tests from a given test file, pass it as command-line argument (the path +must contain a slash, so use e.g. `./script.et` for script in the current +directory): +``` +erebos-tester path/to/script.et ``` To select single test from a file, use `:` separator: ``` -erebos-tester path/to/script.test:TestName +erebos-tester path/to/script.et:TestName ``` Configuration @@ -76,6 +88,11 @@ This is a YAML file with following fields: * `tool`: path to the test tool, which may be overridden by the `--tool` command-line option. * `tests`: glob pattern that expands to all the test script files that should be used. +* `select`: tests or tags to be selected for running by default (if not provided, all tests will be executed); + given as a single `string` or a list of `string`s. +* `exclude`: tests or tags to be excluded from running (unless requested explicitly on command line); + given as a single `string` or a list of `string`s. +* `timeout`: initial timeout in seconds for test steps like `expect`, given as `int` or `float`; defaults to `1` if not specified. Script language --------------- @@ -166,6 +183,7 @@ let re2 = /$str$re1/ # match '.' followed by any character #### boolean Result of comparison operators `==` and `/=`. +Values are `True` and `False`. #### network @@ -196,6 +214,9 @@ Members: `node` : Node on which the process is running. +`pid` +: PID of the corresponding system process, `0` if there is none. + #### asset Represents an asset (file or directory), which can be used during test execution. @@ -205,6 +226,14 @@ Members: `path` : Path to the asset valid during the test execution. +#### `Tag` + +Tag, which can be assigned to a test using the `tag: <Tag>` declaration. + +#### `Signal` + +Type representing unix signals sent to processes. Values are `SIGINT`, `SIGTERM`, etc. + #### list Lists are written using bracket notation: @@ -213,8 +242,14 @@ let numbers = [1, 2, 4] ``` List elements can be of any type, but all elements of a particular list must have the same type. - -Used in the `for` command. +They can be concatenated using the `concat` function, which takes a list of lists as argument: +``` +let list = concat [[1], [2, 3], [4]] # = [1, 2, 3, 4] +``` +Or with the `++` operator: +``` +let list = [1] ++ [2, 3] ++ [4] # = [1, 2, 3, 4] +``` ### Built-in commands @@ -231,13 +266,15 @@ node <name> [on <network>] Create a node on network `<network>` (or context network if omitted) and assign the new node to the variable `<name>`. ``` -spawn as <name> [on (<node> | <network>)] +spawn as <name> [on (<node> | <network>)] [args <arguments>] [killwith <signal>] ``` Spawn a new test process on `<node>` or `<network>` (or one from context) and assign the new process to variable `<name>`. When spawning on network, create a new node for this process. +Extra `<arguments>` to the tool can be given as a list of strings using the `args` keyword. The process is terminated when the variable `<name>` goes out of scope (at the end of the block in which it was created) by closing its stdin. +If the `killwith` clause is present, it is also sent the given `<signal>` at that point. When the process fails to terminate successfully within a timeout, the test fails. ``` @@ -246,7 +283,7 @@ send <string> to <process> Send line with `<string>` to the standard input of `<process>`. ``` -expect <regex> from <process> [capture <var1> [, <var2> ... ]] +expect <regex> from <process> [timeout <timeout>] [capture <var1> [, <var2> ... ]] ``` Check whether `<process>` produces line matching `<regex>` on standard output, and if this does not happen within current timeout, the test fails. Output lines produced before starting this command and not matched by some previous `expect` are accepted as well. @@ -259,6 +296,9 @@ The regular expression can contain capture groups – parts enclosed in parenthe In that case the expect command has to have the `capture` clause with matching number of variable names. Results of the captures are then assigned to the newly created variables as strings. +If the `timeout` clause is used, the current timeout value is multiplied by the given `<timeout>` for this `expect` call. +Timeout of zero can be used to expect a matching output line to have been already produced in the past. + ``` flush [from <proc>] [matching <regex>] ``` @@ -267,6 +307,15 @@ Flush memory of `<proc>` output, so no following `expect` command will match any If the `matching` clause is used, discard only output lines matching `<regex>`. ``` +ignore [from <proc>] [matching <regex>] +``` + +Ignore output lines from `<proc>` (or context process) that match the given +`<regex>` (or all lines if the `matching` clause is not used). Affects both +past and future output of the process; the effect lasts until the end of +the block. + +``` guard <expr> ``` @@ -322,12 +371,52 @@ with <expr>: Execute `<test block>` with `<expr>` as context. ``` +multiply_timeout by <multiplier> +``` + +Modify the timeout used for commands like `expect` by multiplying it with `<multiplier>`. +The effect lasts until the end of the block. + +``` wait ``` Wait for user input before continuing. Useful mostly for debugging or test development. +### Shell interpreter + +**Experimental feature**: Functionality is not fully implemented and behavior may change in incompatible ways between releases. + +Using the `shell` expression, it's possible to embed a shell script inside a test script. +The shell script is not passed to an external interpreter, but rather executed by the tester itself, +which allows the use of variables from the rest of the test script: + +``` +test: + node some_node + let x = "abc" + shell as sh on some_node: + echo $x > some_file + echo ${some_node.ip} >> some_file + cat some_file | sed 's/a/A/' > other_file +``` + +The syntax is intended to be generally similar to the classic Bourne shell, +however, only limited functionality is implemented so far (that includes executing commands, pipelines or input/output redirection). + +The general form of the `shell` expression is: + +``` +shell [as <name>] on <node>: + <shell commands> +``` + +Where `<node>` is the network node on which to run the script (it will be run in the network namespace of the node, and with working directory set to the node root), +and `<name>`, if given, is the name of the variable that will refer to the shell process (this can be used e.g. in the `expect` command to check the standard output of the script). +As with the `spawn` command, the resulting process is terminated at the end of the current scope. + + ### Functions When calling a function, parameters are usually passed using argument keywords @@ -449,13 +538,46 @@ Such defined asset object can then be used in expressions within tests or functi ``` test: - spawn p + spawn as p send to p "use-asset ${my_asset.path}" ``` -The `my_asset.path` expression expands to a strict containing path to the asset -that can be used by the spawn process `p`. The process should not try to modify -the file. +The `my_asset.path` expression expands to a string containing path to the asset +that can be used by the spawned process `p`. The process should not try to +modify the file. + +Assets can be exported for use in other modules using the `export` keyword, +just like other definitions: + +``` +export asset my_asset: + path: ../path/to/file +``` + +### Tags + +Tags are a way to refer to a group of tests, instead of needing to list all their names individually; +for example to mark broken tests, which can then be easily excluded from running until fixed. +Tags are declared using the `tag` keyword on the top level of a module, +and need to be `export`ed if they are to be referenced from outside of that module: + +``` +export tag Broken +``` + +Tags can be assigned to tests in a the test preamble before the first test steps +using `tag: <Tag>` declaration, which can also be given multiple times: + +``` +test SomeBrokenTest: + tag: Broken + tag: OtherTag + spawn as p + ... +``` + +Such tags can then be used instead of test names to select or exclude tests on +command line or in the configuration file. Optional dependencies |