diff options
Diffstat (limited to 'README.md')
| -rw-r--r-- | README.md | 355 |
1 files changed, 325 insertions, 30 deletions
@@ -31,17 +31,15 @@ Usage ----- The `erebos-tester` tool, when executed without any arguments, -looks for a `erebos-tester.yaml` file in the current or any parent directory (see below for details). -Run `erebos-tester --help` for details about command-line parameters. +looks for an `erebos-tester.yaml` file in the current or any parent directory (see below for details). +Run `erebos-tester --help` for details about available command-line parameters. -The tester can be installed from sources or directly via cabal: -``` -cabal install erebos-tester -``` +### Examples -When available in the `PATH`, it can be run to test the [Haskell Erebos implementation](https://erebosprotocol.net/erebos): +When available in the `PATH`, it can be run, for example, to test +the [Haskell Erebos implementation](https://erebosprotocol.net/erebos): ``` -git clone git://erebosprotocol.net/erebos +git clone https://code.erebosprotocol.net/erebos cd erebos cabal build erebos-tester --tool="$(cabal list-bin erebos) test" --verbose @@ -49,23 +47,51 @@ erebos-tester --tool="$(cabal list-bin erebos) test" --verbose or the [C++ one](https://erebosprotocol.net/cpp): ``` -git clone git://erebosprotocol.net/cpp +git clone https://code.erebosprotocol.net/cpp cd cpp cmake -B build cmake --build build erebos-tester --verbose ``` -To run tests from a given test file, pass it as command-line argument: +### Running + +To run all tests from project configuration (see below), run the tester without any argument: +``` +erebos-tester +``` + +To run only some named tests, list the names on command line: +``` +erebos-tester FirstTest SecondTest +``` + +To run tests from a given test file, pass it as command-line argument (the path +must contain a slash, so use e.g. `./script.et` for script in the current +directory): ``` -erebos-tester path/to/script.test +erebos-tester path/to/script.et ``` To select single test from a file, use `:` separator: ``` -erebos-tester path/to/script.test:TestName +erebos-tester path/to/script.et:TestName ``` +### Reports + +By default, `erebos-tester` stops when a test fails, showing backtrace and +values of used variables. That can be changed with the following command-line +options: + +* `--report`: run all the tests, continuing even in the case of error, and + print a short summary of the number of passed and failed test, and a list of + those that failed. + +* `--junit-report=<path>`: run all the tests, and write the report to the file + in `<path>` using the JUnit XML format. + + Configuration ------------- @@ -76,6 +102,11 @@ This is a YAML file with following fields: * `tool`: path to the test tool, which may be overridden by the `--tool` command-line option. * `tests`: glob pattern that expands to all the test script files that should be used. +* `select`: tests or tags to be selected for running by default (if not provided, all tests will be executed); + given as a single `string` or a list of `string`s. +* `exclude`: tests or tags to be excluded from running (unless requested explicitly on command line); + given as a single `string` or a list of `string`s. +* `timeout`: initial timeout in seconds for test steps like `expect`, given as `int` or `float`; defaults to `1` if not specified. Script language --------------- @@ -106,7 +137,14 @@ The script language is strictly typed without any implicit conversions, although types can not be (as of now) declared explicitly and are always inferred. Each expression has specific concrete type, polymorphic types are not supported (yet). -#### integer +Generally, types of expressions should be inferred, but they can also be given +explicitly to any (sub)expression using the `:` notation: +``` +let x = 1 : Integer +let y = (2 : Integer) + (x : Integer) +``` + +#### `Integer` Integer numbers. Entered as decimal literals and used in arithmetic expressions: ``` @@ -115,7 +153,7 @@ let y = 3 let z = x * 2 + y ``` -#### number +#### `Number` Arbitrary-precision numbers. Entered as literals with decimal point or percentage and used in arithmetic expressions: ``` @@ -124,7 +162,7 @@ let y = 34% let z = x * 2.0 + y ``` -#### string +#### `String` String literals are enclosed in double quotes (`"`), using backslash to escape special characters (`"`, `\` and `$`) @@ -147,7 +185,7 @@ let b = 3 let s = "abc ${2*a + b}" # = "abc 7" ``` -#### regex +#### `Regex` Regular expression literals are enclosed in slash characters (`/`): ``` @@ -163,28 +201,32 @@ let re1 = /./ let re2 = /$str$re1/ # match '.' followed by any character ``` -#### boolean +#### `Bool` Result of comparison operators `==` and `/=`. +Values are `True` and `False`. -#### network +#### `Network` Represents network/subnet, created by `subnet` command and used by `subnet`, `node`, `spawn` and network configuration commands. -#### node +#### `Node` Represents network node, created by `node` command or implicitly by `spawn`, and used by `spawn` or network configuration commands. Members: +`ifname` +: Name of the primary network interface of the node. + `ip` -: String representation of node's IP address. +: String representation of the node primary IP address. `network` -: The network which the node belogs to. +: The network which the node belongs to. -#### process +#### `Process` Represents running process. Created by `spawn`, used by `send` and `expect` commands. @@ -193,18 +235,44 @@ Members: `node` : Node on which the process is running. +`pid` +: PID of the corresponding system process, `0` if there is none. + +#### `Asset` + +Represents an asset (file or directory), which can be used during test execution. + +Members: + +`path` +: Path to the asset valid during the test execution. + +#### `Tag` + +Tag, which can be assigned to a test using the `tag: <Tag>` declaration. + +#### `Signal` + +Type representing unix signals sent to processes. Values are `SIGINT`, `SIGTERM`, etc. + #### list -Lists are written using bracket notation: +Lists are written using bracket notation, and brackets are also used to express the type: ``` -let numbers = [1, 2, 4] +let numbers = [1, 2, 4] : [Integer] ``` List elements can be of any type, but all elements of a particular list must have the same type. +They can be concatenated using the `concat` function, which takes a list of lists as argument: +``` +let list = concat [[1], [2, 3], [4]] # = [1, 2, 3, 4] +``` +Or with the `++` operator: +``` +let list = [1] ++ [2, 3] ++ [4] # = [1, 2, 3, 4] +``` -Used in the `for` command. - -### Build-in commands +### Built-in commands ``` subnet <name> [of <network>] @@ -219,13 +287,15 @@ node <name> [on <network>] Create a node on network `<network>` (or context network if omitted) and assign the new node to the variable `<name>`. ``` -spawn as <name> [on (<node> | <network>)] +spawn as <name> [on (<node> | <network>)] [args <arguments>] [killwith <signal>] ``` Spawn a new test process on `<node>` or `<network>` (or one from context) and assign the new process to variable `<name>`. When spawning on network, create a new node for this process. +Extra `<arguments>` to the tool can be given as a list of strings using the `args` keyword. The process is terminated when the variable `<name>` goes out of scope (at the end of the block in which it was created) by closing its stdin. +If the `killwith` clause is present, it is also sent the given `<signal>` at that point. When the process fails to terminate successfully within a timeout, the test fails. ``` @@ -234,7 +304,7 @@ send <string> to <process> Send line with `<string>` to the standard input of `<process>`. ``` -expect <regex> from <process> [capture <var1> [, <var2> ... ]] +expect <regex> from <process> [timeout <timeout>] [capture <var1> [, <var2> ... ]] ``` Check whether `<process>` produces line matching `<regex>` on standard output, and if this does not happen within current timeout, the test fails. Output lines produced before starting this command and not matched by some previous `expect` are accepted as well. @@ -247,11 +317,24 @@ The regular expression can contain capture groups – parts enclosed in parenthe In that case the expect command has to have the `capture` clause with matching number of variable names. Results of the captures are then assigned to the newly created variables as strings. +If the `timeout` clause is used, the current timeout value is multiplied by the given `<timeout>` for this `expect` call. +Timeout of zero can be used to expect a matching output line to have been already produced in the past. + ``` -flush [from <proc>] +flush [from <proc>] [matching <regex>] ``` Flush memory of `<proc>` output, so no following `expect` command will match anything produced up to this point. +If the `matching` clause is used, discard only output lines matching `<regex>`. + +``` +ignore [from <proc>] [matching <regex>] +``` + +Ignore output lines from `<proc>` (or context process) that match the given +`<regex>` (or all lines if the `matching` clause is not used). Affects both +past and future output of the process; the effect lasts until the end of +the block. ``` guard <expr> @@ -309,12 +392,224 @@ with <expr>: Execute `<test block>` with `<expr>` as context. ``` +multiply_timeout by <multiplier> +``` + +Modify the timeout used for commands like `expect` by multiplying it with `<multiplier>`. +The effect lasts until the end of the block. + +``` wait ``` Wait for user input before continuing. Useful mostly for debugging or test development. +### Shell interpreter + +**Experimental feature**: Functionality is not fully implemented and behavior may change in incompatible ways between releases. + +Using the `shell` expression, it's possible to embed a shell script inside a test script. +The shell script is not passed to an external interpreter, but rather executed by the tester itself, +which allows the use of variables from the rest of the test script: + +``` +test: + node some_node + let x = "abc" + shell as sh on some_node: + echo $x > some_file + echo ${some_node.ip} >> some_file + cat some_file | sed 's/a/A/' > other_file +``` + +The syntax is intended to be generally similar to the classic Bourne shell, +however, only limited functionality is implemented so far (that includes executing commands, pipelines or input/output redirection). + +The general form of the `shell` expression is: + +``` +shell [as <name>] on <node>: + <shell commands> +``` + +Where `<node>` is the network node on which to run the script (it will be run in the network namespace of the node, and with working directory set to the node root), +and `<name>`, if given, is the name of the variable that will refer to the shell process (this can be used e.g. in the `expect` command to check the standard output of the script). +As with the `spawn` command, the resulting process is terminated at the end of the current scope. + +By default the shell process exists with failure whenever any command exits with non-zero status. +This behavior can be disabled using the `set +e` command (and re-enabled with `set -e`). + + +### Functions + +When calling a function, parameters are usually passed using argument keywords +(in the case of built-in commands, those keywords are typically prepositions +like `on`, `from`, etc.), and apart from those, there can be at most one +parameter passed without a keyword. This is done in order to avoid the need to +remember parameter order and to make the behavior of each call as clear as +possible, even without looking up the documentation. + +To make the syntax unambiguous, the keywordless parameter can be passed as +a literal (number, string, etc.), or using parentheses. So this is ok: + +``` +expect /something/ from p +``` + +but if the regular expression is stored in a variable, the parameter needs to +be enclosed in parentheses: +``` +expect (re) from p +``` +or in a literal: +``` +expect /$re/ from p +``` + +### Defining functions + +Custom functions can be defined on the top level using `def` keyword, and with +the parameters either followed by `=` sign to return a value: +``` +def quadruple of x = 4 * x +``` + +or followed by `:` to define test block: +``` +def say_hello to p: + send "hello" to p + expect /hi/ from p +``` + +Those then can be invoked elsewhere: +``` +test: + spawn as p + say_hello to p +``` + +When defining a function, the unnamed parameter, if any, must be enclosed in +parentheses: +``` +def twice (x) = 2 * x +``` + +Type of a given parameter can be also given explicitly using the `:` notation: +``` +def say (what : String) to (p : Process): + send what to p +``` + +### Modules, exports and imports + +Each test script file constitutes a module. As such, it can export definitions +for other modules to use, and import definitions from other modules. The name +of each module must match the filename with the file extension removed, and is +given using the `module` declaration. This declaration, if present, must be +given at the beginning of the file, before other declarations. + +For example a file `test/foo.et` can start with: +``` +module foo +``` +This name is also implicitly assigned when the `module` declaration is omitted. + +In case of a more complex hierarchy, individual parts are separated with `.` +and must match names of parent directories. E.g. a file `test/bar/baz.et` +can start with: + +``` +module bar.baz +``` + +Such declared hierarchy is then used to determine the root of the project in +order to find imported modules. + +To export a definition from module, use `export` keyword before `def`: +``` +export def say_hello to p: + send "hello" to p + expect /hi/ from p +``` +or list the exported name in a standalone export declaration: +``` +export say_hello + +... + +def say_hello to p: + send "hello" to p + expect /hi/ from p +``` + +To import module, use `import <name>` statement, which makes all the exported +definitions from the module `<name>` available in the local scope. +``` +module bar.baz + +import foo +``` + +### Assets + +To provide the used test tool with access to auxiliary files needed for the +test execution, asset objects can be defined. The definition is done on the +toplevel using the `asset` keyword, giving the asset object name and a path to +the asset on the filesystem, relative to the directory containing the test +script: + +``` +asset my_asset: + path: ../path/to/file +``` + +Such defined asset object can then be used in expressions within tests or function definitions: + +``` +test: + spawn as p + send to p "use-asset ${my_asset.path}" +``` + +The `my_asset.path` expression expands to a string containing path to the asset +that can be used by the spawned process `p`. The process should not try to +modify the file. + +Assets can be exported for use in other modules using the `export` keyword, +just like other definitions: + +``` +export asset my_asset: + path: ../path/to/file +``` + +### Tags + +Tags are a way to refer to a group of tests, instead of needing to list all their names individually; +for example to mark broken tests, which can then be easily excluded from running until fixed. +Tags are declared using the `tag` keyword on the top level of a module, +and need to be `export`ed if they are to be referenced from outside of that module: + +``` +export tag Broken +``` + +Tags can be assigned to tests in a the test preamble before the first test steps +using `tag: <Tag>` declaration, which can also be given multiple times: + +``` +test SomeBrokenTest: + tag: Broken + tag: OtherTag + spawn as p + ... +``` + +Such tags can then be used instead of test names to select or exclude tests on +command line or in the configuration file. + + Optional dependencies --------------------- |