Running Tests

TestBench doesn't require the use of any special test runner. It's designed so that tests can be executed using nothing more than Ruby. There's no need to create or maintain plugins for editors or CI servers. It's just Ruby.

Using the Ruby Executable

Run test files like any script file by passing the file name to the ruby command.

> ruby test/automated/example.rb
Example Context
  Some passing test
  Some other passing test
  Some failing test
    Assertion failed

Failure: 1

Batch Runner

Runs a batch of files and directories.

API

TestBench::Run.(path, exclude: nil)

Parameters

Name	Description	Type	Default
path	Path name to run	String or Array	test/automated
exclude	Pattern matching files to exclude	String	`*_init.rb`

The batch runner is run from within a Ruby script file.

Here is an example of the batch runner being invoked from a file named automated.rb located in the test directory.

# test/automated.rb

TestBench::Run.('test/automated')

Batch Runner Output

In addition to the test output printed when running a test using the Ruby executable, the batch runner prints a summary of the results of all the tests.

Running test/automated/example.rb
Example Context
  Some passing test
  Some other passing test
  Some failing test
    Assertion failed

Failure: 1

Failure Summary:
- test/automated/example.rb: 1 failure

Attempted 1 file: 1 completed, 0 aborted, 0 not found
3 tests in 0.01 seconds (298.59 tests/sec)
2 passed, 1 failed, 0 skipped

Excluding Files

The batch runner allows files or directories to be excluded.

The runner's exclude parameter allows a shell glob pattern to be specified that excluded test files from the batch run. If a file name matches the pattern, it will be skipped.

TestBench::Run.('test/automated',
  exclude: '{_*,*_init}.rb'
)

In this example above, files that start with underscore or that end with _init.rb are skipped.

Command Line Runner

In addition to being able to run tests using the raw ruby executable, TestBench also provides it's own command line executable that offers a bit more power.

The bench executable can be used to run individual test files or directories containing test files.

Running a Single File

To run a single test file, specify the file path as a command line argument.

> bench test/automated/some_test.rb

Running a Directory

To run a directory of test files, and its subdirectories, specify the directory path as a command line argument.

> bench test/automated/some_directory/

Default Test Directory

By default, when the bench commend is executed with no arguments, it will run all the test files under test/automated.

This default can be changed by setting the environment variable TEST_BENCH_DEFAULT_TEST_PATH.

Piping Into the CLI

Test files and directories can be piped into the CLI via standard input ("stdin").

> echo "test/automated/some_test.rb" | bench

> echo "test/automated/some_directory" | bench

Command Line Switches

The bench command accepts command-line switches that configure how TestBench operates.

Each of the switches also has a corresponding environment variable which allows for TestBench to be configured for a local development environment.

Executing bench the --help or -h switches will print descriptions of the switches.

> bench --help
Usage: bench [options] [paths]

Informational Options:
  Help:
      -h, --help
          Print this help message and exit immediately

Execution Options:
  Abort On Failure:
      -a, --abort-on-failure
          Stops execution if a test fails or a test file aborts

  Exclude File Patterns:
      -x, --exclude PATTERN
          Exclude test files that match PATTERN
          If multiple --exclude arguments are supplied, then files that match any will be excluded
      -X, --no-exclude
          Don't exclude any files
      Default: '*_init.rb'

  Strict:
      -s, --strict
          Prohibit skipped tests and contexts, and require at least one test to be performed
      -S, --no-strict
          Relax strictness
      Default: non strict, unless TEST_BENCH_STRICT is set to 'on'

  Require Library:
      -r, --require LIBRARY
          Require LIBRARY before running any files
      -I, --include DIR
          Add DIR to the load path

  Random Seed:
      --random-seed SEED
          Pseudorandom number seed

Output Options:
  Backtrace Formatting:
      -b, --omit-backtrace PATTERN
          Omits backtrace frames that match PATTERN
          If multiple --omit-backtrace arguments are supplied, then frames that match any will be omitted

  Detail:
      -d, --detail
          Always show details
      -D, --no-detail
          Never show details
      Default: print details when their surrounding context failed, unless TEST_DETAIL is set to 'on' or 'off'

  Device:
      --device DEVICE
          stderr: redirect output to standard error
          null: don't write any output
      Default: stdout

  Verbosity:
      -l, --output-level LEVEL
          all: print output from every file
          not-passing: print output from files that skip tests and contexts or don't perform any tests
          failure: print output only from files that failed or aborted
          abort: print output only from file that aborted
      -q, --quiet
          Sets output verbosity level to 'not-passing'
      Default: all

  Styling:
      -o, --output-styling
          Enable output text styling
      -O, --no-output-styling
          Disable output text styling
      Default: enabled if the output device is an interactive terminal

  Summary:
      --no-summary
          Don't print summary after running files

Paths to test files (and directories containing test files) can be given after any command line arguments or via STDIN (or both).

If no paths are given, the directory 'test/automated' is scanned for test files.

The following environment variables can also control execution:

  TEST_BENCH_ABORT_ON_FAILURE           See --abort-on-failure
  TEST_BENCH_EXCLUDE_FILE_PATTERN       See --exclude
  TEST_BENCH_OUTPUT_SUMMARY             See --no-summary
  TEST_BENCH_STRICT                     See --strict
  TEST_BENCH_RANDOM_SEED                See --random-seed
  TEST_BENCH_FILTER_BACKTRACE_PATTERN   See --filter-backtrace
  TEST_BENCH_OUTPUT_DETAIL              See --detail
  TEST_BENCH_OUTPUT_DEVICE              See --device
  TEST_BENCH_OUTPUT_LEVEL               See --output-level
  TEST_BENCH_OUTPUT_STYLING             See --output-styling
  TEST_BENCH_DEFAULT_TEST_PATH          Specifies default path
  TEST_BENCH_OPTIONS                    Evaluated as command line arguments similar to RUBYOPT