Quick-Test is a small testing framework for Red. It is called quick-test as it was put together quickly to support the development of the "boot-strapped" version of Red/System. Hopefully, it also runs tests quite quickly and, perhaps more importantly, is a quick way to write meaningful tests.
As a result, quick-test can be a little fussy about where files are and it is not particularly fault tolerant.
There are actually two aspects to Quick-Test. The first is quick-test.red and quick-test.reds, a handful of Red and Red/System functions respectively which can be included in a Red or Red/System program to write unit tests. These can easily be used on their own to test Red and Red/System code and are not at all fussy about where files reside.
A temporary addition has been made to this part of Quick-Test. It is quick-unit-test.r which is written in REBOL and used for unit testing the bootstrapping Red compiler.
The second aspect is that Quick-Test is a test suite for Red and Red/System themselves and includes the files used to test Red and Red/System.
There are many improvements that could be made to quick-test. It is expected that quick-test will be replaced at the time when Red and Red/System is re-written so they probably won't get made.
2. Types of Tests
Quick-Test supports four types of tests - tests of Red or Red/System code, tests of Red and Red/System programs, tests of the Red/System compiler behaviour and tests of the Red compiler behaviour. Temporarily, Quick-Test also supports REBOL unit tests for the Red compiler.
The first type of tests are written in Red or Red/System, the other four in REBOL.
3. Quick-Test Components
Quick-Test consists of the following:
quick-test.r - a REBOL script that supports testing compiler and executable output of both Red and Red/System programs.
quick-test.reds - a set of Red/System functions that supports writing tests in Red/System.
quick-test.red - a set of Red functions that supports writing tests in Red.
quick-unit-test.r - a set of REBOL functions that supports writing tests in REBOL.
(The above are stored in the Red/quick-test/ directory.)
run-all.r - a script(s) which runs a set of Red and/or Red/System tests.
- run-all.r in Red/system/tests/ runs the Red/System test suite
- run-all.r in Red/tests/ runs the Red test suite
- run-all.r in Red/ runs all the tests
run-test.r - a script which will run an individual test.
- stored in Red/quick-test/
4. Directory Structure
Quick-Test expects the following Red directory structure:
Red/
run-all.r ;; runs combined Red & Red/System tests
quick-test/ ;; quick-test components
quick-test.r
quick-test.reds
quick-unit-test.r
run-test.r
runnable/ ;; the test executables
;; automatically created by Quick-Test
;; listed in .gitignore
;; this directory can be emptied without ;; ;; impacting quick-test
tests/
run-all.r ;; runs all Red tests
source/
compiler/ ;; red compiler unit tests
runtime/ ;; runtime tests
system/
tests/ ;; the main test directory
run-all.r ;; runs all Red/System tests
source/ ; all test sources
compiler/ ;; tests of the compiler
run-time/ ;; tests of the run time library
units/ ;; base language tests (eg datatype tests)
5. Running Tests
The tests are designed to be run from the command line or a REBOL console session. This must be a REBOL/View console session under Windows.
The following examples assume that they have been run from the Red dir
To run all Red & Red/System tests from the console:
rebol -s run-all.r
To run all Red & Red/System tests from the REBOL console:
do %run-all.r
To run all Red tests from the console:
rebol -s tests/run-all.r
To run all Red tests from the REBOL console:
do %tests/run-all.r
To run all Red/System tests from the console:
rebol -s system/tests/run-all.r
To run all Red/System tests from the REBOL console:
do %system/tests/run-all.r
To run an individual test file from the console:
rebol -s quick-test/run-test.r "<file>"
To run an individual test file from the REBOL console:
do/args %quick-test/run-test.r "<file>"
where <file> is the path to the file to be tested .
e.g. The command to run the logic-test.reds test file which is in the units directory:
(Note: %run-test.r can run both .r and .reds tests).
(Note: When using run-test.r from within a REBOL console session, the file paths to both run-test.r and the test file msut be relative to the directory from which the REBOL session was launched (system/options/path).
5 Running Tests with the "Binary" Compiler
By default, quick-tests runs all the tests using the source code of the compiler. However, it is possible to run the tests using the "binary" compiler.
To run the tests using the "binary" compiler from the command line:
rebol -qs run-all.r --binary <path-to-compiler>
To run the tests from the REBOL console:
do/args %run-all.r "--binary <path-to-compiler>
You can omit the if it is Red/build/bin/red or Red/build/bin/red.exe
5 Running Tests in Batch Mode
The test can be run in batch mode from within a shell script (or similar). In batch mode, the test output is suppressed and the script returns 0 if all tests succeed and 1 if any test fails.
rebol -qs run-all.r --batch
6. Writing Tests
In terms of writing tests, Quick-Test is quite minimal by design. It provides only a simple assert function and a minimal set of test organisation functions.
6.1 Examples
Example 1: A Test of inc.reds - an imaginary function which adds 1 to an integer
Red/System [
Title: "Tests of inc.reds"
File: %source/run-time/inc-test.reds
]
#include %../../quick-test/quick-test.reds
#include %relativepathto/inc.reds
~~~start-file~~~ "inc" ;; start test file
;; initialises totals
===start-group=== "increment an variable" ;; a marker to group tests
;; group name [string!]
--test-- "inc-test-1" ;; start of code for a test
;; test name [string!]
i: 1
--assert 2 = inc i ;; an assertion
;; expression [logic!]
===end-group=== ;; end of group marker
~~~end-file~~~ ;; finish test - print totals
Example 2: A test to check the compiler correctly identifies aan unidentified variable:
change-dir %../ ;; revert to tests/ dir (from runnable)
;; .r test scripts are copied to runnable
;; before execution
;; There is no need to include quick-test.r
;; as it will have been included by either
;; %run-all.r or %run-test.r
~~~start-file~~~ "comp-err" ;; start test
--test-- "compile-error-1"
--compile-this { ;; compiles the suplied string
;; by default automatically inserts
;; Red [] by default
;; this can be changed by setting
;; qt/script-header prior to running
;; the test
;; compiler output is collected in
;; qt/comp-output
i := 1;
}
--assert none <> find qt/comp-output "*** undefined symbol"
--assert none <> find qt/comp-output "at: ["
--assert none <> find qt/comp-output "i := 1"
--assert none <> find qt/comp-output "]"
--clean ;; tidies up temporary files
~~~end-file~~~ ;; ends test and print totals
Example 3: Test the output of a Red or Red/System program
change-dir %../ ;; revert to tests/ directory from runnable/
~~~start-file~~~ "output"
--test-- "hello"
;; --compile-and-run compiles and runs
;; a Red/System program. The output is
;; collected qt/output
--compile-and-run %source/compiler/hello.reds
--assert none <> find qt/output "hello"
--assert none <> find qt/output "world"
~~~end-file~~~
6.2 Test Dialects
6.2.1 quick-test.reds
***start-run*** <title>
<title> : title of the test run - c-string!
Marks the start of the test run and initialises the run totals.
~~~start-file~~~ <title>
<title> : title of the test file - c-string!
Marks the start of an individual test file.
===start-group=== <title>
<title> : title of the test group - c-string!
Marks the start of a group of tests. Using groups in a test is optional.
--test-- <title>
<title> : title of an inidvidual test - c-string!
The start of an individual test. The number of tests in a run and file are counted and reported.
--assert <assertion>
<assertion> : a Red/System expression that returns a logic! value
An assertion. True = pass. False = fail. The number of assertions, passes and failures are counted and reported for each run and file.
--assertf~= <value> <value> <tolerance>
<value> : a float! value
<tolerance> : a float! value
An assertion that two floating point numbers are approximately equal. The tolerance provided is used both as an absolute and relative tolerance. The calculation of the tolerance is a quite simplistic but has proved sufficient for its purpose.
--assertf32~= <value> <value> <tolerance>
<value> : a float32! value
<tolerance> : a float32! value
AssertF~= for float32! values.
===end-group===
Marks the end of a group of tests.
~~~end-file~~~
Marks the end of a test file.
***end-run***
Marks the end of a test run.
6.2.2 quick-test.r
quick-test.r supports tests with two different levels of output. The more verbose of the two displays totals for each file in a run and highlights failing tests.
The second, "quiet mode" displays only a summary of each tests in a concise report. The more detailed report is logged to %quick-test.log in the tests/ directory. This is triggered by using the quiet version of the commands in the dialect.
***start-run*** <title>
<title> : title of the test run - string!
Marks the start of the test run and initialises the run totals.
***start-run-quiet*** <title>
<title> : title of the test run - string!
Marks the start of a "quiet" test run and initialises the run totals.
~~~start-file~~~ <title>
<title> : title of the test file - string!
Marks the start of an individual test file.
===start-group=== <title>
<title> : title of the test group - string!
Marks the start of a group of tests. Using groups in a test is optional.
--test-- <title>
<title> : title of an inidvidual test - string!
The start of an individual test. The number of tests in a run and file are counted and reported.
--compile <file>
<file> : source file - file!
Compiles a Red or Red/System source file. The output from the compiler is captured in qt/comp-ouput. If the compile is successful, qt/exe will contain the path to the executable file. It will be 'none if the compile failed.
--compile-dll <file> <target>
<file> : source file - file!
<target> : The compiler target ID
Compiles a Red or Red/System source file as a dynamic link library. The output from the compiler is captured in qt/comp-output. The dll is stored in the tests/runnable directory. Use the qt/compile-ok? function to check if the compilation was successful.
--compile-red <file> DEPRECATED USE --compile
<file> : Red source file - file!
Compiles a Red source file. The output from the compiler is captured in qt/comp-output. If the compile is successful, qt/exe will contain the path to the executable file. It will be 'none if the compile failed.
Compiles a Red or Red/System source supplied as a string in the test script. A Red header is automatically inserted if one is not supplied. Setting qt/script-header will override the Red [] default. The output from the compiler is captured in qt/comp-ouput. If the compile is successful, qt/exe will contain the path to the executable file. It will be 'none if the compile failed.
--compile-this-red <source code> DEPRECATED USE -- compile-this
<source code> : Red source - string!
Compiles a Red source supplied as a string in the test script. A Red header is automatically inserted if one is not supplied. The output from the compiler is captured in qt/comp-ouput. If the compile is successful, qt/exe will contain the path to the executable file. It will be 'none if the compile failed.
Compiles a Red or Red/System source file and runs it. The output from the compiler is captured in qt/comp-ouput. If the compile is successful, qt/exe will contain the path to the executable file. It will be 'none if the compile failed. The output produced by the executable is captured in qt/output. (It will contain "Compilation failed" in the case of a compilation error.)
--compile-and-run-red <file> DEPRECATED use --compile-and-run
<file> : Red source file - file!
Compiles a Red source file and runs it. The output from the compiler is captured in qt/comp-ouput. If the compile is successful, qt/exe will contain the path to the executable file. It will be 'none if the compile failed. The output produced by the executable is captured in qt/output. (It will contain "Compilation failed" in the case of a compilation error.)
Compiles a Red/System source supplied as a string in the test script and runs it. A Red/System header is automatically inserted if one is not supplied. The output from the compiler is captured in qt/comp-ouput. If the compile is succesful, qt/exe will contain the path to the executable file. It will be 'none if the compile failed. The output produced by the executable is captured in qt/output. (It will contain "Compilation failed" in the case of a compilation error.)
--compile-and-run-this-red <source code> DEPRECATED USE --compile-and-run-this
<file> : Red source - string!
Compiles a Red source supplied as a string in the test script and runs it. A Red/System header is automatically inserted if one is not supplied. The output from the compiler is captured in qt/comp-ouput. If the compile is succesful, qt/exe will contain the path to the executable file. It will be 'none if the compile failed. The output produced by the executable is captured in qt/output. (It will contain "Compilation failed" in the case of a compilation error.)
Compiles, runs and prints the output of a Red or Red/System program supplied as a source file. It is designed to run test programs written using the quick-test.reds test framework.
--add-to-run-totals
Is used after a --compile-and-run or --compile-run-print to collect the totals from a test using the quick-test.reds framework.
--run <file>
<file> : an executable program - file!
Runs an executable program. The output is captured in qt/output.
--run-unit-test <file>
<file> : a REBOL script file - file!
Runs a quick-unit-test.r test written in REBOL.
--run-unit-test-quiet
<file> : a REBOL script file - file!
Runs a quick-unit-test.r test written in REBOL in quiet mode.
--run-script <file>
<file> : a REBOL script - file!
Runs a REBOL script.
--run-script-quiet <file>
<file> : a REBOL script - file!
Runs a REBOL script in quiet mode.
--run-test-file <file>
<file> : a quick-test.red or quick-test.reds test program
Compiles, runs and prints the output of a Red or Red/System test program written using quick-test.reds. It automatically adds the totals from the test to the run-totals.
--run-test-file-quiet <file>
<file> : a quick-test.reds or quick-test.red test program
Compiles, runs and prints the output of a Red or Red/System test program written using quick-test.reds in quiet mode. It automatically adds the totals from the test to the run-totals.
--assert <assertion>
<assertion> : a Red/System expression that returns a logic! value
An assertion. True = pass. False = fail. The number of assertions, passes and failures are counted and reported for each run and file.
--assert-msg? <message>
<message> : an expected compiler messsage - string!
Checks if the compiler has produced the expected message.
Checks if the runtime output included the expected phrase.
--assert-red-printed? <phrase>
<phrase> : expected string in Red runtime output - string!
Checks if the runtime output from a Red program included the expected phrase. Red program output Unicode (either UTF-8 or UTF-16LE). The phrase must be UTF-8 encoded.
--clean
Cleans up the temporary files created by --compile-this and --compile-and-run-this. Can be safely used, but is not necessary, after --compile, --compile-and-run and --run.
===end-group===
Marks the end of a group of tests.
~~~end-file~~~
Marks the end of a test file.
***end-run***
Marks the end of a test run.
***end-run-quiet***
Marks the end of a quiet-mode test run.
--setup-temp-files
Creates a set of temporary files exclusively for a test run to allow multiple tests to be run in parallel. These files should be deleted at the end of the run.
--delete-temp-files
Deletes a set of temporary files created with --setup-temp-files.
--separate-log-file
Creates a separate log file for a test run. This file will be retained after the run for inspection. It needs to be deleted manually.
6.2.3 quick-unit-test.r
Syntatically the quick-unit-test dialect is similar to the quick-test.reds dialect. There is one significant difference between them, quick-unit-test collects all the output from test and provides a means to interogate it.
***start-run*** <title>
<title> : title of the test run - string!
Marks the start of the test run and initialises the run totals.
~~~start-file~~~ <title>
<title> : title of the test file - string!
Marks the start of an individual test file.
===start-group=== <title>
<title> : title of the test group - string!
Marks the start of a group of tests. Using groups in a test is optional.
--test-- <title>
<title> : title of an inidvidual test - string!
The start of an individual test. The number of tests in a run and file are counted and reported.
--assert <assertion>
<assertion> : a Red/System expression that returns a logic! value
An assertion. True = pass. False = fail. The number of assertions, passes and failures are counted and reported for each run and file.
--assert-printed? <value>
<value> : text - string!