Foundry: Testing a Host

Foundry is the test framework JaiScript's own suite is written in, and it works the same for a project built on JaiScript. It is one header, <jaiscript/testing/foundry.hpp>, and one runner source file, source/tests/main_test_runner.cpp, which holds main. It needs nothing beyond the library. A test is a C++ lambda, so it covers plain host code as readily as script run through an engine.

The header is part of the library's include/ folder. The runner is a source file in the repository, compiled into your test program; a project that uses an installed package copies that one file beside its tests.

A suite

inventory_tests.cpp

#include <jaiscript/testing/foundry.hpp>
#include <jaiscript/core/engine.hpp>

using namespace jai::foundry;

int stack_limit(int item_weight) {            // the host code under test
    return item_weight > 10 ? 1 : 20;
}

class inventory_tests : public suite {
public:
    inventory_tests() : suite("Inventory") {}

    void forge_tests() override {
        test("heavy items do not stack", []() {
            check_eq(1, stack_limit(50));         // check_eq(expected, actual)
            check_eq(20, stack_limit(2));
        });
        test("a script totals the bag", []() {
            auto eng = make_engine();
            auto total = eng->execute("int total = 0; for (int w : [3, 4, 5]) { total += w; } total");
            check_eq(int64_t(12), total.as_int());
        });
        test("a script error reaches the host", []() {
            auto eng = make_engine();
            check_throws([&]() { eng->execute("throw \"bag is full\";"); });
        });
    }
};
FOUNDRY_REGISTER(inventory_tests)

A suite derives jai::foundry::suite, passes its name to the base, and adds its tests in forge_tests() with test(name, function).
A failed check throws, which ends that test and records the message. Anything else a test throws fails it the same way (a std::exception with its what(), any other type as an unknown exception), and the remaining tests still run.
FOUNDRY_REGISTER hands the suite to the runner; no list of suites is kept by hand. A class inside a namespace registers under its qualified name: FOUNDRY_REGISTER(my_game::inventory_tests).
make_engine() returns a jai::engine for a test: the runner's backend choice, a 120-second execution budget so a runaway script fails instead of hanging the run, and imports that are rechecked on every use. Register your own bindings on it as your host does.

Build and run

A test program is an executable made of your test files and the runner source, linked to the library like any other host (Building JaiScript into a Host):

CMakeLists.txt

add_subdirectory(JaiScript)
add_executable(my_game_tests
    inventory_tests.cpp
    JaiScript/source/tests/main_test_runner.cpp)
target_link_libraries(my_game_tests PRIVATE JaiScript::jaiscript)

shell

cmake --build build --target my_game_tests
build/my_game_tests

output

/*----------------------------*\
| JaiScript Foundry Test Suite |
\*----------------------------*/
Backend: vm
Performing Tests, One Moment ...
-----------------------------------------
<3 :D All tests passed! (1 suites, 3 tests, 0.003s)
-----------------------------------------

The exit code is 0 when every test passed and 1 when one failed or a filter matched nothing, so the program is a build or CI step as it stands. A failure names its suite, its test and what the check saw. With a fourth test, light items stack to 99, holding check_eq(99, stack_limit(2)):

output

Failed tests:
  x Inventory :: light items stack to 99
      Expected: 99, Actual: 20

-----------------------------------------
x 1 of 4 test(s) failed (0.002s)
-----------------------------------------

What the tests print to std::cout and std::cerr is hidden unless the run is --verbose.

Checks

Check	Passes when
`check(condition)`, `check_true(condition)`, `check_false(condition)`	the condition is true (false for `check_false`)
`check_eq(expected, actual)`, `check_ne(a, b)`	`expected == actual`; `!(a == b)`. A failed `check_eq` prints both values.
`check_gt(value, bound)`, `check_ge`, `check_lt`, `check_le`	`value > bound`, `>=`, `<`, `<=`
`check_near(expected, actual, tolerance)`	the two differ by no more than the tolerance
`check_null(pointer)`, `check_not_null(pointer)`	the pointer compares equal (unequal) to `nullptr`
`check_throws(function)`	the function throws anything
`check_throws<E>(function)`	the function throws an `E`; another type fails

Every check but check_near takes a message as an optional last argument, shown when it fails.

The runner's arguments

Argument	Effect
`--verbose`, `-v`	prints each suite and test as it runs, with its time and its own output; the way to find a test that hangs
`Inventory`	a bare pattern runs every suite whose name contains it, plus any test elsewhere whose name contains it
`"Inventory.heavy"`	the dotted form runs the matching tests of the matching suites only
`--filter=PATTERN`	the same patterns as a named argument (`--gtest_filter=` is accepted too); `*` matches any run of characters
`--backend=vm`, `--backend=interpreter`	the backend of the engines `make_engine()` returns. A host has the VM alone, and it is the default there; the interpreter needs `jaiscript_interpreter`.
`--shard=I/N`	runs every N-th suite starting at suite I (from 0), so N processes cover the suites once between them
`--progress`	a line per suite and a dot per test, for a watchdog that needs to see a long run move

A crash prints the suite and test that were running to stderr. On Windows, setting the environment variable JAI_CRASH_DUMP_DIR to a folder also writes a minidump there.

Setup, teardown and benchmarks

A suite may override pre_test() and post_test(); they run around every test, and post_test() runs after a failed one too.
benchmark(name, function, iterations), called in forge_tests() beside test, times the function over that many calls (1000 unless given) and prints the microseconds per call. Benchmarks run only under --verbose with no filter, and never fail a run.

Other build systems

Compile source/tests/main_test_runner.cpp and your test files into one executable with the settings the library was built with, and link the library. On Windows the runner also uses dbghelp, which MSVC picks up from the source and a GNU linker needs named.
Put test files in the executable itself. A suite registers through a static object, and a linker drops an unused object file out of a static library, taking its suites with it.
A test program has one main, the runner's. Code the tests need from your application comes in as a library or as source files, without the application's own main.

Foundry: Testing a Host​

A suite​

Build and run​

Checks​

The runner's arguments​

Setup, teardown and benchmarks​

Other build systems​