Thread Pool for Host Work​

jai::thread_pool (<jaiscript/thread_pool.hpp>) is a persistent pool of worker threads for the host's own work: asset decoding, pathfinding batches, blocking I/O. It is a plain object with no global state; a host may own several. Each engine runs its parallel_for and parallel_transform regions on a pool of its own and never touches a pool the host makes.

• Two ways in: the shared queue (submit, submit_batch), which any free worker serves, and one worker's own queue (submit_to).
• A continuation always runs, exactly once, after its work returns or throws, on the finishing worker or on the host's thread.
• The posted queue brings results home. post queues from any thread; run_posted() runs the queue on the caller, typically once a frame.
• Exceptions never kill a worker. They go to the on_exception handler, or are dropped.
• The pool synchronizes its own queues and nothing else.

jai::thread_pool pool(4);
const size_t workers = pool.worker_count();
std::vector<int64_t> partial(workers);
for (size_t w = 0; w < workers; ++w) {
    pool.submit_to(w, [&partial, w, workers] {
        for (int64_t i = int64_t(w); i < 1000; i += int64_t(workers)) { partial[w] += i; }
    });
}
pool.wait_idle();
int64_t sum = 0;
for (int64_t p : partial) { sum += p; }
std::cout << sum << "\n";

499500

The API​

A task is a thread_pool::task_type, a std::function<void()>.

Every call, with its defaults and failures: jai::thread_pool calls.

An engine's own pool is sized from thread_pool::performance_core_count() and built with thread_pool::fast_threads, which opts its workers out of the system's power throttling (Parallel worker count).

Exceptions​

The handler receives every exception a pool task or a continuation lets escape, and the std::exceptions of posted tasks, on the thread the task ran on. With no handler, or a handler that throws, the exception is dropped. A task that throws still continues, so a continuation can always close what its task opened:

jai::thread_pool pool(1);
pool.on_exception([](std::exception_ptr error) {
    try {
        std::rethrow_exception(error);
    } catch (const std::exception& e) {
        std::cout << "on_exception: " << e.what() << "\n";
    }
});
pool.submit([] { throw std::runtime_error("decode failed"); },
            [] { std::cout << "then runs anyway\n"; });
pool.wait_idle();

on_exception: decode failed
then runs anyway

The posted queue​

Work that must happen on the host's own thread (uploading a decoded texture, touching the scene) is posted. The loading shape: decode on the workers, continue posted, and let the frame loop pick the result up. ready needs no atomic, since only the draining thread touches it.

jai::thread_pool pool;
std::vector<std::string> levels(3);
std::vector<jai::thread_pool::task_type> loads;
for (size_t i = 0; i < levels.size(); ++i) {
    loads.push_back([&levels, i] { levels[i] = "level_" + std::to_string(i); });
}
bool ready = false;
pool.submit_batch(std::move(loads), [&] { ready = true; }, jai::thread_pool::then_on::posted);

while (!ready) {
    // one frame: input, update, render, then the work the pool handed back
    pool.run_posted();
}
std::cout << levels[0] << " " << levels[1] << " " << levels[2] << "\n";

level_0 level_1 level_2

• A task posted while the drain runs runs in the same drain. A posted task may call run_posted(); the nested drain continues the same queue and runs nothing twice.
• A std::exception from a posted task goes to on_exception and the drain goes on. Any other exception leaves run_posted(), with the tasks not yet run queued ahead of later posts.
• Posting never contends with the workers. Drain from one thread.

Spinning​

A worker out of work spins for the budget before it parks on a condition variable, so a dense run of small batches (fork-join work several times a frame) skips the kernel wake. It spins only when the pool's last two batches arrived within the budget, so a pool fed once a frame or less parks at once; wait_idle() spins on the same budget. Set 0 for a pool whose work arrives irregularly:

jai::thread_pool loader(2);
loader.spin_budget_ns(0);   // loads arrive a few times a second: never spin for the next one

Destruction​

Whatever a task references must outlive the pool: declare the pool after that state. The destructor drops every queued task that has not started (their continuations never run), destroys posted tasks unrun, lets running tasks finish, and joins every worker before it returns. Nothing a still-running task submits or posts during destruction runs.

Thread safety​

• submit, submit_to, submit_batch, post, on_exception, spin_budget_ns and wait_idle may be called from any thread, concurrently, including from inside a task.
• A task must not call wait_idle() on its own pool or destroy it.
• Data two tasks share needs its own synchronization. A submit_batch continuation sees every write its batch made.
• A task may call into an engine only while no other thread uses that engine: engines take no locks. Script work goes parallel through the language's own loops (Parallel Loops).