How It Works
This describes Ratect’s internal pipeline, for anyone extending Ratect or trying to
understand its behavior in detail. For the code itself, see AGENTS.md
for a map of the source layout.
1. CLI parsing
Each binary has its own main.rs, and both parse arguments with
clap: ratect-compat/src/main.rs for Batect’s flat flag
interface (CLI reference), and ratect/src/main.rs for the
subcommand interface (ratect CLI reference). Everything below
this step is shared — both call into ratect-core, which is where the rest of
this document lives. Parsing has to happen before config evaluation (step 2)
can run, since --config-var/--config-vars-file feed into it.
2. Config loading and evaluation (ratect-core/src/config.rs)
This is two separate steps, not one: loading checks what the file says, and evaluation reads what the machine running it supplies — see decisions/0014.
-
load_project(orload_project_native, forratect.toml) starts withConfig::load_from_file_with_git_cache(orload_from_file_native_written_for): the root file is parsed intoConfig/Container/Task/TaskRun/ConfigVariable/ServiceGroupstructs — YAML vianoyalib, TOML viatoml, one struct set for both formats — and its top-levelincludelist (if any) is resolved, with every loaded file’scontainers/tasks/config_variables, and a TOML file’sservices, merged into oneConfig(see how included files combine).Includes are walked breadth-first, so every entry in the root file is reached before any included file’s own, and each file is loaded exactly once however many entries name it. A
type: gitentry clones its repository into~/.ratect/inclfirst (ratect-core/src/git_include.rs), and everything reached through one is confined to that clone and may do only what the entry granted it — see What a bundle may do for the rules, andCONTEXT.mdfor what bundle, grant and boundary each denote.It then checks the result — references, field combinations, and the shape of every value it can judge without the machine — applies
extends(native only), and returns aLoadedProject. No expression is evaluated: a field that may hold one is judged here only when it holds a literal, so a container reading a variable this machine doesn’t have stops nothing that doesn’t use it. TheLoadedProjectkeeps what evaluation needs: the directory each container was declared in, the clone a Git-included container must keep its host paths within, and which container in anextendschain supplied each path-bearing field. Listing tasks needs nothing more. -
LoadedProject::evaluate: given the containers and tasks a caller needs, and the--config-var/--config-vars-fileoverrides it parsed:-
Evaluates expressions (
$VAR,${VAR:-default},<name,<{name}, plus the built-inbatect.project_directory) in every field that takes one:environmentvalues (container, taskrun, and taskcustomise.<container>),localvolume mount host paths (in a Batect-format file, only the object form’s),build_directory,build_args, abuild_secretsentry’spath, abuild_sshentry’spaths, and —ratect.tomlonly —imageandidentity.home. Acachemount’sname/containerare plain strings, matching Batect: nothing to evaluate (see Cache volumes). A value that can’t be evaluated is an error naming the container or task, the field and the expression. -
Host path resolution: after evaluating a host path, if the result is relative, it’s resolved to an absolute path relative to the directory of the file that declared the value — the container’s own, or for an inherited field the ancestor’s that supplied it; the root config’s directory when there’s no
includeinvolved — not the current working directory. Done in this order because an expression can itself resolve to an absolute path, which mustn’t be treated as a relative fragment.batect.project_directoryitself always resolves to the root config’s directory regardless of which file a container came from. Acachemount’s Docker volume name/host directory is resolved later instead (ratect-core/src/cache.rs, viaengine.rs’sresolve_volumes), once--cache-typeand the project’s own cache key are known — neither available at this stage.The resolved path is then checked against the container’s Git boundary, if it has one: a container that came from a Git-included file may only reach inside its own clone or your project directory, unless that include was granted
allow_host_paths. Checked twice: once lexically, with both the check and the path normalized first, sincePath::starts_withdoes not interpret..; then against the real locations, with symlinks resolved as far as the path exists, since a bundle can commit one inside its own clone. A literal path gets the same check at load.
The engine evaluates each task as it reaches it — see step 3 of the task engine below.
See the configuration reference for the full expression syntax, precedence, and error rules.
-
3. Task engine (ratect-core/src/engine.rs)
TaskEngine::run_task(name) first walks the task’s whole prerequisite order —
wildcards expanded, each task once — without running anything, so a prerequisite
naming no task, or a cycle among them, fails the run before any task starts, as in
Batect. Then it runs a recursive async function. The order of its five steps is the
part worth knowing here; for what each one does in detail, read
engine.rs’s own module comment (cargo doc --open -p ratect-core), which is
where that lives and stays current.
- Already executed? If this task has already run successfully in this invocation, return immediately. This is what makes shared prerequisites run only once.
- Cycle detection: a task already in the middle of being run — an ancestor of itself in the current call stack — errors immediately rather than recursing forever. A backstop: the walk above has already refused every cycle.
- Run prerequisites, each through the same recursive function, before the
task’s own container step. They run with
top_level: false, unlike the task actually named on the command line, and with no-- ARGS. The flag keeps--skip-prerequisitesto the named task, and leaves a prerequisite’s resources out of--no-cleanup-after-success. It plays no part in interactive eligibility: every task’s own container is eligible, however deeply nested, as in Batect — see Interactive Mode. The task is then evaluated, before any Docker work of its own: every container in its graph, itsrun.environmentand itscustomiseenvironments go throughLoadedProject::evaluate, and the rest of the task reads only those values. A container no task in the run uses is never evaluated. Its container graph is built then too, and a cycle in it fails the task before it starts — no network is created, as in Batect. A task with norunof its own stops here and succeeds — it exists purely to chain prerequisites (see config reference), matching Batect’s ownTaskRunner. - Create the task’s network, then pull or build every image and create every
container in the task’s container graph on it — the task’s own included — as
soon as each one’s image is ready, and start each dependency once every
container it depends on is ready, before the task’s own container, so it can
reach them by name. The
graph is the container’s own
dependenciesunioned with the task’s, resolved recursively, with anycustomiseoverrides applied to whichever container they target at whatever depth. Every task execution gets its own network — a task’s container is never left on Docker’s shared default bridge — and it is torn down afterwards. With--use-networkan existing network is validated and reused instead, and never removed, since Ratect didn’t create it. See the task lifecycle for the step-by-step and diagrams. - Resolve and run the image.
resolve_imageturns a container’simageorbuild_directoryinto something runnable — pulling (perimage_pull_policy) or building, or erroring if neither is set — and is used identically for the task’s own container and for dependencies. The container then runs with the task’scommand, created on the task’s network (unless its ownnetwork_modesays otherwise), its environment layered as Environment precedence states. Everything else on the container — ports, hostnames, working directory, entrypoint, capabilities, devices, and the rest — is assembled bycontainer_spec.rsfrom the config and handed todocker.rsas plain values; the config reference is the list of what those fields mean, and which accept a task-levelrunoverride.
The “run once”, “pull once” and “build once” guarantees are in-memory and scoped to
a single invocation — nothing persists between runs, so a build_directory
container is rebuilt every time. Pulls and builds are memoized as once-cells holding
their outcome rather than plain sets, so two containers resolving the same image under the same
pull policy share one in-flight operation instead of racing. Dependency readiness, by contrast, is scoped
to a single task execution and discarded when it finishes.
Concurrency follows Batect’s in outline: prerequisites run sequentially, one to
completion after another, even when independent — while one task’s startup is
concurrent. Every container’s image — each dependency’s and the task’s own —
is pulled or built at once, the moment the task’s network is ready, and each
container is created as soon as its own image is (and, under
network_mode = "container:<name>", its target has been created), so a pull, build or create
that fails ends the task as soon as it fails — not once every dependency is
ready, by which time their setup commands have run. Starting is what a
container’s own dependencies gate: independent branches of the graph start
and health-wait at the same time, and a container — already created — starts
once every container it depends on has passed its readiness gate (see decision
0019).
Cleanup is concurrent the same way, in reverse: a container is stopped once every
container depending on it has been, and every container ready to stop is stopped at
once, so independent dependencies wait out their grace periods together.
The task’s network is created before any of that
starts, where Batect creates it alongside — see Differences from
Batect. Running independent prerequisites concurrently too is
a possible Rust-specific enhancement beyond Batect — tracked as
ratect#102 — and
Dependency Readiness has the detail.
Testability
The engine talks to Docker through a ContainerRuntime trait (defined in
ratect-core/src/docker.rs) rather than depending on the concrete Docker client
directly. This is what lets the engine’s prerequisite/cycle/dedup logic be
unit-tested with a fake implementation instead of a real Docker daemon.
4. Docker integration (ratect-core/src/docker.rs)
DockerClient wraps bollard and implements
ContainerRuntime. What each method is for:
pull_imageandbuild_image: fetch or build the image a container needs, streaming the daemon’s progress to the output layer as events — what, if anything, those render as is the selected output style’s decision, notdocker.rs’s (see Logging vs. output).build_imagefirst packs the build directory into an in-memory tar, honouring.dockerignorevia thedockerignorecrate.create_container: creates a container — a dependency’s or the task’s own — ready to be started: Docker’s create, which attaches it to its network, then the user-mapping uploads. It hands the container’s id back the moment that is done (inside its error, if an upload failed after Docker’s create), and the engine registers it for cleanup there and then, so a container whose start later fails or is cut short by another dependency failing is still removed. For a container withidentity, the engine then edits the created container’s identity files, prepares its home and, on a rootful Linux daemon, takes its root-owned caches, and a failure there counts as a failed create. Its own step, separate from starting, because it waits for less: a container is created once its image and the network are ready (and anynetwork_mode = "container:<name>"target has been created), and started only once the containers it depends on are — see Dependency readiness.run_container: starts and streams the task’s own container until it exits. Three attach-and-start paths sit behind it — fully non-interactive, stdin-forwarding, and a real TTY with raw mode and live resize — chosen by whether the task is interactive-eligible and whether Ratect’s own stdin and stdout are terminals. All three attach to the container before Docker’s start and read its output from that attach, as Batect does, so nothing typed or printed early is lost and any log driver works. It does not remove the container: the engine’s cleanup stage removes everything a task created, its own container included, so--no-cleanup-*is interpreted in exactly one place.create_network/remove_network/network_exists: the per-task network, or a service instance’s (decisions/0026), plus the up-front validation that makes--use-networkfail with a clear error rather than an unrelated API failure later.start_background_container/stop_and_remove_container: the same, for a dependency, sidecar or service group’s container — started and left running alongside the task rather than waited on, so no taskcommandapplies; its output is shown only under-o all, read from an attach made before it starts.follow_container: the output of a service group’s container that anotherratect services upstarted, attached to after the fact, under-o all.wait_for_container_healthy/exec_in_container: the two steps of the dependency readiness gate. The first blocks on Docker’s own event stream, replayed from the beginning so a verdict that arrived before the stream opened still counts, and turns an unhealthy verdict into an error carrying the last health check’s exit code and output, when the container and daemon are still there to report them. The second runs onesetup_commandsentry in the running container and returns its exit code and output for the engine to judge.
Nothing here depends on config types: container_spec.rs converts config into a
plain ContainerSpec first, which is what the methods that create and start a container
take; the rest take its id and plain values.
The module’s own comment carries the gotchas — where each path calls Docker’s
start relative to attaching, why cleanup ownership is not to be split again, how
build_ssh paths are classified — and is the thing to read before changing any of
this.
Container creation/start/removal events are logged at debug level via tracing (see
below), as is each setup_commands exec’s raw output — which is why
Troubleshooting warns about pasting a debug
log anywhere.
5. Logging vs. output
Ratect keeps two channels deliberately separate — and output itself follows one rule: stdout is the result, stderr is everything said about producing it (decisions/0017).
- Output is what a user with
RUST_LOGunset would be worse off without, and doesn’t go throughtracing, apart from two terminal-resize failures that are logged instead. The result — a run’s task-container output, atasks listlisting — is stdout. Everything about the run is stderr: errors (Error:), warnings (Warning:), notices (the interrupt notice, what--no-cleanup*left in place), in every style includingquiet; and progress (“Running build…”, “Pulling alpine:3.18…”, “build finished with exit code 0 in 2.3s.”), the one class--output/-oshapes. Progress is the one place the binaries differ:ratectwrites it to stderr,ratect-compatto stdout, where Batect puts it — see Streams. Internally these lines are typed events (ratect-core/src/ui/):engine.rsanddocker.rspost task-execution milestones, and the three stderr classes, to an event sink instead of printing, and the selected output style decides what each renders as —fancy’s live per-container status block on an interactive terminal,simple’s append-only lines otherwise, no progress at all underquiet, orall’s per-container prefixed lines (the one style where even container output routes through the event sink, line-buffered, instead of streaming to stdout directly — the task container’s own lines still landing on stdout). Eachmainnames which stream its progress console is; nothing inside the loggers chooses. - Diagnostics are breadcrumbs — Docker API calls, what resolved to what,
a best-effort retry that then succeeded — via
tracing/tracing-subscriberon stderr, filtered byRUST_LOG(defaults toinfo;ratect-compat’s--log-filedefaults todebugfor Ratect’s own targets). A fatal error is never only one of these:main.rsprints it directly (Error: <message>), so it stays visible underRUST_LOG=offand-o quietalike — except for a run’s failure the run has already reported by another route (the task’s own non-zero exit, a signal or a closed stdout, or a failure to run the task once it had started or to clean up after it, which the engine reported as it happened), which gets no second line, and for a verb that printed its ownError:line for each thing it couldn’t do (ratect includes clean/refresh). A failedratect services upis a run, and reports as one.ratect-compatalso records every one atdebug, so--log-filesays why the run failed unlessRUST_LOGfilters that record out.
Colours (e.g. the exit code in the task summary line, the Error: marker) are,
by default, only emitted when the stream being written to is actually a
terminal — each stream decides for itself, tracing’s lines included — see
Colour for the controls that change that.
Filtering RUST_LOG
Scoping RUST_LOG to Ratect’s own targets (ratect_core, ratect_core::docker,
and ratect-compat’s ratect_compat) rather than every dependency’s is on
Troubleshooting, which owns it.