Detachable Execution

A command launched through a Target's channel can run detached from that channel. A detached command keeps running if the connection that started it drops, and its output and exit status stay observable afterwards. This keeps long-running remote work independent of the network link and provides the basis for reconnecting to a job after the orchestrator restarts.

Detachment is a property of the target layer. Executors drive a target's channel through run_command and consume the returned ChannelProcess, regardless of whether a command runs detached.

run_command as a template method

run_command is a concrete method on BaseTarget. It either runs the command synchronously over a live channel or launches it detached and returns a handle that polls for status:

async def run_command(
    self, cmd, *, cwd=None, env=None, detach: bool | None = None
) -> ChannelProcess:
    if detach is None:
        detach = self.detach_by_default
    if not detach:
        return await self._run_command_sync(cmd, cwd=cwd, env=env)
    job_dir = new_job_dir(cwd or self.working_directory)
    handle = await self._launch(cmd, cwd=cwd, env=env, job_dir=job_dir)
    return _PollingChannelProcess(self, handle)

Both paths share the same signature and return type, so callers are unaffected by the choice. A target supports detachment by implementing a small set of primitives.

note

The primitives carry default implementations that raise NotImplementedError rather than being abstract, so a target that provides its own run_command works without implementing them.

Target primitives

A target that supports detachment implements:

async def _run_command_sync(self, cmd, *, cwd, env) -> ChannelProcess
async def _launch(self, cmd, *, cwd, env, job_dir) -> JobHandle
async def _poll(self, handle) -> int | None          # None while running
async def _read_output(self, handle) -> tuple[bytes, bytes]
async def _send_signal(self, handle, sig) -> None

Primitive	Responsibility
_run_command_sync	Synchronous, channel-bound execution (the detach=False path).
_launch	Start the command so it outlives the launching channel, and return a JobHandle.
_poll	Non-blocking status: None while running, the exit code once finished.
_read_output	The job's captured (stdout, stderr), read independently of the channel.
_send_signal	Signal delivery without a live channel.

JobHandle and marker files

_launch records what is needed to observe and control the job later. The build_detach_command helper wraps the command so it runs as its own background job:

mkdir -p <job_dir> || exit 1;
setsid sh -c '( <cmd> ); echo $? > <job_dir>/exit_code' \
    > <job_dir>/stdout.log 2> <job_dir>/stderr.log < /dev/null &
echo $! > <job_dir>/pid

• setsid places the job in its own session and process group, so its PID equals its PGID. This releases the controlling terminal, keeping the job unaffected by SIGHUP when the channel closes, and it lets a signal reach the whole process tree through kill -- -<pid>, so cancelling a job also stops the children it spawned. On platforms without setsid, such as macOS (used by LocalTarget), the launcher uses nohup and the target resolves the process group with os.getpgid at signal time. The session_leader argument to build_detach_command selects between the two.
• The command runs in a subshell so a top-level exit within it still reaches the exit-code write.
• The exit_code file is the authoritative completion signal and stays readable after the launching channel is gone. A kill -0 <pid> liveness check serves only as a fast path, so a reused PID cannot be mistaken for a still-running job.
• The launcher returns once the job is backgrounded. The returned JobHandle carries the pid and job_dir.

The polling handle

_PollingChannelProcess is the ChannelProcess returned for a detached job. It satisfies the full channel-process contract (wait, communicate, stream, kill, signal, returncode) by calling the target's primitives on demand instead of holding a channel open:

• wait() polls _poll until it returns an exit code, sleeping poll_interval between checks.
• communicate() waits for completion, then reads output with _read_output.
• stream() re-reads the captured logs on each interval and yields new lines. Lines appear with up to poll_interval of latency.
• kill() and signal() call _send_signal, which targets the job's process group so a signal reaches the command and everything it spawned. Delivery may require a round trip, so callers confirm the outcome by continuing to poll.

Because every method probes the target on demand, a dropped connection pauses polling and the next reconnection resumes it, with no effect on the running job.

Per-target default

Each target sets whether run_command detaches when the caller does not specify:

class BaseTarget(...):
    detach_by_default: ClassVar[bool] = True
    poll_interval: ClassVar[float] = 1.0

Target	detach_by_default	Rationale
SSHTarget	True	A remote job runs beyond the lifetime of any single SSH channel.
LocalTarget	False	Execution shares the orchestrator's machine, so the live-streaming subprocess path applies. Detachment stays available on request.

poll_interval sets how often a detached job is polled. Targets that probe over the network keep it modest; local targets can lower it for tighter streaming latency.

Control-plane commands

Short commands that should block and stream over the live channel pass detach=False:

• SSHTarget.mkdir runs mkdir -p with detach=False.
• DockerExecutor runs docker build and docker rmi with detach=False. docker run uses the target's default, so a container on an SSH target detaches with it.

Executors

Executors call run_command and consume the returned ChannelProcess uniformly, so detachment applies to ShellExecutor, PythonEnvironmentExecutor, and DockerExecutor without executor-specific code. PythonFunctionExecutor and PythonExecExecutor invoke Python in the orchestrator process and do not use run_command, so they run in-process with no detached job.

Adding detachment to a target

Implement the five primitives. The shared run_command template, _PollingChannelProcess, and recover provide the rest. A Slurm target, for example, maps _launch to sbatch, _poll to sacct or squeue, _read_output to the Slurm log files, and _send_signal to scancel, and gains detachment without any polling or state logic of its own.

Resume

The same primitives support reconnecting to a job after the orchestrator process restarts. A persisted run-state file records the JobHandle, and recover(task) rebuilds the polling handle from it so a fresh process reattaches to a running job and reads its result.