feat: auto-discover runners (#12)
This commit is contained in:
banteg
+89
-33
@@ -5,13 +5,13 @@ domain model. Use the existing runners (Codex/Claude) as references.
|
||||
|
||||
## Quick checklist
|
||||
|
||||
1. Implement `Runner` in `src/takopi/runners/<engine>.py`.
|
||||
1. Implement `Runner` in `src/takopi/runners/<engine>.py` (usually via
|
||||
`JsonlSubprocessRunner`).
|
||||
2. Emit Takopi events from `takopi.model` and implement resume helpers
|
||||
(`format_resume`, `extract_resume`, `is_resume_line`).
|
||||
3. Register an `EngineBackend` in `src/takopi/engines.py` with setup checks
|
||||
and runner construction.
|
||||
4. Add CLI subcommand in `src/takopi/cli.py`.
|
||||
5. Extend tests (runner contract + engine-specific translation tests).
|
||||
3. Define `BACKEND = EngineBackend(...)` in the runner module (auto-discovered),
|
||||
including `install_cmd` (and `cli_cmd` only if the binary name differs).
|
||||
4. Extend tests (runner contract + engine-specific translation tests).
|
||||
|
||||
---
|
||||
|
||||
@@ -25,20 +25,41 @@ make it easy to drop in another engine without changing the Takopi domain model.
|
||||
- Engine id: `"pi"` (used in config, resume tokens, and CLI subcommand).
|
||||
- Canonical resume line: the engine’s own CLI resume command, e.g.
|
||||
`` `pi --resume <session_id>` ``.
|
||||
- If your engine uses the standard `"<engine> resume <token>"` format, you can
|
||||
reuse `compile_resume_pattern()`. Otherwise, define a custom regex in the
|
||||
runner (like Claude does).
|
||||
- Pick the resume line format you want to support and define a regex for it in
|
||||
the runner (Claude is a good example). If you choose the
|
||||
`"<engine> resume <token>"` shape, you can use that exact regex.
|
||||
|
||||
### 2) Implement `src/takopi/runners/pi.py`
|
||||
|
||||
Skeleton outline:
|
||||
Recommended: `JsonlSubprocessRunner`
|
||||
|
||||
For JSONL CLIs, this base class centralizes subprocess + JSONL plumbing,
|
||||
lock timing, and completion semantics. Your runner usually only needs:
|
||||
|
||||
- `command()` (binary name)
|
||||
- `build_args(...)`
|
||||
- `translate(...)` (map one JSON object to a list of Takopi events)
|
||||
|
||||
Optional hooks for common variants:
|
||||
|
||||
- `stdin_payload(...)`: return `None` if the prompt is passed via argv
|
||||
- `env(...)`: add or redact environment variables
|
||||
- `invalid_json_events(...)`: customize the warning event
|
||||
- `process_error_events(...)`: customize `rc != 0` handling
|
||||
- `stream_end_events(...)`: customize stream-end fallback (no `CompletedEvent`)
|
||||
- `handle_started_event(...)`: customize session-id validation
|
||||
|
||||
If you call `note_event(...)`, your state object must include `note_seq` or
|
||||
override `next_note_id(...)`.
|
||||
|
||||
Skeleton outline (JSONL CLI):
|
||||
|
||||
```py
|
||||
ENGINE: EngineId = "pi"
|
||||
_RESUME_RE = re.compile(r"(?im)^\s*`?pi\s+--resume\s+(?P<token>[^`\\s]+)`?\\s*$")
|
||||
|
||||
@dataclass
|
||||
class PiRunner(SessionLockMixin, ResumeTokenMixin, Runner):
|
||||
class PiRunner(ResumeTokenMixin, JsonlSubprocessRunner):
|
||||
engine: EngineId = ENGINE
|
||||
resume_re: re.Pattern[str] = _RESUME_RE
|
||||
|
||||
@@ -46,30 +67,55 @@ class PiRunner(SessionLockMixin, ResumeTokenMixin, Runner):
|
||||
model: str | None = None
|
||||
allowed_tools: list[str] | None = None
|
||||
|
||||
def _build_args(self, prompt: str, resume: ResumeToken | None) -> list[str]:
|
||||
args = ["--jsonl"]
|
||||
def command(self) -> str:
|
||||
return self.pi_cmd
|
||||
|
||||
def build_args(
|
||||
self, prompt: str, resume: ResumeToken | None, *, state: Any
|
||||
) -> list[str]:
|
||||
_ = prompt, state
|
||||
args = ["--jsonl", "--verbose"]
|
||||
if resume is not None:
|
||||
args.extend(["--resume", resume.value])
|
||||
if self.model is not None:
|
||||
args.extend(["--model", self.model])
|
||||
if self.allowed_tools:
|
||||
args.extend(["--allowed-tools", ",".join(self.allowed_tools)])
|
||||
args.append("--")
|
||||
args.append(prompt)
|
||||
return args
|
||||
|
||||
async def run(
|
||||
self, prompt: str, resume: ResumeToken | None
|
||||
) -> AsyncIterator[TakopiEvent]:
|
||||
async for evt in self._run_with_resume_lock(prompt, resume, self._run):
|
||||
yield evt
|
||||
def stdin_payload(
|
||||
self, prompt: str, resume: ResumeToken | None, *, state: Any
|
||||
) -> bytes | None:
|
||||
_ = resume, state
|
||||
return prompt.encode()
|
||||
|
||||
def translate(
|
||||
self,
|
||||
data: dict[str, Any],
|
||||
*,
|
||||
state: Any,
|
||||
resume: ResumeToken | None,
|
||||
found_session: ResumeToken | None,
|
||||
) -> list[TakopiEvent]:
|
||||
_ = state, resume, found_session
|
||||
...
|
||||
```
|
||||
|
||||
Key implementation notes:
|
||||
|
||||
- Use `SessionLockMixin` to enforce per-session serialization.
|
||||
- Use `ResumeTokenMixin` for `format_resume` / `extract_resume` / `is_resume_line`.
|
||||
- Use `iter_jsonl(...)` + `drain_stderr(...)` from `takopi.utils.streams`.
|
||||
- Use `BaseRunner` (or `JsonlSubprocessRunner`) for per-session serialization.
|
||||
- Mix in `ResumeTokenMixin` (with a `resume_re`) or override
|
||||
`format_resume` / `extract_resume` / `is_resume_line` so the runner owns
|
||||
resume encoding/decoding.
|
||||
- For JSONL CLIs, prefer `JsonlSubprocessRunner` and implement `command`,
|
||||
`build_args`, and `translate` (override `stdin_payload` if the prompt should
|
||||
be passed via argv instead of stdin).
|
||||
- If you don’t use `JsonlSubprocessRunner`, use `iter_jsonl(...)` +
|
||||
`drain_stderr(...)` from `takopi.utils.streams`.
|
||||
- **Minimal mode is supported:** start with exactly one `StartedEvent` and one
|
||||
`CompletedEvent`. `ActionEvent`s are optional and can be added later. If you
|
||||
do emit actions, you can emit only `phase="completed"` notes without tracking
|
||||
pending state.
|
||||
- **Do not truncate** tool outputs in the runner; pass full strings into events.
|
||||
Truncation belongs in renderers.
|
||||
|
||||
@@ -94,13 +140,28 @@ Mapping guidance:
|
||||
If Pi emits warnings/errors before the final event, surface them as completed
|
||||
`ActionEvent`s (e.g., `kind="warning"`).
|
||||
|
||||
### 4) Register engine in `src/takopi/engines.py`
|
||||
### 4) Expose the backend (auto-discovered)
|
||||
|
||||
Add:
|
||||
Takopi discovers runners by importing modules in `takopi.runners` and looking
|
||||
for a module-level `BACKEND: EngineBackend` (from `takopi.backends`).
|
||||
|
||||
- `_pi_check_setup()` that verifies `pi` exists on PATH
|
||||
- `_pi_build_runner()` that reads `[pi]` config and returns `PiRunner`
|
||||
- A new `EngineBackend(id="pi", display_name="Pi", ...)` entry
|
||||
At the bottom of `src/takopi/runners/pi.py`, define:
|
||||
|
||||
```py
|
||||
BACKEND = EngineBackend(
|
||||
id="pi",
|
||||
build_runner=build_runner,
|
||||
install_cmd="npm install -g @acme/pi-cli",
|
||||
)
|
||||
```
|
||||
|
||||
No changes to `engines.py` or `cli.py` are required.
|
||||
|
||||
Only modules that define `BACKEND` are treated as engines. Internal/testing
|
||||
modules (like `mock.py`) should omit it.
|
||||
|
||||
If the CLI binary name differs from the engine id, set `cli_cmd="pi-cli"` on
|
||||
the backend.
|
||||
|
||||
Example config (minimal):
|
||||
|
||||
@@ -110,12 +171,7 @@ model = "pi-large"
|
||||
allowed_tools = ["Bash", "Read"]
|
||||
```
|
||||
|
||||
### 5) Add CLI subcommand
|
||||
|
||||
Expose `takopi pi` alongside `takopi codex` / `takopi claude` by adding a new
|
||||
`@app.command()` in `src/takopi/cli.py`.
|
||||
|
||||
### 6) Tests + fixtures
|
||||
### 5) Tests + fixtures
|
||||
|
||||
- Add `tests/test_pi_runner.py` for translation behavior.
|
||||
- Reuse `tests/test_runner_contract.py` to ensure lock/resume invariants.
|
||||
|
||||
+7
-2
@@ -106,9 +106,14 @@ Transforms takopi events into human-readable text:
|
||||
| `model.py` | Domain types: resume tokens, actions, events, run result |
|
||||
| `runner.py` | Runner protocol + event queue utilities |
|
||||
|
||||
### `engines.py` - Engine backend registry
|
||||
### `backends.py` - Engine backend contracts
|
||||
|
||||
Registers available engines and provides setup checks + runner construction.
|
||||
Defines `EngineBackend`, `SetupIssue`, and the `EngineConfig` type used by
|
||||
runner modules.
|
||||
|
||||
### `engines.py` - Engine backend discovery
|
||||
|
||||
Auto-discovers runner modules in `takopi.runners` that export `BACKEND`.
|
||||
|
||||
### `runners/` - Runner implementations
|
||||
|
||||
|
||||
@@ -94,15 +94,19 @@ Notes:
|
||||
|
||||
## Code changes (by file)
|
||||
|
||||
### 1) `src/takopi/engines.py`
|
||||
### 1) New file: `src/takopi/runners/claude.py`
|
||||
|
||||
Add a new backend:
|
||||
#### Backend export
|
||||
|
||||
* Engine ID: `EngineId("claude")`
|
||||
Expose a module-level `BACKEND = EngineBackend(...)` (from `takopi.backends`).
|
||||
Takopi auto-discovers runners by importing `takopi.runners.*` and looking for
|
||||
`BACKEND`.
|
||||
|
||||
* `check_setup()` should:
|
||||
`BACKEND` should provide:
|
||||
|
||||
* `shutil.which("claude")` must exist.
|
||||
* Engine id: `"claude"`
|
||||
* `install_cmd`:
|
||||
* Install command for `claude` (used by onboarding when missing on PATH).
|
||||
* Error message should include official install options and “run `claude` once to authenticate”.
|
||||
|
||||
* Install methods include install scripts, Homebrew, and npm. ([Claude Code][4])
|
||||
@@ -110,11 +114,7 @@ Add a new backend:
|
||||
|
||||
* `build_runner()` should parse `[claude]` config and instantiate `ClaudeRunner`.
|
||||
|
||||
* `startup_message()` e.g.:
|
||||
|
||||
* `takopi (claude) is ready\npwd: ...`
|
||||
|
||||
### 2) New file: `src/takopi/runners/claude.py`
|
||||
#### Runner implementation
|
||||
|
||||
Implement a new `Runner`:
|
||||
|
||||
@@ -319,7 +319,7 @@ Mirror the existing `CodexRunner` tests patterns.
|
||||
|
||||
1. **Contract & locking**
|
||||
|
||||
* `test_run_serializes_same_session` (stub `_run` like Codex tests)
|
||||
* `test_run_serializes_same_session` (stub `run_impl` like Codex tests)
|
||||
* `test_run_allows_parallel_new_sessions`
|
||||
* `test_run_serializes_new_session_after_session_is_known`:
|
||||
|
||||
@@ -367,7 +367,7 @@ Mirror the existing `CodexRunner` tests patterns.
|
||||
|
||||
## Implementation checklist
|
||||
|
||||
* [ ] Add `ClaudeBackend` in `src/takopi/engines.py` and register in `ENGINES`.
|
||||
* [ ] Export `BACKEND = EngineBackend(...)` from `src/takopi/runners/claude.py`.
|
||||
* [ ] Add `src/takopi/runners/claude.py` implementing the `Runner` protocol.
|
||||
* [ ] Add tests + stub executable fixtures.
|
||||
* [ ] Update README and developing docs.
|
||||
|
||||
@@ -44,8 +44,8 @@ Notes:
|
||||
`claude --resume <session_id>`
|
||||
```
|
||||
|
||||
Runner must implement its own regex (cannot use `compile_resume_pattern` because
|
||||
that only matches `<engine> resume <token>`). Suggested regex:
|
||||
Runner must implement its own regex because the resume format is
|
||||
`claude --resume <session_id>`. Suggested regex:
|
||||
|
||||
```
|
||||
(?im)^\s*`?claude\s+(?:--resume|-r)\s+(?P<token>[^`\s]+)`?\s*$
|
||||
@@ -202,11 +202,9 @@ Add a Claude runner without changing the Takopi domain model:
|
||||
|
||||
1. Create `takopi/runners/claude.py` implementing `Runner` and (custom)
|
||||
resume parsing.
|
||||
2. Update `takopi/engines.py`:
|
||||
- add `claude` backend id
|
||||
- `check_setup`: locate `claude` binary (PATH + common locations)
|
||||
2. Define `BACKEND` in `takopi/runners/claude.py`:
|
||||
- `install_cmd`: install command for the `claude` binary
|
||||
- `build_runner`: read `[claude]` config + construct runner
|
||||
- `startup_message`: `"claude is ready\npwd: <cwd>"`
|
||||
3. Add new docs (this file + `claude-stream-json-cheatsheet.md`).
|
||||
4. Add fixtures in `tests/fixtures/` (see below).
|
||||
5. Add unit tests mirroring `tests/test_codex_*` but for Claude translation
|
||||
|
||||
+19
-1
@@ -169,6 +169,17 @@ Takopi MUST support the following event types:
|
||||
2. `action`
|
||||
3. `completed`
|
||||
|
||||
**Minimal runner mode (supported):**
|
||||
|
||||
Runners MAY emit only:
|
||||
|
||||
- exactly one `started`
|
||||
- exactly one `completed`
|
||||
|
||||
`action` events are optional. If emitted, a runner MAY emit only
|
||||
`phase="completed"` action events (no requirement to emit `started` / `updated`
|
||||
phases or track pending action state).
|
||||
|
||||
### 5.3 Required fields by event type
|
||||
|
||||
#### 5.3.1 `started`
|
||||
@@ -199,6 +210,10 @@ Optional:
|
||||
- `message: str` (freeform status/warning text)
|
||||
- `level: "debug" | "info" | "warning" | "error"`
|
||||
|
||||
Notes:
|
||||
|
||||
- `phase="completed"` alone is valid; `started` / `updated` are optional.
|
||||
|
||||
#### 5.3.3 `completed`
|
||||
|
||||
Required:
|
||||
@@ -424,6 +439,9 @@ The progress renderer SHOULD maintain:
|
||||
- completed actions and status
|
||||
- resume token if known
|
||||
|
||||
The progress renderer MUST tolerate “completed-only” actions (no prior
|
||||
`started` / `updated`) and treat them as standalone steps.
|
||||
|
||||
If the runner emits multiple `action` events for the same `Action.id` while it is still running (e.g., repeated `phase="started"` or `phase="updated"`), the progress renderer SHOULD treat these as updates and collapse them into a single line (replacing the prior running line rather than appending a new one).
|
||||
|
||||
### 8.3 Final rendering
|
||||
@@ -462,7 +480,7 @@ The architecture SHOULD keep this future change localized to a `RunnerRegistry`
|
||||
|
||||
1. **Runner contract tests**
|
||||
- Emits exactly one `started`
|
||||
- All actions have required fields and stable IDs
|
||||
- All actions (if any) have required fields and stable IDs
|
||||
- `completed.resume` matches started token (when present)
|
||||
- Event ordering is preserved
|
||||
- `ok` semantics match intended behavior
|
||||
|
||||
Reference in New Issue
Block a user