feat: plugins and public api (#71)

This commit is contained in:
banteg
2026-01-09 03:23:57 +04:00
committed by GitHub
parent 16c0069aa0
commit f856338b94
32 changed files with 3135 additions and 622 deletions
+23 -2
View File
@@ -5,10 +5,15 @@ This guide explains how to add a **new engine runner** to Takopi.
A *runner* is the adapter between an engine-specific CLI (Codex, Claude Code, …) and Takopis
**normalized event model** (`StartedEvent`, `ActionEvent`, `CompletedEvent`).
If you are building an external plugin package, read `docs/plugins.md` first.
Takopi is designed so that adding a runner usually means **adding one new module** under
`src/takopi/runners/` plus a small **msgspec schema** module under `src/takopi/schemas/`
no changes to the bridge, renderer, or CLI.
When writing code intended for plugins, prefer importing from `takopi.api`
instead of internal modules.
The walkthrough below uses an **imaginary engine** named **Acme** (`acme`) and intentionally mirrors
the patterns used in `runners/claude.py`.
@@ -74,6 +79,12 @@ Choose a stable engine id string. This string becomes:
- The CLI subcommand (`takopi acme`)
- The `ResumeToken.engine`
Engine ids must match the plugin ID regex:
```
^[a-z0-9_]{1,32}$
```
For Acme well use:
- Engine id: `"acme"`
@@ -114,8 +125,18 @@ src/takopi/runners/
acme.py # ← new
```
Takopi discovers engines by importing modules in `takopi.runners` and looking for a
module-level `BACKEND: EngineBackend` (see `takopi.engines`).
Takopi discovers engines via **entrypoints**. Every engine backend must be exposed
as an entrypoint under `takopi.engine_backends`, and the entrypoint name must match
the backend id.
For in-repo engines, add an entrypoint in `pyproject.toml`:
```toml
[project.entry-points."takopi.engine_backends"]
acme = "takopi.runners.acme:BACKEND"
```
For external plugins, use your packages `pyproject.toml` with the same group.
---
+63 -10
View File
@@ -9,10 +9,19 @@ flowchart TB
cli_desc["Entry point, config loading, lock file"]
end
subgraph Plugins["Plugin Layer"]
entrypoints[plugins.py<br/>entrypoint discovery]
engines[engines.py]
transports[transports.py]
commands[commands.py]
api[api.py<br/>public plugin API]
end
subgraph Orchestration["Orchestration Layer"]
router[AutoRouter<br/>router.py]
scheduler[ThreadScheduler<br/>scheduler.py]
projects[ProjectsConfig<br/>config.py]
runtime[TransportRuntime<br/>transport_runtime.py]
end
subgraph Bridge["Bridge Layer"]
@@ -42,8 +51,18 @@ flowchart TB
cli --> router
cli --> scheduler
cli --> projects
cli --> engines
cli --> transports
cli --> commands
engines --> entrypoints
transports --> entrypoints
commands --> entrypoints
router --> runtime
projects --> runtime
router --> tg_bridge
scheduler --> tg_bridge
runtime --> tg_bridge
tg_bridge --> commands
tg_bridge --> runner_bridge
runner_bridge --> runner_proto
runner_proto --> runners
@@ -59,6 +78,21 @@ flowchart TB
---
## Plugin Architecture
Takopi discovers plugins via Python entrypoints and keeps loading lazy:
- **Engine backends** (`takopi.engine_backends`)
- **Transport backends** (`takopi.transport_backends`)
- **Command backends** (`takopi.command_backends`)
Entrypoint names become plugin IDs, are validated up front (reserved names, regex),
and are only loaded when needed. The public surface for plugin authors lives in
`takopi.api`, while transports and commands interact with core routing via
`TransportRuntime`.
---
## Domain Model
```mermaid
@@ -120,19 +154,27 @@ sequenceDiagram
participant RunnerBridge as runner_bridge.py
participant Runner
participant AgentCLI as Agent CLI
participant Command as Command Plugin
User->>Telegram: Send message
Telegram->>Bridge: poll_incoming()
Bridge->>Bridge: Parse directives<br/>(/engine, /project, @branch)
Bridge->>Bridge: Extract resume token<br/>from reply
Bridge->>Bridge: Resolve worktree<br/>(if @branch)
Bridge->>Bridge: Parse slash command
alt Command plugin
Bridge->>Command: handle(ctx)
Command->>RunnerBridge: run_one/run_many (optional)
RunnerBridge->>Telegram: Send progress/final
else Default routing
Bridge->>Bridge: Parse directives<br/>(/engine, /project, @branch)
Bridge->>Bridge: Extract resume token<br/>from reply
Bridge->>Bridge: Resolve worktree<br/>(if @branch)
Bridge->>Scheduler: enqueue(ThreadJob)
Scheduler->>RunnerBridge: handle_message()
Bridge->>Scheduler: enqueue(ThreadJob)
Scheduler->>RunnerBridge: handle_message()
RunnerBridge->>Telegram: Send progress message
RunnerBridge->>Runner: run(prompt, resume)
RunnerBridge->>Telegram: Send progress message
RunnerBridge->>Runner: run(prompt, resume)
end
Runner->>AgentCLI: Spawn subprocess
@@ -217,8 +259,14 @@ sequenceDiagram
flowchart TD
cli[cli.py] --> config[config.py]
cli --> engines[engines.py]
cli --> transports[transports.py]
cli --> commands[commands.py]
cli --> lockfile[lockfile.py]
engines --> plugins[plugins.py]
transports --> plugins
commands --> plugins
engines --> backends[backends.py]
backends --> runners[runners/]
@@ -244,7 +292,10 @@ flowchart TD
pi --> pi_s
cli --> router[router.py]
router --> tg_bridge[telegram/bridge.py]
tg_bridge --> runtime[transport_runtime.py]
runtime --> router
runtime --> config
tg_bridge --> commands
runner --> runner_bridge[runner_bridge.py]
runner_bridge --> tg_bridge
@@ -274,12 +325,13 @@ flowchart LR
subgraph toml_contents["takopi.toml"]
direction TB
global["transport<br/>default_engine"]
global["transport<br/>default_engine<br/>default_project"]
telegram_cfg["[transports.telegram]<br/>bot_token = ...<br/>chat_id = ..."]
plugins_cfg["[plugins]<br/>enabled = [\"...\"]"]
plugins_extra["[plugins.mycommand]<br/>setting = ..."]
claude_cfg["[claude]<br/>model = ..."]
codex_cfg["[codex]<br/>model = ..."]
projects_cfg["[projects.alias]<br/>path = ...<br/>worktrees_dir = ...<br/>default_engine = ..."]
default_proj["[projects]<br/>default = ..."]
end
toml --> toml_contents
@@ -335,6 +387,7 @@ flowchart TD
| Layer | Components | Responsibility |
|-------|------------|----------------|
| **CLI** | `cli.py` | Entry point, config, lock |
| **Plugins** | `plugins.py`, `engines.py`, `transports.py`, `commands.py`, `api.py` | Entrypoint discovery, plugin loading, public API boundary |
| **Orchestration** | `router.py`, `scheduler.py`, `config.py` | Engine selection, job queuing, project config |
| **Bridge** | `telegram/bridge.py`, `runner_bridge.py` | Message handling, execution coordination |
| **Runner** | `runner.py`, `runners/*.py`, `schemas/*.py` | Agent CLI subprocess, JSONL parsing, event translation |
+31 -3
View File
@@ -77,9 +77,14 @@ Defines `Transport`, `MessageRef`, `RenderedMessage`, and `SendOptions`.
Defines a renderer that converts `ProgressState` into `RenderedMessage` outputs.
### `transports.py` - Transport registry
### `transport_runtime.py` - Transport runtime facade
Defines the transport backend protocol, registry helpers, and built-in transport registration.
Provides the `TransportRuntime` helper used by transport backends to resolve
messages, select runners, and format context without depending on internal types.
### `transports.py` - Transport backend loading
Defines the transport backend protocol and entrypoint-backed loading helpers.
### `config_migrations.py` - Config migrations
@@ -165,9 +170,32 @@ See `docs/transports/telegram.md` for outbox behavior, rate limiting, and retry
Defines `EngineBackend`, `SetupIssue`, and the `EngineConfig` type used by
runner modules.
### `plugins.py` - Entrypoint discovery
Centralizes plugin discovery and lazy loading:
- lists IDs without importing plugin modules
- loads a specific entrypoint on demand
- captures load errors for diagnostics
- filters by enabled list (distribution names)
### `commands.py` - Command backend loading
Defines the command backend protocol, command context/executor helpers, and
entrypoint-backed loading for slash-command plugins.
### `ids.py` - Plugin ID validation
Defines the shared ID regex used for plugin IDs and Telegram command names.
### `api.py` - Public plugin API
Re-exports the supported plugin surface from `takopi.api` (stable API boundary).
### `engines.py` - Engine backend discovery
Auto-discovers runner modules in `takopi.runners` that export `BACKEND`.
Loads engine backends via entrypoints (`takopi.engine_backends`), with lazy loading
and enabled list support.
### `runners/` - Runner implementations
+307
View File
@@ -0,0 +1,307 @@
# Plugins
Takopi supports **entrypoint-based plugins** for:
- **Engine backends** (new runner implementations)
- **Transport backends** (new chat/command transports)
- **Command backends** (custom `/command` handlers)
Plugins are **discovered lazily**: Takopi lists IDs without importing plugin code,
and loads a plugin only when it is needed (or when you explicitly request it).
This keeps `takopi --help` fast and prevents broken plugins from bricking the CLI.
See `public-api.md` for the stable API surface you should depend on.
---
## Entrypoint groups
Takopi uses two Python entrypoint groups:
```toml
[project.entry-points."takopi.engine_backends"]
myengine = "myengine.backend:BACKEND"
[project.entry-points."takopi.transport_backends"]
mytransport = "mytransport.backend:BACKEND"
[project.entry-points."takopi.command_backends"]
mycommand = "mycommand.backend:BACKEND"
```
**Rules:**
- The entrypoint **name** is the plugin ID.
- The entrypoint value must resolve to a **backend object**:
- Engine backend -> `EngineBackend`
- Transport backend -> `TransportBackend`
- The backend object **must** have `id == entrypoint name`.
Takopi validates this at load time and will report errors via `takopi plugins --load`.
---
## ID rules
Plugin IDs are used in the CLI and (for engines/projects) in Telegram commands.
They must match:
```
^[a-z0-9_]{1,32}$
```
If an ID does not match, it is skipped and reported as an error.
**Reserved IDs (engines):**
- `cancel` (core chat command)
- `init`, `plugins` (CLI commands)
Engines using these IDs are skipped and reported as errors.
**Reserved IDs (commands):**
- `cancel`, `init`, `plugins`
- Any engine id or project alias (checked at runtime)
Command backends using reserved IDs are skipped and reported as errors.
---
## Enabling plugins
Takopi supports a simple enabled list to control which plugins are visible.
```toml
[plugins]
enabled = ["takopi-transport-slack", "takopi-engine-acme"]
auto_install = false
```
- `enabled = []` (default) -> load all installed plugins.
- If `enabled` is non-empty, **only distributions with matching names** are visible.
- Distribution names are taken from package metadata (case-insensitive).
- If a plugin has no resolvable distribution name and an enabled list is set, it is hidden.
- `auto_install` is **reserved** and not implemented yet.
This enabled list affects:
- Engine subcommands registered in the CLI
- `takopi plugins` output
- Runtime resolution of engines/transports/commands
---
## Discovering plugins
Use the CLI to inspect plugins:
```sh
takopi plugins
takopi plugins --load
```
Behavior:
- `takopi plugins` lists discovered entrypoints **without loading them**.
- `--load` loads each plugin to validate type and surface import errors.
- Errors are shown at the end, grouped by engine/transport and distribution.
- If `[plugins] enabled` is set, entries are still listed but marked `enabled`/`disabled`.
---
## Engine backend plugins
Engine plugins implement a runner for a new engine CLI and expose
an `EngineBackend` object.
Minimal example:
```py
# myengine/backend.py
from __future__ import annotations
from pathlib import Path
from takopi.api import EngineBackend, EngineConfig, Runner
def build_runner(config: EngineConfig, config_path: Path) -> Runner:
_ = config_path
# Parse config if needed; raise ConfigError for invalid config.
return MyEngineRunner(config)
BACKEND = EngineBackend(
id="myengine",
build_runner=build_runner,
cli_cmd="myengine",
install_cmd="pip install myengine",
)
```
`EngineConfig` is the raw config table (dict) from `takopi.toml`:
```toml
[myengine]
model = "..."
```
Read it with `settings.engine_config("myengine", config_path=...)` in Takopi,
or just consume the dict directly in your runner builder.
See `public-api.md` for the runner contract and helper classes like
`JsonlSubprocessRunner` and `EventFactory`.
---
## Transport backend plugins
Transport plugins connect Takopi to new messaging systems (Slack, Discord, etc).
You must provide a `TransportBackend` object with:
- `id` and `description`
- `check_setup()` -> returns `SetupResult` (issues + config path)
- `interactive_setup()` -> optional interactive setup flow
- `lock_token()` -> token fingerprinting for config locks
- `build_and_run()` -> build transport and start the main loop
Minimal skeleton:
```py
# mytransport/backend.py
from __future__ import annotations
from pathlib import Path
from takopi.api import (
EngineBackend,
SetupResult,
TransportBackend,
TransportRuntime,
)
class MyTransportBackend:
id = "mytransport"
description = "MyTransport bot"
def check_setup(
self, engine_backend: EngineBackend, *, transport_override: str | None = None
) -> SetupResult:
_ = engine_backend, transport_override
return SetupResult(issues=[], config_path=Path("takopi.toml"))
def interactive_setup(self, *, force: bool) -> bool:
_ = force
return True
def lock_token(
self, *, transport_config: dict[str, object], config_path: Path
) -> str | None:
_ = transport_config, config_path
return None
def build_and_run(
self,
*,
transport_config: dict[str, object],
config_path: Path,
runtime: TransportRuntime,
final_notify: bool,
default_engine_override: str | None,
) -> None:
_ = (
transport_config,
config_path,
runtime,
final_notify,
default_engine_override,
)
raise NotImplementedError
BACKEND = MyTransportBackend()
```
For most transports, you will want to call `handle_message()` from `takopi.api`
inside your message loop. That function implements progress updates, resume handling,
and cancellation semantics.
---
## Command backend plugins
Command plugins add custom `/command` handlers. A command only runs when the
message starts with `/command` and does **not** collide with engine ids,
project aliases, or reserved command names.
Minimal example:
```py
# mycommand/backend.py
from __future__ import annotations
from takopi.api import CommandContext, CommandResult, RunRequest
class MultiCommand:
id = "multi"
description = "run the prompt on every engine"
async def handle(self, ctx: CommandContext) -> CommandResult | None:
prompt = ctx.args_text.strip()
if not prompt:
return CommandResult(text="usage: /multi <prompt>")
requests = [
RunRequest(prompt=prompt, engine=engine)
for engine in ctx.runtime.available_engine_ids()
]
results = await ctx.executor.run_many(
requests,
mode="capture",
parallel=True,
)
blocks = []
for result in results:
text = result.message.text if result.message else "no output"
blocks.append(f"## {result.engine}\n{text}")
return CommandResult(text="\n\n".join(blocks))
BACKEND = MultiCommand()
```
### Command plugin configuration
Configure command plugins under `[plugins.<id>]`:
```toml
[plugins.multi]
engines = ["codex", "claude"]
```
The parsed dict is available as `ctx.plugin_config` inside `handle()`.
---
## Versioning & compatibility
Takopi exposes a **stable plugin API** via `takopi.api`.
- `TAKOPI_PLUGIN_API_VERSION = 1` is the current API version.
- Depend on a compatible Takopi version range, for example:
```toml
dependencies = ["takopi>=0.11,<0.12"]
```
When the plugin API changes, Takopi will bump the API version and document
any compatibility guidance.
---
## Troubleshooting
Common issues:
- **Plugin missing from CLI**: check the enabled list in `[plugins] enabled`.
- **Plugin not listed**: verify entrypoint group and ID regex.
- **Load failures**: run `takopi plugins --load` and inspect errors.
- **ID mismatch**: ensure `BACKEND.id == entrypoint name`.
+252
View File
@@ -0,0 +1,252 @@
# Public Plugin API
Takopi's **public plugin API** is exported from:
```
takopi.api
```
Anything not imported from `takopi.api` should be considered **internal** and
subject to change. The API version is tracked by `TAKOPI_PLUGIN_API_VERSION`.
---
## Versioning
- Current API version: `TAKOPI_PLUGIN_API_VERSION = 1`
- Plugins should pin to a compatible Takopi range, e.g.:
```toml
dependencies = ["takopi>=0.11,<0.12"]
```
---
## Exported symbols
### Engine backends and runners
| Symbol | Purpose |
|--------|---------|
| `EngineBackend` | Declares an engine backend (id + runner builder) |
| `EngineConfig` | Dict-based engine config table |
| `Runner` | Runner protocol |
| `BaseRunner` | Helper base class with resume locking |
| `JsonlSubprocessRunner` | Helper for JSONL-streaming CLIs |
| `EventFactory` | Helper for building takopi events |
### Transport backends
| Symbol | Purpose |
|--------|---------|
| `TransportBackend` | Transport backend protocol |
| `SetupIssue` | Setup issue for onboarding / validation |
| `SetupResult` | Setup issues + config path |
| `Transport` | Transport protocol (send/edit/delete) |
| `Presenter` | Renders progress to `RenderedMessage` |
| `RenderedMessage` | Rendered text + transport metadata |
| `SendOptions` | Reply/notify/replace flags |
| `MessageRef` | Transport-specific message reference |
| `TransportRuntime` | Transport runtime facade (routers/projects hidden) |
| `ResolvedMessage` | Parsed prompt + resume/context resolution |
| `ResolvedRunner` | Runner selection result |
### Command backends
| Symbol | Purpose |
|--------|---------|
| `CommandBackend` | Slash command plugin protocol |
| `CommandContext` | Context passed to a command handler |
| `CommandExecutor` | Helper to send messages or run engines |
| `CommandResult` | Simple response payload for a command |
| `RunRequest` | Engine run request used by commands |
| `RunResult` | Engine run result (captured output) |
| `RunMode` | `"emit"` (send) or `"capture"` (collect) |
### Core types and helpers
| Symbol | Purpose |
|--------|---------|
| `EngineId` | Engine id type alias |
| `ResumeToken` | Resume token (engine + value) |
| `StartedEvent` / `ActionEvent` / `CompletedEvent` | Core event types |
| `Action` | Action metadata for `ActionEvent` |
| `RunContext` | Project/branch context |
| `ConfigError` | Configuration error type |
| `DirectiveError` | Error raised when parsing directives |
| `RunnerUnavailableError` | Router error when a runner is unavailable |
### Bridge helpers (for transport plugins)
| Symbol | Purpose |
|--------|---------|
| `ExecBridgeConfig` | Transport + presenter config |
| `IncomingMessage` | Normalized incoming message |
| `RunningTask` / `RunningTasks` | Per-message run coordination |
| `handle_message()` | Core message handler used by transports |
---
## Runner contract (engine plugins)
Runners emit events in a strict sequence (see `tests/test_runner_contract.py`):
- Exactly **one** `StartedEvent`
- Exactly **one** `CompletedEvent`
- `CompletedEvent` is **last**
- `CompletedEvent.resume == StartedEvent.resume`
Action events are optional. The minimal valid run is:
```
StartedEvent -> CompletedEvent
```
### Resume tokens
Runners own the resume format:
- `format_resume(token)` returns a command line users can paste
- `extract_resume(text)` parses resume tokens from user text
- `is_resume_line(line)` lets Takopi strip resume lines before running
---
## EngineBackend
```py
EngineBackend(
id: str,
build_runner: Callable[[EngineConfig, Path], Runner],
cli_cmd: str | None = None,
install_cmd: str | None = None,
)
```
- `id` must match the entrypoint name and the ID regex.
- `build_runner` should raise `ConfigError` for invalid config.
- `cli_cmd` is used to check whether the engine CLI is on `PATH`.
- `install_cmd` is surfaced in onboarding output.
---
## TransportBackend
```py
class TransportBackend(Protocol):
id: str
description: str
def check_setup(...) -> SetupResult: ...
def interactive_setup(self, *, force: bool) -> bool: ...
def lock_token(
self, *, transport_config: dict[str, object], config_path: Path
) -> str | None: ...
def build_and_run(
self,
*,
transport_config: dict[str, object],
config_path: Path,
runtime: TransportRuntime,
final_notify: bool,
default_engine_override: str | None,
) -> None: ...
```
Transport backends are responsible for:
- Validating config and onboarding users (`check_setup`, `interactive_setup`)
- Providing a lock token so Takopi can prevent parallel runs
- Starting the transport loop in `build_and_run`
---
## CommandBackend
```py
class CommandBackend(Protocol):
id: str
description: str
async def handle(self, ctx: CommandContext) -> CommandResult | None: ...
```
Command handlers receive a `CommandContext` with:
- the raw command text and parsed args
- the original message + reply metadata
- `config_path` for the active `takopi.toml` (when known)
- `plugin_config` from `[plugins.<id>]` (dict, defaults to `{}`)
- `runtime` (engine/project resolution)
- `executor` (send messages or run engines)
Use `ctx.executor.run_one(...)` or `ctx.executor.run_many(...)` to reuse Takopi's
engine pipeline. Use `mode="capture"` to collect results and build a custom reply.
---
## TransportRuntime helpers
`TransportRuntime` keeps transports away from internal router/project types. Key helpers:
- `resolve_message(text, reply_text)``ResolvedMessage` (prompt, resume token, context)
- `resolve_engine(engine_override, context)``EngineId`
- `resolve_runner(resume_token, engine_override)``ResolvedRunner` (runner + availability info)
- `resolve_run_cwd(context)``Path | None` (raises `ConfigError` for project/worktree issues)
- `format_context_line(context)``str | None`
- `available_engine_ids()` / `missing_engine_ids()` / `engine_ids` / `default_engine`
- `project_aliases()`
- `config_path` (active config path when available)
- `plugin_config(plugin_id)``dict` from `[plugins.<id>]`
---
## Bridge usage (transport plugins)
Most transports can delegate message handling to `handle_message()`. Use
`TransportRuntime` to resolve messages and select a runner:
```py
from takopi.api import (
ExecBridgeConfig,
IncomingMessage,
RunningTask,
RunningTasks,
TransportRuntime,
handle_message,
)
async def on_message(...):
resolved = runtime.resolve_message(text=text, reply_text=reply_text)
entry = runtime.resolve_runner(
resume_token=resolved.resume_token,
engine_override=resolved.engine_override,
)
context_line = runtime.format_context_line(resolved.context)
incoming = IncomingMessage(
channel_id=...,
message_id=...,
text=...,
reply_to=...,
)
await handle_message(
exec_cfg,
runner=entry.runner,
incoming=incoming,
resume_token=resolved.resume_token,
context=resolved.context,
context_line=context_line,
strip_resume_line=runtime.is_resume_line,
running_tasks=running_tasks,
on_thread_known=on_thread_known,
)
```
`handle_message()` implements:
- Progress updates and throttling
- Resume handling
- Cancellation propagation
- Final rendering
This keeps transport backends thin and consistent with core behavior.