docs: restructure docs into diataxis (#121)

This commit is contained in:
banteg
2026-01-13 15:59:27 +04:00
committed by GitHub
parent d0e9a51a0f
commit e292c99ab0
52 changed files with 1538 additions and 1255 deletions
+396
View File
@@ -0,0 +1,396 @@
# Takopi Architecture & Lifecycle
## Layer Diagram
```mermaid
flowchart TB
subgraph CLI["CLI Layer"]
cli[cli.py]
cli_desc["Entry point, config loading, lock file"]
end
subgraph Plugins["Plugin Layer"]
entrypoints[plugins.py<br/>entrypoint discovery]
engines[engines.py]
transports[transports.py]
commands[commands.py]
api[api.py<br/>public plugin API]
end
subgraph Orchestration["Orchestration Layer"]
router[AutoRouter<br/>router.py]
scheduler[ThreadScheduler<br/>scheduler.py]
projects[ProjectsConfig<br/>config.py]
runtime[TransportRuntime<br/>transport_runtime.py]
end
subgraph Bridge["Bridge Layer"]
tg_bridge[telegram/bridge.py<br/>run_main_loop]
runner_bridge[runner_bridge.py<br/>handle_message]
end
subgraph Runner["Runner Layer"]
runner_proto[Runner Protocol<br/>runner.py]
runners[runners/<br/>claude, codex, opencode, pi]
schemas[schemas/<br/>JSONL decoders]
end
subgraph Transport["Transport Layer"]
transport[Transport Protocol]
presenter[Presenter Protocol]
tg_client[telegram/client.py]
tg_render[telegram/render.py]
markdown[markdown.py]
end
subgraph External["External"]
agent_clis[Agent CLIs<br/>claude, codex, pi]
telegram_api[Telegram Bot API]
end
cli --> router
cli --> scheduler
cli --> projects
cli --> engines
cli --> transports
cli --> commands
engines --> entrypoints
transports --> entrypoints
commands --> entrypoints
router --> runtime
projects --> runtime
router --> tg_bridge
scheduler --> tg_bridge
runtime --> tg_bridge
tg_bridge --> commands
tg_bridge --> runner_bridge
runner_bridge --> runner_proto
runner_proto --> runners
runners --> schemas
runners --> agent_clis
runner_bridge --> transport
runner_bridge --> presenter
transport --> tg_client
presenter --> tg_render
presenter --> markdown
tg_client --> telegram_api
```
---
## Plugin Architecture
Takopi discovers plugins via Python entrypoints and keeps loading lazy:
- **Engine backends** (`takopi.engine_backends`)
- **Transport backends** (`takopi.transport_backends`)
- **Command backends** (`takopi.command_backends`)
Entrypoint names become plugin IDs, are validated up front (reserved names, regex),
and are only loaded when needed. The public surface for plugin authors lives in
`takopi.api`, while transports and commands interact with core routing via
`TransportRuntime`.
---
## Domain Model
```mermaid
classDiagram
class ResumeToken {
+engine: EngineId
+value: str
}
class Action {
+id: str
+kind: ActionKind
+title: str
+detail: dict
}
class StartedEvent {
+type: "started"
+engine: EngineId
+resume: ResumeToken
+title: str?
}
class ActionEvent {
+type: "action"
+engine: EngineId
+action: Action
+phase: started|updated|completed
+ok: bool?
+message: str?
}
class CompletedEvent {
+type: "completed"
+engine: EngineId
+ok: bool
+answer: str
+resume: ResumeToken?
+usage: dict?
}
StartedEvent --> ResumeToken
ActionEvent --> Action
CompletedEvent --> ResumeToken
note for Action "ActionKind: command | tool | file_change |\nweb_search | subagent | note | turn | warning | telemetry"
```
---
## Message Lifecycle
```mermaid
sequenceDiagram
participant User
participant Telegram
participant Bridge as telegram/bridge.py
participant Scheduler as ThreadScheduler
participant RunnerBridge as runner_bridge.py
participant Runner
participant AgentCLI as Agent CLI
participant Command as Command Plugin
User->>Telegram: Send message
Telegram->>Bridge: poll_incoming()
Bridge->>Bridge: Parse slash command
alt Command plugin
Bridge->>Command: handle(ctx)
Command->>RunnerBridge: run_one/run_many (optional)
RunnerBridge->>Telegram: Send progress/final
else Default routing
Bridge->>Bridge: Parse directives<br/>(/engine, /project, @branch)
Bridge->>Bridge: Extract resume token<br/>from reply
Bridge->>Bridge: Resolve worktree<br/>(if @branch)
Bridge->>Scheduler: enqueue(ThreadJob)
Scheduler->>RunnerBridge: handle_message()
RunnerBridge->>Telegram: Send progress message
RunnerBridge->>Runner: run(prompt, resume)
end
Runner->>AgentCLI: Spawn subprocess
loop JSONL Stream
AgentCLI-->>Runner: JSONL event
Runner-->>RunnerBridge: TakopiEvent
RunnerBridge->>Telegram: Edit progress message
end
AgentCLI-->>Runner: Completed
Runner-->>RunnerBridge: CompletedEvent
RunnerBridge->>Telegram: Send final answer
RunnerBridge->>Telegram: Delete progress message
```
---
## Runner Execution Flow
```mermaid
flowchart TD
A[runner.run\nprompt, resume_token] --> B[Acquire Session Lock<br/>SessionLockMixin]
B --> C[Build Command]
C --> D{Engine?}
D -->|Claude| D1["claude --print --output-format stream-json<br/>[--resume id] prompt"]
D -->|Codex| D2["codex exec --json<br/>[resume &lt;token&gt;] -"]
D -->|Pi| D3["pi --print --mode json<br/>--session &lt;id&gt; &lt;prompt&gt;"]
D -->|OpenCode| D4["opencode run --format json<br/>[--session id] -- &lt;prompt&gt;"]
D1 --> E[Spawn Subprocess<br/>anyio.open_process]
D2 --> E
D3 --> E
D4 --> E
E --> F[Stream JSONL from stdout]
F --> G[Decode with msgspec]
G --> H[Translate to TakopiEvent]
H --> I[yield event]
I --> F
F -->|EOF| J[Return]
```
---
## Resume Token Flow
```mermaid
sequenceDiagram
participant User
participant Bridge
participant Runner
participant CLI as Agent CLI
Note over User,CLI: New Conversation
User->>Bridge: "fix the bug"
Bridge->>Runner: run(prompt, None)
Runner->>CLI: claude "fix the bug"
CLI-->>Runner: StartedEvent(resume=abc123)
Runner-->>Bridge: Stream events
Bridge->>User: Final message with:<br/>claude --resume abc123<br/>ctx: project @branch
Note over User,CLI: Resume Conversation
User->>Bridge: Reply: "now add tests"
Bridge->>Bridge: extract_resume(reply_text)<br/>→ ResumeToken(claude, abc123)
Bridge->>Bridge: parse_ctx_line()<br/>→ project, branch
Bridge->>Runner: run("now add tests", token)
Runner->>CLI: claude --resume abc123 "now add tests"
CLI-->>Runner: Continues session
Runner-->>Bridge: Stream events
Bridge->>User: Final message
```
---
## Component Dependencies
```mermaid
flowchart TD
cli[cli.py] --> config[config.py]
cli --> engines[engines.py]
cli --> transports[transports.py]
cli --> commands[commands.py]
cli --> lockfile[lockfile.py]
engines --> plugins[plugins.py]
transports --> plugins
commands --> plugins
engines --> backends[backends.py]
backends --> runners[runners/]
backends --> runner[runner.py]
subgraph runners[runners/]
claude[claude.py]
codex[codex.py]
opencode[opencode.py]
pi[pi.py]
end
subgraph schemas[schemas/]
claude_s[claude.py]
codex_s[codex.py]
opencode_s[opencode.py]
pi_s[pi.py]
end
claude --> claude_s
codex --> codex_s
opencode --> opencode_s
pi --> pi_s
cli --> router[router.py]
tg_bridge --> runtime[transport_runtime.py]
runtime --> router
runtime --> config
tg_bridge --> commands
runner --> runner_bridge[runner_bridge.py]
runner_bridge --> tg_bridge
tg_bridge --> client[telegram/client.py]
tg_bridge --> render[telegram/render.py]
client --> transport[transport.py]
runner_bridge --> progress[progress.py]
runner_bridge --> events[events.py]
render --> presenter[presenter.py]
presenter --> markdown[markdown.py]
```
---
## Configuration Structure
```mermaid
flowchart LR
subgraph Config["~/.takopi/"]
toml[takopi.toml]
lock[takopi.lock]
end
subgraph toml_contents["takopi.toml"]
direction TB
global["transport<br/>default_engine<br/>default_project"]
telegram_cfg["[transports.telegram]<br/>bot_token = ...<br/>chat_id = ..."]
plugins_cfg["[plugins]<br/>enabled = [...]"]
plugins_extra["[plugins.mycommand]<br/>setting = ..."]
claude_cfg["[claude]<br/>model = ..."]
codex_cfg["[codex]<br/>model = ..."]
projects_cfg["[projects.alias]<br/>path = ...<br/>worktrees_dir = ...<br/>default_engine = ..."]
end
toml --> toml_contents
```
---
## Thread Scheduling
```mermaid
flowchart TD
subgraph Incoming[Incoming Messages]
m1[Message 1<br/>new thread]
m2[Message 2<br/>reply to thread A]
m3[Message 3<br/>reply to thread A]
m4[Message 4<br/>new thread]
end
subgraph Scheduler[ThreadScheduler]
direction TB
q1[Thread A Queue]
q2[Thread B Queue]
q3[Thread C Queue]
end
subgraph Workers[Worker Tasks]
w1[Worker A]
w2[Worker B]
w3[Worker C]
end
m1 --> q2
m2 --> q1
m3 --> q1
m4 --> q3
q1 --> w1
q2 --> w2
q3 --> w3
w1 --> runner1[Runner.run]
w2 --> runner2[Runner.run]
w3 --> runner3[Runner.run]
note1[Jobs in same thread<br/>execute sequentially]
note2[Different threads<br/>execute in parallel]
```
---
## Summary
| Layer | Components | Responsibility |
|-------|------------|----------------|
| **CLI** | `cli.py` | Entry point, config, lock |
| **Plugins** | `plugins.py`, `engines.py`, `transports.py`, `commands.py`, `api.py` | Entrypoint discovery, plugin loading, public API boundary |
| **Orchestration** | `router.py`, `scheduler.py`, `config.py` | Engine selection, job queuing, project config |
| **Bridge** | `telegram/bridge.py`, `runner_bridge.py` | Message handling, execution coordination |
| **Runner** | `runner.py`, `runners/*.py`, `schemas/*.py` | Agent CLI subprocess, JSONL parsing, event translation |
| **Transport** | `transport.py`, `presenter.py`, `telegram/client.py` | Telegram API, message rendering |
| **Domain** | `model.py`, `progress.py`, `events.py` | Event types, action tracking |
| **Utils** | `worktrees.py`, `utils/*.py`, `markdown.py` | Git worktrees, formatting, paths |
+43
View File
@@ -0,0 +1,43 @@
# Explanation
Explanation docs answer **“how does this work?”** and **“why is it designed this way?”**
If you want step-by-step instructions, go to **[Tutorials](../tutorials/index.md)**.
If you want exact options and contracts, go to **[Reference](../reference/index.md)**.
## How Takopi works end-to-end
- Incoming Telegram message → resolve context (project/branch) → resolve resume token → select runner → stream events → render progress → send final + resume line.
Start here:
- [Architecture](architecture.md)
## Routing, sessions, and continuation
Takopi is stateless by default, but can provide “continuation” in multiple ways:
- reply-to-continue (always available)
- per-topic resume (Telegram forum topics)
- per-chat sessions (auto-resume)
- [Routing & sessions](routing-and-sessions.md)
## Plugins and extensibility
Takopi uses entrypoint-based plugins with lazy discovery so broken plugins dont brick the CLI.
- [Plugin system](plugin-system.md)
## Codebase orientation
If youre making changes, this is the “map of the territory”:
- [Module map](module-map.md)
## Where to look for hard rules
Explanation pages describe intent and tradeoffs. The *hard requirements* live in:
- [Reference: Specification](../reference/specification.md)
- [Reference: Plugin API](../reference/plugin-api.md)
+81
View File
@@ -0,0 +1,81 @@
# Module map
This page is a high-level map of Takopis internal modules: what they do and how they fit together.
## Entry points
| Module | Responsibility |
|--------|----------------|
| `cli.py` | Typer CLI entry point; loads settings, selects engine/transport, runs the transport backend. |
| `telegram/backend.py` | Telegram transport backend: validates config, runs onboarding, builds and runs the Telegram bridge. |
## Orchestration and routing
| Module | Responsibility |
|--------|----------------|
| `runner_bridge.py` | Transport-agnostic orchestration: per-message handler, progress updates, final render, cancellation, resume coordination. |
| `router.py` | Auto-router: resolves resume tokens by polling runners; selects a runner for a message. |
| `scheduler.py` | Per-thread FIFO job queueing with serialization. |
| `transport_runtime.py` | Facade used by transports and commands to resolve messages and runners without importing internal router/project types. |
## Domain model and events
| Module | Responsibility |
|--------|----------------|
| `model.py` | Domain types: resume tokens, events, actions, run results. |
| `runner.py` | Runner protocol and event queue utilities. |
| `events.py` | Event factory helpers for building Takopi events consistently. |
## Rendering and progress
| Module | Responsibility |
|--------|----------------|
| `progress.py` | Progress tracking: reduces takopi events into progress snapshots. |
| `markdown.py` | Markdown formatting for progress/final messages; includes helpers like elapsed formatting. |
| `presenter.py` | Presenter protocol: converts `ProgressState` into transport-specific messages. |
| `transport.py` | Transport protocol: send/edit/delete abstractions and message reference types. |
## Telegram implementation
| Module | Responsibility |
|--------|----------------|
| `telegram/bridge.py` | Telegram bridge loop: polls updates, filters messages, dispatches handlers, coordinates cancellation. |
| `telegram/client.py` | Telegram API wrapper with retry/outbox semantics. |
| `telegram/render.py` | Telegram markdown rendering and trimming. |
| `telegram/onboarding.py` | Interactive setup and setup validation UX. |
| `telegram/commands/*` | In-chat command handlers (`/agent`, `/file`, `/topic`, `/ctx`, `/new`, …). |
## Plugins
| Module | Responsibility |
|--------|----------------|
| `plugins.py` | Entrypoint discovery and lazy loading (capture load errors, filter by enabled list). |
| `engines.py` | Engine backend discovery and loading via entrypoints. |
| `transports.py` | Transport backend discovery and loading via entrypoints. |
| `commands.py` | Command backend discovery and loading via entrypoints; command execution helpers. |
| `ids.py` | Shared ID regex and collision checks for plugin ids and Telegram command names. |
| `api.py` | Public plugin API boundary (`takopi.api` re-exports). |
## Runners and schemas
| Module | Responsibility |
|--------|----------------|
| `runners/*` | Engine runner implementations (Codex, Claude, OpenCode, Pi). |
| `schemas/*` | msgspec schemas / decoders for engine JSONL streams. |
## Configuration and persistence
| Module | Responsibility |
|--------|----------------|
| `settings.py` | Loads `takopi.toml` (TOML + env), validates with pydantic-settings. |
| `config_store.py` | Raw TOML read/write (merge/update without clobbering extra sections). |
| `config_migrations.py` | One-time edits to on-disk config (e.g. legacy Telegram key migration). |
## Utilities
| Module | Responsibility |
|--------|----------------|
| `utils/paths.py` | Path/command relativization helpers. |
| `utils/streams.py` | Async stream helpers (`iter_bytes_lines`, stderr draining). |
| `utils/subprocess.py` | Subprocess management helpers (terminate/kill best-effort). |
+88
View File
@@ -0,0 +1,88 @@
# Plugin system
Takopi uses Python entrypoints to extend engines, transports, and commands.
## Why entrypoints
Entrypoints let Takopi discover plugins without hard dependencies on plugin packages.
Installed distributions declare what they provide, and Takopi can list and load them at runtime.
This makes it possible to:
- Add new engines/transports/commands without changing Takopi itself.
- Ship plugins independently.
- Keep the core CLI small.
## Why discovery is lazy
Takopi lists plugin IDs **without importing plugin code**, then imports a plugin only when:
- it is selected by routing (engine/transport), or
- it is invoked as a command, or
- you explicitly request loading via `takopi plugins --load`.
This keeps `takopi --help` fast and prevents a broken third-party plugin from bricking the CLI.
## Entrypoint rules (what Takopi expects)
Takopi uses three entrypoint groups:
```toml
[project.entry-points."takopi.engine_backends"]
myengine = "myengine.backend:BACKEND"
[project.entry-points."takopi.transport_backends"]
mytransport = "mytransport.backend:BACKEND"
[project.entry-points."takopi.command_backends"]
mycommand = "mycommand.backend:BACKEND"
```
Rules:
- The entrypoint **name** is the plugin id.
- The entrypoint value must resolve to a backend object:
- engine backend: `EngineBackend`
- transport backend: `TransportBackend`
- command backend: `CommandBackend`
- The backend object must have `id == entrypoint name`.
## Why there is an enabled list
Plugin visibility can be restricted via:
```toml
[plugins]
enabled = ["takopi-engine-acme", "takopi-transport-slack"]
```
When set, Takopi filters by **distribution name** (package metadata), not by entrypoint name.
This lets you:
- ship multiple entrypoints from one distribution, and
- enable/disable whole plugin packages predictably.
## IDs and collisions
Entrypoint names become plugin IDs and appear in user-facing surfaces (CLI subcommands, Telegram commands, `/engine` directives).
Takopi validates IDs and rejects collisions with reserved names.
Plugin IDs must match:
```
^[a-z0-9_]{1,32}$
```
Reserved IDs include core chat and CLI command names such as `cancel`, `init`, and `plugins`.
## How to debug discovery and loading
```sh
takopi plugins
takopi plugins --load
```
## Related
- [Write a plugin](../how-to/write-a-plugin.md)
- [Plugin API reference](../reference/plugin-api.md)
+44
View File
@@ -0,0 +1,44 @@
# Routing & sessions
Takopi is **stateless by default**: each message starts a new engine session unless a resume token is present.
## Continuation (how threads persist)
Takopi supports three ways to continue a thread:
1. **Reply-to-continue** (always available)
- Reply to any bot message that contains a resume line in the footer.
- Takopi extracts the resume token and resumes that engine thread.
2. **Forum topics** (optional)
- Topics can store resume tokens per topic and auto-resume new messages in that topic.
- Topic state is stored in `telegram_topics_state.json`.
- Reset with `/new`.
3. **Chat sessions** (optional)
- Set `session_mode = "chat"` to store one resume token per chat (per sender in groups).
- State is stored in `telegram_chat_sessions_state.json`.
- Reset with `/new`.
Reply-to-continue works even if topics or chat sessions are enabled.
## Routing (how Takopi picks a runner)
For each message, Takopi:
- parses directive prefixes (`/engine`, `/project`, `@branch`) from the first non-empty line
- attempts to extract a resume token by polling available runners
- if a resume token is found, routes to the matching runner; otherwise uses the configured default engine
## Serialization (why you dont get overlapping runs)
Takopi allows parallel runs across **different threads**, but enforces serialization within a thread:
- Telegram side: jobs are queued FIFO per thread.
- Runner side: runners enforce per-resume-token locks (so the same session cant be resumed concurrently).
The precise invariants are specified in the [Specification](../reference/specification.md).
## Related
- [Commands & directives](../reference/commands-and-directives.md)
- [Context resolution](../reference/context-resolution.md)