izackp/takopi
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: introduce runner protocol and normalized event model (#7)

Browse Source
This commit is contained in:
banteg
2026-01-01 01:13:55 +04:00
committed by GitHub
parent a9f8967bf4
commit d296c0dbf1
36 changed files with 4749 additions and 1836 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/workflows/ci.yml
+38 -22
View File
@@ -3,23 +3,41 @@ name: CI
on:
push:
branches:
- "**"
tags-ignore:
- "v*"
- "master"
pull_request:
concurrency:
group: ci-${{ github.ref }}
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
test:
name: Python ${{ matrix.python-version }}
checks:
name: ${{ matrix.task }}
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
python-version: ["3.12", "3.13", "3.14"]
include:
- task: format
do_sync: true
command: uv run --no-sync ruff format --check --diff
sync_args: --no-install-project
- task: ruff
do_sync: true
command: uv run --no-sync ruff check . --output-format=github
sync_args: --no-install-project
- task: ty
do_sync: true
command: uv run --no-sync ty check .
sync_args: --no-install-project
- task: pytest
do_sync: true
command: uv run --no-sync pytest
sync_args: ""
- task: build
do_sync: false
command: uv build
sync_args: ""
steps:
- name: Checkout
@@ -28,23 +46,21 @@ jobs:
- name: Install uv
uses: astral-sh/setup-uv@v7
with:
python-version: ${{ matrix.python-version }}
python-version: "3.14"
enable-cache: true
- name: Install dependencies
run: uv sync
if: matrix.do_sync
run: uv sync --frozen ${{ matrix.sync_args }}
- name: Lint (format)
run: uv run ruff format --check
- name: Run check
run: ${{ matrix.command }}
- name: Lint (ruff)
run: uv run ruff check .
- name: Type check
run: uv run ty check .
- name: Tests
run: uv run pytest
- name: Build (wheel + sdist)
run: uv build
- name: Add coverage to summary
if: ${{ always() && matrix.task == 'pytest' }}
run: |
{
echo "## Coverage"
echo ""
uv run --no-sync python -m coverage report || true
} >> "$GITHUB_STEP_SUMMARY"
.github/workflows/release.yml
+1 -1
View File
@@ -18,7 +18,7 @@ jobs:
- name: Install uv
uses: astral-sh/setup-uv@v7
with:
python-version: "3.13"
python-version: "3.14"
enable-cache: true
- name: Check tag matches project version
Makefile
+1 -1
View File
@@ -1,7 +1,7 @@
.PHONY: check
check:
uv run ruff format
uv run ruff format --check
uv run ruff check .
uv run ty check .
uv run pytest
changelog.md
+36 -9
View File
@@ -1,16 +1,43 @@
# changelog
## v0.2.0 (2025-12-31)
### changes
- introduce runner protocol for multi-engine support #8
- normalized event model (`started`, `action`, `completed`)
- actions with stable ids, lifecycle phases, and structured details
- engine-agnostic bridge and renderer
- add `/cancel` command with progress message targeting #4
- migrate async runtime from asyncio to anyio #6
- stream runner events via async iterators (natural backpressure)
- per-thread job queues with serialization for same-thread runs
- emit `completed` as terminal event (carries resume + final answer)
- render resume as `` `codex resume <token>` `` command lines
### breaking
- require python 3.14+
- remove `--profile` flag; configure via `[codex].profile` only
### fixes
- serialize new sessions once resume token is known
- preserve resume tokens in error renders #3
- preserve file-change paths in action events #2
- terminate codex process groups on cancel (POSIX)
- correct resume command matching in bridge
## v0.1.0 (2025-12-29)
initial release.
### features
- telegram bot bridge for openai codex cli using `codex exec` and `codex exec resume`
- stateless session resume via `resume: <uuid>` lines embedded in messages
- real-time progress updates with ~2s throttling, showing commands, tools, and elapsed time
- full markdown rendering with telegram entity support (via markdown-it-py + sulguk)
- concurrent message handling with per-session serialization to prevent race conditions
- automatic telegram token redaction in logs
- telegram bot bridge for openai codex cli via `codex exec`
- stateless session resume via `` `codex resume <token>` `` lines
- real-time progress updates with ~2s throttling
- full markdown rendering with telegram entities (markdown-it-py + sulguk)
- per-session serialization to prevent race conditions
- interactive onboarding guide for first-time setup
- cli options: `--profile`, `--debug`, `--final-notify`, `--version`
- codex profile configuration
- automatic telegram token redaction in logs
- cli options: `--debug`, `--final-notify`, `--version`
developing.md
+87 -51
View File
@@ -1,6 +1,7 @@
# takopi — Developer Guide
# takopi - Developer Guide
This document describes the internal architecture and module responsibilities.
See `specification.md` for the authoritative behavior spec.
## Development Setup
@@ -27,71 +28,95 @@ make check
## Module Responsibilities
### `exec_bridge.py` — Main Entry Point
### `bridge.py` - Telegram bridge loop
The orchestrator module containing:
| Component | Purpose |
|-----------|---------|
| `main()` / `run()` | CLI entry point via Typer |
| `BridgeConfig` | Frozen dataclass holding runtime config |
| `CodexExecRunner` | Spawns `codex exec`, streams JSONL, handles cancellation |
| `poll_updates()` | Async generator that drains backlog, long-polls updates, filters messages |
| `_run_main_loop()` | TaskGroup-based main loop that spawns per-message handlers |
| `handle_message()` | Per-message handler with progress updates |
| `extract_session_id()` | Parses `resume: <uuid>` from message text |
| `truncate_for_telegram()` | Smart truncation preserving resume lines |
| `handle_message()` | Per-message handler with progress updates and final render |
| `ProgressEdits` | Throttled progress edit worker |
| `_handle_cancel()` | `/cancel` routing |
| `truncate_for_telegram()` | Moved to `markdown.py` |
**Key patterns:**
- Per-session locks prevent concurrent resumes to the same `session_id`
- Worker pool with an AnyIO memory stream limits concurrency (default: 16 workers)
- AnyIO task groups manage worker tasks
- Progress edits are throttled to ~2s intervals
- Subprocess stderr is drained to a bounded deque for error reporting
- `poll_updates()` uses Telegram `getUpdates` long-polling with a single server-side updates
queue per bot token; updates are confirmed when a client requests a higher `offset`, so
multiple instances with the same token will race (duplicates and/or missed updates)
- Bridge schedules runs FIFO per thread to avoid concurrent progress messages; runner locks enforce per-thread serialization
- `/cancel` routes by reply-to progress message id (accepts extra text)
- Progress edits are throttled to ~1s intervals and only run when new events arrive
- Resume tokens are runner-formatted command lines (e.g., `` `codex resume <token>` ``)
- Resume parsing is delegated to the active runner (no cross-engine fallback)
### `telegram.py` — Telegram Bot API
### `cli.py` - CLI entry point
Minimal async client wrapping the Bot API:
| Component | Purpose |
|-----------|---------|
| `run()` / `main()` | Typer CLI entry points |
| `_parse_bridge_config()` | Reads config + builds `BridgeConfig` |
```python
class TelegramClient:
async def get_updates(...) # Long-polling
async def send_message(...) # With entities support
async def edit_message_text(...)
async def delete_message(...)
```
### `markdown.py` - Telegram markdown helpers
**Features:**
- Automatic retry on 429 (rate limit) with `retry_after`
- Raises `TelegramAPIError` with payload details on failure
| Function | Purpose |
|----------|---------|
| `render_markdown()` | Markdown → Telegram text + entities |
| `prepare_telegram()` | Render + truncate for Telegram limits |
| `truncate_for_telegram()` | Smart truncation preserving resume lines |
### `exec_render.py` — JSONL Event Rendering
### `runners/codex.py` - Codex runner
Transforms Codex JSONL events into human-readable text:
| Component | Purpose |
|-----------|---------|
| `CodexRunner` | Spawns `codex exec --json`, streams JSONL, emits takopi events |
| `translate_codex_event()` | Normalizes Codex JSONL into the takopi event schema |
| `manage_subprocess()` | Starts a new process group and kills it on cancellation (POSIX) |
**Key patterns:**
- Per-resume locks (WeakValueDictionary) prevent concurrent resumes of the same session
- Event delivery uses a single internal queue to preserve order without per-event tasks
- Stderr is drained into a bounded tail (debug logging only)
- Event callbacks must not raise; callback errors abort the run
### `render.py` - Takopi event rendering
Transforms takopi events into human-readable text:
| Function/Class | Purpose |
|----------------|---------|
| `format_event()` | Core dispatcher returning `(item_num, cli_lines, progress_line, prefix)` |
| `render_event_cli()` | Simplified wrapper for console logging |
| `ExecProgressRenderer` | Stateful renderer tracking recent actions for progress display |
| `render_event_cli()` | Format a takopi event for CLI logs |
| `format_elapsed()` | Formats seconds as `Xh Ym`, `Xm Ys`, or `Xs` |
| `render_markdown()` | Markdown → Telegram text + entities (markdown-it-py + sulguk) |
| `render_markdown()` | Moved to `markdown.py` |
**Supported event types:**
- `thread.started`, `turn.started/completed/failed`
- `item.started/completed` for: `agent_message`, `reasoning`, `command_execution`, `mcp_tool_call`, `web_search`, `file_change`, `error`
- `started`
- `action`
- `completed`
### `config.py` — Configuration Loading
### `model.py` / `runner.py` - Core domain types
| File | Purpose |
|------|---------|
| `model.py` | Domain types: resume tokens, actions, events, run result |
| `runner.py` | Runner protocol + event queue utilities |
### `runners/` - Runner implementations
| File | Purpose |
|------|---------|
| `engines.py` | Engine backend registry (setup checks + runner construction) |
| `runners/codex.py` | Codex runner (JSONL → takopi events) + per-resume locks |
| `runners/mock.py` | Mock runner for tests/demos |
### `config.py` - Configuration loading
```python
def load_telegram_config() -> tuple[dict, Path]:
# Loads ./.codex/takopi.toml, then ~/.codex/takopi.toml
# Loads ./.takopi/takopi.toml, then ~/.takopi/takopi.toml
```
### `logging.py` — Secure Logging Setup
### `logging.py` - Secure logging setup
```python
class RedactTokenFilter:
@@ -101,16 +126,25 @@ def setup_logging(*, debug: bool):
# Configures root logger with redaction filter
```
### `onboarding.py` — Setup Validation
### `onboarding.py` - Setup validation
```python
def check_setup() -> SetupResult:
# Validates codex CLI on PATH and config file
def check_setup(backend: EngineBackend) -> SetupResult:
# Validates engine CLI on PATH and config file
def render_setup_guide(result: SetupResult):
# Displays rich panel with setup instructions
```
## Adding a Runner
1. Implement the `Runner` protocol in `src/takopi/runners/<engine>.py`.
2. Emit Takopi events from `takopi.model` and implement resume helpers
(`format_resume`, `extract_resume`, `is_resume_line`).
3. Register an `EngineBackend` in `src/takopi/engines.py` with setup checks
and runner construction.
4. Extend tests (runner contract + any engine-specific translation tests).
## Data Flow
### New Message Flow
@@ -126,17 +160,18 @@ handle_message() spawned as task
↓
Send initial progress message (silent)
↓
CodexExecRunner.run_serialized()
CodexRunner.run()
├── Spawns: codex exec --json ... -
├── Streams JSONL from stdout
├── Calls on_event() for each event
├── Normalizes JSONL -> takopi events
├── Yields Takopi events (async iterator)
│ ↓
│ ExecProgressRenderer.note_event()
│ ↓
│ Throttled edit_message_text()
└── Returns (session_id, answer, saw_agent_message)
│ ProgressEdits throttled edit_message_text()
└── Ends with completed(resume, ok, answer)
↓
render_final() with resume line
render_final() with resume line (runner-formatted)
↓
Send/edit final message
```
@@ -144,15 +179,16 @@ Send/edit final message
### Resume Flow
Same as above, but:
- `extract_session_id()` finds UUID in message or reply
- Command becomes: `codex exec --json resume <session_id> -`
- Per-session lock serializes concurrent resumes
- Runners parse resume lines (e.g. `` `codex resume <token>` ``)
- Command becomes: `codex exec --json resume <token> -`
- Per-token lock serializes concurrent resumes
## Error Handling
| Scenario | Behavior |
|----------|----------|
| `codex exec` fails (rc≠0) | Shows stderr tail in error message |
| `codex exec` fails (rc != 0) | Emits a warning `action` plus `completed(ok=false, error=...)` |
| Telegram API error | Logged, edit skipped (progress continues) |
| Cancellation | Cancel scope triggers terminate; cancellation is detected via `cancelled_caught` |
| No agent_message | Final shows "error" status |
| Cancellation | Cancel scope terminates the process group (POSIX) and renders `cancelled` |
| Errors in handler | Final render uses `status=error` and preserves resume tokens when known |
| No agent_message (empty answer) | Final shows `error` status |
docs/runner/codex/codex-takopi-events.md
+423
View File
@@ -0,0 +1,423 @@
Here’s a clean way to make “Takopi events” just **3 shapes** while still covering **every `codex exec --json` line type** and preserving the invariants you care about (stable IDs, resume/thread ownership, final answer delivery).
## The 3-event Takopi schema
I’d model it like this (JSON-ish). The important trick is: **your single `action` event needs a `phase`**, otherwise you can’t represent started/updated/completed lifecycles.
### 1) `started`
Emitted once **as soon as you know the resume token** (Codex: `thread.started.thread_id`).
```json
{
"type": "started",
"engine": "codex",
"resume": { "engine": "codex", "value": "0199..." },
"title": "Codex", // optional
"meta": { "raw": { ... } } // optional: for debugging
}
```
### 2) `action`
Emitted for **everything that is progress / updates / warnings / per-item lifecycle**.
```json
{
"type": "action",
"engine": "codex",
"action": {
"id": "item_5",
"kind": "tool", // command | tool | file_change | web_search | note | turn | warning | telemetry
"title": "docs.search", // short label for renderer
"detail": { ... } // structured payload (freeform)
},
"phase": "started", // started | updated | completed
"ok": true, // optional; present when phase=completed (or warnings)
"message": "optional text", // optional; logs/warnings can use this
"level": "info" // optional: debug|info|warning|error
}
```
### 3) `completed`
Emitted once at end-of-run with the **final answer** (from `agent_message`) and final status.
```json
{
"type": "completed",
"engine": "codex",
"resume": { "engine": "codex", "value": "0199..." }, // if known
"ok": true,
"answer": "Done. I updated the docs...",
"error": null,
"usage": { "input_tokens": 24763, "cached_input_tokens": 24448, "output_tokens": 122 } // optional
}
```
Why this fits Takopi cleanly:
* Your `started` corresponds to the old “session.started” concept (runner learns resume token; bridge can now safely serialize per thread).
* Your `action` is “everything that would have been action.started/action.completed/log/error” collapsed into one stream.
* Your `completed` corresponds to final `RunResult` + status, using Codex’s `agent_message` as the answer source.
---
## How everything fits together (end-to-end)
From the bridge/runner point of view:
1. **Bridge receives Telegram prompt**
2. Bridge tries to extract a resume line (`codex resume <uuid>`) from the message/reply (runner-owned parsing).
3. Bridge calls `runner.run(prompt, resumeTokenOrNone, on_event=...)`
4. Codex runner spawns `codex exec --json ...` and reads JSONL line-by-line.
5. The *first moment the runner can know thread identity* is:
* `thread.started` → contains `thread_id` (this is your resume value)
6. Runner must (per Takopi’s concurrency invariant) **acquire the per-thread lock as soon as the new thread token is known**, before emitting `started`.
7. Runner translates subsequent Codex JSONL lines into `action` events for progress rendering.
8. Runner captures the final answer from `item.completed` where `item.type="agent_message"`.
9. Runner emits exactly one `completed` event when the run ends (`turn.completed` or failure), including the captured final answer.
---
## Direct translation: every Codex `exec --json` line → your 3-event schema
Codex emits two categories: **top-level lines** and **item lines**.
### A) Top-level lines
#### `thread.started`
Codex:
```json
{"type":"thread.started","thread_id":"0199..."}
```
→ Takopi:
* emit **`started`**:
* `resume.value = thread_id`
This is exactly the “learn resume tag” moment you described.
---
#### `turn.started`
Codex:
```json
{"type":"turn.started"}
```
→ Takopi (recommended):
* emit **`action`** with a synthetic action id, e.g. `"turn_0"`
* `kind="turn"`, `phase="started"`, `title="turn started"`
You *can* also drop it if your UI doesn’t care, but if you want “every codex type translates”, this maps cleanly into `action`.
---
#### `turn.completed`
Codex includes usage:
```json
{"type":"turn.completed","usage":{...}}
```
→ Takopi:
* emit **`completed`**
* `ok=true`
* `answer = last seen agent_message text` (or `""` if none)
* `usage = usage` (optional)
This is your authoritative “run succeeded” boundary.
---
#### `turn.failed`
Codex:
```json
{"type":"turn.failed","error":{"message":"..."}}
```
→ Takopi:
* emit **`completed`**
* `ok=false`
* `error = error.message`
* `answer = last seen agent_message` (if any; usually empty)
This is “run ended, but failed”.
---
#### Top-level `error` (stream error)
Codex:
```json
{"type":"error","message":"stream error: broken pipe"}
```
Cheatsheet meaning: this is a **fatal stream failure** (not just a tool failure).
→ Takopi:
* if you haven’t emitted `completed` yet: emit **`completed`** with `ok=false` and `error=message`
* if you *already* emitted `completed`, treat it as an extra warning (or ignore; it’s “post-mortem noise”)
---
### B) Item lines: `item.started`, `item.updated`, `item.completed`
All item lines include `item.id` and it is stable across updates/completion.
That means your `action.action.id` should just be `item.id` — perfect match to “stable within a run”.
#### General rule (for any item.* line)
* `action.action.id = item.id`
* `action.phase = started | updated | completed`
* `action.action.kind` derived from `item.type`
* `action.action.detail` contains the relevant item fields (possibly trimmed)
Now, map each `item.type`:
---
## Item-type mapping: `item.type` → `action.kind/title/detail/ok`
Below is a “complete coverage” mapping for all item types listed in the cheatsheet.
### 1) `agent_message` (only `item.completed`)
Codex:
```json
{"type":"item.completed","item":{"id":"item_3","type":"agent_message","text":"..."}}
```
→ Takopi:
* **do not emit an `action`** (recommended)
* instead: **store** `final_answer = item.text`
* final answer will be surfaced by the eventual `completed` event
Reason: you want `completed` to be “final answer delivery”, and you probably don’t want the answer duplicated in progress rendering.
(If you *do* want to render it as it arrives, you can emit an `action` too, but then your renderer must avoid showing it twice.)
---
### 2) `reasoning` (only `item.completed`, if enabled)
Codex gives a text breadcrumb.
→ Takopi `action`:
* `kind="note"`
* `title="reasoning"` (or “thought”)
* `phase="completed"`
* `message=item.text` (or put it under `detail.text`)
This is usually safe to show as a short “what it’s doing” line (or ignore if you don’t want to surface it).
---
### 3) `command_execution` (`item.started` and `item.completed`)
Codex fields include `command`, `exit_code`, `status`, `aggregated_output` (often noisy).
→ Takopi `action`:
* `kind="command"`
* `title=item.command` (or a shortened version like `pytest`)
* `detail={ command, exit_code, status }` (optionally include output tail)
* `phase="started"` on `item.started`
* `phase="completed"` on `item.completed`
* `ok = (item.status == "completed")` (or `exit_code == 0`)
Note: “failed” command becomes `ok=false` but it’s still just an `action` completion — the overall run might still succeed later, depending on agent behavior.
---
### 4) `file_change` (only `item.completed`)
Codex contains `changes[]` and `status`.
→ Takopi `action`:
* `kind="file_change"`
* `title="file changes"`
* `detail={ changes }`
* `phase="completed"`
* `ok = (item.status == "completed")`
This is a great progress line for your UI (“updated docs/…, added …”).
---
### 5) `mcp_tool_call` (`item.started` and `item.completed`)
Codex contains server/tool/arguments/result/error/status. Result can be large; may include base64 in content blocks.
→ Takopi `action`:
* `kind="tool"`
* `title=f"{item.server}.{item.tool}"`
* `detail={ server, tool, arguments, status }`
* on completion, include *summary* of result:
* e.g. `detail.result_summary = { content_blocks: N, has_structured: bool }`
* include `detail.error_message` if failed
* `phase="started"` or `"completed"`
* `ok = (item.status == "completed")`
Recommendation: **do not dump** full `result.content` into `detail` if it can contain large blobs; keep a summary and optionally stash full raw elsewhere for debugging.
---
### 6) `web_search` (only `item.completed`)
Codex includes `query`.
→ Takopi `action`:
* `kind="web_search"`
* `title="web search"`
* `detail={ query }`
* `phase="completed"`
* `ok=true` (this is just “it did a search”; success/failure is typically not expressed here)
---
### 7) `todo_list` (`item.started`, `item.updated`, `item.completed`)
Codex includes checklist items with `completed` booleans.
→ Takopi `action`:
* `kind="note"` (or `"todo"`)
* `title="plan"`
* `detail={ items, done: count_done, total: count_total }`
* `phase` maps 1:1 to started/updated/completed
* `ok=true` when phase completed (optional)
This is the one case where `item.updated` is common; your unified `action` event is exactly the right shape for it.
---
### 8) Item `error` (non-fatal warning as an item; only `item.completed`)
Codex:
```json
{"type":"item.completed","item":{"id":"item_9","type":"error","message":"command output truncated"}}
```
Cheatsheet: this is a **non-fatal warning** (different from top-level fatal `error`).
→ Takopi `action`:
* `kind="warning"` (or `"note"`)
* `title="warning"`
* `message=item.message`
* `level="warning"`
* `phase="completed"`
* `ok=true` (because it’s informational) **or** omit `ok`
---
## Suggested “single-pass” translator logic (pseudocode)
This shows how to implement it without needing more than one pass or complicated buffering:
```python
final_answer = None
resume = None
did_emit_started = False
did_emit_completed = False
turn_index = 0
def emit(evt): on_event(evt)
for line in codex_jsonl_stream:
t = line["type"]
if t == "thread.started":
resume = {"engine": "codex", "value": line["thread_id"]}
# acquire per-thread lock here (for new sessions) before emitting started
emit({"type":"started","engine":"codex","resume":resume,"title":"Codex"})
did_emit_started = True
continue
if t == "turn.started":
emit({"type":"action","engine":"codex",
"action":{"id":f"turn_{turn_index}","kind":"turn","title":"turn started","detail":{}},
"phase":"started"})
continue
if t == "item.started" or t == "item.updated" or t == "item.completed":
item = line["item"]
item_type = item["type"]
item_id = item["id"]
if t == "item.completed" and item_type == "agent_message":
final_answer = item.get("text","")
continue
# map item_type -> kind/title/detail/ok
action_evt = map_item_to_action(item, phase=t.split(".")[1])
emit(action_evt)
continue
if t == "turn.completed":
emit({"type":"completed","engine":"codex","resume":resume,
"ok":True,"answer":final_answer or "",
"error":None,"usage":line.get("usage")})
did_emit_completed = True
continue
if t == "turn.failed":
emit({"type":"completed","engine":"codex","resume":resume,
"ok":False,"answer":final_answer or "",
"error":line["error"]["message"]})
did_emit_completed = True
continue
if t == "error": # fatal stream error
if not did_emit_completed:
emit({"type":"completed","engine":"codex","resume":resume,
"ok":False,"answer":final_answer or "",
"error":line.get("message")})
did_emit_completed = True
continue
# Optional: if stream ends without turn.completed/failed,
# emit completed with ok=False and error="unexpected EOF"
```
This design preserves the Takopi ordering/serialization principles: `started` happens as soon as resume token is known, actions stream in order, and exactly one `completed` closes the run.
---
## One practical note: what “completed” should mean
Even though you *learn* the final answer at `agent_message`, you generally want `completed` to be emitted at the **turn boundary** (`turn.completed` / `turn.failed`), because:
* you can attach usage (`turn.completed.usage`) only there,
* you guarantee `completed` is truly the last event,
* you still use `agent_message` as the authoritative answer payload.
That still matches your intent (“completed is when we get final answer”) because the answer comes from `agent_message`; you just *publish* it at the terminal boundary.
docs/runner/codex/exec-json-cheatsheet.md
+329
View File
@@ -0,0 +1,329 @@
# Codex `exec --json` event cheatsheet
`codex exec --json` writes **one JSON object per line** (JSONL) to stdout. Each
line is a top-level **thread event** with a `type` field.
Below: **all fields** for every line type plus a **full-line example** for each
shape that can be emitted.
## Top-level event lines (non-item)
### `thread.started`
Fields:
- `type`
- `thread_id`
Example:
```json
{"type":"thread.started","thread_id":"0199a213-81c0-7800-8aa1-bbab2a035a53"}
```
### `turn.started`
Fields:
- `type`
Example:
```json
{"type":"turn.started"}
```
### `turn.completed`
Fields:
- `type`
- `usage.input_tokens`
- `usage.cached_input_tokens`
- `usage.output_tokens`
Example:
```json
{"type":"turn.completed","usage":{"input_tokens":24763,"cached_input_tokens":24448,"output_tokens":122}}
```
### `turn.failed`
Fields:
- `type`
- `error.message`
Example:
```json
{"type":"turn.failed","error":{"message":"model response stream ended unexpectedly"}}
```
### `error`
Fields:
- `type`
- `message`
Example:
```json
{"type":"error","message":"stream error: broken pipe"}
```
## Item event lines (`item.*`)
Every item line includes:
- `type` (`item.started`, `item.updated`, or `item.completed`)
- `item.id`
- `item.type`
- fields for the specific `item.type` below
`item.id` is stable for the item; updates/completion reuse the same id.
### `agent_message` (only `item.completed`)
Fields:
- `item.text`
Example:
```json
{"type":"item.completed","item":{"id":"item_3","type":"agent_message","text":"Done. I updated the docs and added examples."}}
```
### `reasoning` (only `item.completed`, if enabled)
Fields:
- `item.text`
Example:
```json
{"type":"item.completed","item":{"id":"item_0","type":"reasoning","text":"**Scanning docs for exec JSON schema**"}}
```
### `command_execution` (`item.started` and `item.completed`)
Fields:
- `item.command`
- `item.aggregated_output`
- `item.exit_code` (null until completion)
- `item.status` (`in_progress`, `completed`, `failed`)
Example (started):
```json
{"type":"item.started","item":{"id":"item_1","type":"command_execution","command":"bash -lc ls","aggregated_output":"","exit_code":null,"status":"in_progress"}}
```
Example (completed, success):
```json
{"type":"item.completed","item":{"id":"item_1","type":"command_execution","command":"bash -lc ls","aggregated_output":"docs\nsrc\n","exit_code":0,"status":"completed"}}
```
Example (completed, failure):
```json
{"type":"item.completed","item":{"id":"item_2","type":"command_execution","command":"bash -lc false","aggregated_output":"","exit_code":1,"status":"failed"}}
```
Note: `aggregated_output` is truncated to **64 KiB**; truncated output ends with
`\n...(truncated)`.
### `file_change` (only `item.completed`)
Fields:
- `item.changes[].path`
- `item.changes[].kind` (`add`, `delete`, `update`)
- `item.status` (`completed`, `failed`)
Example:
```json
{"type":"item.completed","item":{"id":"item_4","type":"file_change","changes":[{"path":"docs/exec-json-cheatsheet.md","kind":"add"},{"path":"docs/exec.md","kind":"update"}],"status":"completed"}}
```
### `mcp_tool_call` (`item.started` and `item.completed`)
Fields:
- `item.server`
- `item.tool`
- `item.arguments` (JSON value; defaults to `null` if absent)
- `item.result` (object or `null`)
- `item.result.content` (array of MCP content blocks)
- `item.result.structured_content` (JSON value or `null`)
- `item.error` (object or `null`)
- `item.error.message` (if `error` is present)
- `item.status` (`in_progress`, `completed`, `failed`)
Example (started):
```json
{"type":"item.started","item":{"id":"item_5","type":"mcp_tool_call","server":"docs","tool":"search","arguments":{"q":"exec --json"},"result":null,"error":null,"status":"in_progress"}}
```
Example (completed, success):
```json
{"type":"item.completed","item":{"id":"item_5","type":"mcp_tool_call","server":"docs","tool":"search","arguments":{"q":"exec --json"},"result":{"content":[{"type":"text","text":"Found 3 matches.","annotations":{"audience":["assistant"],"lastModified":"2025-01-01T00:00:00Z","priority":0.5}}],"structured_content":{"matches":3}},"error":null,"status":"completed"}}
```
Example (completed, failure):
```json
{"type":"item.completed","item":{"id":"item_6","type":"mcp_tool_call","server":"docs","tool":"search","arguments":{"q":"exec --json"},"result":null,"error":{"message":"tool timeout"},"status":"failed"}}
```
### `web_search` (only `item.completed`)
Fields:
- `item.query`
Example:
```json
{"type":"item.completed","item":{"id":"item_7","type":"web_search","query":"codex exec --json schema"}}
```
### `todo_list` (`item.started`, `item.updated`, and `item.completed`)
Fields:
- `item.items[].text`
- `item.items[].completed`
Example (started):
```json
{"type":"item.started","item":{"id":"item_8","type":"todo_list","items":[{"text":"Scan docs","completed":false},{"text":"Write cheatsheet","completed":false}]}}
```
Example (updated):
```json
{"type":"item.updated","item":{"id":"item_8","type":"todo_list","items":[{"text":"Scan docs","completed":true},{"text":"Write cheatsheet","completed":false}]}}
```
Example (completed):
```json
{"type":"item.completed","item":{"id":"item_8","type":"todo_list","items":[{"text":"Scan docs","completed":true},{"text":"Write cheatsheet","completed":true}]}}
```
### `error` (non-fatal warning as an item; only `item.completed`)
Fields:
- `item.message`
Example:
```json
{"type":"item.completed","item":{"id":"item_9","type":"error","message":"command output truncated"}}
```
## MCP content block shapes (`mcp_tool_call.result.content`)
`result.content` is an array of **content blocks**. Each block is one of the
types below; all optional fields may appear depending on the server.
### Text content
Fields:
- `type`
- `text`
- `annotations.audience` (optional)
- `annotations.lastModified` (optional)
- `annotations.priority` (optional)
Example block:
```json
{"type":"text","text":"Hello","annotations":{"audience":["assistant"],"lastModified":"2025-01-01T00:00:00Z","priority":0.5}}
```
### Image content
Fields:
- `type`
- `data` (base64)
- `mimeType`
- `annotations.*` (same as above, optional)
Example block:
```json
{"type":"image","data":"<base64>","mimeType":"image/png","annotations":{"audience":["assistant"]}}
```
### Audio content
Fields:
- `type`
- `data` (base64)
- `mimeType`
- `annotations.*` (optional)
Example block:
```json
{"type":"audio","data":"<base64>","mimeType":"audio/wav","annotations":{"audience":["assistant"]}}
```
### Resource link
Fields:
- `type`
- `name`
- `uri`
- `description` (optional)
- `mimeType` (optional)
- `size` (optional)
- `title` (optional)
- `annotations.*` (optional)
Example block:
```json
{"type":"resource_link","name":"docs/exec.md","uri":"file:///repo/docs/exec.md","description":"Exec docs","mimeType":"text/markdown","size":1234,"title":"exec.md","annotations":{"audience":["assistant"]}}
```
### Embedded resource
Fields:
- `type`
- `resource` (either text or blob contents)
- `annotations.*` (optional)
Example block (embedded text):
```json
{"type":"resource","resource":{"uri":"file:///repo/README.md","text":"Hello","mimeType":"text/markdown"},"annotations":{"audience":["assistant"]}}
```
Example block (embedded blob):
```json
{"type":"resource","resource":{"uri":"file:///repo/image.png","blob":"<base64>","mimeType":"image/png"},"annotations":{"audience":["assistant"]}}
```
## Consumer considerations (rendering + success/failure)
Use this section to decide what to surface to end users vs. what to treat as
machine-only metadata.
### What to render for users
- **Final answer:** render `item.completed` where `item.type = "agent_message"` as
the main response.
- **Progress updates (optional):**
- `item.completed` with `item.type = "reasoning"` can be shown as brief
activity breadcrumbs (only if you want to expose reasoning summaries).
- `item.started` / `item.completed` with `item.type = "command_execution"` can
be shown as “running command …” status lines without printing full output.
- `item.completed` with `item.type = "file_change"` can be rendered as a list
of changed paths and kinds (add/update/delete).
- `item.*` with `item.type = "todo_list"` can be shown as a progress checklist.
- **Errors:** render `type = "error"` and `item.type = "error"` as user-visible
warnings or failures.
### Fields you can safely skip for UX
- `command_execution.aggregated_output` is often noisy; many consumers omit or
truncate it, and rely on `command_execution.status` + `exit_code` instead.
- `mcp_tool_call.result.content` can be large and tool-specific; consider showing
only high-level status unless you know the tool’s schema.
- `usage` fields (`turn.completed.usage.*`) are typically telemetry-only.
### Success and failure signals
- **Turn success:** `type = "turn.completed"` indicates overall success.
- **Turn failure:** `type = "turn.failed"` with `error.message` indicates failure.
- **Item success/failure:** use `item.status` on the item payload:
- `command_execution.status`: `completed` = success, `failed` = failure.
- `file_change.status`: `completed` = patch applied, `failed` = patch failed.
- `mcp_tool_call.status`: `completed` = tool succeeded, `failed` = tool failed.
- **Fatal stream errors:** `type = "error"` means the JSONL stream itself hit an
unrecoverable error.
### Suggested minimal rendering
If you want a compact UI, the following is usually enough:
- Thread/turn lifecycle: `thread.started`, `turn.started`, `turn.completed` or
`turn.failed`
- Final answer: `item.completed` with `item.type = "agent_message"`
- Optional progress: `item.started` / `item.completed` for `command_execution`
and `file_change`
pyproject.toml
+3 -5
View File
@@ -1,11 +1,11 @@
[project]
name = "takopi"
authors = [{name = "banteg"}]
version = "0.1.0"
version = "0.2.0"
description = "Run OpenAI Codex CLI with Telegram as the human-in-the-loop interface."
readme = "readme.md"
license = { file = "LICENSE" }
requires-python = ">=3.12"
requires-python = ">=3.14"
dependencies = [
"anyio>=4.12.0",
"httpx>=0.28.1",
@@ -17,8 +17,6 @@ dependencies = [
classifiers = [
"License :: OSI Approved :: MIT License",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.12",
"Programming Language :: Python :: 3.13",
"Programming Language :: Python :: 3.14",
"Programming Language :: Python :: 3 :: Only",
"Operating System :: OS Independent",
@@ -30,7 +28,7 @@ Repository = "https://github.com/banteg/takopi"
Issues = "https://github.com/banteg/takopi/issues"
[project.scripts]
takopi = "takopi.exec_bridge:main"
takopi = "takopi.cli:main"
[build-system]
requires = ["uv_build>=0.9.18,<0.10.0"]
readme.md
+22 -16
View File
@@ -6,17 +6,17 @@ A Telegram bot that bridges messages to [Codex](https://github.com/openai/codex)
## Features
- **Stateless Resume**: No database required—sessions are resumed via `resume: <uuid>` lines embedded in messages
- **Stateless Resume**: No database required—sessions are resumed via `` `codex resume <token>` `` lines embedded in messages
- **Progress Updates**: Real-time progress edits showing commands, tools, and elapsed time
- **Markdown Rendering**: Full Telegram-compatible markdown with entity support
- **Concurrency**: Handles multiple conversations with per-session serialization
- **Token Redaction**: Automatically redacts Telegram tokens from logs
- **Concurrency**: Parallel runs across threads with per-session serialization
## Quick Start
### Prerequisites
- [uv](https://github.com/astral-sh/uv) package manager
- Python 3.14+
- Codex CLI on PATH
### Installation
@@ -37,13 +37,21 @@ uvx takopi
### Configuration
Create `~/.codex/takopi.toml` (or `.codex/takopi.toml` for a repo-local config):
Create `~/.takopi/takopi.toml` (or `.takopi/takopi.toml` for a repo-local config):
```toml
bot_token = "123456789:ABCdefGHIjklMNOpqrsTUVwxyz"
chat_id = 123456789
[codex]
# Optional: Codex profile name (defined in ~/.codex/config.toml)
profile = "takopi"
# Optional: extra args passed before `codex exec`
extra_args = ["-c", "notify=[]"]
```
Engine-specific settings live under a table named after the engine id (e.g. `[codex]`).
| Key | Description |
|-----|-------------|
| `bot_token` | Telegram Bot API token from [@BotFather](https://t.me/BotFather) |
@@ -60,11 +68,7 @@ Create a Codex profile in `~/.codex/config.toml`:
model = "gpt-5.2-codex"
```
Then run takopi with:
```bash
takopi --profile takopi
```
Then set `profile = "takopi"` under `[codex]` in `~/.takopi/takopi.toml`.
### Options
@@ -72,7 +76,8 @@ takopi --profile takopi
|------|---------|-------------|
| `--final-notify` / `--no-final-notify` | `--final-notify` | Send final response as new message (vs. edit) |
| `--debug` / `--no-debug` | `--no-debug` | Enable verbose logging |
| `--profile NAME` | (codex default) | Codex profile name |
| `--engine ID` | `codex` | Engine backend id |
| `--engine-option KEY=VALUE` | | Engine-specific override (repeatable) |
| `--version` | | Show the version and exit |
## Usage
@@ -83,15 +88,15 @@ Send any message to your bot. The bridge will:
1. Send a silent progress message
2. Stream events from `codex exec`
3. Update progress every ~2 seconds
4. Send final response with session ID
3. Update progress every ~1 second
4. Send final response with a resume token line
### Resume a Session
Reply to a bot message (containing `resume: <uuid>`), or include the resume line in your message:
Reply to a bot message (containing `` `codex resume <token>` ``), or include the resume line in your message:
```
resume: `019b66fc-64c2-7a71-81cd-081c504cfeb2`
`codex resume 019b66fc-64c2-7a71-81cd-081c504cfeb2`
```
### Cancel a Run
@@ -101,13 +106,14 @@ Reply to a progress message with `/cancel` to stop the running execution.
## Notes
- **Startup**: Pending updates are drained (ignored) on startup
- **Progress**: Updates are throttled to ~2s intervals, sent silently
- **Progress**: Updates are throttled to ~1s intervals, sent silently
- **Queueing**: Messages for the same thread queue behind the active run without consuming extra concurrency slots
- **Filtering**: Only accepts messages where chat ID equals sender ID and matches `chat_id`
- **Single instance**: Run exactly one instance per bot token—multiple instances will race for updates
## Development
See [`developing.md`](developing.md).
See [`developing.md`](developing.md) and `specification.md` for architecture and behavior details.
## License
specification.md
+91 -70
View File
@@ -49,7 +49,7 @@ This is a normative spec using **MUST / SHOULD / MAY** language. Sections labele
**Domain Model (Takopi-owned)**
- Defines: `ResumeToken`, `RunResult`, `TakopiEvent`, `Action`.
- Defines: `ResumeToken`, `TakopiEvent`, `Action` (including the terminal `completed` event).
- No Telegram, no subprocess, no engine JSON.
**Runner Interface (Takopi-owned)**
@@ -80,9 +80,9 @@ This is a normative spec using **MUST / SHOULD / MAY** language. Sections labele
Recommended module layout (single-word filenames, clean layering):
- `takopi/model.py`
Domain types: events, actions, resume token, run result.
Domain types: events, actions, resume token.
- `takopi/runner.py`
Runner protocol + shared runner utilities (e.g., `EventQueue` if retained).
Runner protocol.
- `takopi/runners/codex.py`
Codex runner implementation.
- `takopi/runners/mock.py`
@@ -164,63 +164,54 @@ Runners are responsible for producing well-formed Takopi events. Downstream cons
Takopi MUST support the following event types:
1. `session.started`
2. `action.started`
3. `action.completed`
4. `log`
5. `error`
1. `started`
2. `action`
3. `completed`
### 5.3 Required fields by event type
#### 5.3.1 `session.started`
#### 5.3.1 `started`
Required:
- `type: "session.started"`
- `type: "started"`
- `engine: EngineId`
- `resume: ResumeToken`
Optional:
- `title: str` (human-readable session/agent label)
- `meta: dict` (debug/diagnostic payloads)
#### 5.3.2 `action.started`
#### 5.3.2 `action`
Required:
- `type: "action.started"`
- `type: "action"`
- `engine: EngineId`
- `action: Action`
#### 5.3.3 `action.completed`
Required:
- `type: "action.completed"`
- `engine: EngineId`
- `action: Action`
- `ok: bool` (success/failure of the action)
#### 5.3.4 `log`
Required:
- `type: "log"`
- `engine: EngineId`
- `message: str`
- `phase: "started" | "updated" | "completed"`
Optional:
- `level: "debug" | "info" | "warning" | "error"` (default: `"info"`)
- `ok: bool` (typically present when `phase="completed"`)
- `message: str` (freeform status/warning text)
- `level: "debug" | "info" | "warning" | "error"`
#### 5.3.5 `error`
#### 5.3.3 `completed`
Required:
- `type: "error"`
- `type: "completed"`
- `engine: EngineId`
- `message: str`
- `ok: bool` (success/failure of the run)
- `answer: str` (final assistant response text; may be empty)
Optional:
- `detail: str` (stack trace / stderr tail)
- `resume: ResumeToken` (final resume token for the run; new or existing, if known)
- `error: str | None` (fatal error message, if any)
- `usage: dict` (engine usage/telemetry, if provided)
### 5.4 Action schema (MUST, per your Decision #4)
@@ -245,6 +236,10 @@ Action kinds SHOULD be from a stable set (extensible):
- `file_change`
- `web_search`
- `note`
- `turn`
- `warning`
- `telemetry`
- `note`
Runners MAY include additional kinds, but renderers MAY treat unknown kinds as `note`.
@@ -252,6 +247,8 @@ The `detail` dict is **freeform per runner**; no per-kind schema is enforced. Re
The `ok` field semantics are **runner-defined**. For example, a runner MAY treat `grep` exit code 1 (no match) as `ok=True` if contextually appropriate.
**User-visible warnings and errors:** runners SHOULD surface these as `action` events with `phase="completed"` (typically `kind="warning"` or `kind="note"`) and `ok=False`, rather than introducing additional event types.
------
## 6. Runner interface and concurrency semantics
@@ -262,12 +259,11 @@ The `ok` field semantics are **runner-defined**. For example, a runner MAY treat
class Runner(Protocol):
engine: str
async def run(
def run(
self,
prompt: str,
resume: ResumeToken | None,
on_event: Callable[[TakopiEvent], None | Awaitable[None]],
) -> RunResult: ...
) -> AsyncIterator[TakopiEvent]: ...
```
### 6.2 Per-thread serialization (MUST; core invariant)
@@ -276,42 +272,52 @@ class Runner(Protocol):
- Parallel runs are allowed only if they target **different** threads.
- Runs targeting the same thread MUST be queued and executed sequentially.
- If a run attempts to acquire the per-thread lock while another run holds it, the run MUST **queue indefinitely** until the lock is released.
- This invariant MUST be enforced by the runner implementation (even if used outside the bridge).
**Critical requirement for new sessions:**
If `resume is None`, the runner MUST acquire the per-thread lock **as soon as the new thread's ResumeToken becomes known**, and MUST do so **before emitting `session.started`** to downstream consumers.
If `resume is None`, the runner MUST acquire the per-thread lock **as soon as the new thread's ResumeToken becomes known**, and MUST do so **before emitting `started`** to downstream consumers.
This prevents:
- a second run resuming the thread while the original "new session" run is still active
- history corruption due to concurrent engine operations
**Codex note (non-normative):**
For Codex, the resume token typically arrives as the first NDJSON event within ~1–2 seconds. If the subprocess exits before a resume token is observed, no `session.started` can be emitted and the bridge reports an error without a resume line.
**Bridge note (non-normative):**
The bridge may enforce FIFO scheduling per thread to avoid emitting multiple progress messages for the same thread while a run is already in-flight.
### 6.3 RunResult (MUST)
**Codex note (non-normative):**
Codex emits `thread.started` (with `thread_id`) before any `turn.*` / `item.*` events for both new and resumed runs. Codex MAY emit top-level warning `error` lines (e.g., config warnings) before `thread.started`; the Codex runner translates these warnings into `action` events with `phase="completed"` and yields them in the same order as received (so `started` is not guaranteed to be the first yielded event). If the subprocess exits before `thread.started` is observed, no `started` can be emitted and the bridge reports an error without a resume line.
Codex also emits exactly one `agent_message`/`assistant_message` per turn; the runner uses that message text as `completed.answer`.
### 6.3 Run completion event (MUST)
```python
@dataclass(frozen=True, slots=True)
class RunResult:
resume: ResumeToken # final resume token for the run (new or existing)
answer: str # final assistant response text (may be empty on failure)
class CompletedEvent:
type: Literal["completed"]
engine: EngineId
ok: bool # success/failure of the run
resume: ResumeToken | None = None # final resume token for the run (new or existing, if known)
answer: str # final assistant response text (may be empty)
```
`completed` MUST be the final event of a successful run.
### 6.4 Event delivery semantics (MUST)
Event ordering is significant. The system MUST ensure:
- Events are delivered to `on_event` in the same order they are produced by the runner.
- Events are yielded to the consumer in the same order they are produced by the runner.
- Event delivery MUST NOT spawn unbounded background tasks per event.
- If `on_event` raises an exception, the runner MUST abort the run.
- If the consumer stops iteration early (break/cancel/exception), the runner MUST abort the run (best-effort) and release any held resources.
### 6.5 Crash and error handling
If the runner subprocess crashes or exits uncleanly:
- The bridge MUST publish an error status message.
- If `session.started` was received, the bridge MUST include the resume line in the error message.
- If `started` was received, the bridge MUST include the resume line in the error message.
------
@@ -322,7 +328,6 @@ If the runner subprocess crashes or exits uncleanly:
The bridge MUST:
- Poll Telegram updates.
- Execute at most **16 active runs** concurrently across all threads.
- Resolve resume token (from message text or reply target).
- Start runner execution with appropriate cancellation support.
- Maintain progress rendering and Telegram edits (rate-limited).
@@ -332,9 +337,23 @@ The bridge MUST:
**Queuing behavior:**
- Multiple prompts to the same thread are queued and executed sequentially.
- Prompts queued behind an in-flight run MUST NOT count toward the **16 active runs** limit.
- There is no queue depth limit; all prompts are accepted.
### 7.1.1 Scheduling algorithm (MUST)
The bridge MUST implement per-thread FIFO scheduling in a way that does not require spawning one task per queued job.
**Definitions:**
- `ThreadKey := f"{resume.engine}:{resume.value}"`
- `Job := (chat_id, user_msg_id, text, resume: ResumeToken | None)`
**Required behavior:**
- For `resume != None`, the bridge MUST enqueue the job into `pending_by_thread[ThreadKey]` and ensure exactly one worker drains that queue sequentially.
- If a run starts with `resume == None` but later emits `started(resume=token)`, the bridge MUST treat that run as the in-flight job for `ThreadKey(token)` for scheduling purposes until it completes.
- A thread worker MUST exit when its queue is empty; the bridge SHOULD avoid retaining per-thread state for inactive threads.
The bridge MUST NOT:
- parse engine-native events
@@ -350,10 +369,10 @@ The bridge MUST NOT:
The progress renderer and/or final message MUST include the canonical resume line once known:
- If `session.started` has been received, the progress view SHOULD include the resume line.
- If `started` has been received, the progress view SHOULD include the resume line.
- The final message MUST include the resume line.
**Important:** because the resume line may appear during progress updates, runner-level locking for new sessions (§6.2) is REQUIRED.
**Important:** because the resume line may appear during progress updates, the bridge MUST treat `started` as the point at which the thread key becomes known for scheduling and cancellation routing.
### 7.4 Cancellation `/cancel`
@@ -399,9 +418,10 @@ The progress renderer SHOULD maintain:
- session title
- current running actions and their latest summaries
- completed actions and status
- latest log/error lines (bounded tail)
- resume token if known
If the runner emits multiple `action` events for the same `Action.id` while it is still running (e.g., repeated `phase="started"` or `phase="updated"`), the progress renderer SHOULD treat these as updates and collapse them into a single line (replacing the prior running line rather than appending a new one).
### 8.3 Final rendering
Final output MUST include:
@@ -436,25 +456,27 @@ The architecture SHOULD keep this future change localized to a `RunnerRegistry`
### 10.1 Test categories (MUST)
1. **Runner contract tests**
- Emits exactly one `session.started`
- Emits exactly one `started`
- All actions have required fields and stable IDs
- `RunResult.resume` matches session started token
- `completed.resume` matches started token (when present)
- Event ordering is preserved
- `ok` semantics match intended behavior
2. **Per-thread serialization test (critical)**
- Start new session run (resume=None) that emits `session.started` then blocks
- Attempt second run using that resume token before first completes
- Assert second run does not enter execution until first finishes
3. **Bridge progress throttling tests**
2. **Runner serialization tests (critical)**
- Serializes concurrent runs for the same `ResumeToken`
- For `resume=None`, acquires per-thread lock once the token is known (before emitting `started`)
3. **Bridge per-thread scheduling tests (critical)**
- Enqueue two prompts for the same `ResumeToken`
- Assert the bridge does not start the second run until the first completes
4. **Bridge progress throttling tests**
- Edits no more frequently than configured interval
- No edits without changes
- Truncation preserves resume line
4. **Cancellation tests**
5. **Cancellation tests**
- `/cancel` terminates run
- “cancelled” status produced
- resume line included if known
5. **Renderer formatting tests**
- Correct rendering of actions, errors, logs
6. **Renderer formatting tests**
- Correct rendering of actions
- Stable formatting under event sequences
### 10.2 Test tooling guidelines (SHOULD)
@@ -496,10 +518,9 @@ To reduce friction adding new runners, v0.2.0 SHOULD treat engine IDs as strings
- Telegram-only bridge with progress edits + cancellation
- Recommended module split into one-word modules
- Clarify: `ok` semantics are runner-defined, `detail` is freeform
- Clarify: 16 concurrent runs limit, indefinite queue per thread
- Clarify: bridge queues per thread (FIFO)
- Clarify: SIGTERM for cancellation, `/cancel` ignores accompanying text
- Clarify: truncation preserves head + resume line
- Clarify: log level defaults to `info`, callback errors abort run
- Clarify: crash publishes error with resume if known
------
@@ -511,14 +532,14 @@ To reduce friction adding new runners, v0.2.0 SHOULD treat engine IDs as strings
- none in message, none in reply → `resume=None`
3. Bridge sends a progress message: “Running…”
4. Runner starts and emits:
- `session.started(engine="codex", resume={engine:"codex", value:"<uuid>"})`
- `action.started(id="1", kind="command", title="pytest", detail={...})`
- `action.completed(id="1", ok=True, ...)`
- `log("All tests passed")`
- `started(engine="codex", resume={engine:"codex", value:"<uuid>"})`
- `action(id="1", kind="command", title="pytest", detail={...}, phase="started")`
- `action(id="1", ok=True, phase="completed", ...)`
- `completed(resume=..., ok=True, answer="...")`
5. Progress renderer now includes resume line:
- ``codex resume <uuid>``
6. User replies to progress message with follow-up prompt.
7. Bridge extracts resume via runner, chooses same thread, runner queues it behind the in-flight run if still active.
7. Bridge extracts resume via runner, chooses same thread, and queues it behind the in-flight run if still active.
8. Final message includes:
- “done”
- final answer
src/takopi/__init__.py
+1 -1
View File
@@ -1 +1 @@
__version__ = "0.1.0"
__version__ = "0.2.0"
src/takopi/bridge.py
+805
View File
@@ -0,0 +1,805 @@
"""Telegram bridge orchestration for running a single runner and streaming progress."""
from __future__ import annotations
import logging
import re
import time
import inspect
from collections import deque
from collections.abc import AsyncIterator, Awaitable, Callable
from dataclasses import dataclass, field
from typing import Any
import anyio
from .markdown import TELEGRAM_MARKDOWN_LIMIT, prepare_telegram
from .model import CompletedEvent, ResumeToken, StartedEvent, TakopiEvent
from .render import ExecProgressRenderer
from .runner import Runner
from .telegram import BotClient
logger = logging.getLogger(__name__)
def _resolve_resume(
runner: Runner, text: str | None, reply_text: str | None
) -> ResumeToken | None:
return runner.extract_resume(text) or runner.extract_resume(reply_text)
def _is_cancel_command(text: str) -> bool:
stripped = text.strip()
if not stripped:
return False
command = stripped.split(maxsplit=1)[0]
return command == "/cancel" or command.startswith("/cancel@")
_RESUME_COMMAND_RE = re.compile(
r"(?im)^\s*`?(?P<engine>[a-z0-9_-]+)\s+resume\s+(?P<token>(?=[^`\s]*\d)[^`\s]+)`?\s*$"
)
def _resume_attempt(text: str | None) -> tuple[bool, str | None]:
if not text:
return False, None
match = _RESUME_COMMAND_RE.search(text)
if match:
return True, match.group("engine").lower()
return False, None
def _resume_warning_text(engine_hint: str | None, current_engine: str) -> str:
if engine_hint and engine_hint.lower() != current_engine.lower():
return (
f"That looks like a {engine_hint} resume command, but this bot is running "
f"{current_engine}. Starting a new thread."
)
return "Couldn't parse a resume command; starting a new thread."
def _strip_resume_lines(text: str, *, is_resume_line: Callable[[str], bool]) -> str:
stripped_lines: list[str] = []
for line in text.splitlines():
if is_resume_line(line) or _RESUME_COMMAND_RE.match(line):
continue
stripped_lines.append(line)
prompt = "\n".join(stripped_lines).strip()
return prompt or "continue"
async def _send_resume_warning(
bot: BotClient,
chat_id: int,
user_msg_id: int,
engine_hint: str | None,
current_engine: str,
) -> None:
await bot.send_message(
chat_id=chat_id,
text=_resume_warning_text(engine_hint, current_engine),
reply_to_message_id=user_msg_id,
disable_notification=True,
)
PROGRESS_EDIT_EVERY_S = 1.0
async def _send_or_edit_markdown(
bot: BotClient,
*,
chat_id: int,
text: str,
edit_message_id: int | None = None,
reply_to_message_id: int | None = None,
disable_notification: bool = False,
limit: int = TELEGRAM_MARKDOWN_LIMIT,
is_resume_line: Callable[[str], bool] | None = None,
prepared: tuple[str, list[dict[str, Any]] | None] | None = None,
) -> tuple[dict[str, Any] | None, bool]:
if prepared is None:
rendered, entities = prepare_telegram(
text, limit=limit, is_resume_line=is_resume_line
)
else:
rendered, entities = prepared
if edit_message_id is not None:
edited = await bot.edit_message_text(
chat_id=chat_id,
message_id=edit_message_id,
text=rendered,
entities=entities,
)
if edited is not None:
return (edited, True)
return (
await bot.send_message(
chat_id=chat_id,
text=rendered,
entities=entities,
reply_to_message_id=reply_to_message_id,
disable_notification=disable_notification,
),
False,
)
class ProgressEdits:
def __init__(
self,
*,
bot: BotClient,
chat_id: int,
progress_id: int | None,
renderer: ExecProgressRenderer,
started_at: float,
progress_edit_every: float,
clock: Callable[[], float],
sleep: Callable[[float], Awaitable[None]],
limit: int,
last_edit_at: float,
last_rendered: str | None,
is_resume_line: Callable[[str], bool],
) -> None:
self.bot = bot
self.chat_id = chat_id
self.progress_id = progress_id
self.renderer = renderer
self.started_at = started_at
self.progress_edit_every = progress_edit_every
self.clock = clock
self.sleep = sleep
self.limit = limit
self.last_edit_at = last_edit_at
self.last_rendered = last_rendered
self.is_resume_line = is_resume_line
self._event_seq = 0
self._published_seq = 0
self.wakeup = anyio.Event()
async def _wait_for_wakeup(self) -> None:
await self.wakeup.wait()
self.wakeup = anyio.Event()
async def run(self) -> None:
if self.progress_id is None:
return
while True:
await self._wait_for_wakeup()
while self._published_seq < self._event_seq:
await self.sleep(
max(
0.0,
self.last_edit_at + self.progress_edit_every - self.clock(),
)
)
seq_at_render = self._event_seq
now = self.clock()
md = self.renderer.render_progress(now - self.started_at)
rendered, entities = prepare_telegram(
md, limit=self.limit, is_resume_line=self.is_resume_line
)
if rendered != self.last_rendered:
logger.debug(
"[progress] edit message_id=%s md=%s", self.progress_id, md
)
self.last_edit_at = now
edited = await self.bot.edit_message_text(
chat_id=self.chat_id,
message_id=self.progress_id,
text=rendered,
entities=entities,
)
if edited is not None:
self.last_rendered = rendered
self._published_seq = seq_at_render
async def on_event(self, evt: TakopiEvent) -> None:
if not self.renderer.note_event(evt):
return
if self.progress_id is None:
return
self._event_seq += 1
self.wakeup.set()
@dataclass(frozen=True)
class BridgeConfig:
bot: BotClient
runner: Runner
chat_id: int
final_notify: bool
startup_msg: str
progress_edit_every: float = PROGRESS_EDIT_EVERY_S
@dataclass
class RunningTask:
resume: ResumeToken | None = None
resume_ready: anyio.Event = field(default_factory=anyio.Event)
cancel_requested: anyio.Event = field(default_factory=anyio.Event)
done: anyio.Event = field(default_factory=anyio.Event)
async def _send_startup(cfg: BridgeConfig) -> None:
logger.debug("[startup] message: %s", cfg.startup_msg)
sent = await cfg.bot.send_message(chat_id=cfg.chat_id, text=cfg.startup_msg)
if sent is not None:
logger.info("[startup] sent startup message to chat_id=%s", cfg.chat_id)
async def _drain_backlog(cfg: BridgeConfig, offset: int | None) -> int | None:
drained = 0
while True:
updates = await cfg.bot.get_updates(
offset=offset, timeout_s=0, allowed_updates=["message"]
)
if updates is None:
logger.info("[startup] backlog drain failed")
return offset
logger.debug("[startup] backlog updates: %s", updates)
if not updates:
if drained:
logger.info("[startup] drained %s pending update(s)", drained)
return offset
offset = updates[-1]["update_id"] + 1
drained += len(updates)
async def handle_message(
cfg: BridgeConfig,
*,
chat_id: int,
user_msg_id: int,
text: str,
resume_token: ResumeToken | None,
running_tasks: dict[int, RunningTask] | None = None,
on_thread_known: Callable[[ResumeToken, anyio.Event], Awaitable[None]]
| None = None,
clock: Callable[[], float] = time.monotonic,
sleep: Callable[[float], Awaitable[None]] = anyio.sleep,
progress_edit_every: float = PROGRESS_EDIT_EVERY_S,
) -> None:
logger.debug(
"[handle] incoming chat_id=%s message_id=%s resume=%r text=%s",
chat_id,
user_msg_id,
resume_token,
text,
)
started_at = clock()
runner = cfg.runner
is_resume_line = runner.is_resume_line
runner_text = _strip_resume_lines(text, is_resume_line=is_resume_line)
progress_renderer = ExecProgressRenderer(
max_actions=5, resume_formatter=runner.format_resume
)
progress_id: int | None = None
last_edit_at = 0.0
last_rendered: str | None = None
initial_md = progress_renderer.render_progress(
0.0, label=f"working ({runner.engine})"
)
initial_rendered, initial_entities = prepare_telegram(
initial_md, limit=TELEGRAM_MARKDOWN_LIMIT, is_resume_line=is_resume_line
)
logger.debug(
"[progress] send reply_to=%s md=%s rendered=%s entities=%s",
user_msg_id,
initial_md,
initial_rendered,
initial_entities,
)
progress_msg = await cfg.bot.send_message(
chat_id=chat_id,
text=initial_rendered,
entities=initial_entities,
reply_to_message_id=user_msg_id,
disable_notification=True,
)
if progress_msg is not None:
progress_id = int(progress_msg["message_id"])
last_edit_at = clock()
last_rendered = initial_rendered
logger.debug("[progress] sent chat_id=%s message_id=%s", chat_id, progress_id)
edits = ProgressEdits(
bot=cfg.bot,
chat_id=chat_id,
progress_id=progress_id,
renderer=progress_renderer,
started_at=started_at,
progress_edit_every=progress_edit_every,
clock=clock,
sleep=sleep,
limit=TELEGRAM_MARKDOWN_LIMIT,
last_edit_at=last_edit_at,
last_rendered=last_rendered,
is_resume_line=is_resume_line,
)
cancel_exc_type = anyio.get_cancelled_exc_class()
cancelled = False
error: Exception | None = None
resume_token_value: ResumeToken | None = None
answer: str | None = None
run_ok: bool | None = None
run_error: str | None = None
running_task: RunningTask | None = None
if running_tasks is not None and progress_id is not None:
running_task = RunningTask()
running_tasks[progress_id] = running_task
edits_scope = anyio.CancelScope()
async def run_edits() -> None:
try:
with edits_scope:
await edits.run()
except cancel_exc_type:
# Edits are best-effort; cancellation should not bubble into the task group.
return
async with anyio.create_task_group() as tg:
if progress_id is not None:
tg.start_soon(run_edits)
async def run_exec() -> CompletedEvent | None:
nonlocal cancelled
cancel_flag = False
completed: CompletedEvent | None = None
async with anyio.create_task_group() as exec_tg:
async def run_runner() -> None:
nonlocal resume_token_value, completed, answer, run_ok, run_error
try:
async for evt in runner.run(runner_text, resume_token):
if isinstance(evt, StartedEvent):
resume_token_value = evt.resume
if (
running_task is not None
and running_task.resume is None
):
running_task.resume = resume_token_value
running_task.resume_ready.set()
if on_thread_known is not None:
await on_thread_known(
resume_token_value, running_task.done
)
elif isinstance(evt, CompletedEvent):
resume_token_value = evt.resume or resume_token_value
answer = evt.answer
run_ok = evt.ok
run_error = evt.error
completed = evt
await edits.on_event(evt)
finally:
exec_tg.cancel_scope.cancel()
async def wait_cancel() -> None:
nonlocal cancel_flag
if running_task is None:
return
await running_task.cancel_requested.wait()
cancel_flag = True
exec_tg.cancel_scope.cancel()
exec_tg.start_soon(run_runner)
if running_task is not None:
exec_tg.start_soon(wait_cancel)
if cancel_flag:
cancelled = True
return completed
try:
completed = await run_exec()
if completed is not None:
resume_token_value = completed.resume or resume_token_value
answer = completed.answer
run_ok = completed.ok
run_error = completed.error
except Exception as e:
error = e
finally:
if (
running_task is not None
and running_tasks is not None
and progress_id is not None
):
running_task.done.set()
running_tasks.pop(progress_id, None)
if not cancelled and error is None:
await anyio.sleep(0)
edits_scope.cancel()
if error is not None:
elapsed = clock() - started_at
if resume_token_value is None:
resume_token_value = progress_renderer.resume_token
progress_renderer.resume_token = resume_token_value
err_body = f"Error:\n{error}"
final_md = progress_renderer.render_final(elapsed, err_body, status="error")
logger.debug("[error] markdown: %s", final_md)
final_msg, edited = await _send_or_edit_markdown(
cfg.bot,
chat_id=chat_id,
text=final_md,
edit_message_id=progress_id,
reply_to_message_id=user_msg_id,
disable_notification=True,
limit=TELEGRAM_MARKDOWN_LIMIT,
is_resume_line=is_resume_line,
)
if final_msg is None:
return
if progress_id is not None and not edited:
logger.debug("[error] delete progress message_id=%s", progress_id)
await cfg.bot.delete_message(chat_id=chat_id, message_id=progress_id)
return
elapsed = clock() - started_at
if cancelled:
if resume_token_value is None:
resume_token_value = progress_renderer.resume_token
logger.info(
"[handle] cancelled resume=%s elapsed=%.1fs",
resume_token_value.value if resume_token_value else None,
elapsed,
)
progress_renderer.resume_token = resume_token_value
final_md = progress_renderer.render_progress(elapsed, label="`cancelled`")
final_msg, edited = await _send_or_edit_markdown(
cfg.bot,
chat_id=chat_id,
text=final_md,
edit_message_id=progress_id,
reply_to_message_id=user_msg_id,
disable_notification=True,
limit=TELEGRAM_MARKDOWN_LIMIT,
is_resume_line=is_resume_line,
)
if final_msg is None:
return
if progress_id is not None and not edited:
logger.debug("[cancel] delete progress message_id=%s", progress_id)
await cfg.bot.delete_message(chat_id=chat_id, message_id=progress_id)
return
if answer is None:
raise RuntimeError("runner finished without a completed event")
final_answer = answer
if run_ok is False and run_error:
if final_answer.strip():
final_answer = f"{final_answer}\n\nError:\n{run_error}"
else:
final_answer = f"Error:\n{run_error}"
status = (
"error" if run_ok is False else ("done" if final_answer.strip() else "error")
)
if resume_token_value is None:
resume_token_value = progress_renderer.resume_token
progress_renderer.resume_token = resume_token_value
final_md = progress_renderer.render_final(elapsed, final_answer, status=status)
logger.debug("[final] markdown: %s", final_md)
final_rendered, final_entities = prepare_telegram(
final_md, limit=TELEGRAM_MARKDOWN_LIMIT, is_resume_line=is_resume_line
)
can_edit_final = progress_id is not None and final_entities is not None
edit_message_id = None if cfg.final_notify or not can_edit_final else progress_id
if edit_message_id is None:
logger.debug(
"[final] send reply_to=%s rendered=%s entities=%s",
user_msg_id,
final_rendered,
final_entities,
)
else:
logger.debug(
"[final] edit message_id=%s rendered=%s entities=%s",
edit_message_id,
final_rendered,
final_entities,
)
final_msg, edited = await _send_or_edit_markdown(
cfg.bot,
chat_id=chat_id,
text=final_md,
edit_message_id=edit_message_id,
reply_to_message_id=user_msg_id,
disable_notification=False,
limit=TELEGRAM_MARKDOWN_LIMIT,
is_resume_line=is_resume_line,
prepared=(final_rendered, final_entities),
)
if final_msg is None:
return
if progress_id is not None and (edit_message_id is None or not edited):
logger.debug("[final] delete progress message_id=%s", progress_id)
await cfg.bot.delete_message(chat_id=chat_id, message_id=progress_id)
async def poll_updates(cfg: BridgeConfig):
offset: int | None = None
offset = await _drain_backlog(cfg, offset)
await _send_startup(cfg)
while True:
updates = await cfg.bot.get_updates(
offset=offset, timeout_s=50, allowed_updates=["message"]
)
if updates is None:
logger.info("[loop] getUpdates failed")
await anyio.sleep(2)
continue
logger.debug("[loop] updates: %s", updates)
for upd in updates:
offset = upd["update_id"] + 1
msg = upd["message"]
if "text" not in msg:
continue
if not (msg["chat"]["id"] == msg["from"]["id"] == cfg.chat_id):
continue
yield msg
async def _handle_cancel(
cfg: BridgeConfig,
msg: dict[str, Any],
running_tasks: dict[int, RunningTask],
) -> None:
chat_id = msg["chat"]["id"]
user_msg_id = msg["message_id"]
reply = msg.get("reply_to_message")
if not reply:
await cfg.bot.send_message(
chat_id=chat_id,
text="reply to the progress message to cancel.",
reply_to_message_id=user_msg_id,
)
return
progress_id = reply.get("message_id")
if progress_id is None:
await cfg.bot.send_message(
chat_id=chat_id,
text="nothing is currently running for that message.",
reply_to_message_id=user_msg_id,
)
return
running_task = running_tasks.get(int(progress_id))
if running_task is None:
await cfg.bot.send_message(
chat_id=chat_id,
text="nothing is currently running for that message.",
reply_to_message_id=user_msg_id,
)
return
logger.info("[cancel] cancelling progress_message_id=%s", progress_id)
running_task.cancel_requested.set()
async def _wait_for_resume(running_task: RunningTask) -> ResumeToken | None:
if running_task.resume is not None:
return running_task.resume
resume: ResumeToken | None = None
async with anyio.create_task_group() as tg:
async def wait_resume() -> None:
nonlocal resume
await running_task.resume_ready.wait()
resume = running_task.resume
tg.cancel_scope.cancel()
async def wait_done() -> None:
await running_task.done.wait()
tg.cancel_scope.cancel()
tg.start_soon(wait_resume)
tg.start_soon(wait_done)
return resume
async def _send_with_resume(
bot: BotClient,
enqueue: Callable[[int, int, str, ResumeToken], Awaitable[None] | None],
running_task: RunningTask,
chat_id: int,
user_msg_id: int,
text: str,
) -> None:
resume = await _wait_for_resume(running_task)
if resume is None:
await bot.send_message(
chat_id=chat_id,
text="resume token not ready yet; try replying to the final message.",
reply_to_message_id=user_msg_id,
disable_notification=True,
)
return
result = enqueue(chat_id, user_msg_id, text, resume)
if inspect.isawaitable(result):
await result
async def _run_main_loop(
cfg: BridgeConfig,
poller: Callable[[BridgeConfig], AsyncIterator[dict[str, Any]]] = poll_updates,
) -> None:
running_tasks: dict[int, RunningTask] = {}
try:
async with anyio.create_task_group() as tg:
scheduler_lock = anyio.Lock()
@dataclass(frozen=True, slots=True)
class ThreadJob:
chat_id: int
user_msg_id: int
text: str
resume_token: ResumeToken
pending_by_thread: dict[str, deque[ThreadJob]] = {}
active_threads: set[str] = set()
busy_until: dict[str, anyio.Event] = {}
def thread_key(token: ResumeToken) -> str:
return f"{token.engine}:{token.value}"
async def clear_busy(key: str, done: anyio.Event) -> None:
await done.wait()
async with scheduler_lock:
if busy_until.get(key) is done:
busy_until.pop(key, None)
async def note_thread_known(token: ResumeToken, done: anyio.Event) -> None:
key = thread_key(token)
async with scheduler_lock:
current = busy_until.get(key)
if current is None or current.is_set():
busy_until[key] = done
tg.start_soon(clear_busy, key, done)
async def run_job(
chat_id: int,
user_msg_id: int,
text: str,
resume_token: ResumeToken | None,
on_thread_known: Callable[[ResumeToken, anyio.Event], Awaitable[None]]
| None = None,
) -> None:
try:
await handle_message(
cfg,
chat_id=chat_id,
user_msg_id=user_msg_id,
text=text,
resume_token=resume_token,
running_tasks=running_tasks,
on_thread_known=on_thread_known,
progress_edit_every=cfg.progress_edit_every,
)
except Exception:
logger.exception("[handle] worker failed")
async def thread_worker(key: str) -> None:
try:
while True:
async with scheduler_lock:
done = busy_until.get(key)
queue = pending_by_thread.get(key)
if not queue:
pending_by_thread.pop(key, None)
active_threads.discard(key)
return
job = queue.popleft()
if done is not None and not done.is_set():
await done.wait()
await run_job(
job.chat_id,
job.user_msg_id,
job.text,
job.resume_token,
)
finally:
async with scheduler_lock:
active_threads.discard(key)
async def enqueue(
chat_id: int,
user_msg_id: int,
text: str,
resume_token: ResumeToken,
) -> None:
key = thread_key(resume_token)
async with scheduler_lock:
queue = pending_by_thread.get(key)
if queue is None:
queue = deque()
pending_by_thread[key] = queue
queue.append(
ThreadJob(
chat_id=chat_id,
user_msg_id=user_msg_id,
text=text,
resume_token=resume_token,
)
)
if key in active_threads:
return
active_threads.add(key)
tg.start_soon(thread_worker, key)
async for msg in poller(cfg):
text = msg["text"]
user_msg_id = msg["message_id"]
if _is_cancel_command(text):
tg.start_soon(_handle_cancel, cfg, msg, running_tasks)
continue
r = msg.get("reply_to_message") or {}
resume_token = _resolve_resume(cfg.runner, text, r.get("text"))
reply_id = r.get("message_id")
if resume_token is None and reply_id is not None:
running_task = running_tasks.get(int(reply_id))
if running_task is not None:
tg.start_soon(
_send_with_resume,
cfg.bot,
enqueue,
running_task,
msg["chat"]["id"],
user_msg_id,
text,
)
continue
if resume_token is None:
attempt_text, engine_text = _resume_attempt(text)
attempt_reply, engine_reply = _resume_attempt(r.get("text"))
attempt = attempt_text or attempt_reply
if attempt:
tg.start_soon(
_send_resume_warning,
cfg.bot,
msg["chat"]["id"],
user_msg_id,
engine_text or engine_reply,
str(cfg.runner.engine),
)
if resume_token is None:
tg.start_soon(
run_job,
msg["chat"]["id"],
user_msg_id,
text,
None,
note_thread_known,
)
else:
await enqueue(msg["chat"]["id"], user_msg_id, text, resume_token)
finally:
await cfg.bot.close()
src/takopi/cli.py
+138
View File
@@ -0,0 +1,138 @@
from __future__ import annotations
import os
from typing import Any
import anyio
import typer
from . import __version__
from .bridge import BridgeConfig, _run_main_loop
from .config import ConfigError, load_telegram_config
from .engines import (
EngineBackend,
get_backend,
get_engine_config,
list_backend_ids,
parse_engine_overrides,
)
from .logging import setup_logging
from .onboarding import check_setup, render_setup_guide
from .telegram import TelegramClient
def _print_version_and_exit() -> None:
typer.echo(__version__)
raise typer.Exit()
def _version_callback(value: bool) -> None:
if value:
_print_version_and_exit()
def _parse_bridge_config(
*,
final_notify: bool,
backend: EngineBackend,
engine_overrides: dict[str, Any],
) -> BridgeConfig:
startup_pwd = os.getcwd()
config, config_path = load_telegram_config()
try:
token = config["bot_token"]
except KeyError:
raise ConfigError(f"Missing key `bot_token` in {config_path}.") from None
if not isinstance(token, str) or not token.strip():
raise ConfigError(
f"Invalid `bot_token` in {config_path}; expected a non-empty string."
) from None
try:
chat_id_value = config["chat_id"]
except KeyError:
raise ConfigError(f"Missing key `chat_id` in {config_path}.") from None
if isinstance(chat_id_value, bool) or not isinstance(chat_id_value, int):
raise ConfigError(
f"Invalid `chat_id` in {config_path}; expected an integer."
) from None
chat_id = chat_id_value
engine_cfg = get_engine_config(config, backend.id, config_path)
startup_msg = backend.startup_message(startup_pwd)
bot = TelegramClient(token)
runner = backend.build_runner(engine_cfg, engine_overrides, config_path)
return BridgeConfig(
bot=bot,
runner=runner,
chat_id=chat_id,
final_notify=final_notify,
startup_msg=startup_msg,
)
def run(
version: bool = typer.Option(
False,
"--version",
help="Show the version and exit.",
callback=_version_callback,
is_eager=True,
),
final_notify: bool = typer.Option(
True,
"--final-notify/--no-final-notify",
help="Send the final response as a new message (not an edit).",
),
engine: str = typer.Option(
"codex",
"--engine",
help=f"Engine backend id ({', '.join(list_backend_ids())}).",
),
debug: bool = typer.Option(
False,
"--debug/--no-debug",
help="Log engine JSONL, Telegram requests, and rendered messages.",
),
engine_option: list[str] = typer.Option(
[],
"--engine-option",
"-E",
help="Engine-specific override in KEY=VALUE form (repeatable).",
),
) -> None:
setup_logging(debug=debug)
try:
backend = get_backend(engine)
except ConfigError as e:
typer.echo(str(e), err=True)
raise typer.Exit(code=1)
try:
overrides = parse_engine_overrides(engine_option)
except ConfigError as e:
typer.echo(str(e), err=True)
raise typer.Exit(code=1)
setup = check_setup(backend)
if not setup.ok:
render_setup_guide(setup)
raise typer.Exit(code=1)
try:
cfg = _parse_bridge_config(
final_notify=final_notify,
backend=backend,
engine_overrides=overrides,
)
except ConfigError as e:
typer.echo(str(e), err=True)
raise typer.Exit(code=1)
anyio.run(_run_main_loop, cfg)
def main() -> None:
typer.run(run)
if __name__ == "__main__":
main()
src/takopi/config.py
+38 -2
View File
@@ -3,8 +3,10 @@ from __future__ import annotations
import tomllib
from pathlib import Path
LOCAL_CONFIG_NAME = Path(".codex") / "takopi.toml"
HOME_CONFIG_PATH = Path.home() / ".codex" / "takopi.toml"
LOCAL_CONFIG_NAME = Path(".takopi") / "takopi.toml"
HOME_CONFIG_PATH = Path.home() / ".takopi" / "takopi.toml"
LEGACY_LOCAL_CONFIG_NAME = Path(".codex") / "takopi.toml"
LEGACY_HOME_CONFIG_PATH = Path.home() / ".codex" / "takopi.toml"
class ConfigError(RuntimeError):
@@ -18,6 +20,32 @@ def _config_candidates() -> list[Path]:
return candidates
def _legacy_candidates() -> list[Path]:
candidates = [Path.cwd() / LEGACY_LOCAL_CONFIG_NAME, LEGACY_HOME_CONFIG_PATH]
if candidates[0] == candidates[1]:
return [candidates[0]]
return candidates
def _maybe_migrate_legacy(legacy_path: Path, target_path: Path) -> None:
if target_path.exists():
if not target_path.is_file():
raise ConfigError(
f"Config path {target_path} exists but is not a file."
) from None
return
if not legacy_path.is_file():
return
try:
target_path.parent.mkdir(parents=True, exist_ok=True)
raw = legacy_path.read_text(encoding="utf-8")
target_path.write_text(raw, encoding="utf-8")
except OSError as e:
raise ConfigError(
f"Failed to migrate legacy config {legacy_path} to {target_path}: {e}"
) from e
def _read_config(cfg_path: Path) -> dict:
try:
raw = cfg_path.read_text(encoding="utf-8")
@@ -36,11 +64,19 @@ def load_telegram_config(path: str | Path | None = None) -> tuple[dict, Path]:
cfg_path = Path(path).expanduser()
return _read_config(cfg_path), cfg_path
for legacy, target in zip(_legacy_candidates(), _config_candidates(), strict=True):
_maybe_migrate_legacy(legacy, target)
candidates = _config_candidates()
for candidate in candidates:
if candidate.is_file():
return _read_config(candidate), candidate
legacy_candidates = _legacy_candidates()
for candidate in legacy_candidates:
if candidate.is_file():
return _read_config(candidate), candidate
if len(candidates) == 1:
raise ConfigError("Missing takopi config.")
raise ConfigError("Missing takopi config.")
src/takopi/engines.py
+137
View File
@@ -0,0 +1,137 @@
from __future__ import annotations
import shutil
from dataclasses import dataclass
from pathlib import Path
from typing import Any, Callable
from .config import ConfigError
from .runner import Runner
from .runners.codex import CodexRunner
EngineConfig = dict[str, Any]
EngineOverrides = dict[str, Any]
@dataclass(frozen=True, slots=True)
class SetupIssue:
title: str
lines: tuple[str, ...]
@dataclass(frozen=True, slots=True)
class EngineBackend:
id: str
display_name: str
check_setup: Callable[[EngineConfig, Path], list[SetupIssue]]
build_runner: Callable[[EngineConfig, EngineOverrides, Path], Runner]
startup_message: Callable[[str], str]
def _codex_check_setup(_config: EngineConfig, _config_path: Path) -> list[SetupIssue]:
if shutil.which("codex") is None:
return [
SetupIssue(
"Install the Codex CLI",
(" [dim]$[/] npm install -g @openai/codex",),
)
]
return []
def _codex_build_runner(
config: EngineConfig, overrides: EngineOverrides, config_path: Path
) -> Runner:
codex_cmd = shutil.which("codex")
if not codex_cmd:
raise ConfigError(
"codex not found on PATH. Install the Codex CLI with:\n"
" npm install -g @openai/codex\n"
" # or on macOS\n"
" brew install codex"
)
extra_args_value = config.get("extra_args")
if extra_args_value is None:
extra_args = ["-c", "notify=[]"]
elif isinstance(extra_args_value, list) and all(
isinstance(item, str) for item in extra_args_value
):
extra_args = list(extra_args_value)
else:
raise ConfigError(
f"Invalid `codex.extra_args` in {config_path}; expected a list of strings."
)
title = "Codex"
profile_value = config.get("profile")
if profile_value:
if not isinstance(profile_value, str):
raise ConfigError(
f"Invalid `codex.profile` in {config_path}; expected a string."
)
extra_args.extend(["--profile", profile_value])
title = profile_value
if overrides:
unknown = ", ".join(sorted(overrides))
raise ConfigError(f"Unknown codex override(s): {unknown}")
return CodexRunner(codex_cmd=codex_cmd, extra_args=extra_args, title=title)
def _codex_startup_message(cwd: str) -> str:
return f"codex is ready\npwd: {cwd}"
_ENGINE_BACKENDS: dict[str, EngineBackend] = {
"codex": EngineBackend(
id="codex",
display_name="Codex",
check_setup=_codex_check_setup,
build_runner=_codex_build_runner,
startup_message=_codex_startup_message,
),
}
def get_backend(engine_id: str) -> EngineBackend:
try:
return _ENGINE_BACKENDS[engine_id]
except KeyError as exc:
available = ", ".join(sorted(_ENGINE_BACKENDS))
raise ConfigError(
f"Unknown engine {engine_id!r}. Available: {available}."
) from exc
def list_backends() -> list[EngineBackend]:
return list(_ENGINE_BACKENDS.values())
def list_backend_ids() -> list[str]:
return sorted(_ENGINE_BACKENDS)
def parse_engine_overrides(options: list[str]) -> EngineOverrides:
overrides: EngineOverrides = {}
for raw in options:
key, sep, value = raw.partition("=")
if not sep:
raise ConfigError(f"Invalid --engine-option {raw!r}; expected KEY=VALUE.")
key = key.strip()
if not key:
raise ConfigError(f"Invalid --engine-option {raw!r}; expected KEY=VALUE.")
overrides[key] = value
return overrides
def get_engine_config(
config: dict[str, Any], engine_id: str, config_path: Path
) -> EngineConfig:
engine_cfg = config.get(engine_id) or {}
if not isinstance(engine_cfg, dict):
raise ConfigError(
f"Invalid `{engine_id}` config in {config_path}; expected a table."
)
return engine_cfg
src/takopi/exec_bridge.py
-895
View File
@@ -1,895 +0,0 @@
from __future__ import annotations
import inspect
import json
import logging
import os
import re
import shutil
import subprocess
import time
from collections import deque
from collections.abc import AsyncIterator, Awaitable, Callable
from contextlib import asynccontextmanager
from dataclasses import dataclass
from typing import Any
from weakref import WeakValueDictionary
import anyio
import typer
from anyio.abc import ByteReceiveStream, Process
from anyio.streams.text import TextReceiveStream
from . import __version__
from .config import ConfigError, load_telegram_config
from .exec_render import (
ExecProgressRenderer,
render_event_cli,
render_markdown,
)
from .logging import setup_logging
from .onboarding import check_setup, render_setup_guide
from .telegram import TelegramClient
logger = logging.getLogger(__name__)
UUID_PATTERN_TEXT = r"\b[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\b"
UUID_PATTERN = re.compile(UUID_PATTERN_TEXT, re.IGNORECASE)
RESUME_LINE = re.compile(
rf"^\s*resume\s*:\s*`?(?P<id>{UUID_PATTERN_TEXT})`?\s*$",
re.IGNORECASE | re.MULTILINE,
)
def _print_version_and_exit() -> None:
typer.echo(__version__)
raise typer.Exit()
def _version_callback(value: bool) -> None:
if value:
_print_version_and_exit()
def extract_session_id(text: str | None) -> str | None:
if not text:
return None
found: str | None = None
for match in RESUME_LINE.finditer(text):
found = match.group("id")
return found
def resolve_resume_session(text: str | None, reply_text: str | None) -> str | None:
return extract_session_id(text) or extract_session_id(reply_text)
async def _iter_text_lines(stream: ByteReceiveStream) -> AsyncIterator[str]:
text_stream = TextReceiveStream(stream, errors="replace")
buffer = ""
while True:
try:
chunk = await text_stream.receive()
except anyio.EndOfStream:
if buffer:
yield buffer
return
buffer += chunk
while True:
split_at = buffer.find("\n")
if split_at < 0:
break
line = buffer[: split_at + 1]
buffer = buffer[split_at + 1 :]
yield line
async def _drain_stderr(stderr: ByteReceiveStream, tail: deque[str]) -> None:
try:
async for line in _iter_text_lines(stderr):
logger.info("[codex][stderr] %s", line.rstrip())
tail.append(line)
except Exception as e:
logger.debug("[codex][stderr] drain error: %s", e)
async def _wait_for_process(proc: Process, timeout: float) -> bool:
with anyio.move_on_after(timeout) as scope:
await proc.wait()
return scope.cancel_called
@asynccontextmanager
async def manage_subprocess(*args, terminate_timeout: float = 2.0, **kwargs):
proc = await anyio.open_process(args, **kwargs)
try:
yield proc
finally:
if proc.returncode is None:
with anyio.CancelScope(shield=True):
try:
proc.terminate()
except ProcessLookupError:
pass
timed_out = await _wait_for_process(proc, terminate_timeout)
if timed_out:
logger.debug(
"[codex] terminate timed out pid=%s; leaving process to exit",
proc.pid,
)
TELEGRAM_MARKDOWN_LIMIT = 3500
PROGRESS_EDIT_EVERY_S = 2.0
def _clamp_tg_text(text: str, limit: int = TELEGRAM_MARKDOWN_LIMIT) -> str:
if len(text) <= limit:
return text
return text[: limit - 20] + "\n...(truncated)"
def truncate_for_telegram(text: str, limit: int) -> str:
"""
Truncate text to fit Telegram limits while preserving the trailing `resume: ...`
line (if present), otherwise preserving the last non-empty line.
"""
if len(text) <= limit:
return text
lines = text.splitlines()
tail_lines: list[str] | None = None
is_resume_tail = False
for i in range(len(lines) - 1, -1, -1):
line = lines[i]
if "resume" in line and UUID_PATTERN.search(line):
tail_lines = lines[i:]
is_resume_tail = True
break
if tail_lines is None:
for i in range(len(lines) - 1, -1, -1):
if lines[i].strip():
tail_lines = [lines[i]]
break
tail = "\n".join(tail_lines or []).strip("\n")
sep = "\n…\n"
max_tail = limit if is_resume_tail else (limit // 4)
tail = tail[-max_tail:] if max_tail > 0 else ""
head_budget = limit - len(sep) - len(tail)
if head_budget <= 0:
return tail[-limit:] if tail else text[:limit]
head = text[:head_budget].rstrip()
return (head + sep + tail)[:limit]
def prepare_telegram(md: str, *, limit: int) -> tuple[str, list[dict[str, Any]] | None]:
rendered, entities = render_markdown(md)
if len(rendered) > limit:
rendered = truncate_for_telegram(rendered, limit)
return rendered, None
return rendered, entities
async def _send_or_edit_markdown(
bot: TelegramClient,
*,
chat_id: int,
text: str,
edit_message_id: int | None = None,
reply_to_message_id: int | None = None,
disable_notification: bool = False,
limit: int = TELEGRAM_MARKDOWN_LIMIT,
) -> tuple[dict[str, Any] | None, bool]:
if edit_message_id is not None:
rendered, entities = prepare_telegram(text, limit=limit)
edited = await bot.edit_message_text(
chat_id=chat_id,
message_id=edit_message_id,
text=rendered,
entities=entities,
)
if edited is not None:
return (edited, True)
rendered, entities = prepare_telegram(text, limit=limit)
return (
await bot.send_message(
chat_id=chat_id,
text=rendered,
entities=entities,
reply_to_message_id=reply_to_message_id,
disable_notification=disable_notification,
),
False,
)
EventCallback = Callable[[dict[str, Any]], Awaitable[None] | None]
class ProgressEdits:
def __init__(
self,
*,
bot: TelegramClient,
chat_id: int,
progress_id: int | None,
renderer: ExecProgressRenderer,
started_at: float,
progress_edit_every: float,
clock: Callable[[], float],
sleep: Callable[[float], Awaitable[None]],
limit: int,
last_edit_at: float,
last_rendered: str | None,
) -> None:
self.bot = bot
self.chat_id = chat_id
self.progress_id = progress_id
self.renderer = renderer
self.started_at = started_at
self.progress_edit_every = progress_edit_every
self.clock = clock
self.sleep = sleep
self.limit = limit
self.last_edit_at = last_edit_at
self.last_rendered = last_rendered
self._event_seq = 0
self._published_seq = 0
self.wakeup = anyio.Event()
async def _wait_for_wakeup(self) -> None:
await self.wakeup.wait()
self.wakeup = anyio.Event()
async def run(self) -> None:
if self.progress_id is None:
return
while True:
await self._wait_for_wakeup()
while self._published_seq < self._event_seq:
await self.sleep(
max(
0.0,
self.last_edit_at + self.progress_edit_every - self.clock(),
)
)
seq_at_render = self._event_seq
now = self.clock()
md = self.renderer.render_progress(now - self.started_at)
rendered, entities = prepare_telegram(md, limit=self.limit)
if rendered != self.last_rendered:
logger.debug(
"[progress] edit message_id=%s md=%s", self.progress_id, md
)
self.last_edit_at = now
edited = await self.bot.edit_message_text(
chat_id=self.chat_id,
message_id=self.progress_id,
text=rendered,
entities=entities,
)
if edited is not None:
self.last_rendered = rendered
self._published_seq = seq_at_render
async def on_event(self, evt: dict[str, Any]) -> None:
if not self.renderer.note_event(evt):
return
if self.progress_id is None:
return
self._event_seq += 1
self.wakeup.set()
class CodexExecRunner:
def __init__(
self,
codex_cmd: str,
extra_args: list[str],
) -> None:
self.codex_cmd = codex_cmd
self.extra_args = extra_args
# Per-session locks to prevent concurrent resumes to the same session_id.
self._session_locks: WeakValueDictionary[str, anyio.Lock] = (
WeakValueDictionary()
)
async def _lock_for(self, session_id: str) -> anyio.Lock:
lock = self._session_locks.get(session_id)
if lock is None:
lock = anyio.Lock()
self._session_locks[session_id] = lock
return lock
async def run(
self,
prompt: str,
session_id: str | None,
on_event: EventCallback | None = None,
) -> tuple[str, str, bool]:
logger.info("[codex] start run session_id=%r", session_id)
logger.debug("[codex] prompt: %s", prompt)
args = [self.codex_cmd]
args.extend(self.extra_args)
args.extend(["exec", "--json"])
# Always pipe prompt via stdin ("-") to avoid quoting issues.
if session_id:
args.extend(["resume", session_id, "-"])
else:
args.append("-")
cancelled_exc_type = anyio.get_cancelled_exc_class()
cancelled_exc: BaseException | None = None
async with manage_subprocess(
*args,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
) as proc:
if proc.stdin is None or proc.stdout is None or proc.stderr is None:
raise RuntimeError("codex exec failed to open subprocess pipes")
proc_stdin = proc.stdin
proc_stdout = proc.stdout
proc_stderr = proc.stderr
logger.debug("[codex] spawn pid=%s args=%r", proc.pid, args)
stderr_tail: deque[str] = deque(maxlen=200)
rc: int | None = None
found_session: str | None = session_id
last_agent_text: str | None = None
saw_agent_message = False
cli_last_item: int | None = None
cancelled = False
async with anyio.create_task_group() as tg:
tg.start_soon(_drain_stderr, proc_stderr, stderr_tail)
try:
await proc_stdin.send(prompt.encode())
await proc_stdin.aclose()
async for raw_line in _iter_text_lines(proc_stdout):
raw = raw_line.rstrip("\n")
logger.debug("[codex][jsonl] %s", raw)
line = raw.strip()
if not line:
continue
try:
evt = json.loads(line)
except json.JSONDecodeError:
logger.debug("[codex][jsonl] invalid line: %r", line)
continue
cli_last_item, out_lines = render_event_cli(evt, cli_last_item)
for out in out_lines:
logger.info("[codex] %s", out)
if on_event is not None:
try:
res = on_event(evt)
if inspect.isawaitable(res):
await res
except Exception as e:
logger.info("[codex][on_event] callback error: %s", e)
if evt["type"] == "thread.started":
found_session = evt.get("thread_id") or found_session
if evt["type"] == "item.completed":
item = evt.get("item") or {}
if item.get("type") == "agent_message" and isinstance(
item.get("text"), str
):
last_agent_text = item["text"]
saw_agent_message = True
except cancelled_exc_type as exc:
cancelled = True
cancelled_exc = exc
tg.cancel_scope.cancel()
finally:
if not cancelled:
rc = await proc.wait()
if cancelled:
raise cancelled_exc # type: ignore[misc]
logger.debug("[codex] process exit pid=%s rc=%s", proc.pid, rc)
if rc != 0:
tail = "".join(stderr_tail)
raise RuntimeError(f"codex exec failed (rc={rc}). stderr tail:\n{tail}")
if not found_session:
raise RuntimeError(
"codex exec finished but no session_id/thread_id was captured"
)
logger.info("[codex] done run session_id=%r", found_session)
return (
found_session,
(last_agent_text or "(No agent_message captured from JSON stream.)"),
saw_agent_message,
)
async def run_serialized(
self,
prompt: str,
session_id: str | None,
on_event: EventCallback | None = None,
) -> tuple[str, str, bool]:
if session_id:
lock = await self._lock_for(session_id)
async with lock:
return await self.run(prompt, session_id=session_id, on_event=on_event)
session_lock: anyio.Lock | None = None
async def on_event_with_lock(evt: dict[str, Any]) -> None:
nonlocal session_lock
if session_lock is None and evt.get("type") == "thread.started":
thread_id = evt.get("thread_id")
if isinstance(thread_id, str) and thread_id:
session_lock = await self._lock_for(thread_id)
await session_lock.acquire()
if on_event is None:
return
res = on_event(evt)
if inspect.isawaitable(res):
await res
try:
return await self.run(prompt, session_id=None, on_event=on_event_with_lock)
finally:
if session_lock is not None:
session_lock.release()
@dataclass(frozen=True)
class BridgeConfig:
bot: TelegramClient
runner: CodexExecRunner
chat_id: int
final_notify: bool
startup_msg: str
max_concurrency: int
@dataclass
class RunningTask:
scope: anyio.CancelScope
session_id: str | None = None
def _parse_bridge_config(
*,
final_notify: bool,
profile: str | None,
) -> BridgeConfig:
startup_pwd = os.getcwd()
config, config_path = load_telegram_config()
try:
token = config["bot_token"]
except KeyError:
raise ConfigError(f"Missing key `bot_token` in {config_path}.") from None
if not isinstance(token, str) or not token.strip():
raise ConfigError(
f"Invalid `bot_token` in {config_path}; expected a non-empty string."
) from None
try:
chat_id_value = config["chat_id"]
except KeyError:
raise ConfigError(f"Missing key `chat_id` in {config_path}.") from None
if isinstance(chat_id_value, bool) or not isinstance(chat_id_value, int):
raise ConfigError(
f"Invalid `chat_id` in {config_path}; expected an integer."
) from None
chat_id = chat_id_value
codex_cmd = shutil.which("codex")
if not codex_cmd:
raise ConfigError(
"codex not found on PATH. Install the Codex CLI with:\n"
" npm install -g @openai/codex\n"
" # or on macOS\n"
" brew install codex"
)
startup_msg = f"🐙 takopi is ready to help-pi!\npwd: {startup_pwd}"
extra_args = ["-c", "notify=[]"]
if profile:
extra_args.extend(["--profile", profile])
bot = TelegramClient(token)
runner = CodexExecRunner(codex_cmd=codex_cmd, extra_args=extra_args)
return BridgeConfig(
bot=bot,
runner=runner,
chat_id=chat_id,
final_notify=final_notify,
startup_msg=startup_msg,
max_concurrency=16,
)
async def _send_startup(cfg: BridgeConfig) -> None:
logger.debug("[startup] message: %s", cfg.startup_msg)
sent = await cfg.bot.send_message(chat_id=cfg.chat_id, text=cfg.startup_msg)
if sent is not None:
logger.info("[startup] sent startup message to chat_id=%s", cfg.chat_id)
async def _drain_backlog(cfg: BridgeConfig, offset: int | None) -> int | None:
drained = 0
while True:
updates = await cfg.bot.get_updates(
offset=offset, timeout_s=0, allowed_updates=["message"]
)
if updates is None:
logger.info("[startup] backlog drain failed")
return offset
logger.debug("[startup] backlog updates: %s", updates)
if not updates:
if drained:
logger.info("[startup] drained %s pending update(s)", drained)
return offset
offset = updates[-1]["update_id"] + 1
drained += len(updates)
async def handle_message(
cfg: BridgeConfig,
*,
chat_id: int,
user_msg_id: int,
text: str,
resume_session: str | None,
running_tasks: dict[int, RunningTask] | None = None,
clock: Callable[[], float] = time.monotonic,
sleep: Callable[[float], Awaitable[None]] = anyio.sleep,
progress_edit_every: float = PROGRESS_EDIT_EVERY_S,
) -> None:
logger.debug(
"[handle] incoming chat_id=%s message_id=%s resume=%r text=%s",
chat_id,
user_msg_id,
resume_session,
text,
)
started_at = clock()
progress_renderer = ExecProgressRenderer(max_actions=5)
progress_id: int | None = None
last_edit_at = 0.0
last_rendered: str | None = None
initial_md = progress_renderer.render_progress(0.0)
initial_rendered, initial_entities = prepare_telegram(
initial_md, limit=TELEGRAM_MARKDOWN_LIMIT
)
logger.debug(
"[progress] send reply_to=%s md=%s rendered=%s entities=%s",
user_msg_id,
initial_md,
initial_rendered,
initial_entities,
)
progress_msg = await cfg.bot.send_message(
chat_id=chat_id,
text=initial_rendered,
entities=initial_entities,
reply_to_message_id=user_msg_id,
disable_notification=True,
)
if progress_msg is not None:
progress_id = int(progress_msg["message_id"])
last_edit_at = clock()
last_rendered = initial_rendered
logger.debug("[progress] sent chat_id=%s message_id=%s", chat_id, progress_id)
edits = ProgressEdits(
bot=cfg.bot,
chat_id=chat_id,
progress_id=progress_id,
renderer=progress_renderer,
started_at=started_at,
progress_edit_every=progress_edit_every,
clock=clock,
sleep=sleep,
limit=TELEGRAM_MARKDOWN_LIMIT,
last_edit_at=last_edit_at,
last_rendered=last_rendered,
)
exec_scope = anyio.CancelScope()
cancelled = False
error: Exception | None = None
session_id: str | None = None
answer: str | None = None
saw_agent_message: bool | None = None
running_task: RunningTask | None = None
if running_tasks is not None and progress_id is not None:
running_task = RunningTask(scope=exec_scope)
running_tasks[progress_id] = running_task
if resume_session is not None:
running_task.session_id = resume_session
async def on_event(evt: dict[str, Any]) -> None:
if (
running_task is not None
and running_task.session_id is None
and evt.get("type") == "thread.started"
):
thread_id = evt.get("thread_id")
if isinstance(thread_id, str) and thread_id:
running_task.session_id = thread_id
await edits.on_event(evt)
async with anyio.create_task_group() as tg:
if progress_id is not None:
tg.start_soon(edits.run)
try:
with exec_scope:
session_id, answer, saw_agent_message = await cfg.runner.run_serialized(
text, resume_session, on_event=on_event
)
except Exception as e:
error = e
finally:
if running_task is not None:
if running_tasks is not None and progress_id is not None:
running_tasks.pop(progress_id, None)
if exec_scope.cancelled_caught and not cancelled and error is None:
cancelled = True
session_id = progress_renderer.resume_session or resume_session
if not cancelled and error is None:
await anyio.sleep(0)
tg.cancel_scope.cancel()
if error is not None:
err = _clamp_tg_text(f"Error:\n{error}")
logger.debug("[error] send reply_to=%s text=%s", user_msg_id, err)
await _send_or_edit_markdown(
cfg.bot,
chat_id=chat_id,
text=err,
edit_message_id=progress_id,
reply_to_message_id=user_msg_id,
disable_notification=True,
limit=TELEGRAM_MARKDOWN_LIMIT,
)
return
elapsed = clock() - started_at
if cancelled:
if session_id is None:
session_id = progress_renderer.resume_session or resume_session
logger.info(
"[handle] cancelled session_id=%s elapsed=%.1fs", session_id, elapsed
)
progress_renderer.resume_session = session_id
final_md = progress_renderer.render_progress(elapsed, label="`cancelled`")
await _send_or_edit_markdown(
cfg.bot,
chat_id=chat_id,
text=final_md,
edit_message_id=progress_id,
reply_to_message_id=user_msg_id,
disable_notification=True,
limit=TELEGRAM_MARKDOWN_LIMIT,
)
return
if session_id is None or answer is None or saw_agent_message is None:
raise RuntimeError("codex exec finished without a result")
status = "done" if saw_agent_message else "error"
progress_renderer.resume_session = session_id
final_md = progress_renderer.render_final(elapsed, answer, status=status)
logger.debug("[final] markdown: %s", final_md)
final_rendered, final_entities = render_markdown(final_md)
can_edit_final = (
progress_id is not None and len(final_rendered) <= TELEGRAM_MARKDOWN_LIMIT
)
edit_message_id = None if cfg.final_notify or not can_edit_final else progress_id
if edit_message_id is None:
logger.debug(
"[final] send reply_to=%s rendered=%s entities=%s",
user_msg_id,
final_rendered,
final_entities,
)
else:
logger.debug(
"[final] edit message_id=%s rendered=%s entities=%s",
edit_message_id,
final_rendered,
final_entities,
)
final_msg, edited = await _send_or_edit_markdown(
cfg.bot,
chat_id=chat_id,
text=final_md,
edit_message_id=edit_message_id,
reply_to_message_id=user_msg_id,
disable_notification=False,
limit=TELEGRAM_MARKDOWN_LIMIT,
)
if final_msg is None:
return
if progress_id is not None and (edit_message_id is None or not edited):
logger.debug("[final] delete progress message_id=%s", progress_id)
await cfg.bot.delete_message(chat_id=chat_id, message_id=progress_id)
async def poll_updates(cfg: BridgeConfig):
offset: int | None = None
offset = await _drain_backlog(cfg, offset)
await _send_startup(cfg)
while True:
updates = await cfg.bot.get_updates(
offset=offset, timeout_s=50, allowed_updates=["message"]
)
if updates is None:
logger.info("[loop] getUpdates failed")
await anyio.sleep(2)
continue
logger.debug("[loop] updates: %s", updates)
for upd in updates:
offset = upd["update_id"] + 1
msg = upd["message"]
if "text" not in msg:
continue
if not (msg["chat"]["id"] == msg["from"]["id"] == cfg.chat_id):
continue
yield msg
async def _handle_cancel(
cfg: BridgeConfig,
msg: dict[str, Any],
running_tasks: dict[int, RunningTask],
) -> None:
chat_id = msg["chat"]["id"]
user_msg_id = msg["message_id"]
reply = msg.get("reply_to_message")
if not reply:
await cfg.bot.send_message(
chat_id=chat_id,
text="reply to the progress message to cancel.",
reply_to_message_id=user_msg_id,
)
return
progress_id = reply.get("message_id")
if progress_id is None:
await cfg.bot.send_message(
chat_id=chat_id,
text="nothing is currently running for that message.",
reply_to_message_id=user_msg_id,
)
return
running_task = running_tasks.get(int(progress_id))
if running_task is None:
await cfg.bot.send_message(
chat_id=chat_id,
text="nothing is currently running for that message.",
reply_to_message_id=user_msg_id,
)
return
logger.info("[cancel] cancelling progress_message_id=%s", progress_id)
running_task.scope.cancel()
async def _run_main_loop(cfg: BridgeConfig) -> None:
worker_count = max(1, min(cfg.max_concurrency, 16))
send_stream, receive_stream = anyio.create_memory_object_stream(
max_buffer_size=worker_count * 2
)
running_tasks: dict[int, RunningTask] = {}
async def worker() -> None:
while True:
chat_id, user_msg_id, text, resume_session = await receive_stream.receive()
try:
await handle_message(
cfg,
chat_id=chat_id,
user_msg_id=user_msg_id,
text=text,
resume_session=resume_session,
running_tasks=running_tasks,
)
except Exception:
logger.exception("[handle] worker failed")
try:
async with anyio.create_task_group() as tg:
for _ in range(worker_count):
tg.start_soon(worker)
async for msg in poll_updates(cfg):
text = msg["text"]
user_msg_id = msg["message_id"]
if text == "/cancel":
tg.start_soon(_handle_cancel, cfg, msg, running_tasks)
continue
r = msg.get("reply_to_message") or {}
resume_session = resolve_resume_session(text, r.get("text"))
await send_stream.send(
(msg["chat"]["id"], user_msg_id, text, resume_session)
)
finally:
await send_stream.aclose()
await receive_stream.aclose()
await cfg.bot.close()
def run(
version: bool = typer.Option(
False,
"--version",
help="Show the version and exit.",
callback=_version_callback,
is_eager=True,
),
final_notify: bool = typer.Option(
True,
"--final-notify/--no-final-notify",
help="Send the final response as a new message (not an edit).",
),
debug: bool = typer.Option(
False,
"--debug/--no-debug",
help="Log codex JSONL, Telegram requests, and rendered messages.",
),
profile: str | None = typer.Option(
None,
"--profile",
help="Codex profile name to pass to `codex --profile`.",
),
) -> None:
setup_logging(debug=debug)
setup = check_setup()
if not setup.ok:
render_setup_guide(setup)
raise typer.Exit(code=1)
try:
cfg = _parse_bridge_config(
final_notify=final_notify,
profile=profile,
)
except ConfigError as e:
typer.echo(str(e), err=True)
raise typer.Exit(code=1)
anyio.run(_run_main_loop, cfg)
def main() -> None:
typer.run(run)
if __name__ == "__main__":
main()
src/takopi/exec_render.py
-261
View File
@@ -1,261 +0,0 @@
from __future__ import annotations
import re
import textwrap
from collections import deque
from pathlib import Path
from textwrap import indent
from typing import Any
from markdown_it import MarkdownIt
from sulguk import transform_html
STATUS_RUNNING = "▸"
STATUS_DONE = "✓"
STATUS_FAIL = "✗"
HEADER_SEP = " · "
HARD_BREAK = " \n"
MAX_PROGRESS_CMD_LEN = 300
MAX_QUERY_LEN = 60
MAX_PATH_LEN = 40
_md = MarkdownIt("commonmark", {"html": False})
def render_markdown(md: str) -> tuple[str, list[dict[str, Any]]]:
html = _md.render(md or "")
rendered = transform_html(html)
text = re.sub(r"(?m)^(\s*)•", r"\1-", rendered.text)
entities = [dict(e) for e in rendered.entities]
return text, entities
def format_elapsed(elapsed_s: float) -> str:
total = max(0, int(elapsed_s))
minutes, seconds = divmod(total, 60)
hours, minutes = divmod(minutes, 60)
if hours:
return f"{hours}h {minutes:02d}m"
if minutes:
return f"{minutes}m {seconds:02d}s"
return f"{seconds}s"
def format_header(elapsed_s: float, item: int | None, label: str) -> str:
elapsed = format_elapsed(elapsed_s)
parts = [label, elapsed]
if item is not None:
parts.append(f"step {item}")
return HEADER_SEP.join(parts)
def extract_numeric_id(item_id: object, fallback: int | None = None) -> int | None:
if isinstance(item_id, int):
return item_id
if isinstance(item_id, str):
match = re.search(r"(?:item_)?(\d+)", item_id)
if match:
return int(match.group(1))
return fallback
def _shorten(text: str, width: int) -> str:
return textwrap.shorten(text, width=width, placeholder="…")
def _format_change_path(path: str) -> str:
workdir = Path.cwd()
path_obj = Path(path)
if path_obj.is_absolute() and path_obj.is_relative_to(workdir):
return str(path_obj.relative_to(workdir))
return path
def format_event(
event: dict[str, Any],
last_item: int | None,
*,
command_width: int | None = None,
escape_markdown: bool = False,
) -> tuple[int | None, list[str], str | None, str | None]:
lines: list[str] = []
match event["type"]:
case "thread.started":
return last_item, ["thread started"], None, None
case "turn.started":
return last_item, ["turn started"], None, None
case "turn.completed":
return last_item, ["turn completed"], None, None
case "turn.failed":
return last_item, [f"turn failed: {event['error']['message']}"], None, None
case "error":
return last_item, [f"stream error: {event['message']}"], None, None
case "item.started" | "item.updated" | "item.completed" as etype:
item = event["item"]
item_type = item.get("type") or item.get("item_type")
if item_type == "assistant_message":
item_type = "agent_message"
if item_type is None:
return last_item, [], None, None
item_num = extract_numeric_id(item.get("id"), last_item)
last_item = item_num if item_num is not None else last_item
prefix = f"{item_num}. "
if escape_markdown and item_num is not None:
# Avoid ordered-list parsing which renumbers items in MarkdownIt/CommonMark.
prefix = f"{item_num}\\." + " "
match (item_type, etype):
case ("agent_message", "item.completed"):
lines.append("assistant:")
lines.extend(indent(item["text"], " ").splitlines())
return last_item, lines, None, None
case ("reasoning", "item.completed"):
text = item.get("text") or ""
first_line = text.splitlines()[0] if text else ""
line = prefix + first_line
return last_item, [line], line, prefix
case ("command_execution", "item.started"):
command = item["command"]
if command_width is not None:
command = _shorten(command, command_width)
command = f"`{command}`"
line = prefix + f"{STATUS_RUNNING} {command}"
return last_item, [line], line, prefix
case ("command_execution", "item.completed"):
command = item["command"]
if command_width is not None:
command = _shorten(command, command_width)
command = f"`{command}`"
exit_code = item["exit_code"]
if exit_code == 0:
status = STATUS_DONE
exit_part = ""
else:
status = STATUS_FAIL if exit_code is not None else STATUS_DONE
exit_part = (
f" (exit {exit_code})" if exit_code is not None else ""
)
line = prefix + f"{status} {command}{exit_part}"
return last_item, [line], line, prefix
case ("mcp_tool_call", "item.started"):
name = (
".".join(
part for part in (item["server"], item["tool"]) if part
)
or "tool"
)
line = prefix + f"{STATUS_RUNNING} tool: {name}"
return last_item, [line], line, prefix
case ("mcp_tool_call", "item.completed"):
name = (
".".join(
part for part in (item["server"], item["tool"]) if part
)
or "tool"
)
line = prefix + f"{STATUS_DONE} tool: {name}"
return last_item, [line], line, prefix
case ("web_search", "item.completed"):
query = _shorten(item["query"], MAX_QUERY_LEN)
line = prefix + f"{STATUS_DONE} searched: {query}"
return last_item, [line], line, prefix
case ("file_change", "item.completed"):
paths = [
change["path"]
for change in item["changes"]
if change.get("path")
]
if not paths:
total = len(item["changes"])
desc = (
"updated files" if total == 0 else f"updated {total} files"
)
elif len(paths) <= 3:
desc = "updated " + ", ".join(
f"`{_format_change_path(p)}`" for p in paths
)
else:
desc = f"updated {len(paths)} files"
line = prefix + f"{STATUS_DONE} {desc}"
return last_item, [line], line, prefix
case ("error", "item.completed"):
warning = _shorten(item["message"], 120)
line = prefix + f"{STATUS_DONE} warning: {warning}"
return last_item, [line], line, prefix
case _:
return last_item, [], None, None
case _:
return last_item, [], None, None
def render_event_cli(
event: dict[str, Any], last_item: int | None = None
) -> tuple[int | None, list[str]]:
last_item, cli_lines, _, _ = format_event(
event, last_item, command_width=None, escape_markdown=False
)
return last_item, cli_lines
class ExecProgressRenderer:
def __init__(
self,
max_actions: int = 5,
command_width: int | None = MAX_PROGRESS_CMD_LEN,
) -> None:
self.max_actions = max_actions
self.command_width = command_width
self.recent_actions: deque[str] = deque(maxlen=max_actions)
self.last_item: int | None = None
self.resume_session: str | None = None
def note_event(self, event: dict[str, Any]) -> bool:
if event["type"] == "thread.started":
self.resume_session = event["thread_id"]
return True
self.last_item, _, progress_line, progress_prefix = format_event(
event,
self.last_item,
command_width=self.command_width,
escape_markdown=True,
)
if progress_line is None:
return False
# Replace the preceding "running" line for the same item on completion.
if (
event["type"] == "item.completed"
and progress_prefix
and self.recent_actions
):
last = self.recent_actions[-1]
if last.startswith(progress_prefix + f"{STATUS_RUNNING} "):
self.recent_actions.pop()
self.recent_actions.append(progress_line)
return True
def render_progress(self, elapsed_s: float, label: str = "working") -> str:
header = format_header(elapsed_s, self.last_item, label=label)
message = self._assemble(header, list(self.recent_actions))
return self._append_resume(message)
def render_final(self, elapsed_s: float, answer: str, status: str = "done") -> str:
header = format_header(elapsed_s, self.last_item, label=status)
answer = (answer or "").strip()
message = header + ("\n\n" + answer if answer else "")
return self._append_resume(message)
def _append_resume(self, message: str) -> str:
if not self.resume_session:
return message
return message + f"\n\nresume: `{self.resume_session}`"
@staticmethod
def _assemble(header: str, lines: list[str]) -> str:
return header if not lines else header + "\n\n" + HARD_BREAK.join(lines)
src/takopi/markdown.py
+83
View File
@@ -0,0 +1,83 @@
"""Markdown rendering and truncation helpers for Telegram constraints."""
from __future__ import annotations
import re
from typing import Any, Callable
from markdown_it import MarkdownIt
from sulguk import transform_html
TELEGRAM_MARKDOWN_LIMIT = 3500
_md = MarkdownIt("commonmark", {"html": False})
def render_markdown(md: str) -> tuple[str, list[dict[str, Any]]]:
html = _md.render(md or "")
rendered = transform_html(html)
text = re.sub(r"(?m)^(\s*)•", r"\1-", rendered.text)
entities = [dict(e) for e in rendered.entities]
return text, entities
def truncate_for_telegram(
text: str, limit: int, *, is_resume_line: Callable[[str], bool]
) -> str:
"""
Truncate text to fit Telegram limits while preserving the trailing resume command
line (if present), otherwise preserving the last non-empty line.
"""
if len(text) <= limit:
return text
lines = text.splitlines()
tail_lines: list[str] | None = None
is_resume_tail = False
for i in range(len(lines) - 1, -1, -1):
line = lines[i]
if is_resume_line(line):
tail_lines = lines[i:]
is_resume_tail = True
break
if tail_lines is None:
for i in range(len(lines) - 1, -1, -1):
if lines[i].strip():
tail_lines = [lines[i]]
break
tail = "\n".join(tail_lines or []).strip("\n")
sep = "\n…\n"
max_tail = limit if is_resume_tail else (limit // 4)
tail = tail[-max_tail:] if max_tail > 0 else ""
head_budget = limit - len(sep) - len(tail)
if head_budget <= 0:
return tail[-limit:] if tail else text[:limit]
head = text[:head_budget].rstrip()
return (head + sep + tail)[:limit]
def prepare_telegram(
md: str,
*,
limit: int,
is_resume_line: Callable[[str], bool] | None = None,
) -> tuple[str, list[dict[str, Any]] | None]:
rendered, entities = render_markdown(md)
if len(rendered) > limit:
if is_resume_line is None:
def _never_resume_line(_line: str) -> bool:
return False
is_resume_line = _never_resume_line
rendered = truncate_for_telegram(rendered, limit, is_resume_line=is_resume_line)
return rendered, None
return rendered, entities
src/takopi/model.py
+76
View File
@@ -0,0 +1,76 @@
"""Takopi domain model types (events, actions, resume tokens)."""
from __future__ import annotations
from dataclasses import dataclass, field
from typing import Any, Literal, TypeAlias
EngineId: TypeAlias = str
ActionKind: TypeAlias = Literal[
"command",
"tool",
"file_change",
"web_search",
"note",
"turn",
"warning",
"telemetry",
]
TakopiEventType: TypeAlias = Literal[
"started",
"action",
"completed",
]
ActionPhase: TypeAlias = Literal["started", "updated", "completed"]
ActionLevel: TypeAlias = Literal["debug", "info", "warning", "error"]
@dataclass(frozen=True, slots=True)
class ResumeToken:
engine: EngineId
value: str
@dataclass(frozen=True, slots=True)
class Action:
id: str
kind: ActionKind
title: str
detail: dict[str, Any] = field(default_factory=dict)
@dataclass(frozen=True, slots=True)
class StartedEvent:
type: Literal["started"] = field(default="started", init=False)
engine: EngineId
resume: ResumeToken
title: str | None = None
meta: dict[str, Any] | None = None
@dataclass(frozen=True, slots=True)
class ActionEvent:
type: Literal["action"] = field(default="action", init=False)
engine: EngineId
action: Action
phase: ActionPhase
ok: bool | None = None
message: str | None = None
level: ActionLevel | None = None
@dataclass(frozen=True, slots=True)
class CompletedEvent:
type: Literal["completed"] = field(default="completed", init=False)
engine: EngineId
ok: bool
answer: str
resume: ResumeToken | None = None
error: str | None = None
usage: dict[str, Any] | None = None
TakopiEvent: TypeAlias = StartedEvent | ActionEvent | CompletedEvent
src/takopi/onboarding.py
+37 -38
View File
@@ -1,6 +1,5 @@
from __future__ import annotations
import shutil
from dataclasses import dataclass
from pathlib import Path
@@ -8,32 +7,52 @@ from rich.console import Console
from rich.panel import Panel
from .config import ConfigError, HOME_CONFIG_PATH, load_telegram_config
from .engines import EngineBackend, SetupIssue
_OCTOPUS = "\N{OCTOPUS}"
@dataclass(slots=True)
class SetupResult:
missing_codex: bool = False
missing_or_invalid_config: bool = False
issues: list[SetupIssue]
config_path: Path = HOME_CONFIG_PATH
@property
def ok(self) -> bool:
return not (self.missing_codex or self.missing_or_invalid_config)
return not self.issues
def check_setup() -> SetupResult:
missing_codex = shutil.which("codex") is None
def _config_issue(path: Path) -> SetupIssue:
config_display = _config_path_display(path)
return SetupIssue(
"Create a config",
(
f" [dim]{config_display}[/]",
"",
' [cyan]bot_token[/] = [green]"123456789:ABCdef..."[/]',
" [cyan]chat_id[/] = [green]123456789[/]",
"",
"[dim]" + ("-" * 56) + "[/]",
"",
"[bold]Getting your Telegram credentials:[/]",
"",
" [cyan]bot_token[/] create a bot with [link=https://t.me/BotFather]@BotFather[/]",
" [cyan]chat_id[/] message [link=https://t.me/myidbot]@myidbot[/] to get your id",
),
)
def check_setup(backend: EngineBackend) -> SetupResult:
issues: list[SetupIssue] = []
config_path = HOME_CONFIG_PATH
config: dict = {}
try:
config, config_path = load_telegram_config()
except ConfigError:
return SetupResult(
missing_codex=missing_codex,
missing_or_invalid_config=True,
config_path=HOME_CONFIG_PATH,
)
issues.extend(backend.check_setup({}, config_path))
issues.append(_config_issue(config_path))
return SetupResult(issues=issues, config_path=config_path)
token = config.get("bot_token")
chat_id = config.get("chat_id")
@@ -41,11 +60,11 @@ def check_setup() -> SetupResult:
missing_or_invalid_config = not (isinstance(token, str) and token.strip())
missing_or_invalid_config |= type(chat_id) is not int
return SetupResult(
missing_codex=missing_codex,
missing_or_invalid_config=missing_or_invalid_config,
config_path=config_path,
)
issues.extend(backend.check_setup(config, config_path))
if missing_or_invalid_config:
issues.append(_config_issue(config_path))
return SetupResult(issues=issues, config_path=config_path)
def _config_path_display(path: Path) -> str:
@@ -72,28 +91,8 @@ def render_setup_guide(result: SetupResult) -> None:
parts.extend(lines)
parts.append("")
if result.missing_codex:
add_step(
"Install the Codex CLI",
" [dim]$[/] npm install -g @openai/codex",
)
if result.missing_or_invalid_config:
config_display = _config_path_display(result.config_path)
add_step(
"Create a config",
f" [dim]{config_display}[/]",
"",
' [cyan]bot_token[/] = [green]"123456789:ABCdef..."[/]',
" [cyan]chat_id[/] = [green]123456789[/]",
"",
"[dim]" + ("-" * 56) + "[/]",
"",
"[bold]Getting your Telegram credentials:[/]",
"",
" [cyan]bot_token[/] create a bot with [link=https://t.me/BotFather]@BotFather[/]",
" [cyan]chat_id[/] message [link=https://t.me/myidbot]@myidbot[/] to get your id",
)
for issue in result.issues:
add_step(issue.title, *issue.lines)
panel = Panel(
"\n".join(parts).rstrip(),
src/takopi/render.py
+259
View File
@@ -0,0 +1,259 @@
"""Pure renderers for Takopi events (no engine-native event handling)."""
from __future__ import annotations
import textwrap
from collections import deque
from typing import Callable
from .model import Action, ActionEvent, ResumeToken, StartedEvent, TakopiEvent
STATUS_RUNNING = "▸"
STATUS_UPDATE = "↻"
STATUS_DONE = "✓"
STATUS_FAIL = "✗"
HEADER_SEP = " · "
HARD_BREAK = " \n"
MAX_PROGRESS_CMD_LEN = 300
MAX_FILE_CHANGES_INLINE = 3
FILE_CHANGE_PREFIX = {"add": "+", "delete": "-", "update": "~"}
def format_elapsed(elapsed_s: float) -> str:
total = max(0, int(elapsed_s))
minutes, seconds = divmod(total, 60)
hours, minutes = divmod(minutes, 60)
if hours:
return f"{hours}h {minutes:02d}m"
if minutes:
return f"{minutes}m {seconds:02d}s"
return f"{seconds}s"
def format_header(elapsed_s: float, item: int | None, label: str) -> str:
elapsed = format_elapsed(elapsed_s)
parts = [label, elapsed]
if item is not None:
parts.append(f"step {item}")
return HEADER_SEP.join(parts)
def shorten(text: str, width: int | None) -> str:
if width is None:
return text
return textwrap.shorten(text, width=width, placeholder="…")
def action_status_symbol(
action: Action, *, completed: bool, ok: bool | None = None
) -> str:
if not completed:
return STATUS_RUNNING
if ok is not None:
return STATUS_DONE if ok else STATUS_FAIL
detail = action.detail or {}
exit_code = detail.get("exit_code")
if isinstance(exit_code, int) and exit_code != 0:
return STATUS_FAIL
return STATUS_DONE
def action_exit_suffix(action: Action) -> str:
detail = action.detail or {}
exit_code = detail.get("exit_code")
if isinstance(exit_code, int) and exit_code != 0:
return f" (exit {exit_code})"
return ""
def format_file_change_title(action: Action, *, command_width: int | None) -> str:
title = str(action.title or "")
detail = action.detail or {}
changes = detail.get("changes")
if isinstance(changes, list) and changes:
rendered: list[str] = []
for raw in changes:
if not isinstance(raw, dict):
continue
path = raw.get("path")
if not isinstance(path, str) or not path:
continue
kind = raw.get("kind")
prefix = FILE_CHANGE_PREFIX.get(kind, "~") if isinstance(kind, str) else "~"
rendered.append(f"{prefix}{path}")
if rendered:
if len(rendered) > MAX_FILE_CHANGES_INLINE:
remaining = len(rendered) - MAX_FILE_CHANGES_INLINE
rendered = rendered[:MAX_FILE_CHANGES_INLINE] + [f"…(+{remaining})"]
inline = shorten(", ".join(rendered), command_width)
return f"files: {inline}"
return f"files: {shorten(title, command_width)}"
def format_action_title(action: Action, *, command_width: int | None) -> str:
title = str(action.title or "")
kind = action.kind
if kind == "command":
title = shorten(title, command_width)
return f"`{title}`"
if kind == "tool":
title = shorten(title, command_width)
return f"tool: {title}"
if kind == "web_search":
title = shorten(title, command_width)
return f"searched: {title}"
if kind == "file_change":
return format_file_change_title(action, command_width=command_width)
if kind in {"note", "warning"}:
return shorten(title, command_width)
return shorten(title, command_width)
def phase_status_and_suffix(event: ActionEvent) -> tuple[str, str]:
action = event.action
match event.phase:
case "completed":
status = action_status_symbol(action, completed=True, ok=event.ok)
suffix = action_exit_suffix(action)
return status, suffix
case "updated":
return STATUS_UPDATE, ""
case _:
return STATUS_RUNNING, ""
def render_event_cli(event: TakopiEvent) -> list[str]:
match event:
case StartedEvent(engine=engine):
return [str(engine)]
case ActionEvent() as action_event:
action = action_event.action
if action.kind == "turn":
return []
status, suffix = phase_status_and_suffix(action_event)
title = format_action_title(action, command_width=MAX_PROGRESS_CMD_LEN)
return [f"{status} {title}{suffix}"]
case _:
return []
class ExecProgressRenderer:
def __init__(
self,
max_actions: int = 5,
command_width: int | None = MAX_PROGRESS_CMD_LEN,
resume_formatter: Callable[[ResumeToken], str] | None = None,
show_title: bool = False,
) -> None:
self.max_actions = max_actions
self.command_width = command_width
self.recent_actions: deque[str] = deque(maxlen=max_actions)
self._recent_action_ids: deque[str] = deque(maxlen=max_actions)
self._recent_action_completed: deque[bool] = deque(maxlen=max_actions)
self.action_count = 0
self._started_counts: dict[str, int] = {}
self.resume_token: ResumeToken | None = None
self.session_title: str | None = None
self._resume_formatter = resume_formatter
self.show_title = show_title
def note_event(self, event: TakopiEvent) -> bool:
match event:
case StartedEvent(resume=resume, title=title):
self.resume_token = resume
self.session_title = title
return True
case ActionEvent(action=action, phase=phase, ok=ok):
if action.kind == "turn":
return False
action_id = str(action.id or "")
if not action_id:
return False
completed = phase == "completed"
if completed:
is_update = False
else:
started_count = self._started_counts.get(action_id, 0)
is_update = phase == "updated" or started_count > 0
if started_count == 0:
self.action_count += 1
self._started_counts[action_id] = 1
elif phase == "started":
self._started_counts[action_id] = started_count + 1
else:
self._started_counts[action_id] = started_count
case _:
return False
if completed:
count = self._started_counts.get(action_id, 0)
if count <= 0:
self.action_count += 1
elif count == 1:
self._started_counts.pop(action_id, None)
else:
self._started_counts[action_id] = count - 1
status = (
STATUS_UPDATE
if (is_update and not completed)
else action_status_symbol(action, completed=completed, ok=ok)
)
title = format_action_title(action, command_width=self.command_width)
suffix = action_exit_suffix(action) if completed else ""
line = f"{status} {title}{suffix}"
self._append_action(action_id, completed=completed, line=line)
return True
def _append_action(self, action_id: str, *, completed: bool, line: str) -> None:
for i in range(len(self._recent_action_ids) - 1, -1, -1):
if (
self._recent_action_ids[i] == action_id
and not self._recent_action_completed[i]
):
self.recent_actions[i] = line
if completed:
self._recent_action_completed[i] = True
return
if len(self.recent_actions) >= self.max_actions:
self.recent_actions.popleft()
self._recent_action_ids.popleft()
self._recent_action_completed.popleft()
self.recent_actions.append(line)
self._recent_action_ids.append(action_id)
self._recent_action_completed.append(completed)
def render_progress(self, elapsed_s: float, label: str = "working") -> str:
step = self.action_count or None
header = format_header(elapsed_s, step, label=self._label_with_title(label))
message = self._assemble(header, list(self.recent_actions))
return self._append_resume(message)
def render_final(self, elapsed_s: float, answer: str, status: str = "done") -> str:
step = self.action_count or None
header = format_header(elapsed_s, step, label=self._label_with_title(status))
answer = (answer or "").strip()
message = header + ("\n\n" + answer if answer else "")
return self._append_resume(message)
def _label_with_title(self, label: str) -> str:
if self.show_title and self.session_title:
return f"{label} ({self.session_title})"
return label
def _append_resume(self, message: str) -> str:
if not self.resume_token or self._resume_formatter is None:
return message
return message + "\n\n" + self._resume_formatter(self.resume_token)
@staticmethod
def _assemble(header: str, lines: list[str]) -> str:
return header if not lines else header + "\n\n" + HARD_BREAK.join(lines)
src/takopi/runner.py
+55
View File
@@ -0,0 +1,55 @@
"""Runner protocol and shared runner definitions."""
from __future__ import annotations
import re
from collections.abc import AsyncIterator
from typing import Protocol
from .model import EngineId, ResumeToken, TakopiEvent
def compile_resume_pattern(engine: EngineId) -> re.Pattern[str]:
name = re.escape(str(engine))
return re.compile(rf"(?im)^\s*`?{name}\s+resume\s+(?P<token>[^`\s]+)`?\s*$")
class ResumeRunnerMixin:
engine: EngineId
resume_re: re.Pattern[str]
def format_resume(self, token: ResumeToken) -> str:
if token.engine != self.engine:
raise RuntimeError(f"resume token is for engine {token.engine!r}")
return f"`{self.engine} resume {token.value}`"
def is_resume_line(self, line: str) -> bool:
return bool(self.resume_re.match(line))
def extract_resume(self, text: str | None) -> ResumeToken | None:
if not text:
return None
found: str | None = None
for match in self.resume_re.finditer(text):
token = match.group("token")
if token:
found = token
if not found:
return None
return ResumeToken(engine=self.engine, value=found)
class Runner(Protocol):
engine: str
def is_resume_line(self, line: str) -> bool: ...
def format_resume(self, token: ResumeToken) -> str: ...
def extract_resume(self, text: str | None) -> ResumeToken | None: ...
def run(
self,
prompt: str,
resume: ResumeToken | None,
) -> AsyncIterator[TakopiEvent]: ...
src/takopi/runners/__init__.py
+1
View File
@@ -0,0 +1 @@
"""Runner implementations."""
src/takopi/runners/codex.py
+758
View File
@@ -0,0 +1,758 @@
from __future__ import annotations
import json
import logging
import os
import signal
import subprocess
from collections import deque
from collections.abc import AsyncIterator
from contextlib import asynccontextmanager
from dataclasses import dataclass
from typing import Any, cast
from weakref import WeakValueDictionary
import anyio
from anyio.abc import ByteReceiveStream, Process
from anyio.streams.text import TextReceiveStream
from ..model import (
Action,
ActionEvent,
ActionKind,
ActionLevel,
ActionPhase,
CompletedEvent,
EngineId,
ResumeToken,
StartedEvent,
TakopiEvent,
)
from ..runner import ResumeRunnerMixin, Runner, compile_resume_pattern
logger = logging.getLogger(__name__)
ENGINE: EngineId = EngineId("codex")
STDERR_TAIL_LINES = 200
_ACTION_KIND_MAP: dict[str, ActionKind] = {
"command_execution": "command",
"mcp_tool_call": "tool",
"tool_call": "tool",
"web_search": "web_search",
"file_change": "file_change",
"reasoning": "note",
"todo_list": "note",
}
_RESUME_RE = compile_resume_pattern(ENGINE)
def _started_event(token: ResumeToken, *, title: str) -> StartedEvent:
return StartedEvent(engine=token.engine, resume=token, title=title)
def _completed_event(
*,
resume: ResumeToken | None,
ok: bool,
answer: str,
error: str | None = None,
usage: dict[str, Any] | None = None,
) -> TakopiEvent:
return CompletedEvent(
engine=ENGINE,
ok=ok,
answer=answer,
resume=resume,
error=error,
usage=usage,
)
def _action_event(
*,
phase: ActionPhase,
action_id: str,
kind: ActionKind,
title: str,
detail: dict[str, Any] | None = None,
ok: bool | None = None,
message: str | None = None,
level: ActionLevel | None = None,
) -> TakopiEvent:
action = Action(
id=action_id,
kind=kind,
title=title,
detail=detail or {},
)
return ActionEvent(
engine=ENGINE,
action=action,
phase=phase,
ok=ok,
message=message,
level=level,
)
def _note_completed(
action_id: str,
message: str,
*,
ok: bool = False,
detail: dict[str, Any] | None = None,
) -> TakopiEvent:
return _action_event(
phase="completed",
action_id=action_id,
kind="warning",
title=message,
detail=detail,
ok=ok,
message=message,
level="warning" if not ok else "info",
)
def _short_tool_name(item: dict[str, Any]) -> str:
name = ".".join(part for part in (item.get("server"), item.get("tool")) if part)
return name or "tool"
def _summarize_tool_result(result: Any) -> dict[str, Any] | None:
if not isinstance(result, dict):
return None
summary: dict[str, Any] = {}
content = result.get("content")
if isinstance(content, list):
summary["content_blocks"] = len(content)
elif content is not None:
summary["content_blocks"] = 1
if "structured" in result:
summary["has_structured"] = bool(result.get("structured"))
return summary or None
def _format_change_summary(item: dict[str, Any]) -> str:
changes = item.get("changes") or []
paths = [c.get("path") for c in changes if c.get("path")]
if not paths:
total = len(changes)
if total <= 0:
return "files"
return f"{total} files"
return ", ".join(str(path) for path in paths)
@dataclass(frozen=True, slots=True)
class _TodoSummary:
done: int
total: int
next_text: str | None
def _summarize_todo_list(items: Any) -> _TodoSummary:
if not isinstance(items, list):
return _TodoSummary(done=0, total=0, next_text=None)
done = 0
total = 0
next_text: str | None = None
for raw_item in items:
if not isinstance(raw_item, dict):
continue
total += 1
completed = raw_item.get("completed") is True
if completed:
done += 1
continue
if next_text is None:
text = raw_item.get("text")
next_text = str(text) if text is not None else None
return _TodoSummary(done=done, total=total, next_text=next_text)
def _todo_title(summary: _TodoSummary) -> str:
if summary.total <= 0:
return "todo"
if summary.next_text:
return f"todo {summary.done}/{summary.total}: {summary.next_text}"
return f"todo {summary.done}/{summary.total}: done"
def _translate_item_event(etype: str, item: dict[str, Any]) -> list[TakopiEvent]:
item_type = item.get("type") or item.get("item_type")
if item_type == "assistant_message":
item_type = "agent_message"
if not item_type:
return []
if item_type == "agent_message":
return []
action_id = item.get("id")
if not isinstance(action_id, str) or not action_id:
logger.debug("[codex] missing item id in codex event: %r", item)
return []
phase = cast(ActionPhase, etype.split(".")[-1])
if item_type == "error":
if phase != "completed":
return []
message = str(item.get("message") or "codex item error")
return [
_action_event(
phase="completed",
action_id=action_id,
kind="warning",
title=message,
detail={"message": message},
ok=False,
message=message,
level="warning",
)
]
kind = _ACTION_KIND_MAP.get(item_type)
if kind is None:
return []
if kind == "command":
title = str(item.get("command") or "")
if phase in {"started", "updated"}:
return [
_action_event(
phase=phase,
action_id=action_id,
kind=kind,
title=title,
)
]
if phase == "completed":
exit_code = item.get("exit_code")
ok = item.get("status") != "failed"
if isinstance(exit_code, int):
ok = ok and exit_code == 0
detail = {
"exit_code": exit_code,
"status": item.get("status"),
}
return [
_action_event(
phase="completed",
action_id=action_id,
kind=kind,
title=title,
detail=detail,
ok=ok,
)
]
if kind == "tool":
tool_name = _short_tool_name(item)
title = tool_name
detail = {
"server": item.get("server"),
"tool": item.get("tool"),
"status": item.get("status"),
}
if "arguments" in item:
detail["arguments"] = item.get("arguments")
if item_type == "tool_call":
name = item.get("name")
tool_name = str(name) if name else "tool"
title = tool_name
detail = {"name": name, "status": item.get("status")}
if "arguments" in item:
detail["arguments"] = item.get("arguments")
if phase in {"started", "updated"}:
return [
_action_event(
phase=phase,
action_id=action_id,
kind=kind,
title=title,
detail=detail,
)
]
if phase == "completed":
ok = item.get("status") != "failed" and not item.get("error")
error = item.get("error")
if error:
detail["error_message"] = str(
error.get("message") if isinstance(error, dict) else error
)
result_summary = _summarize_tool_result(item.get("result"))
if result_summary is not None:
detail["result_summary"] = result_summary
return [
_action_event(
phase="completed",
action_id=action_id,
kind=kind,
title=title,
detail=detail,
ok=ok,
)
]
if kind == "web_search":
title = str(item.get("query") or "")
detail = {"query": item.get("query")}
if phase in {"started", "updated"}:
return [
_action_event(
phase=phase,
action_id=action_id,
kind=kind,
title=title,
detail=detail,
)
]
if phase == "completed":
return [
_action_event(
phase="completed",
action_id=action_id,
kind=kind,
title=title,
detail=detail,
ok=True,
)
]
if kind == "file_change":
if phase != "completed":
return []
title = _format_change_summary(item)
detail = {
"changes": item.get("changes") or [],
"status": item.get("status"),
"error": item.get("error"),
}
ok = item.get("status") != "failed"
return [
_action_event(
phase="completed",
action_id=action_id,
kind=kind,
title=title,
detail=detail,
ok=ok,
)
]
if kind == "note":
if item_type == "todo_list":
summary = _summarize_todo_list(item.get("items"))
title = _todo_title(summary)
detail = {"done": summary.done, "total": summary.total}
else:
title = str(item.get("text") or "")
detail = None
if phase in {"started", "updated"}:
return [
_action_event(
phase=phase,
action_id=action_id,
kind=kind,
title=title,
detail=detail,
)
]
if phase == "completed":
return [
_action_event(
phase="completed",
action_id=action_id,
kind=kind,
title=title,
detail=detail,
ok=True,
)
]
return []
def translate_codex_event(event: dict[str, Any], *, title: str) -> list[TakopiEvent]:
etype = event.get("type")
if etype == "thread.started":
thread_id = event.get("thread_id")
if thread_id:
token = ResumeToken(engine=ENGINE, value=str(thread_id))
return [_started_event(token, title=title)]
logger.debug("[codex] codex thread.started missing thread_id: %r", event)
return []
if etype in {"item.started", "item.updated", "item.completed"}:
item = event.get("item") or {}
return _translate_item_event(etype, item)
return []
async def _iter_text_lines(stream: ByteReceiveStream):
text_stream = TextReceiveStream(stream, errors="replace")
buffer = ""
while True:
try:
chunk = await text_stream.receive()
except anyio.EndOfStream:
if buffer:
yield buffer
return
buffer += chunk
while True:
split_at = buffer.find("\n")
if split_at < 0:
break
line = buffer[: split_at + 1]
buffer = buffer[split_at + 1 :]
yield line
async def _drain_stderr(stderr: ByteReceiveStream, chunks: deque[str]) -> None:
try:
async for line in _iter_text_lines(stderr):
logger.debug("[codex][stderr] %s", line.rstrip())
chunks.append(line)
except Exception as e:
logger.debug("[codex][stderr] drain error: %s", e)
async def _wait_for_process(proc: Process, timeout: float) -> bool:
with anyio.move_on_after(timeout) as scope:
await proc.wait()
return scope.cancel_called
def _terminate_process(proc: Process) -> None:
if proc.returncode is not None:
return
if os.name == "posix" and proc.pid is not None:
try:
os.killpg(proc.pid, signal.SIGTERM)
return
except ProcessLookupError:
return
except Exception as e:
logger.debug("[codex] failed to terminate process group: %s", e)
try:
proc.terminate()
except ProcessLookupError:
return
def _kill_process(proc: Process) -> None:
if proc.returncode is not None:
return
if os.name == "posix" and proc.pid is not None:
try:
os.killpg(proc.pid, signal.SIGKILL)
return
except ProcessLookupError:
return
except Exception as e:
logger.debug("[codex] failed to kill process group: %s", e)
try:
proc.kill()
except ProcessLookupError:
return
@asynccontextmanager
async def manage_subprocess(*args, **kwargs):
"""Ensure subprocesses receive SIGTERM, then SIGKILL after a 2s timeout."""
if os.name == "posix":
kwargs.setdefault("start_new_session", True)
proc = await anyio.open_process(args, **kwargs)
try:
yield proc
finally:
if proc.returncode is None:
with anyio.CancelScope(shield=True):
_terminate_process(proc)
timed_out = await _wait_for_process(proc, timeout=2.0)
if timed_out:
_kill_process(proc)
await proc.wait()
class CodexRunner(ResumeRunnerMixin, Runner):
engine: EngineId = ENGINE
resume_re = _RESUME_RE
def __init__(
self,
*,
codex_cmd: str,
extra_args: list[str],
title: str = "Codex",
) -> None:
self.codex_cmd = codex_cmd
self.extra_args = extra_args
self.session_title = title
self._session_locks: WeakValueDictionary[str, anyio.Lock] = (
WeakValueDictionary()
)
def _lock_for(self, token: ResumeToken) -> anyio.Lock:
key = f"{token.engine}:{token.value}"
lock = self._session_locks.get(key)
if lock is None:
lock = anyio.Lock()
self._session_locks[key] = lock
return lock
async def run(
self, prompt: str, resume: ResumeToken | None
) -> AsyncIterator[TakopiEvent]:
resume_token = resume
if resume_token is not None and resume_token.engine != ENGINE:
raise RuntimeError(
f"resume token is for engine {resume_token.engine!r}, not {ENGINE!r}"
)
if resume_token is None:
async for evt in self._run(prompt, resume_token):
yield evt
return
lock = self._lock_for(resume_token)
async with lock:
async for evt in self._run(prompt, resume_token):
yield evt
async def _run( # noqa: C901
self,
prompt: str,
resume_token: ResumeToken | None,
) -> AsyncIterator[TakopiEvent]:
logger.info(
"[codex] start run resume=%r", resume_token.value if resume_token else None
)
logger.debug("[codex] prompt: %s", prompt)
args = [self.codex_cmd]
args.extend(self.extra_args)
args.extend(["exec", "--json"])
if resume_token:
args.extend(["resume", resume_token.value, "-"])
else:
args.append("-")
session_lock: anyio.Lock | None = None
session_lock_acquired = False
try:
async with manage_subprocess(
*args,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
) as proc:
if proc.stdin is None or proc.stdout is None or proc.stderr is None:
raise RuntimeError("codex exec failed to open subprocess pipes")
proc_stdin = proc.stdin
proc_stdout = proc.stdout
proc_stderr = proc.stderr
logger.debug("[codex] spawn pid=%s args=%r", proc.pid, args)
stderr_chunks: deque[str] = deque(maxlen=STDERR_TAIL_LINES)
rc: int | None = None
expected_session: ResumeToken | None = resume_token
found_session: ResumeToken | None = None
final_answer: str | None = None
note_seq = 0
did_emit_completed = False
turn_index = 0
def next_note_id() -> str:
nonlocal note_seq
note_seq += 1
return f"codex.note.{note_seq}"
async with anyio.create_task_group() as tg:
tg.start_soon(_drain_stderr, proc_stderr, stderr_chunks)
await proc_stdin.send(prompt.encode())
await proc_stdin.aclose()
async for raw_line in _iter_text_lines(proc_stdout):
raw = raw_line.rstrip("\n")
logger.debug("[codex][jsonl] %s", raw)
line = raw.strip()
if not line:
continue
if did_emit_completed:
continue
try:
evt = json.loads(line)
except json.JSONDecodeError:
logger.debug("[codex] invalid json line: %s", line)
note = _note_completed(
next_note_id(),
"invalid JSON from codex; ignoring line",
ok=False,
detail={"line": line},
)
yield note
continue
etype = evt.get("type")
if etype == "error":
message = str(evt.get("message") or "codex error")
fatal_flag = evt.get("fatal")
fatal = fatal_flag is True or fatal_flag is None
if fatal:
resume_for_completed = found_session or resume_token
yield _completed_event(
resume=resume_for_completed,
ok=False,
answer=final_answer or "",
error=message,
)
did_emit_completed = True
continue
note = _note_completed(
next_note_id(),
message,
ok=False,
detail={
"code": evt.get("code"),
"fatal": evt.get("fatal"),
},
)
yield note
continue
if etype == "turn.failed":
error = evt.get("error") or {}
message = str(error.get("message") or "codex turn failed")
resume_for_completed = found_session or resume_token
yield _completed_event(
resume=resume_for_completed,
ok=False,
answer=final_answer or "",
error=message,
)
did_emit_completed = True
continue
if etype == "turn.rate_limited":
retry_ms = evt.get("retry_after_ms")
message = "rate limited"
if isinstance(retry_ms, int):
message = f"rate limited (retry after {retry_ms}ms)"
note = _note_completed(next_note_id(), message, ok=False)
yield note
continue
if etype == "turn.started":
action_id = f"turn_{turn_index}"
turn_index += 1
yield _action_event(
phase="started",
action_id=action_id,
kind="turn",
title="turn started",
)
continue
if etype == "turn.completed":
resume_for_completed = found_session or resume_token
yield _completed_event(
resume=resume_for_completed,
ok=True,
answer=final_answer or "",
usage=evt.get("usage"),
)
did_emit_completed = True
continue
if evt.get("type") == "item.completed":
item = evt.get("item") or {}
item_type = item.get("type") or item.get("item_type")
if item_type == "assistant_message":
item_type = "agent_message"
if item_type == "agent_message" and isinstance(
item.get("text"), str
):
if final_answer is None:
final_answer = item["text"]
else:
logger.debug(
"[codex] emitted multiple agent messages; using the last one"
)
final_answer = item["text"]
for out_evt in translate_codex_event(
evt, title=self.session_title
):
if isinstance(out_evt, StartedEvent):
session = out_evt.resume
if found_session is None:
if session.engine != ENGINE:
raise RuntimeError(
f"codex emitted session token for engine {session.engine!r}"
)
if (
expected_session is not None
and session != expected_session
):
message = "codex emitted a different session id than expected"
raise RuntimeError(message)
if expected_session is None:
session_lock = self._lock_for(session)
await session_lock.acquire()
session_lock_acquired = True
found_session = session
yield out_evt
continue
yield out_evt
rc = await proc.wait()
logger.debug("[codex] process exit pid=%s rc=%s", proc.pid, rc)
if did_emit_completed:
return
if rc != 0:
stderr_text = "".join(stderr_chunks)
message = f"codex exec failed (rc={rc})."
yield _note_completed(
next_note_id(),
message,
ok=False,
detail={"stderr_tail": stderr_text},
)
resume_for_completed = found_session or resume_token
yield _completed_event(
resume=resume_for_completed,
ok=False,
answer=final_answer or "",
error=message,
)
return
if not found_session:
message = (
"codex exec finished but no session_id/thread_id was captured"
)
resume_for_completed = resume_token
yield _completed_event(
resume=resume_for_completed,
ok=False,
answer=final_answer or "",
error=message,
)
return
logger.info("[codex] done run session=%s", found_session.value)
yield _completed_event(
resume=found_session,
ok=True,
answer=final_answer or "",
)
finally:
if session_lock is not None and session_lock_acquired:
session_lock.release()
src/takopi/runners/mock.py
+232
View File
@@ -0,0 +1,232 @@
from __future__ import annotations
import uuid
from collections.abc import AsyncIterator, Awaitable, Callable, Iterable
from dataclasses import dataclass, replace
from typing import TypeAlias
from weakref import WeakValueDictionary
import anyio
from ..model import (
ActionEvent,
CompletedEvent,
EngineId,
ResumeToken,
StartedEvent,
TakopiEvent,
)
from ..runner import ResumeRunnerMixin, Runner, compile_resume_pattern
ENGINE: EngineId = EngineId("mock")
@dataclass(frozen=True, slots=True)
class Emit:
event: TakopiEvent
at: float | None = None
@dataclass(frozen=True, slots=True)
class Advance:
now: float
@dataclass(frozen=True, slots=True)
class Sleep:
seconds: float
@dataclass(frozen=True, slots=True)
class Wait:
event: anyio.Event
@dataclass(frozen=True, slots=True)
class Return:
answer: str
@dataclass(frozen=True, slots=True)
class Raise:
error: Exception
ScriptStep: TypeAlias = Emit | Advance | Sleep | Wait | Return | Raise
def _resume_token(engine: EngineId, value: str | None) -> ResumeToken:
return ResumeToken(engine=engine, value=value or uuid.uuid4().hex)
class MockRunner(ResumeRunnerMixin, Runner):
engine: EngineId
def __init__(
self,
*,
events: Iterable[TakopiEvent] | None = None,
answer: str = "",
engine: EngineId = ENGINE,
resume_value: str | None = None,
title: str | None = None,
) -> None:
self.engine = engine
self._events = list(events or [])
self._answer = answer
self._resume_value = resume_value
self.title = title or str(engine).title()
self._session_locks: WeakValueDictionary[str, anyio.Lock] = (
WeakValueDictionary()
)
self.resume_re = compile_resume_pattern(engine)
def _lock_for(self, token: ResumeToken) -> anyio.Lock:
key = f"{token.engine}:{token.value}"
lock = self._session_locks.get(key)
if lock is None:
lock = anyio.Lock()
self._session_locks[key] = lock
return lock
async def run(
self, prompt: str, resume: ResumeToken | None
) -> AsyncIterator[TakopiEvent]:
_ = prompt
token_value = None
if resume is not None:
if resume.engine != self.engine:
raise RuntimeError(
f"resume token is for engine {resume.engine!r}, not {self.engine!r}"
)
token_value = resume.value
if token_value is None:
token_value = self._resume_value
token = _resume_token(self.engine, token_value)
session_evt = StartedEvent(
engine=self.engine,
resume=token,
title=self.title,
)
lock = self._lock_for(token)
async with lock:
yield session_evt
for event in self._events:
event_out: TakopiEvent = event
if (
isinstance(event_out, ActionEvent)
and event_out.phase == "completed"
):
if event_out.ok is None:
event_out = replace(event_out, ok=True)
yield event_out
await anyio.sleep(0)
yield CompletedEvent(
engine=self.engine,
resume=token,
ok=True,
answer=self._answer,
)
class ScriptRunner(MockRunner):
def __init__(
self,
script: Iterable[ScriptStep],
*,
engine: EngineId = ENGINE,
resume_value: str | None = None,
emit_session_start: bool = True,
sleep: Callable[[float], Awaitable[None]] = anyio.sleep,
advance: Callable[[float], None] | None = None,
default_answer: str = "",
title: str | None = None,
) -> None:
super().__init__(
events=[],
answer=default_answer,
engine=engine,
resume_value=resume_value,
title=title,
)
self.calls: list[tuple[str, ResumeToken | None]] = []
self._script = list(script)
self._emit_session_start = emit_session_start
self._sleep = sleep
self._advance = advance
def _advance_to(self, now: float) -> None:
if self._advance is None:
raise RuntimeError("ScriptRunner advance callback is not configured.")
self._advance(now)
async def run(
self, prompt: str, resume: ResumeToken | None
) -> AsyncIterator[TakopiEvent]:
self.calls.append((prompt, resume))
_ = prompt
token_value = None
if resume is not None:
if resume.engine != self.engine:
raise RuntimeError(
f"resume token is for engine {resume.engine!r}, not {self.engine!r}"
)
token_value = resume.value
if token_value is None:
token_value = self._resume_value
token = _resume_token(self.engine, token_value)
session_evt = StartedEvent(
engine=self.engine,
resume=token,
title=self.title,
)
lock = self._lock_for(token)
async with lock:
if self._emit_session_start:
yield session_evt
await anyio.sleep(0)
for step in self._script:
if isinstance(step, Emit):
if step.at is not None:
self._advance_to(step.at)
event_out: TakopiEvent = step.event
if (
isinstance(event_out, ActionEvent)
and event_out.phase == "completed"
):
if event_out.ok is None:
event_out = replace(event_out, ok=True)
yield event_out
await anyio.sleep(0)
continue
if isinstance(step, Advance):
self._advance_to(step.now)
continue
if isinstance(step, Sleep):
await self._sleep(step.seconds)
continue
if isinstance(step, Wait):
await step.event.wait()
continue
if isinstance(step, Raise):
raise step.error
if isinstance(step, Return):
yield CompletedEvent(
engine=self.engine,
resume=token,
ok=True,
answer=step.answer,
)
return
raise RuntimeError(f"Unhandled script step: {step!r}")
yield CompletedEvent(
engine=self.engine,
resume=token,
ok=True,
answer=self._answer,
)
src/takopi/telegram.py
+33 -1
View File
@@ -1,7 +1,7 @@
from __future__ import annotations
import logging
from typing import Any
from typing import Any, Protocol
import httpx
@@ -11,6 +11,38 @@ logger = logging.getLogger(__name__)
logger.addFilter(RedactTokenFilter())
class BotClient(Protocol):
async def close(self) -> None: ...
async def get_updates(
self,
offset: int | None,
timeout_s: int = 50,
allowed_updates: list[str] | None = None,
) -> list[dict] | None: ...
async def send_message(
self,
chat_id: int,
text: str,
reply_to_message_id: int | None = None,
disable_notification: bool | None = False,
entities: list[dict] | None = None,
parse_mode: str | None = None,
) -> dict | None: ...
async def edit_message_text(
self,
chat_id: int,
message_id: int,
text: str,
entities: list[dict] | None = None,
parse_mode: str | None = None,
) -> dict | None: ...
async def delete_message(self, chat_id: int, message_id: int) -> bool: ...
class TelegramClient:
def __init__(
self,
tests/__init__.py
+1
View File
@@ -0,0 +1 @@
"""Test helpers package."""
tests/factories.py
+64
View File
@@ -0,0 +1,64 @@
from __future__ import annotations
from typing import Any
from takopi.model import (
Action,
ActionEvent,
ActionKind,
EngineId,
ResumeToken,
StartedEvent,
TakopiEvent,
)
def session_started(engine: str, value: str, title: str = "Codex") -> TakopiEvent:
engine_id = EngineId(engine)
return StartedEvent(
engine=engine_id,
resume=ResumeToken(engine=engine_id, value=value),
title=title,
)
def action_started(
action_id: str,
kind: ActionKind,
title: str,
detail: dict[str, Any] | None = None,
engine: str = "codex",
) -> TakopiEvent:
engine_id = EngineId(engine)
return ActionEvent(
engine=engine_id,
action=Action(
id=action_id,
kind=kind,
title=title,
detail=detail or {},
),
phase="started",
)
def action_completed(
action_id: str,
kind: ActionKind,
title: str,
ok: bool,
detail: dict[str, Any] | None = None,
engine: str = "codex",
) -> TakopiEvent:
engine_id = EngineId(engine)
return ActionEvent(
engine=engine_id,
action=Action(
id=action_id,
kind=kind,
title=title,
detail=detail or {},
),
phase="completed",
ok=ok,
)
tests/test_exec_bridge.py
+430 -267
View File
@@ -1,106 +1,116 @@
import uuid
import anyio
import pytest
from takopi.exec_bridge import (
extract_session_id,
prepare_telegram,
resolve_resume_session,
truncate_for_telegram,
)
from takopi import engines
from takopi.markdown import prepare_telegram, truncate_for_telegram
from takopi.model import EngineId, ResumeToken, TakopiEvent
from takopi.runners.codex import CodexRunner
from takopi.runners.mock import Advance, Emit, Raise, Return, ScriptRunner, Sleep, Wait
from tests.factories import action_completed, action_started
CODEX_ENGINE = EngineId("codex")
def _patch_config(monkeypatch, config):
from pathlib import Path
from takopi import exec_bridge
from takopi import cli
monkeypatch.setattr(
exec_bridge,
cli,
"load_telegram_config",
lambda: (config, Path("takopi.toml")),
)
def test_parse_bridge_config_rejects_empty_token(monkeypatch) -> None:
from takopi import exec_bridge
from takopi import cli
_patch_config(monkeypatch, {"bot_token": " ", "chat_id": 123})
with pytest.raises(exec_bridge.ConfigError, match="bot_token"):
exec_bridge._parse_bridge_config(final_notify=True, profile=None)
with pytest.raises(cli.ConfigError, match="bot_token"):
cli._parse_bridge_config(
final_notify=True,
backend=engines.get_backend("codex"),
engine_overrides={},
)
def test_parse_bridge_config_rejects_string_chat_id(monkeypatch) -> None:
from takopi import exec_bridge
from takopi import cli
_patch_config(monkeypatch, {"bot_token": "token", "chat_id": "123"})
with pytest.raises(exec_bridge.ConfigError, match="chat_id"):
exec_bridge._parse_bridge_config(final_notify=True, profile=None)
with pytest.raises(cli.ConfigError, match="chat_id"):
cli._parse_bridge_config(
final_notify=True,
backend=engines.get_backend("codex"),
engine_overrides={},
)
def test_extract_session_id_finds_uuid_v7() -> None:
def test_codex_extract_resume_finds_command() -> None:
uuid = "019b66fc-64c2-7a71-81cd-081c504cfeb2"
text = f"resume: `{uuid}`"
runner = CodexRunner(codex_cmd="codex", extra_args=[])
text = f"`codex resume {uuid}`"
assert extract_session_id(text) == uuid
assert runner.extract_resume(text) == ResumeToken(engine=CODEX_ENGINE, value=uuid)
def test_extract_session_id_requires_resume_line() -> None:
uuid = "019b66fc-64c2-7a71-81cd-081c504cfeb2"
text = f"here is a uuid {uuid}"
assert extract_session_id(text) is None
def test_extract_session_id_uses_last_resume_line() -> None:
def test_codex_extract_resume_uses_last_resume_line() -> None:
uuid_first = "019b66fc-64c2-7a71-81cd-081c504cfeb2"
uuid_last = "123e4567-e89b-12d3-a456-426614174000"
text = f"resume: `{uuid_first}`\n\nresume: `{uuid_last}`"
runner = CodexRunner(codex_cmd="codex", extra_args=[])
text = f"`codex resume {uuid_first}`\n\n`codex resume {uuid_last}`"
assert extract_session_id(text) == uuid_last
def test_extract_session_id_ignores_malformed_resume_line() -> None:
text = "resume: not-a-uuid"
assert extract_session_id(text) is None
def test_resolve_resume_session_prefers_message_text() -> None:
uuid_message = "123e4567-e89b-12d3-a456-426614174000"
uuid_reply = "019b66fc-64c2-7a71-81cd-081c504cfeb2"
assert (
resolve_resume_session(f"resume: `{uuid_message}`", f"resume: `{uuid_reply}`")
== uuid_message
assert runner.extract_resume(text) == ResumeToken(
engine=CODEX_ENGINE, value=uuid_last
)
def test_resolve_resume_session_uses_reply_when_missing() -> None:
uuid_reply = "019b66fc-64c2-7a71-81cd-081c504cfeb2"
def test_codex_extract_resume_ignores_malformed_resume_line() -> None:
runner = CodexRunner(codex_cmd="codex", extra_args=[])
text = "codex resume"
assert (
resolve_resume_session("no resume here", f"resume: `{uuid_reply}`")
== uuid_reply
)
assert runner.extract_resume(text) is None
def test_codex_extract_resume_accepts_plain_line() -> None:
uuid = "019b66fc-64c2-7a71-81cd-081c504cfeb2"
runner = CodexRunner(codex_cmd="codex", extra_args=[])
text = f"codex resume {uuid}"
assert runner.extract_resume(text) == ResumeToken(engine=CODEX_ENGINE, value=uuid)
def test_codex_extract_resume_accepts_uuid7() -> None:
uuid7 = getattr(uuid, "uuid7", None)
assert uuid7 is not None
token = str(uuid7())
runner = CodexRunner(codex_cmd="codex", extra_args=[])
text = f"`codex resume {token}`"
assert runner.extract_resume(text) == ResumeToken(engine=CODEX_ENGINE, value=token)
def test_truncate_for_telegram_preserves_resume_line() -> None:
uuid = "019b66fc-64c2-7a71-81cd-081c504cfeb2"
md = ("x" * 10_000) + f"\nresume: `{uuid}`"
md = ("x" * 10_000) + f"\n`codex resume {uuid}`"
out = truncate_for_telegram(md, 400)
runner = CodexRunner(codex_cmd="codex", extra_args=[])
out = truncate_for_telegram(md, 400, is_resume_line=runner.is_resume_line)
assert len(out) <= 400
assert uuid in out
assert out.rstrip().endswith(f"resume: `{uuid}`")
assert f"codex resume {uuid}" in out
assert out.rstrip().endswith(f"`codex resume {uuid}`")
def test_truncate_for_telegram_keeps_last_non_empty_line() -> None:
md = "intro\n\n" + ("x" * 500) + "\nlast line"
out = truncate_for_telegram(md, 120)
out = truncate_for_telegram(md, 120, is_resume_line=lambda _line: False)
assert len(out) <= 120
assert out.rstrip().endswith("last line")
@@ -168,18 +178,19 @@ class _FakeBot:
self.delete_calls.append({"chat_id": chat_id, "message_id": message_id})
return True
async def get_updates(
self,
offset: int | None,
timeout_s: int = 50,
allowed_updates: list[str] | None = None,
) -> list[dict] | None:
_ = offset
_ = timeout_s
_ = allowed_updates
return []
class _FakeRunner:
def __init__(self, *, answer: str, saw_agent_message: bool = True) -> None:
self._answer = answer
self._saw_agent_message = saw_agent_message
async def run_serialized(self, *_args, **_kwargs) -> tuple[str, str, bool]:
return (
"019b66fc-64c2-7a71-81cd-081c504cfeb2",
self._answer,
self._saw_agent_message,
)
async def close(self) -> None:
return None
class _FakeClock:
@@ -211,54 +222,28 @@ class _FakeClock:
await self._sleep_event.wait()
class _FakeRunnerWithEvents:
def __init__(
self,
*,
events: list[dict],
times: list[float],
clock: _FakeClock,
answer: str = "ok",
session_id: str = "019b66fc-64c2-7a71-81cd-081c504cfeb2",
advance_after: float | None = None,
hold: anyio.Event | None = None,
) -> None:
self._events = events
self._times = times
self._clock = clock
self._answer = answer
self._session_id = session_id
self._advance_after = advance_after
self._hold = hold
async def run_serialized(self, *_args, **kwargs) -> tuple[str, str, bool]:
on_event = kwargs.get("on_event")
if on_event is not None:
for when, event in zip(self._times, self._events, strict=False):
self._clock.set(when)
await on_event(event)
await anyio.sleep(0)
if self._advance_after is not None:
self._clock.set(self._advance_after)
await anyio.sleep(0)
if self._hold is not None:
await self._hold.wait()
return (self._session_id, self._answer, True)
def _return_runner(
*, answer: str = "ok", resume_value: str | None = None
) -> ScriptRunner:
return ScriptRunner(
[Return(answer=answer)],
engine=CODEX_ENGINE,
resume_value=resume_value,
)
@pytest.mark.anyio
async def test_final_notify_sends_loud_final_message() -> None:
from takopi.exec_bridge import BridgeConfig, handle_message
from takopi.bridge import BridgeConfig, handle_message
bot = _FakeBot()
runner = _FakeRunner(answer="ok")
runner = _return_runner(answer="ok")
cfg = BridgeConfig(
bot=bot, # type: ignore[arg-type]
runner=runner, # type: ignore[arg-type]
bot=bot,
runner=runner,
chat_id=123,
final_notify=True,
startup_msg="",
max_concurrency=1,
)
await handle_message(
@@ -266,7 +251,7 @@ async def test_final_notify_sends_loud_final_message() -> None:
chat_id=123,
user_msg_id=10,
text="hi",
resume_session=None,
resume_token=None,
)
assert len(bot.send_calls) == 2
@@ -275,18 +260,47 @@ async def test_final_notify_sends_loud_final_message() -> None:
@pytest.mark.anyio
async def test_new_final_message_forces_notification_when_too_long_to_edit() -> None:
from takopi.exec_bridge import BridgeConfig, handle_message
async def test_handle_message_strips_resume_line_from_prompt() -> None:
from takopi.bridge import BridgeConfig, handle_message
bot = _FakeBot()
runner = _FakeRunner(answer="x" * 10_000)
runner = ScriptRunner([Return(answer="ok")], engine=CODEX_ENGINE)
cfg = BridgeConfig(
bot=bot, # type: ignore[arg-type]
runner=runner, # type: ignore[arg-type]
bot=bot,
runner=runner,
chat_id=123,
final_notify=True,
startup_msg="",
)
resume = ResumeToken(engine=CODEX_ENGINE, value="sid")
text = "do this\n`codex resume sid`\nand that"
await handle_message(
cfg,
chat_id=123,
user_msg_id=10,
text=text,
resume_token=resume,
)
assert runner.calls
prompt, passed_resume = runner.calls[0]
assert prompt == "do this\nand that"
assert passed_resume == resume
@pytest.mark.anyio
async def test_new_final_message_forces_notification_when_too_long_to_edit() -> None:
from takopi.bridge import BridgeConfig, handle_message
bot = _FakeBot()
runner = _return_runner(answer="x" * 10_000)
cfg = BridgeConfig(
bot=bot,
runner=runner,
chat_id=123,
final_notify=False,
startup_msg="",
max_concurrency=1,
)
await handle_message(
@@ -294,7 +308,7 @@ async def test_new_final_message_forces_notification_when_too_long_to_edit() ->
chat_id=123,
user_msg_id=10,
text="hi",
resume_session=None,
resume_token=None,
)
assert len(bot.send_calls) == 2
@@ -304,43 +318,30 @@ async def test_new_final_message_forces_notification_when_too_long_to_edit() ->
@pytest.mark.anyio
async def test_progress_edits_are_rate_limited() -> None:
from takopi.exec_bridge import BridgeConfig, handle_message
from takopi.bridge import BridgeConfig, handle_message
bot = _FakeBot()
clock = _FakeClock()
events = [
{
"type": "item.started",
"item": {
"id": "item_0",
"type": "command_execution",
"command": "echo 1",
"status": "in_progress",
},
},
{
"type": "item.started",
"item": {
"id": "item_1",
"type": "command_execution",
"command": "echo 2",
"status": "in_progress",
},
},
events: list[TakopiEvent] = [
action_started("item_0", "command", "echo 1"),
action_started("item_1", "command", "echo 2"),
]
runner = _FakeRunnerWithEvents(
events=events,
times=[0.2, 0.4],
clock=clock,
advance_after=1.0,
runner = ScriptRunner(
[
Emit(events[0], at=0.2),
Emit(events[1], at=0.4),
Advance(1.0),
Return(answer="ok"),
],
engine=CODEX_ENGINE,
advance=clock.set,
)
cfg = BridgeConfig(
bot=bot, # type: ignore[arg-type]
runner=runner, # type: ignore[arg-type]
bot=bot,
runner=runner,
chat_id=123,
final_notify=True,
startup_msg="",
max_concurrency=1,
)
await handle_message(
@@ -348,7 +349,7 @@ async def test_progress_edits_are_rate_limited() -> None:
chat_id=123,
user_msg_id=10,
text="hi",
resume_session=None,
resume_token=None,
clock=clock,
sleep=clock.sleep,
progress_edit_every=1.0,
@@ -360,45 +361,31 @@ async def test_progress_edits_are_rate_limited() -> None:
@pytest.mark.anyio
async def test_progress_edits_do_not_sleep_again_without_new_events() -> None:
from takopi.exec_bridge import BridgeConfig, handle_message
from takopi.bridge import BridgeConfig, handle_message
bot = _FakeBot()
clock = _FakeClock()
hold = anyio.Event()
events = [
{
"type": "item.started",
"item": {
"id": "item_0",
"type": "command_execution",
"command": "echo 1",
"status": "in_progress",
},
},
{
"type": "item.started",
"item": {
"id": "item_1",
"type": "command_execution",
"command": "echo 2",
"status": "in_progress",
},
},
events: list[TakopiEvent] = [
action_started("item_0", "command", "echo 1"),
action_started("item_1", "command", "echo 2"),
]
runner = _FakeRunnerWithEvents(
events=events,
times=[0.2, 0.4],
clock=clock,
advance_after=None,
hold=hold,
runner = ScriptRunner(
[
Emit(events[0], at=0.2),
Emit(events[1], at=0.4),
Wait(hold),
Return(answer="ok"),
],
engine=CODEX_ENGINE,
advance=clock.set,
)
cfg = BridgeConfig(
bot=bot, # type: ignore[arg-type]
runner=runner, # type: ignore[arg-type]
bot=bot,
runner=runner,
chat_id=123,
final_notify=True,
startup_msg="",
max_concurrency=1,
)
async def run_handle_message() -> None:
@@ -407,7 +394,7 @@ async def test_progress_edits_do_not_sleep_again_without_new_events() -> None:
chat_id=123,
user_msg_id=10,
text="hi",
resume_session=None,
resume_token=None,
clock=clock,
sleep=clock.sleep,
progress_edit_every=1.0,
@@ -443,46 +430,37 @@ async def test_progress_edits_do_not_sleep_again_without_new_events() -> None:
@pytest.mark.anyio
async def test_bridge_flow_sends_progress_edits_and_final_resume() -> None:
from takopi.exec_bridge import BridgeConfig, handle_message
from takopi.bridge import BridgeConfig, handle_message
bot = _FakeBot()
clock = _FakeClock()
events = [
{
"type": "item.started",
"item": {
"id": "item_0",
"type": "command_execution",
"command": "echo ok",
"status": "in_progress",
},
},
{
"type": "item.completed",
"item": {
"id": "item_0",
"type": "command_execution",
"command": "echo ok",
"exit_code": 0,
"status": "completed",
},
},
events: list[TakopiEvent] = [
action_started("item_0", "command", "echo ok"),
action_completed(
"item_0",
"command",
"echo ok",
ok=True,
detail={"exit_code": 0},
),
]
session_id = "123e4567-e89b-12d3-a456-426614174000"
runner = _FakeRunnerWithEvents(
events=events,
times=[0.0, 2.1],
clock=clock,
answer="done",
session_id=session_id,
runner = ScriptRunner(
[
Emit(events[0], at=0.0),
Emit(events[1], at=2.1),
Return(answer="done"),
],
engine=CODEX_ENGINE,
advance=clock.set,
resume_value=session_id,
)
cfg = BridgeConfig(
bot=bot, # type: ignore[arg-type]
runner=runner, # type: ignore[arg-type]
bot=bot,
runner=runner,
chat_id=123,
final_notify=True,
startup_msg="",
max_concurrency=1,
)
await handle_message(
@@ -490,7 +468,7 @@ async def test_bridge_flow_sends_progress_edits_and_final_resume() -> None:
chat_id=123,
user_msg_id=42,
text="do it",
resume_session=None,
resume_token=None,
clock=clock,
sleep=clock.sleep,
progress_edit_every=1.0,
@@ -500,23 +478,22 @@ async def test_bridge_flow_sends_progress_edits_and_final_resume() -> None:
assert "working" in bot.send_calls[0]["text"]
assert len(bot.edit_calls) >= 1
assert session_id in bot.send_calls[-1]["text"]
assert "resume:" in bot.send_calls[-1]["text"].lower()
assert "codex resume" in bot.send_calls[-1]["text"].lower()
assert len(bot.delete_calls) == 1
@pytest.mark.anyio
async def test_handle_cancel_without_reply_prompts_user() -> None:
from takopi.exec_bridge import BridgeConfig, _handle_cancel
from takopi.bridge import BridgeConfig, _handle_cancel
bot = _FakeBot()
runner = _FakeRunner(answer="ok")
runner = _return_runner(answer="ok")
cfg = BridgeConfig(
bot=bot, # type: ignore[arg-type]
runner=runner, # type: ignore[arg-type]
bot=bot,
runner=runner,
chat_id=123,
final_notify=True,
startup_msg="",
max_concurrency=1,
)
msg = {"chat": {"id": 123}, "message_id": 10}
running_tasks: dict = {}
@@ -529,17 +506,16 @@ async def test_handle_cancel_without_reply_prompts_user() -> None:
@pytest.mark.anyio
async def test_handle_cancel_with_no_progress_message_says_nothing_running() -> None:
from takopi.exec_bridge import BridgeConfig, _handle_cancel
from takopi.bridge import BridgeConfig, _handle_cancel
bot = _FakeBot()
runner = _FakeRunner(answer="ok")
runner = _return_runner(answer="ok")
cfg = BridgeConfig(
bot=bot, # type: ignore[arg-type]
runner=runner, # type: ignore[arg-type]
bot=bot,
runner=runner,
chat_id=123,
final_notify=True,
startup_msg="",
max_concurrency=1,
)
msg = {
"chat": {"id": 123},
@@ -556,17 +532,16 @@ async def test_handle_cancel_with_no_progress_message_says_nothing_running() ->
@pytest.mark.anyio
async def test_handle_cancel_with_finished_task_says_nothing_running() -> None:
from takopi.exec_bridge import BridgeConfig, _handle_cancel
from takopi.bridge import BridgeConfig, _handle_cancel
bot = _FakeBot()
runner = _FakeRunner(answer="ok")
runner = _return_runner(answer="ok")
cfg = BridgeConfig(
bot=bot, # type: ignore[arg-type]
runner=runner, # type: ignore[arg-type]
bot=bot,
runner=runner,
chat_id=123,
final_notify=True,
startup_msg="",
max_concurrency=1,
)
progress_id = 99
msg = {
@@ -584,17 +559,16 @@ async def test_handle_cancel_with_finished_task_says_nothing_running() -> None:
@pytest.mark.anyio
async def test_handle_cancel_cancels_running_task() -> None:
from takopi.exec_bridge import BridgeConfig, _handle_cancel
from takopi.bridge import BridgeConfig, _handle_cancel
bot = _FakeBot()
runner = _FakeRunner(answer="ok")
runner = _return_runner(answer="ok")
cfg = BridgeConfig(
bot=bot, # type: ignore[arg-type]
runner=runner, # type: ignore[arg-type]
bot=bot,
runner=runner,
chat_id=123,
final_notify=True,
startup_msg="",
max_concurrency=1,
)
progress_id = 42
msg = {
@@ -603,49 +577,33 @@ async def test_handle_cancel_cancels_running_task() -> None:
"reply_to_message": {"message_id": progress_id},
}
from takopi.exec_bridge import RunningTask
from takopi.bridge import RunningTask
cancelled_event = anyio.Event()
cancel_scope = anyio.CancelScope()
running_task = RunningTask(scope=cancel_scope)
async def sleeper() -> None:
with cancel_scope:
try:
await anyio.sleep(10)
except anyio.get_cancelled_exc_class():
cancelled_event.set()
return
async with anyio.create_task_group() as tg:
tg.start_soon(sleeper)
running_tasks = {progress_id: running_task}
await _handle_cancel(cfg, msg, running_tasks)
await cancelled_event.wait()
running_task = RunningTask()
running_tasks = {progress_id: running_task}
await _handle_cancel(cfg, msg, running_tasks)
assert running_task.cancel_requested.is_set() is True
assert len(bot.send_calls) == 0 # No error message sent
@pytest.mark.anyio
async def test_handle_cancel_only_cancels_matching_progress_message() -> None:
from takopi.exec_bridge import BridgeConfig, _handle_cancel
from takopi.bridge import BridgeConfig, _handle_cancel
bot = _FakeBot()
runner = _FakeRunner(answer="ok")
runner = _return_runner(answer="ok")
cfg = BridgeConfig(
bot=bot, # type: ignore[arg-type]
runner=runner, # type: ignore[arg-type]
bot=bot,
runner=runner,
chat_id=123,
final_notify=True,
startup_msg="",
max_concurrency=1,
)
from takopi.exec_bridge import RunningTask
from takopi.bridge import RunningTask
scope_first = anyio.CancelScope()
scope_second = anyio.CancelScope()
task_first = RunningTask(scope=scope_first)
task_second = RunningTask(scope=scope_second)
task_first = RunningTask()
task_second = RunningTask()
msg = {
"chat": {"id": 123},
"message_id": 10,
@@ -655,37 +613,56 @@ async def test_handle_cancel_only_cancels_matching_progress_message() -> None:
await _handle_cancel(cfg, msg, running_tasks)
assert scope_first.cancel_called is True
assert scope_second.cancel_called is False
assert task_first.cancel_requested.is_set() is True
assert task_second.cancel_requested.is_set() is False
assert len(bot.send_calls) == 0
class _FakeRunnerCancellable:
def __init__(self, session_id: str = "019b66fc-64c2-7a71-81cd-081c504cfeb2"):
self._session_id = session_id
def test_cancel_command_accepts_extra_text() -> None:
from takopi.bridge import _is_cancel_command
async def run_serialized(self, *_args, **kwargs) -> tuple[str, str, bool]:
on_event = kwargs.get("on_event")
if on_event:
await on_event({"type": "thread.started", "thread_id": self._session_id})
await anyio.sleep(10) # Will be cancelled
return (self._session_id, "ok", True)
assert _is_cancel_command("/cancel now") is True
assert _is_cancel_command("/cancel@takopi please") is True
assert _is_cancel_command("/cancelled") is False
def test_resume_attempt_does_not_trigger_on_plain_resume_word() -> None:
from takopi.bridge import _resume_attempt
attempt, engine = _resume_attempt("resume abc123")
assert attempt is False
assert engine is None
def test_resume_warning_for_other_engine() -> None:
from takopi.bridge import _resume_attempt, _resume_warning_text
attempt, engine = _resume_attempt("claude resume abc123")
assert attempt is True
assert engine == "claude"
warning = _resume_warning_text(engine, "codex")
assert "claude" in warning.lower()
assert "codex" in warning.lower()
@pytest.mark.anyio
async def test_handle_message_cancelled_renders_cancelled_state() -> None:
from takopi.exec_bridge import BridgeConfig, handle_message
from takopi.bridge import BridgeConfig, handle_message
bot = _FakeBot()
session_id = "019b66fc-64c2-7a71-81cd-081c504cfeb2"
runner = _FakeRunnerCancellable(session_id=session_id)
hold = anyio.Event()
runner = ScriptRunner(
[Wait(hold)],
engine=CODEX_ENGINE,
resume_value=session_id,
)
cfg = BridgeConfig(
bot=bot, # type: ignore[arg-type]
runner=runner, # type: ignore[arg-type]
bot=bot,
runner=runner,
chat_id=123,
final_notify=True,
startup_msg="",
max_concurrency=1,
)
running_tasks: dict = {}
@@ -695,7 +672,7 @@ async def test_handle_message_cancelled_renders_cancelled_state() -> None:
chat_id=123,
user_msg_id=10,
text="do something",
resume_session=None,
resume_token=None,
running_tasks=running_tasks,
)
@@ -706,10 +683,196 @@ async def test_handle_message_cancelled_renders_cancelled_state() -> None:
break
await anyio.sleep(0)
assert running_tasks
running_tasks[next(iter(running_tasks))].scope.cancel()
running_task = running_tasks[next(iter(running_tasks))]
with anyio.fail_after(1):
await running_task.resume_ready.wait()
running_task.cancel_requested.set()
assert len(bot.send_calls) == 1 # Progress message
assert len(bot.edit_calls) >= 1
last_edit = bot.edit_calls[-1]["text"]
assert "cancelled" in last_edit.lower()
assert session_id in last_edit
@pytest.mark.anyio
async def test_handle_message_error_preserves_resume_token() -> None:
from takopi.bridge import BridgeConfig, handle_message
bot = _FakeBot()
session_id = "019b66fc-64c2-7a71-81cd-081c504cfeb2"
runner = ScriptRunner(
[Raise(RuntimeError("boom"))],
engine=CODEX_ENGINE,
resume_value=session_id,
)
cfg = BridgeConfig(
bot=bot,
runner=runner,
chat_id=123,
final_notify=True,
startup_msg="",
)
await handle_message(
cfg,
chat_id=123,
user_msg_id=10,
text="do something",
resume_token=None,
)
assert bot.edit_calls
last_edit = bot.edit_calls[-1]["text"]
assert "error" in last_edit.lower()
assert session_id in last_edit
assert "codex resume" in last_edit.lower()
@pytest.mark.anyio
async def test_send_with_resume_waits_for_token() -> None:
from takopi.bridge import RunningTask, _send_with_resume
bot = _FakeBot()
sent: list[tuple[int, int, str, ResumeToken | None]] = []
def enqueue(chat_id: int, user_msg_id: int, text: str, resume: ResumeToken) -> None:
sent.append((chat_id, user_msg_id, text, resume))
running_task = RunningTask()
async def trigger_resume() -> None:
await anyio.sleep(0)
running_task.resume = ResumeToken(engine=CODEX_ENGINE, value="abc123")
running_task.resume_ready.set()
async with anyio.create_task_group() as tg:
tg.start_soon(trigger_resume)
await _send_with_resume(
bot,
enqueue,
running_task,
123,
10,
"hello",
)
assert sent == [
(123, 10, "hello", ResumeToken(engine=CODEX_ENGINE, value="abc123"))
]
@pytest.mark.anyio
async def test_send_with_resume_reports_when_missing() -> None:
from takopi.bridge import RunningTask, _send_with_resume
bot = _FakeBot()
sent: list[tuple[int, int, str, ResumeToken | None]] = []
def enqueue(chat_id: int, user_msg_id: int, text: str, resume: ResumeToken) -> None:
sent.append((chat_id, user_msg_id, text, resume))
running_task = RunningTask()
running_task.done.set()
await _send_with_resume(
bot,
enqueue,
running_task,
123,
10,
"hello",
)
assert sent == []
assert bot.send_calls
assert "resume token" in bot.send_calls[-1]["text"].lower()
@pytest.mark.anyio
async def test_run_main_loop_routes_reply_to_running_resume() -> None:
from takopi.bridge import BridgeConfig, _run_main_loop
progress_ready = anyio.Event()
stop_polling = anyio.Event()
reply_ready = anyio.Event()
hold = anyio.Event()
class _BotWithProgress(_FakeBot):
def __init__(self) -> None:
super().__init__()
self.progress_id: int | None = None
async def send_message(
self,
chat_id: int,
text: str,
reply_to_message_id: int | None = None,
disable_notification: bool | None = False,
entities: list[dict] | None = None,
parse_mode: str | None = None,
) -> dict:
msg = await super().send_message(
chat_id=chat_id,
text=text,
reply_to_message_id=reply_to_message_id,
disable_notification=disable_notification,
entities=entities,
parse_mode=parse_mode,
)
if self.progress_id is None and reply_to_message_id is not None:
self.progress_id = int(msg["message_id"])
progress_ready.set()
return msg
bot = _BotWithProgress()
resume_value = "abc123"
runner = ScriptRunner(
[Wait(hold), Sleep(0.05), Return(answer="ok")],
engine=CODEX_ENGINE,
resume_value=resume_value,
)
cfg = BridgeConfig(
bot=bot,
runner=runner,
chat_id=123,
final_notify=True,
startup_msg="",
)
async def poller(_cfg: BridgeConfig):
yield {
"message_id": 1,
"text": "first",
"chat": {"id": 123},
"from": {"id": 123},
}
await progress_ready.wait()
assert bot.progress_id is not None
reply_ready.set()
yield {
"message_id": 2,
"text": "followup",
"chat": {"id": 123},
"from": {"id": 123},
"reply_to_message": {"message_id": bot.progress_id},
}
await stop_polling.wait()
async with anyio.create_task_group() as tg:
tg.start_soon(_run_main_loop, cfg, poller)
try:
with anyio.fail_after(2):
await reply_ready.wait()
await anyio.sleep(0)
hold.set()
with anyio.fail_after(2):
while len(runner.calls) < 2:
await anyio.sleep(0)
assert runner.calls[1][1] == ResumeToken(
engine=CODEX_ENGINE, value=resume_value
)
finally:
hold.set()
stop_polling.set()
tg.cancel_scope.cancel()
tests/test_exec_render.py
+170 -131
View File
@@ -1,146 +1,116 @@
import json
from pathlib import Path
from typing import cast
from types import SimpleNamespace
from takopi.exec_render import ExecProgressRenderer, render_event_cli, render_markdown
def _loads(lines: str) -> list[dict]:
return [json.loads(line) for line in lines.strip().splitlines() if line.strip()]
FIXTURE_PATH = Path(__file__).resolve().parent / "fixtures" / "codex.jsonl"
ALL_FORMATS_FIXTURE_PATH = (
Path(__file__).resolve().parent / "fixtures" / "codex_exec_json_all_formats.jsonl"
)
ALL_FORMATS_GOLDEN_PATH = (
Path(__file__).resolve().parent / "fixtures" / "codex_exec_json_all_formats.txt"
from takopi.markdown import render_markdown
from takopi.model import TakopiEvent
from takopi.render import ExecProgressRenderer, render_event_cli
from tests.factories import (
action_completed,
action_started,
session_started,
)
SAMPLE_STREAM = """
{"type":"thread.started","thread_id":"0199a213-81c0-7800-8aa1-bbab2a035a53"}
{"type":"turn.started"}
{"type":"item.completed","item":{"id":"item_0","type":"reasoning","text":"**Searching for README files**"}}
{"type":"item.started","item":{"id":"item_1","type":"command_execution","command":"bash -lc ls","aggregated_output":"","status":"in_progress"}}
{"type":"item.completed","item":{"id":"item_1","type":"command_execution","command":"bash -lc ls","aggregated_output":"2025-09-11\\nAGENTS.md\\nCHANGELOG.md\\ncliff.toml\\ncodex-cli\\ncodex-rs\\ndocs\\nexamples\\nflake.lock\\nflake.nix\\nLICENSE\\nnode_modules\\nNOTICE\\npackage.json\\npnpm-lock.yaml\\npnpm-workspace.yaml\\nPNPM.md\\nREADME.md\\nscripts\\nsdk\\ntmp\\n","exit_code":0,"status":"completed"}}
{"type":"item.completed","item":{"id":"item_2","type":"reasoning","text":"**Checking repository root for README**"}}
{"type":"item.completed","item":{"id":"item_3","type":"agent_message","text":"Yep — there’s a `README.md` in the repository root."}}
{"type":"turn.completed","usage":{"input_tokens":24763,"cached_input_tokens":24448,"output_tokens":122}}
"""
def _format_resume(token) -> str:
return f"`codex resume {token.value}`"
def test_render_event_cli_sample_stream() -> None:
last_turn = None
SAMPLE_EVENTS: list[TakopiEvent] = [
session_started("codex", "0199a213-81c0-7800-8aa1-bbab2a035a53", title="Codex"),
action_started("a-1", "command", "bash -lc ls"),
action_completed(
"a-1",
"command",
"bash -lc ls",
ok=True,
detail={"exit_code": 0},
),
action_completed("a-2", "note", "Checking repository root for README", ok=True),
]
def test_render_event_cli_sample_events() -> None:
out: list[str] = []
for evt in _loads(SAMPLE_STREAM):
last_turn, lines = render_event_cli(evt, last_turn)
out.extend(lines)
for evt in SAMPLE_EVENTS:
out.extend(render_event_cli(evt))
assert out == [
"thread started",
"turn started",
"0. **Searching for README files**",
"1. ▸ `bash -lc ls`",
"1. ✓ `bash -lc ls`",
"2. **Checking repository root for README**",
"assistant:",
" Yep — there’s a `README.md` in the repository root.",
"turn completed",
"codex",
"▸ `bash -lc ls`",
"✓ `bash -lc ls`",
"✓ Checking repository root for README",
]
def test_render_event_cli_real_run_fixture() -> None:
events = _loads(FIXTURE_PATH.read_text(encoding="utf-8"))
last_turn = None
def test_render_event_cli_handles_action_kinds() -> None:
events: list[TakopiEvent] = [
action_completed(
"c-1", "command", "pytest -q", ok=False, detail={"exit_code": 1}
),
action_completed(
"s-1",
"web_search",
"python jsonlines parser handle unknown fields",
ok=True,
),
action_completed("t-1", "tool", "github.search_issues", ok=True),
action_completed(
"f-1",
"file_change",
"2 files",
ok=True,
detail={
"changes": [
{"path": "README.md", "kind": "add"},
{"path": "src/compute_answer.py", "kind": "update"},
]
},
),
action_completed("n-1", "note", "stream error", ok=False),
]
out: list[str] = []
for evt in events:
last_turn, lines = render_event_cli(evt, last_turn)
out.extend(lines)
out.extend(render_event_cli(evt))
print("\n".join(out))
assert out[0] == "thread started"
assert "turn started" in out
assert any(line.startswith("0. ▸ `") for line in out)
assert any(line.startswith("0. ✓ `") for line in out)
assert "assistant:" in out
assert any("takopi" in line for line in out)
assert out[-1] == "turn completed"
def test_render_event_cli_all_formats_fixture() -> None:
events = _loads(ALL_FORMATS_FIXTURE_PATH.read_text(encoding="utf-8"))
last_turn = None
out: list[str] = []
for evt in events:
last_turn, lines = render_event_cli(evt, last_turn)
out.extend(lines)
assert "thread started" in out
assert "turn started" in out
assert any(line.startswith("stream error:") for line in out)
assert any(line.startswith("4. ▸ `pytest -q`") for line in out)
assert any("✗ `pytest -q` (exit 1)" in line for line in out)
assert any(line.startswith("✗ `pytest -q` (exit 1)") for line in out)
assert any(
"searched: python jsonlines parser handle unknown fields" in line
for line in out
)
assert any("tool: github.search_issues" in line for line in out)
assert any("updated `src/compute_answer.py`" in line for line in out)
assert any(
line.startswith(
"turn failed: Aborted: required dependency `npm` is missing; cannot continue."
)
for line in out
)
assert "assistant:" in out
assert any("Legacy schema example" in line for line in out)
def test_render_event_cli_all_formats_golden() -> None:
events = _loads(ALL_FORMATS_FIXTURE_PATH.read_text(encoding="utf-8"))
last_turn = None
out: list[str] = []
for evt in events:
last_turn, lines = render_event_cli(evt, last_turn)
out.extend(lines)
print("\n".join(out))
expected = ALL_FORMATS_GOLDEN_PATH.read_text(encoding="utf-8").rstrip("\n")
assert "\n".join(out) == expected
assert any("files: +README.md, ~src/compute_answer.py" in line for line in out)
assert any(line.startswith("✗ stream error") for line in out)
def test_progress_renderer_renders_progress_and_final() -> None:
r = ExecProgressRenderer(max_actions=5)
for evt in _loads(SAMPLE_STREAM):
r = ExecProgressRenderer(max_actions=5, resume_formatter=_format_resume)
for evt in SAMPLE_EVENTS:
r.note_event(evt)
progress = r.render_progress(3.0)
assert progress.startswith("working · 3s · step 3")
assert "1\\. ✓ `bash -lc ls`" in progress
assert "resume: `0199a213-81c0-7800-8aa1-bbab2a035a53`" in progress
assert progress.startswith("working · 3s · step 2")
assert "✓ `bash -lc ls`" in progress
assert "`codex resume 0199a213-81c0-7800-8aa1-bbab2a035a53`" in progress
final = r.render_final(3.0, "answer", status="done")
assert final.startswith("done · 3s · step 3")
assert "running:" not in final
assert "ran:" not in final
assert final.startswith("done · 3s · step 2")
assert "answer" in final
assert final.rstrip().endswith("resume: `0199a213-81c0-7800-8aa1-bbab2a035a53`")
assert final.rstrip().endswith(
"`codex resume 0199a213-81c0-7800-8aa1-bbab2a035a53`"
)
def test_progress_renderer_clamps_actions_and_ignores_unknown() -> None:
r = ExecProgressRenderer(max_actions=3, command_width=20)
events = [
{
"type": "item.completed",
"item": {
"id": f"item_{i}",
"type": "command_execution",
"command": f"echo {i}",
"exit_code": 0,
"status": "completed",
},
}
action_completed(
f"item_{i}",
"command",
f"echo {i}",
ok=True,
detail={"exit_code": 0},
)
for i in range(6)
]
@@ -148,33 +118,102 @@ def test_progress_renderer_clamps_actions_and_ignores_unknown() -> None:
assert r.note_event(evt) is True
assert len(r.recent_actions) == 3
assert r.recent_actions[0].startswith("3\\. ")
assert r.recent_actions[-1].startswith("5\\. ")
assert r.note_event({"type": "mystery"}) is False
assert "echo 3" in r.recent_actions[0]
assert "echo 5" in r.recent_actions[-1]
mystery = SimpleNamespace(type="mystery")
assert r.note_event(cast(TakopiEvent, mystery)) is False
def test_progress_renderer_preserves_item_ids_in_telegram_text() -> None:
def test_progress_renderer_renders_commands_in_markdown() -> None:
r = ExecProgressRenderer(max_actions=5, command_width=None)
for i in (30, 31, 32):
r.note_event(
{
"type": "item.completed",
"item": {
"id": f"item_{i}",
"type": "command_execution",
"command": f"echo {i}",
"exit_code": 0,
"status": "completed",
},
}
action_completed(
f"item_{i}",
"command",
f"echo {i}",
ok=True,
detail={"exit_code": 0},
)
)
md = r.render_progress(0.0)
assert "30\\." in md
assert "31\\." in md
assert "32\\." in md
text, _ = render_markdown(md)
assert "30. ✓ echo 30" in text
assert "31. ✓ echo 31" in text
assert "32. ✓ echo 32" in text
assert "✓ echo 30" in text
assert "✓ echo 31" in text
assert "✓ echo 32" in text
def test_progress_renderer_handles_duplicate_action_ids() -> None:
r = ExecProgressRenderer(max_actions=5)
events = [
action_started("dup", "command", "echo first"),
action_completed(
"dup",
"command",
"echo first",
ok=True,
detail={"exit_code": 0},
),
action_started("dup", "command", "echo second"),
action_completed(
"dup",
"command",
"echo second",
ok=True,
detail={"exit_code": 0},
),
]
for evt in events:
assert r.note_event(evt) is True
assert len(r.recent_actions) == 2
assert r.recent_actions[0].startswith("✓ ")
assert "echo first" in r.recent_actions[0]
assert r.recent_actions[1].startswith("✓ ")
assert "echo second" in r.recent_actions[1]
def test_progress_renderer_collapses_action_updates() -> None:
r = ExecProgressRenderer(max_actions=5)
events = [
action_started("a-1", "command", "echo one"),
action_started("a-1", "command", "echo two"),
action_completed(
"a-1",
"command",
"echo two",
ok=True,
detail={"exit_code": 0},
),
]
for evt in events:
assert r.note_event(evt) is True
assert r.action_count == 1
assert len(r.recent_actions) == 1
assert r.recent_actions[0].startswith("✓ ")
assert "echo two" in r.recent_actions[0]
def test_progress_renderer_deterministic_output() -> None:
events = [
action_started("a-1", "command", "echo ok"),
action_completed(
"a-1",
"command",
"echo ok",
ok=True,
detail={"exit_code": 0},
),
]
r1 = ExecProgressRenderer(max_actions=5)
r2 = ExecProgressRenderer(max_actions=5)
for evt in events:
r1.note_event(evt)
r2.note_event(evt)
assert r1.render_progress(1.0) == r2.render_progress(1.0)
tests/test_exec_runner.py
+263 -47
View File
@@ -1,88 +1,304 @@
import anyio
import pytest
from takopi.exec_bridge import CodexExecRunner, EventCallback
from collections.abc import AsyncIterator
from takopi.model import (
ActionEvent,
CompletedEvent,
EngineId,
ResumeToken,
StartedEvent,
TakopiEvent,
)
from takopi.runners.codex import CodexRunner
CODEX_ENGINE = EngineId("codex")
@pytest.mark.anyio
async def test_run_serialized_serializes_same_session() -> None:
runner = CodexExecRunner(codex_cmd="codex", extra_args=[])
async def test_run_serializes_same_session() -> None:
runner = CodexRunner(codex_cmd="codex", extra_args=[])
gate = anyio.Event()
in_flight = 0
max_in_flight = 0
async def run_stub(*_args, **_kwargs):
async def run_stub(*_args, **_kwargs) -> AsyncIterator[TakopiEvent]:
nonlocal in_flight, max_in_flight
in_flight += 1
max_in_flight = max(max_in_flight, in_flight)
await gate.wait()
in_flight -= 1
return ("sid", "ok", True)
try:
await gate.wait()
yield CompletedEvent(
engine=CODEX_ENGINE,
resume=ResumeToken(engine=CODEX_ENGINE, value="sid"),
ok=True,
answer="ok",
)
finally:
in_flight -= 1
runner.run = run_stub # type: ignore[assignment]
runner._run = run_stub # type: ignore[assignment]
async def drain(prompt: str, resume: ResumeToken | None) -> None:
async for _event in runner.run(prompt, resume):
pass
token = ResumeToken(engine=CODEX_ENGINE, value="sid")
async with anyio.create_task_group() as tg:
tg.start_soon(runner.run_serialized, "a", "sid")
tg.start_soon(runner.run_serialized, "b", "sid")
tg.start_soon(drain, "a", token)
tg.start_soon(drain, "b", token)
await anyio.sleep(0)
gate.set()
assert max_in_flight == 1
@pytest.mark.anyio
async def test_run_serialized_allows_parallel_new_sessions() -> None:
runner = CodexExecRunner(codex_cmd="codex", extra_args=[])
async def test_run_allows_parallel_new_sessions() -> None:
runner = CodexRunner(codex_cmd="codex", extra_args=[])
gate = anyio.Event()
in_flight = 0
max_in_flight = 0
async def run_stub(*_args, **_kwargs):
async def run_stub(*_args, **_kwargs) -> AsyncIterator[TakopiEvent]:
nonlocal in_flight, max_in_flight
in_flight += 1
max_in_flight = max(max_in_flight, in_flight)
await gate.wait()
in_flight -= 1
return ("sid", "ok", True)
try:
await gate.wait()
yield CompletedEvent(
engine=CODEX_ENGINE,
resume=ResumeToken(engine=CODEX_ENGINE, value="sid"),
ok=True,
answer="ok",
)
finally:
in_flight -= 1
runner.run = run_stub # type: ignore[assignment]
runner._run = run_stub # type: ignore[assignment]
async def drain(prompt: str, resume: ResumeToken | None) -> None:
async for _event in runner.run(prompt, resume):
pass
async with anyio.create_task_group() as tg:
tg.start_soon(runner.run_serialized, "a", None)
tg.start_soon(runner.run_serialized, "b", None)
with anyio.move_on_after(1):
while max_in_flight < 2:
await anyio.sleep(0)
tg.start_soon(drain, "a", None)
tg.start_soon(drain, "b", None)
await anyio.sleep(0)
gate.set()
assert max_in_flight == 2
@pytest.mark.anyio
async def test_new_session_holds_lock_for_resumes() -> None:
runner = CodexExecRunner(codex_cmd="codex", extra_args=[])
finish = anyio.Event()
resume_started = anyio.Event()
async def test_run_allows_parallel_different_sessions() -> None:
runner = CodexRunner(codex_cmd="codex", extra_args=[])
gate = anyio.Event()
in_flight = 0
max_in_flight = 0
async def run_stub(
_prompt: str,
session_id: str | None,
on_event: EventCallback | None = None,
) -> tuple[str, str, bool]:
if session_id is None:
if on_event:
await on_event({"type": "thread.started", "thread_id": "sid"})
await finish.wait()
return ("sid", "ok", True)
resume_started.set()
return ("sid", "ok", True)
async def run_stub(*_args, **_kwargs) -> AsyncIterator[TakopiEvent]:
nonlocal in_flight, max_in_flight
in_flight += 1
max_in_flight = max(max_in_flight, in_flight)
try:
await gate.wait()
yield CompletedEvent(
engine=CODEX_ENGINE,
resume=ResumeToken(engine=CODEX_ENGINE, value="sid"),
ok=True,
answer="ok",
)
finally:
in_flight -= 1
runner.run = run_stub # type: ignore[assignment]
runner._run = run_stub # type: ignore[assignment]
async def drain(prompt: str, resume: ResumeToken | None) -> None:
async for _event in runner.run(prompt, resume):
pass
token_a = ResumeToken(engine=CODEX_ENGINE, value="sid-a")
token_b = ResumeToken(engine=CODEX_ENGINE, value="sid-b")
async with anyio.create_task_group() as tg:
tg.start_soon(drain, "a", token_a)
tg.start_soon(drain, "b", token_b)
await anyio.sleep(0)
gate.set()
assert max_in_flight == 2
@pytest.mark.anyio
async def test_run_serializes_new_session_after_session_is_known(
tmp_path, monkeypatch
) -> None:
gate_path = tmp_path / "gate"
resume_marker = tmp_path / "resume_started"
thread_id = "019b73c4-0c3f-7701-a0bb-aac6b4d8a3bc"
codex_path = tmp_path / "codex"
codex_path.write_text(
"#!/usr/bin/env python3\n"
"import json\n"
"import os\n"
"import sys\n"
"import time\n"
"\n"
"gate = os.environ['CODEX_TEST_GATE']\n"
"resume_marker = os.environ['CODEX_TEST_RESUME_MARKER']\n"
"thread_id = os.environ['CODEX_TEST_THREAD_ID']\n"
"\n"
"args = sys.argv[1:]\n"
"if 'resume' in args:\n"
" print(json.dumps({'type': 'thread.started', 'thread_id': thread_id}), flush=True)\n"
" with open(resume_marker, 'w', encoding='utf-8') as f:\n"
" f.write('started')\n"
" f.flush()\n"
" sys.exit(0)\n"
"\n"
"print(json.dumps({'type': 'thread.started', 'thread_id': thread_id}), flush=True)\n"
"while not os.path.exists(gate):\n"
" time.sleep(0.001)\n"
"sys.exit(0)\n",
encoding="utf-8",
)
codex_path.chmod(0o755)
monkeypatch.setenv("CODEX_TEST_GATE", str(gate_path))
monkeypatch.setenv("CODEX_TEST_RESUME_MARKER", str(resume_marker))
monkeypatch.setenv("CODEX_TEST_THREAD_ID", thread_id)
runner = CodexRunner(codex_cmd=str(codex_path), extra_args=[])
session_started = anyio.Event()
resume_value: str | None = None
new_done = anyio.Event()
async def run_new() -> None:
nonlocal resume_value
async for event in runner.run("hello", None):
if isinstance(event, StartedEvent):
resume_value = event.resume.value
session_started.set()
new_done.set()
async def run_resume() -> None:
assert resume_value is not None
async for _event in runner.run(
"resume", ResumeToken(engine=CODEX_ENGINE, value=resume_value)
):
pass
async with anyio.create_task_group() as tg:
tg.start_soon(runner.run_serialized, "first", None)
await anyio.sleep(0)
tg.start_soon(runner.run_serialized, "resume", "sid")
await anyio.sleep(0)
assert not resume_started.is_set()
finish.set()
tg.start_soon(run_new)
await session_started.wait()
tg.start_soon(run_resume)
await anyio.sleep(0.01)
assert not resume_marker.exists()
gate_path.write_text("go", encoding="utf-8")
await new_done.wait()
with anyio.fail_after(2):
while not resume_marker.exists():
await anyio.sleep(0.001)
@pytest.mark.anyio
async def test_codex_runner_preserves_warning_order(tmp_path) -> None:
thread_id = "019b73c4-0c3f-7701-a0bb-aac6b4d8a3bc"
codex_path = tmp_path / "codex"
codex_path.write_text(
"#!/usr/bin/env python3\n"
"import json\n"
"import sys\n"
"\n"
"sys.stdin.read()\n"
"print(json.dumps({'type': 'error', 'message': 'warning one', 'fatal': False}), flush=True)\n"
f"print(json.dumps({{'type': 'thread.started', 'thread_id': '{thread_id}'}}), flush=True)\n"
"print(json.dumps({'type': 'item.completed', 'item': {'id': 'item_0', 'type': 'agent_message', 'text': 'ok'}}), flush=True)\n",
encoding="utf-8",
)
codex_path.chmod(0o755)
runner = CodexRunner(codex_cmd=str(codex_path), extra_args=[])
seen = [evt async for evt in runner.run("hi", None)]
assert len(seen) == 3
assert isinstance(seen[0], ActionEvent)
assert seen[0].phase == "completed"
assert seen[0].ok is False
assert seen[0].action.kind == "warning"
assert seen[0].action.title == "warning one"
assert isinstance(seen[1], StartedEvent)
assert seen[1].resume.value == thread_id
assert isinstance(seen[2], CompletedEvent)
assert seen[2].resume == seen[1].resume
assert seen[2].answer == "ok"
@pytest.mark.anyio
async def test_run_serializes_two_new_sessions_same_thread(
tmp_path, monkeypatch
) -> None:
gate_path = tmp_path / "gate"
thread_id = "019b73c4-0c3f-7701-a0bb-aac6b4d8a3bc"
codex_path = tmp_path / "codex"
codex_path.write_text(
"#!/usr/bin/env python3\n"
"import json\n"
"import os\n"
"import sys\n"
"import time\n"
"\n"
"gate = os.environ['CODEX_TEST_GATE']\n"
"thread_id = os.environ['CODEX_TEST_THREAD_ID']\n"
"\n"
"print(json.dumps({'type': 'thread.started', 'thread_id': thread_id}), flush=True)\n"
"while not os.path.exists(gate):\n"
" time.sleep(0.001)\n"
"sys.exit(0)\n",
encoding="utf-8",
)
codex_path.chmod(0o755)
monkeypatch.setenv("CODEX_TEST_GATE", str(gate_path))
monkeypatch.setenv("CODEX_TEST_THREAD_ID", thread_id)
runner = CodexRunner(codex_cmd=str(codex_path), extra_args=[])
started_first = anyio.Event()
started_second = anyio.Event()
async def run_first() -> None:
async for event in runner.run("one", None):
if isinstance(event, StartedEvent):
started_first.set()
async def run_second() -> None:
async for event in runner.run("two", None):
if isinstance(event, StartedEvent):
started_second.set()
async with anyio.create_task_group() as tg:
tg.start_soon(run_first)
tg.start_soon(run_second)
with anyio.fail_after(2):
while not (started_first.is_set() or started_second.is_set()):
await anyio.sleep(0.001)
assert not (started_first.is_set() and started_second.is_set())
gate_path.write_text("go", encoding="utf-8")
with anyio.fail_after(2):
await started_first.wait()
await started_second.wait()
tests/test_onboarding.py
+17 -11
View File
@@ -2,46 +2,52 @@ from __future__ import annotations
from pathlib import Path
from takopi import onboarding
from takopi import engines, onboarding
def test_check_setup_marks_missing_codex(monkeypatch, tmp_path: Path) -> None:
monkeypatch.setattr(onboarding.shutil, "which", lambda _name: None)
backend = engines.get_backend("codex")
monkeypatch.setattr(engines.shutil, "which", lambda _name: None)
monkeypatch.setattr(
onboarding,
"load_telegram_config",
lambda: ({"bot_token": "token", "chat_id": 123}, tmp_path / "takopi.toml"),
)
result = onboarding.check_setup()
result = onboarding.check_setup(backend)
assert result.missing_codex is True
assert result.missing_or_invalid_config is False
titles = {issue.title for issue in result.issues}
assert "Install the Codex CLI" in titles
assert "Create a config" not in titles
assert result.ok is False
def test_check_setup_marks_missing_config(monkeypatch) -> None:
monkeypatch.setattr(onboarding.shutil, "which", lambda _name: "/usr/bin/codex")
backend = engines.get_backend("codex")
monkeypatch.setattr(engines.shutil, "which", lambda _name: "/usr/bin/codex")
def _raise() -> None:
raise onboarding.ConfigError("Missing config file")
monkeypatch.setattr(onboarding, "load_telegram_config", _raise)
result = onboarding.check_setup()
result = onboarding.check_setup(backend)
assert result.missing_or_invalid_config is True
titles = {issue.title for issue in result.issues}
assert "Create a config" in titles
assert result.config_path == onboarding.HOME_CONFIG_PATH
def test_check_setup_marks_invalid_chat_id(monkeypatch, tmp_path: Path) -> None:
monkeypatch.setattr(onboarding.shutil, "which", lambda _name: "/usr/bin/codex")
backend = engines.get_backend("codex")
monkeypatch.setattr(engines.shutil, "which", lambda _name: "/usr/bin/codex")
monkeypatch.setattr(
onboarding,
"load_telegram_config",
lambda: ({"bot_token": "token", "chat_id": "123"}, tmp_path / "takopi.toml"),
)
result = onboarding.check_setup()
result = onboarding.check_setup(backend)
assert result.missing_or_invalid_config is True
titles = {issue.title for issue in result.issues}
assert "Create a config" in titles
tests/test_rendering.py
+1 -1
View File
@@ -1,4 +1,4 @@
from takopi.exec_render import render_markdown
from takopi.markdown import render_markdown
def test_render_markdown_basic_entities() -> None:
tests/test_runner_contract.py
+106
View File
@@ -0,0 +1,106 @@
import anyio
import pytest
from collections.abc import AsyncGenerator
from typing import cast
from takopi.model import (
Action,
ActionEvent,
CompletedEvent,
EngineId,
ResumeToken,
StartedEvent,
TakopiEvent,
)
from takopi.runners.mock import Emit, Return, ScriptRunner, Wait
from tests.factories import action_started
CODEX_ENGINE = EngineId("codex")
@pytest.mark.anyio
async def test_runner_contract_session_started_and_order() -> None:
raw_completed: TakopiEvent = ActionEvent(
engine=CODEX_ENGINE,
action=Action(
id="a-1",
kind="command",
title="echo ok",
detail={"exit_code": 0},
),
phase="completed",
)
script = [
Emit(action_started("a-1", "command", "echo ok")),
Emit(raw_completed),
Return(answer="done"),
]
runner = ScriptRunner(script, engine=CODEX_ENGINE, resume_value="abc123")
seen = [evt async for evt in runner.run("hi", None)]
session_events = [evt for evt in seen if isinstance(evt, StartedEvent)]
assert len(session_events) == 1
completed_events = [evt for evt in seen if isinstance(evt, CompletedEvent)]
assert len(completed_events) == 1
assert seen[-1].type == "completed"
session_idx = seen.index(session_events[0])
completed_idx = seen.index(completed_events[0])
assert session_idx < completed_idx
assert completed_events[0].resume == session_events[0].resume
assert completed_events[0].answer == "done"
assert [evt.type for evt in seen if evt.type not in {"started", "completed"}] == [
"action",
"action",
]
completed_event = next(
evt for evt in seen if isinstance(evt, ActionEvent) and evt.phase == "completed"
)
assert completed_event.type == "action"
assert completed_event.ok is True
action = completed_event.action
assert action.id == "a-1"
assert action.kind == "command"
assert action.title == "echo ok"
@pytest.mark.anyio
async def test_runner_contract_resume_matches_session_started() -> None:
runner = ScriptRunner(
[Return(answer="ok")], engine=CODEX_ENGINE, resume_value="sid"
)
seen = [evt async for evt in runner.run("hello", None)]
session = next(evt for evt in seen if isinstance(evt, StartedEvent))
completed = next(evt for evt in seen if isinstance(evt, CompletedEvent))
assert completed.resume == session.resume
assert isinstance(completed.resume, ResumeToken)
@pytest.mark.anyio
async def test_runner_releases_lock_when_consumer_closes() -> None:
gate = anyio.Event()
runner = ScriptRunner([Wait(gate)], engine=CODEX_ENGINE, resume_value="sid")
gen = cast(AsyncGenerator[TakopiEvent, None], runner.run("hello", None))
try:
while True:
evt = await anext(gen)
if isinstance(evt, StartedEvent):
break
finally:
await gen.aclose()
gen2 = cast(
AsyncGenerator[TakopiEvent, None],
runner.run("again", ResumeToken(engine=CODEX_ENGINE, value="sid")),
)
try:
while True:
evt2 = await anext(gen2)
if isinstance(evt2, StartedEvent):
break
finally:
await gen2.aclose()
tests/test_subprocess.py
+11 -4
View File
@@ -2,16 +2,23 @@ import sys
import pytest
from takopi import exec_bridge
from takopi.runners import codex
@pytest.mark.anyio
async def test_manage_subprocess_kills_when_terminate_times_out() -> None:
async with exec_bridge.manage_subprocess(
async def test_manage_subprocess_kills_when_terminate_times_out(
monkeypatch,
) -> None:
async def fake_wait_for_process(_proc, timeout: float) -> bool:
_ = timeout
return True
monkeypatch.setattr(codex, "_wait_for_process", fake_wait_for_process)
async with codex.manage_subprocess(
sys.executable,
"-c",
"import signal, time; signal.signal(signal.SIGTERM, signal.SIG_IGN); time.sleep(10)",
terminate_timeout=0.01,
) as proc:
assert proc.returncode is None
uv.lock
Generated
+2 -2
View File
@@ -1,6 +1,6 @@
version = 1
revision = 3
requires-python = ">=3.12"
requires-python = ">=3.14"
[[package]]
name = "anyio"
@@ -430,7 +430,7 @@ wheels = [
[[package]]
name = "takopi"
version = "0.1.0"
version = "0.2.0"
source = { editable = "." }
dependencies = [
{ name = "anyio" },