webterm/docs/ROADMAP.md

Roadmap: Migration to xterm.js 6.0 with Bun

This document outlines the plan for bundling xterm.js 6.0 directly, replacing the dependency on textual-serve's bundled textual.js.

Status: ✅ Complete

The migration has been implemented on the upstream-xterm branch.

What Was Done

• Phase 1: Tooling Setup - Added package.json, bunfig.toml, Makefile targets
• Phase 2: Terminal Client - Created terminal.ts with full WebSocket protocol
• Phase 3: Server Integration - Updated HTML template, removed monkey-patch
• Phase 4: Configuration - Added data-scrollback attribute support
• Phase 5: Remove Dependency - Dropped textual-serve from pyproject.toml
• Phase 6: Documentation - Updated README.md and ARCHITECTURE.md

Key Outcomes

Metric	Before	After
textual-serve dependency	Required	❌ Removed
Scrollback history	0 (none)	1000 (configurable)
Font configuration	Monkey-patch workaround	Direct configuration
Bundle size	502 KB + 381 KB fonts	560 KB total
xterm.js version	Unknown (5.x?)	6.0.0

Files Changed

Added:
  package.json              # xterm.js 6.0 + addons
  bunfig.toml               # Bun configuration
  src/.../static/js/terminal.ts   # TypeScript source
  src/.../static/js/terminal.js   # Pre-built bundle (committed)
  src/.../static/css/xterm.css    # xterm.js styles

Modified:
  pyproject.toml            # Removed textual-serve dependency
  Makefile                  # Added bundle/bundle-watch targets
  .gitignore                # Added node_modules/
  src/.../local_server.py   # Simplified HTML template
  docs/ARCHITECTURE.md      # Updated file structure
  README.md                 # Added frontend dev instructions

For Users

No action required. The pre-built terminal.js bundle is committed to the repo, so:

pip install git+https://github.com/rcarmo/textual-webterm.git@upstream-xterm

Works without needing Node.js or Bun.

For Developers

To modify the frontend:

# Install Bun (https://bun.sh)
curl -fsSL https://bun.sh/install | bash

# Install dependencies and build
make bundle

# Or watch for changes during development
make bundle-watch

---

Background Analysis

The sections below document the original analysis that led to this migration.

What textual-serve Provides

Asset	Size	What We Use	Required?
static/js/textual.js	502 KB	xterm.js + WebSocket client	Yes
static/css/xterm.css	4.6 KB	Terminal styling	Yes
static/fonts/RobotoMono*.ttf	381 KB	Roboto Mono font	No (we override font)
static/images/background.png	58 KB	Background image	No
Total	948 KB

What textual.js Bundle Contains

The minified textual.js bundles:

xterm.js (core terminal)
├── @xterm/addon-fit        (auto-resize to container)
├── @xterm/addon-webgl      (GPU-accelerated rendering)
├── @xterm/addon-canvas     (fallback 2D canvas renderer)
├── @xterm/addon-unicode11  (wide character support)
├── @xterm/addon-web-links  (clickable URLs)
├── @xterm/addon-clipboard  (clipboard integration)
└── WebSocket client wrapper (class w)

Hardcoded Configuration in textual.js

new Terminal({
  allowProposedApi: true,
  fontSize: /* from data-font-size attribute */,
  scrollback: 0,           // ❌ No scrollback history
  fontFamily: "'Roboto Mono', Monaco, 'Courier New', monospace"  // ❌ Hardcoded
})

WebSocket Protocol (Fully Compatible)

The protocol is simple JSON arrays. Our server already implements this:

Direction	Message	Description
Client → Server	["stdin", "data"]	Terminal input
Client → Server	["resize", {width: N, height: M}]	Window resize
Client → Server	["ping", data]	Keep-alive
Server → Client	["stdout", "data"]	Terminal output (text)
Server → Client	Binary frame	Terminal output (binary)
Server → Client	["pong", data]	Keep-alive response

Current Workarounds

1. Font override: Canvas monkey-patch in HTML to replace hardcoded font family
2. No scrollback: Users cannot scroll back through terminal history

---

Tradeoffs Analysis

Option A: Keep textual-serve Dependency

Pros	Cons
Zero build tooling	Hardcoded font requires workaround
Automatic updates via pip	No scrollback (scrollback: 0)
Maintained by Textualize	No theme customization
	Carries unused fonts/images (381 KB)
	Tied to textual-serve release cycle
	Unknown xterm.js version (likely 5.x)

Option B: Bundle xterm.js 6.0 Directly

Pros	Cons
Full configuration control	Requires Bun toolchain
Scrollback history support	~150-200 KB bundle to maintain
Custom themes/colors	Must track xterm.js updates
Latest xterm.js 6.0 features	Initial setup effort (2-3 days)
Smaller bundle (no unused fonts)
Can drop textual-serve dependency

xterm.js 6.0 Features We'd Gain

Feature	Benefit
Synchronized output (DEC 2026)	Smoother rapid output rendering
Ligature support	Better programming font rendering
Progress addon	Visual progress indicators
Shadow DOM support	Better CSS encapsulation
ESM support	Modern module loading
Performance improvements	Faster search, less memory
OSC 52 clipboard	Secure clipboard from terminal

---

Implementation Plan (Completed)

Phase 1: Tooling Setup ✅

Goal: Establish Bun-based build pipeline

src/textual_webterm/
├── static/
│   ├── js/
│   │   └── terminal.ts      # New: our xterm wrapper
│   ├── css/
│   │   └── xterm.css        # Copied from xterm.js package
│   └── monospace.css        # Existing
├── package.json             # New: npm dependencies
└── bunfig.toml              # New: Bun configuration

Tasks:

• Create package.json with xterm.js 6.0 dependencies
• Create bunfig.toml for build configuration
• Add Makefile targets: make bundle, make bundle-watch
• Add .gitignore entries for node_modules/
• Document Bun installation in README

package.json (final):

{
  "name": "textual-webterm-frontend",
  "private": true,
  "type": "module",
  "dependencies": {
    "@xterm/xterm": "^6.0.0",
    "@xterm/addon-fit": "^0.10.0",
    "@xterm/addon-webgl": "^0.18.0",
    "@xterm/addon-canvas": "^0.7.0",
    "@xterm/addon-unicode11": "^0.8.0",
    "@xterm/addon-web-links": "^0.11.0",
    "@xterm/addon-clipboard": "^0.2.0"
  },
  "devDependencies": {
    "typescript": "^5.7.0"
  },
  "scripts": {
    "build": "bun build src/textual_webterm/static/js/terminal.ts --outfile=src/textual_webterm/static/js/terminal.js --minify --target=browser",
    "watch": "bun build src/textual_webterm/static/js/terminal.ts --outfile=src/textual_webterm/static/js/terminal.js --watch --target=browser"
  }
}

Phase 2: Terminal Client Implementation ✅

Goal: Create terminal.ts that replicates textual.js functionality

Tasks:

• Implement Terminal wrapper class
• WebSocket connection with reconnection logic
• Message protocol handling (stdin, resize, ping/pong)
• Addon initialization (fit, webgl, canvas, unicode11, web-links, clipboard)
• Configurable options via data attributes or window config

See src/textual_webterm/static/js/terminal.ts for the full implementation (~230 lines).

Phase 3: Server Integration ✅

Goal: Update local_server.py to use new bundle

Tasks:

• Update HTML template to load our bundle instead of textual.js
• Remove canvas monkey-patch workaround
• Add data attributes for scrollback, theme configuration
• Copy xterm.css to our static folder
• Update static file routes

Phase 4: Configuration Support ✅

Goal: Make terminal appearance configurable

Tasks:

• Pass config to HTML template via data attributes (data-scrollback, data-font-size)
• Add terminal config to CLI (--scrollback, --font-family) - Future enhancement
• Add terminal config to TOML manifest files - Future enhancement

Phase 5: Remove textual-serve Dependency ✅

Goal: Eliminate dependency once our bundle is stable

Tasks:

• Remove textual-serve from pyproject.toml dependencies
• Update ARCHITECTURE.md to document new frontend
• Update README.md with build instructions
• Commit pre-built bundle so users don't need Bun

Phase 6: Testing & Polish

Goal: Ensure reliability across browsers

Tasks:

• Cross-browser testing (Chrome, Firefox, Safari, Edge)
• Mobile browser testing (iOS Safari, Chrome Android)
• WebGL fallback to Canvas (implemented in terminal.ts)
• Reconnection logic (implemented with exponential backoff)
• Performance comparison vs textual.js
• Bundle size: 560 KB (acceptable for full xterm.js + addons)

---

Build Integration (Reference)

Makefile Additions

# Frontend build
.PHONY: bundle bundle-watch bundle-clean

bundle: node_modules
	bun run build

bundle-watch: node_modules
	bun run watch

bundle-clean:
	rm -rf node_modules src/textual_webterm/static/js/terminal.js

node_modules: package.json
	bun install

Dockerfile Changes

# Add Bun for frontend build
RUN curl -fsSL https://bun.sh/install | bash
ENV PATH="/root/.bun/bin:${PATH}"

# Build frontend
COPY package.json bunfig.toml ./
RUN bun install
COPY src/textual_webterm/static/js/terminal.ts src/textual_webterm/static/js/
RUN bun run build

CI/CD Considerations

• Pre-commit hook to verify terminal.js matches terminal.ts
• Or: commit built bundle to repo (simpler for users without Bun)
• GitHub Actions step to build and verify bundle

---

Risk Mitigation

Risk	Mitigation
xterm.js 6.0 breaking changes	Pin exact version, test thoroughly
Bun compatibility issues	Fall back to esbuild if needed
WebSocket protocol mismatch	Keep protocol identical to textual.js
Performance regression	Benchmark before/after, keep WebGL
Missing addon features	Test each addon explicitly

---

Timeline Estimate

Phase	Effort	Dependencies
Phase 1: Tooling Setup	0.5 days	None
Phase 2: Terminal Client	1-2 days	Phase 1
Phase 3: Server Integration	0.5 days	Phase 2
Phase 4: Configuration	0.5 days	Phase 3
Phase 5: Remove Dependency	0.5 days	Phase 4
Phase 6: Testing	1 day	Phase 5
Total	4-5 days

---

Decision Checkpoints

1. After Phase 2: Verify terminal.ts works in isolation before integrating
2. After Phase 3: Side-by-side comparison with textual.js
3. After Phase 5: Confirm no regressions before removing dependency
4. After Phase 6: Final sign-off for release

---

Success Criteria

• Terminal renders correctly in Chrome, Firefox, Safari
• Scrollback history works (configurable limit)
• Custom fonts load without workarounds
• WebGL rendering enabled with Canvas fallback
• Bundle size: 560 KB (larger than target due to full addon suite, but acceptable)
• No textual-serve dependency in pyproject.toml
• All existing tests pass (302 tests)
• Documentation updated

---

---

Future: Go Reimplementation

This section analyzes what it would take to reimplement textual-webterm in Go for lighter deployment.

Status: 📋 Planning

Not yet started. This would be a separate project (textual-webterm-go) providing a lightweight alternative.

Executive Summary

Most functionality can be reimplemented in Go with mature libraries. The main challenge is the terminal emulator (pyte equivalent) - GoPyte exists but is less battle-tested than Python's pyte. Benefits would be a single static binary, lower memory footprint, and better concurrency.

---

Component Mapping

Python Component	Go Equivalent	Library	Maturity
aiohttp (HTTP/WS server)	net/http + websocket	gorilla/websocket or nhooyr.io/websocket	⭐⭐⭐⭐⭐ Excellent
pyte (terminal emulator)	GoPyte	github.com/scottpeterman/gopyte	⭐⭐⭐ Good
PTY handling	go-pty	github.com/aymanbagabas/go-pty	⭐⭐⭐⭐ Very Good
asyncio (concurrency)	goroutines/channels	stdlib	⭐⭐⭐⭐⭐ Native
SSE	Custom handler	stdlib net/http	⭐⭐⭐⭐ Simple
Docker stats	Docker SDK	github.com/docker/docker/client	⭐⭐⭐⭐⭐ Official
SVG generation	SVGo	github.com/ajstarks/svgo	⭐⭐⭐⭐⭐ Mature
YAML parsing	yaml.v3	gopkg.in/yaml.v3	⭐⭐⭐⭐⭐ Standard
CLI	cobra	github.com/spf13/cobra	⭐⭐⭐⭐⭐ Standard

---

What We'd Gain

Benefit	Impact
Single static binary	No Python/pip dependency, simpler deployment
Lower memory	~10-20MB vs ~50-100MB for Python
Better concurrency	Goroutines vs asyncio - more intuitive
Faster startup	Instant vs Python interpreter load
Cross-compilation	Easy builds for Linux/macOS/Windows/ARM
Smaller Docker image	~20MB vs ~200MB+ with Python

What We'd Lose

Loss	Impact
Textual app support	Cannot run Python Textual apps directly
Rapid prototyping	Go requires more boilerplate
pyte maturity	GoPyte is less proven

---

Required Go Dependencies

// go.mod
module github.com/rcarmo/textual-webterm-go

go 1.22

require (
    // HTTP/WebSocket
    github.com/gorilla/websocket v1.5.1
    
    // Terminal emulation
    github.com/scottpeterman/gopyte v0.1.0
    
    // PTY handling
    github.com/aymanbagabas/go-pty v0.2.2
    
    // Docker stats
    github.com/docker/docker v25.0.0
    
    // SVG generation
    github.com/ajstarks/svgo v0.0.0-20211024235047-1546f124cd8b
    
    // CLI
    github.com/spf13/cobra v1.8.0
    
    // YAML parsing
    gopkg.in/yaml.v3 v3.0.1
)

---

Implementation Plan

Phase 1: Project Setup & Core Server (2 days)

Goal: Basic HTTP server with WebSocket support

Tasks:

• Initialize Go module with dependencies
• Create basic HTTP server with routing
• Implement WebSocket upgrade handler
• Port JSON message protocol (stdin, resize, ping/pong)
• Add graceful shutdown handling

Files:

cmd/
  webterm/
    main.go           # Entry point
internal/
  server/
    server.go         # HTTP server
    websocket.go      # WebSocket handler
    routes.go         # Route definitions

Phase 2: PTY Session Management (2 days)

Goal: Spawn and manage terminal sessions

Tasks:

• Integrate go-pty for PTY creation
• Implement session lifecycle (create, resize, close)
• Build session manager with route mapping
• Add replay buffer for reconnection
• Handle concurrent session access

Files:

internal/
  session/
    manager.go        # Session registry
    terminal.go       # PTY session wrapper
    buffer.go         # Replay ring buffer

Phase 3: Terminal Emulation (2 days)

Goal: Parse ANSI sequences for screen state

Tasks:

• Integrate GoPyte terminal emulator
• Feed PTY output through emulator
• Extract screen buffer for screenshots
• Implement dirty tracking for cache invalidation
• Handle resize events

Files:

internal/
  terminal/
    emulator.go       # GoPyte wrapper
    screen.go         # Screen buffer access

Phase 4: SVG Screenshot Generation (1.5 days)

Goal: Generate terminal screenshots as SVG

Tasks:

• Port character positioning logic from Python
• Implement ANSI color palette (16 + 256 + truecolor)
• Handle box-drawing character scaling
• Add screenshot caching with ETag support
• Implement cache TTL backoff

Files:

internal/
  screenshot/
    svg.go            # SVG renderer
    colors.go         # ANSI color handling
    cache.go          # Screenshot cache

Phase 5: Dashboard & SSE (1.5 days)

Goal: Landing page with live updates

Tasks:

• Embed static assets (HTML, CSS, xterm.js bundle)
• Implement SSE endpoint for activity notifications
• Port dashboard HTML template
• Add tile rendering with screenshots

Files:

internal/
  dashboard/
    handler.go        # Dashboard HTTP handler
    sse.go            # Server-Sent Events
  static/
    embed.go          # Embedded assets

Phase 6: Docker Stats (1 day)

Goal: CPU sparklines for compose mode

Tasks:

• Integrate Docker SDK client
• Implement container stats polling
• Calculate CPU percentage from deltas
• Generate sparkline SVGs
• Filter by compose project label

Files:

internal/
  docker/
    stats.go          # Stats collector
    sparkline.go      # SVG sparkline

Phase 7: CLI & Configuration (1 day)

Goal: Feature-complete CLI

Tasks:

• Implement Cobra CLI with flags
• Parse YAML landing manifests
• Parse Docker Compose manifests
• Add version command
• Environment variable support

Files:

cmd/
  webterm/
    main.go           # CLI entry point
internal/
  config/
    config.go         # Configuration types
    manifest.go       # YAML parsing

Phase 8: Testing & Polish (2 days)

Goal: Production-ready release

Tasks:

• Unit tests for core components
• Integration tests for WebSocket protocol
• Cross-browser testing
• Build scripts for multiple platforms
• Docker image (FROM scratch)
• Documentation

Files:

Makefile              # Build targets
Dockerfile            # Multi-stage build
README.md             # Usage docs

---

Effort Summary

Phase	Component	Days
1	Project Setup & Core Server	2
2	PTY Session Management	2
3	Terminal Emulation	2
4	SVG Screenshot Generation	1.5
5	Dashboard & SSE	1.5
6	Docker Stats	1
7	CLI & Configuration	1
8	Testing & Polish	2
Total		13 days

---

File Structure (Final)

textual-webterm-go/
├── cmd/
│   └── webterm/
│       └── main.go
├── internal/
│   ├── config/
│   │   ├── config.go
│   │   └── manifest.go
│   ├── dashboard/
│   │   ├── handler.go
│   │   └── sse.go
│   ├── docker/
│   │   ├── sparkline.go
│   │   └── stats.go
│   ├── screenshot/
│   │   ├── cache.go
│   │   ├── colors.go
│   │   └── svg.go
│   ├── server/
│   │   ├── routes.go
│   │   ├── server.go
│   │   └── websocket.go
│   ├── session/
│   │   ├── buffer.go
│   │   ├── manager.go
│   │   └── terminal.go
│   ├── static/
│   │   └── embed.go
│   └── terminal/
│       ├── emulator.go
│       └── screen.go
├── static/
│   ├── css/
│   │   └── xterm.css
│   └── js/
│       └── terminal.js
├── go.mod
├── go.sum
├── Makefile
├── Dockerfile
└── README.md

---

Build & Release

Makefile Targets

.PHONY: build build-all test clean

BINARY := webterm
VERSION := $(shell git describe --tags --always)
LDFLAGS := -s -w -X main.version=$(VERSION)

build:
	go build -ldflags "$(LDFLAGS)" -o bin/$(BINARY) ./cmd/webterm

build-all:
	GOOS=linux GOARCH=amd64 go build -ldflags "$(LDFLAGS)" -o bin/$(BINARY)-linux-amd64 ./cmd/webterm
	GOOS=linux GOARCH=arm64 go build -ldflags "$(LDFLAGS)" -o bin/$(BINARY)-linux-arm64 ./cmd/webterm
	GOOS=darwin GOARCH=amd64 go build -ldflags "$(LDFLAGS)" -o bin/$(BINARY)-darwin-amd64 ./cmd/webterm
	GOOS=darwin GOARCH=arm64 go build -ldflags "$(LDFLAGS)" -o bin/$(BINARY)-darwin-arm64 ./cmd/webterm

test:
	go test -v ./...

clean:
	rm -rf bin/

Minimal Docker Image

FROM golang:1.22-alpine AS builder
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -ldflags "-s -w" -o webterm ./cmd/webterm

FROM scratch
COPY --from=builder /app/webterm /webterm
ENTRYPOINT ["/webterm"]

Result: ~15-20MB Docker image vs ~200MB+ for Python version.

---

Decision Criteria

Proceed with Go reimplementation if:

• Deployment size is critical (embedded, edge, IoT)
• No need for Textual app support
• Want single-binary distribution
• Memory constraints matter

Keep Python version if:

• Need Textual app support
• Rapid iteration is priority
• Team more familiar with Python
• Current deployment size is acceptable

---

References

• GoPyte: https://github.com/scottpeterman/gopyte
• go-pty: https://github.com/aymanbagabas/go-pty
• Gorilla WebSocket: https://github.com/gorilla/websocket
• Docker Go SDK: https://pkg.go.dev/github.com/docker/docker/client
• SVGo: https://github.com/ajstarks/svgo
• Cobra CLI: https://github.com/spf13/cobra