# Streaming & WebRTC Design Document

> **Status:** Approved — Implementation in Progress  
> **Target:** Sub-800ms first-chunk latency for short TTS; full browser WebRTC for bidirectional audio  
> **Scope:** New endpoints on existing `bolo-tts-server`; no changes to inference model

---

## 1. Executive Summary

This document describes the addition of three new surfaces to Bolo Voice Service:

1. **Streaming HTTP TTS** (`POST /v1/stream/speech`) — Sentence-level synthesis with fast presets, server-side crossfade, and chunked HTTP delivery.
2. **Bidirectional WebSocket** (`GET /v1/stream/ws`) — Persistent socket for realtime TTS + ASR in a single connection.
3. **Full WebRTC** (`/v1/webrtc/*`) — Browser-compatible peer connection with Opus RTP transport, STUN/TUN, DataChannel text bridge, and bidirectional audio.

All existing endpoints remain untouched. The core `Phase3Pipeline` inference model is not modified.

---

## 2. The Latency Constraint

OmniVoice is a **masked diffusion TTS model** with two rigid stages:

- **Stage 0 (Qwen3 backbone):** Iterative diffusion over the *entire* target token sequence. Requires `num_step` full forward passes before any token is known.
- **Stage 1 (DAC decoder):** Needs the *complete* token tensor before any audio waveform can be produced.

**Implication:** True token-level or word-level streaming audio generation is mathematically impossible with this model. The best achievable pattern is **sentence-level chunk streaming**.

### Latency by Preset (short sentence, ~2–3s audio)

| Preset | `num_step` | Quality | First-Chunk Latency (GPU) |
|--------|-----------|---------|---------------------------|
| `quality` | 32 | Best | ~1.5–2.5s |
| `balanced` | 16 | Good | ~1.0–1.5s |
| `fast` | 8 | Slight loss | ~500–800ms |
| `ultra` | 4 | Noticeable | ~300–600ms |

For longer text, each ~10–15s sentence chunk streams independently. Total time is still proportional to text length, but audio starts playing after the first chunk.

---

## 3. Streaming HTTP API

### 3.1 Endpoint

```
POST /v1/stream/speech
Authorization: Bearer <token>
Content-Type: application/json
Transfer-Encoding: chunked   (server response)
```

### 3.2 Request Body

```json
{
  "input": "Hello world. This is streaming text-to-speech.",
  "voice": "vp_1",
  "preset": "fast",
  "language": "en",
  "speed": 1.0,
  "response_format": "wav"
}
```

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `input` | string | required | Text to synthesize |
| `voice` | string | `"auto"` | Profile ID or built-in name |
| `preset` | string | `"fast"` | `quality`/`balanced`/`fast`/`ultra` → maps to `num_step` |
| `language` | string | `"en"` | Language code |
| `speed` | float | `1.0` | Playback speed |
| `response_format` | string | `"wav"` | `wav` or `pcm` (chunked delivery only) |

### 3.3 Response

The server returns `Transfer-Encoding: chunked` with `Content-Type: audio/wav`.

Each chunk is a valid WAV file segment. The server handles crossfading between sentences, so the client simply appends chunks to an audio buffer.

```
HTTP/1.1 200 OK
Content-Type: audio/wav
Transfer-Encoding: chunked

<chunk 0: WAV "Hello world.">
<chunk 1: WAV "This is streaming text-to-speech.">
```

### 3.4 How It Works

```
Client sends text
  ↓
Server splits text into sentences (existing chunk_text_punctuation)
  ↓
For each sentence:
  a. Build prompt (reuse existing frontend.prepare_prompt_device)
  b. Run Stage 0 with preset num_step (e.g., 8 for "fast")
  c. Run Stage 1 decode
  d. Apply postprocessing (fade_and_pad_audio, peak_normalize)
  e. Crossfade with previous sentence (server-side)
  f. Encode as WAV chunk
  g. Emit chunk via chunked HTTP
  ↓
While chunk N is being streamed, chunk N+1 is already generating in parallel
```

### 3.5 Server-Side Crossfade

The existing `cross_fade_chunks` function in `bolo-tts-infer` is used. Each sentence's audio is crossfaded with the previous sentence using a 300ms overlap. The resulting continuous audio is split into HTTP chunks at sentence boundaries.

**Why server-side?** The client receives already-smooth audio. No client-side audio processing needed.

---

## 4. WebSocket Streaming API

### 4.1 Endpoint

```
GET /v1/stream/ws
Authorization: Bearer <token>   (or X-VAPI-SECRET for Vapi compat)
Upgrade: websocket
```

### 4.2 Protocol

**Client → Server (JSON control):**

```json
// TTS request
{ "type": "tts", "input": "Hello", "voice": "vp_1", "preset": "fast" }

// Start ASR
{ "type": "asr.start", "sampleRate": 16000, "channels": 1 }

// Stop ASR
{ "type": "asr.stop" }
```

**Client → Server (binary audio for ASR):**
- Binary WebSocket frames containing PCM s16le mono audio at the declared sample rate.

**Server → Client (JSON):**

```json
// TTS audio chunk
{ "type": "tts.audio", "format": "wav", "data": "<base64>", "final": false }

// TTS done
{ "type": "tts.audio", "format": "wav", "data": "<base64>", "final": true }

// ASR result
{ "type": "asr.result", "text": "transcribed text", "final": true }

// ASR partial (if sliding window enabled)
{ "type": "asr.result", "text": "partial text", "final": false }

// Error
{ "type": "error", "message": "...", "code": 422 }
```

### 4.3 Architecture

The WebSocket handler spawns three concurrent tasks:

1. **Receiver task:** Parses incoming JSON + binary frames, routes to TTS queue or ASR buffer.
2. **TTS worker:** `tokio::task::spawn_blocking` loop. Pops TTS requests from queue, synthesizes sentence-by-sentence, sends audio chunks via a `mpsc::Sender` back to the WebSocket task.
3. **ASR worker:** Buffers binary audio in a `Vec<i16>`. Runs energy-based VAD. When silence exceeds threshold (default 500ms), wraps buffered audio as WAV, transcribes via `SpeechRuntime::transcribe_audio`, sends result.

---

## 5. WebRTC API

### 5.1 Dependencies

| Crate | Version | Purpose |
|-------|---------|---------|
| `str0m` | `0.20.0` | Sans-IO WebRTC peer connection, DTLS, SRTP, ICE |
| `opus` | `0.3.1` | Opus encoder/decoder for RTP audio |

`str0m` is chosen over `webrtc` crate because:
- **Sans-IO**: no internal threads; we drive it from a single tokio UDP loop
- **Smaller surface area**: no callback hell, no internal async runtime
- **Better integration** with existing axum/tokio architecture

### 5.2 Signaling Endpoints

#### Offer/Answer

```bash
POST /v1/webrtc/offer
Authorization: Bearer <token>
Content-Type: application/json

{
  "sdp": "v=0\r\no=- ...",
  "voice": "vp_1",
  "preset": "fast",
  "language": "en"
}
```

**Response:**
```json
{
  "peer_id": "peer_abc123",
  "sdp": "v=0\r\no=- ...",
  "candidates": ["candidate:..."]
}
```

#### ICE Candidates

```bash
POST /v1/webrtc/ice
Authorization: Bearer <token>
Content-Type: application/json

{
  "peer_id": "peer_abc123",
  "candidates": ["candidate:..."]
}
```

**Response:**
```json
{ "candidates": ["candidate:..."] }
```

### 5.3 Media Pipeline

```
┌─────────────┐      UDP (SRTP)      ┌─────────────────────────┐
│   Browser   │  ←────────────────→  │  str0m Rtc instance     │
│  (microphone)│                      │  opus_decode → PCM 24kHz │
│              │                      │  → VAD → Whisper ASR    │
│  (speakers)  │  ←────────────────→  │  TTS pipeline           │
│              │                      │  → opus_encode → RTP    │
└─────────────┘                      └─────────────────────────┘
```

**Incoming (Browser → Server):**
1. Browser sends Opus RTP packets via UDP
2. `str0m` decrypts SRTP, depacketizes
3. `opus::Decoder` (48kHz/Mono) → `Vec<f32>` PCM
4. Resample to 24kHz (or decode directly to 24kHz if negotiated)
5. Buffer in `Vec<f32>` for VAD
6. VAD detects silence → wrap as WAV → `SpeechRuntime::transcribe_audio`
7. ASR result sent via **DataChannel** (`label: "control"`)

**Outgoing (Server → Browser):**
1. DataChannel receives text from browser (or ASR result)
2. `StreamingSynthesizer` splits into sentences, synthesizes with preset
3. Postprocess → `Vec<f32>` at 24kHz
4. `opus::Encoder` (24kHz/Mono, 20ms frames = 480 samples)
5. `str0m Writer::write` with RTP timestamp incrementing by 480
6. `str0m` packetizes, encrypts SRTP, transmits via UDP

### 5.4 Opus Configuration

**Encoder (TTS → Browser):**
- Sample rate: 24000 Hz
- Channels: 1 (mono)
- Application: `opus::Application::Audio`
- Frame size: 20ms (480 samples @ 24kHz)
- Bitrate: auto (opus selects based on complexity)

**Decoder (Browser → ASR):**
- Sample rate: 48000 Hz (browser default, then downsample to 24kHz)
- Channels: 1 (mono)
- Frame size: 20ms (960 samples @ 48kHz)

**Note:** Browsers always offer `opus/48000/2`. The server accepts this and downsamples after decode. Alternatively, we could restrict the SDP to `opus/24000/1` but browser support is less predictable.

### 5.5 DataChannel Protocol

**Label:** `control`

**Client → Server (text input for TTS):**
```json
{"type":"tts","text":"Hello world","preset":"fast"}
```

**Server → Client (ASR results):**
```json
{"type":"asr","text":"Hello world","final":true}
```

**Server → Client (TTS status):**
```json
{"type":"tts.status","state":"generating","progress":0.5}
```

### 5.6 STUN / TURN Configuration

The server must provide its own ICE candidates for the browser to connect.

**Candidate types:**
1. **Host candidate:** Local IP + UDP port (always present)
2. **Server-reflexive candidate:** Discovered via STUN binding request to a public STUN server
3. **Relay candidate:** Via configured TURN server (optional but recommended for NAT traversal)

**CLI flags:**
```bash
--webrtc-udp-port 3478          # UDP port for media traffic
--webrtc-public-ip 1.2.3.4      # Public IP advertised in host candidate
--stun-server stun.l.google.com:19302
--turn-server turn:my-turn.com:3478
--turn-username alice
--turn-password secret
```

**Default behavior:**
- STUN: `stun.l.google.com:19302` (free public STUN)
- TURN: none (direct P2P only)
- If TURN is configured, a relay candidate is added (the TURN server address is advertised)

**Important:** TURN requires a separate TURN server. The Bolo server does not include a TURN server. You must run one externally:

**Option A: Self-hosted coturn (recommended for production)**
```bash
# Install coturn
sudo apt-get install coturn

# /etc/turnserver.conf
listening-port=3478
fingerprint
lt-cred-mech
user=alice:secret
realm=my-domain.com

# Start
turnserver -c /etc/turnserver.conf
```

**Option B: Cloud TURN service (easiest)**
- Twilio Network Traversal Service (~$0.40/GB)
- Xirsys (developer-friendly pricing)
- Cloudflare Calls (if available in your region)

**Option C: No TURN (acceptable for same-network testing)**
If both browser and server are on the same network or have public IPs, host candidates alone are sufficient.

---

### 5.6a Reverse Proxy

**Do you need a reverse proxy for WebRTC?**

- **Signaling (HTTP):** Yes — same as existing endpoints. Use nginx/Caddy/traefik for HTTPS, rate limiting, etc.
- **Media (UDP/RTP):** **No** — WebRTC media is peer-to-peer (or relayed through TURN). It never flows through your HTTP reverse proxy.

**Firewall rules:**
```bash
# TCP for HTTP/WebSocket APIs (same as before)
sudo ufw allow 8000/tcp

# UDP for WebRTC media (new)
sudo ufw allow 3478/udp
```

**nginx config example:**
```nginx
server {
    listen 443 ssl;
    server_name voice.example.com;

    # All HTTP/WebSocket signaling goes here
    location / {
        proxy_pass http://127.0.0.1:8000;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
    }
}
```

**Note:** Do NOT proxy UDP 3478 through nginx. The Bolo server binds this port directly.

**Zoraxy config example:**
```
# In Zoraxy web UI (http://zoraxy:port)
# 1. Add HTTP Proxy Rule
#    - Domain: voice.example.com
#    - Target: http://127.0.0.1:8000
#    - Enable WebSocket support: YES (required for /v1/stream/ws and /v1/ws/transcribe)
#    - TLS/HTTPS: Enable with your certificate
#
# 2. Advanced > Custom Headers (optional but recommended):
#    X-Forwarded-For  $remote_addr
#    X-Forwarded-Proto $scheme
#
# 3. UDP Port Forwarding (MUST be done at firewall/router level, NOT in Zoraxy):
#    Router/Firewall: forward UDP 3478 → server-internal-ip:3478
#    Or if server has public IP: sudo ufw allow 3478/udp
```

**Zoraxy-specific notes:**
- Zoraxy handles all HTTP/HTTPS signaling (including WebSocket upgrades)
- Zoraxy's **Stream Proxy** can forward UDP traffic to your Bolo server
- Make sure "Enable WebSocket support" is turned ON in the proxy rule
- If using Zoracy's built-in TLS, the certificate must cover your domain

**DMZ setup (when router DMZ points to Zoraxy server):**

```
Internet
    │
    ▼
[Router] ──DMZ──► [Zoraxy Server]
    │                   │
    │                   ├── HTTP Proxy ──► Bolo HTTP (192.168.1.109:8880)
    │                   │
    │                   └── Stream Proxy ──► Bolo UDP (192.168.1.109:3478)
    │
    └── UDP 3478 ────────► [Zoraxy] ──► [Bolo Server]
```

**Zoraxy Stream Proxy setup for UDP forwarding:**

```
# In Zoraxy web UI (http://zoraxy:port)
# 1. Go to "Stream Proxy" (TCP/UDP forwarding)
# 2. Add UDP Stream Rule:
#    - Listen Port: 3478
#    - Protocol: UDP
#    - Target: 192.168.1.109:3478
#    - Description: Bolo WebRTC UDP
# 3. Save and apply
```

This is the **recommended approach** when Zoraxy is your DMZ host:
1. Router DMZ → Zoraxy (handles all incoming traffic)
2. Zoraxy HTTP Proxy → Bolo HTTP (existing)
3. Zoraxy Stream Proxy → Bolo UDP (new)

**No router port forwarding needed** — DMZ sends everything to Zoraxy, and Zoraxy routes UDP internally.

**Alternative router-level forwarding** (if you prefer not to use Stream Proxy):

```
Router config:
  DMZ host: Zoraxy server (192.168.1.x)     ← handles everything EXCEPT forwarded ports
  Forward UDP 3478 → Bolo server (192.168.1.109)   ← ADD THIS RULE
```

Most routers allow DMZ + specific port forwarding overrides.

**If neither option works, fallback to TURN:**
```toml
turn_server = "turn:my-turn.com:3478"
turn_username = "alice"
turn_password = "secret"
```
All media flows through TURN server. No direct UDP needed.
turn_server = "turn:my-turn.com:3478"
turn_username = "alice"
turn_password = "secret"
```
All WebRTC media goes through the TURN server. The browser and Bolo server don't need direct UDP connectivity.

### 5.7 str0m Run Loop

A dedicated tokio task (`webrtc_media_loop`) runs outside the axum HTTP server:

```rust
async fn webrtc_media_loop(
    udp_socket: UdpSocket,
    mut command_rx: mpsc::Receiver<MediaCommand>,
    peers: Arc<Mutex<HashMap<PeerId, Rtc>>>,
) {
    let mut buf = vec![0u8; 2000];
    loop {
        // Drain all str0m outputs (transmit packets, events, timeouts)
        for (peer_id, rtc) in peers.lock().unwrap().iter_mut() {
            while let Ok(output) = rtc.poll_output() {
                match output {
                    Output::Transmit(t) => {
                        let _ = udp_socket.send_to(&t.contents, t.destination).await;
                    }
                    Output::Event(e) => handle_event(peer_id, e).await,
                    Output::Timeout(deadline) => { /* record earliest deadline */ }
                }
            }
        }

        // Wait for next UDP packet, timeout, or app command
        let timeout = get_earliest_deadline(&peers);
        tokio::select! {
            Ok((n, src)) = udp_socket.recv_from(&mut buf) => {
                // Demux packet to correct peer by (src_ip, src_port)
                if let Some(peer) = find_peer_by_address(&peers, src) {
                    let input = Input::Receive(Instant::now(), Receive { ... });
                    peer.handle_input(input).unwrap();
                }
            }
            Some(cmd) = command_rx.recv() => {
                match cmd {
                    MediaCommand::CreatePeer { peer_id, rtc } => { peers.insert(peer_id, rtc); }
                    MediaCommand::WriteAudio { peer_id, data } => { /* feed to str0m writer */ }
                    MediaCommand::ClosePeer(peer_id) => { peers.remove(&peer_id); }
                }
            }
            _ = tokio::time::sleep_until(timeout) => {
                for (_, rtc) in peers.lock().unwrap().iter_mut() {
                    rtc.handle_input(Input::Timeout(Instant::now())).unwrap();
                }
            }
        }
    }
}
```

---

## 6. Shared Optimizations

### 6.1 Voice Clone Prompt Cache

Repeated use of the same voice profile currently re-runs:
1. Audio loading + resampling
2. HuBERT encoding → DAC tokenization
3. Optional Whisper ASR for ref_text

**Optimization:** LRU cache keyed by `profile_id` or `sha256(ref_audio_bytes) + ref_text + preprocess`.

```rust
pub struct VoicePromptCache {
    cache: LruCache<String, DeviceVoiceClonePrompt>,
    capacity: usize,
}
```

**Integration:** Inside `Phase3Pipeline::materialize_device_request()`, check cache before creating prompt from audio. Populate cache on miss.

**Impact:** Saves 200–800ms per repeated-voice synthesis.

### 6.2 Model Pre-warming

`Stage1RuntimePlan` lazily loads the Stage 1 model via `OnceLock`. For streaming, add explicit warmup at server startup.

```rust
impl Stage1RuntimePlan {
    pub fn warmup(&self) -> Result<()> {
        let _ = self.model()?; // Trigger OnceLock initialization
        Ok(())
    }
}
```

Call in `main.rs` after runtime is ready but before accepting traffic.

### 6.3 Fast Preset Mapping

The `Stage0DeterministicConfig` already supports `num_step` as a parameter. Streaming presets are purely a mapping layer:

```rust
impl StreamingPreset {
    pub fn to_stage0_config(&self) -> Stage0DeterministicConfig {
        let mut config = Stage0DeterministicConfig::default();
        config.num_step = match self {
            Self::Quality => 32,
            Self::Balanced => 16,
            Self::Fast => 8,
            Self::Ultra => 4,
        };
        config
    }
}
```

---

## 7. New Configuration

### 7.1 ServerArgs Additions

```rust
#[derive(Debug, Clone, Parser)]
#[command(name = "bolo-voice-service")]
pub struct ServerArgs {
    // ... existing fields ...

    /// Default preset for streaming endpoints.
    #[arg(long, default_value = "fast")]
    pub streaming_preset: String,

    /// Enable WebRTC endpoints.
    #[arg(long)]
    pub enable_webrtc: bool,

    /// UDP port for WebRTC media traffic.
    #[arg(long, default_value = "3478")]
    pub webrtc_udp_port: u16,

    /// Public IP address advertised in ICE host candidate.
    /// If unset, uses the bind address.
    #[arg(long)]
    pub webrtc_public_ip: Option<String>,

    /// STUN server for ICE candidate gathering.
    #[arg(long, default_value = "stun.l.google.com:19302")]
    pub stun_server: String,

    /// TURN server URI (e.g. "turn:my-turn.com:3478").
    #[arg(long)]
    pub turn_server: Option<String>,

    /// TURN username.
    #[arg(long)]
    pub turn_username: Option<String>,

    /// TURN password.
    #[arg(long)]
    pub turn_password: Option<String>,
}
```

### 7.2 ConfigFile Additions

```toml
# bolo-voice-service.toml
streaming_preset = "fast"
enable_webrtc = true
webrtc_udp_port = 3478
webrtc_public_ip = "1.2.3.4"
stun_server = "stun.l.google.com:19302"
turn_server = "turn:my-turn.com:3478"
turn_username = "alice"
turn_password = "secret"
```

---

## 8. Implementation Phases

### Phase 1: Streaming HTTP (Days 1–3)

1. Add `streaming.rs` module with `StreamingSynthesizer`
2. Add `streaming_handler.rs` with `POST /v1/stream/speech`
3. Update `server.rs` `build_router()`
4. Add `StreamingPreset` to `params.rs`
5. Add voice prompt LRU cache to `bolo-tts-infer`
6. Test with curl

### Phase 2: WebSocket Bidirectional (Days 4–5)

1. Reuse `StreamingSynthesizer`
2. Add `GET /v1/stream/ws` handler
3. Implement JSON + binary message protocol
4. Integrate ASR pipeline (buffer → VAD → transcribe)
5. Test with HTML client

### Phase 3: ASR Streaming Improvements (Days 6–7)

1. Tune VAD in existing `/v1/ws/transcribe` (configurable threshold + silence timeout)
2. Add sliding-window partial transcription to `/v1/stream/ws`
3. Add ASR config to `ConfigFile`

### Phase 4: WebRTC Full Stack (Days 8–14)

1. Add `str0m` + `opus` dependencies
2. Create `webrtc/` module tree
3. Implement signaling endpoints (`/v1/webrtc/offer`, `/v1/webrtc/ice`)
4. Implement media loop task
5. Implement Opus encode/decode
6. Implement DataChannel protocol
7. Add STUN/TURN candidate gathering
8. Add WebRTC config to `ServerArgs`
9. Test with browser

### Phase 5: Polish & Documentation (Days 15–17)

1. Error handling (WebRTC disconnects, Opus failures)
2. Metrics (streaming latency histograms, connection counts)
3. Update `AGENTS.md`, `skills.txt`, `vapi-support.txt`
4. Browser client example HTML/JS
5. CI/Docker updates for libopus dependency

---

## 9. Testing Strategy

### Unit Tests

- `StreamingSynthesizer::split_sentences` — verify `chunk_text_punctuation` integration
- `StreamingSynthesizer::crossfade_chunks` — verify smooth audio boundaries
- `VoicePromptCache` — hit/miss behavior, eviction
- `OpusCodec` — encode/decode roundtrip, 24kHz fidelity

### Integration Tests

- `POST /v1/stream/speech` with `preset=fast` — verify chunked response, valid WAV chunks
- `GET /v1/stream/ws` — WebSocket handshake, tts message, audio chunk receipt
- `GET /v1/stream/ws` — binary audio upload, asr result receipt
- `POST /v1/webrtc/offer` — SDP offer/answer exchange, valid answer SDP
- Browser e2e test: connect WebRTC, send DataChannel text, receive Opus audio

### Performance Benchmarks

- Measure first-chunk latency for `preset={quality,balanced,fast,ultra}`
- Measure total latency for 10-word, 30-word, 60-word texts
- Measure voice cache hit improvement (with/without cache)

---

## 10. Browser Client Example

### WebSocket Client

```html
<!DOCTYPE html>
<html>
<head><title>Bolo Streaming</title></head>
<body>
  <button id="connect">Connect</button>
  <button id="speak">Speak</button>
  <button id="listen">Listen</button>
  <audio id="player" controls></audio>

  <script>
    const ws = new WebSocket('wss://localhost:8000/v1/stream/ws');
    ws.binaryType = 'arraybuffer';

    ws.onopen = () => {
      ws.send(JSON.stringify({
        type: 'tts',
        input: 'Hello from the browser.',
        voice: 'auto',
        preset: 'fast'
      }));
    };

    const chunks = [];
    ws.onmessage = (event) => {
      const msg = JSON.parse(event.data);
      if (msg.type === 'tts.audio') {
        const bytes = Uint8Array.from(atob(msg.data), c => c.charCodeAt(0));
        chunks.push(bytes);
        if (msg.final) {
          const blob = new Blob(chunks, { type: 'audio/wav' });
          document.getElementById('player').src = URL.createObjectURL(blob);
        }
      }
      if (msg.type === 'asr.result') {
        console.log('ASR:', msg.text);
      }
    };
  </script>
</body>
</html>
```

### WebRTC Client (Simplified)

```javascript
const pc = new RTCPeerConnection({
  iceServers: [{ urls: 'stun:stun.l.google.com:19302' }]
});

// Add microphone audio track
const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
stream.getTracks().forEach(track => pc.addTrack(track, stream));

// Create data channel for text
const dc = pc.createDataChannel('control');
dc.onmessage = (e) => {
  const msg = JSON.parse(e.data);
  if (msg.type === 'asr') console.log('ASR:', msg.text);
  if (msg.type === 'tts.status') console.log('TTS status:', msg.state);
};

// Send TTS request
dc.send(JSON.stringify({ type: 'tts', text: 'Hello', preset: 'fast' }));

// Create offer and send to server
const offer = await pc.createOffer();
await pc.setLocalDescription(offer);

const res = await fetch('/v1/webrtc/offer', {
  method: 'POST',
  headers: { 'Authorization': 'Bearer TOKEN', 'Content-Type': 'application/json' },
  body: JSON.stringify({ sdp: offer.sdp })
});
const { peer_id, sdp, candidates } = await res.json();
await pc.setRemoteDescription({ type: 'answer', sdp });

// Send ICE candidates
dc.onopen = () => {
  pc.onicecandidate = async (e) => {
    if (e.candidate) {
      await fetch('/v1/webrtc/ice', {
        method: 'POST',
        headers: { 'Authorization': 'Bearer TOKEN', 'Content-Type': 'application/json' },
        body: JSON.stringify({ peer_id, candidates: [e.candidate.candidate] })
      });
    }
  };
};
```

---

## 11. Risks & Mitigations

| Risk | Likelihood | Impact | Mitigation |
|------|-----------|--------|------------|
| **libopus build failure** | Medium | High | Add `libopus-dev` to Dockerfile/CI. Provide fallback to WAV-only streaming if opus unavailable. |
| **str0m crypto backend conflict** | Low | Medium | Use `rust-crypto` feature (pure Rust). Document backend selection. |
| **Browser SDP quirks** | Medium | Medium | Test Chrome/Firefox/Safari. Provide WebSocket fallback endpoint. |
| **UDP port firewall/NAT** | High | High | Document port requirements. TURN relay mitigates most cases. |
| **500ms target not met on CPU** | High | High | Document GPU requirement for streaming presets. Add latency telemetry. |
| **Memory leak in peer map** | Medium | High | Implement peer timeout (5min idle → disconnect). Add connection count metrics. |
| **Opus decode drift** | Low | Medium | Use 20ms fixed frames. Reset decoder on jitter > 100ms. |

---

## 12. File Inventory

### New Files

| Path | Description |
|------|-------------|
| `docs/streaming-webrtc-plan.md` | This document |
| `crates/bolo-tts-server/src/streaming.rs` | `StreamingSynthesizer`, sentence splitting, chunk loop |
| `crates/bolo-tts-server/src/streaming_handler.rs` | HTTP + WebSocket handlers |
| `crates/bolo-tts-server/src/webrtc/mod.rs` | WebRTC module root |
| `crates/bolo-tts-server/src/webrtc/signaling.rs` | SDP/ICE HTTP handlers |
| `crates/bolo-tts-server/src/webrtc/media_loop.rs` | UDP socket + str0m run loop |
| `crates/bolo-tts-server/src/webrtc/session.rs` | Per-peer session state |
| `crates/bolo-tts-server/src/webrtc/opus_codec.rs` | Opus encode/decode wrapper |
| `crates/bolo-tts-infer/src/cache.rs` | `VoicePromptCache` LRU |

### Modified Files

| Path | Changes |
|------|---------|
| `crates/bolo-tts-server/Cargo.toml` | Add `str0m`, `opus` dependencies |
| `crates/bolo-tts-server/src/server.rs` | Add new routes |
| `crates/bolo-tts-server/src/args.rs` | Add streaming + WebRTC CLI flags |
| `crates/bolo-tts-server/src/runtime.rs` | Add `warmup()` method, WebRTC config |
| `crates/bolo-tts-server/src/lib.rs` | Export new modules |
| `crates/bolo-tts-server/src/main.rs` | Start media loop task, call warmup |
| `crates/bolo-tts-infer/src/pipeline.rs` | Integrate voice prompt cache |
| `crates/bolo-tts-infer/src/stage1_decoder.rs` | Add `warmup()` method |
| `AGENTS.md` | Document new endpoints |

---

*Document version: 1.0*  
*Last updated: 2026-06-02*