# Bolo WebRTC API — Real-Time Bidirectional Voice

> **Status:** Implemented  
> **Last updated:** 2026-06-08  
> **Scope:** Browser-compatible WebRTC peer connection for bidirectional audio (microphone → ASR → TTS → speakers)

---

## 1. Overview

The Bolo Voice Service exposes a **WebRTC** endpoint that lets browsers (or any WebRTC client) establish a peer-to-peer UDP connection for real-time bidirectional audio:

- **In** (browser → server): Opus-encoded microphone audio → ASR transcription
- **Out** (server → browser): TTS-generated audio streamed as Opus RTP packets
- **Control**: JSON messages exchanged over a WebRTC DataChannel

This enables use cases like:
- **Voice chatbots** — user speaks, bot transcribes, thinks, and replies with synthesized speech
- **Echo chamber / voice repeater** — transcribe and immediately play back in a different voice
- **Real-time voice assistants** — low-latency speech I/O without polling HTTP

---

## 2. Architecture

```
┌─────────────┐      UDP (SRTP)      ┌─────────────────────────────────────┐
│   Browser   │  ←────────────────→  │  Bolo Server                        │
│             │     ICE + DTLS         │                                     │
│ ┌─────────┐ │                      │ ┌─────────────────────────────────┐   │
│ │ getUser │ │  opus/48000/2        │ │ str0m (WebRTC stack)            │   │
│ │ Media   │ │  ─────────────────→  │ │  • ICE, DTLS, SRTP, SCTP        │   │
│ └─────────┘ │                      │ │  • DataChannel (SCTP)           │   │
│             │                      │ └─────────────────────────────────┘   │
│ ┌─────────┐ │                      │           │                         │
│ │ <audio> │ │  opus/48000/2        │     ┌─────┴─────┐                   │
│ │ element │ │  ←─────────────────  │     │           │                   │
│ └─────────┘ │                      │ ┌───▼────┐ ┌────▼────┐              │
│             │                      │ │ Opus   │ │ Opus    │              │
│  DataChannel│  JSON control       │ │ Decoder│ │ Encoder │              │
│  (control)  │  ←───────────────→  │ │48k→24k │ │24k→opus │              │
│             │                      │ └───┬────┘ └────┬────┘              │
│             │                      │     │           │                   │
│             │                      │ ┌───▼────┐ ┌────▼────┐              │
│             │                      │ │ VAD +  │ │ Streaming│              │
│             │                      │ │ ASR    │ │ Synthesizer│              │
│             │                      │ │Whisper │ │ (TTS)    │              │
│             │                      │ └────────┘ └──────────┘              │
└─────────────┘                      └─────────────────────────────────────┘
```

### Components

| Component | Library | Purpose |
|-----------|---------|---------|
| WebRTC stack | `str0m 0.20` | Sans-IO ICE, DTLS, SRTP, SCTP |
| Opus codec | `opus 0.3` | Encode (TTS→browser) / Decode (browser→ASR) |
| ASR | Whisper (GGUF) | Transcribe incoming audio |
| TTS | OmniVoice (Candle) | Synthesize outgoing audio |
| UDP socket | `tokio::net::UdpSocket` | Raw media packet I/O |

---

## 3. Enabling WebRTC

WebRTC is **disabled by default**. Enable it with the `--enable-webrtc` flag or TOML config:

```toml
# bolo-voice-service.toml
enable_webrtc = true
webrtc_udp_port = 3478
webrtc_public_ip = "1.2.3.4"    # your public IP advertised in ICE host candidate
stun_server = "stun.l.google.com:19302"
# Optional TURN fallback:
# turn_server = "turn:my-turn.com:3478"
# turn_username = "alice"
# turn_password = "secret"
```

CLI flags:
```bash
./bolo-tts-server \
  --enable-webrtc true \
  --webrtc-udp-port 3478 \
  --webrtc-public-ip "1.2.3.4" \
  --stun-server "stun.l.google.com:19302"
```

**Firewall:** The server binds the HTTP port (e.g. `8880/tcp`) and the WebRTC UDP port (e.g. `3478/udp`). Both must be reachable from the browser.

---

## 4. Signaling Endpoints

WebRTC requires an out-of-band signaling step to exchange SDP offers/answers and ICE candidates before the peer-to-peer UDP connection can begin.

### 4.1 POST /v1/webrtc/offer

**Purpose:** Exchange SDP offer/answer and create a peer connection.

**Request:**
```http
POST /v1/webrtc/offer
Authorization: Bearer <api-key>
Content-Type: application/json

{
  "sdp": "v=0\r\no=- ...",
  "voice": "auto",
  "preset": "fast",
  "language": "en"
}
```

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `sdp` | string | required | Browser's SDP offer |
| `voice` | string | `"auto"` | Voice profile or built-in name |
| `preset` | string | `"fast"` | `quality`/`balanced`/`fast`/`ultra` → `num_step` |
| `language` | string | `"en"` | Language code |

**Response:**
```json
{
  "peer_id": "peer_1717845123456",
  "sdp": "v=0\r\no=- ...",
  "candidates": [
    "candidate:1 1 UDP 2130706431 192.168.1.109 3478 typ host",
    "candidate:2 1 UDP 1694498815 119.73.109.158 54321 typ srflx raddr 0.0.0.0 rport 0"
  ]
}
```

| Field | Description |
|-------|-------------|
| `peer_id` | Unique peer ID for subsequent ICE and control messages |
| `sdp` | Server's SDP answer |
| `candidates` | Server's ICE candidates (host + srflx + relay if configured) |

**Errors:**
- `401` — Missing or invalid API key
- `422` — WebRTC not enabled (`--enable-webrtc false`)
- `422` — Invalid SDP offer

### 4.2 POST /v1/webrtc/ice

**Purpose:** Add remote (browser) ICE candidates after the initial offer/answer.

**Request:**
```http
POST /v1/webrtc/ice
Authorization: Bearer <api-key>
Content-Type: application/json

{
  "peer_id": "peer_1717845123456",
  "candidates": [
    "candidate:1 1 UDP 2130706431 192.168.1.50 54320 typ host",
    "candidate:2 1 UDP 1694498815 203.0.113.1 54321 typ srflx raddr 192.168.1.50 rport 54320"
  ]
}
```

**Response:**
```json
{
  "candidates": [
    "candidate:1 1 UDP 2130706431 192.168.1.50 54320 typ host",
    "candidate:2 1 UDP 1694498815 203.0.113.1 54321 typ srflx raddr 192.168.1.50 rport 54320"
  ]
}
```

Returns only the candidates that were successfully parsed and forwarded to the ICE stack.

**Note:** If the browser sends all candidates inline in the initial SDP offer (not trickle ICE), this endpoint is optional.

---

## 5. Media Pipeline

Once the peer connection is established, media flows over UDP via SRTP.

### 5.1 Incoming Audio (Browser → Server)

1. Browser encodes microphone audio as **Opus 48kHz stereo**
2. str0m receives SRTP packets, decrypts, depacketizes
3. `OpusDecoder` decodes to **48kHz f32 PCM**
4. Downsampled to **24kHz mono** (the server's internal rate)
5. Buffered in a `Vec<f32>` ring buffer
6. **Energy-based VAD** runs on every 20ms frame:
   - RMS > 0.01 → speech detected, reset silence counter
   - RMS <= 0.01 → increment silence counter
   - Silence > 500ms + at least 500ms of audio → flush buffer as `PeerEvent::AsrAudio`
7. ASR (Whisper) transcribes the buffered audio → text
8. Text sent back to browser via **DataChannel** as `{"type":"asr.result","text":"...","final":true}`

### 5.2 Outgoing Audio (Server → Browser)

1. DataChannel receives a control message (e.g., `{"type":"tts","text":"Hello"}`)
2. `StreamingSynthesizer` splits text into sentences, runs TTS pipeline with fast preset
3. Post-processing: remove silence, peak normalize, fade in/out
4. WAV chunks decoded back to **24kHz f32 PCM**
5. `OpusEncoder` (24kHz mono) encodes 20ms frames into Opus packets
6. str0m `Writer::write()` sends each packet with an **incrementing RTP timestamp** (+960 per 20ms frame at 48kHz media clock)
7. str0m packetizes, SRTP-encrypts, and transmits via UDP
8. Browser's `<audio>` element or `RTCPeerConnection.ontrack` plays the decoded audio

### 5.3 Opus Configuration

| Direction | Sample Rate | Channels | Frame Size | Application |
|-----------|-------------|----------|------------|-------------|
| Encode (TTS→browser) | 24 kHz | Mono | 20ms (480 samples) | `opus::Application::Audio` |
| Decode (browser→ASR) | 48 kHz → 24 kHz | Mono | 20ms (960 samples @ 48k) | Standard decode |

Browsers always offer `opus/48000/2`. The server accepts this and resamples after decode.

---

## 6. DataChannel Protocol

Label: `control`

### 6.1 Client → Server

| Message | Type | Description |
|---------|------|-------------|
| TTS request | `{"type":"tts","text":"Hello world","preset":"fast","voice":"auto","language":"en","speed":1.0}` | Synthesize text and stream audio back |
| Echo test | `{"type":"echo","payload":{...}}` | Server replies with `"ok"` |

**TTS fields:**
- `text` (required) — Text to synthesize
- `preset` — `quality`, `balanced`, `fast`, `ultra`
- `voice` — Profile ID or built-in name
- `language` — ISO language code
- `speed` — Playback speed multiplier
- `seed` — Random seed for reproducibility

### 6.2 Server → Client

| Message | Type | Description |
|---------|------|-------------|
| ASR result | `{"type":"asr.result","text":"Hello world","final":true}` | Transcription of user's speech |
| TTS status | `{"type":"tts.status","state":"started"}` | Synthesis began |
| TTS status | `{"type":"tts.status","state":"done"}` | Synthesis finished |
| TTS status | `{"type":"tts.status","state":"error","message":"..."}` | Synthesis failed |
| Error | `{"type":"error","message":"..."}` | Generic control error |

---

## 7. Echo Chamber Mode

The built-in **echo chamber** behavior automatically bridges ASR → TTS:

1. User speaks into microphone
2. After ~500ms silence, audio is transcribed
3. The transcribed text is **automatically** sent to the TTS synthesizer
4. Synthesized audio is streamed back to the browser
5. User hears their own words spoken back in the TTS voice

This is implemented server-side in the event processor and requires no client-side logic beyond the standard WebRTC connection.

---

## 8. Security

| Layer | Protection |
|-------|-----------|
| Signaling (HTTP) | `Authorization: Bearer <api-key>` required on `/v1/webrtc/offer` and `/v1/webrtc/ice` |
| Media (UDP/SRTP) | DTLS handshake + SRTP encryption negotiated by str0m |
| DataChannel | Encrypted inside the DTLS/SCTP tunnel |

The API key is validated during signaling. Once the peer connection is established, media and DataChannel traffic is encrypted end-to-end by WebRTC's standard DTLS/SRTP stack.

---

## 9. Complete Browser Example

```html
<!DOCTYPE html>
<html>
<head><title>Bolo WebRTC Echo Chamber</title></head>
<body>
  <button id="connect">Connect</button>
  <button id="disconnect" disabled>Disconnect</button>
  <div id="log" style="font-family:monospace;white-space:pre-wrap;background:#1e293b;color:#e2e8f0;padding:16px;border-radius:8px;margin-top:20px;min-height:120px;"></div>
  <audio id="player" autoplay controls style="width:100%;margin-top:20px;"></audio>

  <script>
    const API_HOST = '192.168.1.109'; // your Bolo server IP
    const API_BASE = `http://${API_HOST}:8880`;
    const API_KEY = 'qwerasdf';

    let pc, dc, localStream;
    const log = msg => {
      console.log(msg);
      document.getElementById('log').textContent += msg + '\n';
    };

    async function connect() {
      document.getElementById('connect').disabled = true;
      log('Getting microphone...');
      localStream = await navigator.mediaDevices.getUserMedia({ audio: true });

      log('Creating peer connection...');
      pc = new RTCPeerConnection({ iceServers: [{ urls: 'stun:stun.l.google.com:19302' }] });
      localStream.getTracks().forEach(t => pc.addTrack(t, localStream));

      pc.ontrack = e => {
        log('Remote audio track received');
        document.getElementById('player').srcObject = e.streams[0];
      };

      dc = pc.createDataChannel('control');
      dc.onopen = () => log('DataChannel open');
      dc.onmessage = e => {
        try { log('DC: ' + JSON.stringify(JSON.parse(e.data))); }
        catch { log('DC raw: ' + e.data); }
      };

      // Wait for ICE gathering
      await new Promise(resolve => {
        pc.onicegatheringstatechange = () => { if (pc.iceGatheringState === 'complete') resolve(); };
        setTimeout(resolve, 3000); // fallback
      });

      const offer = await pc.createOffer();
      await pc.setLocalDescription(offer);

      log('Sending offer...');
      const res = await fetch(`${API_BASE}/v1/webrtc/offer`, {
        method: 'POST',
        headers: { 'Authorization': `Bearer ${API_KEY}`, 'Content-Type': 'application/json' },
        body: JSON.stringify({ sdp: pc.localDescription.sdp })
      });
      const { peer_id, sdp, candidates } = await res.json();
      log(`Got answer (peer_id=${peer_id})`);

      await pc.setRemoteDescription({ type: 'answer', sdp });
      candidates.forEach(async c => {
        try { await pc.addIceCandidate({ candidate: c, sdpMid: '0', sdpMLineIndex: 0 }); }
        catch(e) { log('ICE err: ' + e.message); }
      });

      document.getElementById('disconnect').disabled = false;
      log('Connected! Speak into your mic.');
    }

    function disconnect() {
      if (dc) dc.close();
      if (pc) pc.close();
      if (localStream) localStream.getTracks().forEach(t => t.stop());
      document.getElementById('connect').disabled = false;
      document.getElementById('disconnect').disabled = true;
      log('Disconnected.');
    }
  </script>
</body>
</html>
```

Save this as `webrtc_echo.html` and open it in Chrome, Firefox, or Edge. HTTPS is **required** for `getUserMedia()` in production; for LAN testing you can use `http://` if the browser is configured to allow insecure origins for the server's IP.

---

## 10. Configuration Reference

| Flag / TOML Key | Default | Description |
|-----------------|---------|-------------|
| `--enable-webrtc` | `false` | Enable WebRTC endpoints and media loop |
| `--webrtc-udp-port` | `3478` | UDP port for WebRTC media traffic |
| `--webrtc-public-ip` | — | Public IP advertised in ICE host candidate |
| `--stun-server` | `stun.l.google.com:19302` | STUN server for srflx candidate discovery |
| `--turn-server` | — | TURN relay server URI |
| `--turn-username` | — | TURN username |
| `--turn-password` | — | TURN password |

---

## 11. Troubleshooting

| Symptom | Cause | Fix |
|---------|-------|-----|
| "WebRTC is not enabled" (422) | `--enable-webrtc` is false | Enable in config or CLI |
| "failed to bind WebRTC UDP socket" | Port already in use | Change `--webrtc-udp-port` or stop conflicting process |
| No audio heard after connect | Firewall blocking UDP | Open the WebRTC UDP port in `ufw` / router |
| Candidates not reaching server | NAT / symmetric NAT | Configure a TURN relay |
| Connection drops after ~5 min | Idle timeout | Send keepalive DataChannel pings or reconnect |
| `getUserMedia` fails in browser | Not HTTPS | Serve the HTML over HTTPS or allow insecure origin |

---

## 12. Risks & Limitations

1. **Model latency:** OmniVoice is a diffusion model. Even with `preset=fast` (8 steps), TTS latency is ~500–800ms for a short sentence. This is inherent to the architecture.
2. **CPU streaming:** On CPU, first-chunk latency can exceed 2s. GPU is strongly recommended for real-time use.
3. **One peer per connection:** Each browser tab gets its own `peer_id`. There is no broadcast/multicast.
4. **No TURN server built-in:** You must run an external coturn or cloud TURN service if direct UDP is blocked.
5. **Sentence-level streaming:** Audio is chunked at sentence boundaries. There is no true word-level or phoneme-level streaming.

---

*Document version: 1.0*  
*Last updated: 2026-06-08*