# WebSocket Streaming API Reference

> Endpoint: `wss://<host>/v1/stream/ws?api_key=<key>`
>
> Binary type: `arraybuffer`
> Audio format: Int16 PCM, little-endian (s16le)

## Overview

The Bolo Voice Service WebSocket endpoint provides **real-time, bidirectional** speech processing through a modular pipeline:

```
Audio In → VAD → ASR → TTS → Audio Out
```

1. **VAD (Voice Activity Detection)** — Silero neural VAD detects speech/silence
2. **ASR (Automatic Speech Recognition)** — Whisper transcribes speech to text
3. **TTS (Text-to-Speech)** — OmniVoice synthesizes text back to speech

All three blocks are **hot-swappable at runtime** via the WebSocket protocol. You can enable/disable individual blocks, reconfigure them, or swap implementations without reconnecting.

## Demo UI

Try the interactive echo chamber at:
- `https://<host>/websocket`

Features:
- Real-time voice recording and echo playback
- Modular pipeline visualization with status indicators (AEC, VAD, Diarization, Gate, ASR, TTS)
- Live timing stats (ASR latency, TTS first chunk, TTS total, end-to-end)
- Structured event log with category/level filtering
- Voice profile creation (record → transcribe → create) with optional DeepFilterNet noise removal
- Multi-language ASR support
- Acoustic echo cancellation (AEC) via NLMS adaptive filter

## Authentication

Pass the API key as a query parameter:
```
wss://bolovoice.apps.minascode.com/v1/stream/ws?api_key=qwerasdf
<!-- also available at: wss://omnivoice.apps.minascode.com/v1/stream/ws?api_key=qwerasdf -->
```

Or via the `Authorization: Bearer <key>` header (browser WebSockets only support query params).

---

## Client → Server Messages

### `tts` — Text-to-Speech Request

Request the server to synthesize speech from text.

```json
{
  "type": "tts",
  "input": "Hello, how are you?",
  "voice": "alloy",
  "preset": "fast",
  "language": "en",
  "speed": 1.0,
  "instruct": "female, low pitch",
  "seed": 42
}
```

| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `input` | string | **Yes** | — | Text to synthesize |
| `voice` | string | No | `auto` | Voice ID (built-in like `alloy`, or custom profile like `vp_3`) |
| `preset` | string | No | `fast` | Quality/latency tradeoff: `quality`, `balanced`, `fast`, `ultra` |
| `language` | string | No | auto | Language code: `en`, `es`, `ur`, etc. |
| `speed` | float | No | — | Playback speed multiplier |
| `instruct` | string | No | — | Voice design instruction (e.g. `female, low pitch`) |
| `seed` | uint | No | — | Random seed for reproducible output |

**Response flow:**
- Server sends one or more `tts.audio` messages (WAV chunks, base64)
- Server sends `tts.done` when complete

**Example minimal TTS:**
```json
{"type": "tts", "input": "Hello world", "preset": "fast"}
```

---

### `asr.start` — Start Speech Recognition

Tell the server to begin listening for audio on the WebSocket binary channel.

```json
{
  "type": "asr.start",
  "sample_rate": 16000,
  "channels": 1,
  "language": "en"
}
```

| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `sample_rate` | uint | No | `16000` | Audio sample rate in Hz (8000, 16000 supported) |
| `channels` | uint | No | `1` | Number of audio channels (1 = mono, 2 = stereo) |
| `language` | string | No | auto-detect | Language hint for ASR: `en`, `ur`, `es`, etc. Comma-separated for multiple hints. Empty string = auto-detect. |

**Audio streaming:**
After sending `asr.start`, send raw PCM audio as **binary WebSocket messages**.
- Format: Int16, little-endian (s16le)
- Chunk size: any (server buffers internally)
- Example: 4096 samples = 256ms at 16kHz

**ASR behavior:**
- Server uses Silero VAD (neural) if available, falling back to energy-based VAD
- Speech is detected → buffered → transcribed on silence
- Results are sent as `asr.result` with `final: false` (partial) or `final: true` (complete)

**Sending a second `asr.start` updates the language hint without restarting the stream.**

---

### `asr.stop` — Stop Speech Recognition

Stop listening. Any buffered audio is flushed to ASR for final transcription.

```json
{"type": "asr.stop"}
```

---

### `asr.segment` — Browser VAD Mode (Client-Side Segmentation)

When using client-side VAD (like `@ricky0123/vad-web`), send this to tell the server that **each subsequent binary message is a complete utterance** — skip server-side VAD entirely.

```json
{
  "type": "asr.segment",
  "language": "en"
}
```

After this, each binary message is sent directly to ASR as a full utterance.

| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `language` | string | No | auto-detect | Language hint for ASR |

---

### `pipeline.config` — Reconfigure Pipeline Blocks

Hot-swap or reconfigure pipeline blocks at runtime without reconnecting.

```json
{
  "type": "pipeline.config",
  "blocks": [
    {"block_type": "SileroVadBlock", "name": "vad", "enabled": true},
    {"block_type": "WhisperAsrBlock", "name": "asr", "enabled": true},
    {"block_type": "OmniVoiceTtsBlock", "name": "tts", "enabled": true}
  ]
}
```

| Field | Type | Description |
|-------|------|-------------|
| `blocks` | array | Ordered list of blocks. Omitted blocks are disabled. |

**Response:** Server sends `pipeline.configured` confirming the new block chain.

---

### `log.config` — Configure Logging

Change log level or categories at runtime.

```json
{
  "type": "log.config",
  "min_level": "debug",
  "categories": ["pipeline", "asr", "tts"]
}
```

| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `min_level` | string | No | `info` | Minimum level: `error`, `warn`, `info`, `debug`, `trace` |
| `categories` | array | No | all | Which categories to log. Empty = all categories. |

**Levels (inclusive):** trace < debug < info < warn < error

---

### `log.history` — Query Recent Logs

Request recent events from the server's ring buffer.

```json
{
  "type": "log.history",
  "limit": 100
}
```

**Response:** Server sends `log.history` with events array.

---

## Server → Client Messages

### `tts.audio` — Audio Chunk

```json
{
  "type": "tts.audio",
  "format": "wav",
  "data": "<base64-encoded-wav-chunk>",
  "final_chunk": false
}
```

| Field | Type | Description |
|-------|------|-------------|
| `format` | string | Always `"wav"` |
| `data` | string | Base64-encoded WAV audio chunk |
| `final_chunk` | bool | `true` if this is the last chunk |

**Usage:** Decode base64 → play as WAV audio (sample rate 24kHz).

---

### `tts.done` — TTS Complete

```json
{"type": "tts.done"}
```

All audio chunks have been sent.

---

### `asr.result` — Transcription Result

```json
{
  "type": "asr.result",
  "text": "Hello how are you",
  "final": true,
  "duration_ms": 2340,
  "lang_hint": "en",
  "detected_lang": null
}
```

| Field | Type | Description |
|-------|------|-------------|
| `text` | string | Transcribed text |
| `final` | bool | `true` = final result, `false` = partial/interim |
| `duration_ms` | uint | How long ASR took in milliseconds |
| `lang_hint` | string | The language hint that was used |
| `detected_lang` | string | Whisper auto-detected language (null if not available) |

**Partial results** (`final: false`) are sent during active speech for real-time feedback.
**Final results** (`final: true`) are sent after silence is detected.

---

### `log.batch` — Structured Log Events

Batched log events from the server (sent every ~50ms or 50 events).

```json
{
  "type": "log.batch",
  "events": [
    {
      "timestamp_ms": 1704067200000,
      "session_id": "ws-1234",
      "level": "info",
      "category": "asr",
      "block_id": "whisper",
      "stage": "transcribe",
      "message": "Transcribed 'hello'",
      "data": null,
      "elapsed_ms": 1234,
      "source": "server"
    }
  ]
}
```

| Field | Type | Description |
|-------|------|-------------|
| `timestamp_ms` | uint64 | Unix timestamp in milliseconds |
| `session_id` | string | WebSocket session ID |
| `level` | string | `error`, `warn`, `info`, `debug`, `trace` |
| `category` | string | `pipeline`, `websocket`, `asr`, `tts`, `vad`, `client`, `server` |
| `block_id` | string | Which pipeline block generated this (e.g. `silero`, `whisper`, `omnivoice`) |
| `stage` | string | Pipeline stage name |
| `message` | string | Log message |
| `data` | object | Structured data (varies by event) |
| `elapsed_ms` | uint | Timing for this operation |
| `source` | string | `server` or `client` |

---

### `log.history` — Log History Response

```json
{
  "type": "log.history",
  "events": [ /* array of LogEvent */ ]
}
```

Sent in response to `log.history` request.

---

### `pipeline.configured` — Pipeline Configuration Confirmed

```json
{
  "type": "pipeline.configured",
  "blocks": [
    {"block_type": "SileroVadBlock", "name": "vad", "enabled": true},
    {"block_type": "WhisperAsrBlock", "name": "asr", "enabled": true},
    {"block_type": "OmniVoiceTtsBlock", "name": "tts", "enabled": true}
  ]
}
```

Sent after `pipeline.config` is processed.

---

### `status` — Status/Info Message

```json
{
  "type": "status",
  "message": "TTS: First chunk ready in 850ms",
  "level": "info"
}
```

| Level | Color | Examples |
|-------|-------|----------|
| `info` | Blue | `TTS: First chunk ready in 850ms` |
| `success` | Green | `ASR: Transcribed "Hello" in 2340ms` |
| `warning` | Amber | `VAD: Utterance too short, discarding` |
| `error` | Red | `Error: invalid message` |
| `vad` | Purple | `VAD: Speech detected (prob=0.923)` |

---

### `error` — Error Message

```json
{
  "type": "error",
  "message": "runtime is not ready",
  "code": 503
}
```

---

## Usage Patterns

### Pattern 1: TTS Only (No ASR)

```javascript
const ws = new WebSocket('wss://host/v1/stream/ws?api_key=xxx');

ws.onopen = () => {
  ws.send(JSON.stringify({
    type: 'tts',
    input: 'Welcome to the system',
    voice: 'alloy',
    preset: 'fast'
  }));
};

ws.onmessage = (event) => {
  const msg = JSON.parse(event.data);
  if (msg.type === 'tts.audio') playAudio(msg.data);
};
```

**No `asr.start` needed. No microphone needed.**

---

### Pattern 2: ASR Only (No TTS)

```javascript
const ws = new WebSocket('wss://host/v1/stream/ws?api_key=xxx');

ws.onopen = () => {
  ws.send(JSON.stringify({
    type: 'asr.start',
    sample_rate: 16000,
    channels: 1,
    language: 'en'
  }));
  startMicrophoneStream(ws);
};

ws.onmessage = (event) => {
  const msg = JSON.parse(event.data);
  if (msg.type === 'asr.result' && msg.final) {
    console.log('Transcribed:', msg.text);
  }
};
```

**No `tts` messages sent. No audio playback needed.**

---

### Pattern 3: Full Echo Chamber (ASR + TTS)

```javascript
const ws = new WebSocket('wss://host/v1/stream/ws?api_key=xxx');

ws.onopen = () => {
  ws.send(JSON.stringify({type: 'asr.start', sample_rate: 16000, channels: 1}));
  startMicrophoneStream(ws);
};

ws.onmessage = (event) => {
  const msg = JSON.parse(event.data);
  if (msg.type === 'asr.result' && msg.final) {
    ws.send(JSON.stringify({type: 'tts', input: msg.text, preset: 'fast'}));
  } else if (msg.type === 'tts.audio') {
    playAudio(msg.data);
  }
};
```

---

### Pattern 4: Client-Side VAD (No Server VAD)

Use when you have your own VAD (e.g., browser ONNX VAD):

```javascript
ws.onopen = () => {
  ws.send(JSON.stringify({type: 'asr.segment', language: 'en'}));
  // Now each binary message is a complete utterance
};

// On speech end from your VAD:
ws.send(pcm16.buffer); // Complete utterance

// Server sends asr.result immediately
```

---

### Pattern 5: Modular Pipeline Control

Reconfigure blocks at runtime:

```javascript
// Disable TTS temporarily
ws.send(JSON.stringify({
  type: 'pipeline.config',
  blocks: [
    {block_type: 'SileroVadBlock', name: 'vad', enabled: true},
    {block_type: 'WhisperAsrBlock', name: 'asr', enabled: true}
  ]
}));

// Re-enable TTS later
ws.send(JSON.stringify({
  type: 'pipeline.config',
  blocks: [
    {block_type: 'SileroVadBlock', name: 'vad', enabled: true},
    {block_type: 'WhisperAsrBlock', name: 'asr', enabled: true},
    {block_type: 'OmniVoiceTtsBlock', name: 'tts', enabled: true}
  ]
}));
```

---

### Pattern 6: Structured Logging

```javascript
// Set log level to debug
ws.send(JSON.stringify({type: 'log.config', min_level: 'debug', categories: []}));

// Request recent history
ws.send(JSON.stringify({type: 'log.history', limit: 50}));

ws.onmessage = (event) => {
  const msg = JSON.parse(event.data);
  if (msg.type === 'log.batch') {
    for (const ev of msg.events) {
      console.log(`[${ev.level}] [${ev.category}] ${ev.message} (${ev.elapsed_ms}ms)`);
    }
  }
};
```

---

## Audio Format Reference

### PCM → WAV conversion (browser)

```javascript
// Float32 [-1, 1] → Int16 PCM
const pcm16 = new Int16Array(floatSamples.length);
for (let i = 0; i < floatSamples.length; i++) {
  pcm16[i] = Math.max(-1, Math.min(1, floatSamples[i])) * 0x7FFF;
}
ws.send(pcm16.buffer); // Send as ArrayBuffer
```

### WAV playback (browser)

```javascript
const bytes = Uint8Array.from(atob(msg.data), c => c.charCodeAt(0));
const audioBuffer = await audioCtx.decodeAudioData(bytes.buffer.slice(0));
const source = audioCtx.createBufferSource();
source.buffer = audioBuffer;
source.connect(audioCtx.destination);
source.start();
```

---

## Timing Reference

Typical latencies on GPU (CUDA):

| Step | Typical Latency | Description |
|------|----------------|-------------|
| TTS first chunk | 500–1200ms | Time to first audio from text input |
| TTS total | 1500–4000ms | Total synthesis time depending on text length |
| ASR (1s audio) | 800–2000ms | Transcription time |
| ASR (3s audio) | 1500–3000ms | Transcription time |
| VAD detection | 32ms/frame | Silero VAD processes 512-sample frames at 16kHz |
| End-to-end echo | 2000–5000ms | Speech end → TTS audio start |

---

## Server Configuration

VAD behavior can be configured at server startup:

```bash
# Enable/disable VAD (default: true)
bolo-tts-server --enable-vad true

# Adjust VAD threshold (default: 0.01)
bolo-tts-server --vad-threshold 0.02

# Silence timeout in ms (default: 1000)
bolo-tts-server --vad-silence-ms 800

# Partial ASR window in ms (default: 5000)
bolo-tts-server --asr-partial-window-ms 2000
```

Or in TOML config:
```toml
enable_vad = true
vad_threshold = 0.02
vad_silence_ms = 800
asr_partial_window_ms = 2000
```

---

## Architecture

### Modular Pipeline

The server uses a block-based architecture where each processing step is a swappable block:

| Block | Default | Description |
|-------|---------|-------------|
| AEC | `AecBlock` | Acoustic echo cancellation (NLMS adaptive filter) — cancels TTS playback echo from mic input |
| VAD | `SileroVadBlock` | Voice activity detection (Silero ONNX) |
| Diarization | `PyannoteVadBlock` | Speaker identification (Pyannote segmentation ONNX) |
| Gate | `SpeakerGateBlock` | Primary speaker filter — drops non-primary speaker audio |
| ASR | `WhisperAsrBlock` | Speech-to-text (Whisper) |
| TTS | `OmniVoiceTtsBlock` | Text-to-speech (OmniVoice) |

Blocks implement the `AudioBlock` trait with methods:
- `process(audio, sample_rate, logger)` → processed audio + metadata
- `configure(config)` → runtime reconfiguration
- `flush(logger)` → flush buffered state

### Parallel Logging

Log events flow through a non-blocking pipeline:
1. Block logs event via `ParallelLogger`
2. Event pushed to `SyncSender` (std channel)
3. `LogExporter` drains channel in tokio task
4. Events batched (50ms or 50 events) and sent to client via `log.batch`
5. Ring buffer retains last 1000 events per session for history queries

### Concurrency

- Audio processing runs in `tokio::task::spawn_blocking`
- Logging is fully non-blocking (never blocks audio pipeline)
- WebSocket I/O is async (no blocking calls in main loop)
- Per-session semaphore limits concurrent synthesis to 1