---
name: bolo-voice-service
description: Text-to-speech and speech-to-text service with voice cloning, voice profile management, and ASR transcription. Use when the user wants to synthesize speech from text, clone voices, transcribe audio, manage voice profiles, or build voice-enabled AI agent skills.
---

# Bolo Voice Service — Text-to-Speech & Speech-to-Text Service

Bolo Voice Service is an OpenAI-compatible TTS/ASR server with voice cloning, voice profile management, and Whisper-based transcription. It exposes a REST API on `http://<host>:<port>` (default `http://localhost:8888`). All JSON request/response bodies unless noted.

## When to Use This Skill

- User asks to convert text to spoken audio
- User wants to clone a voice from a reference audio sample
- User wants to transcribe audio files to text
- User wants to list, create, or manage voice profiles
- User wants to build an AI agent that speaks or listens
- User wants to integrate TTS/ASR into a chatbot, voice assistant, or accessibility tool
- User mentions OpenAI `/v1/audio/speech` or `/v1/audio/transcriptions` endpoints

## Quick Start

```bash
# Start the server (if not already running)
cargo run -p bolo-tts-server -- --api-key my-key

# Text-to-speech (basic)
curl -X POST http://localhost:8888/v1/audio/speech \
  -H "Authorization: Bearer my-key" \
  -H "Content-Type: application/json" \
  -d '{"model":"default","input":"Hello world","voice":"auto"}' \
  --output speech.wav
```

## Authentication

All endpoints except `/health`, `/`, `/skills.txt`, `/AI_SKILL.md`, and `/CHANGELOG.md` require a Bearer token:

```
Authorization: Bearer <api-key>
```

Set via `--api-key` CLI flag or `BOLO_TTS_API_KEY` env var.

## Endpoints

### Text-to-Speech

Synthesize speech from text.

```
POST /v1/audio/speech
POST /clone                                   # requires ref_audio (voice cloning)
POST /design                                  # requires instruct (voice design)
```

**JSON body:**

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `model` | string | `"default"` | Model ID |
| `input` | string | required | Text to synthesize |
| `voice` | string/object | `"auto"` | Profile ID, `{"id":"vp_xxx"}`, built-in name, or `"auto"` |
| `instructions` | string | — | Voice design instructions (only for `/design`) |
| `response_format` | string | `"wav"` | `"wav"`, `"mp3"`, or `"pcm"` |
| `stream_format` | string | `"audio"` | `"audio"` or `"sse"` |
| `speed` | float | `1.0` | Playback speed |
| `ref_audio` | string (data URI) | — | Base64 WAV inline audio |
| `language` | string | `"en"` | Language code |
| `duration` | float | auto | Target duration in seconds |
| `instruct` | string | — | Voice style instruction |
| `ref_text` | string | — | Reference transcript for cloning |
| `seed` | integer | random | Random seed for reproducibility |
| `num_step` | integer | `16` | Number of diffusion steps |
| `guidance_scale` | float | `3.0` | Classifier-free guidance scale |
| `t_shift` | float | `6.0` | Time shift parameter |
| `layer_penalty_factor` | float | `1.2` | Layer penalty |
| `position_temperature` | float | `1.0` | Position temperature |
| `class_temperature` | float | `1.0` | Class temperature |
| `preprocess_prompt` | bool | `true` | Enable prompt preprocessing |
| `postprocess_output` | bool | `true` | Enable output postprocessing |
| `denoise` | bool | `true` | Enable denoising |
| `audio_chunk_duration` | float | `15.0` | Audio chunk duration in seconds |
| `audio_chunk_threshold` | float | `30.0` | Long-form chunking threshold |

**Multipart fields:** Same as above, with `ref_audio` as a file upload or data URI text field.

**Voice resolution rules:**
- String `"vp_1"` or object `{"id":"vp_1"}` → resolves custom profile by ID
- Built-in names (`"alloy"`, `"ash"`, `"coral"`, `"echo"`, `"fable"`, `"ballad"`, `"onyx"`, `"nova"`, `"sage"`, `"shimmer"`, `"verse"`, `"marin"`, `"cedar"`) → auto mode
- `"auto"` or absent/empty → auto mode
- Custom profile + explicit `ref_audio` → 422 error

**SSE streaming:** When `stream_format: "sse"`, the response is a Server-Sent Events stream:

```
data: {"type":"audio","data":"<base64_chunk>","index":0}

data: {"type":"done","reason":"length"}

data: [DONE]
```

### Speech-to-Text (Transcription)

Transcribe audio to text using Whisper ASR.

```
POST /v1/audio/transcriptions
```

**Multipart fields:**

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `file` | file upload | yes | Audio file (WAV, MP3, or OGG) |
| `model` | string | yes | Model name (e.g. `"whisper-1"`) |
| `response_format` | string | no | `"json"` (default) or `"text"` |
| `language` | string | no | Language hint |
| `prompt` | string | no | Transcription prompt |
| `temperature` | float | no | Sampling temperature |

**Limits:**

| Constraint | Value |
|------------|-------|
| Max file size | 10 MB |
| Max duration | 5 minutes (auto-chunked internally) |
| Chunking | Audio >12s is split into overlapping segments server-side. Client sends one file, receives one transcript. |

**Format support:**

- **WAV** — primary format, decoded via Symphonia + hound fallback
- **MP3** — decoded via Symphonia; falls back to `ffmpeg` if decode fails
- **OGG** — decoded via Symphonia (Opus/Vorbis); falls back to `ffmpeg` if decode fails
- **Fallback** — if Symphonia fails, the server tries `ffmpeg` CLI to convert to 24kHz mono 16-bit WAV. Ensure `ffmpeg` is installed on the server path.

**Responses:**

```json
// response_format: "json" (default)
{ "text": "transcribed text here" }

// response_format: "text"
transcribed text here
```

### Voice Profiles (OpenAI-compatible)

```
GET  /v1/audio/voices                          # List all voices (built-in + custom)
```

Response:

```json
{
  "object": "list",
  "data": [
    { "voice_id": "alloy", "name": "alloy", "type": "builtin" },
    { "voice_id": "vp_1",  "name": "My Profile", "type": "custom" }
  ]
}
```

### Voice Profiles (custom CRUD)

```
GET    /v1/voices                              # List all voices (same as /v1/audio/voices)
POST   /v1/voices/profiles                     # Create profile (multipart: name, description, file, ref_text, password)
GET    /v1/voices/profiles                     # List all custom profiles
GET    /v1/voices/profiles/{id}                # Get profile details
PUT    /v1/voices/profiles/{id}                # Update profile (name, description, ref_text, password)
DELETE /v1/voices/profiles/{id}                # Delete profile
```

**Profile metadata (GET response):**

```json
{
  "id": "vp_1",
  "name": "My Voice",
  "description": "A custom voice profile",
  "ref_text": "Reference transcript of the spoken audio",
  "created_at": "2026-05-17T12:00:00Z",
  "updated_at": "2026-05-17T12:00:00Z"
}
```

**`ref_text` (optional):** When provided at creation or via `PUT`, synthesis with this profile skips Whisper ASR and uses the stored transcript directly. This avoids ASR errors on noisy audio or uncommon names.

**`password` (optional):** When provided at creation or via `PUT`, the profile is password-protected. The server stores only a bcrypt hash — never the plaintext. Synthesis requires `voice_password` in the request. Send `"password": ""` to remove protection.

**Password-protected profile synthesis:**
```bash
curl -X POST http://localhost:8888/v1/audio/speech \
  -H "Authorization: Bearer my-key" \
  -d '{"model":"default","input":"Hello","voice":"vp_1","voice_password":"mysecret"}' \
  --output speech.wav
```

**Legacy endpoints (also available):**

```
POST   /profiles                               # Create profile (field: ref_audio)
GET    /profiles                               # List profiles
GET    /profiles/{id}                          # Get profile
PUT    /profiles/{id}                          # Update profile
DELETE /profiles/{id}                          # Delete profile
```

### Health & Info (public, no auth)

```
GET  /              → { "status": "ok", "service": "bolo-voice-service", "author": "Bolo Voice Service" }
GET  /health        → 200 { "status": "ok" } or 503 { "status": "starting" }
GET  /skills.txt    → plain-text API reference
GET  /AI_SKILL.md   → Markdown — full agent skill definition
GET  /CHANGELOG.md  → Markdown — chronological change history
```

### Models

```
GET  /v1/models
```

Returns an OpenAI-compatible list of available models.

## Error Responses

All errors return JSON:

```json
{
  "error": {
    "message": "description of what went wrong",
    "type": "validation_error | auth_error | not_found | timeout_error | service_error"
  }
}
```

| HTTP Status | Meaning |
|-------------|---------|
| 200 | Success |
| 401 | Missing or invalid API key / voice_password |
| 404 | Resource not found (profile, etc.) |
| 422 | Validation error (missing field, conflict, etc.) |
| 503 | Runtime not ready yet |

## Example Workflows

### Basic TTS

```bash
curl -X POST http://localhost:8888/v1/audio/speech \
  -H "Authorization: Bearer my-key" \
  -H "Content-Type: application/json" \
  -d '{"model":"default","input":"Welcome to Bolo Voice Service","voice":"auto"}' \
  --output welcome.wav
```

### Transcribe Audio

```bash
curl -X POST http://localhost:8888/v1/audio/transcriptions \
  -H "Authorization: Bearer my-key" \
  -F "file=@recording.wav" \
  -F "model=whisper-1" \
  -F "response_format=json"
```

### Voice Cloning via Multipart

```bash
curl -X POST http://localhost:8888/clone \
  -H "Authorization: Bearer my-key" \
  -F "model=default" \
  -F "input=This is my cloned voice" \
  -F "ref_audio=@reference.wav;type=audio/wav"
```

### Create and Use Voice Profile

```bash
# Create
curl -X POST http://localhost:8888/v1/voices/profiles \
  -H "Authorization: Bearer my-key" \
  -F "name=My Voice" \
  -F "description=My custom voice" \
  -F "file=@sample.wav"

# Use in TTS
curl -X POST http://localhost:8888/v1/audio/speech \
  -H "Authorization: Bearer my-key" \
  -H "Content-Type: application/json" \
  -d '{"model":"default","input":"Hello from my profile","voice":"vp_1"}'
```

### SSE Streaming

```bash
curl -X POST http://localhost:8888/v1/audio/speech \
  -H "Authorization: Bearer my-key" \
  -H "Content-Type: application/json" \
  -d '{"model":"default","input":"Stream this long text","voice":"auto","stream_format":"sse"}' \
  --no-buffer
```

## Audio Format Details

- **Input:** WAV, MP3, OGG (autodetected from file extension or magic bytes)
- **Output:** WAV (24kHz mono 16-bit PCM), MP3, or raw PCM
- **Profile storage:** 24kHz mono 16-bit WAV on disk
- **Transcription input:** Any supported audio format, resampled automatically

## Building an AI Agent Skill with Bolo Voice Service

When integrating Bolo Voice Service into an AI agent:

1. **TTS for voice responses:** Use `/v1/audio/speech` to convert the agent's text responses to spoken audio. Use SSE streaming (`stream_format: "sse"`) for real-time playback.

2. **Voice cloning for personalized agents:** Upload a reference audio sample via `/clone` or create a voice profile via `/v1/voices/profiles` to give the agent a specific voice.

3. **Transcription for voice input:** Accept user audio recordings and transcribe them via `/v1/audio/transcriptions` to get text that the agent can process.

4. **Voice profile management:** Let users create, list, and manage multiple voice profiles for different characters or use cases.

5. **Error handling:** Always check HTTP status codes. Handle 401 by re-authenticating, 422 by validating inputs, 404 by checking profile IDs.

6. **Audio validation:** Uploaded audio must be non-empty and decodable (WAV/MP3/OGG). Corrupt audio returns 422.

## Urdu Voice Generation Guide

### Base Configuration

```
POST /v1/audio/speech
Authorization: Bearer <token>
Content-Type: application/json
```

```json
{
  "model": "<model_name>",
  "input": "Text to synthesize — see formatting rules below",
  "voice": "<profile_id>",
  "response_format": "mp3"
}
```

### Text Formatting Rules (CRITICAL for Urdu)

- **Rule 1:** ALWAYS use native Urdu script (اردو) for Urdu words and phrases.
- **Rule 2:** Keep English names and technical terms in Latin script — e.g. `Samra`, `AI`, `message`, `WhatsApp`.
- **Rule 3:** NEVER send Roman Urdu transliteration to the API (e.g. do NOT send `"maine apnay"` — translate to `"میں نے اپنے"`).
- **Rule 4:** The `input` field should contain a natural mix of native Urdu + Latin-script English names/terms.

### Translation Pipeline

1. Identify proper nouns, names, and technical terms → keep in Latin script.
2. Translate all remaining text into native Urdu script (اردو).
3. Combine and send.

### Voice Selection

- **Custom profile:** Use the `profile_id` assigned by the server (e.g. `"vp_1"`). Do NOT include a `ref_audio` — the server resolves the stored profile automatically.
  - Sending a `profile_id` in `voice` plus an explicit `ref_audio` in the same request will return `422 Conflict`.
- **Built-in presets:** The server supports 14 built-in voices (`auto`, `alloy`, `ash`, `ballad`, `coral`, `echo`, `fable`, `onyx`, `nova`, `sage`, `shimmer`, `verse`, `marin`, `cedar`). These do not need a `ref_audio`.
- **Voice design:** Send an `instructions` or `instruct` field (e.g. `"female, young, british accent"`) to generate a custom voice on-the-fly without a profile.

### Response Handling

| Status | Meaning |
|--------|---------|
| 200 | Valid audio binary (MP3/WAV/PCM). Save and deliver as a voice message. |
| 422 | Invalid input — model mismatch, empty text, conflicting `voice` + `ref_audio`, corrupt audio. |
| 404 | Profile/voice ID not found. |
| 503 | Server runtime not ready (model still loading). |

### Audio Formats for Uploads

- Uploads: WAV, MP3, OGG accepted.
- Auto-conversion to 24kHz mono 16-bit WAV internally.
- Empty/corrupt audio → 422.

### Urdu Translation Examples

| User Request | API Input |
|---|---|
| Send to Samra: How are you? | `Samra, آپ کیسے ہیں؟` |
| Tell Ami I am coming late | `Ami, میں دیر سے آؤں گا` |
| Say thanks for the help | `آپ کی مدد کے لیے شکریہ` |
| I made my AI speak in my voice | `میں نے اپنے AI سے اپنی آواز میں بات کروانا شروع کر دی` |