╔══════════════════════════════════════════════════════════════╗ ║ Bolo Voice Service API — Agent Skills Reference ║ ╚══════════════════════════════════════════════════════════════════╝ Base URL: http://:[/base_path] Auth: Bearer in the Authorization header Format: All request/response bodies are JSON unless noted. Multipart endpoints accept multipart/form-data. ────────────────────────────────────────────────────────────── 1. HEALTH & INFO (public, no auth required) ────────────────────────────────────────────────────────────── GET / → { "status": "ok", "service": "bolo-voice-service", "author": "Bolo Voice Service" } GET /health → 200 { "status": "ok", ... } → 503 { "status": "starting", ... } (runtime still loading) GET /skills.txt → plain text — this document (public, no auth) GET /AI_SKILL.md → Markdown — verbose agent skill definition (public, no auth) GET /CHANGELOG.md → Markdown — chronological change history for this service (public, no auth) ────────────────────────────────────────────────────────────── 2. TEXT-TO-SPEECH (auth required) ────────────────────────────────────────────────────────────── POST /v1/audio/speech (general purpose) POST /clone (requires ref_audio or voice profile) POST /design (requires instruct) Accepts JSON body or multipart/form-data. JSON body: { "model": " | default", "input": "Text to synthesize", "voice": " | {\"id\": \"\"} | \"auto\" | ", "instructions": "Optional voice design text", // all endpoints accept instruct "response_format": "wav | mp3 | pcm", // default: "wav" "stream_format": "audio | sse", // default: "audio" "speed": 1.0, "ref_audio": "data:audio/wav;base64,", // inline audio "language": "en", "duration": 5.0, "instruct": "Voice style instruction", "ref_text": "Reference transcript", "voice_password": "secret123", // required if profile is password-protected "seed": 42, "num_step": 16, "guidance_scale": 3.0, "t_shift": 6.0, "layer_penalty_factor": 1.2, "position_temperature": 1.0, "class_temperature": 1.0, "preprocess_prompt": true, "postprocess_output": true, "denoise": true, "audio_chunk_duration": 15.0, "audio_chunk_threshold": 30.0, "asr_model": "whisper" } Multipart fields: model, input, voice, instructions, response_format, stream_format, speed, ref_audio (file upload), plus any extra fields above as text parts. Voice resolution rules: - String like "vp_1" → resolves custom profile by ID - Object like {"id": "vp_1"} → same - Built-in names ("alloy", "ash", "coral", "echo", "fable", "ballad", "onyx", "nova", "sage", "shimmer", "verse", "marin", "cedar") → auto mode (no ref_audio needed) - "auto" or absent/empty → auto mode - Custom profile + explicit ref_audio → 422 error - Password-protected profile without voice_password → 401 error Responses: wav → audio/wav binary stream mp3 → audio/mpeg binary stream pcm → audio/pcm raw i16 PCM binary stream sse → text/event-stream with delta/done events SSE events: event: data: {"type":"speech.audio.delta","audio":""} event: data: {"type":"speech.audio.done","usage":{...}} event: data: [DONE] 422 errors: invalid model, empty input, bad voice ref, conflicting params 404 errors: profile/voice ID not found 503 errors: runtime not ready ────────────────────────────────────────────────────────────── 2a. STREAMING HTTP TTS (auth required) ────────────────────────────────────────────────────────────── POST /v1/stream/speech JSON body: { "input": "Text to synthesize", "preset": "fast", // fast | balanced | quality | ultra "voice": "alloy", "language": "en", "speed": 1.0, "response_format": "wav" // wav | pcm } Response: Transfer-Encoding: chunked Content-Type: audio/wav (or audio/pcm) Each chunk is a valid WAV segment. Client appends chunks. Server handles crossfade between sentences. Preset mapping: quality=32 steps, balanced=16, fast=8, ultra=4. Target first-chunk latency: ~500–800ms on GPU with "fast". 422 errors: invalid preset, empty input 503 errors: runtime not ready ────────────────────────────────────────────────────────────── 2b. BIDIRECTIONAL WEBSOCKET (auth required) ────────────────────────────────────────────────────────────── GET /v1/stream/ws (Upgrade: websocket) Protocol: Client → Server (JSON control): { "type": "tts", "input": "Hello", "voice": "auto", "preset": "fast" } { "type": "asr.start", "sampleRate": 16000, "channels": 1 } { "type": "asr.stop" } Client → Server (binary): PCM s16le frames for ASR input Server → Client (JSON): { "type": "tts.audio", "format": "wav", "data": "", "final": false } { "type": "tts.audio", "format": "wav", "data": "", "final": true } { "type": "asr.result", "text": "transcribed text", "final": true } { "type": "asr.result", "text": "partial text", "final": false } { "type": "error", "message": "...", "code": 422 } ASR: energy-based VAD, 500ms silence → final transcription. Optional sliding-window partials every ~5s (configurable). ────────────────────────────────────────────────────────────── 2c. WEBRTC (auth required — only when --enable-webrtc) ────────────────────────────────────────────────────────────── POST /v1/webrtc/offer { "sdp": "v=0\r\no=- ...", "voice": "auto", "preset": "fast" } → { "peer_id": "peer_...", "sdp": "v=0\r\no=- ...", "candidates": [...] } POST /v1/webrtc/ice { "peer_id": "peer_...", "candidates": ["candidate:..."] } → { "ok": true } Media pipeline: Browser microphone → Opus RTP → Server → decode → 24kHz PCM → ASR TTS → 24kHz PCM → Opus encode → RTP → Browser speakers Control protocol via DataChannel (label: "control"): Client → Server: {"type":"tts","text":"Hello","preset":"fast"} Server → Client: {"type":"asr","text":"Hello","final":true} Config: --enable-webrtc, --webrtc-udp-port, --webrtc-public-ip, --stun-server, --turn-server, --turn-username, --turn-password ────────────────────────────────────────────────────────────── 3. VOICE PROFILES — CRUD (auth required) ────────────────────────────────────────────────────────────── POST /v1/voices/profiles → create profile GET /v1/voices/profiles → list profiles GET /v1/voices/profiles/{id} → get profile by ID PUT /v1/voices/profiles/{id} → update name/description DELETE /v1/voices/profiles/{id} → delete profile POST /v1/voices/profiles (multipart): file (required): audio file (wav/mp3/ogg/opus/flac/aac) name (optional): profile name (default: "unnamed") description (optional): profile description ref_text (optional): reference transcript for the audio - When provided, synthesis skips Whisper ASR and uses this transcript directly. - This avoids ASR errors (e.g. poor transcription of the profile audio). language (optional): language code for the profile (e.g. "en", "ur", "zh") - When provided, this language is used as the default for synthesis with this profile. - If the TTS request also specifies a language, the request-level language takes precedence. password | voice_password (optional): password-protect the profile - Only the creator knows the plaintext password. - Server stores the bcrypt hash (never the plaintext). - Anyone using this profile for synthesis must provide voice_password. Response (create/get/list): { "id": "vp_1", "name": "My Voice", "description": "Warm male voice", "ref_text": "Reference transcript", "language": "en", "created_at": "2025-01-01T00:00:00.000000Z", "updated_at": "2025-01-01T00:00:00.000000Z", "audio_path": "/path/to/profiles/vp_1/ref_audio.wav", "format": "wav", "sample_rate": 24000, "duration_seconds": 3.5 } PUT /v1/voices/profiles/{id} (JSON): { "name": "...", "description": "...", "ref_text": "...", "language": "...", "password": "..." } (at least one required) → returns updated profile NOTE: password is re-hashed with bcrypt on update. Send empty string "" to remove. DELETE → 204 No Content (or 404 if not found) Audio upload rules: - Accepts WAV, MP3, OGG, OPUS, FLAC, AAC - Auto-converted to 24kHz mono 16-bit WAV - Empty or corrupt audio → 422 ────────────────────────────────────────────────────────────── 3a. VOICE PROFILES — BACKFILL (auth required) ────────────────────────────────────────────────────────────── POST /v1/voices/profiles/backfill Auto-fills missing ref_text and/or language for every custom profile by running Whisper ASR on the stored reference audio. Profiles that already have both fields are left untouched. Idempotent — safe to re-run; only missing fields are (re)computed. Per profile: - language missing → detected via Whisper language ID - ref_text missing → transcribed via Whisper ASR (language is detected first so it can hint the transcription) Request body: none (empty). All custom profiles are processed. Response (200): { "object": "list", "processed": 3, "updated": 2, "skipped": 1, "failed": 0, "data": [ { "id": "vp_1", "name": "My Voice", "ref_text": "transcribed transcript", "language": "en", "status": "updated" }, { "id": "vp_2", "name": "Already set", "ref_text": "...", "language": "en", "status": "skipped" }, { "id": "vp_3", "name": "Bad audio", "status": "error", "error": "failed to read profile audio: ..." } ] } status values: "updated" | "skipped" | "error" 401: missing/invalid auth 503: runtime not ready (ASR model still loading) 408: backfill exceeded the request timeout ────────────────────────────────────────────────────────────── 4. LIST ALL VOICES (auth required) ────────────────────────────────────────────────────────────── GET /v1/voices → { "object": "list", "data": [ { "id": "alloy", "type": "builtin", "name": "alloy", "description": "Neutral, well-rounded voice" }, { "id": "vp_1", "type": "custom", "name": "My Voice", "description": "...", "language": "en", "created_at": "...", "audio_path": "...", "format": "wav", "sample_rate": 24000, "duration_seconds": 3.5 } ] } Built-in voices (14): auto, alloy, ash, ballad, coral, echo, fable, onyx, nova, sage, shimmer, verse, marin, cedar ────────────────────────────────────────────────────────────── 8. VAPI-COMPATIBLE CUSTOM TRANSCRIBER (WebSocket, auth required) ────────────────────────────────────────────────────────────── GET /v1/ws/transcribe (WebSocket upgrade) Implements Vapi's custom transcriber protocol for real-time speech-to-text using the configured Whisper ASR model. Protocol: 1. Vapi connects via WebSocket 2. Vapi sends a start message: { "type": "start", "encoding": "linear16", "container": "raw", "sampleRate": 16000, "channels": 2 } 3. Vapi streams binary PCM s16le audio frames 4. Server detects utterance boundaries via energy-based VAD and transcribes each utterance with Whisper 5. Server sends transcription results: { "type": "transcriber-response", "transcription": "transcribed text", "channel": "customer", "transcriptType": "final" } 6. On disconnect, remaining audio is transcribed Authentication: - X-VAPI-SECRET header (matched against --vapi-secret) - Falls back to Authorization: Bearer 401 errors: missing or invalid authentication 503 errors: ASR model not loaded / runtime not ready ────────────────────────────────────────────────────────────── 5. SPEECH-TO-TEXT / TRANSCRIPTION (auth required) ────────────────────────────────────────────────────────────── POST /v1/audio/transcriptions Multipart fields: file (required): Audio file upload (WAV, MP3, OGG) model (required): Model name, e.g. "whisper-1" response_format: "json" (default) or "text" language: Language hint (e.g. "en", "ur", "hi", "fr") prompt: Accepted for API compatibility, NOT passed to Whisper decoder (injection causes hallucinations) temperature: Sampling temperature (0.0 – 1.0) script_mode: "auto" (default) or "native"; controls Devanagari→Urdu transliteration and English loanword preservation: • auto: Urdu Arabic + English loanwords converted back to Latin (سیٹنگز → settings) • native: Urdu Arabic only, English loanwords remain in Urdu script "devanagari" rejected as invalid Urdu / Hindi script normalization: When language=ur is set: 1. Whisper gets the <|ur|> language token to bias toward Urdu (safe — part of Whisper's standard input format) 2. Post-processing detects any Devanagari characters and converts them to Urdu Arabic script via a character map + common word dictionary (or external LLM if configured) 3. English loanword preservation (script_mode=auto only): Urdu-script English words like "سیٹنگز" are converted back to Latin ("settings") via dictionary or external LLM 4. script_mode=auto: Devanagari→Urdu + English preserved script_mode=native: Devanagari→Urdu only, English loanwords remain in Urdu script Prompt injection into Whisper decoder is UNSAFE — causes hallucination loops and timestamp token leakage. The prompt field is accepted for API compatibility only. Limits: Max file size: 10 MB Auto-chunking: Audio longer than ~12 seconds is split into overlapping segments server-side. Results are concatenated — the client sends one file and receives one transcript. Format support: WAV: primary format, decoded via Symphonia + hound fallback MP3: decoded via Symphonia OGG: decoded via Symphonia (Opus/Vorbis containers) Fallback: If Symphonia fails, the server tries `ffmpeg` CLI (outputs raw PCM s16le, bypasses hound WAV header parsing) Responses: json → 200 { "text": "transcribed content" } text → 200 text/plain body with raw transcription 422 errors: missing file or model, corrupt audio, file too large 503 errors: ASR model not loaded / runtime not ready ────────────────────────────────────────────────────────────── 6. OPENAI-COMPATIBLE VOICE ENDPOINTS (auth required) ────────────────────────────────────────────────────────────── GET /v1/audio/voices → list custom voices → { "object": "list", "data": [ { "id": "vp_1", "name": "...", "object": "audio.voice", "created_at": "..." } ] } POST /v1/audio/voices → create voice (multipart) Same as /v1/voices/profiles but returns OpenAI shape GET /v1/audio/voices/{id} → get voice detail DELETE /v1/audio/voices/{id} → delete voice → 204 ────────────────────────────────────────────────────────────── 7. LEGACY ENDPOINTS (auth required) ────────────────────────────────────────────────────────────── GET /profiles → same as /v1/voices/profiles POST /profiles → same as /v1/voices/profiles (multipart) GET /profiles/{id} → same as /v1/voices/profiles/{id} DELETE /profiles/{id} → same as /v1/voices/profiles/{id} ────────────────────────────────────────────────────────────── 8. MODELS (auth required) ────────────────────────────────────────────────────────────── GET /v1/models → { "object": "list", "data": [ { "id": "", "object": "model", "created": 0, "owned_by": "Bolo Voice Service" } ] } ────────────────────────────────────────────────────────────── 7. VAPI-COMPATIBLE TTS WEBHOOK (auth optional via --vapi-secret) ────────────────────────────────────────────────────────────── POST /v1/synthesize Accepts Vapi's custom TTS webhook format and returns raw PCM audio. Request body: { "message": { "type": "voice-request", "text": "Text to synthesize", "sampleRate": 24000 } } Authentication (in priority order): 1. X-VAPI-SECRET header (matched against --vapi-secret) 2. Authorization: Bearer (standard auth) Supported sample rates: 8000, 16000, 22050, 24000, 44100 Hz Response: Content-Type: application/octet-stream Body: raw PCM s16le mono audio at the requested sample rate 400 errors: invalid message type, unsupported sample rate, empty text 422 errors: missing or malformed request body Example: curl -X POST http://localhost:8080/v1/synthesize \ -H "Content-Type: application/json" \ -H "X-VAPI-SECRET: your-secret" \ -d '{ "message": { "type": "voice-request", "text": "Hello from Vapi!", "sampleRate": 24000 } }' --output speech.pcm ────────────────────────────────────────────────────────────── ERROR RESPONSES ────────────────────────────────────────────────────────────── All errors: { "error": { "message": "...", "type": "...", "code": ... } } 400 — Bad request / invalid input 401 — Missing or invalid Authorization header 404 — Resource not found (profile, voice) 415 — Unsupported media type 422 — Validation error (bad params, empty audio, conflicts) 500 — Internal server error 503 — Runtime not ready ────────────────────────────────────────────────────────────── EXAMPLE: Basic TTS (JSON) ────────────────────────────────────────────────────────────── curl -X POST http://localhost:8080/v1/audio/speech \ -H "Authorization: Bearer test_key" \ -H "Content-Type: application/json" \ -d '{ "model": "default", "input": "Hello, this is a test.", "voice": "alloy", "response_format": "wav" }' --output speech.wav ────────────────────────────────────────────────────────────── EXAMPLE: Clone with multipart ────────────────────────────────────────────────────────────── curl -X POST http://localhost:8080/clone \ -H "Authorization: Bearer test_key" \ -F 'model=default' \ -F 'input=Hello, clone this voice.' \ -F 'ref_audio=@/path/to/sample.wav' \ --output cloned.wav ────────────────────────────────────────────────────────────── EXAMPLE: Transcribe audio ────────────────────────────────────────────────────────────── curl -X POST http://localhost:8080/v1/audio/transcriptions \ -H "Authorization: Bearer test_key" \ -F 'file=@recording.wav' \ -F 'model=whisper-1' \ -F 'response_format=json' → {"text":"transcribed text here"} ────────────────────────────────────────────────────────────── EXAMPLE: Create profile (no password) ────────────────────────────────────────────────────────────── curl -X POST http://localhost:8080/v1/voices/profiles \ -H "Authorization: Bearer test_key" \ -F 'file=@sample.wav' \ -F 'name=My Voice' \ -F 'description=Warm recording' \ -F 'ref_text=This is the spoken text from the sample audio' ────────────────────────────────────────────────────────────── EXAMPLE: Create password-protected profile ────────────────────────────────────────────────────────────── curl -X POST http://localhost:8080/v1/voices/profiles \ -H "Authorization: Bearer test_key" \ -F 'file=@sample.wav' \ -F 'name=Premium Voice' \ -F 'password=mysecret' # Response: { "id": "vp_2", ... } # Only someone with "mysecret" can use vp_2 for synthesis. ────────────────────────────────────────────────────────────── EXAMPLE: Synthesize with a password-protected profile ────────────────────────────────────────────────────────────── curl -X POST http://localhost:8080/v1/audio/speech \ -H "Authorization: Bearer test_key" \ -H "Content-Type: application/json" \ -d '{ "model": "default", "input": "Hello", "voice": "vp_2", "voice_password": "mysecret" }' ────────────────────────────────────────────────────────────── EXAMPLE: SSE streaming ────────────────────────────────────────────────────────────── curl -N -X POST http://localhost:8080/v1/audio/speech \ -H "Authorization: Bearer test_key" \ -H "Content-Type: application/json" \ -d '{ "model": "default", "input": "Stream this text.", "voice": "auto", "stream_format": "sse" }' ────────────────────────────────────────────────────────────── 9. URDU VOICE GENERATION GUIDE ────────────────────────────────────────────────────────────── Base Configuration: Endpoint: POST /v1/audio/speech Auth: Bearer in the Authorization header Content-Type: application/json or multipart/form-data Request Format: { "model": "", "input": "Text to synthesize — see formatting rules below", "voice": "", "response_format": "mp3" } Text Formatting Rules (CRITICAL for Urdu): Rule 1: ALWAYS use native Urdu script (اردو) for Urdu words and phrases. Rule 2: Keep English names and technical terms in Latin script — e.g. "Samra", "AI", "message", "WhatsApp". Rule 3: NEVER send Roman Urdu transliteration (e.g. do NOT send "maine apnay" — translate to "میں نے اپنے"). Rule 4: The input field should contain a natural mix of native Urdu + Latin-script English names/terms. Translation Pipeline: 1. Identify proper nouns, names, and technical terms → keep in Latin script. 2. Translate all remaining text into native Urdu script (اردو). 3. Combine and send. Voice Selection: Custom profile: Use the profile_id assigned by the server (e.g. "vp_1"). Do NOT include a ref_audio — the server resolves the stored profile automatically. Note: Sending a profile_id in voice plus an explicit ref_audio in the same request will return 422 Conflict. Built-in presets: The server supports 14 built-in voices (auto, alloy, ash, ballad, coral, echo, fable, onyx, nova, sage, shimmer, verse, marin, cedar). These do not need a ref_audio. Voice design: Send an instructions or instruct field (e.g. "female, young, british accent") to generate a custom voice on-the-fly without a profile. Response Handling: 200 → Valid audio binary (MP3/WAV/PCM). Save and deliver as a voice message. 422 → Invalid input. Common causes: model name mismatch, empty text, conflicting voice + ref_audio, corrupt uploaded audio. 404 → Profile/voice ID not found. 503 → Server runtime not ready (model still loading). Supported Audio Formats for Uploads: Uploads (for profile creation or inline ref_audio): WAV, MP3, OGG accepted. Auto-conversion: Server converts uploaded audio to 24kHz mono 16-bit WAV internally. Empty/corrupt audio → 422 error. Examples — Urdu Translation Pipeline: User Request → API Input ───────────────────────────────────────────────────────── Send to Samra: How are you? → "Samra, آپ کیسے ہیں؟" Tell Ami I am coming late → "Ami, میں دیر سے آؤں گا" Say thanks for the help → "آپ کی مدد کے لیے شکریہ" I made my AI speak in my voice → "میں نے اپنے AI سے اپنی آواز میں بات کروانا شروع کر دی"