{ "nbformat": 4, "nbformat_minor": 0, "metadata": { "accelerator": "GPU", "colab": { "gpuType": "T4", "provenance": [], "toc_visible": true }, "kernelspec": { "display_name": "Python 3", "name": "python3" }, "language_info": { "name": "python" } }, "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# VoiceStudio on Google Colab\n", "\n", "Run the full [VoiceStudio](https://github.com/debpalash/VoiceStudio) app — voice cloning, voice design, video dubbing, and TTS in 646 languages — on a free Colab GPU, web UI included.\n", "\n", "**Before you start:** `Runtime → Change runtime type → T4 GPU` (the notebook also works on CPU, but generation is much slower).\n", "\n", "Run the cells top to bottom. Every cell is idempotent — re-running after a hiccup is always safe.\n", "\n", "**Contents**\n", "\n", "- **Part 1 — Setup & launch (cells 1-7):** GPU check, install, backend launch, open the web UI, API smoke test.\n", "- **Part 2 — Feature tour (cells 8-20):** each major feature as a self-contained API demo with inline playback — multilingual TTS, voice cloning, voice design, voice profiles, transcription, AI-watermark detection, the OpenAI-compatible API, a multi-voice story, a chaptered audiobook, video dubbing, and vocal isolation.\n", "- **Part 3 — Troubleshooting.**\n", "\n", "**How this notebook works (and why):**\n", "\n", "| Piece | Approach | Why |\n", "|---|---|---|\n", "| Backend install | `uv pip install --system .` | Mirrors the official Docker image: installs into Colab's Python and keeps Colab's preinstalled CUDA PyTorch when it matches our pinned torch, instead of re-downloading multi-GB wheels into a venv. The pins (`deploy/torch-constraints.txt`) are passed explicitly so torch, torchaudio and torchvision cannot drift apart (#1357). |\n", "| Web UI | Built in-notebook with `bun` (~2 min, once per session) | Official releases ship desktop installers only — there is no prebuilt standalone web bundle to download. The backend serves the built SPA itself from `frontend/dist`. |\n", "| Opening the app | Colab's built-in kernel port proxy | No third-party tunnel binaries; the URL is private to your Google session. (A `cloudflared` public-URL alternative is documented in the launch cell.) |\n", "\n", "Issues with this notebook are VoiceStudio issues — report them at [debpalash/VoiceStudio/issues](https://github.com/debpalash/VoiceStudio/issues).\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Part 1 — Setup & launch\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── 1. GPU check ────────────────────────────────────────────────────────────\n", "# Confirms the runtime has an NVIDIA GPU. Everything still works on CPU, but\n", "# a single sentence can take minutes instead of seconds.\n", "import shutil\n", "import subprocess\n", "\n", "if shutil.which(\"nvidia-smi\"):\n", " print(subprocess.run([\"nvidia-smi\"], capture_output=True, text=True).stdout)\n", " print(\"GPU detected — you're good to go.\")\n", "else:\n", " print(\"=\" * 74)\n", " print(\"WARNING: no NVIDIA GPU in this runtime.\")\n", " print(\"Fix: menu bar -> Runtime -> Change runtime type -> T4 GPU,\")\n", " print(\"then re-run this notebook from the top.\")\n", " print(\"(Continuing anyway works, but generation will be very slow on CPU.)\")\n", " print(\"=\" * 74)\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── 2. Clone + install (idempotent; first run ~5-8 min) ─────────────────────\n", "import os\n", "import shutil\n", "import subprocess\n", "import sys\n", "\n", "REPO_URL = \"https://github.com/debpalash/VoiceStudio\"\n", "REPO_DIR = \"/content/VoiceStudio\"\n", "BUN = \"/root/.bun/bin/bun\"\n", "\n", "# Generous network timeouts: Colab -> PyPI is usually fast, but model/wheel\n", "# CDNs occasionally stall and uv's default timeout is short.\n", "os.environ.setdefault(\"UV_HTTP_TIMEOUT\", \"300\")\n", "\n", "def run(cmd, *, cwd=None, what=\"\"):\n", " \"\"\"Stream a command's output; stop the cell with an actionable message on failure.\"\"\"\n", " shown = cmd if isinstance(cmd, str) else \" \".join(cmd)\n", " print(f\"\\n$ {shown}\")\n", " rc = subprocess.run(cmd, cwd=cwd, shell=isinstance(cmd, str)).returncode\n", " if rc != 0:\n", " raise SystemExit(\n", " f\"\\nFAILED: {what or shown} (exit code {rc}).\\n\"\n", " \"Scroll up for the underlying error. Re-running this cell is safe.\\n\"\n", " \"Still stuck? Open an issue with the output above:\\n\"\n", " \" https://github.com/debpalash/VoiceStudio/issues\"\n", " )\n", "\n", "# 2a. Clone the repo (reused if already present)\n", "if os.path.isdir(os.path.join(REPO_DIR, \".git\")):\n", " print(f\"Repo already present at {REPO_DIR} — reusing it.\")\n", "else:\n", " run([\"git\", \"clone\", \"--depth\", \"1\", REPO_URL, REPO_DIR], what=\"git clone\")\n", "\n", "# 2b. System packages (ffmpeg: audio/video processing; libsndfile1: soundfile)\n", "run(\"apt-get -qq update && apt-get -qq install -y ffmpeg libsndfile1\", what=\"apt-get install\")\n", "\n", "# 2c. bun — builds the web UI (see the intro cell for why we build it here)\n", "if not os.path.exists(BUN):\n", " run(\"curl -fsSL https://bun.sh/install | bash\", what=\"bun installer\")\n", "os.environ[\"PATH\"] = os.path.dirname(BUN) + os.pathsep + os.environ[\"PATH\"]\n", "run([BUN, \"--version\"], what=\"bun version check\")\n", "\n", "# 2d. Build the frontend (skipped once frontend/dist/index.html exists)\n", "dist_index = os.path.join(REPO_DIR, \"frontend\", \"dist\", \"index.html\")\n", "if os.path.exists(dist_index):\n", " print(\"frontend/dist already built — skipping.\")\n", "else:\n", " # The repo is a bun workspace: install at the repo root, build in frontend/.\n", " run([BUN, \"install\", \"--frozen-lockfile\"], cwd=REPO_DIR, what=\"bun install (workspace)\")\n", " run([BUN, \"run\", \"--cwd\", \"frontend\", \"build\"], cwd=REPO_DIR, what=\"frontend build (vite)\")\n", " if not os.path.exists(dist_index):\n", " raise SystemExit(\n", " \"The frontend build finished but frontend/dist/index.html is missing —\\n\"\n", " \"scroll up for vite errors, then re-run this cell.\"\n", " )\n", "\n", "# 2e. Backend dependencies — same command the official Docker image uses.\n", "if not shutil.which(\"uv\"):\n", " run([sys.executable, \"-m\", \"pip\", \"install\", \"-q\", \"uv\"], what=\"pip install uv\")\n", "# --constraint: `uv pip install` ignores pyproject's `[tool.uv]\n", "# constraint-dependencies` (a project-API setting), so without it torch,\n", "# torchaudio and torchvision resolve on their bare lower bounds and Colab's\n", "# preinstalled torchvision can be left behind a newer torch -- which fails at\n", "# import with \"operator torchvision::nms does not exist\" (#1357).\n", "run([\"uv\", \"pip\", \"install\", \"--system\", \"--no-cache\",\n", " \"--constraint\", \"deploy/torch-constraints.txt\", \".\"],\n", " cwd=REPO_DIR, what=\"backend install (uv pip install --system .)\")\n", "\n", "# 2f. cuDNN 8 side-install for CTranslate2 (WhisperX ASR on GPU). PyTorch 2.8+\n", "# ships cuDNN 9; scripts/setup.py places cuDNN 8 libs where the backend\n", "# preloads them from (/.venv/.../cudnn8_compat). We create the .venv\n", "# directory only so the script has its expected target — no actual venv is\n", "# used on Colab. Non-fatal: without it, transcription falls back to the\n", "# torch-native Whisper backend automatically.\n", "os.makedirs(os.path.join(REPO_DIR, \".venv\"), exist_ok=True)\n", "run([sys.executable, os.path.join(REPO_DIR, \"scripts\", \"setup.py\")], what=\"cuDNN 8 compat setup\")\n", "\n", "# 2g. Sanity check in a fresh interpreter (this kernel may hold a stale torch).\n", "# Imports the backend's own model stack, not just torch — Colab's system\n", "# Python mixes preinstalled and freshly-resolved wheels, and a torchaudio or\n", "# transformers that can't load together only shows up when the model module is\n", "# imported (#1229). Catching it here beats a 5-minute health timeout in cell 5.\n", "run([sys.executable, \"-c\",\n", " # Versions FIRST: if the model-stack import below fails, the cell output\n", " # still shows what was actually installed, which is the single most\n", " # useful line for diagnosing a Colab environment (#1229).\n", " \"import torch, torchaudio, uvicorn, fastapi, transformers; \"\n", " \"print(f'torch {torch.__version__}, torchaudio {torchaudio.__version__}, \"\n", " \"transformers {transformers.__version__}, \"\n", " \"CUDA available: {torch.cuda.is_available()}'); \"\n", " \"from omnivoice.models.omnivoice import OmniVoice; \"\n", " \"from transformers import HiggsAudioV2TokenizerModel; \"\n", " \"print('Install OK - backend model stack imports cleanly')\"],\n", " cwd=REPO_DIR, what=\"import sanity check\")\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── 3. (Optional) Hugging Face token ────────────────────────────────────────\n", "# Only needed for GATED models — e.g. pyannote speaker diarization, used by\n", "# video dubbing for multi-speaker detection. TTS, voice cloning, and voice\n", "# design all work WITHOUT a token, so feel free to skip this cell.\n", "#\n", "# To use one: accept the model terms at\n", "# https://huggingface.co/pyannote/speaker-diarization-3.1\n", "# then add a Colab Secret named HF_TOKEN (key icon in the left sidebar),\n", "# toggle \"Notebook access\" ON, and re-run this cell.\n", "import os\n", "\n", "try:\n", " from google.colab import userdata\n", " _token = userdata.get(\"HF_TOKEN\")\n", " os.environ[\"HF_TOKEN\"] = _token\n", " print(\"HF_TOKEN loaded from Colab Secrets — gated models (diarization) enabled.\")\n", "except Exception:\n", " print(\"No HF_TOKEN Colab Secret found — skipping. (Everything except gated \"\n", " \"diarization models works without it.)\")\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── 4. (Optional, recommended) Pre-download the default voice model ─────────\n", "# The default TTS engine fetches k2-fsa/OmniVoice (a few GB) on its very first\n", "# generation. Downloading it up front makes the first request fast instead of\n", "# a several-minute stall. Safe to re-run: already-downloaded files are reused.\n", "from huggingface_hub import snapshot_download\n", "\n", "path = snapshot_download(\"k2-fsa/OmniVoice\")\n", "print(\"Default TTS model cached at:\", path)\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── 5. Launch the backend ───────────────────────────────────────────────────\n", "# Starts uvicorn on port 3900 and waits for /health. Re-running this cell when\n", "# the backend is already up just reports its status.\n", "import json\n", "import os\n", "import subprocess\n", "import sys\n", "import time\n", "import urllib.request\n", "\n", "REPO_DIR = \"/content/VoiceStudio\"\n", "PORT = 3900\n", "LOG_PATH = \"/content/omnivoice_backend.log\"\n", "HEALTH_URL = f\"http://127.0.0.1:{PORT}/health\"\n", "\n", "def health():\n", " try:\n", " with urllib.request.urlopen(HEALTH_URL, timeout=5) as r:\n", " return json.load(r)\n", " except Exception:\n", " return None\n", "\n", "info = health()\n", "if info:\n", " print(f\"Backend already running — {info}\")\n", "else:\n", " # Clear any half-dead process from a previous run of this cell.\n", " subprocess.run(f\"kill -9 $(lsof -t -i:{PORT}) 2>/dev/null || true\", shell=True)\n", " time.sleep(1)\n", "\n", " env = os.environ.copy()\n", " # Headless-server deployment, same as the official Docker image: relaxes\n", " # the desktop-only loopback origin gate so the proxied browser session can\n", " # reach the settings/system routes.\n", " env[\"OMNIVOICE_SERVER_MODE\"] = \"1\"\n", " # Voices, projects, and generated audio live here. Ephemeral! See the\n", " # troubleshooting cell for persisting it to Google Drive.\n", " env[\"OMNIVOICE_DATA_DIR\"] = \"/content/omnivoice_data\"\n", " env[\"PYTHONUNBUFFERED\"] = \"1\"\n", "\n", " log = open(LOG_PATH, \"ab\")\n", " proc = subprocess.Popen(\n", " [sys.executable, \"-m\", \"uvicorn\", \"main:app\",\n", " \"--app-dir\", \"backend\", \"--host\", \"127.0.0.1\", \"--port\", str(PORT)],\n", " cwd=REPO_DIR, env=env, stdout=log, stderr=subprocess.STDOUT,\n", " )\n", " print(f\"Backend starting (PID {proc.pid}); log: {LOG_PATH}\")\n", "\n", " deadline = time.time() + 300\n", " while time.time() < deadline:\n", " if proc.poll() is not None:\n", " break # process died — report below\n", " info = health()\n", " if info:\n", " break\n", " print(\".\", end=\"\", flush=True)\n", " time.sleep(3)\n", " print()\n", "\n", " if info:\n", " print(f\"Backend is up — {info}\")\n", " if \"cuda\" not in str(info.get(\"device\", \"\")):\n", " print(\"NOTE: device is not CUDA — generation will be slow. \"\n", " \"See cell 1 to enable the GPU runtime.\")\n", " else:\n", " try:\n", " with open(LOG_PATH, \"r\", errors=\"replace\") as f:\n", " tail = \"\".join(f.readlines()[-40:])\n", " except OSError:\n", " tail = \"(no log file found)\"\n", " raise SystemExit(\n", " \"Backend did not become healthy within 5 minutes.\\n\"\n", " f\"--- last lines of {LOG_PATH} ---\\n{tail}\\n\"\n", " \"--- end of log ---\\n\"\n", " \"Fix the error above (usually a missing dependency: re-run cell 2),\\n\"\n", " \"then re-run this cell. Still stuck? Attach the log to an issue:\\n\"\n", " \" https://github.com/debpalash/VoiceStudio/issues\"\n", " )\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── 6. Open the app ─────────────────────────────────────────────────────────\n", "# Serves localhost:3900 to YOUR browser through Colab's built-in kernel port\n", "# proxy — authenticated to your Google session, no third-party tunnel binary.\n", "# A new browser tab opens with the full VoiceStudio UI (allow pop-ups\n", "# for colab.research.google.com if nothing appears).\n", "#\n", "# Want a PUBLIC URL instead (e.g. to open the app on your phone)? Cloudflare's\n", "# quick tunnel works well — run in a new cell:\n", "# !wget -q https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64 -O /usr/local/bin/cloudflared\n", "# !chmod +x /usr/local/bin/cloudflared\n", "# !nohup cloudflared tunnel --url http://127.0.0.1:3900 --no-autoupdate > /content/cloudflared.log 2>&1 &\n", "# !sleep 5 && grep -o \"https://.*trycloudflare.com\" /content/cloudflared.log | head -1\n", "# Anyone with that URL can reach your session — set a share PIN in the app's\n", "# Settings first.\n", "from google.colab import output\n", "\n", "output.serve_kernel_port_as_window(3900)\n", "print(\"If no tab opened, click the link above (and allow pop-ups).\")\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── 7. Smoke test: generate speech through the API ──────────────────────────\n", "# Proves the whole stack end-to-end without touching the UI: /health, then a\n", "# real TTS generation played inline. POST /generate returns the WAV directly.\n", "import json\n", "import urllib.error\n", "import urllib.parse\n", "import urllib.request\n", "\n", "BASE = \"http://127.0.0.1:3900\"\n", "\n", "with urllib.request.urlopen(f\"{BASE}/health\", timeout=10) as r:\n", " print(\"GET /health ->\", json.load(r))\n", "\n", "data = urllib.parse.urlencode({\n", " \"text\": \"VoiceStudio is up and running on Google Colab.\",\n", "}).encode()\n", "print(\"Generating... (first-ever generation loads the model — if you skipped \"\n", " \"cell 4 it also downloads it, which can take several minutes)\")\n", "try:\n", " with urllib.request.urlopen(\n", " urllib.request.Request(f\"{BASE}/generate\", data=data), timeout=1800\n", " ) as r:\n", " wav = r.read()\n", " print(f\"OK — {len(wav)} bytes, duration {r.headers.get('X-Audio-Duration')}s, \"\n", " f\"generated in {r.headers.get('X-Gen-Time')}s (seed {r.headers.get('X-Seed')})\")\n", "except urllib.error.HTTPError as e:\n", " detail = e.read().decode(\"utf-8\", errors=\"replace\")\n", " raise SystemExit(\n", " f\"Generation failed: HTTP {e.code}\\n{detail}\\n\"\n", " \"Check /content/omnivoice_backend.log for the full trace; on a fresh\\n\"\n", " \"session the usual cause is an interrupted model download — re-run\\n\"\n", " \"cell 4, then this cell.\"\n", " )\n", "\n", "OUT = \"/content/omnivoice_smoke.wav\"\n", "with open(OUT, \"wb\") as f:\n", " f.write(wav)\n", "\n", "from IPython.display import Audio, display\n", "display(Audio(OUT))\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Part 2 — Feature tour\n", "\n", "Each cell below demos one major feature straight through the backend's HTTP API — the same API the desktop app and the web UI use. Every cell is self-contained (it regenerates anything it needs), plays its result inline, and states its expected runtime honestly. Requirements: the backend from cell 5 must be running, and **run cell 8 (helpers) once first**.\n", "\n", "Endpoints used, per section: `/generate` (9-12), `/design/describe` (11), `/profiles` (12), `/transcribe` (13), `/watermark/status` + `/watermark/detect` (14), `/v1/audio/*` (15), `/longform/render` (16), `/audiobook` (17), `/dub/*` + `/jobs/*` (18-19).\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 8. Feature-tour helpers\n", "\n", "Shared plumbing for the tour: a `/generate` wrapper that saves WAVs under `/content/omnivoice_demos/`, inline playback, and loud failures that point at the backend log. Run this once before any feature cell.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── 8. Feature-tour helpers (run once before cells 9-20) ────────────────────\n", "import os\n", "\n", "import requests\n", "from IPython.display import Audio, display\n", "\n", "BASE = \"http://127.0.0.1:3900\"\n", "DEMO_DIR = \"/content/omnivoice_demos\"\n", "LOG_HINT = (\"Backend log: /content/omnivoice_backend.log — attach it when \"\n", " \"reporting: https://github.com/debpalash/VoiceStudio/issues\")\n", "os.makedirs(DEMO_DIR, exist_ok=True)\n", "\n", "# Shared sample lines (cells 9/10/12/13 reuse these to stay self-contained).\n", "EN_TEXT = \"This sentence was generated locally, on a free cloud graphics card.\"\n", "ES_TEXT = \"La clonación de voz ya no necesita la nube: todo sucede aquí mismo.\"\n", "BN_TEXT = \"আমার কণ্ঠস্বর এখন ছয়শো ছেচল্লিশটি ভাষায় কথা বলতে পারে।\"\n", "\n", "def check_backend():\n", " try:\n", " requests.get(f\"{BASE}/health\", timeout=5).raise_for_status()\n", " except Exception:\n", " raise SystemExit(\"The backend is not answering — run cell 5 first. \" + LOG_HINT)\n", "\n", "def api_fail(resp, doing):\n", " raise SystemExit(f\"{doing} failed: HTTP {resp.status_code}\\n\"\n", " f\"{resp.text[:2000]}\\n{LOG_HINT}\")\n", "\n", "def speak(text, out_name, files=None, timeout=1800, **form):\n", " \"\"\"POST /generate; save the returned WAV under DEMO_DIR; return (path, headers).\"\"\"\n", " check_backend()\n", " data = {\"text\": text, **{k: str(v) for k, v in form.items()}}\n", " r = requests.post(f\"{BASE}/generate\", data=data, files=files, timeout=timeout)\n", " if r.status_code != 200:\n", " api_fail(r, f\"/generate ({out_name})\")\n", " path = os.path.join(DEMO_DIR, out_name)\n", " with open(path, \"wb\") as f:\n", " f.write(r.content)\n", " return path, r.headers\n", "\n", "def ensure_wav(out_name, text, **form):\n", " \"\"\"Idempotent speak(): reuse the file if an earlier cell already made it.\"\"\"\n", " path = os.path.join(DEMO_DIR, out_name)\n", " if os.path.exists(path) and os.path.getsize(path) > 0:\n", " return path\n", " return speak(text, out_name, **form)[0]\n", "\n", "def play(path, label=None):\n", " if label:\n", " print(label)\n", " display(Audio(path))\n", "\n", "print(\"Helpers ready. Demo artifacts will land in\", DEMO_DIR)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 9. Text-to-speech in many languages\n", "\n", "Three short generations — English, Spanish, Bengali — through `POST /generate`, with the per-request seed and generation time from the response headers. No `language` parameter is passed: the engine autodetects from the text. Expected runtime: ~5-20 s per line on a warm T4 (the very first generation loads the model — minutes if cell 4 was skipped).\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── 9. TTS in many languages ────────────────────────────────────────────────\n", "try:\n", " speak\n", "except NameError:\n", " raise SystemExit(\"Run cell 8 (feature-tour helpers) first.\")\n", "\n", "for code, label, text in [(\"en\", \"English\", EN_TEXT),\n", " (\"es\", \"Spanish\", ES_TEXT),\n", " (\"bn\", \"Bengali\", BN_TEXT)]:\n", " path, h = speak(text, f\"tts_{code}.wav\")\n", " print(f\"{label}: seed={h.get('X-Seed')} gen_time={h.get('X-Gen-Time')}s \"\n", " f\"duration={h.get('X-Audio-Duration')}s\")\n", " play(path)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 10. Voice cloning (zero-shot)\n", "\n", "Clone a voice from a single short reference clip — no training. To stay self-contained, the reference is the English WAV from cell 9 (regenerated if missing): the cloned line should come back in that same voice. A commented variant shows how to clone **your own** voice from an uploaded 3-10 s clip. Expected runtime: ~10-30 s on a warm T4.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── 10. Voice cloning ───────────────────────────────────────────────────────\n", "try:\n", " ensure_wav\n", "except NameError:\n", " raise SystemExit(\"Run cell 8 (feature-tour helpers) first.\")\n", "\n", "ref_path = ensure_wav(\"tts_en.wav\", EN_TEXT)\n", "ref_text = EN_TEXT # transcript of the reference clip (skips auto-transcription)\n", "\n", "with open(ref_path, \"rb\") as ref:\n", " clone_path, h = speak(\n", " \"And this is the very same voice, cloned from three seconds of audio.\",\n", " \"clone_demo.wav\",\n", " files={\"ref_audio\": (\"reference.wav\", ref, \"audio/wav\")},\n", " ref_text=ref_text,\n", " )\n", "print(f\"Cloned in {h.get('X-Gen-Time')}s (seed {h.get('X-Seed')})\")\n", "play(ref_path, \"Reference voice:\")\n", "play(clone_path, \"Cloned voice, new sentence:\")\n", "\n", "# ── Clone YOUR OWN voice instead ──\n", "# from google.colab import files\n", "# uploaded = files.upload() # pick a clean 3-10 s clip of one speaker\n", "# my_clip = next(iter(uploaded))\n", "# with open(my_clip, \"rb\") as ref:\n", "# my_clone, _ = speak(\"Any text you like, in my voice.\", \"my_clone.wav\",\n", "# files={\"ref_audio\": (my_clip, ref)})\n", "# # (no ref_text -> the backend transcribes the clip automatically)\n", "# play(my_clone)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 11. Voice design\n", "\n", "Describe a voice in plain English; `POST /design/describe` maps the description onto the engine's design-parameter space (gender, age, pitch, accent, style) and returns a validated `instruct` string, which `/generate` then uses to synthesize that voice from nothing — no reference audio. The same sentence is rendered with two contrasting designs. Expected runtime: ~10-30 s per voice on a warm T4.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── 11. Voice design ────────────────────────────────────────────────────────\n", "try:\n", " speak\n", "except NameError:\n", " raise SystemExit(\"Run cell 8 (feature-tour helpers) first.\")\n", "\n", "SENTENCE = \"Every voice you hear in this notebook was invented by a description.\"\n", "designs = [\n", " (\"design_a.wav\", \"a deep, elderly male narrator with a low pitch and a british accent\"),\n", " (\"design_b.wav\", \"a young, energetic female voice with a high pitch\"),\n", "]\n", "\n", "for out_name, description in designs:\n", " r = requests.post(f\"{BASE}/design/describe\", json={\"description\": description}, timeout=30)\n", " if r.status_code != 200:\n", " api_fail(r, \"/design/describe\")\n", " d = r.json()\n", " print(f'\"{description}\"')\n", " print(f\" -> instruct: {d['instruct']!r}\")\n", " if d.get(\"unmatched\"):\n", " print(f\" -> not mappable (ignored): {d['unmatched']}\")\n", " path, h = speak(SENTENCE, out_name, instruct=d[\"instruct\"])\n", " play(path, f\" gen_time={h.get('X-Gen-Time')}s\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 12. Voice profiles — save, list, reuse\n", "\n", "Voices become reusable assets: save the cell-9 narrator and the cell-11 designed voice as named profiles (`POST /profiles`), list them (`GET /profiles`), then generate by `profile_id` alone — no reference upload, no instruct string. Cells 16-17 reuse these two profiles. Idempotent: existing profiles with the same names are reused. Expected runtime: seconds, plus one ~10-30 s generation.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── 12. Voice profiles ──────────────────────────────────────────────────────\n", "try:\n", " ensure_wav\n", "except NameError:\n", " raise SystemExit(\"Run cell 8 (feature-tour helpers) first.\")\n", "\n", "check_backend()\n", "\n", "def ensure_profile(name, wav_path, ref_text):\n", " \"\"\"Create a clone profile from a WAV, or reuse it if this name exists.\"\"\"\n", " r = requests.get(f\"{BASE}/profiles\", timeout=30)\n", " if r.status_code != 200:\n", " api_fail(r, \"GET /profiles\")\n", " for p in r.json():\n", " if p.get(\"name\") == name:\n", " print(f\"Profile {name!r} already exists (id={p['id']}) — reusing.\")\n", " return p[\"id\"]\n", " with open(wav_path, \"rb\") as ref:\n", " r = requests.post(\n", " f\"{BASE}/profiles\",\n", " data={\"name\": name, \"ref_text\": ref_text, \"kind\": \"clone\"},\n", " files={\"ref_audio\": (os.path.basename(wav_path), ref, \"audio/wav\")},\n", " timeout=120,\n", " )\n", " if r.status_code != 200:\n", " api_fail(r, \"POST /profiles\")\n", " created = r.json()\n", " print(f\"Created profile {name!r} (id={created['id']})\")\n", " return created[\"id\"]\n", "\n", "NARRATOR_ID = ensure_profile(\n", " \"Colab Narrator\", ensure_wav(\"tts_en.wav\", EN_TEXT), EN_TEXT)\n", "GUEST_SENTENCE = \"Every voice you hear in this notebook was invented by a description.\"\n", "GUEST_ID = ensure_profile(\n", " \"Colab Guest\",\n", " ensure_wav(\"design_b.wav\", GUEST_SENTENCE,\n", " instruct=\"female, young, high pitch\"),\n", " GUEST_SENTENCE)\n", "\n", "profiles = requests.get(f\"{BASE}/profiles\", timeout=30).json()\n", "print(f\"\\n{len(profiles)} saved profile(s):\")\n", "for p in profiles:\n", " print(f\" {p['id']} {p.get('kind', '?'):6s} {p.get('name')}\")\n", "\n", "path, h = speak(\"Generating by profile id — no reference clip attached this time.\",\n", " \"profile_demo.wav\", profile_id=NARRATOR_ID)\n", "play(path, f\"\\nVoice of profile {NARRATOR_ID} (gen_time={h.get('X-Gen-Time')}s):\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 13. Transcription (speech-to-text)\n", "\n", "The round trip: the WAV that TTS produced in cell 9 goes back through `POST /transcribe`, and the recognized text should match the original sentence. Expected runtime: the **first** transcription downloads an ASR model (roughly 1-3 GB, a few minutes); afterwards it's seconds.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── 13. Transcription: TTS -> ASR round trip ────────────────────────────────\n", "try:\n", " ensure_wav\n", "except NameError:\n", " raise SystemExit(\"Run cell 8 (feature-tour helpers) first.\")\n", "\n", "wav = ensure_wav(\"tts_en.wav\", EN_TEXT)\n", "print(\"Transcribing... (first run downloads an ASR model — a few minutes)\")\n", "with open(wav, \"rb\") as f:\n", " r = requests.post(f\"{BASE}/transcribe\",\n", " files={\"audio\": (\"tts_en.wav\", f, \"audio/wav\")},\n", " timeout=1800)\n", "if r.status_code != 200:\n", " api_fail(r, \"POST /transcribe\")\n", "res = r.json()\n", "print(f\"Engine: {res.get('engine')} \"\n", " f\"(audio {res.get('duration_s')}s, transcribed in {res.get('transcription_time_s')}s)\")\n", "print(f\" TTS input : {EN_TEXT}\")\n", "print(f\" ASR output: {res.get('text', '').strip()}\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 14. AI-watermark detection\n", "\n", "Every WAV VoiceStudio generates carries an inaudible AudioSeal watermark by default, so AI-generated audio can be identified later. `POST /watermark/detect` on a generated clip should report high confidence; the same check on a plain ffmpeg-made tone should not. Expected runtime: seconds (first call loads the small detector model).\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── 14. AI watermark: detect generated vs. plain audio ──────────────────────\n", "import subprocess\n", "\n", "try:\n", " ensure_wav\n", "except NameError:\n", " raise SystemExit(\"Run cell 8 (feature-tour helpers) first.\")\n", "\n", "status = requests.get(f\"{BASE}/watermark/status\", timeout=30)\n", "if status.status_code != 200:\n", " api_fail(status, \"GET /watermark/status\")\n", "print(\"Watermark status:\", status.json())\n", "\n", "def detect(path):\n", " with open(path, \"rb\") as f:\n", " r = requests.post(f\"{BASE}/watermark/detect\",\n", " files={\"file\": (os.path.basename(path), f, \"audio/wav\")},\n", " timeout=300)\n", " if r.status_code != 200:\n", " api_fail(r, \"POST /watermark/detect\")\n", " return r.json()\n", "\n", "generated = ensure_wav(\"tts_en.wav\", EN_TEXT)\n", "res = detect(generated)\n", "print(f\"\\nVoiceStudio-generated clip: watermarked={res.get('is_watermarked')} \"\n", " f\"confidence={res.get('confidence')}\")\n", "\n", "plain = os.path.join(DEMO_DIR, \"plain_tone.wav\")\n", "if not os.path.exists(plain):\n", " subprocess.run([\"ffmpeg\", \"-hide_banner\", \"-loglevel\", \"error\", \"-y\",\n", " \"-f\", \"lavfi\", \"-i\", \"sine=frequency=440:duration=3\", plain],\n", " check=True)\n", "res = detect(plain)\n", "print(f\"Plain (non-AI) tone: watermarked={res.get('is_watermarked')} \"\n", " f\"confidence={res.get('confidence')}\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 15. OpenAI-compatible API\n", "\n", "Anything that speaks OpenAI's audio API works against `http://127.0.0.1:3900/v1` unchanged — here the official `openai` Python client does TTS (`/v1/audio/speech`) and STT (`/v1/audio/transcriptions`) against the local engine. No key is checked; `voice` also accepts your profile ids from cell 12. Expected runtime: seconds (warm models).\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── 15. OpenAI-compatible API with the official openai client ───────────────\n", "import subprocess\n", "import sys\n", "\n", "try:\n", " ensure_wav\n", "except NameError:\n", " raise SystemExit(\"Run cell 8 (feature-tour helpers) first.\")\n", "\n", "check_backend()\n", "subprocess.run([sys.executable, \"-m\", \"pip\", \"install\", \"-q\", \"openai\"], check=True)\n", "from openai import OpenAI\n", "\n", "client = OpenAI(base_url=f\"{BASE}/v1\", api_key=\"none\") # any string — nothing checks it\n", "\n", "speech = client.audio.speech.create(\n", " model=\"tts-1\", voice=\"alloy\", response_format=\"wav\",\n", " input=\"This request thinks it is talking to OpenAI. It is not.\",\n", ")\n", "oa_path = os.path.join(DEMO_DIR, \"openai_compat.wav\")\n", "with open(oa_path, \"wb\") as f:\n", " f.write(speech.read())\n", "play(oa_path, \"POST /v1/audio/speech ->\")\n", "\n", "with open(oa_path, \"rb\") as f:\n", " tr = client.audio.transcriptions.create(model=\"whisper-1\", file=f)\n", "print(\"POST /v1/audio/transcriptions ->\", tr.text.strip())\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 16. Multi-voice story\n", "\n", "Two saved profiles perform a four-line scene through `POST /longform/render` — the Stories Editor's backend: per-span voices, inter-line pauses, and a single mixed MP3 out, with SSE progress along the way. Requires the profiles from cell 12. Expected runtime: ~1-2 min on a warm T4 (four TTS spans + encode).\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── 16. Multi-voice story via /longform/render ──────────────────────────────\n", "import json as _json\n", "\n", "try:\n", " NARRATOR_ID, GUEST_ID\n", "except NameError:\n", " raise SystemExit(\"Run cell 12 (voice profiles) first — this cell reuses its two profiles.\")\n", "\n", "payload = {\n", " \"default_voice\": NARRATOR_ID,\n", " \"format\": \"mp3\",\n", " \"chapters\": [{\n", " \"title\": \"A very short scene\",\n", " \"spans\": [\n", " {\"voice_id\": NARRATOR_ID, \"text\": \"Have you noticed we are running inside a notebook?\", \"pause_ms_after\": 300},\n", " {\"voice_id\": GUEST_ID, \"text\": \"I have. And nobody uploaded a single voice actor.\", \"pause_ms_after\": 300},\n", " {\"voice_id\": NARRATOR_ID, \"text\": \"Two profiles, four lines, one rendered story.\", \"pause_ms_after\": 300},\n", " {\"voice_id\": GUEST_ID, \"text\": \"Roll the credits.\", \"pause_ms_after\": 0},\n", " ],\n", " }],\n", "}\n", "\n", "check_backend()\n", "out_name = None\n", "with requests.post(f\"{BASE}/longform/render\", json=payload,\n", " stream=True, timeout=(10, 1800)) as r:\n", " if r.status_code != 200:\n", " api_fail(r, \"POST /longform/render\")\n", " for line in r.iter_lines(decode_unicode=True):\n", " if not line or not line.startswith(\"data:\"):\n", " continue\n", " evt = _json.loads(line[len(\"data:\"):].strip())\n", " etype = evt.get(\"type\")\n", " if etype == \"error\":\n", " raise SystemExit(f\"Story render failed: {evt}\\n{LOG_HINT}\")\n", " if etype == \"done\":\n", " out_name = evt[\"output\"]\n", " print(f\"Done: {evt.get('chapters')} chapter(s), {evt.get('duration_s')}s of audio\")\n", " else:\n", " print(f\" [{etype}] {evt.get('title') or evt.get('text', '')!s:.60}\")\n", "\n", "if not out_name:\n", " raise SystemExit(f\"Stream ended without a 'done' event. {LOG_HINT}\")\n", "story_path = os.path.join(\"/content/omnivoice_data/outputs\", out_name)\n", "play(story_path, f\"Rendered story ({story_path}):\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 17. Audiobook (chaptered m4b)\n", "\n", "`POST /audiobook` takes a plain-text script — `# Heading` starts a chapter, `[pause 400ms]` inserts silence — and renders a chapterized, tagged `.m4b` with SSE progress (the same renderer handles resume after interruptions). Narrated by the cell-12 profile. Expected runtime: ~1-2 min on a warm T4 for this two-chapter miniature.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── 17. Audiobook: script -> chaptered m4b ──────────────────────────────────\n", "import base64\n", "import json as _json\n", "\n", "from IPython.display import HTML\n", "\n", "try:\n", " NARRATOR_ID\n", "except NameError:\n", " raise SystemExit(\"Run cell 12 (voice profiles) first — this cell narrates with its profile.\")\n", "\n", "SCRIPT = \"\"\"# Chapter One: The Setup\n", "It began, as most experiments do, with a borrowed graphics card. [pause 400ms] Nothing was installed, and nothing was certain.\n", "\n", "# Chapter Two: The Payoff\n", "Two chapters later, the machine could read aloud. [pause 400ms] The end.\n", "\"\"\"\n", "\n", "payload = {\n", " \"text\": SCRIPT,\n", " \"default_voice\": NARRATOR_ID,\n", " \"format\": \"m4b\",\n", " \"metadata\": {\"title\": \"The Colab Miniature\", \"author\": \"VoiceStudio\",\n", " \"narrator\": \"Colab Narrator\"},\n", "}\n", "\n", "check_backend()\n", "done = None\n", "with requests.post(f\"{BASE}/audiobook\", json=payload,\n", " stream=True, timeout=(10, 1800)) as r:\n", " if r.status_code != 200:\n", " api_fail(r, \"POST /audiobook\")\n", " for line in r.iter_lines(decode_unicode=True):\n", " if not line or not line.startswith(\"data:\"):\n", " continue\n", " evt = _json.loads(line[len(\"data:\"):].strip())\n", " if evt.get(\"type\") == \"error\":\n", " raise SystemExit(f\"Audiobook render failed: {evt}\\n{LOG_HINT}\")\n", " if evt.get(\"type\") == \"done\":\n", " done = evt\n", " else:\n", " print(f\" [{evt.get('type')}] {evt.get('title') or ''}\")\n", "\n", "if not done:\n", " raise SystemExit(f\"Stream ended without a 'done' event. {LOG_HINT}\")\n", "book_path = os.path.join(\"/content/omnivoice_data/outputs\", done[\"output\"])\n", "print(f\"Done: {done.get('chapters')} chapters, {done.get('duration_s')}s -> {book_path}\")\n", "\n", "# Inline playback: IPython's Audio widget doesn't know .m4b, so embed the\n", "# (small) file as an HTML5 audio tag — Chrome plays AAC-in-MP4 natively.\n", "b64 = base64.b64encode(open(book_path, \"rb\").read()).decode(\"ascii\")\n", "display(HTML(f''))\n", "print(\"Chapter markers show up in audiobook players (Apple Books, etc.) — \"\n", " \"download the file from the Colab file browser to try it.\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 18. Video dubbing — heavy, optional\n", "\n", "The flagship pipeline, kept honest and miniature: a 6-second synthetic clip (color frame + the cell-9 English narration) is dubbed into Spanish — upload → prep (audio extract + Demucs vocal separation) → transcribe → translate → voice-cloned TTS → mux — all through the same job API the app uses.\n", "\n", "**Run this cell only if you have 5-15 minutes**: the first run downloads the Demucs separation model and (if cell 13 didn't run) an ASR model. Translation here uses the free Google web endpoint via `deep-translator` (installed in-cell); for a fully offline dub the backend also supports `provider=\"nllb\"` (a ~2.5 GB one-time model download).\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── 18. Video dubbing (mini): English clip -> Spanish dub ───────────────────\n", "import subprocess\n", "import sys\n", "import time\n", "\n", "try:\n", " ensure_wav\n", "except NameError:\n", " raise SystemExit(\"Run cell 8 (feature-tour helpers) first.\")\n", "\n", "from IPython.display import Video\n", "\n", "check_backend()\n", "TARGET_LANG, TARGET_LABEL = \"es\", \"Spanish\"\n", "\n", "# deep-translator backs the default \"google\" translation provider (free web\n", "# endpoint, no key). Offline alternative: provider=\"nllb\" below (~2.5 GB model).\n", "subprocess.run([sys.executable, \"-m\", \"pip\", \"install\", \"-q\", \"deep-translator\"], check=True)\n", "\n", "# 18a. Build a tiny source video: a colored frame carrying the EN narration.\n", "src_wav = ensure_wav(\"tts_en.wav\", EN_TEXT)\n", "src_mp4 = os.path.join(DEMO_DIR, \"dub_source.mp4\")\n", "if not os.path.exists(src_mp4):\n", " subprocess.run([\"ffmpeg\", \"-hide_banner\", \"-loglevel\", \"error\", \"-y\",\n", " \"-f\", \"lavfi\", \"-i\", \"color=c=steelblue:s=640x360:r=25\",\n", " \"-i\", src_wav, \"-shortest\",\n", " \"-c:v\", \"libx264\", \"-pix_fmt\", \"yuv420p\", \"-c:a\", \"aac\",\n", " src_mp4], check=True)\n", "print(f\"Source clip: {src_mp4} ({os.path.getsize(src_mp4)} bytes)\")\n", "\n", "def poll_task(task_id, what, timeout_s=1200):\n", " \"\"\"Poll the persisted job row until the background task finishes.\"\"\"\n", " deadline = time.time() + timeout_s\n", " while time.time() < deadline:\n", " r = requests.get(f\"{BASE}/jobs/{task_id}\", timeout=30)\n", " status = r.json().get(\"status\") if r.status_code == 200 else \"pending\"\n", " if status == \"done\":\n", " print(f\" {what}: done\")\n", " return\n", " if status in (\"failed\", \"cancelled\"):\n", " raise SystemExit(f\"{what} {status}: {r.text[:1500]}\\n{LOG_HINT}\")\n", " print(\".\", end=\"\", flush=True)\n", " time.sleep(5)\n", " raise SystemExit(f\"{what} timed out after {timeout_s}s. {LOG_HINT}\")\n", "\n", "# 18b. Upload -> prep task (audio extract + Demucs separation; model download\n", "# on first run).\n", "with open(src_mp4, \"rb\") as f:\n", " r = requests.post(f\"{BASE}/dub/upload\",\n", " files={\"video\": (\"dub_source.mp4\", f, \"video/mp4\")},\n", " data={\"input_type\": \"video\"}, timeout=300)\n", "if r.status_code != 202:\n", " api_fail(r, \"POST /dub/upload\")\n", "job_id, prep_task = r.json()[\"job_id\"], r.json()[\"task_id\"]\n", "print(f\"Dub job {job_id} — preparing (Demucs separation; first run downloads its model)\")\n", "poll_task(prep_task, \"prep\")\n", "\n", "# 18c. Transcribe the source audio (synchronous endpoint).\n", "print(\"Transcribing source audio...\")\n", "r = requests.post(f\"{BASE}/dub/transcribe/{job_id}\", timeout=1800)\n", "if r.status_code != 200:\n", " api_fail(r, \"POST /dub/transcribe\")\n", "segments = r.json()[\"segments\"]\n", "for s in segments:\n", " print(f\" [{s['start']:.2f}-{s['end']:.2f}s] {s['text']}\")\n", "\n", "# 18d. Translate the segments.\n", "r = requests.post(f\"{BASE}/dub/translate\", json={\n", " \"job_id\": job_id,\n", " \"target_lang\": TARGET_LANG,\n", " \"provider\": \"google\", # or \"nllb\" for the fully offline translator\n", " \"segments\": [{\"id\": str(s[\"id\"]), \"text\": s[\"text\"],\n", " \"start\": s[\"start\"], \"end\": s[\"end\"]} for s in segments],\n", "}, timeout=600)\n", "if r.status_code != 200:\n", " api_fail(r, \"POST /dub/translate\")\n", "translated = {t[\"id\"]: t[\"text\"] for t in r.json()[\"translated\"]}\n", "for sid, text in translated.items():\n", " print(f\" {TARGET_LANG} #{sid}: {text}\")\n", "\n", "# 18e. Generate the dubbed track — TTS voice-cloned from the source speaker.\n", "r = requests.post(f\"{BASE}/dub/generate/{job_id}\", json={\n", " \"language\": TARGET_LABEL,\n", " \"language_code\": TARGET_LANG,\n", " \"segments\": [{\"start\": s[\"start\"], \"end\": s[\"end\"],\n", " \"text\": translated.get(str(s[\"id\"]), s[\"text\"])}\n", " for s in segments],\n", "}, timeout=120)\n", "if r.status_code != 200:\n", " api_fail(r, \"POST /dub/generate\")\n", "print(f\"Rendering dubbed track (task {r.json()['task_id']})\")\n", "poll_task(r.json()[\"task_id\"], \"dub generate\")\n", "\n", "# 18f. Mux and fetch the dubbed video.\n", "r = requests.get(f\"{BASE}/dub/download/{job_id}\",\n", " params={\"include_tracks\": TARGET_LANG, \"default_track\": TARGET_LANG},\n", " timeout=1800)\n", "if r.status_code != 200:\n", " api_fail(r, \"GET /dub/download\")\n", "dub_mp4 = os.path.join(DEMO_DIR, f\"dubbed_{TARGET_LANG}.mp4\")\n", "with open(dub_mp4, \"wb\") as f:\n", " f.write(r.content)\n", "print(f\"Dubbed video: {dub_mp4} ({os.path.getsize(dub_mp4)} bytes)\")\n", "display(Video(dub_mp4, embed=True, width=480))\n", "DUB_JOB_ID = job_id # cell 19 (vocal isolation) reuses this job\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 19. Vocal isolation (stems)\n", "\n", "Dub prep already ran Demucs source separation, so the stems exist as job artifacts: `GET /dub/export-stems/{job_id}` returns the dubbed vocals and the original background as a zip, played here separately. Requires cell 18's job. (There is no job-free standalone isolation endpoint — separation rides the dub pipeline, so this cell does too.) Expected runtime: seconds.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── 19. Vocal isolation: fetch and play the separated stems ─────────────────\n", "import io\n", "import zipfile\n", "\n", "try:\n", " DUB_JOB_ID\n", "except NameError:\n", " raise SystemExit(\"Run cell 18 (video dubbing) first — the stems are artifacts of its job.\")\n", "\n", "r = requests.get(f\"{BASE}/dub/export-stems/{DUB_JOB_ID}\", timeout=300)\n", "if r.status_code != 200:\n", " api_fail(r, \"GET /dub/export-stems\")\n", "\n", "stems_dir = os.path.join(DEMO_DIR, \"stems\")\n", "os.makedirs(stems_dir, exist_ok=True)\n", "with zipfile.ZipFile(io.BytesIO(r.content)) as zf:\n", " names = zf.namelist()\n", " zf.extractall(stems_dir)\n", "print(\"Stems:\", names)\n", "for name in names:\n", " play(os.path.join(stems_dir, name), name)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 20. Everything you just generated\n", "\n", "A closing inventory: every artifact the tour produced, with paths and sizes. Remember Colab storage is ephemeral — download anything you want to keep (file browser in the left sidebar), or mount Drive as shown in the troubleshooting section.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── 20. Tour summary: artifacts on disk ─────────────────────────────────────\n", "import os\n", "\n", "def _ls(root, label):\n", " print(f\"\\n{label} ({root})\")\n", " if not os.path.isdir(root):\n", " print(\" (nothing here — the cells that write to it were not run)\")\n", " return\n", " for dirpath, _dirs, files in sorted(os.walk(root)):\n", " for name in sorted(files):\n", " p = os.path.join(dirpath, name)\n", " print(f\" {os.path.getsize(p):>10,} B {os.path.relpath(p, root)}\")\n", "\n", "_ls(\"/content/omnivoice_demos\", \"Feature-tour artifacts\")\n", "_ls(\"/content/omnivoice_data/outputs\", \"Backend outputs (history, stories, audiobooks)\")\n", "\n", "try:\n", " r = requests.get(f\"{BASE}/profiles\", timeout=15)\n", " if r.status_code == 200:\n", " print(f\"\\nSaved voice profiles: \"\n", " f\"{', '.join(p['name'] for p in r.json()) or '(none)'}\")\n", "except Exception:\n", " pass # backend may already be stopped — the file listing above still stands\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Part 3 — Troubleshooting\n", "\n", "**\"device\": \"cpu\" in /health, or generation is crawling** — the runtime has no GPU. `Runtime → Change runtime type → T4 GPU`, then `Runtime → Run all`. Colab's free tier sometimes has no GPUs available; trying again later usually works.\n", "\n", "**First generation takes minutes** — that's the default model (a few GB) downloading and loading. Run cell 4 to front-load the download; after the first generation the model stays warm and requests take seconds.\n", "\n", "**Session limits (free tier)** — Colab disconnects idle sessions and caps total runtime (up to ~12 h, often less). When the VM is recycled, **everything under `/content` is deleted**: the repo, models, and your `omnivoice_data` (voices, projects, generated audio). To keep your data across sessions, mount Google Drive and point the data dir at it *before* running cell 5:\n", "\n", "```python\n", "from google.colab import drive\n", "drive.mount('/content/drive')\n", "import os\n", "os.environ['OMNIVOICE_DATA_DIR'] = '/content/drive/MyDrive/omnivoice_data'\n", "```\n", "\n", "**VRAM** — a T4 has 16 GB: plenty for TTS, cloning, and voice design. Full dubbing pipelines (separation + transcription + diarization + TTS) run closer to the limit; the app evicts idle engines automatically, but very long videos may need shorter chunks.\n", "\n", "**UI tab is blank or errors out** — re-run cell 5 (it verifies `/health` without restarting a healthy backend), then cell 6. Pop-ups must be allowed for `colab.research.google.com`.\n", "\n", "**Feature-tour cells fail with `NameError` / \"Run cell 8 first\"** — the tour's helper functions live in cell 8; run it once after every kernel restart.\n", "\n", "**Backend logs** — `/content/omnivoice_backend.log`. The launch, smoke-test, and feature cells print the relevant tail or point here on failure; attach the full file when reporting an issue.\n", "\n", "**Reporting issues** — this notebook is maintained in the main repo: [debpalash/VoiceStudio/issues](https://github.com/debpalash/VoiceStudio/issues).\n" ] } ] }