{ "cells": [ { "cell_type": "markdown", "id": "license-header", "metadata": {}, "source": [ "" ] }, { "cell_type": "markdown", "id": "intro", "metadata": {}, "source": [ "### Cosmos3 Generator inference with NVIDIA NIM\n", "\n", "This notebook runs the prebuilt `Cosmos3-Generator` NIM container and sends video-generation requests to its HTTP API. It mirrors the Reasoner NIM cookbook structure, but uses the Generator NIM API contract:\n", "\n", "1. Launch `nvcr.io/nim/nvidia/cosmos3-generator:1.0.0`.\n", "2. Wait for `GET /v1/health/ready`.\n", "3. Inspect `/v1/models`, `/v1/metadata`, and `/v1/version`.\n", "4. Send Text2Video and Image2Video requests to `POST /v1/infer`.\n", "5. Decode the JSON response field `b64_video` into `.mp4` files and preview them.\n", "\n", "The NIM container downloads model artifacts from NGC into a local cache on first launch. You need an `NGC_API_KEY` and Docker login to `nvcr.io` before starting the container.\n", "\n", "Useful links:\n", "- Container [page](https://catalog.ngc.nvidia.com/orgs/nim/nvidia/containers/cosmos3-generator/-) on NGC\n", "- [Documentation](https://docs.nvidia.com/nim/cosmos/latest/introduction.html) for the Cosmos NIMs, each page have a table specific to Cosmos3-Generator NIM" ] }, { "cell_type": "markdown", "id": "capabilities", "metadata": {}, "source": [ "## Capability scope\n", "\n", "`Cosmos3-Generator` NIM currently exposes only the two video-generation modes below. It infers the mode automatically from request fields; there is no explicit `mode` field.\n", "\n", "| Workflow | Request shape | Output | Supported here? |\n", "| --- | --- | --- | --- |\n", "| Text2Video | non-empty `prompt`, no `image` | JSON with `b64_video` | Yes |\n", "| Image2Video | `image` plus optional `prompt` | JSON with `b64_video` | Yes |\n", "| Text-to-image | — | image | No |\n", "| Video-to-video | — | video | No |\n", "| Sound/audio generation | — | video + audio | No |\n", "| Action modes | — | video/action | No |\n", "| Transfer controls | — | controlled video | No |\n", "\n", "The request schema used in this notebook is Cosmos3-specific: `prompt`, `negative_prompt`, `image`, `seed`, `guidance_scale`, `steps`, `resolution`, `num_output_frames`, and `fps`. Unknown request fields are rejected by the server." ] }, { "cell_type": "markdown", "id": "launch-intro", "metadata": {}, "source": [ "## 1. Launch the NIM container\n", "\n", "Set `NGC_API_KEY` before launching Jupyter, or set it in the notebook environment. Authenticate Docker once with:\n", "\n", "```bash\n", "echo \"$NGC_API_KEY\" | docker login nvcr.io --username '$oauthtoken' --password-stdin\n", "```\n", "\n", "The default launch below uses Nano (`NIM_MODEL_SIZE=nano`), FP8 (`NIM_PRECISION=fp8`), and the latency profile (`NIM_PERF_PROFILE=latency`). Set `NIM_MODEL_SIZE=super` before running the launch cell to serve the larger model." ] }, { "cell_type": "code", "execution_count": null, "id": "helpers", "metadata": {}, "outputs": [], "source": [ "from pathlib import Path\n", "import base64\n", "import json\n", "import mimetypes\n", "import os\n", "import subprocess\n", "import time\n", "\n", "import requests\n", "from IPython.display import Video, display\n", "\n", "\n", "def find_repo_root() -> Path:\n", " try:\n", " return Path(\n", " subprocess.check_output([\"git\", \"rev-parse\", \"--show-toplevel\"], text=True).strip()\n", " ).resolve()\n", " except Exception:\n", " return Path.cwd().resolve()\n", "\n", "\n", "COSMOS_ROOT = find_repo_root()\n", "AUDIOVISUAL_ROOT = COSMOS_ROOT / \"cookbooks\" / \"cosmos3\" / \"generator\" / \"audiovisual\"\n", "ASSETS_ROOT = AUDIOVISUAL_ROOT / \"assets\"\n", "OUTPUT_DIR = AUDIOVISUAL_ROOT / \"outputs\" / \"cosmos3_generator_nim\"\n", "OUTPUT_DIR.mkdir(parents=True, exist_ok=True)\n", "\n", "I2V_IMAGE = ASSETS_ROOT / \"images\" / \"image2video\" / \"humanoid_robot.jpg\"\n", "assert I2V_IMAGE.exists(), I2V_IMAGE\n", "\n", "NIM_PORT = int(os.environ.get(\"NIM_PORT\", \"8000\"))\n", "BASE_URL = f\"http://127.0.0.1:{NIM_PORT}\"\n", "\n", "\n", "def file_data_uri(path: Path) -> str:\n", " mime = mimetypes.guess_type(path.name)[0] or \"application/octet-stream\"\n", " encoded = base64.b64encode(path.read_bytes()).decode(\"ascii\")\n", " return f\"data:{mime};base64,{encoded}\"\n", "\n", "\n", "def write_b64_video(response_json: dict, path: Path) -> Path:\n", " data = response_json[\"b64_video\"]\n", " path.write_bytes(base64.b64decode(data))\n", " return path\n", "\n", "\n", "def post_infer(payload: dict, output_name: str) -> Path:\n", " response = requests.post(\n", " f\"{BASE_URL}/v1/infer\",\n", " json=payload,\n", " headers={\"Accept\": \"application/json\", \"Content-Type\": \"application/json\"},\n", " timeout=60 * 60,\n", " )\n", " response.raise_for_status()\n", " output_path = OUTPUT_DIR / output_name\n", " return write_b64_video(response.json(), output_path)\n", "\n", "\n", "print(\"cosmos root:\", COSMOS_ROOT)\n", "print(\"audiovisual root:\", AUDIOVISUAL_ROOT)\n", "print(\"i2v image:\", I2V_IMAGE)\n", "print(\"outputs:\", OUTPUT_DIR)\n", "print(\"NIM base URL:\", BASE_URL)\n", "print(\"I2V data URI prefix:\", file_data_uri(I2V_IMAGE)[:64] + \"...\")\n" ] }, { "cell_type": "code", "execution_count": null, "id": "launch-docker", "metadata": {}, "outputs": [], "source": [ "%%bash\n", "set -euo pipefail\n", "\n", ": \"${NGC_API_KEY:?Set NGC_API_KEY before launching the Cosmos3-Generator NIM}\"\n", "\n", "export CONTAINER_NAME=\"${CONTAINER_NAME:-nvidia-cosmos3-generator}\"\n", "export IMG_NAME=\"${IMG_NAME:-nvcr.io/nim/nvidia/cosmos3-generator:1.0.0}\"\n", "export NIM_MODEL_SIZE=\"${NIM_MODEL_SIZE:-nano}\"\n", "export NIM_PRECISION=\"${NIM_PRECISION:-fp8}\"\n", "export NIM_PERF_PROFILE=\"${NIM_PERF_PROFILE:-latency}\"\n", "export LOCAL_NIM_CACHE=\"${LOCAL_NIM_CACHE:-$HOME/.cache/nim}\"\n", "export NIM_PORT=\"${NIM_PORT:-8000}\"\n", "mkdir -p \"$LOCAL_NIM_CACHE\"\n", "chmod -R 777 \"$LOCAL_NIM_CACHE\" 2>/dev/null || true\n", "\n", "# The container name must be free. If a previous run is still up, stop it first:\n", "# docker stop nvidia-cosmos3-generator\n", "docker run -d --rm --name=\"$CONTAINER_NAME\" \\\n", " --runtime=nvidia \\\n", " --gpus all \\\n", " --shm-size=32GB \\\n", " --ulimit nofile=65536:65536 \\\n", " -e NGC_API_KEY=\"$NGC_API_KEY\" \\\n", " -e NIM_MODEL_SIZE=\"$NIM_MODEL_SIZE\" \\\n", " -e NIM_PRECISION=\"$NIM_PRECISION\" \\\n", " -e NIM_PERF_PROFILE=\"$NIM_PERF_PROFILE\" \\\n", " -v \"$LOCAL_NIM_CACHE:/opt/nim/.cache\" \\\n", " -p \"${NIM_PORT}:8000\" \\\n", " \"$IMG_NAME\"\n", "\n", "echo \"NIM container '$CONTAINER_NAME' launching on port $NIM_PORT\"\n", "echo \"model_size=$NIM_MODEL_SIZE precision=$NIM_PRECISION profile=$NIM_PERF_PROFILE\"\n" ] }, { "cell_type": "markdown", "id": "wait-intro", "metadata": {}, "source": [ "## 2. Wait for readiness\n", "\n", "The first run downloads model artifacts and may spend additional time compiling/warming the engine. The server is ready for inference when `GET /v1/health/ready` returns HTTP 200." ] }, { "cell_type": "code", "execution_count": null, "id": "wait-ready", "metadata": {}, "outputs": [], "source": [ "%%bash\n", "set -euo pipefail\n", "\n", "PORT=\"${NIM_PORT:-8000}\"\n", "CONTAINER_NAME=\"${CONTAINER_NAME:-nvidia-cosmos3-generator}\"\n", "\n", "echo \"Waiting for Cosmos3-Generator NIM on port ${PORT}...\"\n", "\n", "for i in $(seq 1 3600); do\n", " if curl -fsS \"http://127.0.0.1:${PORT}/v1/health/ready\" >/dev/null 2>&1; then\n", " echo \"NIM server is ready.\"\n", " exit 0\n", " fi\n", " if ! docker ps --format '{{.Names}}' | grep -q \"^${CONTAINER_NAME}$\"; then\n", " echo \"Container '${CONTAINER_NAME}' is no longer running. Recent logs:\"\n", " docker logs --tail 160 \"$CONTAINER_NAME\" 2>&1 || true\n", " exit 1\n", " fi\n", " if (( i % 30 == 0 )); then\n", " echo \"Still waiting... recent logs:\"\n", " docker logs --tail 20 \"$CONTAINER_NAME\" 2>&1 || true\n", " fi\n", " sleep 2\n", "done\n", "\n", "echo \"Timed out waiting for NIM server. Recent logs:\"\n", "docker logs --tail 160 \"$CONTAINER_NAME\" 2>&1 || true\n", "exit 1\n" ] }, { "cell_type": "markdown", "id": "inspect-intro", "metadata": {}, "source": [ "## 3. Inspect the running service\n", "\n", "`Cosmos3-Generator` exposes standard NIM health/metadata endpoints plus `/v1/version` and a live OpenAPI schema at `/openapi.json`." ] }, { "cell_type": "code", "execution_count": null, "id": "inspect-service", "metadata": {}, "outputs": [], "source": [ "for endpoint in [\"/v1/models\", \"/v1/metadata\", \"/v1/version\"]:\n", " response = requests.get(f\"{BASE_URL}{endpoint}\", timeout=30)\n", " response.raise_for_status()\n", " print(f\"\\n### {endpoint}\")\n", " print(json.dumps(response.json(), indent=2)[:4000])\n" ] }, { "cell_type": "markdown", "id": "schema-intro", "metadata": {}, "source": [ "## 4. Request parameters\n", "\n", "Useful constraints for `POST /v1/infer`:\n", "\n", "| Field | Constraint / default |\n", "| --- | --- |\n", "| `guidance_scale` | float in `[1.0, 7.0]`, default `6.0` |\n", "| `steps` | int in `[1, 100]`, default `35` |\n", "| `resolution` | `256`, `480`, `720`, optionally suffixed with `_16_9`, `_1_1`, `_9_16`, `_4_3`, `_3_4`; bare tiers mean 16:9 |\n", "| `num_output_frames` | must follow the `4k+1` cadence: `25, 29, 33, ...`; per-tier caps are `256 <= 397`, `480 <= 297`, `720 <= 197` |\n", "| `fps` | float in `[1.0, 60.0]`, recommended `10` to `30`, default `24.0` |\n", "\n", "The examples use `resolution=\"256\"` and `num_output_frames=25` so a smoke test completes faster. Increase these values for quality once the workflow is verified." ] }, { "cell_type": "markdown", "id": "t2v-intro", "metadata": {}, "source": [ "## 5. Text2Video\n", "\n", "For Text2Video, provide a non-empty `prompt` and omit `image`. The response is JSON; decode `response[\"b64_video\"]` to get the MP4 bytes." ] }, { "cell_type": "code", "execution_count": null, "id": "t2v-request", "metadata": {}, "outputs": [], "source": [ "t2v_payload = {\n", " \"prompt\": \"A humanoid robot walks through a futuristic warehouse, inspecting shelves of mechanical components. Photorealistic, cinematic lighting.\",\n", " \"seed\": 42,\n", " \"guidance_scale\": 6.0,\n", " \"steps\": 35,\n", " \"resolution\": \"256\",\n", " \"num_output_frames\": 25,\n", " \"fps\": 24.0,\n", "}\n", "\n", "t2v_path = post_infer(t2v_payload, \"cosmos3_generator_nim_t2v.mp4\")\n", "print(\"wrote\", t2v_path, t2v_path.stat().st_size, \"bytes\")\n", "display(Video(str(t2v_path), embed=True, html_attributes=\"controls loop\"))\n" ] }, { "cell_type": "markdown", "id": "i2v-intro", "metadata": {}, "source": [ "## 6. Image2Video\n", "\n", "For Image2Video, provide an `image` field. The field accepts raw base64, a `data:image/...;base64,...` URI, or a public URL when URL inputs are enabled. This example uses a checked-in local image encoded as a data URI, so the NIM container does not need to see the host filesystem." ] }, { "cell_type": "code", "execution_count": null, "id": "i2v-request", "metadata": {}, "outputs": [], "source": [ "i2v_payload = {\n", " \"prompt\": \"The humanoid robot performs a controlled standing backflip in a modern living room, then lands steadily on both feet.\",\n", " \"image\": file_data_uri(I2V_IMAGE),\n", " \"seed\": 123,\n", " \"guidance_scale\": 6.0,\n", " \"steps\": 35,\n", " \"resolution\": \"256\",\n", " \"num_output_frames\": 25,\n", " \"fps\": 24.0,\n", "}\n", "\n", "i2v_path = post_infer(i2v_payload, \"cosmos3_generator_nim_i2v.mp4\")\n", "print(\"wrote\", i2v_path, i2v_path.stat().st_size, \"bytes\")\n", "display(Video(str(i2v_path), embed=True, html_attributes=\"controls loop\"))\n" ] }, { "cell_type": "markdown", "id": "playback-note", "metadata": {}, "source": [ "## Playback note\n", "\n", "The generated `b64_video` payload is an MP4 container. Some environments may not preview VP9-in-MP4 inline. If the notebook preview does not play, verify the written file with a player such as `ffplay`, `mpv`, or IINA, or re-encode externally to H.264." ] }, { "cell_type": "markdown", "id": "cleanup-intro", "metadata": {}, "source": [ "## 7. Optional cleanup\n", "\n", "Stop the detached container when you are done. The cached artifacts stay in `LOCAL_NIM_CACHE` for future launches." ] }, { "cell_type": "code", "execution_count": null, "id": "cleanup", "metadata": {}, "outputs": [], "source": [ "%%bash\n", "set -euo pipefail\n", "CONTAINER_NAME=\"${CONTAINER_NAME:-nvidia-cosmos3-generator}\"\n", "docker stop \"$CONTAINER_NAME\"\n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.13" } }, "nbformat": 4, "nbformat_minor": 5 }