> ## Documentation Index
> Fetch the complete documentation index at: https://lmsysorg-cursor-cookbook-dsv4-amd-fp8-experts-env-d201.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Cosmos3

export const DiffusionModelTags = ({tags = []}) => {
  const normalizedTags = Array.isArray(tags) ? tags : [tags];
  return <div className="not-prose sgd-model-tags">
      {normalizedTags.map(tag => <span key={tag} className="sgd-chip">
          {tag}
        </span>)}
    </div>;
};

<DiffusionModelTags tags={["omnimodal", "image + video", "sound + action", "world model", "robot policy"]} />

## 1. Model Introduction

[NVIDIA Cosmos3](https://huggingface.co/collections/nvidia/cosmos3) is an omnimodal world-model family spanning text/image/video generation, optional synchronized sound, and robot action prediction. Its main advantage is breadth: the same native SGLang pipeline can serve media-generation checkpoints and the DROID policy checkpoint without routing through an LLM sampler.

Choose Nano for the broadest modality coverage and lower deployment cost, Super for the larger 64B image/video model, and a specialized checkpoint when only T2I or I2V is needed. Sound and action are checkpoint-specific heads, so they are not available from every Cosmos3 repository.

| Model                                    | Status    | Notes                                                        |
| ---------------------------------------- | --------- | ------------------------------------------------------------ |
| `nvidia/Cosmos3-Nano`                    | Supported | T2I, T2V, I2V, V2V, joint sound, and action                  |
| `nvidia/Cosmos3-Super`                   | Supported | T2I, T2V, I2V, and V2V; use multi-GPU for the 64B checkpoint |
| `nvidia/Cosmos3-Super-Text2Image`        | Supported | T2I-specialized checkpoint                                   |
| `nvidia/Cosmos3-Super-Image2Video`       | Supported | I2V-specialized checkpoint                                   |
| `nvidia/Cosmos3-Nano-Policy-DROID`       | Supported | DROID policy action generation                               |
| `nvidia/Cosmos3-Edge`                    | Supported | 4B dense model for T2I, T2V, I2V, V2V, and action generation |
| `nvidia/Cosmos3-Edge-Policy-DROID`       | Supported | 4B DROID policy action generation                            |
| `nvidia/Cosmos3-Super-Text2Image-4Step`  | Supported | 64B T2I checkpoint distilled to a fixed 4-step schedule      |
| `nvidia/Cosmos3-Super-Image2Video-4Step` | Supported | 64B I2V checkpoint distilled to a fixed 4-step schedule      |

Sound and action generation require the corresponding checkpoint heads. The pipeline reads the transformer and scheduler configs at startup, so Edge and distilled checkpoints do not require architecture-specific server flags. Non-distilled checkpoints use the flow-native `FlowUniPCMultistepScheduler`; distilled checkpoints use the fixed sigma schedule stored in the checkpoint.

The default `flow_shift` is `3.0` for T2I, `10.0` for non-Edge video and all action modes, and `3.0` for Edge video modes. Distilled checkpoints bake the schedule into their sigmas and do not use a request-level `flow_shift`.

## 2. Installation

Install SGLang with the diffusion dependencies:

```bash Command theme={null}
pip install -e "python[diffusion]"
```

Cosmos3 guardrails are enabled by default when the package is available:

```bash Command theme={null}
pip install "cosmos-guardrail==0.3.1"
```

`cosmos-guardrail` downloads gated NVIDIA guardrail weights, so pass a Hugging Face token if your environment needs one. If the package is not installed, SGLang skips Cosmos3 guardrails and logs a warning. To disable Cosmos3 guardrails for local experiments, set `SGLANG_DISABLE_COSMOS3_GUARDRAILS=1` before starting the server.

There may be problems loading the Cosmos-1.0-Guardrail weights on Ascend NPU. If the *\_pickle.UnpicklingError* error occurs during startup, you should change `weight_only=True` to `weights_only=False` parameter in *cosmos\_guardrail/cosmos\_utils.py*:

```
#!/usr/bin/env bash
COSMOS_GUARDRAIL_DIR="$(dirname "$(python -c 'import cosmos_guardrail; print(cosmos_guardrail.__file__)')")"
sed -i 's/weights_only=True/weights_only=False/g' "$COSMOS_GUARDRAIL_DIR/cosmos_utils.py"
```

## 3. Serve Cosmos3

Serve `Cosmos3-Nano` directly from the Hugging Face model ID:

```bash Command theme={null}
sglang serve \
  --model-path nvidia/Cosmos3-Nano \
  --num-gpus 1
```

With `--performance-mode auto`, a Cosmos3 Nano checkpoint keeps its DiT and
VAE resident when every selected GPU has at least 120 GiB available at
startup. Below that threshold, auto mode retains the conservative DiT
component-offload policy. This high-memory override is limited to Nano;
Cosmos3 Super checkpoints keep their existing multi-GPU placement defaults.

For `Cosmos3-Super`, split the model across multiple GPUs:

```bash Command theme={null}
sglang serve \
  --model-path nvidia/Cosmos3-Super \
  --num-gpus 4
```

The server also accepts the specialized `nvidia/Cosmos3-Super-Text2Image` and `nvidia/Cosmos3-Super-Image2Video` checkpoint IDs.

### Edge checkpoints

`Cosmos3-Edge` is a 4B dense model and can be served on one GPU:

```bash Command theme={null}
sglang serve \
  --model-path nvidia/Cosmos3-Edge \
  --num-gpus 1
```

Edge is trained for 256p and 480p generation. Its default video configuration is `832x480` with `guidance_scale=5.0`; its default image configuration is `640x640` with `guidance_scale=7.0`. Supported sizes are `832x480`, `480x832`, `640x480`, `480x640`, `480x480`, `640x640`, `448x256`, `256x448`, and `256x256`.

Serve the Edge DROID policy checkpoint with the same single-GPU configuration, replacing the model path with `nvidia/Cosmos3-Edge-Policy-DROID`.

### Distilled checkpoints

The distilled Super checkpoints are 64B models. Use multiple GPUs unless the complete model and request workload fit on one GPU:

```bash Command theme={null}
sglang serve \
  --model-path nvidia/Cosmos3-Super-Text2Image-4Step \
  --num-gpus 4
```

For distilled I2V, replace the model path with `nvidia/Cosmos3-Super-Image2Video-4Step`. SGLang detects both checkpoints from `scheduler/scheduler_config.json`, uses the checkpoint's fixed four-step sigma schedule, and forces `guidance_scale=1.0`. Do not tune `num_inference_steps` or `flow_shift` for these checkpoints.

## 4. OpenAI-Compatible Requests

### Text to image

Cosmos3 text-to-image uses `/v1/images/generations`. The default Cosmos3 image response is `b64_json`, matching vLLM-Omni's examples.

```bash Command theme={null}
curl -sS -X POST http://127.0.0.1:30010/v1/images/generations \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "A warehouse robot folds a blue cloth on a clean workbench.",
    "size": "1280x720",
    "n": 1,
    "num_inference_steps": 35,
    "guidance_scale": 6.0,
    "flow_shift": 3.0,
    "seed": 0,
    "extra_args": {
      "use_resolution_template": false,
      "guardrails": true
    }
  }'
```

With a server running `nvidia/Cosmos3-Super-Text2Image-4Step`, omit the scheduler controls and use `guidance_scale=1.0`:

```bash Command theme={null}
curl -sS -X POST http://127.0.0.1:30010/v1/images/generations \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "A warehouse robot folds a blue cloth on a clean workbench.",
    "size": "640x640",
    "n": 1,
    "guidance_scale": 1.0,
    "seed": 0,
    "extra_args": {
      "use_resolution_template": false,
      "guardrails": true
    }
  }'
```

### Text to video with sound

Use `/v1/videos` to create an asynchronous job, then poll the job and download the completed MP4. Set `generate_sound=true` to generate and mux a stereo 48 kHz audio track; omit it for a silent video.

```bash Command theme={null}
job_id=$(curl -sS -X POST http://127.0.0.1:30010/v1/videos \
  --form-string "prompt=A small warehouse robot moves a blue box across a clean floor." \
  --form-string "negative_prompt=blurry, distorted, low quality" \
  --form-string "size=1280x720" \
  --form-string "num_frames=81" \
  --form-string "fps=24" \
  --form-string "num_inference_steps=35" \
  --form-string "guidance_scale=4.0" \
  --form-string "flow_shift=10.0" \
  --form-string "generate_sound=true" \
  --form-string "seed=42" \
  --form-string 'extra_params={"guardrails":true,"use_resolution_template":false,"use_duration_template":false}' \
  | python -c 'import json, sys; print(json.load(sys.stdin)["id"])')

while true; do
  status=$(curl -sS "http://127.0.0.1:30010/v1/videos/${job_id}" \
    | python -c 'import json, sys; print(json.load(sys.stdin)["status"])')
  [ "$status" = "completed" ] && break
  [ "$status" = "failed" ] && exit 1
  sleep 1
done

curl -sS -L "http://127.0.0.1:30010/v1/videos/${job_id}/content" \
  -o cosmos3_t2v.mp4
```

### Image to video

This mirrors the official `nvidia/Cosmos3-Nano` Hugging Face image-to-video example:

```python Python theme={null}
import json
import time
from pathlib import Path

import requests
from huggingface_hub import snapshot_download

base_url = "http://127.0.0.1:30010"
model_dir = Path(snapshot_download("nvidia/Cosmos3-Nano"))
asset_dir = model_dir / "assets"

prompt = json.dumps(json.loads((asset_dir / "example_i2v_prompt.json").read_text()))
negative_prompt = json.dumps(
    json.loads((asset_dir / "negative_prompt.json").read_text())
)

data = {
    "prompt": prompt,
    "negative_prompt": negative_prompt,
    "size": "1280x720",
    "num_frames": "189",
    "fps": "24",
    "num_inference_steps": "35",
    "guidance_scale": "6.0",
    "max_sequence_length": "4096",
    "flow_shift": "10.0",
    "seed": "1111",
    "extra_params": json.dumps(
        {
            "use_resolution_template": False,
            "use_duration_template": False,
            "guardrails": True,
        }
    ),
}

with (asset_dir / "example_i2v_input.jpg").open("rb") as image:
    response = requests.post(
        f"{base_url}/v1/videos",
        data=data,
        files={"input_reference": ("example_i2v_input.jpg", image, "image/jpeg")},
        timeout=60,
    )
response.raise_for_status()
video_id = response.json()["id"]

while True:
    job = requests.get(f"{base_url}/v1/videos/{video_id}", timeout=30).json()
    if job["status"] == "completed":
        break
    if job["status"] == "failed":
        raise RuntimeError(job.get("error") or "Video generation failed")
    time.sleep(1)

response = requests.get(f"{base_url}/v1/videos/{video_id}/content", timeout=300)
response.raise_for_status()
Path("cosmos3_i2v.mp4").write_bytes(response.content)
```

For the distilled I2V checkpoint, use the same API with a server running `nvidia/Cosmos3-Super-Image2Video-4Step`. The recommended request is 480p and does not specify scheduler controls:

```bash Command theme={null}
job_id=$(curl -sS -X POST http://127.0.0.1:30010/v1/videos \
  --form-string "prompt=A warehouse robot carefully places a blue box on a shelf." \
  --form "input_reference=@first_frame.png;type=image/png" \
  --form-string "size=832x480" \
  --form-string "num_frames=189" \
  --form-string "fps=24" \
  --form-string "guidance_scale=1.0" \
  --form-string "seed=42" \
  --form-string 'extra_params={"guardrails":true,"use_resolution_template":false,"use_duration_template":false}' \
  | python -c 'import json, sys; print(json.load(sys.stdin)["id"])')
```

Poll and download this job with the same status and content endpoints used by the T2V example.

### Video to video

Upload a source video with `video_reference`. Cosmos3 keeps latent frames `[0, 1]` by default and generates the remaining frames. Use `condition_frame_indexes` to select different latent frames, and `condition_video_keep` to take conditioning frames from the start or end of the source.

```bash Command theme={null}
job_id=$(curl -sS -X POST http://127.0.0.1:30010/v1/videos \
  --form-string "prompt=A robotic arm pours liquid into a glass on a white tabletop." \
  --form "video_reference=@robot_pouring.mp4;type=video/mp4" \
  --form-string "size=1280x704" \
  --form-string "num_frames=45" \
  --form-string "fps=24" \
  --form-string "num_inference_steps=35" \
  --form-string "guidance_scale=6.0" \
  --form-string 'condition_frame_indexes=[0,1]' \
  --form-string "condition_video_keep=first" \
  | python -c 'import json, sys; print(json.load(sys.stdin)["id"])')
```

Poll and download this job with the same status and content endpoints used by the T2V example.

### Action generation

For DROID policy generation, start a single-GPU server with either the Nano or Edge policy checkpoint. Cosmos3 action generation does not currently support CFG or sequence parallelism.

```bash Command theme={null}
sglang serve \
  --model-path nvidia/Cosmos3-Nano-Policy-DROID \
  --num-gpus 1
```

Use `nvidia/Cosmos3-Edge-Policy-DROID` in the same command to serve the smaller 4B policy checkpoint.

`policy` and `inverse_dynamics` return actions, so their canonical API is the synchronous `/v1/actions/generations` endpoint. The following request predicts a 16-step action chunk from one observation image. `action_horizon=16` maps to the model's `num_frames=17` convention.

```python Python theme={null}
import base64
from pathlib import Path

import requests

image_b64 = base64.b64encode(Path("observation.png").read_bytes()).decode()
response = requests.post(
    "http://127.0.0.1:30010/v1/actions/generations",
    json={
        "input": {
            "task": "Put the pot to the left of the purple item.",
            "observation": {
                "image": {"b64_json": image_b64},
            },
        },
        "parameters": {
            "action_mode": "policy",
            "action_horizon": 16,
            "domain_name": "droid_lerobot",
            "height": 480,
            "width": 832,
            "fps": 5,
            "num_inference_steps": 30,
            "guidance_scale": 1.0,
            "seed": 42,
        },
    },
    timeout=300,
)
response.raise_for_status()
action = response.json()["data"][0]["action"]
print(action["shape"], action["values"])
```

Use `GET /v1/actions/metadata` to inspect the action modes, default horizon, padded action dimension, and accepted observation modalities. Msgpack requests and the `/v1/actions/realtime` websocket use the same action envelope.

`inverse_dynamics` also uses `/v1/actions/generations`; set `action_mode="inverse_dynamics"` and pass an observation video URL or server-local path as `input.observation.video`. Select the embodiment head with `domain_name` or `domain_id`; set `raw_action_dim` explicitly when it cannot be inferred from the domain name.

`forward_dynamics` is intentionally different: it consumes an action array and predicts video, so it remains on `/v1/videos`. Action-producing modes submitted to `/v1/videos` return HTTP 400 with the canonical action endpoint in the error message.

## 5. Cosmos3 Parameters

Cosmos3 supports the standard SGLang video and image fields such as `size`, `num_frames`, `fps`, `num_inference_steps`, `guidance_scale`, `negative_prompt`, and `seed`. For distilled checkpoints, SGLang replaces `num_inference_steps` with the checkpoint's fixed four-step schedule and forces `guidance_scale=1.0`; negative-prompt CFG and request-level `flow_shift` do not apply.

Top-level Cosmos3 request fields:

* `max_sequence_length`: maximum text token length used by the Cosmos3 tokenizer.
* `flow_shift`: per-request scheduler shift for non-distilled checkpoints. If omitted, SGLang uses `--flow-shift`, then the mode default (`3.0` for T2I, `10.0` for non-Edge video and all action modes, or `3.0` for Edge video).
* `guidance_interval`: optional `[start, end]` noise interval for CFG. Non-distilled T2I defaults to `[400, 1000]`; video modes guide at every step.

Cosmos3 omnimodal fields are accepted as extra JSON fields or multipart form fields:

* `generate_sound`: generate a sound track whose duration follows `num_frames / fps`.
* `sound_duration`: explicit sound duration in seconds; takes precedence over the derived duration.
* `condition_frame_indexes`: V2V latent-frame indexes to keep from the source video; defaults to `[0, 1]`.
* `condition_video_keep`: use the `first` or `last` source frames for V2V conditioning.
* `action_mode`: `policy`, `forward_dynamics`, or `inverse_dynamics`.
* `domain_name` / `domain_id`: select the action embodiment head.
* `raw_action_dim`: number of active action dimensions; inferred for known domain names.
* `action`: action array with shape `[T, D]`, required by `forward_dynamics`.
* `action_fps`: action-token frame rate for temporal mRoPE; defaults to the video FPS.
* `action_view_point`: viewpoint used in the structured action caption.
* `action_normalization`: dataset normalization mode, such as `quantile`, `meanstd`, or `minmax`.

Put model-specific compatibility knobs in `extra_params` for video requests, or `extra_args` for image requests:

* `use_duration_template`: whether to append SGLang's generated duration suffix to video prompts.
* `use_resolution_template`: accepted for vLLM-Omni request compatibility.
* `use_system_prompt`: whether to add the Cosmos3 system prompt to the chat template.
* `guardrails` or `use_guardrails`: per-request guardrail toggle when the server started with guardrails enabled.