> ## Documentation Index
> Fetch the complete documentation index at: https://lmsysorg-cursor-cookbook-dsv4-amd-fp8-experts-env-d201.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Radix Cache Eviction Policies

When the KV cache pool is full, the radix cache reclaims space by evicting cached prefixes. The eviction policy decides which prefix goes first. The default, `lru`, is the right choice for most workloads; the others trade recency for hit frequency, reuse history, or request priority.

## How eviction picks a victim

Eviction only ever considers **evictable leaves**: nodes whose KV is present, unlocked (no in-flight request holds them), and not shadowed by a child that still holds KV. The root is never evictable.

The policy scores each candidate and the **lowest score is evicted first**. Once a leaf is evicted, its parent may become a leaf and re-enter the candidate set, so eviction walks a branch from its tip toward the root.

A policy only scores. It does not decide how much to free, and it cannot pin KV in memory — a node protected by the policy is still evicted if reclaiming everything else is not enough.

## Available policies

Select one with `--radix-eviction-policy`. All of them fall back to least-recently-used order within a tie.

| Policy          | Evicts first                                                                                                                      | Use when                                                                                                                                                                       |
| --------------- | --------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `lru` (default) | The prefix unused for longest.                                                                                                    | General serving. Matches how prefix reuse decays with time.                                                                                                                    |
| `lfu`           | The prefix with the fewest cache hits, then the least recent.                                                                     | A small set of prompts is reused far more than the rest, and you want those to survive bursts of one-off traffic.                                                              |
| `slru`          | Prefixes still in the probationary segment, then the least recent within a segment.                                               | Like `lfu`, but you want a hard floor on how much a one-off prefix can displace a proven one. See [`slru` parameters](#slru-parameters).                                       |
| `priority`      | The prefix belonging to the lowest-priority request, then the least recent.                                                       | You run [priority scheduling](/docs/advanced_features/server_arguments) and want cache retention to follow the same ranking as admission.                                      |
| `tlru`          | The tail of a conversation beyond what its next prefill needs to meet the TTFT budget ("TEL-safe" tokens), then the least recent. | Agentic / multi-turn workloads where tail TTFT matters more than raw hit rate. See [`tlru` parameters](#tlru-parameters). Requires the unified radix cache (the default tree). |

Notes on the scoring inputs:

* **Hit count** (`lfu`, `slru`) counts how many times a node was matched by a later request. It is not incremented for chunked-prefill steps, for evicted nodes, or under write-back HiCache.
* **Request priority** (`priority`) is the `priority` field of the request that inserted the prefix; a node reached by several requests keeps the highest priority among them. Without priority scheduling every node is priority `0` and this policy is equivalent to `lru`.

`fifo`, `mru`, and `filo` also exist in the policy registry but are not offered on the command line. They are reachable only by out-of-tree code that extends the choice list, and exist for experiments rather than serving.

## Tuning a policy

Some policies take parameters. Pass them as a json object to `--radix-eviction-policy-config`; the keys are the policy's own, so they are only valid for the policy you selected.

```bash Command theme={null}
python3 -m sglang.launch_server \
  --model-path MODEL_PATH \
  --radix-eviction-policy slru \
  --radix-eviction-policy-config '{"protected_threshold": 4}'
```

Omit the flag to accept every default. An unrecognized key fails at startup rather than being ignored:

```
TypeError: SLRUStrategy.__init__() got an unexpected keyword argument 'protected_treshold'
```

<Note>
  `--radix-eviction-policy-config` is not supported by the experimental Rust tree core (`SGLANG_UNIFIED_RADIX_TREE_CORE_BACKEND=rust`), which builds its strategy from the policy name alone. Passing both fails at startup.
</Note>

### Policy parameters

`slru` and `tlru` take parameters. `lru`, `lfu`, and `priority` take none, so `--radix-eviction-policy-config` has no effect with them and any key is an error.

#### `slru` parameters

`slru` splits the cache into a **probationary** segment and a **protected** segment. A prefix enters probationary, and is promoted to protected once it has been hit enough times. Everything probationary is evicted before anything protected.

| Key                   | Type | Default | Meaning                                                           |
| --------------------- | ---- | ------- | ----------------------------------------------------------------- |
| `protected_threshold` | int  | `2`     | Hit count at which a prefix is promoted to the protected segment. |

Raising it makes promotion harder, so the protected set stays small and closer to your genuinely hot prefixes; a prefix hit three times stays probationary at `4` but is protected at the default `2`. Lowering it to `1` promotes any prefix that is reused even once, which approaches `lru` with a one-hit grace period.

#### `tlru` parameters

`tlru` (Tail-Optimized LRU, [arXiv:2510.15152](https://arxiv.org/abs/2510.15152)) protects only the cached history a conversation's next prefill needs to stay under a TTFT budget; the rest of its tail is evicted first, and eviction continues in plain recency order once the TEL-safe tokens run out.

| Key                    | Type | Meaning                                                                                                                                                                                                                                                                               |
| ---------------------- | ---- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `threshold`            | int  | Tail-latency threshold ξ, in tokens: a conversation only keeps enough cache to hold its next prefill under ξ uncached tokens. Convert from a TTFT target by dividing it by the measured ms per uncached token. The paper states ξ in blocks, so multiply its values by `--page-size`. |
| `next_prompt_estimate` | int  | Estimated tokens the next turn of a conversation will add; use the trace's empirical mean.                                                                                                                                                                                            |

Both keys are required in practice: `threshold` must be greater than `next_prompt_estimate` (only the difference affects behaviour), and values at or above it reduce T-LRU to plain LRU, which startup rejects.

```bash Command theme={null}
python3 -m sglang.launch_server \
  --model-path MODEL_PATH \
  --radix-eviction-policy tlru \
  --radix-eviction-policy-config '{"threshold": 4096, "next_prompt_estimate": 512}'
```

## Which policy to pick

Start with `lru` and change it only against a measured cache hit rate — the counters are exposed under `--enable-metrics`. `lfu` and `slru` help when your traffic has a stable hot set that a recency-only policy keeps flushing; they hurt when prefix popularity shifts over time, because a prefix that earned a high hit count keeps its advantage after it stops being useful.