News · Infrastructure · News
Buoyant open-sources Kinference for Kubernetes and vLLM routing
Buoyant, the company behind Linkerd, has open-sourced Kinference. It is an Apache 2.0 inference control plane for Kubernetes and vLLM. The project sits in front of a fleet of vLLM workers and decides where each request goes. The first release is v0.1.0-alpha.7, written in Rust.
For teams serving many open-weight models to many internal clients, Kinference targets the layer above the model server. Instead of kube-proxy picking a random pod per connection, it applies declarative routing policies, KV-cache-aware load balancing and per-tenant metrics in one layer.
In one line: Buoyant has released Kinference, an Apache 2.0 control plane for Kubernetes and vLLM that uses declarative policies and KV-cache-aware routing to cut median time-to-first-token by about 95 percent on shared-prefix traffic, as reported in Buoyant's own benchmark.
What Kinference is
Kinference has two parts: a traffic gateway and an inference handler. The handler exposes a gRPC interface. It takes a request and returns the chosen endpoint plus a possibly mutated body. It pairs with a proxy such as the Buoyant Gateway. Kinference watches vLLM KV-cache events and engine metrics. It currently uses NVIDIA's dynamo-kv-router crate to process cache events.
Configuration lives in five Kubernetes CRDs: GatewayBackend, InferencePolicy, InferencePool, InferenceProvider and InferenceWorkerSet. It runs on any GPU-enabled Kubernetes cluster running vLLM. Buoyant's benchmark used three L40S GPUs per pool.
How declarative routing works
InferencePolicy is the CRD that does the routing. Rules can match on tenant, requested model, GPU pool state and semantic tags. Buoyant's examples cover five patterns.
- Simple to cheaper: route low-complexity prompts to a smaller model, unless cache overlap makes the large model faster.
- Burst to cloud: skip workers with over two queued prompts, then fall back to providers like Bedrock.
- PII stays local: remove external providers when a PII tag fires, and fail closed with onEmpty: Reject.
- Batch on idle GPUs: send batch requests only to workers under seventy percent KV-cache usage.
- Model aliasing: publish one stable name, such as standard-model, backed by several models.
Most rules end with minimize: PredictedTTFT. Kinference predicts time-to-first-token for each candidate endpoint using cache overlap and queue length. Policies live in git and can be validated offline with the kinfctl CLI. Multi-tenant controls include per-tenant metrics. Rate limits are planned, according to Buoyant's comparison table.
Benchmark results and caveats
Buoyant published a benchmark against plain Kubernetes load balancing. Each arm used three vLLM 0.31.0 pods, each on one L40S GPU. The model was Qwen3.8-27B-FP8. Load came from GuideLLM 0.8.1. The baseline was a ClusterIP Service, where kube-proxy picks a random pod per connection.
Median time-to-first-token results:
- Control workload with no shared prefix: 6.92 seconds with Kubernetes Service, 6.83 seconds with Kinference. No significant difference.
- Shared-prefix workload: 6.88 seconds down to 0.31 seconds. That is about a 95 percent drop.
- Multiturn workload: 1.70 seconds down to 0.59 seconds. That is about a 65 percent drop.
Median end-to-end latency fell 45 percent on shared-prefix and 32 percent on multiturn. On control traffic, p95 TTFT dropped from 21.46 seconds to 12.33 seconds. Buoyant attributes this to queue-aware balancing.
The capacity test measured goodput: requests per minute meeting a TTFT SLO. Peak goodput was 103.0 versus 14.0 requests per minute on shared-prefix, about 7.4 times higher. On multiturn it was 82.7 versus 23.8, about 3.5 times higher. The Service baseline never met the SLO at any tested concurrency.
These numbers are self-reported. The baseline is random routing, not another cache-aware router like llm-d or Dynamo. Buoyant also notes its SLOs of eight seconds and four seconds suit offline agents, not chat.
Kinference vs llm-d, Dynamo and vLLM Production Stack
Kinference's distinct angle is governance. Policies, aliases, provider fallback and tenant accounting are the main story. llm-d and Dynamo lead on raw serving performance, disaggregation and engine coverage. Kinference currently supports only vLLM. SGLang and TensorRT-LLM are planned. Disaggregated prefill and decode is not disclosed.
For a small team running one model family, Kinference may be enough. For a platform team mixing engines, llm-d or Dynamo is more mature. Kinference is at v0.1.0-alpha.7, llm-d is at v0.7 and Dynamo is at 1.x.
What I would try
At Latent we run RAG chatbots and agentic coding for client projects. I would not move a client workload onto Kinference today. It is early and the benchmark compares only against random routing.
I would watch two things. First, independent benchmarks against llm-d or Dynamo on shared-prefix RAG traffic. If the TTFT gain holds, it changes the cost and latency arithmetic for multi-tenant serving. Second, whether the InferencePolicy CRDs become stable enough to keep routing rules in the client repo, reviewed like code. This is the same control thinking I described in giving 19 developers AI access through one proxy. For RAG work, a PII stays local policy is close to the intent-first RAG design I use for client chatbots.
For now, nothing to do yet, just watch. I would add Kinference to the internal watchlist and test it on a non-production shared-prefix RAG benchmark.
Quick answers
What is Kinference?
Kinference is an open source inference control plane from Buoyant for Kubernetes and vLLM. It applies routing policies, KV-cache-aware load balancing and per-tenant metrics in one layer. It is Apache 2.0 licensed and currently at v0.1.0-alpha.7.
How much does Kinference reduce time-to-first-token?
Buoyant's own benchmark reports a median TTFT drop from 6.88 seconds to 0.31 seconds on a shared-prefix workload, about 95 percent. On control traffic without shared prefixes, there was no significant difference.
Does Kinference support engines other than vLLM?
Not yet. Kinference currently supports vLLM. Buoyant's comparison table lists SGLang and TensorRT-LLM as planned, but there is no release date in the source.