Serve · Servex

Serving configs that meet the SLO on fewer GPUs

Servex searches TensorRT-LLM, Triton and NIM serving configurations against your real traffic shape, benchmarks p50, p95 and p99 on real GPUs, and packages the config that meets the SLO at the lowest cost.

Serving resultExample
Endpoint
chat-70b
SLO
p95 < 45 ms
Measured
p95 41.8 ms
GPUs
2 → 1 H100
Package
Triton + TensorRT-LLM
Example. Values are illustrative.

Capabilities

What Servex does

01

Config search

Batching, concurrency, KV cache, CUDA Graphs, scheduling and engine flags searched together.

02

Traffic replay

Your load shapes and sequence-length distributions drive every benchmark.

03

SLO-aware objective

Optimizes for p50, p95 or p99 at the lowest GPU count, not peak throughput alone.

04

One-step packaging

Ships a TensorRT-LLM engine, Triton repository or NIM container.

05

Continuous re-tuning

Re-runs when the model, GPU or traffic mix changes beyond a threshold.

06

Right-sizing advisor

Fleet-level GPU sizing recommendations.

Planned

Traffic-aware tuning

Tuned for your traffic, not a benchmark

The best batch size for a synthetic benchmark is rarely the best one for your real request mix. Servex replays your traffic shape while it searches.

Same engine, three serving configsExample
p95 latencyThroughputExample
Default config
p95 74 ms
1,100 tok/s
Benchmark-tuned
p95 50 ms
1,640 tok/s
Servex, traffic replay
p95 42 ms
1,912 tok/s
Example. Values are illustrative, not measured results.
Example serving configuration. Values are illustrative.
ParameterDefaultKept
Max batch size864
Schedulerstaticin-flight batching
KV cacheBF16, page 16FP8, page 32
CUDA Graphsoffon, 6 shapes
Instances per GPU12

How it works

Inside a Servex run

  1. 01Hour

    Model the traffic

    Fit request rate and sequence lengths from a sample.

  2. 02Hours

    Search configs

    Propose configs; Simyx prunes them.

  3. 03Hours

    Load test

    Replay traffic on real GPUs and measure tail latency.

  4. 04Hour

    Package

    Build the deployable with its recipe.

Specification

Inputs, outputs and hardware

ItemDetailStatus
InputEngine or model, traffic sample, SLOPilot
Serving stacksTensorRT-LLM, Triton Inference ServerPilot
NIM packagingNIM container outputNext
Metricsp50, p95, p99, throughput, GPU countPilot
TargetsH100, H200; Blackwell nextPilot · Planned
Right-sizing advisorFleet-level GPU sizingPlanned

FAQ

Questions engineers ask

Do you need production traffic?

A sample of request timings and sequence lengths is enough. No prompts or outputs are needed.

Will it change our serving stack?

No. Servex outputs standard TensorRT-LLM, Triton or NIM artifacts that drop into the stack you run.

How are tail latencies measured?

Under replayed load at the concurrency you expect, for long enough to stabilize p99.

What about multi-tenant GPUs?

You can constrain Servex to a share of a GPU; fairness across tenants is an open design question for the pilot.

Next step

Tune one production endpoint

Bring an endpoint and a traffic sample to the pilot. Servex returns a packaged config and a load-test report.