01
Config search
Batching, concurrency, KV cache, CUDA Graphs, scheduling and engine flags searched together.
Serve · Servex
Servex searches TensorRT-LLM, Triton and NIM serving configurations against your real traffic shape, benchmarks p50, p95 and p99 on real GPUs, and packages the config that meets the SLO at the lowest cost.
Capabilities
01
Batching, concurrency, KV cache, CUDA Graphs, scheduling and engine flags searched together.
02
Your load shapes and sequence-length distributions drive every benchmark.
03
Optimizes for p50, p95 or p99 at the lowest GPU count, not peak throughput alone.
04
Ships a TensorRT-LLM engine, Triton repository or NIM container.
05
Re-runs when the model, GPU or traffic mix changes beyond a threshold.
06
Fleet-level GPU sizing recommendations.
Traffic-aware tuning
The best batch size for a synthetic benchmark is rarely the best one for your real request mix. Servex replays your traffic shape while it searches.
| Parameter | Default | Kept |
|---|---|---|
| Max batch size | 8 | 64 |
| Scheduler | static | in-flight batching |
| KV cache | BF16, page 16 | FP8, page 32 |
| CUDA Graphs | off | on, 6 shapes |
| Instances per GPU | 1 | 2 |
How it works
Fit request rate and sequence lengths from a sample.
Propose configs; Simyx prunes them.
Replay traffic on real GPUs and measure tail latency.
Build the deployable with its recipe.
Specification
| Item | Detail | Status |
|---|---|---|
| Input | Engine or model, traffic sample, SLO | Pilot |
| Serving stacks | TensorRT-LLM, Triton Inference Server | Pilot |
| NIM packaging | NIM container output | Next |
| Metrics | p50, p95, p99, throughput, GPU count | Pilot |
| Targets | H100, H200; Blackwell next | Pilot · Planned |
| Right-sizing advisor | Fleet-level GPU sizing | Planned |
Place in the loop
It combines Kernl's kernels and Quantra's precision into one engine and ships it.
FAQ
A sample of request timings and sequence lengths is enough. No prompts or outputs are needed.
No. Servex outputs standard TensorRT-LLM, Triton or NIM artifacts that drop into the stack you run.
Under replayed load at the concurrency you expect, for long enough to stabilize p99.
You can constrain Servex to a share of a GPU; fairness across tenants is an open design question for the pilot.
Next step
Bring an endpoint and a traffic sample to the pilot. Servex returns a packaged config and a load-test report.