Technology

Engineered for the bottlenecks that actually matter.

Every LLM workload has one primary bottleneck. FlowServe's optimizations are precision-built to hit each — memory bandwidth, time-to-first-token, and scheduling stability — without a redeployment or stack change.

// FlowServe architecture — one plane, three layers

Control

Unified API & console

OpenAI-compatible endpoints. One interface for training and inference. Sovereign — no cloud dependency.

Orchestration

Gang + memory-aware scheduler

Placement that prevents OOM. Adapter catalog with zero drift. Mixed train/inference on shared silicon.

Runtime

vLLM engine + optimizations

TensorRT-LLM & NVIDIA Dynamo, In-Flight Batching, PDL, Skip Softmax, TurboQuant KV-cache, DDTree — toggleable per-job.

// primary bottleneck: memory bandwidth

TurboQuant KV-cache

1–4 bit KV-cache quantization delivering up to 16× compression, runtime-toggleable without redeployment. It directly targets memory bandwidth — the dominant bottleneck for quantized inference — improving TTFT, TPOT, and throughput on memory-bound hardware without accuracy loss.

1–4 bitup to 16×runtime-toggleable

KV-cache footprint per 1B params

FP16 baseline1.00×

TurboQuant 4-bit~4×

TurboQuant 1-bitup to 16×

Lower is better. Compression is directional and workload-dependent; measured live during pilot.

// primary bottleneck: time-to-first-token

DDTree speculative decoding

A block-diffusion drafter replaces autoregressive drafting, generating candidate token trees the main model verifies in parallel — near-draft-model speed with full-model accuracy. A tree-depth slider (1–16) and optional draft model tune it per workload. Ideal for latency-sensitive chat where first-token speed is the user-facing metric.

parallel verifylower TTFTfull-model accuracy

// candidate token tree — accepted path in orange

Workload map

Where each optimization applies.

The primary bottleneck differs by workload. FlowServe maps them so you toggle the right optimization — no redeployment.

Metric	Inference (Quantized)	LoRA / QLoRA	Full-Parameter	DPO / RLHF
Primary bottleneck	Memory bandwidth	Compute & memory bw	Compute & inter-GPU net	Compute & memory cap.
Key KPIs	TTFT, TPOT, throughput	Train throughput, TTC	Train throughput, TTC	Reward acc., stability
VRAM / 1B params	<2GB	<4GB (<24GB @ 7B)	>16GB (>300GB @ 7B)	>20GB (multi-copy)
Compute intensity	Low–medium	Medium	Very high	Very high
Multi-node net sensitivity	Low	Low–medium	Very high	Very high

Runtime stack

Every optimization is a toggle, not a rebuild.

inference

TensorRT-LLM · NVIDIA Dynamo

Blackwell-optimized runtime — up to 10× lower inference cost for open-source models, toggleable per-job.

fine-tuning

SFT · ASFT · PEFT/LoRA

KL-anchored ASFT and LoRA with Flash Attention 2. Every adapter version auto-cataloged.

deployment

Bare metal · EKS · MAAS · vSphere

On-prem, cloud, or fully air-gapped — the same control plane, no egress.

Governance & observability

Production controls, not a demo.

Every endpoint is authenticated, every credential encrypted, every process accounted for — and fully observable while it runs.

Observability

vLLM metrics

tokens/sec, queue depth, KV-cache utilization — per-job.

Live log streaming

Real-time tailing of vLLM logs per node.

Test endpoint & chat drawer

One-click /v1/models probe before traffic; in-UI OpenAI-compatible streaming chat.

Security

Encrypted credentials

Fernet-backed credential store, disk-only keys.

JWT authentication

HTTPBearer guards every API endpoint.

Clean teardown & process scanning

SIGTERM → grace → SIGKILL; /proc scanned for stray vLLM processes. VRAM freed, zero zombies.

What's shipping next

The roadmap is operational depth.

01

Per-adapter metrics

Aggregation across fine-tuned catalog entries.

02

Auto-A/B testing

DDTree vs autoregressive by rolling acceptance rate.

03

Terraform-state sync

Reconcile registered nodes with live infrastructure.

04

Multi-node gang for ASFT

Large ASFT runs across multiple GPU nodes.

How we compare

FlowServe vs. general model-serving platforms.

Where a general orchestration platform (OICM+) offers breadth, FlowServe goes deep on the inference and fine-tuning runtime — with optimizations and VRAM discipline built for it.

Capability	FlowServe	General platforms (OICM+)
Inference stack	vLLM + TurboQuant KV-cache + DDTree speculative decoding	vLLM / TGI backends, standard deployment
Inference optimizations	NVIDIA toggles, LoRA hot-swap, auto max-model-len clamping	Standard vLLM/TGI params, per-request timeout
VRAM management	SIGTERM/SIGKILL process scan, zero zombie vLLMs	Kubernetes pod lifecycle, resource quotas
Fine-tuning	SFT (standard, ORPO, POFT) + ASFT (KL-anchored)	SFT, DPO, RLHF, LoRA, seq2seq via UI
Model catalog	Auto-catalog with lineage (base, dataset, LoRA rank)	Registered models + versions, bundles, benchmarks
Hardware	NVIDIA, AMD Instinct, Intel Gaudi 3	NVIDIA + AMD (ROCm)

Source: OICM+ Documentation v1.11.0 — oicm.docs.openinnovation.ai

See these numbers on your own hardware.

We run a benchmarked pilot on your models and silicon — air-gapped if required.

Request a paid pilot →