ONNXVoice AIAgentic AICost OptimisationLLM Engineering

SOLV.AI: CUTTING VOICE AI INFERENCE COST BY 99.9% WITH ONNX ENSEMBLES

The brief: build an AI voice complaint management system for an FMCG brand that categorises, analyses, and responds to customer calls automatically. The first prototype used GPT-3.5 end-to-end and cost $1,500 per million tokens. The shipped system costs $1.83. Here's how.

99.9%

Cost reduction

65%

Latency reduction

100%

Category accuracy

$1.83/M

vs $1,500 GPT-3.5

THE ORIGINAL PIPELINE (AND ITS PROBLEM)

GPT-3.5-turbo is excellent at NLU but catastrophically expensive when every customer call goes through it. FMCG complaint volumes are high (thousands of calls daily), meaning the cost model doesn't survive contact with production traffic.

THE ONNX ENSEMBLE APPROACH

Input audio (STT via Groq Whisper) ↓ Dual-model ONNX ensemble (12ms total): ├── DistilBERT-MNLI (zero-shot classification, INT8) └── MiniLM-L6-v2 (semantic similarity scoring) ↓ VADER sentiment analysis (rule-based, ~0ms) ↓ 6-state FSM agent orchestration ├── State: Greeting / Intent / Clarify / Resolve / Escalate / Close └── 5 specialised agents (one per resolution path) ↓ Groq LLM (primary) → Ollama local (fallback) ↓ Dual TTS: ElevenLabs (primary) → pyttsx3 (fallback)

WHY ONNX + DISTILBERT INSTEAD OF GPT?

Intent classification is a solved problem for narrow domains. DistilBERT-MNLI in ONNX INT8 achieves 100% accuracy on the FMCG complaint taxonomy (product quality, delivery, billing, returns) at 12ms latency. GPT-3.5 adds latency, cost, and non-determinism to a task that doesn't require it.

The 6-state FSM ensures predictable state transitions. Agentic loops with LLMs are powerful but unpredictable. For a voice interaction where every second counts, a deterministic FSM with LLM generation only at the response layer gives you the best of both worlds.

THE DUAL-FALLBACK ARCHITECTURE

Every critical path has a fallback. If Groq's API is unavailable, the system routes to a locally-running Ollama instance (Llama-3.1-8B). If ElevenLabs TTS is down, pyttsx3 handles synthesis locally. Zero downtime in production.

GitHub Profile →