ONNXVoice AIAgentic AICost OptimisationLLM Engineering

SOLV.AI: CUTTING VOICE AI INFERENCE COST BY 99.9% WITH ONNX ENSEMBLES

The brief: build an AI voice complaint management system for an FMCG brand that categorises, analyses, and responds to customer calls automatically. The first prototype used GPT-3.5 end-to-end and cost $1,500 per million tokens. The shipped system costs $1.83. Here's how.

99.9%

Cost reduction

65%

Latency reduction

100%

Category accuracy

$1.83/M

vs $1,500 GPT-3.5

THE ORIGINAL PIPELINE (AND ITS PROBLEM)

GPT-3.5-turbo is excellent at NLU but catastrophically expensive when every customer call goes through it. FMCG complaint volumes are high — thousands of calls daily — meaning the cost model doesn't survive contact with production traffic.

THE ONNX ENSEMBLE APPROACH

Input audio (STT via Groq Whisper) ↓ Dual-model ONNX ensemble (12ms total): ├── DistilBERT-MNLI (zero-shot classification, INT8) └── MiniLM-L6-v2 (semantic similarity scoring) ↓ VADER sentiment analysis (rule-based, ~0ms) ↓ 6-state FSM agent orchestration ├── State: Greeting / Intent / Clarify / Resolve / Escalate / Close └── 5 specialised agents (one per resolution path) ↓ Groq LLM (primary) → Ollama local (fallback) ↓ Dual TTS: ElevenLabs (primary) → pyttsx3 (fallback)

WHY ONNX + DISTILBERT INSTEAD OF GPT?

Intent classification is a solved problem for narrow domains. DistilBERT-MNLI in ONNX INT8 achieves 100% accuracy on the FMCG complaint taxonomy (product quality, delivery, billing, returns) at 12ms latency. GPT-3.5 adds latency, cost, and non-determinism to a task that doesn't require it.

The 6-state FSM ensures predictable state transitions. Agentic loops with LLMs are powerful but unpredictable — for a voice interaction where every second counts, a deterministic FSM with LLM generation only at the response layer gives you the best of both worlds.

THE DUAL-FALLBACK ARCHITECTURE

Every critical path has a fallback. If Groq's API is unavailable, the system routes to a locally-running Ollama instance (Llama-3.1-8B). If ElevenLabs TTS is down, pyttsx3 handles synthesis locally. Zero downtime in production.

GitHub Profile →