SOLV.AI: CUTTING VOICE AI INFERENCE COST BY 99.9% WITH ONNX ENSEMBLES
The brief: build an AI voice complaint management system for an FMCG brand that categorises, analyses, and responds to customer calls automatically. The first prototype used GPT-3.5 end-to-end and cost $1,500 per million tokens. The shipped system costs $1.83. Here's how.
Cost reduction
Latency reduction
Category accuracy
vs $1,500 GPT-3.5
THE ORIGINAL PIPELINE (AND ITS PROBLEM)
GPT-3.5-turbo is excellent at NLU but catastrophically expensive when every customer call goes through it. FMCG complaint volumes are high (thousands of calls daily), meaning the cost model doesn't survive contact with production traffic.
THE ONNX ENSEMBLE APPROACH
WHY ONNX + DISTILBERT INSTEAD OF GPT?
Intent classification is a solved problem for narrow domains. DistilBERT-MNLI in ONNX INT8 achieves 100% accuracy on the FMCG complaint taxonomy (product quality, delivery, billing, returns) at 12ms latency. GPT-3.5 adds latency, cost, and non-determinism to a task that doesn't require it.
The 6-state FSM ensures predictable state transitions. Agentic loops with LLMs are powerful but unpredictable. For a voice interaction where every second counts, a deterministic FSM with LLM generation only at the response layer gives you the best of both worlds.
THE DUAL-FALLBACK ARCHITECTURE
Every critical path has a fallback. If Groq's API is unavailable, the system routes to a locally-running Ollama instance (Llama-3.1-8B). If ElevenLabs TTS is down, pyttsx3 handles synthesis locally. Zero downtime in production.
Written by Neal Daftary, AI & ML Engineer, Nirma University, Ahmedabad