Small language models: where they earn a place in an agentic stack
A working note on sub-10B models in 2026 — what they do well, where they break, and the three places in my own products where one would change the economics.
Status: in progress. This is a research note, not a completed project. The landscape section is sourced; the experiments section is the plan, to be replaced with results as I run them.
Why I’m looking at this
Three of my products pay a large-model bill for work that is narrow, repetitive and format-constrained: the voice agent’s qualification turns in Studio 3Sixty5, the PII-stripping and classification steps in IQnition’s gateway, and the per-listing condition and feature extraction in Perch. None of those calls needs a frontier model’s breadth. All of them need low latency, predictable structure, and — for the children’s product — data that never leaves a boundary I control. That is the SLM pitch exactly, and I want to know how much of it holds up in 2026.
The argument
The clearest statement of the case is NVIDIA Research’s 2025 position paper, which defines an SLM as a model that runs on consumer hardware with latency low enough to serve one user’s agent, and treats most models under about 10B parameters as qualifying (Belcak et al., 2025). Their claim has three parts: SLMs are already strong enough for the sub-tasks agents actually issue; agent work is narrow, repetitive and non-conversational, which suits small specialised models better than general ones; and the economics are decisive — they estimate serving a 7B model at 10–30× lower cost in latency, energy and compute than a 70–175B model, with fine-tuning measured in GPU-hours rather than GPU-days. They also estimate that 40–70 % of the LLM calls in several open agent frameworks could be handled by SLMs, and lay out a conversion recipe: log the non-human-facing calls, cluster them into recurring tasks, pick a small model per task, fine-tune with LoRA or QLoRA, iterate.
The honest counterweight in the same paper is the list of barriers: sunk investment in centralised LLM serving, benchmarks that measure generalist ability rather than agent-task fitness, and simple attention. None are technical. That matches my experience — the hard part of adopting a small model is not the model.
The landscape, late 2026
The 2–4B class is where on-device agent work has settled. The models that come up repeatedly are Qwen3-4B, Gemma 4 E4B, Phi-4-mini (3.8B), SmolLM3-3B and Ministral-3B, with Qwen3.5 sub-1B variants at the very small end (BentoML). Several are multimodal, most support function calling, and context windows run from 32K to 256K.
For tool calling specifically, one 2026 comparison puts all three of Qwen3-4B, Gemma 4 E4B and Phi-4-mini on modern phones at 4-bit quantisation, at roughly 30–40 tokens/s on a recent iPhone and 12–19 tokens/s on mid-range Android; out of the box they score in the 80s on function-calling benchmarks, and after a small QLoRA fine-tune on a few hundred examples all three clear 95 % joint accuracy on a five-tool task (ertas.ai). The spread between models shrinks after fine-tuning, which supports the paper’s thesis: for a defined task, the fine-tune matters more than the base.
Where it breaks
The most useful source I found is a single-developer case study of shipping two on-device models in a mobile app (arXiv 2604.24636). Its conclusion is blunt: the most reliable on-device LLM feature is the one where the model does the least. The failure modes are the ones that matter in production, not on benchmarks — format violations in a large share of early calls, quality that degrades after a handful of generations in one session, character counting and language drift, 5–10 second model initialisation, and a hard memory floor (8 GB for the ~2.6B model) that forced a two-tier model strategy. The eight heuristics that came out of it — minimal output schema, stacked parsing fallbacks, failure reasons in retry prompts, session rotation, deterministic fallback for everything, creative work to the model and factual work to deterministic sources — read like the workflow-engineering section of my own media engine.
Two further limits from the landscape sources: small models carry little factual knowledge, so they need retrieval for anything that depends on facts rather than form; and the very small models are prone to reasoning loops without sampling guardrails.
What this means for my stack
Mapping the argument onto the three products:
| Product | Candidate call | Why an SLM fits | What it needs |
|---|---|---|---|
| IQnition gateway | PII stripping, role/consent classification, difficulty scoring | Narrow, format-constrained, and the strongest privacy case — children’s data never reaches a third party if the strip runs inside the boundary | A fine-tuned 1–4B model behind the same gateway interface; evals against the knowledge graph unchanged |
| Studio 3Sixty5 voice agent | Intent capture, slot extraction, CRM write formatting per pathway node | One job per node is already the design; each node is a small-model task | Latency budget per turn; deterministic fallback to the pathway’s default on parse failure |
| Perch | Listing feature and condition extraction, formula inputs | Repetitive per-listing extraction into a fixed schema; cost per listing dominates | Vision-capable small model (Gemma 4 / Ministral class); mesh with existing CV condition step |
The pattern across all three is the paper’s heterogeneous design: a small specialised model handles the repetitive call, and the large model is reserved for the open-ended turn — the seller proposal narrative, the study-plan explanation, the exception a pathway can’t resolve. The gateway and the orchestrator are already the right place to make that routing decision; this is a routing-table change, not an architecture change.
Experiments (planned)
- IQnition PII strip. Fine-tune a sub-4B model on de-identification for the gateway’s input schema; measure recall on identifiers against the current approach, and latency inside the gateway.
- Pathway node extraction. Take one voice pathway, log the non-human-facing turns, and run the paper’s conversion recipe end to end — cluster, select, QLoRA, evaluate against the current model on joint accuracy and turn latency.
- Perch extraction cost. Replace the per-listing extraction call with a vision-capable small model and measure cost per listing and schema-compliance rate.
Results, when I have them, replace this section.
Sources
- Small Language Models are the Future of Agentic AI — Belcak et al., NVIDIA Research, 2025
- The Best Open-Source Small Language Models in 2026 — BentoML
- On-device tool calling in 2026: Qwen3, Gemma 4, Phi-4 — ertas.ai
- Less Is More: Engineering Challenges of On-Device SLM Integration in a Mobile Application — arXiv 2604.24636