Public leaderboards rank models on broad benchmarks that rarely resemble what a specific product actually needs the model to do. A model ranking highly on general reasoning benchmarks can underperform a lower-ranked model on your specific task — structured extraction from a particular document format, say, or a narrow classification job — because leaderboard rankings average across a wide spread of tasks, and your task may sit far from that average.
The only reliable comparison is evaluating candidates against your own eval set, built from real inputs with known-correct outputs, which is the same discipline covered in the piece on fine-tuning versus RAG versus prompting. Run the actual task through each candidate model, measure accuracy on your own definition of correct, and only then bring cost and latency into the comparison — evaluating cost before accuracy tells you nothing about whether the cheap model is actually usable for the job.
Open-source models deserve genuine consideration rather than automatic exclusion, particularly for narrow, well-defined tasks where a smaller model can be fine-tuned to match or exceed a larger general-purpose model's performance at meaningfully lower per-request cost. The trade is operational — you're now hosting and maintaining infrastructure rather than calling an API — and that trade is worth making for high-volume tasks and rarely worth it for low-volume, exploratory ones.
Latency and rate limits matter more in production than most evaluations account for upfront. A model that scores marginally higher on accuracy and takes three times as long to respond may be the wrong choice for a user-facing feature where response time affects whether the feature gets used at all. Weight the evaluation by what the product actually needs — sometimes that's maximum accuracy regardless of speed, sometimes it's the fastest model that clears an acceptable accuracy bar, and conflating the two produces a choice optimised for the wrong thing.
