A three-hour meeting to pick a model. Then they picked the one I suggested at the start
About one project that got delayed seven weeks because the team couldn't agree on Claude, ChatGPT, Gemini, or a local. And about how this usually gets cured.
The meeting was on a Tuesday. On Zoom — six of us: CTO, two backend engineers, product, security, and me. One question on the agenda: "which model do we pick for feature X."
Three hours later the meeting ended without a decision. The format chosen was "let's think until Friday." By Friday no one had thought, because there was a prod incident that week and nobody cared about models anymore.
Over the next six weeks we convened on this question four more times. Each time a new argument joined in — "new Sonnet just dropped," "Gemini now has a two-million context," "Llama 3.3 is now slightly better on a benchmark that, by the way, isn't relevant to us." Feature X went nowhere for those seven weeks.
In week seven I lost patience. I said: "Take Claude Sonnet, build a prototype in three days on your twenty real examples, and let's see." They did. The prototype worked. They took Claude Sonnet. Still running. Feature X shipped.
Those seven weeks cost the client roughly a million rubles in wasted time of four people. Plus the demoralisation — something I don't know how to count, but which I think was of similar order.
This is a typical story that repeats for me two or three times a year. Picking a model is a task it's very easy to drown in, because:
— new models ship every three to four months, — benchmarks flip, — every engineer has their personal opinion ("I tried Gemini, liked it"), — industry blogs publish "why GPT-4o beat Claude at X" (and, naturally, "why Claude beat GPT-4o at Y"), — nobody in the team wants to be the first to say "whatever, take this one," because it sounds unprofessional.
As a result, instead of working, the team discusses tools. And that would be fine if the discussion had a point. Usually it doesn't.
Why it doesn't.
The benchmarks you're debating with — MMLU, GPQA, HumanEval and the rest of the synthetics — measure how a model handles standard abstract tasks. That's probably irrelevant to you. You need to know whether this specific model handles your specific task at a reasonable cost. Which gets verified in half a day on your data, not seven weeks of argument.
When I do have to ground a decision — because a client wants a "reasoned choice," not "I said so" — I split models roughly like this.
Claude (Sonnet / Opus) — long contexts, tasks with nuanced instructions, refactors, careful text work. I use it for most of my real scenarios, because it is, first, disciplined (follows instructions), and second, rarely hallucinates on facts it doesn't know (it usually says so honestly). Downside — pricier per token. A mid-size project: $50–200/month at normal load.
ChatGPT (GPT-4o / o-series) — multimodal, real time, chat interfaces for end users. Massive plugin ecosystem. Downside — hallucinations on niche facts are more visible than Claude's. Few write about it, but I convinced myself of it several times this year, including once with reputational cost for a client who nearly sent an email citing invented research.
Gemini (2.x) — if the whole company lives in Google Workspace, this is the shortest path. Long contexts (in 2026, no longer unique). Strong on multimodal with charts. Downside — the ecosystem. Outside Google it feels like someone else's kitchen.
Local (Llama 3.x, Mistral, Qwen) — privacy, cost at scale, edge scenarios. If you hold data you cannot send to any cloud — decision closed, go local. Downside — you need an engineer, a server, a GPU. And the quality gap with flagships on hard tasks is still real in 2026.
I collapse the choice to three questions in this order:
First — privacy. Can't send? Local. Stop reading.
Second — volume. Over 100,000 requests per day? Local or cheapest API. Price it on real load.
Third — everything else — take Claude or ChatGPT, run 20 examples, take the winner. Three hours of work. Not three weeks.
One thing I say to clients a lot, as the external person:
Don't migrate between models every three months "because a newer one dropped." Stability costs more than 2% on a benchmark. Every migration is rewriting prompts, re-tuning, retesting. Work that adds zero value for the customer — they don't care whether you're on Sonnet 3.5 or Sonnet 4.
One client this year spent two weeks migrating from Sonnet to Opus because "Opus is now smarter." A month later we noticed that for their task Opus wasn't notably smarter, just three times more expensive. Migrated back. Another week. Minus three weeks of real work.
(Actually good that they came back. Worse would have been staying.)
A model is not a religion. It's a tool for a task. In 2026 tools are plentiful and roughly comparable; choice takes half a day on your data, not three hours of a six-person Zoom.
If your "let's pick a model" meeting lasts more than an hour, you're not debating models. You're debating the fear of choosing wrong. That's a different problem, and not one a benchmark comparison can solve.