mathiasgrimm.com
[ ← back ]

blog

Newest isn't best: the model I picked is three generations old

Mathias Grimm··23 min read
Newest isn't best: the model I picked is three generations old

I'm building a pipeline that translates books from English to Brazilian Portuguese. The translation has to follow a 46 rule style guide and keep every inline formatting marker where it was.

Before committing to a model, I ran the first chapter of a real book (67 blocks, about 6,900 words) through 14 models, then reran the interesting ones with corrected settings and added Google's current Flash lineup, including gemini-3.8-flash on the day it launched. That was 30 runs on chapter 1. Then every configuration with a recorded setting again on chapter 2, 27 more.

In total: 57 runs, 18 models, four vendors, one day, and about $35 in API spend. I initially estimated it at around $50 and 60 runs, and that's the number I posted on Twitter, but $35 is what the runs actually add up to.

The setup

Every run got the same input: the chapter split into blocks, 12 blocks per request as JSON, and the model asked to return JSON segments keyed by block id. That's how my production pipeline works. Same system prompt, temperature 0 where the API allows it, and the vendor's own recommended reasoning setting where there is one. Every call went straight to the vendor's raw HTTP API, nothing of mine in the loop.

I scored two things:

  • Closeness: how much of the model's Portuguese matches the professional human translation of the same chapter. I strip the formatting codes, split both texts into words, and let Python's difflib find the longest runs of words the two have in common, the same way a diff tool finds the unchanged lines between two versions of a file. The score is the share of words that sit in the same sequence in both texts. A 68% means about two words in three are where the human put them.
  • Completeness: how many of the blocks came back, and whether every formatting marker survived.

One caveat before the findings. Closeness is a lexical proxy against one human's word choices. If a model writes "desmentir" where the human wrote "refutar", that counts as a miss, even though both are fine for "disprove". If it moves a clause to the front of the sentence, that is a run of misses. So a translation can score lower just because it said the same thing in different words, and gaps under a point are noise. The real quality test is a blind read by the human translator, and that's the next step. Still, the proxy catches the models that drop text, break the contract, or drift too far from the register.

Chapter 2, block by block Closeness of each of the 88 blocks to the human translation, one row per configuration, best chapter first. Darker is closer. The same measure as the table, recomputed per block. Dashed cells are blocks the run never returned. blocks in reading order chapter gemini-3.1-pro-preview medium 71.1 gemini-3.5-flash off 69.5 claude-fable-5-1 high 69.0 grok-4.3 none 69.0 claude-fable-5-1 low 68.8 gpt-5.6-luna low 68.7 claude-fable-5-1 medium 68.6 gpt-5.6-luna medium 68.6 claude-sonnet-5 high 68.5 gpt-4.1 none 68.5 gemini-3.6-flash minimal 68.3 gemini-3.5-flash-lite minimal 68.2 claude-opus-5 high 67.9 gpt-5.6-sol low 67.7 claude-sonnet-5 low 67.5 gpt-5.6-terra medium 67.5 gpt-5.6-terra low 67.3 grok-4.5 low 67.3 claude-opus-5 low 67.1 gpt-5.6-sol medium 66.8 grok-4.6 low 66.8 gemini-3.7-flash off 66.2 gpt-5 medium 65.7 gemini-3.5-flash medium 65.1 gemini-3.8-flash low 65.0 gemini-3.8-flash off 64.0 gpt-5-mini medium 57.9 median of the 27 runs 1 10 20 30 40 50 60 70 80 88 45% 90% closeness of the block not returned heading study questions Google OpenAI Anthropic xAI

Each row is one run, each column one of the 88 blocks of chapter 2, and darker means closer to the human translation. Where one model struggles, most of the others do too: the light stripes cross the whole grid. The lightest column, block 68, is a Bible verse the human quoted from a different Portuguese Bible than the models used. Dashed cells are blocks a run never returned. The rows look alike because the models are alike: the passage changes the score three times more than the model does. That's the argument of this post in one picture. Quality is close enough that cost and time can decide.

What I found

1. Completeness

Every model that returned all 67 blocks of chapter 1 landed in a tight band, 62% to 68% closeness. No model was dramatically closer to the human than another. What actually separated them was whether a model silently dropped whole blocks while returning perfectly valid JSON.

Two did, at every setting I tried. Google's gemini-3.5-flash returned 55 of 67 blocks at its default setting and 63 of 67 with thinking off. OpenAI's gpt-5.6-luna returned 57 of 67 at medium and 50 of 67 at low. A big part of their low scores is simply the missing text. In production my validator would catch the gaps and re-request them, but a model that needs a safety net for a quarter of its output, as gpt-5.6-luna at low did, is not a model I want to build on.

Chapter 2 made it clearer. With thinking off, gemini-3.5-flash returned all 88 blocks and scored 69.5%. At its default thinking level it lost a whole chunk again, 76 of 88: the reply hit the output limit after about 134,000 thinking tokens and the JSON was cut mid-string. gpt-5.6-luna dropped blocks on both chapters at both settings, only two on chapter 2, and with the text present it scored 68.7% for $0.02.

2. What Gemini recommended vs what I measured

I asked Gemini which of its own models I should use. It recommended the newest one, gemini-3.8-flash, and told me to stay away from the cheap gemini-3.5-flash-lite: too weak for a 46-rule style guide, it said, the repair loops would eat any savings.

When I actually measured it, it was more or less the opposite. On chapter 1, gemini-3.5-flash-lite tied for the top of the table with claude-fable-5-1 at effort low, both at 68.4%. gemini-3.5-flash-lite came back with all 67 blocks and every marker intact, in 41 seconds, for $0.04. The recommended gemini-3.8-flash returned every block too, but it scored near the bottom of the complete runs: 64.5% with thinking low and 63.9% with thinking off.

Taking gemini-3.5-flash-lite as the 3.5 entry, closeness went down with every newer Flash generation, on both chapters and on their average:

Gemini Flash: closeness by generation Average of the two chapters. Newest on the right. 3.8-flash at thinking low, the thinking-off run hollow. 60% 62% 64% 66% 68% 70% limit 61.7% gemini-3.5-flash-lite gemini-3.6-flash gemini-3.7-flash gemini-3.8-flash 68.3 67.9 66.2 64.8 off 64.0

Google's Flash lineup, two-chapter average. Closeness falls with every newer generation, and gemini-3.5-flash-lite, the cheapest of the four, is at the top. The line is the closeness limit from the ranking further down, 10% under the best run.

On chapter 2, gemini-3.8-flash scored 65.0% at low and 64.0% with thinking off, again the lowest complete runs except for gpt-5-mini, while gemini-3.5-flash-lite scored 68.2%, 3.2 points above it at 60% of the price. gemini-3.5-flash-lite came twelfth on that chapter, inside a cluster the metric can't order, and the gap to gemini-3.8-flash held.

So at least for this task, and for this Flash lineup, newer was not better. A model "engineered for long-horizon software engineering, autonomous agents, and complex enterprise workflows", in Google's own words, is not what I needed for constrained styling work, and the vendor's own chat assistant doesn't seem to know that.

3. Reasoning effort

I ran Anthropic's three current models at every effort level, and OpenAI's 5.6 tier at low against its medium results.

On Claude, closeness didn't move. claude-fable-5-1 scored 68.4 at low, 67.6 at medium and 67.7 at high on chapter 1, and 68.8, 68.6 and 69.0 on chapter 2. claude-opus-5 and claude-sonnet-5 gained at most a point at high, for about double the cost. What moved was the bill: output tokens, which include the thinking, more than halved from high to low, and cost and time with them. $2.68 and 545 seconds at high became $1.23 and 273 seconds at low, for the same closeness.

On OpenAI, gpt-5.6-terra and gpt-5.6-sol lost about a point at low on chapter 1, and on chapter 2 the gap was gone: gpt-5.6-terra 67.3 at low against 67.5 at medium, gpt-5.6-sol 67.7 against 66.8. So low is the setting there too.

More effort: what it did to closeness and to the bill Every model that ran at more than one setting and returned every block. One dot per setting, darker is more effort. limit 61.7% limit $2.16 closeness, average (best at the left) cost per book, log scale 60% 62% 64% 66% 68% 70% $0.50 $1 $2 $5 $10 $20 claude-fable-5-1 68.6 68.3 68.1 $13.69 $21.72 $27.50 claude-opus-5 67.1 66.7 $6.27 $12.44 claude-sonnet-5 67.7 67.0 $2.50 $5.76 gpt-5.6-sol 67.1 67.0 $3.15 $3.71 gpt-5.6-terra 67.5 66.8 $1.83 $1.98 gemini-3.8-flash 64.8 64.0 $0.73 $0.74 lowest setting middle highest

The six models I ran at more than one setting, closeness on the left and book cost on the right. On Claude, going from low to high roughly doubles the price and moves closeness by less than a point. On OpenAI's 5.6 tier the step from low to medium adds 8 to 18% to the price for under a point. On gemini-3.8-flash, thinking low against off changed the price by a cent. The lines are the limits from the ranking further down: closeness 10% under the best run, cost five times the cheapest.

In these two chapters, on all three vendors, the extra thinking was billed output that did not improve the translation by more than a point. I'd start every reasoning model at the floor and measure up. In fairness, that is exactly what Anthropic's own docs say to do.

4. The whole field

All 27 configurations with a recorded setting, sorted by rank, with the average of the two chapters and cost and time scaled to a 200-page book, about 71,000 words at this book's density. Times are with the chunks run one after the other, as in the benchmark, so they are a ceiling. Setting is the thinking, reasoning or effort level, and the two highlighted rows are the picks. Three chapter 1 Anthropic runs where I hadn't logged the effort are left out.

Score is how far a run sits from the best complete run on each axis, measured in units of what I'd still accept on that axis, added up. The best on every axis would score 0.

My limits:

  • closeness within 10% of the best average, which is about one word in fourteen
  • cost up to 5x the cheapest
  • time up to 10x the fastest

score = (68.6 - average) / 6.86 + (cost - $0.43) / $1.72 + (time - 7 min) / 63 min

A term of 1 means at the edge of a limit. A run is within limits only if all three terms are 1 or less, and the limits column names the ones it failed.

Lower is better. Rank follows the score, so the lowest score is rank 1. The runs that dropped blocks are placed last, since part of their cost and time is text they never produced.

There are no weights, only those three limits. And it is still a value number, not translation quality: rank 4 means the fourth-best trade of closeness against cost and time, not the fourth-best Portuguese.

Rank Score Limits Vendor Model Setting Ch. 1 Ch. 2 Average Book cost Book time Blocks 1 0.04 in Google gemini-3.5-flash-lite minimal 68.4% 68.2% 68.3% $0.43 7 min all, all 2 0.35 in xAI grok-4.3 none 67.5% 69.0% 68.2% $0.65 18 min all, all 3 0.36 in Google gemini-3.6-flash minimal 67.5% 68.3% 67.9% $0.73 12 min all, all 4 0.54 in Google gemini-3.7-flash off 66.2% 66.2% 66.2% $0.74 8 min all, all 5 0.74 in Google gemini-3.8-flash low 64.5% 65.0% 64.8% $0.73 8 min all, all 6 0.86 in Google gemini-3.8-flash off 63.9% 64.0% 64.0% $0.74 8 min all, all 7 1.06 in OpenAI gpt-4.1 n/a 63.2% 68.5% 65.8% $1.46 11 min all, all 8 1.39 in xAI grok-4.5 low 66.5% 67.3% 66.9% $1.41 45 min all, all 9 1.41 in OpenAI gpt-5.6-terra low 66.4% 67.3% 66.8% $1.83 29 min all, all 10 1.43 in OpenAI gpt-5.6-terra medium 67.4% 67.5% 67.5% $1.98 31 min all, all 11 1.73 in xAI grok-4.6 low 66.1% 66.8% 66.4% $1.69 52 min all, all 12 1.84 out, cost Anthropic claude-sonnet-5 low 66.6% 67.5% 67.0% $2.50 34 min all, all 13 2.17 out, cost OpenAI gpt-5.6-sol low 66.4% 67.7% 67.1% $3.15 32 min all, all 14 2.78 out, cost OpenAI gpt-5.6-sol medium 67.2% 66.8% 67.0% $3.71 50 min all, all 15 2.88 out, closeness, time OpenAI gpt-5-mini medium 62.4% 57.9% 60.1% $1.11 1 h 28 min all, all 16 4.22 out, cost Anthropic claude-opus-5 low 66.3% 67.1% 66.7% $6.27 44 min all, all 17 4.33 out, cost, time Anthropic claude-sonnet-5 high 66.8% 68.5% 67.7% $5.76 1 h 20 min all, all 18 4.78 out, cost Google gemini-3.1-pro-preview medium 65.0% 71.1% 68.0% $6.99 1 h 06 min all, all 19 5.24 out, cost, time OpenAI gpt-5 medium 63.6% 65.7% 64.7% $6.11 1 h 37 min all, all 20 8.35 out, cost Anthropic claude-fable-5-1 low 68.4% 68.8% 68.6% $13.69 51 min all, all 21 8.36 out, cost, time Anthropic claude-opus-5 high 66.3% 67.9% 67.1% $12.44 1 h 24 min all, all 22 13.43 out, cost, time Anthropic claude-fable-5-1 medium 67.6% 68.6% 68.1% $21.72 1 h 15 min all, all 23 17.05 out, cost, time Anthropic claude-fable-5-1 high 67.7% 69.0% 68.3% $27.50 1 h 34 min all, all 24 0.94 out, blocks OpenAI gpt-5.6-luna low 57.9% 68.7% 63.3% $0.18 18 min 50/67, 86/88 25 1.32 out, blocks Google gemini-3.5-flash off 61.3% 69.5% 65.4% $1.77 13 min 63/67, all 26 1.75 out, blocks, closeness OpenAI gpt-5.6-luna medium 46.7% 68.6% 57.6% $0.20 17 min 57/67, 86/88 27 7.04 out, blocks, cost Google gemini-3.5-flash medium 60.0% 65.1% 62.5% $9.74 57 min 55/67, 76/88

Google: closeness falls from gemini-3.6-flash to gemini-3.8-flash on both chapters, and gemini-3.5-flash-lite matches or beats every newer Flash at the lowest price. Only gemini-3.5-flash drops blocks. The chart is in finding 2.

Anthropic: effort moves the bill more than the score. Low and high sit within about a point of each other on both chapters, with low at about half the cost and time.

Anthropic: closeness by model Average of the two chapters. All three at effort low, the setting that costs least and scores the same. 60% 62% 64% 66% 68% 70% claude-sonnet-5 claude-opus-5 claude-fable-5-1 limit 61.7% 67.0 66.7 68.6

Anthropic at effort low. claude-fable-5-1 is the highest average in the whole benchmark, 0.3 above gemini-3.5-flash-lite, at 32 times the price.

OpenAI: the 5.6 tier at low matches medium on chapter 2. gpt-5.6-luna is the cheapest run in the benchmark and incomplete on both chapters.

OpenAI: closeness by model Average of the two chapters, reasoning models at medium. gpt-5.6-luna is left out, it dropped blocks. 56% 60% 64% 68% gpt-4.1 gpt-5-mini gpt-5 gpt-5.6-terra gpt-5.6-sol limit 61.7% 65.8 60.1 64.7 67.5 67.0

OpenAI at medium. The 5.6 tier is a clear step up from gpt-5 and gpt-5-mini, and gpt-4.1 lands between them.

xAI: grok-4.3 beats its two successors on both chapters at a third to half of their time, and it's the only one of the three that can switch reasoning off.

xAI: closeness by model Average of the two chapters. grok-4.3 with reasoning off; 4.5 and 4.6 at low, the least they allow. 60% 62% 64% 66% 68% 70% grok-4.3 grok-4.5 grok-4.6 limit 61.7% 68.2 66.9 66.4

xAI: grok-4.3 with reasoning off scored above its two newer siblings, which can't switch reasoning off.

How far each run sits from the best, on each limit Each column is one term of the score. The line at 1 is the limit, green within it, red past it. Dots past 3 sit at the edge with their value. closeness term 0 1 2 3 cost term 0 1 2 3 3.4 3.1 3.8 3.3 7.7 7.0 12.3 15.7 5.4 time term 0 1 2 3 1 gemini-3.5-flash-lite minimal 0.04 2 grok-4.3 none 0.35 3 gemini-3.6-flash minimal 0.36 4 gemini-3.7-flash off 0.54 5 gemini-3.8-flash low 0.74 6 gemini-3.8-flash off 0.86 7 gpt-4.1 n/a 1.06 8 grok-4.5 low 1.39 9 gpt-5.6-terra low 1.41 10 gpt-5.6-terra medium 1.43 11 grok-4.6 low 1.73 12 claude-sonnet-5 low 1.84 13 gpt-5.6-sol low 2.17 14 gpt-5.6-sol medium 2.78 15 gpt-5-mini medium 2.88 16 claude-opus-5 low 4.22 17 claude-sonnet-5 high 4.33 18 gemini-3.1-pro-preview medium 4.78 19 gpt-5 medium 5.24 20 claude-fable-5-1 low 8.35 21 claude-opus-5 high 8.36 22 claude-fable-5-1 medium 13.43 23 claude-fable-5-1 high 17.05 24 gpt-5.6-luna low 0.94 25 gemini-3.5-flash off 1.32 26 gpt-5.6-luna medium 1.75 27 gemini-3.5-flash medium 7.04 score past 3, value shown the limit Google OpenAI Anthropic xAI

How far each run sits from the best on each of the three limits, with the limit drawn as the line at 1. The top eleven clear all three. claude-fable-5-1 fails on cost alone, by a wide margin. gpt-5-mini fails on closeness and time. Two of the four runs at the bottom would pass every limit, but they dropped blocks.

Closeness, per configuration Two-chapter average, best at the left. The line is the closeness limit, green within it, red past it. Muted rows dropped blocks. 70% 68% 66% 64% 62% 60% 58% 56% limit 61.7% closeness to the human translation, two-chapter average claude-fable-5-1 low 68.6 gemini-3.5-flash-lite minimal 68.3 claude-fable-5-1 high 68.3 grok-4.3 none 68.2 claude-fable-5-1 medium 68.1 gemini-3.1-pro-preview medium 68.0 gemini-3.6-flash minimal 67.9 claude-sonnet-5 high 67.7 gpt-5.6-terra medium 67.5 claude-opus-5 high 67.1 gpt-5.6-sol low 67.1 claude-sonnet-5 low 67.0 gpt-5.6-sol medium 67.0 grok-4.5 low 66.9 gpt-5.6-terra low 66.8 claude-opus-5 low 66.7 grok-4.6 low 66.4 gemini-3.7-flash off 66.2 gpt-4.1 n/a 65.8 gemini-3.5-flash off 65.4 gemini-3.8-flash low 64.8 gpt-5 medium 64.7 gemini-3.8-flash off 64.0 gpt-5.6-luna low 63.3 gemini-3.5-flash medium 62.5 gpt-5-mini medium 60.1 gpt-5.6-luna medium 57.6 returned every block dropped blocks the closeness limit Google OpenAI Anthropic xAI

Every configuration by two-chapter average closeness, best first. Every complete run except gpt-5-mini sits within five points of the best, which is the whole tie. The cost and time charts in the next section cover the same rows, and there the complete runs spread from $0.43 to $27.50 and from 7 minutes to an hour and a half.

Of the 23 complete runs, only two are not beaten on closeness, cost and time at once: gemini-3.5-flash-lite at minimal (68.3%, $0.43, 7 min) and claude-fable-5-1 at low (68.6%, $13.69, 51 min). claude-fable-5-1's edge is 0.3 points, inside the noise, at 32x the cost and 7x the time. Read gaps under a point as ties, and gemini-3.5-flash-lite is the only run nothing beats.

What I decided

The score is not that different between the top models. It's pretty much tied. On chapter 1 gemini-3.5-flash-lite came first at 68.4% and grok-4.3 sixth at 67.5%. On chapter 2 grok-4.3 came fourth at 69.0% and gemini-3.5-flash-lite twelfth at 68.2%. That looks like a big move, but gemini-3.5-flash-lite's own score moved 0.2 points, and everything from fourth to twelfth sits inside 0.8 points. So the operational numbers decide: cost and runtime.

What a 200-page book costs, per configuration About 71,000 words, scaled from both chapters. The line is the cost limit, green within it, red past it. Muted rows dropped blocks. $0.20 $0.50 $1 $2 $5 $10 $20 limit $2.16 cost per book, log scale gpt-5.6-luna low $0.18 gpt-5.6-luna medium $0.20 gemini-3.5-flash-lite minimal $0.43 grok-4.3 none $0.65 gemini-3.8-flash low $0.73 gemini-3.6-flash minimal $0.73 gemini-3.7-flash off $0.74 gemini-3.8-flash off $0.74 gpt-5-mini medium $1.11 grok-4.5 low $1.41 gpt-4.1 n/a $1.46 grok-4.6 low $1.69 gemini-3.5-flash off $1.77 gpt-5.6-terra low $1.83 gpt-5.6-terra medium $1.98 claude-sonnet-5 low $2.50 gpt-5.6-sol low $3.15 gpt-5.6-sol medium $3.71 claude-sonnet-5 high $5.76 gpt-5 medium $6.11 claude-opus-5 low $6.27 gemini-3.1-pro-preview medium $6.99 gemini-3.5-flash medium $9.74 claude-opus-5 high $12.44 claude-fable-5-1 low $13.69 claude-fable-5-1 medium $21.72 claude-fable-5-1 high $27.50 returned every block dropped blocks the cost limit Google OpenAI Anthropic xAI

What each configuration would cost for a 200-page book, on a log scale. The line is the cost limit, 5x the cheapest. gemini-3.5-flash-lite and grok-4.3 are the two cheapest runs that returned every block. The Flash line sits at $0.73, gpt-5.6-terra just under the limit, and claude-fable-5-1 at 30 to 60 times gemini-3.5-flash-lite.

How long a 200-page book takes, per configuration Chunks run one after the other, as in the benchmark. The line is the time limit, green within it, red past it. Muted rows dropped blocks. 5 min 10 min 20 min 30 min 60 min 120 min limit 72 min time per book, log scale gemini-3.5-flash-lite minimal 7 min gemini-3.8-flash low 8 min gemini-3.8-flash off 8 min gemini-3.7-flash off 8 min gpt-4.1 n/a 11 min gemini-3.6-flash minimal 12 min gemini-3.5-flash off 13 min gpt-5.6-luna medium 17 min grok-4.3 none 18 min gpt-5.6-luna low 18 min gpt-5.6-terra low 29 min gpt-5.6-terra medium 31 min gpt-5.6-sol low 32 min claude-sonnet-5 low 34 min claude-opus-5 low 44 min grok-4.5 low 45 min gpt-5.6-sol medium 50 min claude-fable-5-1 low 51 min grok-4.6 low 52 min gemini-3.5-flash medium 57 min gemini-3.1-pro-preview medium 1 h 06 min claude-fable-5-1 medium 1 h 15 min claude-sonnet-5 high 1 h 20 min claude-opus-5 high 1 h 24 min gpt-5-mini medium 1 h 28 min claude-fable-5-1 high 1 h 34 min gpt-5 medium 1 h 37 min returned every block dropped blocks the time limit Google OpenAI Anthropic xAI

The same book by time, with the chunks run one after the other. gemini-3.5-flash-lite is the fastest at 7 minutes, the Flash line is done inside 12, grok-4.3 takes 18. Six runs are past the 72-minute limit, and every Claude run at high effort is among them. A pipeline that sends chunks in parallel shortens every row; the order stays.

That's the pick:

  • Primary: gemini-3.5-flash-lite at thinking minimal. The cheapest and fastest complete run on both chapters, 68.4% and 68.2%, every block and marker intact: $0.43 and 7 minutes for a 200-page book.
  • Fallback: grok-4.3 at no reasoning. Top cluster on both chapters at 67.5% and 69.0%, every block on both, $0.65 and 18 minutes a book, and a different vendor, so one outage can't take both.
  • Two settings I'm carrying into the pipeline for Gemini. Always send the output-token limit yourself: the model's default is what gemini-3.5-flash's thinking ran into when it cut a reply mid-string. And ask for JSON through the API's schema option, not the prompt: every complete gemini-3.5-flash-lite run had it on.

On those numbers gemini-3.5-flash-lite wins: 0.6 to 0.7x grok-4.3's cost and 2 to 3x its speed, with the same completeness and marker fidelity. The price gap is about 20 cents a book, and the speed gap is 11 minutes a book only with the chunks run one after the other. Sent in parallel, it's seconds per chapter.

Conclusion

Eighteen models, two chapters, one human translation to measure against, and the top of the table is a tie. The best complete runs sit inside a point of each other, and the heat map says why: the passage changes the score three times more than the model does. On closeness alone I couldn't pick.

So cost and time picked. gemini-3.5-flash-lite at thinking minimal returned every block on both chapters for $0.43 and 7 minutes a 200-page book. grok-4.3 with reasoning off did the same for $0.65 and 18 minutes, from a different vendor, so one outage can't take both. claude-fable-5-1 at high effort tied them on closeness at 64 times gemini-3.5-flash-lite's price.

Where the models did differ was not the Portuguese. Two of them dropped blocks. More effort bought price, not closeness. And Gemini's own recommendation, gemini-3.8-flash, lost to gemini-3.5-flash-lite, three generations older, on both chapters.

Closeness is still a lexical proxy, and the real test is a blind read by the human translator. That's the next step. Until then I'd read this ranking as tiers, not positions: the two chapters agreed on the tiers and shuffled the order inside them.

If you take one thing from this, take the method, not the pick: run the candidates on your own task, count what they finish before you grade how well they did it, begin at the cheapest effort, and let a person make the last call.

Photo by Patrick Fore on Unsplash

Mathias Grimm

Mathias Grimm

Software engineer, currently doing tech support at Laravel. Writing about PHP, image processing and the craft of building software. Sometimes about the code, sometimes about what it means to write it. Father and husband. Follower of Christ.

[ follow ]