Rankings you can't game: why we whale-filter

A usage leaderboard is supposed to answer one question: what are developers actually choosing to run in production? That signal breaks the moment one customer’s traffic can single-handedly decide the outcome. It already happened, on a major aggregator, in public.

What happened

In May 2026, a free promotional model called Hy3 reached #1 on OpenRouter’s public model usage rankings — not by a small margin, but with 7.7 trillion tokens routed, a 54% lead over the next-highest model. The catch: that volume wasn’t broad developer adoption. It was consistent with a single heavy application routing enormous volume through a model that happened to be free during a promotional window. The analysis was picked up widely — an HN thread on it ran to 150 points and 112 comments — because it made an obvious point obvious: a usage leaderboard that one customer can top by routing enough tokens isn’t measuring what people think it’s measuring.

This isn’t a knock on OpenRouter’s product. Usage rankings are a genuinely useful idea — they’re the only leaderboard type in the LLM-aggregator space, and they get cited constantly as a market signal. The Hy3 episode is a knock on unfiltered usage rankings specifically: if the ranking counts raw tokens with no cap on how much any single account can contribute, a whale distorts the whole board, and the leaderboard stops being a demand signal and starts being a story about one customer’s architecture choices.

Why this matters beyond one incident

If you’re picking a model based on “what’s popular,” a gameable leaderboard actively misleads you. Worse, it’s not hard to game on purpose — a lab wanting its model to look popular during a free-tier or promo window has every incentive to route (or pay someone to route) enormous volume through it for exactly as long as the promo lasts, then let the ranking coast on the resulting spike. Nothing about an unfiltered token-count leaderboard catches that.

What we do differently

Our rankings methodology is published, not just claimed, and it’s built around three specific controls:

Whale filtering. No single account’s traffic counts for more than 5% of any model’s ranked volume. However much one customer routes, it cannot move a model’s position by more than one band. This is the direct fix for the Hy3 scenario: a single application generating trillions of tokens gets capped rather than counted in full.

Promo-price flags. Any model priced at zero or steeply discounted by its provider gets flagged on the leaderboard, and free-tier traffic is ranked separately from paid traffic. Free usage tells you what’s cheap enough to experiment with liberally. Paid usage tells you what people trust enough to spend real money on. Conflating the two — which is exactly what let a free promotional model hit #1 — produces a number that looks like a quality signal but is actually a price signal.

No self-dealing. Traffic from our own health checks, playground demos, and verification pipelines is excluded from ranking volume. A leaderboard shouldn’t include the operator’s own test traffic padding the numbers.

On top of the filters: rolling aliases (a -latest tag, for instance) fold into their underlying model rather than counting as a separate leaderboard entry, so version churn doesn’t fragment or inflate a model’s standing.

What we publish, and what we don’t

Every ranking shows the counted token volume, the number of distinct accounts behind that volume, week-over-week change, and any active promo-price or whale flags. If a methodology change alters historical rankings, that change is dated in the changelog and the old snapshots stay available rather than quietly disappearing.

We also don’t do a few things on purpose: we don’t rank by revenue (it just privileges the most expensive models), we don’t accept payment for leaderboard placement, and we don’t editorially pin models above their measured position. If a lab thinks its numbers are wrong, there’s a public corrections process rather than a back channel.

The point

Rankings are only worth citing if they can’t be bought, gamed, or accidentally distorted by one customer’s traffic pattern. The Hy3 incident is the clearest public proof that unfiltered usage rankings fail that test. Ours are built with the filters specifically designed to prevent the failure mode that already happened elsewhere — read the full mechanics at /rankings/methodology, browse the current leaderboard at /rankings, and check any individual model’s standing on its own model page.