OpenRouter Handed Me a Free Mystery Model. I Put It Through a Real Scorecard.

I've got eleven Claude Code sessions running on a stealth model called ox-alpha that costs exactly nothing — for now. Twenty-four hours in, my honest read: not perfect, but good. Instead of trusting that, I pointed a scripted test battery at it and kept the transcripts. The engine is real: 600,000-token needle retrieval, honest answers when the answer is nothing, working vision. The envelope around the engine is broken in three ways that matter to anyone doing agentic work. And three independent fingerprints all point in the same direction: Zhipu.

Liking it isn't the same as trusting it

To be clear up front: I am not here to tell you the model is bad. Twenty-four hours in, my honest verdict is the one above — not perfect, but genuinely good. Is it at Fable or Opus or Grok level yet? No — and unlike most "is it frontier yet" arguments, I can put numbers under that answer instead of adjectives. On the exact same three structural prompts, Opus-5 answered in 4.8s, 3.2s, and 2.6s, and Fable in 5.6s, 2.7s, and 1.9s — both honoring every contract. Ox-alpha took 69 seconds, 20 seconds, and 12 seconds on those identical prompts — and got all three wrong. Nothing in a day of real work says the top seats change hands; the Critic keeps its Anthropic badge for now. But free came closer to those seats than free has any right to, and that gap is worth measuring precisely. The question this post answers is narrower: which parts of it are good enough to depend on, and which parts will bite you on a Tuesday. Liking a tool earns it a place in your rotation. Trusting one earns it a place in your architecture. Those are different thresholds, and only one of them comes with receipts.

The model is called Ox Alpha. On OpenRouter it lives at stealth/ox-alpha, it went live on August 20, and it is priced at zero. Not "free tier with limits" — the pricing fields on the API are literally "0" and "0". A third-party provider built it, chose to stay anonymous during the preview, and OpenRouter just routes to it.

I moved a pile of real work onto it. Eleven concurrent Claude Code sessions on the last ps snapshot, running through OpenRouter, doing the kind of long-horizon coding and ops work I'd normally spread across models like Kimi, Opus, Grok, and Ollama Cloud's GLM shelf. I've been testing it for about 24 hours now. Do I like what I see? Yes. Is it 100% perfect? No. But it is good — and "good but unproven" is exactly the state where discipline matters, because good-enough-to-depend-on is a claim, and claims need evidence.

That's the trap with any new model, free or not: a good first day is an anecdote wearing a lab coat. It's the same trap as reading a vendor's spec sheet and assuming the parts you care about work the way the table says — advertised capability is not capability, and the only cure is a probe against the live endpoint.

So I built a probe. A proper one.

Don't guess — inspect.
Especially when the thing you're inspecting is free, anonymous, and eating your workweek.

The envelope, verified

First, what does the API itself claim? Pulled live from OpenRouter's model registry:

Claim	Value from the registry
Context window	1,048,576 tokens
Max output	131,072 tokens
Modalities	text+image+video → text
Tool calling / structured output	both listed as supported
Reasoning	mandatory — always on, effort dial only goes low / high / max. There is no off switch.
Pricing	$0 input, $0 output
Tokenizer	listed as `"Other"` — no vendor attribution
Upstream	reported simply as `"Stealth"` on every response

Two things in that table deserve a second look. The mandatory reasoning means even "reply with the word PONG" burns thinking budget on every call — more on the cost of that below. And the anonymous tokenizer is the thread I'll pull later, because it turns out to be the loudest clue about who built this thing.

How I tested — and who graded it

A confession that cuts both ways: the agent that wrote and ran this battery runs on ox-alpha itself. I used the model to grade the model.

I'm not pretending that's ideal. I am saying it's manageable, because the battery was built so that nothing rides on judgment. Every test has a ground truth computed by the script — the expected product of two integers, the expected letter count in a sentence the script generated, the expected vault codes hidden in a log the script assembled. The grader is regex and arithmetic, not vibes. If ox-alpha wanted to flatter itself, it would have to alter arithmetic.

The battery:

Objective reasoning — multi-digit multiplication, a chained word problem with a non-integer midpoint, case-insensitive letter counting, and a bat-and-ball-style trap with changed numbers so a memorized answer fails.
Structured output — strict JSON Schema extraction, three runs.
Tool calling — single call, parallel calls across cities plus math, and a forced tool_choice.
Long context — synthetic logs with unique audit needles planted at 10%, 50%, and 90% depth, plus a fourth needle that was never planted. Asking about the phantom one is the honesty probe: a model that fills from vibes will report a color for something that doesn't exist.
Vision — a rendered image with known text at two sizes and four shapes whose colors and positions the script knows.
Latency — time-to-first-byte and tokens/sec on streamed calls.

What held up

Test	Result	Detail
Reasoning battery	4/4	847×396, the decimal chain (86.95 exactly), letter count, the changed-numbers trap. All clean.
Long-context retrieval	clean through 601k	All three needles found at 120k / 200k / 400k / 601,687 reported prompt tokens.
Honesty at depth	perfect	The never-planted needle came back `NONE` at every depth. No confabulation anywhere in the haystack.
Vision	5/5	Read the large code, read the small-print serial number, placed all four shapes correctly — including noting the triangle points downward.
Effort dial	real	Same multiplication task: 56 completion tokens at low effort, 97 at high, 322 at max. Correct all three times.
Streaming speed	usable	~6.7s to first token warm, ~33 tokens/sec sustained.

The long-context result is the headline. This is not "the API accepted a big prompt." The needles were buried in filler text at specific depths, and retrieval was perfect from shallow to 600k-plus — including the part most models fail, which is admitting a needle isn't there. Whatever is under the anonymity badge, somebody taught it to distinguish knowing from guessing. I have opinions about why that might be. They come later.

Where it broke

Three structural failures, all on the axes the marketing copy leads with — coding, agents, production workloads. Each one reproduced cleanly.

1. Strict JSON schema is theater through this route

The registry advertises structured output. I sent a strict JSON Schema asking for a fixed shape — order ID, customer tier, items with quantities, total. Three runs. Zero parseable responses. Here's the actual opening of the reply, unedited:

Since no specific schema was provided, here's the order extracted
into a standard JSON structure:

```json
{
  "order_id": "ORD-7712",
  ...invented field names from here...

The model never saw my schema. The parameter is listed as supported; somewhere between OpenRouter and the anonymous upstream, it silently evaporates — and the model politely announces that no schema arrived. Your validation layer gets markdown fences and invented field names instead of the contract you asked for.

2. Parallel tool calls collapse into illegal JSON

I asked for two weather lookups and a calculation. This is the bread and butter of an agent loop. The model emitted two tool calls — but the weather call's arguments looked like this, verbatim from the transcript:

{"city":"Tokyo","city":"Paris"}

Two identical keys in one object. That is not valid JSON in any meaningful sense — the spec explicitly says duplicate names are ambiguous. What happens next depends on which parser your stack uses: Python keeps Paris, some C++ parsers keep Tokyo, strict validators reject the whole call. One request, three different outcomes, none of them flagged as an error.

This is the nastiest class of bug there is: not a crash, but silent environment-dependent data loss inside the exact workflow the model is sold for. A missing tool call throws. A mangled one quietly does the wrong job.

3. Forced tool_choice is ignored

The API supports forcing a specific tool. I forced calculator on a message that said "just say hello." The model replied with a greeting. Its hidden reasoning trace, which the route does expose, said it all:

"The user just wants a simple greeting. No tools needed."

Correct social instinct. Wrong contract. The parameter said call the calculator; nobody enforced it.

4. Past ~650k tokens, it fails like a liar, not like a server

The advertised context is 1,048,576 tokens. Retrieval stayed perfect at 601,687. Then I pushed the haystack bigger, and at roughly 660,000 tokens something interesting happened:

HTTP 200 OK
finish_reason: "stop"
content: ""          <- a="" back="" bytes="" code="" count="" even="" not="" null="" token="" usage:="" zero="">

No error code. No truncation notice. A success-shaped response containing nothing. If your pipeline checks status codes — and most do — it will treat silence as success and propagate emptiness downstream. The honest behaviors here are a 400, or a truncated flag, or at minimum a usage object telling you how much of your prompt survived. I got a smile and shrug.

Latency footnote. Cold start is rough: the very first call took 87 seconds to return the literal word PONG. Warm calls run 3–40 seconds depending on how much mandatory thinking the router applies. And because reasoning cannot be switched off, a trivial extraction that should cost ~80 tokens burned 500–700 on hidden thinking. Budget accordingly.

Then I made it fight the neighbors

A scorecard against itself is incomplete. So I replayed the three structural tests verbatim against seven named competitors on the same aggregator, same prompts, same day. Total spend for the entire comparison matrix: about eleven cents.

Model	Strict schema	Parallel tools	Forced choice	List price (in/out per M)
stealth/ox-alpha	✗ 0/3 parseable	✗ duplicate keys	✗ ignored	free (preview)
z-ai/glm-5.3	✗ same failure	✓	✗ ignored	$1.40 / $4.40
~z-ai/glm-latest	✗ empty reply	✓	✗ ignored	$1.40 / $4.40
x-ai/grok-4.6	✓	✓	✓	$2.00 / $6.00
~deepseek/deepseek-v4-flash-latest	✓	✓	✓	$0.065 / $0.18
anthropic/claude-opus-5	✓	✓	✓	$5.00 / $25.00
anthropic/claude-opus-5-fast	✓	✓	✓	$10.00 / $50.00
~anthropic/claude-fable-latest	✓	✓	✓	$10.00 / $50.00

Read that table twice, because it kills two lazy takes at once.

First take killed: "that's just OpenRouter mangling parameters." No. Five other providers' routes honored the exact same parameters on the exact same day. Only the stealth route and the Zhipu routes dropped them.

Second take killed: "free wins because free." DeepSeek-V4-Flash passed every single test ox-alpha failed, at $0.065 per million input tokens. The cheapest model in the comparison is also the cheapest way to avoid engineering around ox-alpha's holes. Free isn't a price once you pay in workarounds.

So what is it? Three fingerprints say Zhipu

The community has been guessing since launch day, mostly landing on a Zhipu / Z.ai GLM-family model. I wasn't going to settle identity by vibes, so I ran three independent forensic probes.

Fingerprint one: the tokenizer matches GLM to the token. I sent six adversarial strings — plain English, mixed Chinese-English, snake_case code, 300 repeated delimiters, mathematical-alphabet Unicode with emoji, a 200-digit run — through ox-alpha and through candidate families, then compared token counts string by string. Against GLM-5.2, the difference was exactly 75 tokens on every single string: prose, CJK, delimiters, digits, fraktur, all of it. A constant offset means template overhead; identical per-string counts mean the same tokenizer making the same decisions on every stress case. Meanwhile DeepSeek drifted to −142 on digit runs and Kimi to −138, and Gemma disagreed wildly everywhere. Six strings, zero divergence, only against GLM. That's not coincidence; that's lineage.

(Bonus leak: the constant 75-token offset means the stealth route injects its own hidden system preamble into every conversation. Something is whispering instructions to your model before you speak.)

Fingerprint two: identical failure handwriting. Remember the schema-test confession — "Since no specific schema was provided, here's the order extracted into a standard JSON structure." GLM-5.3, routed separately and billed normally, opened its own failed attempt with: "Since no schema was attached, here's the order extracted into a standard JSON structure." Same sentence shape, same fence, same invented field renames. Two different accounts, two different routes, one shared serving stack behaving identically badly.

Fingerprint three: only the Zhipu family ignores forced tools. Across the whole matrix, exactly three entries blew off tool_choice — the stealth model and the two Zhipu entries. Grok, DeepSeek, and both Claude seats obeyed the contract.

Token counting, failure mode, and parameter handling all pointing the same direction puts me near 85% confidence on a GLM-family upstream — up from the 65–75% I'd have guessed from capability vibes alone. One honest wrinkle keeps me off 95%: ox-alpha mangled the parallel-tool test in a way retail GLM-5.3 did not, so whatever this is, it isn't byte-for-byte yesterday's checkpoint. Different build, same bloodline.

Free is a launch price

Say the quiet part out loud: this is free the way a dealer's first sample is free.

Every indicator says limited-time preview: a $0.00 pricing row, an expiration date decades out in the registry, an anonymous provider buying market data with compute they already have. None of that is a complaint — it's a rational launch strategy, and smart people are getting real work done on their dime. But build your architecture around the current envelope and read the fine print later, and the bill arrives all at once: the day the preview ends, pricing appears, and whatever the free period taught your pipelines to tolerate becomes your new dependency surface.

So the question is not "is it good?" It's "what survives the meter starting?" The engine will still be the engine when it costs money — 600k-token retrieval doesn't evaporate with a pricing change. The envelope may improve overnight too, because these particular failures smell like plumbing, not weights: the schema and tool-choice drops look like translation gaps between OpenRouter and the upstream, and plumbing can be fixed without touching a checkpoint. But you don't get to count on that. What you get to do is architect so that neither the free price nor the broken envelope is load-bearing in anything you keep.

If you're running it today

The rules I'm applying to my own fleet, effective immediately:

Never trust response_format on this route. Ask for JSON in the prompt, strip fences, validate against the schema yourself, reject loudly on violation.
Put a duplicate-key detector on tool dispatch. Parse arguments with a hook that flags repeated keys, and split them into separate calls before execution.
Treat empty completions as retryable failures. Check finish_reason AND non-empty content AND a usage object. Any one missing: back off and retry, never pass silence downstream.
Cap prompts around 600k tokens despite the advertised million. The cliff sits somewhere past 600k and it's shaped like a success.
Validate one call end-to-end before any batch. Same rule I apply to every paid API; free endpoints deserve it even more, because nobody reads errors off a free meter.

The whole thing in five lines

# the engine is real
600k-token needle retrieval, perfect, and honest when the needle isn't there

# the envelope is not
json_schema -> silently dropped   parallel calls -> duplicate keys
forced choice -> ignored          ~660k prompt -> HTTP 200, zero bytes

# the lineage leaves prints
tokenizer, failure handwriting, and parameter handling all say Zhipu

# the price is temporary
free is how you buy market share; architecture like the meter is running

# the discipline that found all of this
don't guess -- inspect

Pages

Saturday, August 22, 2026