Model compatibility

How the LEX Compatibility Score rates models against LEX AI: the probe and benchmark halves, the twenty-three fidelity checks, the four tiers and hard gates, and where to read current scores.

A
Written By Atakan KayaLast updated 3 months ago

Model compatibility

LEX AI drives whichever model your coding tool is set to, and the choice matters more than it does for ordinary chat: the workflow depends on the model following a step loop, reading the shipped Lex handbook before it writes code, and halting instead of improvising. The LEX Compatibility Score (LCS) is how that is measured and published.

What the score measures

LCS scores models, not LEX AI. The agents, prompts and workflow artefacts are held fixed and only the model varies, so the number answers one question: how well does this model drive this product. Validating LEX AI itself before a release is the orthogonal axis, and a separate internal gate.

The two halves

One number from 0 to 100, split evenly:

  • A LEX-fidelity probe (50%) — a short purpose-built task run against the real shipped artefacts.
  • Published vendor tool-use benchmarks (50%) — tau2-bench 30%, BFCL multi-turn 20%, Terminal-Bench 2.0 20%, MCP-Universe 15%, SWE-bench Verified 15% of that half.

The benchmark half only enters the composite once at least half of its weight is populated. Below that the model keeps its probe-only score and the benchmark figure is displayed marked "not blended", so a single cherry-picked benchmark result cannot move a ranking.

Inside the fidelity probe

The probe is a weighted pass rate over 23 checks, each citing a rule in the Lex handbook that ships with the product. Eleven of them are places where a Django habit contradicts a Lex rule, and those are weighted double; the remaining twelve are Lex-only contracts. Twenty are decided statically from the code the model produced, three need a human reader.

Two design decisions are worth knowing, because they are what make the number honest:

  • The brief never states a Lex rule. Every rule is discoverable only from the shipped documentation, so what is measured is whether the model goes looking rather than whether it can follow an instruction it was handed.
  • Undecided checks are excluded from both sides of the fraction. A partial run reports what it actually established instead of counting silence as failure.

The probe is one focused task, not a full twenty-step build. copilot-cli is the reference harness, because new models normally reach GitHub Copilot first; a model Copilot does not carry is scored through its vendor CLI, and the harness is printed under the model name so a mixed comparison is never hidden.

The four tiers

Tier

Label

Score

What it means

A

Recommended

88 and above

Safe for unattended production workflows.

B

Supported

76 and above

Production-capable; review the hardening steps.

C

Usable with supervision

62 and above

Completes the workflow, but expect to check artefacts per step.

D

Not recommended

Below 62, or any hard gate

Fails the workflow or a hard gate. Do not use for delivery.

Hard gates

Four failures drop a model to the bottom tier regardless of its average, because each one breaks the guarantees the workflow rests on:

  • no_project_output — produced nothing beyond framework scaffolding.
  • protocol_violation — broke the coordinator loop by skipping steps or inventing tools.
  • unsafe_git — performed git operations the workflow reserves for the MCP server.
  • low_internal — fidelity below 50, applied automatically.

Confidence is derived, not asserted

A record's confidence comes from how many graded runs stand behind it: three or more is high, two medium, one low, none untested. A record carried over from an earlier evaluation round is marked provisional. No score claims more certainty than the evidence supports.

Where to see the current scores

Run lex ai-faq and read the "which models work best" question. The table is rendered live from the evaluation log inside the installed package, so the published ranking cannot drift from the recorded evidence, and the FAQ also links a public page. Consult it there rather than from a copy — this article deliberately does not reproduce the rankings, which change as models are re-run.

At present every shipped record was scored under the previous rubric. Those rows are kept but quarantined: they sort below current records and each is labelled as not comparable and awaiting a re-run. Because of that, no model currently reaches the recommended tier, and the FAQ says so plainly rather than promoting the highest legacy score into a recommendation.

Working with a lower-tier model

The workflow still runs; what changes is the supervision cost. Read each step's artefact before letting the run advance, and hold the assistant to the halt-on-ok: false rule. The server-side gates help here — Edit and Test both refuse to finalize while a mandatory item is outstanding, which catches a weaker model before its output reaches your repository. See Execution model.

Was this helpful?

Your feedback shapes what we write next.