Methodology

SDKScore is a compiler-graded benchmark. Everything below is stated so that a hostile reader can reproduce a score and prove us wrong. Where the method is weak, it says so.

01What is measured

One question: when an AI coding agent is asked to write a first integration against a given SDK, does the code it produces type-check against that SDK's real, published types?

That is not the same as "is this SDK good", and it is not a measure of the agent's general ability. It is a proxy for one specific thing that has become expensive: an SDK whose surface confuses agents produces broken first drafts, and the cost lands on the SDK's own support channel.

02Version pinning

Each SDK gets a throwaway npm project containing nothing but that package at an exact pinned version (no ranges), TypeScript, and a fixed tsconfig.json. Types are read from the installed package. Never from documentation, never from a hand-written stub.

{
  "compilerOptions": {
    "strict": true,
    "target": "ES2022",
    "module": "NodeNext",
    "moduleResolution": "NodeNext",
    "noEmit": true,
    "skipLibCheck": true,
    "types": ["node"]
  }
}

skipLibCheck is on deliberately: we grade the agent's file, not pre-existing type errors inside the SDK's own .d.ts files. The pinned version and the TypeScript version are recorded in every results.json, so a score always refers to a specific pair.

03Tasks and prompts

Each SDK has four tasks. Tasks are chosen to be the things a developer actually writes in the first week: auth, the main write path, the main read path, and one thing that is easy to get subtly wrong (webhook verification, pagination, streaming).

• Prompts are published in full in config/sdks.json. There is no hidden prompt.
• The same prompt string goes to every model, verbatim.
• Prompts describe the goal, not the API. They never name the method the agent is expected to call.
• Prompts state the pinned major version, because a real developer would know it from their package.json.

04Generation

Every sample is produced under identical conditions:

The one-attempt rule matters. Agents with a compile-and-fix loop score far higher on everything, which measures the harness rather than the SDK. SDKScore deliberately grades the first draft, because that is what tells you whether your API surface is guessable.

05Grading

A task passes only if both checks pass:

1. Compilation. tsc --noEmit over the generated file with the config above, against the pinned SDK's real types. Any error attributable to the generated file fails the task. Every error string is published.
2. Pattern checks. A small, published set of required and forbidden string/regex patterns per task. See below.

Grading is a pure function of the generated file. Rerunning the grader on the same file always yields the same verdict. There is no partial credit inside a task: it compiles clean and satisfies its patterns, or it fails.

06Pattern checks

The compiler misses one important failure mode: code that type-checks but uses an API the vendor has verifiably deprecated. Pattern checks cover exactly that gap, under strict rules:

• A forbidden pattern comes in exactly two flavours, and we say which. Deprecation patterns are only added when the removal or deprecation is documented by the SDK vendor (a changelog entry, a migration guide, or a @deprecated tag), or is provably absent from the pinned build; the citation is stored next to the pattern. Spec guards forbid an API the task prompt itself explicitly rules out (e.g. a task that asks for the low-level Git Data endpoints forbids the one-call file-contents shortcut). A spec guard is never a claim that the API is deprecated. All of them are itemised in the suite's authoring notes.
• A required pattern only asserts something the task explicitly asked for (e.g. "verify the signature" requires the verification call to appear at all).
• A handful of compiler diagnostics are declared not a signal and filtered out, per SDK, in the public suite (ignoreTscErrors). The bar is narrow: the diagnostic must be unanswerable from the information the prompt gives. Today there is exactly one: Stripe's dated apiVersion literal, which changes with every minor release inside the pinned major, so no written value can be correct across the pinned range.
• Patterns are never used to encode stylistic preference, naming, or structure.
• All patterns ship in the public task suite. If you can't see it, it isn't applied.

Every pattern failure on the leaderboard is shown verbatim, with the reason attached.

07Scoring and bands

Per model: score = round(100 × passed ÷ total). With four tasks the per-model score is one of 0, 25, 50, 75, 100. The scale is deliberately coarse: four samples do not support finer resolution. The overall score is the unweighted mean across models. Tasks are equally weighted; there are no bonus points.

Trend is the change in overall score versus the previous published run. A trend can move because the agent changed, because the SDK released a new pinned version, or because a task was added. The run record says which.

08Why there is no LLM judge

No model grades anything, anywhere in this pipeline. Not as a judge, not as a tie-breaker, not to "sanity check" a failure.

• Reproducibility. Anyone can rerun tsc and get our exact verdict. An LLM judge makes a score unfalsifiable.
• Conflict of interest. The models being benchmarked are made by the same vendors whose SDKs may appear on the board. A model must not grade work adjacent to its own maker.
• Known failure mode. LLM judges reward fluent, plausible code. That is exactly the code we are trying to catch when it doesn't compile.

The cost of this choice is coverage, and it is stated plainly in limitations.

09Contesting a score

If a task is unfair, the score is wrong, and we want to know. Open a public issue on the benchmark repository with the SDK id, the task id, and the model. Useful grounds:

• The prompt is ambiguous, or is solvable only by guessing an undocumented symbol.
• A forbidden pattern flags an API that is not actually deprecated. Link the vendor's docs.
• The pinned version is stale relative to a release that changed the surface.
• The compiler error comes from the environment, not the generated code.
• The task isn't representative of real first-week usage of your SDK.

Disputes are handled in public and the outcome is recorded. Accepted disputes change the task suite, and the whole matrix is rerun. Never a single SDK in isolation. Vendors get no private channel to adjust their own numbers, and there is no way to pay to be removed from the board.

10Limitations

• Compilation ≠ correctness. Code can type-check perfectly and still be wrong at runtime: wrong endpoint, wrong idempotency handling, wrong retry semantics, silently swallowed errors. We do not execute anything, so we cannot see any of that. A 100 means "nothing detectably wrong", not "correct".
• TypeScript only (v1). Python, Go, Java and Rust SDKs are not covered. A TS-only result says nothing about your other clients.
• Claude-family agents only (v1). Two models. It is not a cross-vendor model ranking, and we don't present it as one.
• Four tasks is a small sample. One task flipping moves a per-model score by 25 points. Small differences between adjacent SDKs are noise; treat the bands, not the digits, as the signal.
• Task choice is a judgement call. We pick the tasks. A different four tasks would produce different numbers. That's why prompts are public and contestable.
• Sampling variance. Generation is not fully deterministic. Grading is; generation isn't. Repeated runs of the same matrix can differ.
• No credit for great docs. If your docs are excellent but not in the model's training data or context, that won't show up here.
• Age bias. SDKs whose current API predates the models' training cutoff have a structural advantage. A recent breaking rename will show up as a low score even if the new API is better designed.

11Reproducing a run

The harness, the task suite, and every published results.json are open. To check a single score by hand:

npm i <package>@<pinned-version> typescript
# paste the prompt from config/sdks.json into your agent, headless, one attempt
npx tsc --noEmit src/index.ts

If your verdict differs from ours, that is a bug report and we want it. Every results.json carries generatedAt, the harness versions, and the pinned SDK version needed to reconstruct the exact conditions.

← Back to the leaderboard