# SDKScore

> SDKScore is an independent, continuously re-run benchmark that measures how correctly AI coding agents write real integration code against popular TypeScript SDKs. Each SDK is installed at a pinned version into a throwaway project, an agent is given four public, realistic integration tasks with no human in the loop and no access to the grader, and the generated code is graded deterministically by the TypeScript compiler against the SDK's real published types. A score is simply 100 × passed ÷ total tasks. There are no LLM judges and no subjective rubric: a task passes only if it compiles clean and avoids verified-deprecated APIs.

Canonical site: https://sdkscore.dev/
Machine-readable results: https://sdkscore.dev/results.json
Methodology: https://sdkscore.dev/methodology
Contact: hello@sdkscore.dev

## Current leaderboard

Benchmark run: 2026-07-24T19:18:45.177Z
Models: haiku-4.5, sonnet-5
SDKs scored: 10

| Rank | SDK | haiku-4.5 | sonnet-5 | Overall |
| --- | --- | --- | --- | --- |
| 1 | Hono | 100 | 100 | 100 |
| 2 | Octokit (GitHub REST) | 100 | 100 | 100 |
| 3 | Supabase | 100 | 100 | 100 |
| 4 | Anthropic | 75 | 100 | 88 |
| 5 | Stripe | 75 | 100 | 88 |
| 6 | Drizzle ORM | 25 | 100 | 63 |
| 7 | Resend | 25 | 100 | 63 |
| 8 | Zod | 25 | 100 | 63 |
| 9 | OpenAI | 25 | 75 | 50 |
| 10 | Vercel AI SDK | 0 | 75 | 38 |

Scores are 0–100. Colour bands used on the site and in badges: 90–100 green, 75–89 yellow-green, 60–74 yellow, below 60 red.

## Key findings

Each line is generated from results.json and is safe to quote with attribution to SDKScore (https://sdkscore.dev/).

- In the July 2026 SDKScore run, haiku-4.5 scored 0/100 on Vercel AI SDK.
- The strongest model in the July 2026 SDKScore run was sonnet-5, averaging 95/100 across 10 SDKs.
- The lowest-scoring SDK in the July 2026 SDKScore run was Vercel AI SDK at 38/100, averaged over 2 models.

## Methodology summary

- **Pinned versions.** Every SDK is installed at an exact version into an isolated sandbox containing only that SDK, `typescript` and `@types/node`. Types come from the real published package, never from documentation.
- **Public, identical prompts.** Four realistic first-week integration tasks per SDK. The same prompts are given to every model.
- **Headless generation.** The agent writes the file with no human in the loop, no retries, and no visibility into the grader.
- **Deterministic grading.** `tsc --noEmit` against the pinned types, plus published pattern checks for APIs verified as deprecated. **No LLM judges, no vibes-based scoring.**
- **Known limitation.** Compilation is a lower bound on correctness, not proof of it: code can compile and still be wrong at runtime. Scores measure whether an agent can produce type-correct integration code, nothing more.
- **Contestable.** Any failing task can be publicly contested; the exact compiler diagnostics for every failure are published on the site and in results.json.

## Data

- [results.json](https://sdkscore.dev/results.json): every SDK, every model, every task, pass/fail, and the raw `tsc` diagnostics for each failure.
- [Badges](https://sdkscore.dev/badges/): per-SDK SVG score badges, e.g. `https://sdkscore.dev/badges/stripe.svg`.