--- title: Static Checks vs Agent Task Tests description: "The two layers of testing whether AI agents can use a site: deterministic static checks that verify discovery files, structure, and protocol manifests, and agent task tests that run real AI agents against a task. What each catches, what each misses, and how to combine them." last_updated: 2026-11-04 canonical_url: https://agent-ready.dev/static-checks-vs-agent-task-tests --- # Static Checks vs Agent Task Tests > The two layers of testing whether AI agents can use a site: deterministic static checks that verify discovery files, structure, and protocol manifests, and agent task tests that run real AI agents against a task. What each catches, what each misses, and how to combine them. ## At a glance | Criterion | Static checks | Agent task tests | |---|---|---| | What runs | Fetches and parsers, sometimes a headless browser | An AI agent (LLM) attempting a task | | Question answered | Are the signals agents rely on present and correct? | Did this agent finish this task? | | Repeatability | Same site, same result | Varies run to run; needs repeated runs | | When it fails | Names the file or rule, and the fix | A trace to read and diagnose | | Setup | A URL | Tasks and success criteria you define | | Coverage | Every page and manifest it can reach | The tasks you wrote | | Discovery | Tested directly | Usually skipped — the agent is given the endpoint | | Changes when | Your site changes | Your site or the model changes | | Cost per run | A few HTTP requests | Model usage × agents × repeats | | Good CI gate? | Yes | Noisy; better on a schedule | | Examples | Agent Ready, isitagentready.com, Lighthouse Agentic Browsing | Microsoft Scope, commercial agent-evaluation services | ## What is a static check? A static check verifies something concrete about your site without asking an AI model to do anything. It fetches a URL, parses the response, and compares it with a rule: llms.txt exists and has an H1, the page serves Markdown for `Accept: text/markdown`, robots.txt does not block the AI crawlers you meant to allow, `/.well-known/mcp.json` describes an MCP server that actually answers. With no model in the loop, the result is deterministic: the same site gives the same result every run, and a failure points at a specific file and fix. Some static tools drive a headless browser — Lighthouse's Agentic Browsing category () reads the rendered accessibility tree and measures layout shift — but they are still deterministic audits, not agents. ## What is an agent task test? An agent task test gives a real AI agent — a coding agent such as Claude Code or GitHub Copilot, or a browsing agent — a job to do against your product and records what happened: "make a first authenticated API call", "install the SDK and list resources", "find the price of the Pro plan". You define success; the tool reports pass or fail with a trace. It measures the outcome directly, which no checklist can. Microsoft's open-source Scope () shows the shape: give agents the same task, define reusable success criteria, compare the evidence over repeated runs. ## Why doesn't a clean static scan guarantee agents succeed? The signals being present is necessary, not sufficient. A static check can confirm llms.txt is well-formed, not that the linked pages explain your API clearly enough for an agent to call it correctly. It can confirm the MCP server card at `/.well-known/mcp.json` points at a live server, not that the tool descriptions lead an agent to the right tool. Anything that depends on an agent's judgement is only observable by running one — that is the gap task tests close. ## Why doesn't a passed task test mean agents can find you? Most task tests start after discovery. To run a task against your API, CLI, or MCP server, the harness usually hands the agent the endpoint, package name, or docs URL. That is the right design for measuring task completion, but it never asks whether an agent on its own would have found the endpoint. Discovery is what static checks measure: whether llms.txt points at the right docs, whether your MCP server, agent card, or API catalog is advertised at its well-known path, whether pages read without JavaScript. A product that passes every task test when given its own URL can still be invisible to an agent that has to find it. ## Why does determinism matter? Task tests inherit the variability of the model inside them: the same agent on the same task can pass one run and fail the next, and each model release shifts the baseline with no change to your site. Good task-testing tools repeat each task and compare across runs. Three consequences: - **CI gates.** A gate should fail when your commit broke something. Deterministic checks do; a task test can fail on an off run. - **Regression tracking.** A static score drop after a deploy means the deploy changed something. A task pass-rate drop might be the deploy or a new model. - **Cost.** A static scan is a few HTTP requests. A task suite costs model usage per task, per agent, per repeat. ## Which tools do which? - **Static, fetch-based:** Agent Ready (site-wide, 72 checks across discovery files, page structure, llms.txt, and agent-protocol manifests) and Cloudflare's isitagentready.com (single URL, no sign-up). Comparison: - **Static, browser-based:** Lighthouse's Agentic Browsing category — one rendered page, covering WebMCP, the accessibility tree, layout stability, llms.txt, and ARD manifests. Comparison: - **Task-based:** Microsoft Scope (open source, MIT, you write the tasks and criteria), and commercial services such as Lightsage that run a panel of coding agents against your API, CLI, or MCP server in sandboxes. Third-party tool descriptions checked against each tool's own documentation on 2026-11-04. ## Should you use both? Yes, in order. Static checks are cheap, deterministic, and site-wide: run them first and keep them running — every deploy, as a CI gate, on a schedule. They clear the missing files and broken manifests that would otherwise fail every task test for boring reasons. Then run task tests on the few flows that matter most — first API call, install, checkout — before releases, and read the traces. When one fails, check the static result first: a missing Markdown mirror, unreachable manifest, or docs blocked to AI crawlers often explains it faster than the trace. Together the layers cover whether agents can find you and whether they can finish the job. ## Where does Agent Ready fit? Agent Ready is a static checker, deliberately. It runs 72 deterministic checks — 15 site-wide, 25 per-page from the Vercel Agent Readability Spec, 10 on llms.txt structure, 22 on agent-protocol manifests — plus a separate 23-check accessibility suite. Every check maps to a published methodology entry () and a plain-English fix, and it runs from the web, a REST API, a CLI, an MCP server, and a GitHub Action. It does not run AI agents against your site; pair it with a task-testing tool for outcome data on specific agents. ## Frequently asked questions ### What is the difference between a static check and an agent task test? A static check fetches your site and verifies something concrete and deterministic: does llms.txt exist and parse, does the page serve a Markdown mirror, does /.well-known/mcp.json describe a reachable MCP server. An agent task test hands a real AI agent a job, such as making a first API call or completing a checkout, and records whether it finishes. Static checks tell you which preconditions are missing. Task tests tell you whether the outcome happens. ### If my site passes a static scan, will AI agents succeed on it? Not necessarily. A static scan proves the signals agents rely on are present and well-formed; it cannot prove an agent will reason its way through your docs, choose the right endpoint, or recover from an error. That is what task tests measure. The reverse also holds: a site that fails basic static checks gives agents less to work with, so static failures are usually the first thing to fix when a task test fails. ### Why are agent task test results different every run? Because the thing under test includes a language model. The same agent on the same task can take a different path each time, and results shift again when the model or agent version updates — even if your site has not changed. Serious task-testing tools account for this by repeating each task several times and comparing the evidence across runs, which also multiplies the cost. ### Can I use agent task tests as a CI gate? You can, but they make a noisy one. A gate needs to fail when your change broke something and pass when it did not; run-to-run variance and model updates make task results fail for reasons unrelated to your commit. Deterministic static checks are the usual CI gate, with task tests run on a schedule or before a release, where a human reads the traces. ### Does Agent Ready run AI agents against my site? No. Agent Ready is a static checker by design: it runs 72 deterministic checks across discovery files, the Vercel Agent Readability Spec, llms.txt structure, and agent-protocol manifests, plus a separate 23-check accessibility suite, and it returns the same result for the same site every time. It covers the discovery and conformance layer. For whether a specific agent completes a specific task, pair it with a task-testing tool. --- Read the full guide on the web: ## Sitemap See the full [sitemap](https://agent-ready.dev/sitemap.md) for all pages on agent-ready.dev.