What is the difference between a static check and an agent task test?

A static check fetches your site and verifies something concrete and deterministic: does llms.txt exist and parse, does the page serve a Markdown mirror, does /.well-known/mcp.json describe a reachable MCP server. An agent task test hands a real AI agent a job, such as making a first API call or completing a checkout, and records whether it finishes. Static checks tell you which preconditions are missing. Task tests tell you whether the outcome happens.

If my site passes a static scan, will AI agents succeed on it?

Not necessarily. A static scan proves the signals agents rely on are present and well-formed; it cannot prove an agent will reason its way through your docs, choose the right endpoint, or recover from an error. That is what task tests measure. The reverse also holds: a site that fails basic static checks gives agents less to work with, so static failures are usually the first thing to fix when a task test fails.

Why are agent task test results different every run?

Because the thing under test includes a language model. The same agent on the same task can take a different path each time, and results shift again when the model or agent version updates — even if your site has not changed. Serious task-testing tools account for this by repeating each task several times and comparing the evidence across runs, which also multiplies the cost.

Can I use agent task tests as a CI gate?

You can, but they make a noisy one. A gate needs to fail when your change broke something and pass when it did not; run-to-run variance and model updates make task results fail for reasons unrelated to your commit. Deterministic static checks are the usual CI gate, with task tests run on a schedule or before a release, where a human reads the traces.

Does Agent Ready run AI agents against my site?

No. Agent Ready is a static checker by design: it runs 72 deterministic checks across discovery files, the Vercel Agent Readability Spec, llms.txt structure, and agent-protocol manifests, plus a separate 23-check accessibility suite, and it returns the same result for the same site every time. It covers the discovery and conformance layer. For whether a specific agent completes a specific task, pair it with a task-testing tool.

Static Checks vs Agent Task Tests

At a glance

Criterion	Static checks	Agent task tests
What runs	Fetches and parsers, sometimes a headless browser	An AI agent (LLM) attempting a task
Question answered	Are the signals agents rely on present and correct?	Did this agent finish this task?
Repeatability	Same site, same result	Varies run to run; needs repeated runs
When it fails	Names the file or rule, and the fix	A trace to read and diagnose
Setup	A URL	Tasks and success criteria you define
Coverage	Every page and manifest it can reach	The tasks you wrote
Discovery	Tested directly	Usually skipped — the agent is given the endpoint
Changes when	Your site changes	Your site or the model changes
Cost per run	A few HTTP requests	Model usage × agents × repeats
Good CI gate?	Yes	Noisy; better on a schedule
Examples	Agent Ready, isitagentready.com, Lighthouse Agentic Browsing	Microsoft Scope, commercial agent-evaluation services

What is a static check?

A static check verifies something concrete about your site without asking an AI model to do anything. It fetches a URL, parses what comes back, and compares it with a rule: llms.txt exists and has an H1, the page serves a Markdown version when asked with Accept: text/markdown, robots.txt does not block the AI crawlers you meant to allow, /.well-known/mcp.json describes an MCP server that actually answers.

Because there is no model in the loop, the result is deterministic: the same site produces the same result on every run, and a failure points at a specific file and a specific fix. Some static tools drive a headless browser — Lighthouse’s Agentic Browsing category reads the rendered accessibility tree and measures layout shift — but they are still deterministic audits, not agents.

What is an agent task test?

An agent task test gives a real AI agent — a coding agent such as Claude Code or GitHub Copilot, or a browsing agent — a job to do against your product, then records what happened. The task might be “make a first authenticated API call”, “install the SDK and list resources”, or “find the price of the Pro plan”. You define what success looks like, and the tool reports pass or fail with a trace of what the agent tried.

This measures the outcome directly, which is its great strength: no checklist can tell you whether an agent will actually reason its way through your docs. Microsoft’s open-source Scope is a good example of the shape: you give agents the same task, define reusable success criteria, and compare the evidence over repeated runs.

Why doesn’t a clean static scan guarantee agents succeed?

Because the signals being present is necessary, not sufficient. A static check can confirm your llms.txt is well-formed; it cannot confirm the pages it links to explain your API clearly enough for an agent to call it correctly. It can confirm an MCP server card at /.well-known/mcp.json points at a live server; it cannot confirm the tool descriptions lead an agent to pick the right tool. Anything that depends on an agent’s judgement — reading, choosing, recovering from an error — is only observable by running one. That gap is real, and it is exactly what task tests exist to close.

Why doesn’t a passed task test mean agents can find you?

Because most task tests start after discovery. To run a task against your API, CLI, or MCP server, the test harness usually hands the agent the endpoint, the package name, or the docs URL. That is the right design for measuring whether the agent can complete the task — but it means the test never asks whether an agent working on its own would have found the endpoint in the first place.

Discovery is precisely what static checks measure: whether your llms.txt points at the right docs, whether your MCP server, agent card, or API catalog is advertised at its well-known path, whether your pages can be read without running JavaScript. A product that passes every task test when given its own URL can still be invisible to an agent that has to find it.

Why does determinism matter?

Agent task tests inherit the variability of the model inside them. The same agent on the same task can succeed on one run and fail on the next, and every model or agent release shifts the baseline again — with no change to your site. That is not a flaw in any particular tool; it is why good task-testing tools repeat each task and compare across runs.

It has three practical consequences:

CI gates. A gate should fail when your commit broke something. Deterministic checks do that; a task test can fail because the model had an off run.
Regression tracking. A static score that drops after a deploy means the deploy changed something. A task pass rate that drops might mean the deploy, or a new model version.
Cost. A static scan is a handful of HTTP requests. A task suite costs model usage for every task, every agent, and every repeat, which shapes how often you can afford to run it.

Which tools do which?

Static, fetch-based: Agent Ready (site-wide, 72 checks across discovery files, page structure, llms.txt, and agent-protocol manifests) and Cloudflare’s isitagentready.com (single URL, no sign-up). See how the two compare.
Static, browser-based: Lighthouse’s Agentic Browsing category, which audits one rendered page for WebMCP, the accessibility tree, layout stability, llms.txt, and ARD manifests. See Lighthouse vs Agent Ready.
Task-based: Microsoft Scope (open source, MIT, you write the tasks and criteria), and commercial services such as Lightsage that run a panel of coding agents against your API, CLI, or MCP server in sandboxes.

Third-party tool descriptions checked against each tool’s own documentation on 2026-11-04. These products change quickly — check their current docs before relying on a detail.

Should you use both?

Yes, and in a particular order. Static checks are cheap, deterministic, and cover the whole site, so run them first and keep them running: on every deploy, as a CI gate, and on a schedule. They clear away the missing files and broken manifests that would otherwise make every task test fail for boring reasons.

Then run task tests on the handful of flows that matter most — the first API call, the install, the checkout — before releases or on a slower schedule, and read the traces. When a task test fails, look at the static result first: a missing Markdown mirror, an unreachable manifest, or docs blocked to AI crawlers often explains the failure faster than the trace does. Neither layer is a substitute for the other; together they cover whether agents can find you and whether they can finish the job.

Where does Agent Ready fit?

Agent Ready is a static checker, and deliberately so. It runs 72 deterministic checks — 15 site-wide, 25 per-page from the Vercel Agent Readability Spec, 10 on llms.txt structure, and 22 on agent-protocol manifests — plus a separate 23-check accessibility suite. Every check maps to a published methodology entry and a plain-English fix, and it runs from the web, a REST API, a CLI, an MCP server, and a GitHub Action for CI. It does not run AI agents against your site. It covers the discovery and conformance layer, so pair it with a task-testing tool if you need outcome data for specific agents.

Frequently asked questions

What is the difference between a static check and an agent task test?: A static check fetches your site and verifies something concrete and deterministic: does llms.txt exist and parse, does the page serve a Markdown mirror, does /.well-known/mcp.json describe a reachable MCP server. An agent task test hands a real AI agent a job, such as making a first API call or completing a checkout, and records whether it finishes. Static checks tell you which preconditions are missing. Task tests tell you whether the outcome happens.
If my site passes a static scan, will AI agents succeed on it?: Not necessarily. A static scan proves the signals agents rely on are present and well-formed; it cannot prove an agent will reason its way through your docs, choose the right endpoint, or recover from an error. That is what task tests measure. The reverse also holds: a site that fails basic static checks gives agents less to work with, so static failures are usually the first thing to fix when a task test fails.
Why are agent task test results different every run?: Because the thing under test includes a language model. The same agent on the same task can take a different path each time, and results shift again when the model or agent version updates — even if your site has not changed. Serious task-testing tools account for this by repeating each task several times and comparing the evidence across runs, which also multiplies the cost.
Can I use agent task tests as a CI gate?: You can, but they make a noisy one. A gate needs to fail when your change broke something and pass when it did not; run-to-run variance and model updates make task results fail for reasons unrelated to your commit. Deterministic static checks are the usual CI gate, with task tests run on a schedule or before a release, where a human reads the traces.
Does Agent Ready run AI agents against my site?: No. Agent Ready is a static checker by design: it runs 72 deterministic checks across discovery files, the Vercel Agent Readability Spec, llms.txt structure, and agent-protocol manifests, plus a separate 23-check accessibility suite, and it returns the same result for the same site every time. It covers the discovery and conformance layer. For whether a specific agent completes a specific task, pair it with a task-testing tool.