Skip to main content
The AI Access API checks robots.txt policy and sends HTTP requests using AI crawler user-agents from Athena’s network. Use the evidence to investigate request failures or share diagnostics with your host or CDN. These probes do not verify access from real crawler IP ranges or guarantee indexing or citations.

​
What you get back

  • result.evidence contains the authoritative per-agent observations. http records true for a passed probe, false for a denied probe and null for unknown access. robots reports policy independently, or is null when the file could not be evaluated. network is athena and version is 2.
  • diagnostics records each request’s HTTP status, verdict, response headers, error code, duration and UTC timestamp. Use these fingerprints to correlate requests in your server or CDN logs. Policy-only tokens use not_tested with no HTTP status or request error.
  • log formats the request diagnostics for a support ticket. Its overall status uses result.evidence.status to distinguish unknown access from a block.
The older result.status, result.http and result.robots fields remain for compatibility. They combine crawler roles and cannot distinguish unknown access. When Google-Extended has no HTTP observation, the legacy result.http.gemini.training boolean uses the measured Googlebot response as a compatibility proxy. It does not establish Gemini training access or permission. Google-Extended’s robots.txt policy stays independent, and its raw HTTP evidence remains null, as Google-Extended has no dedicated HTTP user-agent. New integrations should use result.evidence and per-agent diagnostics. Sitemaps are not checked by this probe.

​
Separate search, retrieval and training

​
Evidence status values

result.evidence.status summarizes HTTP observations, independently of robots.txt. Google-Extended is excluded because it has no HTTP crawler. A challenge can return HTTP 200. Challenges, rate limits, network failures, timeouts and ordinary server errors mean unknown access. A denied request is also evidence about Athena’s probe, rather than proof that the real crawler is blocked. Confirm actual crawler traffic using your server or CDN logs.

​
Rate limits and authentication

Use standard AthenaHQ API key authentication. The endpoint allows 30 requests per minute per organization. Exceeding the limit returns 429 with Retry-After.

​
Endpoint reference

See POST /api/v1/ai-access/check for the full request and response schema.