Skip to main content

Open in AthenaHQ

app.athenahq.ai/site-health/properties

​
Purpose

A crawl scope defines the specific part of your website that Site Health crawls to discover pages for auditing. Customers use this page to create, edit, run, and track these scopes. While the standard Site Health report automatically samples pages, a crawl scope allows you to list every single page the crawler finds. This ensures you can comprehensively audit, assign, and track your site’s health across all intended pages.

​
What’s on the page

  • Page header: Displays the page title (“Crawl scope”), a brief description of its purpose, and actions to return to the main overview or create a new scope.
  • Crawl scope cards: A list of cards, each representing a configured crawl scope.
    • Header and status: Shows the scope’s name, its root URL, a summary of its rules (e.g., “example.com and 2 subdomains”), and a status badge indicating the crawl’s current state (e.g., Disabled, Auditing, Crawling, Complete, Partial, Failed, Cancelled).
    • Pages found: The number of live pages currently stored in your inventory for this scope.
    • Last crawl: The date and time the most recent crawl was started.
    • Last run: A summary of the last crawl’s results, detailing how many pages were found, skipped, blocked, or failed.
    • Error messages: If a crawl failed, an alert message explains the error.
  • Add/Edit crawl scope dialog: Opens when adding or editing a scope.
    • Name: The display name for your scope.
    • Site URL: The root website URL. This cannot be changed once the scope is saved and pages are stored under it.
    • Advanced crawl settings: A collapsible section with fine-grained rules.
      • Subdomains: Choose to crawl “Root host only”, “Some subdomains” (pick from a checklist of up to 200 known subdomains), or “All subdomains”.
      • Sitemaps: Choose “Every sitemap” or restrict the crawl to “Some sitemaps” (pick from a checklist of up to 200 declared files).
      • Only crawl under: A text area to define specific path prefixes to include (one per line).
      • Never crawl under: A text area to define path prefixes to exclude. This overrides the include list.
      • Page limit: A maximum cap on pages discovered (up to 250,000).

​
What you can do here

  • Navigate back: Click Back to Site Health to return to the main overview page.
  • Create a scope: Click Add crawl scope to open the configuration dialog and define a new area of your site to crawl. Click Save and continue to finalize it.
  • Edit a scope: Click Edit scope on a card to adjust its name or advanced settings. Click Save scope when finished.
  • Start a crawl: Click Run crawl on a scope’s card. This triggers a discovery crawl to find pages based on the scope’s rules. A success toast (“Crawl started for [Name]”) appears.
  • Cancel a crawl: While a crawl is active, the run button becomes Cancel run. Clicking it opens a confirmation dialog (“Cancel this crawl?”). Confirming (“Cancel crawl”) marks the run as cancelled, but pages found so far remain in the inventory. Choosing “Keep crawling” dismisses the prompt.
  • View detailed findings: Click Pages and findings (or Open scope if no pages have been found yet) to navigate to a detailed view of the pages and audit findings associated with that scope.

​
Data shown

The page displays the crawl scopes you have configured for your active website, alongside the latest results and page counts from their automated crawls. When configuring a scope’s advanced settings, the lists of candidate subdomains and sitemaps are sourced directly from what Athena has previously discovered on your site (e.g., from your robots.txt file, redirects, and earlier crawls).

​
Common workflows

Creating a new crawl scope:
  1. Click Add crawl scope in the top right corner.
  2. Enter a Name and the Site URL.
  3. (Optional) Expand Advanced crawl settings to restrict the crawler to specific subdomains, sitemaps, or path prefixes.
  4. Click Save and continue.
Running a discovery crawl:
  1. Locate the scope card you want to update on the main page.
  2. Click Run crawl. The card’s status badge will update to show the crawl is running.
  3. Wait for the crawl to finish, or click Cancel run if you need to stop it early.
  4. Once finished, click Pages and findings to review the discovered pages.
Narrowing an existing scope:
  1. Click Edit scope on an existing card.
  2. Expand Advanced crawl settings.
  3. Add specific paths to the Never crawl under text area, or uncheck subdomains/sitemaps.
  4. Click Save scope. (Note: The new restrictions apply to the next crawl. Pages previously found outside this new scope will remain until a fresh crawl retires them.)

​
Empty, loading, and error states

  • Empty state: If you haven’t created any scopes, you will see a “No crawl scope yet” message explaining that adding one allows you to track every page the crawler finds. It includes an Add crawl scope button.
  • Disabled feature: If your account doesn’t have Site Health enabled, you see “Site Health is not available here”.
  • No website selected: If you haven’t selected an active website in your workspace, the page prompts you to “Select a website”.
  • Loading state: While loading, the page displays skeleton placeholder blocks indicating where the cards will appear.
  • Error state: If the scopes fail to load, a red callout titled “Crawl scopes could not be loaded” appears with the error details and a Try again button to retry the fetch.
  • Linked from: The main Site Health overview page, and the Site Health header’s options menu (the three-dot icon).
  • Links to: The main Site Health overview (via the “Back to Site Health” button), and the individual property detail pages detailing pages and findings.

​
Common support questions

Why can’t I edit the Site URL of my scope? The root Site URL acts as the core identity of the scope. Once you save the scope and it begins storing pages under that URL, it becomes permanently fixed to prevent data mismatches. If you need a different root URL, you must create a new scope. I excluded a path in the settings, but those pages are still showing up. Why? Changes to your scope’s advanced settings (like excluding paths or subdomains) only apply to the next crawl you run. Pages found by earlier, broader crawls stay in your inventory until a new crawl runs and officially retires them. I clicked Cancel run, but it said the crawl had already finished? If you click Cancel run right as the system completes the crawl in the background, Athena will notify you (“That crawl had already finished”) rather than showing a cancellation success message. Why are the action buttons disabled? Adding, editing, running, or cancelling a crawl scope requires “Write” access for analytics. If you only have view access, these buttons will appear disabled. You can hover over a disabled button to see a tooltip explaining the required permission. The “Run crawl” button is also disabled while a crawl is actively running.