Skip to main content

Open in AthenaHQ

app.athenahq.ai/oracle

​
Purpose

The Oracle page lets you run AI-response accuracy checks and fact-verify your Knowledge Base against third-party sources and collected AI responses. It helps marketers ensure AI models accurately describe their brand and identify which sources or AI responses are introducing false claims. From this page, you can search across synthesized claims, review open discrepancies, resolve them by selecting the correct claim, and track your brand’s overall accuracy and hallucination rates across different AI models over time.

​
What’s on the page

Page header Shows the page title, an info tooltip explaining the feature, and an action button to run a new analysis. Search section Lets you search synthesized AI claims about your brand. Submitting a query generates an AI summary with inline citations pointing to specific responses. Metric cards Shows “Discrepancy rate” (with a sparkline trend of recent runs) and “Open discrepancies” (with a sub-count of high-impact discrepancies) at a glance. High-impact discrepancies list Displays the top 5 most severe and high-confidence unresolved discrepancies between AI responses and your Knowledge Base. Includes a link to see all open discrepancies. Coverage dashboard Contains deeper analytics charts. Shows an “Accuracy by model” chart and a “Topics AI gets wrong” chart. Both can be clicked to drill down into specific models or topics. Fact verification (Hero validation card) Surfaces the latest completed fact-verification run. It displays the contradiction count, the top contradicting third-party sources, and buttons to view sources, verify facts, or view past runs. Check responses dialog Allows you to pick a date range and start a new accuracy analysis run. Shows response counts found in the selected range and progress while the run is executing. KB Analysis Run Form dialog Lets you pick which Knowledge Base Pillars/topics to verify against third-party sources and collected AI responses, then start the run. Run History drawer A full history of all past accuracy and fact-verification runs. Includes a type filter and lets you click through to review findings from each run.
  • Run History table columns:
    • Date: When the run was created (date and time).
    • Type: A badge indicating “Response check” (an accuracy run) or “Fact verify” (a Knowledge Base validation run).
    • Responses / claims: Total responses analyzed for accuracy runs, or candidate claims checked for fact-verify runs.
    • Findings: Number of discrepancies or contradictions found in that run.
    • Discrepancy rate: Findings divided by responses as a percentage (only shown for accuracy runs).
    • Status: The run’s current status (Complete, Failed, Cancelled, or Running).
Validation Drawer Shows the ranked source list for a selected fact-verification run. Clicking a source opens the contradictions drawer.
  • KB Source Rank table columns:
    • Domain: The contradicting source’s domain, or “Knowledge base” for internal inconsistencies. Features the tooltip: “Places where your KB topics contradict each other (internal inconsistencies)”.
    • Contradictions: Count of contradictions attributed to that domain.
KB Source Contradictions drawer Steps through every contradiction (or internal consistency issue) for a selected source, one at a time, providing Previous and Next navigation buttons. Oracle Review Modal A full discrepancy review workflow. Lets you browse a paginated list of discrepancies, view both claims side-by-side along with their sources and citations, see AI research verdicts (and confidence scores), and resolve each discrepancy with optional notes. Responses Drawer A shared slide-out panel that shows full AI response details (including the model, sentiment, sources, and competitors mentioned) when you click a “View response” link from a discrepancy or search result. Run progress overlays Persistent floating progress indicators positioned at the bottom-right of the screen for in-flight accuracy or fact-verification runs. They include cancel and hide controls and remain visible as you navigate the page.

​
What you can do here

  • Run analysis (dropdown): Opens a dropdown in the page header with options for “Check responses” (starts an accuracy analysis run) and “Verify facts” (opens the fact-verification topic picker).
  • Check responses: Opens the dialog to select a date range and start a new accuracy run.
  • Verify facts: Opens the topic picker dialog to start a fact-verification run against selected Pillars.
  • Run first analysis: If no runs exist, click this in the empty state to open the Check responses dialog and start your first run.
  • Start (in Check responses dialog): Begins the accuracy analysis over the chosen date window.
  • Cancel (in Check responses dialog): Closes the dialog without starting a run.
  • Dismiss (while running): Hides the run dialog while the run continues processing in the background.
  • Review discrepancies (post-run dialog): After a run completes, this closes the dialog and opens the Oracle Review Modal filtered to unresolved discrepancies.
  • Cancel run (progress overlay): Immediately cancels an in-progress accuracy or fact-verification run from the floating progress indicator (no confirmation shown).
  • Hide progress (overlay close icon): Hides the floating run-progress overlay from your screen without cancelling the run itself.
  • Search input: Type a query and press Enter to search synthesized claims and generate a summarized response with citations.
  • Citation pill / Response link (search summary): Click an inline citation pill to open the Responses drawer for that AI response, scrolled and highlighted to the relevant excerpt.
  • See all (high-impact discrepancies): Opens the Oracle Review Modal to show all unresolved discrepancies.
  • Open discrepancy row: Clicking a row in the high-impact list opens the Oracle Review Modal focused specifically on that discrepancy.
  • Coverage tabs (Overview / Gap analysis): Switches the lower dashboard view between metrics/charts and gap analysis.
  • Model accuracy chart bar/segment: Click a model’s bar to drill into its accuracy trend.
  • Hallucinated categories chart bar: Click a topic category bar to view a list of wrong claims for that topic, which can then lead to specific discrepancy details.
  • Count vs % wrong toggle: Switches the hallucinated categories chart metric between raw wrong-claim counts and the percentage of wrong claims.
  • Verify facts / Configure run (hero card): Opens the KB Analysis Run Form to select topics.
  • View sources (hero validation card): Opens the Validation Drawer to see sources ranked by their contradiction count.
  • View past runs (hero validation card link): Opens the Run History drawer.
  • Select a source domain row: Inside the Validation Drawer, click a domain to open the KB Source Contradictions drawer.
  • Previous / Next (contradictions drawer): Steps through individual contradictions or consistency issues within a specific source domain.
  • View response (discrepancy/contradiction card): Opens the Responses drawer to read the full AI response containing the flagged claim.
  • Open site (external link icon): Opens the source domain in a new browser tab.
  • Start verification (KB Analysis Run Form): Kicks off a fact-verification run scoped only to the Pillars/topics you selected.
  • Cancel (KB Analysis Run Form): Closes the topic picker without starting the run.
  • Review them / Review (pending-only pillars link): Navigates to the Knowledge Base review page to approve unapproved claims.
  • Knowledge Base tile (empty validation state): Takes you to your Knowledge Base.
  • Select A/B claim (Review Modal): Click a claim side to mark it as correct.
  • Edit (locked resolved discrepancy): Unlocks a previously dismissed or resolved (e.g., both valid) discrepancy card so you can change its resolution.
  • Add a note (Review Modal): Expands a text area where you can attach a note to explain your resolution decision.
  • Resolve discrepancy (implicit via selection + navigation): Your resolution choice (claim A, claim B, both valid, or dismissed) along with notes are saved automatically as you navigate or close the modal.
  • Filter dropdown (Review Modal): Change the dropdown to view unresolved, resolved, or dismissed discrepancies.
  • Type filter dropdown (Run History): Filter the Run History list by “All runs”, “Response check”, or “Fact verify”.
  • Click a run row (Run History): Opens the Review Modal (for accuracy runs) or Validation Drawer (for fact-verify runs) associated with that past run.

​
Data shown

  • Accuracy analysis runs: The results of AI responses being fact-checked against your Knowledge Base.
  • Fact-verification runs: The results of your Knowledge Base claims being checked against third-party sources and collected AI responses.
  • Your tracked Knowledge Base claims: The topics and Pillars you have configured in your Knowledge Base.
  • Collected AI responses: Responses gathered for your brand across various tracked models (e.g., ChatGPT, Claude, Gemini, Perplexity, Copilot, Grok, DeepSeek).

​
Common workflows

Run a fresh accuracy check
  1. Click the “Run analysis” dropdown in the top right and select “Check responses” (or click “Run first analysis” if you have no runs yet).
  2. Select a date range to determine the response window.
  3. Review the number of responses found (if it exceeds the cap, the system notes how many will be sampled).
  4. Click “Start”.
  5. Track the progress via the floating overlay or the dialog’s progress bar.
  6. When complete, view the discrepancy count in the dialog and optionally click “Review discrepancies” to jump straight into resolving them.
Verify Knowledge Base facts against external sources
  1. Click the “Run analysis” dropdown and select “Verify facts” (or click “Verify facts” on the validation hero card).
  2. In the picker dialog, select one or more ready Pillars/topics you want to check.
  3. Click “Start verification”.
  4. Track the run’s progress using the floating overlay in the bottom right.
  5. Once complete, view the contradiction count and top sources on the hero card, or open the Validation Drawer to see the fully ranked source list.
Review and resolve a discrepancy
  1. Open a discrepancy from the high-impact list, search results, or the Run History drawer.
  2. In the Oracle Review Modal, read Claim A and Claim B side-by-side, along with their sources, citations, and the AI research verdict (and confidence score) if present.
  3. Select which claim is correct, mark both as valid, or dismiss the discrepancy.
  4. Optionally, click “Add a note” to explain your reasoning for your team.
  5. The resolution saves automatically as you navigate to the next discrepancy or close the modal.
Investigate a contradicting source from a fact-verification run
  1. Open the hero validation card or open the Run History drawer to select a completed fact-verification run.
  2. In the Validation Drawer, view the list of sources ranked by the number of contradictions they generated.
  3. Click a source domain row to open the KB Source Contradictions drawer.
  4. Step through each contradiction using the Previous and Next buttons. You can optionally click links to open the underlying AI response or visit the source site directly.

​
Empty, loading, and error states

  • Empty: If the website has never run Oracle, you will see a “No analysis run yet” state with a “Run first analysis” button. If there are no open issues, the high-impact list displays “No unresolved discrepancies. Run an analysis to find new ones.” The Verify Facts topic picker shows “No pillars yet. Add one in Knowledge Base first.” if your KB is empty. The Validation Drawer shows “No verification run selected.” when empty, and the contradictions drawer shows “No contradictions found for this source.” or “No internal KB inconsistencies found.” The Run History table displays “No runs yet.” if history is empty.
  • Loading: While the latest run status loads, skeleton cards mirroring the Overview layout appear. If a first run is executing, a “Running your first analysis” state appears with a spinner. You will also see spinning loading indicators in dialogs and drawers while fetching contradictions, run details, or claims data.
  • Error: Red toast notifications appear for failed actions, like “Failed to start analysis. Please try again.” or “Failed to cancel run. Please try again.” If drawers fail to load data, an inline error reads “Couldn’t load contradictions/consistency issues. Try again.” If there is a billing or plan verification issue, a full-page “Couldn’t verify access” error screen appears.
  • Linked from: The main sidebar navigation (Oracle nav item), Knowledge Base flows that deep-link to the page (e.g., after reviewing a KB analysis), and legacy bookmarks for older tabs.
  • Links to: The Knowledge Base page, the Knowledge Base review page, and the Content page (for legacy drafts and remediation previews).

​
Common support questions

How do I see what AI is getting wrong? You can view the “Topics AI gets wrong” chart in the Coverage section, or review the high-impact discrepancies list on the Overview tab. To see all issues, click “See all” to open the Oracle Review Modal and browse the full list. Why can’t I select my topics when I try to verify facts? The Knowledge Base analysis run form only allows you to select “ready” topics that contain approved claims. If your topics only contain unreviewed or pending claims, they will appear in a notice with a link to review them, but they cannot be selected until approved. What’s the difference between “Check responses” and “Verify facts”? “Check responses” runs an accuracy analysis, fact-checking recent AI responses against your Knowledge Base. “Verify facts” runs a Knowledge Base validation, checking your approved KB claims against third-party sources and collected AI responses to ensure your own facts are correct and internally consistent. How do I correct a discrepancy? Open the discrepancy in the Oracle Review Modal. Compare Claim A and Claim B, then click the correct side, or mark them both as valid or dismiss the issue. Your choice is saved automatically and updates the system’s “known bad” claims list to improve future tracking. What does the confidence score mean? The confidence score is an AI research verdict indicating how certain the system is about which claim is correct. Higher scores mean strong evidence was found supporting that side.