Case study · AskOrac

An AI agent that knows when to say it doesn't know.

AskOrac trains an AI agent on a business's website and documents, answers visitors with citations, and brings in a person when its confidence drops.

Visit askorac.com
Challenge
Support chatbots guess when they aren't sure, and visitors often have no route to a person.
What we built
An agent trained on a business's own content that scores its confidence on every answer.
Result
Live with a free plan, answering with sources and handing uncertain questions to a person.
askorac.com
AskOrac homepage with the headline AI agents trained on your own docs and a live chat panel
Type
AI support platform
Status
Live, free plan open
In development since
April 2026
Platform
Web app and embeddable widget
Stack
Next.js, Supabase, pgvector

3

parallel retrieval passes: vector, keyword and entity

4

weighted signals in every confidence score

24 h

sync cycle, re-processing only content that changed

105

API routes behind the platform

Taken from the AskOrac codebase and askorac.com, in October 2026.

01

The problem

Support teams answer the same questions all day, and the answers already exist somewhere on their own site. A chatbot trained on that content looks like the obvious fix, and for a simple question it is.

The failure comes with harder questions. Given weak context, a language model doesn't stop. It writes a fluent answer from whatever it found, and the visitor can't tell. When that happens there is often no route to a person, and per-resolution pricing makes the monthly bill hard to predict.

AskOrac starts from a different question: how sure is the system of this answer, and what should happen when it isn't?

02

Who it's for

AskOrac is for small and growing businesses with real content and no appetite for a long integration project.

E-commerce stores

Product and policy questions answered from the store's own pages, with the current page given priority.

SaaS and in-app support

Help-centre answers on marketing pages and inside the product, with context passed in from the page.

Publishers and documentation teams

Large content libraries that stay current through daily sync and feed polling.

03

What we built

What happens to every questionInterface illustration
Visitor asks
VectorKeywordEntity

Confidence

Answers with sourcesEnough evidence to reply.
Brings in a personWeak evidence, so Orac doesn't guess.

Crawl and sync

Sitemap discovery, PDF and document ingestion, RSS polling, and daily change detection by content hash.

Grounded answers

Cited responses from hybrid retrieval, with re-ranking that weighs the visitor's current page and the conversation so far.

Confidence-based routing

A score on every answer decides whether Orac replies, falls back or brings in a person.

Human handoff

A live conversation inbox with teams, presence, canned responses, internal notes and SLA tracking.

Intelligence

Knowledge gaps found from real questions, with draft answers an owner can review and save as verified knowledge.

Actions and leads

Custom actions, lead capture and outbound webhooks that connect conversations to other systems.

Multi-model and BYOK

OpenAI, Anthropic and Google models, with bring-your-own-key for unlimited messages on paid plans.

One-tag widget

A single script tag that runs on any site, isolated in an iframe from the host page.

AskOrac crawler panel listing sitemap, pages, chunks and the last sync
Sitemap crawl and delta sync, from askorac.com. Figures shown are illustrative.
Bring-your-own-key panel with OpenAI, Anthropic and Google providers
Bring your own provider key. Keys are encrypted with AES-256-GCM.

04

Architecture

A single Next.js application serves the dashboard, the widget and the chat API. Postgres holds tenant data, documents and vectors together, so a query can filter by project without crossing systems.

Surfaces

  • WidgetScript tag, iframe, pinned postMessage origin
  • DashboardInbox, knowledge, analytics, billing

Conversation engine

  • PreflightLimits, project, BYOK, escalation
  • Channel adaptersThe widget today, more on the same contract

Retrieval

  • Query rewriteResolves follow-ups in context
  • Hybrid searchVector, keyword and entity in parallel
  • Re-rank and scoreMulti-signal ranking, then confidence

Data and background

  • Postgres with pgvectorHNSW vectors and text search together
  • Scheduled jobsCrawl, sync, RSS, SLA evaluation
  • Model providersOpenAI, Anthropic, Google

05

Key engineering decisions

1

Keep vectors in Postgres

Why

Chunks, documents and tenants share one database, so isolation, joins and backups stay simple, with no second system to keep in step.

Trade-off

Filtered vector search needs careful tuning at scale, as the first hard problem below shows.

2

Prefer "I don't have that" to a guess

Why

Strict mode raises the retrieval threshold and adds a confidence gate. Below it, the project's fallback message goes out in place of an improvised answer.

Trade-off

Coverage drops. For technical and product content, accuracy is worth more.

3

Score confidence from retrieval, not from the model

Why

The score combines similarity, chunk availability, keyword coverage and source diversity. Those can be measured and explained, and they drive routing and gap detection.

Trade-off

A strong retrieval score doesn't prove the answer is right, so citations and review still matter.

4

Pull decisions out of the chat route

Why

Limits, lifecycle and escalation moved into a channel-agnostic engine that returns a typed outcome. New channels become adapters instead of rebuilds.

Trade-off

We stopped at the first phase on purpose. Rewriting the streaming path too would have put the core revenue path at risk.

5

Telemetry never touches the response

Why

Each chat turn records per-stage latency, and the write is fire-and-forget. That gave us real traces without slowing a single reply.

Trade-off

A failed write loses a trace. Empty stages are kept as meaningful, so analysis has to handle them.

06

The hardest problems

Filtered vector search at multi-tenant scale

Problem

Production traces showed vector search at a 2,087 ms median and 4,073 ms at p95, with wide swings between adjacent requests. The HNSW index is global, so a project filter either dropped most results or pushed Postgres into a sequential scan.

What we did

We enabled pgvector's iterative scan so the graph search keeps expanding until enough results survive the filter. We used relaxed ordering, because the re-ranker re-sorts every result anyway and strict ordering would have bought nothing.

Result

Retrieval stopped being the bottleneck. Later tracing put most of the roughly 3.5-second median in the model call.

07

What shipped

Live today

  • Free, Pro and Business plans with public sign-up
  • Site crawl, document ingestion and 24-hour sync
  • Cited answers with hybrid retrieval and confidence scoring
  • Human handoff with teams, SLAs and canned responses
  • Knowledge-gap detection with draft answers
  • Custom actions, lead capture and webhooks
  • Multi-provider models and bring-your-own-key

Still to come

  • More channels on the conversation engine, starting with those customers ask for
  • A voice channel, currently in design
Next case studyShelf Inn

Let's build.

Tell us about your project. We respond within 24 hours with a detailed plan of attack.

Start a Project