Fusion Intel
Building a Knowledge Engine from First Principles
Connor England
Please do interrupt with questions!
About Me
- Current: Applied AI Developer & Solutions Architect, Fusion Health
- Before: Co-founded SmartSave (fintech, USTE processing). Built Pidgeon Health (open-source C# healthcare data platform, used at Fusion and beyond)
- Early Career: QA Lead, 3 food manufacturing plants (HACCP/SQF)
- Certs: AWS Solutions Architect, CompTIA Security+, DeepLearning.AI Data Engineering
- Flight: Lifelong love for aviation and aerospace. Currently working on PPL/IR.
What We'll Cover
This talk is about
- Fusion Intel: multi-domain RAG platform (designed + 80% coded by me)
- Modular AI development that absorbs improvements fast
- Lessons learned, applied to Boltline
This talk will
- Derive requirements from domain constraints
- Walk through architecture and implementation
- Share strong opinions, loosely held
- Get into the nitty gritty!
Overview
The Problem, The Principles, The Constraints
What Is Fusion Intel?
Accurate, cited answers from our own documentation, powered by AI.
The problem with general AI
General AI guesses. Dangerous in our domain.
What RAG changes
Search our docs first. Every claim traceable to source.
Why Build This?
First: RFP Bottleneck
- RFPs are the bloodflow of growth
- 40-80 hrs per RFP
- Clear first safe use case
Next: Cross-Domain Intelligence Multiplier
- Same knowledge bases serve all departments
- Build once, scale from first principles
Build vs. Buy
Rovo (Atlassian's AI) tested and rejected. Couldn't enforce citations, hallucinated frequently.
Team Usability
Technically sound but usable for BD and comms professionals, not just engineers.
Compliance
Teams already using ChatGPT internally with zero guardrails. Hallucinations going into client-facing materials.
Constraints & Opportunities
Domain Constraints
- Multi-department, one platform
- HIPAA: full data control, no RAG-in-a-box
- Docs scattered across Confluence + OneDrive
- Cross-domain sharing with clear boundaries
Technical Constraints
- RAG degrades at scale (46K pages, 131K tickets)
- Vector search misses exact terms
- LLMs are sycophantic: guess over refuse
- AI moves fast: every component swappable
How It Works
Core Mechanics
A wrong RFP claim can cost millions in failed contracts. If we can't cite the source, we can't use the answer.
Tradeoff: helpfulness vs. honesty
What we know so far
Functional
- Citation-backed responses
- "Insufficient Evidence" as a first-class output
- Multi-pass retrieval with fallback advisory
- Source authority tiers (Gold 3x, Silver 2x, Ref 1x)
Non-Functional
- 100% citation presence on answered queries
- <1% unsupported claims
The RAG Flow
System Architecture
collection='RFP', content_type_filter=['manual','release_notes']. The model never sees Legal or Support content.
Users search by meaning and by exact terms (release versions, compliance codes, prescription identifiers). The retrieval layer needs to handle both well.
Tradeoff: precision vs. cost
Hybrid Search
Neither search alone is enough. Hybrid + cross-encoder reranking: +15-20% precision.
Model Strategy
classify/route
generate
check
Haiku: Classifier ~$0.01/q
Routes queries to domain, decomposes compound questions. Structured JSON output.
Cohere: Reranker ~$0.003/q
Cross-encoder rescores top 15 → top 5. +15-20% precision. Evaluated in the eval harness.
Sonnet: Generator ~$0.15/q
Generates cited responses from reranked chunks. Every claim traces to source.
Sometimes the evidence just isn't there. That's signal, not a dead end, as long as the system tells the truth instead of guessing.
Tradeoff: honesty vs. sycophancy
Confidence Scoring
HIGH
Gold sources, high relevance
MEDIUM
Partial match, verify
INSUFFICIENT
No evidence, system refuses
Teaching the System to Refuse
- LLMs guess to be helpful. Harness blocks generation below threshold.
- Every refusal logged with search context.
with citations
evidence logged
prioritized backlog
The list grows
Functional
- Citation-backed responses
- "Insufficient Evidence"
- Multi-pass retrieval
- Source authority tiers
- Hybrid search (vector + BM25 + rerank)
- Query decomposition
- Circuit breaker
- Confidence scoring (rule-based)
- Configurable model router
Non-Functional
- 100% citation presence
- <1% unsupported claims
- P95 latency <12s
- Monthly cost <$800
Data Pipeline & Quality
Ingestion, Scale, and Optimization
46K Confluence pages and 131K Jira tickets. The system has to stay current without drowning in noise.
Tradeoff: volume vs. quality
The Ingestion Pipeline
Confluence page created/updated triggers an Atlassian webhook. Event-driven means zero polling overhead.
Service Bus absorbs bursts, retries failures, dead-letters poison events. Decouples ingestion from source.
Labels map to taxonomy axes. Determines which Weaviate class(es) receive the content. Removals trigger soft-delete.
Canonical version stored in Blob Storage. Vector DB is derived, never authoritative. Always rebuildable.
Semantic chunking: heading boundaries first, then paragraphs, then sentences. 300–500 tokens with 50-token overlap.
Content-hash deduplication ensures idempotency. Embedding model generates vectors for each chunk.
Vectors land in the correct Weaviate class with full metadata: source ID, heading path, authority tier, timestamps.
The Laffer Curve of RAG
Recall drops from 99% at 10K vectors to 85% at 10M‡. Class isolation keeps us in the safe zone.
- "Ingest everything" = noise wins
- Our defense: 2K-5K vectors per query, not 600K+
† HNSW (Hierarchical Navigable Small World) is the approximate nearest-neighbor algorithm most vector databases use to find similar chunks quickly.
‡ Sarkar, "HNSW at Scale: Why Your RAG System Gets Worse as the Vector Database Grows," Towards Data Science, Jan 2026
Document Curation & Class Isolation
One collection per domain. Tags route content in. The harness filters by content_type at query time.
Growing stronger
Functional
- Citation-backed responses
- "Insufficient Evidence"
- Multi-pass retrieval
- Source authority tiers
- Hybrid search (vector + BM25 + rerank)
- Query decomposition
- Circuit breaker
- Confidence scoring (rule-based)
- Configurable model router
- Idempotent ingestion
- Golden answer cache for recurring queries
- Domain-isolated vector classes
Non-Functional
- 100% citation presence
- <1% unsupported claims
- P95 latency <12s
- Monthly cost <$800
- Recurring queries served from verified cache
Proving It Works
Evaluation & Feedback
RFPs repeat the same questions. Users catch errors the system misses. Both signals should feed back in.
Tradeoff: build speed vs. feedback quality
Golden Answer Cache
"Is Fusion EHR HIPAA compliant?" gets asked on every RFP. The answer never changes.
RAG response
"likely stable answer?"
"Save for future?"
answer + citations
(like ChatGPT memory)
similarity > 0.95?
The Eval Harness
- 50 real RFP questions + adversarial probes
- CI/CD triggers on doc updates, model swaps, weekly regression
correct top-5 chunks
across 50-question set
a real Confluence source
time end-to-end
Eval Types Matrix
These eval types apply beyond just RFP. They validate any domain deployment.
| Eval Type | What It Measures | Applies To |
|---|---|---|
| Retrieval Precision@5 | Right chunks in top 5? | All domains |
| Answer Faithfulness | Every claim supported by source? | All domains |
| Citation Accuracy | Citations point to real sources? | All domains |
| Confidence Calibration | High confidence = actually right? | All domains |
| Refusal Accuracy | Refuses when should, accepts when should? | All domains |
| Gap Report Quality | Recommendations actionable? | RFP, Support |
| Cross-Domain Leakage | RBAC prevents bleed? | Multi-domain |
| Chunking Boundary | Answers split across chunks? | All domains |
Almost there
Functional
- Citation-backed responses
- "Insufficient Evidence"
- Multi-pass retrieval
- Source authority tiers
- Hybrid search (vector + BM25 + rerank)
- Query decomposition
- Circuit breaker
- Confidence scoring (rule-based)
- Configurable model router
- Idempotent ingestion
- Golden answer cache for recurring queries
- Domain-isolated vector classes
- Feedback loop (thumbs up/down + corrections)
- Gap tracking (refusals feed backlog)
Non-Functional
- 100% citation presence
- <1% unsupported claims
- P95 latency <12s
- Monthly cost <$800
- Recurring queries served from verified cache
- >90% accuracy on 50-question validation
How It Scales
Multi-Domain & Future-Proofing
Multiple departments need grounded intelligence. We can't rebuild for each one.
Tradeoff: flexibility vs. simplicity
Designing for Multi-Domain from Day One
- Two-axis taxonomy: routing x content-type
- Adding a department = YAML config change
- RBAC gates who sees what
Swappable Components
Models commoditize. The moat is the harness. Every component behind an abstraction. Upgrades are config changes.
Azure Architecture
- Container Apps (scale-to-zero). Self-hosted Weaviate (data sovereignty). Service Bus (durable messaging).
What This Costs
| Component | Monthly |
|---|---|
| Container Apps (3 containers, scale-to-zero) | $80 – $150 |
| Weaviate (self-hosted) | $15 – $25 |
| APIM + networking | ~$50 |
| Anthropic API (Haiku + Sonnet) | $100 – $400 |
| Storage + queue | $50 – $130 |
| Total | $295 – $755 /mo |
What We Gained
Revenue Pipeline†
- Q1 2026: biggest by 1.4x
- +3 deals/quarter potential
- $6-10M above target
- More at-bats per year
Operational Improvements‡
- SLA adherence improving
- NPS + eNPS growing
- Augmented workforce, not chatbot
- Domain too complex for off-the-shelf
† Numbers shared at Fusion Health Town Hall, 3/18/2026
‡ Monthly reporting from the Operations team (Value & Automation division, under which the AI function is housed)
Closing the Loop
| Requirement | Design Decision |
|---|---|
| Citation-backed responses | Sonnet generates inline citations from retrieved chunks |
| Insufficient Evidence | Three-tier confidence scoring, rule-based refusal |
| Multi-domain routing | Two-axis taxonomy, config-driven Weaviate classes |
| RBAC | Azure AD + tag-scoped class access |
| Idempotent ingestion | Content-hash dedup + canonical blob snapshots |
| P95 <12s | Haiku classifier, APIM caching, hybrid search |
| Cost <$800/mo | Scale-to-zero containers, two-model split |
| Gap tracking | Refusal logs → content backlog pipeline |
| >90% accuracy | 50-question eval harness, built first |
| Recurring query optimization | Golden answer cache: verified responses bypass RAG pipeline |
| Catching confident-but-wrong | User feedback loop: thumbs up/down + corrections, weekly review |
If I Had More Time
Agentic Canvas Workflows
RFP section building in-platform
Deeper Multimodal
LMS videos, product demos, ticket screenshots as first-class knowledge sources.
Predictive Gap Analysis
Pattern analysis on refusal logs to surface documentation gaps before they're queried.
Teams Integration
MCP bot where work actually happens
How This Translates to Boltline
Modular Harness Architecture
FedRAMP-ready, evals catch regressions on swap
Work Plans, BOMs, and Part Traceability
Work plans + BOMs as RAG corpus
Hybrid Search for Exact Matches
Part #s and serial #s need exact match
Corpus Quality at Scale
Per-customer class scoping prevents degradation
Thank You
Questions? Let's go deep on anything.
Connor England
A Note on Harnesses
The moat is the harness, not the model. Two projects shaped this thinking.
Block's Goose
Open-source, LLM-agnostic agent framework. MCP as the sole integration standard. Local-first, no vendor calls. Now in the Linux Foundation alongside MCP itself.
Stripe's Minions
Built on a Goose fork. 1,300+ PRs/week, zero human-written code. Key innovation: "Blueprints" (hybrid deterministic + agentic nodes). ~500 tools available, ~15 curated per task.
Context
Tool Call
Tool
Revision
How we applied this
- Blueprints: confidence + citation = deterministic. Retrieval + generation = agentic.
- Pre-hydration: context assembled before LLM sees anything.
- Tool curation: Legal query only sees Legal_* classes. Scoped per domain.
- MCP: one protocol, any interface. Same brain everywhere.
- Harness-level RLS: Weaviate has no row-level security. The harness injects collection + content_type filters at query time. Deterministic, fast, configurable per role.
UX Decisions Under the Hood
- Streaming responses. Users see generation in real-time, reducing perceived latency by ~60%.
- Citations rendered as clickable links to source Confluence pages, with excerpt previews on hover.
- Confidence badge shown prominently. Users learn to trust the system because it tells them when it's unsure.
- Query history with re-ask capability. Refine without retyping.
- Admin view shows the full retrieval trace: what was searched, what was retrieved, what was filtered, what was cited.
Chunking Strategy
- Semantic chunking: split on heading boundaries first, then paragraph breaks, then sentence boundaries.
- Target: 300–500 tokens per chunk. Enough context to be useful, small enough for precise retrieval.
- Overlap: 50-token sliding window between chunks to preserve context at boundaries.
- Metadata preserved per chunk: source page ID, heading hierarchy, authority tier, last-modified date, content-type tag.
The RAG Quality Cliff
Two independent studies confirm the same pattern: vector search accuracy degrades meaningfully as corpus size grows, even with good embeddings.
EyeLevel Research, 2024
Tested Pinecone + OpenAI ada-002 embeddings across 1K, 10K, and 100K pages. 310 real answer-bearing pages + filler. 92 test questions, human-evaluated.
Expected degradation: 10-12% per 100K pages. A "page" is a full document page (PDF, HTML), not a chunk.
Sarkar (TDS), Jan 2026
Focused on HNSW algorithm behavior. With fixed index parameters (M=32, ef_search=128), recall degrades as vector count scales.
A "vector" is a single chunk embedding. 46K Confluence pages at ~10-20 chunks each = 600K-1M vectors.
Takeaways
- Noise ratio drives degradation. More irrelevant content = harder to surface the right chunks.
- Mitigations compound: class isolation + authority tiers + hybrid search + reranking.
- "Ingest everything" has a ceiling. Class decomposition delays it indefinitely.
- For Boltline: multi-tenant customers with large multimodal data need class-scoped search from day one.
EyeLevel, "Do Vector Databases Lose Accuracy at Scale?", 2024 | Sarkar, "HNSW at Scale," TDS, Jan 2026