Fusion Intel
Building a Knowledge Engine from First Principles
Connor England
Please do interrupt with questions!
About Me
- Current: Applied AI Developer & Solutions Architect, Fusion Health
- Before: Co-founded SmartSave (fintech, USTE processing). Built Pidgeon Health (open-source C# healthcare data platform, used at Fusion and beyond)
- Early Career: QA Lead, 3 food manufacturing plants (HACCP/SQF)
- Certs: AWS Solutions Architect, CompTIA Security+, DeepLearning.AI Data Engineering
What We'll Cover
This talk is about
- Fusion Intel: multi-domain RAG platform (designed + 80% coded by me)
- Modular AI development that absorbs improvements fast
- Lessons learned, applied to Boltline
This talk will
- Derive requirements from domain constraints
- Walk through architecture and implementation
- Share strong opinions, loosely held
- Get into the nitty gritty!
What Is Fusion Intel?
A platform that lets our teams ask questions and get accurate, cited answers from our own documentation, powered by AI.
The problem with general AI
Trained on public data, not ours. Guesses when it doesn't know.
What RAG changes
Retrieval-Augmented Generation: searches our vetted docs first, then answers. Every claim traceable to source.
Why Build This?
First: RFP Bottleneck
- RFPs are the bloodflow of business growth
- 40-80 hrs per RFP, SMEs pulled off product work
- Clear first use case: massive upside, limited downside
Next: Cross-Domain Intelligence Multiplier
- Same knowledge bases serve Legal, Compliance, Support, CSM
- Build once well, then scale from first principles
Constraints & Opportunities
Domain Constraints
- Multi-department, one platform (RFP, Legal, Support, CSM)
- HIPAA compliance: full data control required, no RAG-in-a-box
- Docs scattered across Confluence + OneDrive, no unified search
- Cross-domain sharing with clear department boundaries
Technical Constraints
- RAG quality degrades at scale (46K pages, 131K tickets)
- Vector search misses exact terms (drug names, product codes)
- LLMs are sycophantic: will guess instead of refuse
- AI moves fast: every component must be swappable
The core value prop is genuinely accurate intelligence. If we don't have provenance, we don't have an app.
Helpfulness vs. honesty. We chose honesty.
- First Pass: Weaviate hybrid search across authority-tiered classes
- Second Pass: Confluence deep search via Atlassian MCP. If found, recommend tagging into corpora.
- Third Pass: Intelligence advisory. What was searched, what's missing, who to ask next.
What we know so far
Functional
- Citation-backed responses
- "Insufficient Evidence" as a first-class output
- Multi-pass retrieval with fallback advisory
- Feedback loop for confident-but-wrong responses
Non-Functional
- 100% citation presence on answered queries
- <1% unsupported claims
- Recurring queries served from verified cache
The RAG Flow
Instead of asking a general AI model and hoping it knows the answer, we feed it our vetted information first. Here's how.
Multiple departments need this. We can't rebuild for each one. One architecture, many consumers, clear boundaries.
Flexibility vs. simplicity. Config-driven over code-driven.
Designing for Multi-Domain from Day One
RFP response is the first use case. But Legal, Compliance, Client Success, and Support all need the same capability. The architecture serves all of them without retooling.
- Each intersection maps to a Weaviate class. Adding a new department = a YAML config change. No code changes. RBAC gates who sees what.
- "Interface" = HL7 interface capabilities (speaking with other vendor systems like pharmacy, lab, EHR).
- When tags are removed, the system soft-deletes (archives) vectors rather than persisting them. This prevents mistakes from polluting signal.
Building the list
Functional
- Citation-backed responses
- "Insufficient Evidence" as first-class output
- Multi-pass retrieval with fallback advisory
- Two-axis tag routing
- Config-driven domain extensibility
- RBAC-scoped access per department
- Soft-delete on tag removal
Non-Functional
- 100% citation presence
- <1% unsupported claims
- Idempotent ingestion (reprocess = zero duplicates)
System Architecture
Knowledge changes constantly, and we have a lot of it. 46K pages, 131K tickets. Stay current without drowning in noise.
Volume vs. quality. Curate aggressively, ingest idempotently.
The Ingestion Pipeline
Confluence page created/updated triggers an Atlassian webhook. Event-driven means zero polling overhead.
Service Bus absorbs bursts, retries failures, dead-letters poison events. Decouples ingestion from source.
Labels map to taxonomy axes. Determines which Weaviate class(es) receive the content. Removals trigger soft-delete.
Canonical version stored in Blob Storage. Vector DB is derived, never authoritative. Always rebuildable.
Semantic chunking: heading boundaries first, then paragraphs, then sentences. 300–500 tokens with 50-token overlap.
Content-hash deduplication ensures idempotency. Embedding model generates vectors for each chunk.
Vectors land in the correct Weaviate class with full metadata: source ID, heading path, authority tier, timestamps.
The Laffer Curve of RAG
More data isn't always better. With default HNSW† settings, recall drops from 99% at 10K vectors to 85% at 10M‡. At Fusion's scale (~600K-1M chunk vectors), naive ingestion puts us in the degradation zone.
- "Ingest everything" trap: noise dilutes signal as vector count grows
- Our defense: domain-isolated classes, 2K-5K vectors per query scope
- Authority tiers: Gold 3x / Silver 2x / Reference 1x scoring per class
† HNSW (Hierarchical Navigable Small World) is the approximate nearest-neighbor algorithm most vector databases use to find similar chunks quickly.
‡ Sarkar, "HNSW at Scale: Why Your RAG System Gets Worse as the Vector Database Grows," Towards Data Science, Jan 2026
Swappable Components
Models commoditize. The moat is the harness. Every component behind an abstraction. Upgrades are config changes.
"eMAR" and "electronic medication administration" are the same concept. "Barcode scanning" is an exact term. Search has to handle both.
Precision vs. cost. Hybrid search + smart model routing.
Why Hybrid Search
Pure vector misses exact terms (drug names, product codes). Pure keyword misses semantic similarity. Hybrid with RRF merges both ranked lists. Cross-encoder reranking: 15 candidates to 5. +15–20% precision. Adds 200–400ms.
Two-Model Pipeline
classify/route
generate
check
Haiku: Classifier & Router ~$0.01/query
Classifies domain, detects compound queries, decomposes into sub-queries. Structured output via tool_use. Independently testable.
Sonnet: Generator ~$0.15/query
Generates citation-backed responses from reranked chunks. Every claim must trace to source. Confidence scoring is deterministic, happens after generation.
2 gold sources, relevance 0.91
Integration Guide, SLA Matrix
- Circuit breaker: embedding down = keyword-only fallback, system stays up
- Model router: config surface. New models or compliance boundaries = no code changes.
The list grows
Functional
- Citation-backed responses
- "Insufficient Evidence" as first-class output
- Multi-pass retrieval with fallback advisory
- Two-axis tag routing
- Config-driven domain extensibility
- RBAC-scoped access
- Soft-delete on tag removal
- Query decomposition for compound questions
- Circuit breaker / graceful degradation
- Configurable model router
Non-Functional
- 100% citation presence
- <1% unsupported claims
- Idempotent ingestion
- P95 latency <12s
- Monthly cost <$800
The system will be wrong sometimes. The question isn't whether. It's whether it knows when.
Trust vs. coverage. We'd rather refuse than guess.
Confidence Scoring
HIGH
Multiple Gold sources agree
Gold = 3x weight
MEDIUM
Silver sources, partial coverage
Silver = 2x weight
INSUFFICIENT
Refuses to answer, logs the gap
Reference = 1x weight
- Rule-based, not learned. Fully explainable, no black-box confidence.
- Gold sources count 3x in the weighted confidence calculation. Two gold sources matching = weighted score of 6. One silver = 2. The threshold for HIGH is 5+.
- Compound queries: overall confidence = minimum across sub-queries.
Teaching the System to Refuse
- LLMs are sycophantic. Will guess to be "helpful."
- Claude chosen for refusal backbone, but model-level isn't enough
- Harness enforcement: confidence scoring is code, not model judgment. Below threshold = generation blocked.
- Every refusal logged: what was searched, what was missing
with citations
evidence logged
prioritized backlog
Almost there
Functional
- Citation-backed responses
- "Insufficient Evidence"
- Multi-pass retrieval + fallback advisory
- Two-axis tag routing
- Config-driven extensibility
- RBAC-scoped access
- Soft-delete on tag removal
- Query decomposition
- Circuit breaker
- Configurable model router
- Golden answer cache for recurring queries
- User feedback loop (thumbs up/down + corrections)
- Gap tracking: refusals feed content backlog
- Source authority tiers
Non-Functional
- 100% citation presence
- <1% unsupported claims
- Idempotent ingestion
- P95 <12s
- Monthly cost <$800
- >90% accuracy on 50-question validation set
RFPs ask the same questions across submissions. Users will catch errors the system can't. Both signals need to feed back into the loop.
Repetition is an optimization opportunity. User corrections are free eval data.
The Eval Harness
- 50 questions: real RFP questions + adversarial out-of-scope probes
- Built first, not last. The eval harness existed before the generation pipeline.
- CI/CD integration: automated triggers after large doc updates, model switches, chunking changes, weekly regression
- User feedback: thumbs up/down + correction text on every response. Catches "confident but wrong."
correct top-5 chunks
across 50-question set
a real Confluence source
time end-to-end
Eval Types Matrix
These eval types apply beyond just RFP. They validate any domain deployment.
| Eval Type | What It Measures | Applies To |
|---|---|---|
| Retrieval Precision@5 | Right chunks in top 5? | All domains |
| Answer Faithfulness | Every claim supported by source? | All domains |
| Citation Accuracy | Citations point to real sources? | All domains |
| Confidence Calibration | High confidence = actually right? | All domains |
| Refusal Accuracy | Refuses when should, accepts when should? | All domains |
| Gap Report Quality | Recommendations actionable? | RFP, Support |
| Cross-Domain Leakage | RBAC prevents bleed? | Multi-domain |
| Chunking Boundary | Answers split across chunks? | All domains |
Golden Answer Cache
RFPs ask the same questions across submissions. Verified answers shouldn't go through the full pipeline every time.
similarity > 0.95?
no LLM call, no retrieval
retrieve, rerank, generate
How it works
- BD team approves a response with a "verify" action
- Question + verified answer stored with embedding
- New queries checked against cache before pipeline runs
- Served with a "Verified" badge so users know it's human-reviewed
Why it matters
- Zero latency, zero API cost for recurring questions
- Consistency: same question always gets same answer across RFPs
- Trust: "verified" badge signals human review
- Translates directly to Boltline (recurring work plan queries)
Azure Architecture
- Container Apps over AKS: simpler, scale-to-zero saves ~40%. Service Bus over Redis: durable, dead-letter. Self-hosted Weaviate over Pinecone: data sovereignty, no vendor lock-in.
What This Costs
| Component | Monthly |
|---|---|
| Container Apps (3 containers, scale-to-zero) | $80 – $150 |
| Weaviate (self-hosted) | $15 – $25 |
| APIM + networking | ~$50 |
| Anthropic API (Haiku + Sonnet) | $100 – $400 |
| Storage + queue | $50 – $130 |
| Total | $295 – $755 /mo |
What We Gained
Revenue Pipeline†
- Q1 2026 was our biggest Q1 by a factor of 1.4x
- Strong possibility of adding 3 more deals per quarter into the pipeline for the year
- Estimated additional $6-10M above initial target
- More at-bats per year: faster RFP turnaround means we can pursue deals we previously had to pass on
Operational Improvements‡
- Multimodal improvements driving better SLA adherence on support
- NPS for client satisfaction and eNPS for support team both growing steadily
- Better responses, easier troubleshooting through augmented workforce (internal)
- Our domain is too complex for an Intercom/Finn-style chatbot. Augmenting our own people is moving the needle.
† Numbers shared at Fusion Health Town Hall, 3/18/2026
‡ Monthly reporting from the Operations team (Value & Automation division, under which the AI function is housed)
Closing the Loop
Every requirement traces to a design decision.
| Requirement | Design Decision |
|---|---|
| Citation-backed responses | Sonnet generates inline citations from retrieved chunks |
| Insufficient Evidence | Three-tier confidence scoring, rule-based refusal |
| Multi-domain routing | Two-axis taxonomy, config-driven Weaviate classes |
| RBAC | Azure AD + tag-scoped class access |
| Idempotent ingestion | Content-hash dedup + canonical blob snapshots |
| P95 <12s | Haiku classifier, APIM caching, hybrid search |
| Cost <$800/mo | Scale-to-zero containers, two-model split |
| Gap tracking | Refusal logs → content backlog pipeline |
| >90% accuracy | 50-question eval harness, built first |
| Recurring query optimization | Golden answer cache: verified responses bypass RAG pipeline |
| Catching confident-but-wrong | User feedback loop: thumbs up/down + corrections, weekly review |
If I Had More Time
Agentic Canvas Workflows
RFP section building + citation implantation in a document canvas (like Claude Artifacts), so BD pros iterate on documents inside the platform.
Golden Answer Cache
Verified answer store for recurring questions. Human-approved responses bypass the full RAG pipeline. Highest-ROI optimization for repetitive query patterns.
Cross-Domain Intelligence Synthesis
Queries spanning multiple department corpora with appropriate access controls. Synthesize across silos while respecting RBAC.
Teams Integration
Extending the MCP server pattern to a Microsoft Teams bot, where teams actually work. Same intelligence, delivered where the conversations happen.
How This Translates to Boltline
Modular Harness Architecture
FedRAMP, ~9 approved models. Harness performs regardless. Evals catch regressions on swap.
Work Plans, BOMs, and Part Traceability
Work plans + BOMs as RAG corpus. Technicians query specs, get cited answers from actual docs.
Hybrid Search for Exact Matches
Part numbers, serial numbers, QR codes. Exact-match miss is a showstopper. Hybrid search handles both.
Corpus Quality at Scale
Multi-tenant multimodal data. Class-scoped search prevents degradation. Evals catch regressions.
Thank You
Questions? Let's go deep on anything.
Connor England
A Note on Harnesses
Models are commoditizing fast. The real moat is the harness: deterministic workflows that govern when and how intelligence gets applied. Two projects shaped how I think about this.
Block's Goose
Open-source, LLM-agnostic agent framework. MCP as the sole integration standard. Local-first, no vendor calls. Now in the Linux Foundation alongside MCP itself.
Stripe's Minions
Built on a Goose fork. 1,300+ PRs/week, zero human-written code. Key innovation: "Blueprints" (hybrid deterministic + agentic nodes). ~500 tools available, ~15 curated per task.
Context
Tool Call
Tool
Revision
Where these influenced Fusion Intel
- Blueprints pattern (from Minions): our confidence scoring and citation linking are deterministic nodes. Retrieval and generation are the agentic nodes. The LLM never decides whether to refuse; the harness does.
- Context pre-hydration (from Minions): we retrieve, rerank, and assemble the context payload before the generation model sees anything. The LLM's job is writing, not searching.
- Tool curation (from Minions): a Legal query only sees Legal_* Weaviate classes + General. We scope tools per domain rather than exposing everything.
- MCP as universal integration (from Goose): one protocol for all tool access. Same brain, any interface.
UX Decisions Under the Hood
- Streaming responses. Users see generation in real-time, reducing perceived latency by ~60%.
- Citations rendered as clickable links to source Confluence pages, with excerpt previews on hover.
- Confidence badge shown prominently. Users learn to trust the system because it tells them when it's unsure.
- Query history with re-ask capability. Refine without retyping.
- Admin view shows the full retrieval trace: what was searched, what was retrieved, what was filtered, what was cited.
Chunking Strategy
- Semantic chunking: split on heading boundaries first, then paragraph breaks, then sentence boundaries.
- Target: 300–500 tokens per chunk. Enough context to be useful, small enough for precise retrieval.
- Overlap: 50-token sliding window between chunks to preserve context at boundaries.
- Metadata preserved per chunk: source page ID, heading hierarchy, authority tier, last-modified date, content-type tag.
The RAG Quality Cliff
Two independent studies confirm the same pattern: vector search accuracy degrades meaningfully as corpus size grows, even with good embeddings.
EyeLevel Research, 2024
Tested Pinecone + OpenAI ada-002 embeddings across 1K, 10K, and 100K pages. 310 real answer-bearing pages + filler. 92 test questions, human-evaluated.
Expected degradation: 10-12% per 100K pages. A "page" is a full document page (PDF, HTML), not a chunk.
Sarkar (TDS), Jan 2026
Focused on HNSW algorithm behavior. With fixed index parameters (M=32, ef_search=128), recall degrades as vector count scales.
A "vector" is a single chunk embedding. 46K Confluence pages at ~10-20 chunks each = 600K-1M vectors.
Why this matters for system design
- The degradation is a function of noise ratio, not a magic number. The more irrelevant content in your index relative to any given query, the harder it is to surface the right chunks. Both studies confirm this independently.
- Mitigation strategies compound. Domain-isolated classes (search 2K-5K vectors, not 1M). Authority tier weighting (gold sources float above noise). Hybrid search (BM25 catches exact matches that vector search misses at scale). Cross-encoder reranking (re-scores the top-N for true relevance). Each layer recovers precision that raw vector search loses.
- The "ingest everything" strategy has a structural ceiling. Teams adopting "ingest all docs of types X, Y, Z" into one index will hit this wall. The question isn't if, it's when. Class decomposition and curation delay the cliff indefinitely.
- For Boltline: this matters even more in a multi-tenant manufacturing context. Customers with large volumes of data, especially multimodal (work plan docs, CAD references, inspection photos, test logs), need thoughtful class design and search strategies from day one. A platform that lets customer data grow unchecked into a monolithic index will hemorrhage effectiveness at scale. The right approach is guiding customers toward well-structured classes, curated ingestion policies, and domain-scoped search, so that the system gets more useful as data grows instead of less.
EyeLevel, "Do Vector Databases Lose Accuracy at Scale?", 2024 | Sarkar, "HNSW at Scale," TDS, Jan 2026