Engineeringcognitive searchwith zero hallucination.
Hybrid Dense/Sparse Vector Search & Ground-Truth Verification
14 million technical documents. 40 different repos. Engineers spending hours hunting for answers.
At Cortex Labs, 1,200 software and systems engineers were losing valuable hours daily searching through fragmented Confluence wikis, GitHub markdown files, Google Docs, and Jira tickets.
Early experiments with generic LLM chat interfaces produced dangerous hallucinations — generating fabricated API methods and referencing deprecated internal security protocols.
Cortex engaged Codedway to architect a production-grade enterprise RAG engine that guarantees cryptographic citation provenance for every sentence produced.
Hybrid dense-sparse retrieval before generation.
Data Audit (Wk 1–2)
Parsing hierarchical documentation silos, extracting code snippets, tables, and metadata tags.
Vector Pipeline (Wk 3–5)
Fine-tuning embedding models and deploying Qdrant clusters with HNSW index partitioning.
Cross-Encoder (Wk 6–9)
Integrating reciprocal rank fusion and lightweight GPU-accelerated secondary re-ranking.
Streaming UI (Wk 10–12)
Developing keyboard-first command palette search with sub-100ms instant streaming answers.
“Codedway gave us a search engine our engineers actually trust. When it cites a document, the link and code snippet are 100% accurate every single time.”
Hybrid Dense/Sparse Indexing
Simultaneous vector embeddings and BM25 lexical tokenization delivering state-of-the-art semantic precision.
Reciprocal Rank Fusion (RRF)
Algorithmic merge pipeline synthesizing keyword and dense vector candidate sets into a coherent ranking.
Cross-Encoder Re-ranking
Edge-hosted transformer model re-scoring the top 50 retrieval results within an 18ms latency budget.
Ground-Truth Citation Engine
Strict cryptographic chunk verification ensuring generated prose maps directly to immutable document hashes.
Real-Time Incremental Indexer
Webhook listener reacting to GitHub commits and wiki revisions with sub-second differential vector updates.
Command-K Search Palette
Blazing-fast client UI built with Next.js Server Components, instant keyboard shortcuts, and code syntax highlighting.
Numbers don't lie.
“The search speed is mindblowing. Our engineering onboarding time dropped by 40% in the first quarter of deployment.”
Enterprise knowledge management at scale suffers from fragmented document silos, stale indexes, and unreliable keyword search. Cortex Labs engaged Codedway to architect a cognitive enterprise search engine with zero hallucination risk.
Achieving 98.4% Mean Reciprocal Rank (MRR@10), the hybrid dense-sparse vector pipeline returns precision answers backed by direct cryptographic document citations in under 140ms.
The Challenge
- Information Fragmentation: Unifying disparate data formats—PDFs, Jira tickets, Notion pages, and Slack threads—into a single vector embedding space.
- Contextual Freshness: Real-time incremental indexing of updated documents without requiring nightly re-clustering.
- Low-Latency Retrieval: Keeping embedding generation, vector distance calculations, and cross-encoder re-ranking within a strict 150ms SLA.
Architecture & Implementation
- Hybrid Embedding Pipeline: Combining dense vectors (OpenAI text-embedding-3-large) with sparse lexical tokens (BM25) stored in Qdrant.
- Cross-Encoder Re-Ranking: Lightweight secondary re-ranker deployed on GPU edge workers to re-score top-50 candidates.
- Next.js Web Interface: Keyboard-driven, ultra-fast search UI built with server components, streaming response blocks, and markdown previewing.
NexaCommerce — Global Commerce API & Headless Infra
TYPESCRIPT · NODE.JS · STRIPEHave a project? Let's scope it.
We'll tell you honestly if we're the right fit.