Production Multimodal RAG Pipeline and Evaluation Framework

The context

An AI assistant for industrial equipment maintenance and diagnostics: technicians query, in natural language, a corpus of OEM manuals and technical documentation, often scanned, dense with tables and diagrams. It is the use case where naive RAG fails, and the one where my dual background in industry and AI applies most directly. As founding engineer, I owned the whole chain: ingestion, retrieval and evaluation. This page describes the architecture and the measured results.

The ingestion pipeline

From raw PDF to vector index, on a CPU Databricks cluster backed by a GPU Hugging Face inference endpoint:

• OCR by vision-language model served on a dedicated Hugging Face inference endpoint with autoscaling and scale-to-zero so GPU cost is only paid when used, cold-start warm-up, page-batch processing and recursive subdivision on failure: a failing batch is split and retried, partial success is accepted, and lost pages are tracked.
• Structured-output image annotation: every diagram or photo is described by a vision model with contextual anchoring (the prompt embeds the surrounding markdown and the image’s exact position), mandatory reasoning-first fields and a confidence score. Images are served through a CDN and rendered directly in the conversation with their explanation: a technician sees the original annotated diagram, not a paragraph paraphrasing it.
• Perceptual deduplication before paying for inference: perceptual hashing eliminates repeated images, saving 52% of vision calls on a real manual (216 references, 103 unique).
• Token-aware chunking: semantic splitting measured in the embedding model’s own tokenizer (500 to 1,500 tokens), merge-then-split with atomic image blocks, section breadcrumb preserved on every fragment, and tables serialized as JSON Lines rather than markdown through a custom serializer, for a more precise vector retrieval.
• Page attribution by fuzzy matching: OCR loses page provenance, so it is rebuilt through an exact, prefix, then RapidFuzz sliding-window cascade, so that every chat answer is sourced like a bibliography: chapter, paragraph, page and annotation, funneling down to the exact citation.
• 3,072-dimension embeddings and a Databricks Vector Search index with idempotent upserts and filterable metadata by asset, manufacturer and document category.

Retrieval

Maintenance forbids purely semantic retrieval: an exact error code or part reference cannot be found by similarity.

• Hybrid search: vector similarity and BM25 keyword search, fused through Reciprocal Rank Fusion, capturing both natural-language questions and exact identifiers.
• Neural reranking of retrieved passages, then U-shaped reordering (Lost in the Middle) so the decisive passages sit where the model reads them best.
• Confidence tiers that decide what happens next: above the threshold, answer; in the middle band, rewrite and replay the query; below it, switch to web search to raise the confidence level, and failing that, an answer that states its uncertainties rather than hiding them.

Industrialization

• Databricks Asset Bundles across four targets: per-developer environments, then dev, test and prod, run by a service principal with ACLs declared in the bundle.
• Environment topology lives in code: a single function derives catalog, host, CDN and bucket from the current workspace, with zero per-environment config files.
• Three-tier resilience by failure class: exponential backoff for transients, progressive batch subdivision for overflows, periodic HTTP-client recycling for degraded connections.
• A quality gate instead of silent degradation: beyond 10% of unannotated images, the whole document is rejected rather than indexing incomplete chunks.
• Cost as a design constraint: the GPU is isolated on the on-demand inference endpoint while the Databricks pipeline stays on a single-node CPU cluster, token and payload quotas are computed before every call, and DBU-versus-latency trade-offs are documented in the git history.

The evaluation framework

The question to settle: build or buy. The answer came from a benchmark, not a conviction.

• Up to seven candidate pipelines compared against the commercial reference and the platform’s out-of-the-box solution, behind a single interface where only the retriever changes.
• The generator is pinned and the rewritten query is replayed for every candidate: the only free variable between two systems is retrieval. That is what separates a benchmark from an impression.
• An evaluation corpus stratified by document pathology: seeded KMeans clustering over the structural features of 183 PDFs, then targeted groups (image-dense, table-dense, corrupted text layer, language switching).
• Generated test sets plus a multimodal control: Ragas synthetic questions over a knowledge graph, and a hand-written multimodal slice to probe what text-only benchmarks cannot see.
• Six metrics swept by retrieval depth (k from 1 to 10) with 95% confidence intervals, judge provenance persisted per metric, and idempotent, resumable runs.

The results

• The in-house pipeline beats the commercial baseline on Precision@k 0.82 vs 0.68 and Recall@k 0.70 vs 0.56.
• Against the platform’s out-of-the-box solution: 41% lower retrieval latency (P95 at 1.4s vs 2.3s) and about 40% fewer context tokens per query, at comparable answer quality.
• The multimodal slice inverts the text benchmark’s conclusion: +26 points of precision and +28 points of faithfulness on questions about diagrams and tables, where the product’s value lives.
• The final recommendation was written with its counter-evidence: the out-of-the-box solution wins on purely textual queries, and the report says so.

What the project demonstrates

Evaluation as a decision instrument rather than a formality: the first benchmark, biased by having the full agent graph in the loop, was invalidated and discarded within 48 hours; the last one settled a build-vs-buy decision with its numbers, its limits and its counter-arguments. In between, a production pipeline where every stage, cost, quality and provenance, is measured.