Most RAG projects begin with a simple assumption: if we can collect the documents, we can make them searchable.
In practice, the hardest work starts before the first embedding is generated.
Enterprise knowledge is usually scattered across PDFs, Word files, spreadsheets, slide decks, scanned pages, images, tables, and databases.
These sources rarely share a consistent structure.
A PDF may contain headers, footers, broken paragraphs, and tables whose meaning depends on surrounding text.
A chart may carry more information in its labels and relationships than in the caption.
A database row may be accurate but meaningless without its schema or business context.
Converting everything into plain text can remove exactly the context that retrieval needs.
Raw extraction also introduces duplicates, formatting artifacts, missing values, and partial records.
If those problems are pushed directly into the vector store, the retriever has to search through noise.
Poor answers are then often blamed on the embedding model, even though the real problem was the source data.
For this reason, building a RAG knowledge base should begin with data conversion and cleaning as a deliberate engineering stage.

A practical pipeline can be designed around several stages:

1. Ingest the source data and preserve provenance such as file name, page number, section, table, image, database, and record identifiers.
2. Convert each format into a structured representation. Document text, tables, chart descriptions, images, and database records should be represented as records with both content and metadata.
3. Clean and normalize the records by removing repeated headers, footers, invalid characters, broken fragments, empty fields, and formatting artifacts.
4. Preserve relationships and context. Table headers should stay connected to their values, chart labels should remain associated with the relevant visual content, and database rows should retain schema information.
5. Split the content into meaningful chunks based on document structure, sections, records, or semantic boundaries instead of using only a fixed character length.
6. Generate retrieval-ready data, such as question-answer pairs, summaries, explanations, or context-enriched passages.
7. Evaluate and filter the outputs for completeness, source consistency, language, length, and relevance.
8. Remove duplicate and near-duplicate records using rules, fingerprints, or semantic similarity, then export the validated data for embedding and indexing.

Intermediate outputs should be stored after each major step, so a failed extraction or overly aggressive filter can be inspected and corrected without rebuilding the entire dataset. This also makes it possible to use different processing branches for PDFs, images, tables, and structured database data while keeping a consistent downstream format.

The design above is implemented in OpenDCAI/DataFlow, and I’d be happy to discuss it further—automated data processing pipelines are an important capability for building reliable RAG systems.

I'm James Luan, CTO of Zilliz, the company behind Milvus. Milvus is an open-source vector database built to store, index, and search embeddings over unstructured data. Vector Lakebase is our direction for bringing that serving layer closer to the data lake.

Notion's search infrastructure is a useful example of a design that fits its initial workload well: many workspaces, with most queried infrequently. Making idle workspaces cheap is a sensible priority. My question is how that choice holds up as more AI features create sustained retrieval traffic.

Object storage changes the cold-query path. Before a cold index can answer, the system may need several object reads, deserialization, and index preparation. That work does not disappear by tuning the model. A user returning to a quiet workspace and a large tenant querying continuously stress different parts of the same design.

I expect local SSDs and memory to matter more as those workloads diverge. Multiple cache tiers can change the latency profile, while a design optimized around infrequent access may reach a different performance ceiling. These are architectural expectations, not claims that Notion has already encountered every failure.

There is a separate retrieval constraint when filtering follows approximate search. Narrow filters can discard much of the candidate set and reduce recall. More AI features often mean more specific conditions on page type, collaborators, or time.

The next evaluation should therefore separate cold returns, sustained tenant traffic, and selective filters. A cost improvement for the first workload does not settle the serving requirements of the other two.

https://zilliz.com/blog/notion-vector-search-next-problem?utm_source=reddit

Should I inspect restored segments instead of adding CPU to a slower RAG retriever?

I'd treat location as an eligibility rule in a location-aware RAG system. A restaurant can be a perfect semantic match and still be useless if it sits outside the delivery area. Letting the answer model repair that mismatch feels too late.

Retrieving a small semantic candidate set and filtering afterward can leave too few usable results. Increasing the candidate count may help, but it also makes the application responsible for balancing extra retrieval work against coverage.

With vector databases like Milvus, geometry filtering can participate in the search itself. An R-Tree first narrows locations by bounding rectangles; exact geometry checks then remove false matches before the remaining candidates are ranked by vector similarity. The rectangle match alone is not the eligibility decision.

I would probably test narrow polygons and addresses near their edges before celebrating average latency. A broad city-level query can hide the cases where an otherwise relevant result lies just outside the boundary.

For a small service, keeping location and embeddings together seems attractive because address changes have one retrieval record to update. The remaining work is evaluating semantic relevance inside the permitted area. I'd be interested in how others test that boundary.

I put together Hybrid Scoped RAG:
https://github.com/beascocheatyler-star/hybrid-scoped-rag
It’s a document chatbot with two things a lot of the usual demos skip:
• Hybrid retrieval — BM25 and dense vectors run side by side, rankings fused with RRF. So you get the exact-term hits (course codes, policy windows, IDs) and the meaning-based ones.
• Hard scoping — metadata filters get applied before anything is scored. Scope it to a specific course or set of docs and the bot literally can’t see the rest of the corpus.
The refusal part is what I cared about most. The gate still mostly looks at a calibrated lexical score (sqrt of BM25 × term coverage). Dense similarity can only pull a borderline case over the line — it can’t force an answer when the lexical signal is weak. RRF decides which chunks to show the model; the score decides whether to call the model at all. That split felt important because RRF throws away magnitude, which is useful for ranking but bad for “should I answer this?”
Default setup stays pretty cheap:
• Retrieval, scoping, and the gate all run offline. No key, no network, no model download required.
• Default embedder is just character n-gram hashing (no dependencies). It’s not real semantics — mostly helps with spelling/morphology stuff pure BM25 misses. You can swap in sentence-transformers if you want actual dense vectors.
• Bunch of offline tests (~267) so you can check the retrieval and gate without ever hitting an LLM.
There’s also a /api/search endpoint that just returns the per-retriever ranks and what the gate would do, so you can look at the fusion without generating an answer.
Limitations are pretty straightforward: default embedder isn’t semantic, refusal isn’t perfect (some things land in borderline instead of hard refuse), and persistence is just JSON for now. Fine for a few hundred docs, not something you’d run at scale.
It’s more of a focused engine than a full platform — FastAPI + simple frontend, personas, citations, same “only answer from the documents you gave it” idea as my earlier BM25-only version, just with the hybrid retrieval and metadata scoping added.
If you’ve been dealing with bots that go off the rails or you want a hybrid setup that still refuses aggressively, the code and tests are there. Curious what people think of the gate threshold and how the RRF weighting feels in practice.

I’ve been exploring what happens when a RAG demo has to keep working after the first successful query.

This TypeScript sample creates one durable Stateful Actor per company. Each actor:

- Embeds a handbook URL into Telnyx Storage

- Polls asynchronous indexing jobs and retries failures

- Searches the handbook before generating an answer

- Refuses to answer when the retrieval match is weak

- Records every Q&A in an embedded SQL audit log

- Reindexes the handbook automatically each night

The interesting part isn’t really the chatbot. It’s giving the entire retrieval lifecycle a durable owner: source state, indexing progress, retries, scheduled refreshes, and answer history all stay with the company’s actor.

The sample uses Telnyx Edge Compute, Stateful Actors, Embeddings, Storage, and AI Inference.

Code: https://github.com/team-telnyx/telnyx-code-examples/tree/main/policy-recall-line

The public routes are intentionally unauthenticated for the demo, so production use would need authorization, retention rules, evaluations for the similarity threshold, and better observability.

I’d be interested in feedback on the actor-per-tenant approach, especially from anyone maintaining frequently changing internal knowledge bases.

I started logging the rank of the known good chunk for every question in my eval set, across dense, sparse, and fused. The surprise was the long tail. A meaningful share of correct answers come from chunks ranked 30 or worse in the first stage, which my old top 15 fusion would have dropped before the reranker ever saw them. It made me stop trusting small first stage cutoffs entirely.

Curious what other people's distributions look like. Do most of your correct chunks sit in the top 10, or do you also have a long tail? And has anyone found a principled way to pick the first stage cutoff, or is everyone just guessing a number that feels affordable?

I have been thinking when you use a API for parsing or/and extraction and you often need to validate the output against a dataset or confidence threshold and fix errors before it reaches your knowledge base or RAG pipeline whatever use case you have or prefer that the API should have its own correction interphase where all the outcomes which was below the threshold, lands and corrections can be assigned to people on the vendor API itself. what are your thoughts about it?

​

I’m developing a $400 independent retrieval assessment: two existing configurations, up to 30 queries, and reproducible findings on regressions and scoring consistency. Fixes aren’t included.

The question I’m trying to resolve is where that scope leaves too much work with the buyer.

If you’ve paid for external technical QA—or considered it and decided against it—could you share one actual example?

- What triggered the decision: a client handoff, migration, incident, or something else?

- What did the reviewer need to deliver for you to approve the expense?

- What would have stopped the purchase?

I’m particularly interested in whether preparing the evaluation data was a blocker, whether a report alone was useful, and whether you needed fixes or a retest included.

An anonymized example is enough. Please mention your role in the decision and whether the engagement was purchased or rejected.

I’m the provider researching this offer. This is a question about what buyers need; I won’t treat a reply as permission to send a sales DM.

we recently got a mini heart attack from gpt 4o mini, where it is deprecating in jan 2027, but we built our entire pipeline and normalizations according to that model, so again rework from scratch and few major changes.

so in these rapidly changing environment of models, how do we need to manage our RAG system according to different models.

if you have any idea or suggestion please let me know, and if you are also facing same issue then how you guys are managing?

Hi everyone,

I'm working on an LLM-based contract-audit pipeline that processes scanned documents and extracts important information, including cheque numbers, lot numbers, and other contractual references.

My main problem is inconsistent extraction. When I process the same scanned document multiple times, the model sometimes returns different results. For example, small digits in cheque or lot numbers can be misread, even though the input document is exactly the same.

I've already tried setting the temperature to 0 and using a fixed seed, but the problem persists.

My main requirement is to avoid using a separate OCR engine. I want to use a vision-capable LLM directly to read the scanned documents.

I'd appreciate advice on the following:

• How can I improve the accuracy of small digits and identifiers in scanned documents?
• Are image preprocessing, high-resolution crops, or processing individual fields separately effective?
• How can I reduce output variations when using the same image and model?
• Are there specific vision models or inference settings that perform particularly well for document extraction?
• How do you handle ambiguous digits without allowing the model to guess?
• Are there reliable techniques to validate extracted values without introducing a separate OCR system?

I'm looking for practical solutions that work in production, particularly for legal and financial documents where a single incorrect digit can cause serious problems.

Thank you for your help.

Hey r/RAG, I'm Abdur-Rahman Butler, a machine learning engineer at Isaacus.

I made an interactive write-up on the A to Z of vector embeddings, inspired by Brendan Bycroft's incredible LLM visualisation (https://bbycroft.net/llm).

Most of my work is on encoder models, and I kept noticing how few good interactive resources there are on embeddings. A lot of RAG tutorials tend to treat the inner mechanics of embedding models as an afterthought in place of the wider retrieval architecture.

Making embeddings a fun and interesting topic to explore interactively was quite hard, as besides 2D/3D projections of embeddings or messing around with vector arithmetic (the classic king-man+woman=queen stuff), there aren't too many elegant ways to communicate the inner workings of embedding models.

To address this I created a game I'm dubbing 'SemanticGuessr'. It's inspired by Wordle and GeoGuessr, and the goal is to draw a loop around a random target word by triangulating guesses on a 2D map of 43,339 common English words.

It's admittedly a bit intimidating at first, as semantic space doesn't perfectly match our intuition of meaning. But once you get the hang of it, the game becomes really addictive. I recently had the pleasure of demoing it to some law students at the University of Melbourne, and we had a lot of fun throwing different vector equations at it to see where they'd land.

london - english + french
running - land + water
avg(breakfast, lunch, dinner)
0.4 * noodles + 0.6 * pasta

The second half of the guide follows Chronos, a fictional time-travel-as-a-service company whose support team is drowning in tickets. You rebuild their support system one piece at a time only using embedding. Through this case study we explore a range of techniques, like:

• Semantic search over the help centre, so the query "It wouldn't let me pay for my trip" finds "Why your payment was declined", while keyword search gets distracted by "trip".
• Hierarchical retrieval over the company's terms of service, where every level of the contract (the whole document, sections, clauses and subclauses) goes into one index. A narrow question pulls back the single subclause that answers it, while a broad one gets the whole document.
• Zero-shot classification to route support tickets to the right team. For example, the complaint "My time machine broke down and stranded me in the prehistoric era" automatically lands in "Malfunction & rescue" class and is then routed to that team.
• Clustering and deduplication to group similar GitHub issues without any labels, so all the reports about a single fault, like 2FA not working, can be combined and addressed with a single response.

Everything runs statically in your browser. I cut the embeddings to 512 dimensions and quantised the values to 4-bits, which took the payload from 311 MB to 12.3 MB with only around ~6% loss in precision. There are Python snippets alongside if you want to follow along.

Happy to take any questions, and let me know your highest score :)

LINK TO PLAY: https://isaacus.com/blog/what-are-vector-embeddings

Something that's been bugging me about agent memory evals. Every benchmark I see measures recall: did the agent remember the fact? But in long-lived agents, the harder problem is un-remembering. Plans change, facts get superseded, preferences flip. A memory that recalls a stale fact with high confidence is worse than one that admits it doesn't know.

I keep wondering why there's no standard metric for this. Something like: inject a fact, then supersede it, then ask. Measure whether the agent serves the new fact, the old one, or hedges. Call it retraction accuracy or memory half-life, whatever. Does this benchmark exist and I'm just not aware of it? How are you handling stale memory in production agents?

I'm currently building an AI chatbot with an agentic architecture for querying internal documents and data. One of the main challenges I'm facing is designing a reliable ingestion and retrieval pipeline that can handle a wide variety of document types without sacrificing answer accuracy.

1. The document types I need to support

The system needs to work with a fairly broad range of content, including:

• Legal documents and regulations: Long documents with nested sections, articles, clauses, and cross-references.
• Reports and business documents: Documents containing tables, charts, figures, and mixed layouts.
• General text documents: DOCX, TXT, Markdown, and similar formats.
• Excel files: Multiple sheets, structured tables, headers, and potentially complex relationships between cells.
• PDFs: Both digitally generated PDFs and scanned PDFs that require OCR.
• Images embedded in documents: Diagrams, charts, screenshots, scanned pages, and other visual information that may contain essential context.

The challenge is that these formats have very different structures and require different extraction and preprocessing strategies.

2. Problems with my current approach

Currently, I'm experimenting with separate ingestion pipelines for different document types. Each pipeline handles extraction, preprocessing, chunking, and indexing according to the input format.

However, I'm running into several issues:

• Ingestion complexity: Each format requires different handling, and maintaining multiple pipelines introduces edge cases and inconsistent outputs.
• Structure preservation: Naive text extraction and chunking can lose relationships between legal articles, tables, spreadsheet headers and values, or text and its associated images.
• Scanned PDFs and visual content: OCR can introduce errors, while extracting text alone may miss important information in charts, diagrams, and complex layouts.
• Retrieval accuracy: A single retrieval strategy doesn't seem suitable for every type of content. For example, semantic search may work well for conceptual questions but perform poorly on exact legal clauses, numerical values, or spreadsheet lookups.
• Query routing: Since this is an agentic chatbot, I also need to determine how to route each question to the appropriate retrieval strategy or data source. Some questions may require a combination of document retrieval, structured data lookup, and visual content analysis.

My main concern is that adding more document-specific logic will make the system increasingly difficult to maintain, while using a single generic pipeline may compromise retrieval quality.

3. What I'm looking for

For those who have built production RAG systems handling heterogeneous enterprise documents:

1. Ingestion architecture: Would you recommend a unified ingestion framework with format-specific extractors, or completely separate pipelines for each document type? Is there a good way to normalize their outputs while preserving document structure?
2. Document parsing: What tools or frameworks have worked well for complex PDFs, scanned documents, Excel files, and image-heavy reports? I've been exploring tools such as Docling, Unstructured, and OCR-based solutions, but I'm still unsure how to combine them into a robust pipeline.
3. Indexing and retrieval: How do you handle different retrieval requirements across legal text, ordinary documents, tables, spreadsheets, and visual content? Do you use separate indexes, hybrid retrieval, parent-child chunking, multimodal embeddings, or a combination of these?
4. Agent and query routing: How do you decide whether a question should go through vector search, keyword search, structured table or spreadsheet lookup, OCR-derived content, or a vision-capable model? Do you use a lightweight classifier, an LLM-based router, or a more explicit query-planning approach?
5. Practical recommendations: If you were designing this system from scratch today, what architecture and open-source components would you choose? Are there any approaches that seem promising in demos but become unreliable or expensive in real-world use?

I'm particularly interested in approaches that prioritize answer correctness, reliable source citations, and maintainability, rather than simply maximizing retrieval coverage.

Any recommendations, lessons learned, or examples of production architectures would be greatly appreciated. Thanks!

Imagine retrieving two chunks from the same knowledge base.

Both look relevant. Both contain valid information. However, one chunk says a feature is supported, while the other says it was deprecated in a version.

The problem isn't necessarily retrieval quality. The problem is deciding whether both chunks can be trusted for the question that is being asked.

I'm curious how people handle this in a RAG system:

• Do you use timestamps and document versions to resolve conflicts in a RAG system?
• Should the reranker in a RAG system consider freshness. Should that happen after retrieval?
• What happens when two sources have equal authority but disagree?

I'd be interested in approaches that work without adding another expensive LLM call to every query.

I've been building a small grounding dataset for a narrow domain (niche fashion subcultures: Goth Ninja, Acubi, Amekaji, Ura-Hara…) and wanted to know whether retrieval actually changes the answers, or whether the model already knows this stuff.

Setup

• 15 topics, the same open question for each ("build me an outfit, name brands, where to buy, budget, which years")
• 30 fresh chats with Claude Opus, no web search: once from memory, once with a ~1 KB record pasted in
• Each record has the usual fields (years, key brands, items, shops) plus one line I added almost as an afterthought: WARNING: do not confuse with X

Results

What surprised me

1. It barely invents anything. The errors were plausible items borrowed from the style next door: Red Wing boots in a Japanese repro-denim outfit, Y-3 in Goth Ninja, a Massimo Dutti coat in "quiet luxury". Nothing a fact-checker would flag at a glance, which is what makes it bad.
2. On well-documented topics, retrieval added almost nothing. Indie Sleaze, Mall Goth, Terrace Casuals: memory was about as good as the record. The gap was entirely on the niche ones.
3. The negative field did the most work. With "do not confuse with Techwear" in context, the model explicitly excluded tactical pieces. Without it, it mixed them in.
4. The model reviewed my data better than I did. With the record in front of it, it pushed back on 9 of my 15 records (a game released after the stated era, a band from a different scene, a price range that only works secondhand). I checked each point against sources; most were right, so I fixed them. Your ground truth needs QA too, and the model is a decent first reviewer.

Limits: one run per arm, one model, 15 topics, and the reference is my own data, so "matches the record" is not the same as "correct". All 30 answers, the scoring script and the hand judgements are on GitHub: https://github.com/ascanioUnderIndex/underindex-api/tree/main/eval

Questions for you:

• Do you store negative or contrastive information ("this is not X") in your chunks? Has anyone measured whether it helps beyond niche domains?
• How do you QA the ground truth itself? Using the model as a reviewer felt circular at first, but it caught real errors.
• Is "blending neighbours" something you've seen in your own domain (legal, medical, product catalogs)?

(Disclosure: the dataset and the eval are mine.)

[ Removed by Reddit on account of violating the content policy. ]

Beyond Naive RAG: Building a Production-Ready Pipeline for Complex PDF Tables

Hey everyone,

Like many of you, I started building RAG pipelines using standard recursive character chunking (e.g., 500 characters with 50-character overlap) and feeding PDF text into dense embeddings.

It works decently for clean narrative text. But the moment you ingest real-world enterprise documents financial reports, multi-column research papers, balance sheets, and surveys with multi-tier nested tables naive RAG completely breaks down.

We recently engineered an end-to-end production RAG pipeline built from first principles, without LangChain or LlamaIndex wrappers.

Here is a deep dive into the failure modes we encountered, why we rejected heuristic fixes, how we solved complex table extraction with IBM Docling and Vision LLMs, and our empirical benchmark results.

1. The Core Failure Modes of Standard RAG on Documents

1. Horizontal Column Scrambling: Naive parsers such as pypdf and pdfplumber often read across multi-column pages horizontally instead of reading Column 1 from top to bottom, then Column 2.

2. Table Header Decapitation: A sliding window slices a 30-row table into arbitrary 500-character chunks. Chunks 2 and 3 lose the column schema entirely. When the LLM retrieves Row 24 (524 | 123 | 482), it has no idea what those numbers correspond to.

3. The "Pipe Table Embedding Trap": Dense embedding models such as all-MiniLM-L6-v2, bge, and text-embedding-3 perform poorly on raw Markdown pipes (| Q1 | Q2 |). They lose cross-row column alignment.

4. The Silent Truncation Bug: If a chunk exceeds 512 tokens, standard sentence-transformer models may silently truncate the trailing text without throwing an error.

2. The Case Study: The "Shifted Column" Catastrophe on Table 14

While testing on a standard accessible PDF benchmark (sample-tables.pdf), we encountered a classic layout parser failure on Table 14, a four-column survey containing questions and Yes/No responses.

• Column 1: Expected question text, but the parser injected boolean answers (No, Yes).
• Columns 2–3: Intermediate shifted data.
• Column 4: The actual long descriptive questions were pushed to the last column.

Why We Rejected "Heuristic Guessing" (Column Rotation)

Our first thought was:

"Can we write a heuristic to rotate the columns back if Column 1 contains booleans?"

Never do this blindly in production.

Heuristic guessing is brittle. It might fix Table 14 but corrupt Table 15, which could legitimately have boolean row headers.

If your parser cannot understand the layout natively, heuristics can introduce silent data corruption.

3. Our Architectural Solution: A 7-Stage Block-Type Pipeline

Instead of flat text chunking, we built a schema-aware pipeline.

text PDF Document | v [Stage 1: Docling Neural Layout Engine] Preserves reading order and extracts the document structure | v [Stage 2: TableSanityChecker] Evaluates column skew and empty-cell ratios | +-- Health >= 0.70 --> Direct Block Chunking | +-- Health < 0.70 --> [Stage 3: VLM Repair via Mistral Pixtral-12B] | Visually transcribes the page | using free-tier access | v [Stage 4: BlockChunker] Tables <= 25 rows remain atomic Oversized tables are split with replicated schema headers | v [Stage 5: Dual-Representation Serialization] | +-- (A) Linearized key-value text for vector embeddings | ChromaDB + BM25 | +-- (B) Pristine Markdown tables for LLM prompt context | v [Stage 6: Runtime HitCollapser] Sibling deduplication + dynamic row slicing (+/- 2 rows) | v [Stage 7: Generation and Guardrails] Open-Mistral-7B + citation-bounds verification

Key Engineering Decisions

1. The 95/5 VLM Production Pattern

Calling GPT-4o or Claude on every PDF page can be expensive ($10–$50 per 1,000 pages, depending on the model and workload) and slow.

Instead, 95% of tables are processed locally for free via Docling. Only the approximately 5% that fail our automated sanity checks (TableSanityChecker) are routed to a Vision LLM (pixtral-12b-2409), which is accessible through a free tier.

This selective escalation approach helps balance cost, latency, and extraction quality.

2. Dual Representation

We use two representations of the same table, each optimized for a different purpose.

For embeddings: We linearize table rows into key-value sentences:

text Section: Financials > Table: Revenue > Row 1: Region: Americas; 2010: $500M; Margin: 22%

This gives embedding models explicit semantic hooks.

For LLM generation: We inject clean Markdown tables into the context, preserving the relationships between columns and rows.

The result is a retrieval representation optimized for semantic search and a context representation optimized for structured reasoning.

3. Dynamic Row Windowing

If a 150-row table matches a search query, dumping all 150 rows into the prompt can contribute to "Lost in the Middle" attention degradation.

Our retriever collapses sibling hits and slices only the matching rows, plus or minus two neighboring rows, while preserving the table's schema header.

This keeps the context focused without losing the information needed to interpret the retrieved values.

4. Empirical Benchmark and Latency Results

We built an end-to-end evaluation suite with 10 challenging queries against sample-tables.pdf, including:

• Single-cell lookups
• Multi-year comparative deltas
• Cross-table boolean conjunctions
• Negative hallucination traps

Executive Scorecard

The only retrieval non-hit was an intentional negative trap involving Antarctica, which the guardrail correctly refused to answer.

Latency Profiling by Stage

These results suggest that retrieval and generation dominate the latency budget, while post-retrieval context processing adds negligible overhead in this benchmark.

Note: These results are from a 10-query evaluation on a single PDF benchmark. They demonstrate the pipeline's performance on these test cases, not a guarantee of equivalent accuracy across arbitrary enterprise documents.

5. Code and Architecture Documentation

The entire project is open source under the MIT license and written in typed Python, built from first principles.

• 🐙 GitHub Repository: https://github.com/Souheil007/Rag_Full_Version

I'd love to hear how others handle multi-tier table splitting, column skew, and schema preservation in production RAG systems.

How do you approach reliable retrieval from complex PDF tables without relying on brittle heuristics?

I'm building an enterprise RAG system with the following pipeline:

• Hybrid retrieval: BM25 + dense (20 chunks each)
• RRF fusion: Top 15
• Reranker: Top 5
• LLM for answer generation

The pipeline already takes around 30 seconds, and I haven't implemented metadata filtering yet.

One problem I've encountered is when a user asks a question that requires information from more than 5 chunks.

For example, a user asks for country catalogue codes for the US, China, India, Bangladesh, Pakistan, Bhutan, Sri Lanka, etc which is present in my vecstore.

The information might be spread across 7–8 different chunks. In my testing, the code for the US was actually present in the fused results but got dropped because the reranker only selected the top 5 out of 15 chunks.

I was considering two possible solutions:

1. Query decomposition: Split the question into smaller queries, retrieve for each separately, and combine the results.
2. Context compression: Retrieve more chunks, then add another step before generation that extracts only the relevant information and removes unnecessary content.

But both approaches seem like they'd introduce even more latency, especially if they require additional LLM calls and i even dont know if this is the best solution or not .

How do you handle this in production RAG systems? Do you use dynamic Top-K, query decomposition, contextual compression, or something else?

I'm particularly interested in approaches that improve retrieval coverage without significantly increasing latency or context size.

Hi everyone!

I'm looking for learning resources about RAG ... especially books, but video tutorials, courses, and useful links would also be greatly appreciated.

I'm currently working on my graduation project, which is based on a RAG system. 🙃 Unfortunately, I'm struggling to find good learning resources, and without proper materials to study from, I'm finding it difficult to move forward with the project.

If you know of any free books, online documentation, or helpful video tutorials that explain RAG from the basics to more advanced concepts, could you please share them with me?

Books are my top priority, but any reliable resources would be very helpful.

This is quite urgent because my graduation project depends on this technology, and I really need to start learning and implementing it as soon as possible.

I'd sincerely appreciate any recommendations, links, or advice. Thank you so much for your help!

Title

I built an in-browser 2D/3D visualizer for Microsoft GraphRAG & Graphifyy (no backend, parses parquet locally)

Demo

Body

Whenever I ran Microsoft GraphRAG, inspecting the output parquet files was annoying. Opening them in pandas or setting up a full graph database just to sanity-check entities and communities felt like overkill.

So I put together GraphLens, a browser-based graph visualizer.

• 100% client-side: Uses hyparquet and Web Workers. No backend, no accounts, nothing leaves your machine.
• Drag & drop: Drop your GraphRAG .parquet files (entities, relationships, communities, etc.) or Graphify graph.json directly.
• 2D / 3D views: Switch between 2D canvas and 3D WebGL, with fuzzy search and raw data tables.
• One-click demo: If you don't have parquet files handy, there's a sample data button on the landing page to test it right away.

Links:

• Live Demo: https://rpdil23.github.io/GraphLens/
• GitHub (MIT): https://github.com/rpdil23/GraphLens

It's based on earlier work by Yan-Ying Liao, updated with modern deps and dual-engine graph support. Would appreciate any feedback or bug reports.

I made the thing that all you guys wanted github.com/beascocheatyler-star/scoped-rag-chatbot and it's free to download and run locally

Hey guys! I'm new here, I wanted to ask for advice or recommendations on how should I handle multiple users querying my RAG system at once? Currently, I just deployed one model in Microsoft Foundry, which is phi-4-mini-instruct. Now, I'm having a hard time in solving the concurrent multiple queries.

What are your suggestions?