I recently built a complete RAG reranking pipeline to test how different reranking strategies actually impact the final LLM output. I compared FOUR rerankers (Cross-Encoder, LLM-based, Cohere, and Jina) on the same questions over 5 payment-industry PDFs (payment regulations, PCI DSS, and central bank reports—1,507 chunks total).

For each one, I tracked the ranking before and after reranking to see exactly WHY the order changed.

What the project covers:

• The core weakness of standard vector search (embeddings & cosine similarity)
• Why we retrieve Top 20 but send only Top 5 to the LLM (tokens, latency, noise)
• Bi-encoder vs Cross-encoder: reading the question and chunk separately vs together
• LLM reranking strategies: Pointwise vs Pairwise vs Listwise
• How to use dedicated reranking APIs (Cohere and Jina)
• How to design one shared helper so all four rerankers plug into the same pipeline
• How to evaluate rerankers using before/after positions, Hit@5, and average latency

KEY TAKEAWAYS:

• Similarity ≠ Relevance: A chunk can sit close to the query in vector space and still not answer it. Vector search finds chunks about the topic. A strong reranker picks the chunk that actually answers it.
• Small cross-encoders can make things WORSE: My local cross-encoder rewarded word matching, penalized scrambled PDF text, and actually pushed both correct answer chunks out of the Top 5.
• LLMs understand context: The LLM reranker understood the question context (e.g., that "3.3" belongs to Requirement 3) and placed both answer chunks 1st and 2nd.
• Cohere was a lifesaver: It lifted an answer chunk from 15th to 1st, turning a completely missed answer into a correct, cited one.
• The Verdict: For this specific project, Cohere and Jina gave the best balance of accuracy and speed. Trade-off: cost per call and sending your documents to a third party.
• Always measure, don't guess: Run every reranker on the SAME Top 20 so differences come from the reranker alone.

TECH STACK: Python, Qdrant, gemini-embedding-2-preview, Hugging Face Transformers + PyTorch, cross-encoder/ms-marco-MiniLM-L-6-v2, LLM-based listwise reranker, Cohere & Jina APIs.

If you want to see the full code walkthrough or clone the repo to test it yourself, here are the links:

• Full Code (clone it, fork it, break it): https://github.com/saurabhkamal/RAG-ReRanker-Comparison
• Full Video Walkthrough: https://www.youtube.com/watch?v=dh3DKpYHoX8

Would love to hear.

I’ve been exploring what happens when a RAG demo has to keep working after the first successful query.

This TypeScript sample creates one durable Stateful Actor per company. Each actor:

- Embeds a handbook URL into Telnyx Storage

- Polls asynchronous indexing jobs and retries failures

- Searches the handbook before generating an answer

- Refuses to answer when the retrieval match is weak

- Records every Q&A in an embedded SQL audit log

- Reindexes the handbook automatically each night

The interesting part isn’t really the chatbot. It’s giving the entire retrieval lifecycle a durable owner: source state, indexing progress, retries, scheduled refreshes, and answer history all stay with the company’s actor.

The sample uses Telnyx Edge Compute, Stateful Actors, Embeddings, Storage, and AI Inference.

Code: https://github.com/team-telnyx/telnyx-code-examples/tree/main/policy-recall-line

The public routes are intentionally unauthenticated for the demo, so production use would need authorization, retention rules, evaluations for the similarity threshold, and better observability.

I’d be interested in feedback on the actor-per-tenant approach, especially from anyone maintaining frequently changing internal knowledge bases.

I put together Hybrid Scoped RAG:
https://github.com/beascocheatyler-star/hybrid-scoped-rag
It’s a document chatbot with two things a lot of the usual demos skip:
• Hybrid retrieval — BM25 and dense vectors run side by side, rankings fused with RRF. So you get the exact-term hits (course codes, policy windows, IDs) and the meaning-based ones.
• Hard scoping — metadata filters get applied before anything is scored. Scope it to a specific course or set of docs and the bot literally can’t see the rest of the corpus.
The refusal part is what I cared about most. The gate still mostly looks at a calibrated lexical score (sqrt of BM25 × term coverage). Dense similarity can only pull a borderline case over the line — it can’t force an answer when the lexical signal is weak. RRF decides which chunks to show the model; the score decides whether to call the model at all. That split felt important because RRF throws away magnitude, which is useful for ranking but bad for “should I answer this?”
Default setup stays pretty cheap:
• Retrieval, scoping, and the gate all run offline. No key, no network, no model download required.
• Default embedder is just character n-gram hashing (no dependencies). It’s not real semantics — mostly helps with spelling/morphology stuff pure BM25 misses. You can swap in sentence-transformers if you want actual dense vectors.
• Bunch of offline tests (~267) so you can check the retrieval and gate without ever hitting an LLM.
There’s also a /api/search endpoint that just returns the per-retriever ranks and what the gate would do, so you can look at the fusion without generating an answer.
Limitations are pretty straightforward: default embedder isn’t semantic, refusal isn’t perfect (some things land in borderline instead of hard refuse), and persistence is just JSON for now. Fine for a few hundred docs, not something you’d run at scale.
It’s more of a focused engine than a full platform — FastAPI + simple frontend, personas, citations, same “only answer from the documents you gave it” idea as my earlier BM25-only version, just with the hybrid retrieval and metadata scoping added.
If you’ve been dealing with bots that go off the rails or you want a hybrid setup that still refuses aggressively, the code and tests are there. Curious what people think of the gate threshold and how the RRF weighting feels in practice.

Hello everyone!

I’m currently working on a project where we’ve built an AI agent that operates in English, and the client now wants to expand its capabilities to German. I’ve been actively researching the best approach to make this transition while maintaining the quality and reliability of the entire system.

The challenge is that this isn’t just about translating text. We need to ensure that the digitization pipeline, retrieval pipeline, and agent pipeline continue to work effectively in German. Across the application, we have around 50–60 LLM calls, although not all of them are triggered by a single user query.

The core data provided by the client is in German. Initially, I suggested using an external translation service, such as Google Translate, to convert the data into English. However, I’m concerned that translation alone may fail to preserve cultural nuances, contextual meaning, and linguistic coherence.

This is where I’m stuck, and I’d love to hear your thoughts.

What would be the best approach to building a reliable multilingual AI system in this scenario?

- Should we translate the German data into English, process it, and translate the responses back?

- Should we make the entire pipeline multilingual, including document processing, embeddings, retrieval, and agent reasoning?

- Are there any hybrid approaches that could offer the best balance between accuracy, latency, and cost?

If you have worked on multilingual RAG systems, agentic AI, or cross-lingual information retrieval, I’d really appreciate your recommendations.

Research papers, technical blogs, and other useful resources would be especially helpful.

Thanks in advance!

I'm James Luan, CTO of Zilliz, the company behind Milvus. Milvus is an open-source vector database built to store, index, and search embeddings over unstructured data. Vector Lakebase is our direction for bringing that serving layer closer to the data lake.

Notion's search infrastructure is a useful example of a design that fits its initial workload well: many workspaces, with most queried infrequently. Making idle workspaces cheap is a sensible priority. My question is how that choice holds up as more AI features create sustained retrieval traffic.

Object storage changes the cold-query path. Before a cold index can answer, the system may need several object reads, deserialization, and index preparation. That work does not disappear by tuning the model. A user returning to a quiet workspace and a large tenant querying continuously stress different parts of the same design.

I expect local SSDs and memory to matter more as those workloads diverge. Multiple cache tiers can change the latency profile, while a design optimized around infrequent access may reach a different performance ceiling. These are architectural expectations, not claims that Notion has already encountered every failure.

There is a separate retrieval constraint when filtering follows approximate search. Narrow filters can discard much of the candidate set and reduce recall. More AI features often mean more specific conditions on page type, collaborators, or time.

The next evaluation should therefore separate cold returns, sustained tenant traffic, and selective filters. A cost improvement for the first workload does not settle the serving requirements of the other two.

https://zilliz.com/blog/notion-vector-search-next-problem?utm_source=reddit

Should I inspect restored segments instead of adding CPU to a slower RAG retriever?

I'd treat location as an eligibility rule in a location-aware RAG system. A restaurant can be a perfect semantic match and still be useless if it sits outside the delivery area. Letting the answer model repair that mismatch feels too late.

Retrieving a small semantic candidate set and filtering afterward can leave too few usable results. Increasing the candidate count may help, but it also makes the application responsible for balancing extra retrieval work against coverage.

With vector databases like Milvus, geometry filtering can participate in the search itself. An R-Tree first narrows locations by bounding rectangles; exact geometry checks then remove false matches before the remaining candidates are ranked by vector similarity. The rectangle match alone is not the eligibility decision.

I would probably test narrow polygons and addresses near their edges before celebrating average latency. A broad city-level query can hide the cases where an otherwise relevant result lies just outside the boundary.

For a small service, keeping location and embeddings together seems attractive because address changes have one retrieval record to update. The remaining work is evaluating semantic relevance inside the permitted area. I'd be interested in how others test that boundary.

I started logging the rank of the known good chunk for every question in my eval set, across dense, sparse, and fused. The surprise was the long tail. A meaningful share of correct answers come from chunks ranked 30 or worse in the first stage, which my old top 15 fusion would have dropped before the reranker ever saw them. It made me stop trusting small first stage cutoffs entirely.

Curious what other people's distributions look like. Do most of your correct chunks sit in the top 10, or do you also have a long tail? And has anyone found a principled way to pick the first stage cutoff, or is everyone just guessing a number that feels affordable?

I have been thinking when you use a API for parsing or/and extraction and you often need to validate the output against a dataset or confidence threshold and fix errors before it reaches your knowledge base or RAG pipeline whatever use case you have or prefer that the API should have its own correction interphase where all the outcomes which was below the threshold, lands and corrections can be assigned to people on the vendor API itself. what are your thoughts about it?

​

I’m developing a $400 independent retrieval assessment: two existing configurations, up to 30 queries, and reproducible findings on regressions and scoring consistency. Fixes aren’t included.

The question I’m trying to resolve is where that scope leaves too much work with the buyer.

If you’ve paid for external technical QA—or considered it and decided against it—could you share one actual example?

- What triggered the decision: a client handoff, migration, incident, or something else?

- What did the reviewer need to deliver for you to approve the expense?

- What would have stopped the purchase?

I’m particularly interested in whether preparing the evaluation data was a blocker, whether a report alone was useful, and whether you needed fixes or a retest included.

An anonymized example is enough. Please mention your role in the decision and whether the engagement was purchased or rejected.

I’m the provider researching this offer. This is a question about what buyers need; I won’t treat a reply as permission to send a sales DM.

we recently got a mini heart attack from gpt 4o mini, where it is deprecating in jan 2027, but we built our entire pipeline and normalizations according to that model, so again rework from scratch and few major changes.

so in these rapidly changing environment of models, how do we need to manage our RAG system according to different models.

if you have any idea or suggestion please let me know, and if you are also facing same issue then how you guys are managing?

Built a local scoped RAG chatbot that only answers from the docs you upload and refuses anything outside that corpus before calling the LLM. Uses your own Anthropic key, BM25 + term-coverage for the scope gate, and a persona config so the same engine can be a course TA or support bot. Design decisions were mine (AI helped with some of the code). Looking for honest feedback on whether this approach makes sense: https://github.com/beascocheatyler-star/scoped-rag-chatbot

Long context vs RAG, measured: on 1.07M tokens of contracts, hybrid search put 93% of the evidence in 16K tokens

Every frontier model now takes ~1M tokens, so we wanted numbers instead of opinions on whether retrieval is still worth it. The question: if search gets a budget of N tokens, how much of the answer does it actually find?

Setup

• 3 public datasets with the answering evidence marked as character spans: Qasper (86 papers, 439K tokens), LegalBench-RAG (100 contracts, 1.07M tokens), QMSum (35 meeting transcripts, 632K tokens)
• Hybrid search: BM25 + Qwen3-Embedding-0.6B, fused with reciprocal rank fusion
• Two chunkings: fixed 512-token windows vs ~350-token structure-aware units with a file/section header
• Score: lay the results end to end, best first, until they fill N tokens, and count the share of the marked evidence inside. No LLM in the loop: this measures what retrieval hands the model.

Recall within budget (structure-aware units, hybrid):

What stood out

• Contracts: 93% of the evidence fit in 16K tokens, 1.5% of the corpus. Every curve is flat long before 1M.
• Chunking mattered as much as budget. To reach 90% on contracts, fixed 512-token windows needed 64K-128K tokens; units with headers needed ~16K.
• Papers were hardest because their questions don't say which paper they mean ("What datasets do they use?"). A model reading every paper has the same problem.
• 54-62% of the evidence sits in the middle half of its document, which is where long-context models read worst.

Caveats: these are small corpora (35-100 documents), so read the shape of the curves rather than the levels. And for a small, stable corpus, preloading it and caching can beat RAG outright.

Full write-up with charts, the scoring code and a comparison with published long-context-vs-RAG studies: https://deployed-ai.io/blog/long-context-vs-rag/

Disclosure: this is from our blog at Deployed. Happy to answer questions about the setup.